Text-based image search engine and content-based image retrieval (CBIR) have achieved much progress in commercial and academic community respectively. However, few attempts have been conducted to integrate the two techniques for image retrieval in web context. In this paper, based on a novel web image data model, i.e. Fine-Grained Web Image Model (FGWIM), a flexible and extensible framework for web image retrieval is proposed, which incorporates highlevel semantics and low-level visual features of Web images and supports the visual part of MPEG-7 standard. FGWIM model describes the web image data in several levels of abstraction by fine-grained and structured representation, and gives multiple choices at each level, which provides a good flexibility and extensibility for further feature extraction, similarity measurement, integration of semantic and visual features. Based on FGWIM model and the framework, a web image retrieval system prototype is implemented.