Abstract: We develop a two-stage framework for parsing and understanding images: a process of image segmentation - grouping pixels to form regions of coherent color and texture, and a process of recognition - comparing assemblies of such regions, hypothesized to correspond to a single object, with views of stored prototypes. We treat segmenting images into regions as an optimization problem: partition the image into regions such that there is high similarity within a region and low similarity across regions. This is formalized as the minimization of the normalized cut between regions. Using ideas from spectral graph theory, the minimization can be set up as an eigenvalue problem. Visual attributes such as color, texture, contour and motion are encoded in this framework by suitable specification of graph edge weights. The recognition problem requires us to compare assemblies of image regions with previously stored proto-typical views of known objects. We have devised a novel algorithm for shape matching based on a relational descriptor called the shape context. This enables us to compute similarity measures between shapes which, together with similarity measures for texture and color, can be used for object recognition. The shape matching algorithm has yielded excellent results on a variety of different 2D and 3D recognition problems. This paper draws on joint work with Serge Belongie, Thomas Leung, Greg Mori, Jan Puzicha and Jianbo Shi.