A new method for retrieving video segments from a digital video database is proposed. A unique signature for each video segment is generated from the analysis and tracking of interesting objects between each frame of the video region. The MPEG-7 descriptors [Introduction to MPEG-7] consisting of the edge histogram, homogenous texture, dominant colors and color structure (188 features in all) are extracted from corresponding objects in successive frames for the duration of the considered sequence. The average of the MPEG-7 descriptors is computed for each object across all frames in the region thus resulting in a unique signature for the region. The stored signature which best matches this new signature (based on the earth mover distance [The Earth's Mover's Distance, Multi-Dimensional Scaling, and Color-Based Image Retrieval] between the signatures) is used for retrieving the video segment.