Human faces are commonly found in video streams and provide useful information for video content analysis. This paper presents a robust face tracking system to extract multiple face sequences from MPEG video without human intervention. Specifically, a view-based DCT-domain face detection algorithm is first applied periodically to capture mostly frontal and slight slanting faces of variable sizes and locations. The face tracker then searches the target faces in local areas across frames in both the forward and backward directions. The tracking combines color histogram matching and skin-color adaptation to provide robust tracking. This paper focuses on developing effective techniques that exploits the features in DCT domain and the characteristics of video compression standards like MPEGs. The effectiveness of the algorithm is demonstrated using a range of videos obtained from multiple sources like the news and movies.