In this paper, a novel robust tracking algorithm in compressed video is proposed. Within the framework of video compression standards, we consider how to accurately estimate motion of an object by utilizing motion vectors available in compressed video together with derived confidence measures. These confidence measures are based on DCT coefficients, spatial continuity of motion and texture measure of the object. We perform tracking directly on the compressed data and also consider tracking of an object with image scale change. In order to achieve robust tracking, we develop a system which enables us to detect object appearance change such as illumination change and occlusion by exploring the confidence measures derived above. Preliminary results indicate that our tracking algorithm works well with a variety of video sequences.