This paper presents a multi-state hierarchical approach for facial feature tracking. A hierarchical formulation of statistical shape models is proposed to characterize both global shape constraints of human faces and local structural details of facial components. Gabor wavelets and gray level profiles are integrated for effective and efficient representation of feature points. Furthermore, multi-state local shape models are presented to deal with shape variations of facial components. Meanwhile, face pose estimation helps improve shape constraints for the feature search. Both facial component states and feature point positions are dynamically estimated using a multi-modal tracking approach. Experimental results demonstrate that the proposed method accurately and robustly tracks facial features under different facial expressions and pose variations.