Most eye trackers based on active IR illumination requir distinctive bright pupil effect to work well. However, due to a variety of factors such as eye closure, eye occlusion, and external illuminations interference, pupils are not bright enough for these methods to work well. This tends to significantly limit their scope of application. In this paper, we present a new real time eye tracking methodology that works under variable and realistic lighting conditions and various face orientations. By combining the conventional appearance based object recognition method (SVM) and object tracking method (mean shift) with Kalaman filtering based on active IR illumination, our technique is able to benefit from the strengths of different techniques and overcome their respective limitations. Experimental study shows significant improvement of our technique over the existing techniques.