Kernel-based tracking approaches have proven to be more efficient in computation compared to other tracking approaches such as particle filtering. However, existing kernel-based tracking approaches suffer from the well-known "singularity" problem. In this paper, we propose a novel object tracking framework to handle this problem by using a control-based observer design. Specifically, we formulate object tracking as a recursive inverse problem, thus unifying several approaches to video tracking, including kernel-based tracking, into a consistent theoretical framework. Moreover, we interpret the inverse equation as a measurement process and supplement it by introducing state dynamics as a constraint. The augmented recursive inverse equation forms a state-space model, which is solved by using a control-based optimal observer. By exploiting observability theory from control engineering, we extend the current approach to kernel-based tracking and provide explicit criteria for kernel design and dynamics evaluation. The tracking performance of our approach has been demonstrated on both synthetic and real-world video data.