Tracking objects using multiple cues yields more robust results. The well-known hidden Markov model (HMM) provides a powerful framework to incorporate multiple cues by expanding its observation. However, a plain HMM does not capture the inter-correlation between measurements of neighboring states when computing the transition probabilities. This can seriously damage the racking performance. To overcome this difficulty, in this paper, we propose a new HMM framework targeted at contour-based object racking. A joint probability data association filter (JPDAF) is used to compute the HMM?s transition probabilities, taking into account the intercorrelated neighboring measurements. To ensure real-time performance, we have further developed an efficient method to calculate the data association probability via dynamic programming, which allows the proposed JPDAF-HMM to run comfortably a 30 frames/sec. This new tracking framework not only can easily incorporate various image cues (e.g., edge intensity, foreground region color and background region color), but also offers an on-line learning process to adapt to changes in the scene. To evaluate its tracking performance, we have applied the proposed JPDAF-HMM in various real- world video sequences. We report promising racking results in complex environments.