It has been shown that integrating multiple cues will increase the reliability and robustness of a vision system in situations that no single cue is reliable. In this paper, we propose a method by fusing multiple cues (i.e., the color cue and the edge cue). In contrast to previous work, we propose a novel shape similarity measure which includes the spatial distribution of, the number of, and the gradient intensity of the edge points. We integrate this shape similarity measure with our recently proposed SMOG-based color similarity measure in the framework of particle filter (PF). Experimental results demonstrate the high robustness and effectiveness of our method in handling appearance changes, cluttered background, moving camera, and occlusions.