Tracking multiple people and their behaviors is a fundamental task for visually mediated interaction. We present VIGOUR, a platform for simultaneously tracking of multiple people and recognition of their behaviors for high-level interpretation. Through perceptual integration, different types of visual information are fused to combine the benefits of different techniques. Robust low-level visual cues such as skin color and motion are used to focus attention and facilitate real-time tracking. VIGOUR tracks behaviors using gestures and head pose to produce a high-level behavior representation for subsequent interpretation. The system is able to track three people and recognize their gestures simultaneously in real time.