This paper considers the problem of designing a real-time surveillance system, equipped with a single camera on a pan/tilt platform, to track multiple moving targets within the camera's field of regard. The objective is to maintain motion trajectory information of as many of these targets as possible, and for as long as possible. Because the camera can only capture a fraction of the field of regard at any one time, it may not be possible to track all targets of interest at once. The problem can hence be viewed as one of time-sharinga scarce resource (the camera) among multiple contending users (the targets).For this, we propose a system architecture that consists of two modules: one that handles the high-level time-allocation (target scheduling) issue within a queuing theory framework, and another that handles low-level localized detection and tracking of a target, based on principles of recursive filtering (estimation) and feedback control. Effective operation of the system relies on ability to accurately determine the two key parameters for each target: how much tracking time to allocate to it, and how often to re-schedule it.