We present an approach for persistent tracking of moving objects observed by non-overlapping and moving cameras. Our approach robustly recovers the geometry of non-overlapping views using a moving camera that pans across the scene. We address the tracking problem by modeling the appearance and motion of the moving regions. The appearance of the detected blobs is described by multiple spatial distributions models of blobs' colors and edges. This representation is invariant to 2D rigid and scale transformation. It provides a rich description of the detected regions, and produces an efficient blob similarity measure for tracking. The motion model is obtained using a Kalman Filter (KF) process, which predicts the position of the moving objects while taking into account the camera motion. Tracking is performed by the maximization of a joint probability model combining objects' appearance and motion. The novelty of our approach consists in defining a spatio-temporal Joint Probability Data Association Filter (JPDAF) for integrating multiple cues. The proposed method tracks a large number of moving people with partial and total occlusions and provides automatic handoff of tracked objects. We demonstrate the performance of the system on several real video surveillance sequences.