While the geometric aspects of structure and motion estimation from uncalibrated images are well understood, and it has great promise in applications, it has not seen widespread use. In this paper we combine SSD tracking with incremental structure computation into a system computing both motion and structure on-line from video. We show how in combination the structure estimation and tracking benefit each other, resulting in both better structure and more robust tracking. Particularly, through the 3D structure, our method can manage visibility constraints, add new image patches to track as they come into view and remove ones that are occluded or fail. This allows tracking over larger pose variations than possible with conventional SSD tracking (e.g. going around an object or scene where new parts come into view.) Experiments demonstrate tracking and capture of a scene from a camera trajectory covering different sides without mutual visibility.