A new approach to tracking weakly modeled objects in a semantically rich domain is presented. We define a closed-world as a space-time region of an image sequence in which the complete taxonomy of objects is known, and in which each pixel should be explained as belonging to one of those objects. Given contextual object information, context-specific features can be dynamically selected as the basis for tracking. A context-specific feature is one that has been chosen based upon the context to maximize the chance of successful tracking between frames. Our work is motivated by the goal of video annotation-the semi-automatic generation of symbolic descriptions of action taking place in a contextually-rich dynamic scene. We describe how contextual knowledge in the "football domain" can be applied to closed-world football player tracking and present the details of our implementation. We include tracking results based on hundreds of images that demonstrate the wide range of tracking situations the algorithm successfully handles as well as a few examples of where the algorithm fails.