To recognize and retrieve soccer game scenes, the movement of the players and the ball must be analyzed. This paper describes a method of tracking the players and the ball and estimating their 3D positions from video images recorded from TV broadcasting. Our system detects the players and tracks them by predicting their motions. Our system determines the ball from among ball candidates by considering the motion continuity of the ball. The camera parameters (pan, tilt, and zoom) are estimated by detecting line regions on the field in every frame and by matching them to the model. Because our method uses not only straight lines but also circular ones for the matching, it can be applied to the whole area of the field. Our system also estimates the 3D position of the ball by fitting a 3D physical model to the observed ball trajectory. The effectiveness of the method is shown by experiments with actual image sequences.