The problem of automatic sports video classification is considered. We develop a multistage decision making system that is founded on the concept of cues, i.e. pieces of visual evidence, characteristic of certain categories of sports that are extracted from key frames. The main decision making mechanism is a decision tree which generate hypotheses concerning the semantics of the sports video content. The final stage of the decision making process is a Hidden Markov Model system which bridges the gap between the semantic content categorisation defined by the user and the actual visual content categories. The latter is often ambiguous, as the same visual content may be attributed to different sport categories, depending on the context. We demonstrate experimentally that the contextual post-processing of the decision tree outputs by HMMs significantly improves the performance of the sports video classification system.