Semantic video content extraction and selection are critical steps in sports video analysis and editing. The identification of video segments can be from various semantic perspec-tives, e.g. certain event, player or emotional state. In this paper, we examined the possibility of automatically identifying shots with "happy" or "sad" emotion from broadcast sports video. Our proposed model first performs the sports highlight extraction to obtain candidate shots that possibly contain emotion information and then classifies these shots into either "happy" or "sad" emotion groups using Hidden Markov Model based method. The final experimental results are satisfactory.