As bearer of high level semantics, audio signal is being more and more used in content-based multimedia retrieval. In this paper, we investigate the ball hit detection for sports games and propose a novel approach to detect ball hits. By employing Energy Peak Detection (EPD) and Mel Frequency Cepstral Coefficient-based (MFCC-based) Refinement (MBR), high precision (91%) and adequate recall (73%) of ball hit detection are achieved with a low computational complexity and an easy training process. The proposed algorithm can be applied in audio content-based highlight detection systems and provide valuable information for semantical understanding of sports games.