Automatic detection of semantic events in sport videos is a challenging task. In this paper, we propose a multimodal multilayer statistical inference framework for semantic sports video analysis using Dynamic Bayesian Networks (DBNs). Based on this framework, three instances including factorial hierarchical hidden Markov model (FHHMM), coupled hierarchical hidden Markov model (CHHMM), and product hierarchical hidden Markov model (PHHMM), are constructed and compared. Play-break detection in soccer videos is used as a testbed with hierarchical hidden Markov model (HHMM) as a baseline. Experimental results indicate the superior capability of the PHHMM, because it not only effectively models dynamic interactions between different modalities, but also sufficiently utilizes context constraints in multilayer structures.