Interpreting video is a challenging problem in Computer Vision with promising applications, such as video surveillance and indexing. The focus of this paper is determining if a scenario occurs in a video taken from a moving airplane. Our paradigm for scenario recognition uses Dynamic Belief Networks (DBNs) in a hierarchical fashion. DBNs provide a method for propagating statistical information over time. Larger scenarios are made up of smaller scenarios and actions. DBNs are ideal for situations where prior knowledge is available about the scenarios of interest. This prior knowledge is encoded in the structure of the network. The statistical parameters of the network can either be specified by the user or learned from input sequences.