This article presents a methodology to build efficient real-time semantic video understanding systems addressing real world problems. In our case, semantic video under- standing consists in the recognition of predefined scenario models in a given application domain starting from a pixel analysis up to a symbolic description of what is happening in the scene viewed by cameras. This methodology proposes to use evaluation to acquire knowledge of programs and to represent this knowledge with appropriate formalisms. First, to obtain efficiency, a formalism enables to model video processing programs and their associated parameter adaptation rules. These rules are written by experts after performing a technical evaluation. Second, a scenario for- malism enables experts to model their needs and to easily refine their scenario models to adapt them to real-life situa- tions. This refinement is performed with an end-user evalu- ation. This second part ensures that systems match end-user expectations. Results are reported for scenario recognition performances on real video sequences taken from a bank agency monitoring application.