This paper presents a framework of a system for the query and retrieval of video data based on video events in huge video repositories. The events are formulated using domain-independent event primitivies which are represented by spatio-temporal relationships between objects in the video scenes. Complex events are expressible as combinations of simpler events. This facilitates support of event queries from a variety of points of view. In addition, the framework is expected to provide adaptability of the framework to multiple domains.