This paper presents an architecture for cognitive analysis of streaming video, in which a new module can easily be plugged in, to add to or even compete with existing functionality. This allows the implementers to focus on the key scientific issues instead of struggling with the details of the implementation. The architecture is distributed and runs independently of the underlying computer architecture and can run transparently across one or many different operating systems in a larger distributed system. This architecture focuses on several key Computer Vision issues, such as multi-level global and local control, automatic dataflow based on auto-descriptive selfregulating independent modules that come together to form a whole based on the characteristics of the individual and the needs of the system rather than a static flow diagram.