We describe an architecture for implementing scene understanding algorithms in the visual surveillance domain. To achieve a high level description of events observed by multiple cameras, many inter-related, event-driven processes must be executed. We use the agent paradigm to provide a framework in which these processes can be managed. Each camera has an associated agent, which detects and tracks moving regions of interest. This is used to construct and update object agents. Each camera is calibrated so that image co-ordinates can be transformed into ground plane locations. By comparing properties, two object agents can infer that they have the same referent, i.e. that two cameras are observing the same entity, and as a consequence merge identities. Each object's trajectory is classified with a type of activity, with reference to a ground plane agent. This agent stores a hidden Markov model of learned activity patterns. We demonstrate objects simultaneously tracked in two cameras, which infer this shared observation. The combination of the agent framework, and visual surveillance application provides an excellent environment for development and evaluation of scene understanding algorithms.