This paper presents Olympus, a modular processing architecture for a distributed ambient intelligence. The system is aimed at detailed reporting of people wandering and gesturing in complex indoor environments. The design of the architecture has been driven by two main principles: reliable algorithm testing and system scalability. The first goal has been achieved through the development of Zeus, a real time 3D rendering engine that provides simulated sensory inputs supported by automatically generated ground truth for performance evaluation. The rendering engine is supported by Cronos, a flexible tool for the synthesis of choreographed motion of people visiting museums, based on modified force fields. Scalability has been achieved by developing Hermes, a modular architecture for multi-platform video grabbing, MPEG4 compression, stream delivery and processing using a LAN as a distributed processing environment. A set of processing modules has been developed to increase the realism of generated synthetic images which have been used to develop and evaluate algorithms for people detection.