In the field of virtual reality technology, it is an important issue to construct the real 3-D virtual environment of the actual event. Furthermore various systems that can represent these environments have been proposed. This paper proposes a method to acquire 3-D information of a dynamic real environment in all directions by using a stereo omni-directional system (SOS), which we developed. This system can acquire all direction color images and stereo image pairs in real time. All cameras on this system operate simultaneously, and all images can be obtained at the same time. From the stereo image pairs, depth maps can be calculated easily with high accuracy. In the depth map, the moving region in the real environment can be extracted stably. A virtual environment of a real environment can be constructed separately for static and dynamic environment purposes.