This paper presents a 3D scene reconstruction method, based on space carving, that works with a hand-held camera. In our system, the intrinsic and extrinsic parameters of the camera are determined at the moment of image capture, as opposed to other systems that rely on fixed pre-calibrated camera setups. In order to do this we place a special calibration pattern in the scene in such a way that it does not alter scene visibility. However, the calibration pattern may be partially occluded by the objects of interest in the scene. This has led us to adopt a calibration method based on model recognition. Scene reconstruction is obtained from the set of input images by an adaptive space-carving algorithm that uses not only photometric information but also segmentation information. The segmentation information of a given input image is determined by a robust statistical test based on an approximate model of the scene's background. Such model is computed from a set of images of the scene's background that are warped in such a way that they match the geometry of the desired camera.