A rendering algorithm based on physical phenomenon and well-defined model data are needed to generate a synthetic image realistically. In this paper, inverse rendering based on an environment map is proposed to obtain BRDF of objects. Many researchers try to find BRDF by analyzing many pictures taken with well-controlled camera and light position. But this method is available at a laboratory environment. Limits of previous works can be conquered by using Inverse rendering based on an environment map and it is possible to get BRDF of objects at a general environment. Inputs are an image, an environment map and 3D geometry data of models. Objects of an input image are divided into three parts — diffuse reflection, glossy reflection and diffuse texture — and BRDF of objects are recovered. Using BRDF of objects, it is possible to add virtual objects and to remove real objects from an input image.