In this paper, we propose a novel method for building maps of unknown environment based on relationship between human actions and objects by a mobile robot. Most of previous works on map-building tasks by robots focused on generating obstacle maps for path-planning. In addition, model-based object recognition techniques are also used for searching particular objects. It is, however, difficult in reality to prepare a lot of models in advance for recognizing various objects in unknown environment. On the other hand, human can often recognize objects not from their appearances but by watching other person taking actions on them. This is because the function and/or the usage of the objects are closely related with human actions. Therefore, we introduce conceptual models of human actions and objects for classifying objects by observing human activities. Finally, we demonstrate a map can be built in unknown environment.