In this paper we develop a first step towards the recognition of hand activity by detecting objects subject to manipulation, and use the results to build a visual summary of events. The motivation is to extract information from hand activity without requiring that the wearer is explicit as in gesture-based interaction. Our method uses simple image measurements within a probabilistic framework and allows real-time implementation.