A novel head tracking driven camera mouse system, called "hMouse", is developed for manipulating hand-free perceptual user interfaces. The system consists of a robust real-time head tracker, a head pose/motion estimator, and a virtual mouse control module. For the hMouse tracker, we propose a 2D detection/tracking complementary switching strategy with an interactive loop. Based on the reliable tracking results, hMouse calculates the user?s head roll, tilt, yaw, scaling, horizontal, and vertical motion for further mouse control. Cursor position is navigated and fine tuned by calculating the relative position of tracking window in image space and the user?s head tilt or yaw rotation. After mouse cursor is navigated to the desired location, head roll rotation triggers virtual mouse button clicks. Experimental results demonstrate that hMouse succeeds under the circumstances of user jumping, extreme movement, large degree rotation, turning around, hand/object occlusion, part face out of camera shooting region, and multi-user occlusion. It provides alternative solutions for convenient device control, which encourages the application of interactive computer games, machine guidance, robot control, and machine access for disabilities and elders.