Pattern-based augmented reality systems are considered the most promising approach for accurately registering virtual objects with real-time video feeds. The problem with existing solutions is the lack of robustness to partial occlusions of the pattern, which is important when attempting natural interactions with the virtual objects.

This paper describes a fast and accurate vision-based pattern tracking system that allows for autocalibrated 3D augmentations of virtual objects onto known planar patterns. The tracking system is shown to be robust to changes in pattern scale, orientation, and most importantly partial occlusions. A method to detect a hand over top of the pattern is then described, along with a method to render the hand on top of the virtual objects.