Hand detection and tracking play important roles in human computer interaction (HCI) applications, as well as surveillance. We propose a self initializing and self correcting tracking technique that is robust to different skin color, illumination and shadow irregularities. Self initialization is achieved from a detector that has relatively high false positive rate. The detected hands are then tracked backwards and forward in time using mean shift trackers initialized at each hand to find the candidate tracks for possible objects in the test sequence. Observed tracks are merged and weighed to find the real trajectories. Simple actions can be inferred extracting each object from the scene and interpreting their locations within each frame. Extraction is possible using the color histograms of the objects built during the detection phase. We apply the technique here to simple hand tracking with good results, without the need for training for skin color.