Among head gestures, nodding and head-shaking are very common and used often. Thus the detection of such gestures is basic to a visual understanding of human responses. However it is difficult to detect them in real-time, because nodding and head-shaking are fairly small and fast head movements.In this paper, we propose an approach for detecting nodding and head-shaking in real time from a single color video stream by directly detecting and tracking a point between the eyes, or what we call the "between-eyes". Along a circle of a certain radius centered at the "between-eyes", the pixel value has two cycles of bright parts (forehead and nose bridge) and dark parts (eyes and brows). The output of the proposed circle-frequency filter has a local maximum at these characteristic points. To distinguish the true "between-eyes" from similar characteristic points in other face parts, we do a confirmation with eye detection.Once the "between-eyes" is detected, a small area around it is copied as a template and the system enters the tracking mode. Combining with the circle-frequency filtering and the template, the tracking is done not by searching around but by selecting candidates using the template; the template is then updated. Due to this special tracking algorithm, the system can track the "between-eyes" stably and accurately. It runs at 13 frames/sec rate without special hardware. By analyzing the movement of the point, we can detect nodding and head-shaking. Some experimental results are shown.