This paper describes a face detection and recognition system in color image sequences with a novel scheme to model skin color in the RGB color-space using neural networks. In our approach, there are no limitations regarding human skin color. This method eliminates the difficulty of describing non-skin samples by approximating non-skin color from skin samples in the VLSI Systems Laboratory skin database. The neural network algorithm based face detection is performed by using a multilayer feed-forward neural network trained with back-propagation learning algorithm in conjunction with a modular approach utilizing the distance based learning for reducing the structural complexity of the network by analyzing each frame in the video sequence. The recognition is performed based on Composite Principal Component Analysis (CPCA) algorithm. This algorithm is better equipped to recognize faces under the conditions of varying illumination and pose compared to the conventional PCA. The system is capable of detecting and recognizing faces at the rate of 10 frames per second when the frame resolution is 320 ? 240 and the color depth is 24-bit.