We achieve two goals in this paper: (1) to build a novel appearance-based object representation that takes into account variations in contrast often found in training images; (2) to develop a robust appearance-based detection scheme that can handle outliers such as occlusion and structured noise. To build the representation, we decompose the input ensemble into two subspaces: a principal subspace (within-subspace) and its orthogonal complement (out-of-subspace). Before computing the principal subspace, we remove any dependency on contrast that the training set might exhibit. To account for pixel outliers in test images, we model the residual signal in the out-of-subspace by a probabilistic mixture model of an inlier distribution and a uniform outlier distribution. The mixture model, in turn, facilitates the robust estimation of the within-subspace coefficients. We show our methodology leads to an effective classifier for separating images of eyes from non-eyes extracted from the FERET dataset.