We represent local spatial structure in a color image using feature matrices that are computed from an image region. Feature matrices contain significantly more information about local image structure than previous representations. Although feature matrices are useful for surface recognition, this representation depends on the spectral properties of the scene illumination. Using a finite dimensional linear model for surface spectral reflectance with the same number of parameters as the number of color bands, we show that illumination changes correspond to linear transformations of the feature matrices and that surface rotations correspond to circular shifts of the matrices. From these relationships we derive an algorithm for illumination and geometry invariant recognition of local surface structure. We demonstrate the algorithm with a series of experiments on images of real objects.