We present spectral histogram representations for visual modeling. Based on a generative process, the representation is derived by partitioning the frequency domain into small disjoint regions and assuming independence among the regions. This gives rise to a set of filters and a representation consisting of marginal distributions of those filter responses. A distinct advantage of our representation is that it can be effectively used for different classification and recognition tasks, which is demonstrated by experiments and comparisons in texture classification, face recognition, and appearance-based 3D object recognition. The marked improvement over existing methods justifies our principle that effective a priori knowledge should be derived from physical generative processes.