We present a novel method for joint segmentation and pixelwise classification of images, classifying each pixel in the image into one of a set of broad categories. We propose a 2-step approach for this problem, first estimating image structure through dense region segmentation, which provides initial spatial grouping (superpixels), then performing recognition by classifying each superpixel according to its features. Two types of region features are investigated: perceptual grouping features derived from neighborhood relations in the superpixel graph, and a histogram of pixel textons within the superpixel. Region classification is performed by boosting for perceptual features and histogram matching for texton features. We also introduce a novel extension of multi-class boosting: MAP estimation in the space of classifier ensemble outputs. Extensive results on aerial imagery are presented using a label vocabulary of trees, roads, vehicles, grass, shadows, and buildings. We evaluate the two methods across the categories, and compare them to the standard approach of classifying image blocks without prior segmentation. In our experiments perceptual features using multi-class boosting provide the best performance.