1. A computer-implemented method for obtaining labels for an input image, the method comprising: receiving, by one or more processors, a global classification model and a plurality of local classification models for determining image labels representing image content; receiving, by the one or more processors, the input image for labeling; segmenting the input image into a plurality of super-pixels, each super-pixel comprising a spatially-contiguous block of pixels; and determining the features of the input image from appearance characteristics of the super-pixels by: determining a unary feature vector for each of the super-pixels in the input image, each unary feature vector representing one or more of: location of a center of the super-pixel and gradient information of the super-pixel, and determining pair-wise feature vectors representing relative appearance of neighboring super-pixels; applying, by the one or more processors, the global classification model to features of the input image to determine initial labels for the input image; selecting, by the one or more processors based on the initial labels, a local classification model from the plurality of local classification models; applying, by the one or more processors, the selected local classification model to the features of the input image to determine refined labels for the input image; and storing, by the one or more processors, the refined labels in association with the input image to a storage medium; wherein the global classification model is applied to a set of feature vectors including the unary feature vectors and the pair-wise feature vectors, and parameters of the global classification model are determined by minimizing an objective function that evaluates consistency between the determined unary feature vectors and the determined pair-wise feature vectors.