1. A computer-implemented method for training a classification model for determining labels for an input image, the method comprising: receiving, by one or more processors, a plurality of labeled or partially labeled training images, wherein each labeled or partially labeled training image comprises at least one label associated with a pixel in the training image, the label indicating an object class from a plurality of possible object classes associated with the pixel; training, by the one or more processors, a global classification model to learn a global mapping between features of the training images and the labels; clustering, by the one or more processors, the plurality of labeled training images based on label-based descriptors into a plurality of clusters, the label-based descriptors representing label-based image features; training, by the one or more processors, a plurality of local classification models to learn a local mapping between features of the training images in each cluster and the labels of the training images in the cluster; and storing, by the one or more processors, the global classification model and the plurality of local classification models to a storage medium; wherein training the global classification model comprises: segmenting each training image into a plurality of super-pixels, each super-pixel comprising a spatially-contiguous block of pixels; determining the features of each training image from appearance characteristics of the super-pixels; and training the global classification model based on a set of feature vectors representing the features; and wherein determining the features of each training image comprises: determining a unary feature vector for each of the super-pixels in each of the training images, each unary feature vector representing one or both of location of a center of the super-pixel and gradient information of the super-pixel; determining pair-wise feature vectors representing relative appearance of neighboring super-pixels; and including the unary feature vectors and the pair-wise feature vectors in the set of features vectors for training the global classification model.