1. A computer-implemented method comprising: utilizing two or more classifiers to calculate, for an input image, probability scores for a plurality of classes based on visual information extracted from the input image and semantic relationships in a classification hierarchy, wherein each of the two or more classifiers is associated with a given one of two or more levels in the classification hierarchy with each level in the classification hierarchy comprising a subset of the plurality of classes; classifying the input image based on the calculated probability scores; training the two or more classifiers to calculate probability scores for respective subset of the plurality of classes; and performing label inference to refine classification probabilities in the two or more classifiers based on semantic relationships in the classification hierarchy; wherein training the two or more classifiers and performing label inference comprises at least one of: taking as input a graph structure having initial values for nodes corresponding to classification probabilities in the two or more classifier and outputting the graph structure with modified values for the nodes; utilizing a multi-task learning based loss function that jointly optimizes classifiers associated with each of the two or more levels in the classification hierarchy; and utilizing a random walk process that smooths classification probabilities over two or more classes in a same semantic path in the classification hierarchy.