12. A computer program product comprising a computer readable storage medium for storing computer readable program code which, when executed, causes a computer: to utilize two or more classifiers to calculate, for an input image, probability scores for a plurality of classes based on visual information extracted from the input image and semantic relationships in a classification hierarchy, wherein each of the two or more classifiers is associated with a given one of two or more levels in the classification hierarchy with each level in the classification hierarchy comprising a subset of the plurality of classes; to classify the input image based on the calculated probability scores; to train the two or more classifiers to calculate probability scores for respective subset of the plurality of classes; and to perform label inference to refine classification probabilities in the two or more classifiers based on semantic relationships in the classification hierarchy; wherein training the two or more classifiers and performing label inference comprises at least one of: taking as input a graph structure having initial values for nodes corresponding to classification probabilities in the two or more classifier and outputting the graph structure with modified values for the nodes; utilizing a multi-task learning based loss function that jointly optimizes classifiers associated with each of the two or more levels in the classification hierarchy; and utilizing a random walk process that smooths classification probabilities over two or more classes in a same semantic path in the classification hierarchy.
17. The computer program product of claim 12 , wherein the classification hierarchy comprises a tree structure and performing label inference to refine the classification probabilities in the two or more classifiers comprises, for two or more parent nodes having classification probabilities within a designated threshold from one another, adjusting the classification probabilities for the two or more parent nodes based on the classification probabilities for two or more leaf nodes corresponding to the two or more parent nodes, wherein the two or more parent nodes are in a higher level of the classification hierarchy relative to the two or more leaf nodes.