1. A method for classifying an image, comprising: inferring location information of an object of interest in an input representation of the image; determining foreground object features and background object features from the input representation of the image; pooling the foreground object features separately from the background object features using the location information to form a new representation of the image, the new representation being different than the input representation of the image; and classifying the image based on the new representation of the image.