1. A method comprising: extracting one or more features from a patch in an image of a scene; generating an initial prediction of a material category of the patch from candidate material categories based on the one or more features; identifying a scene in the image; identifying one or more objects in the image; obtaining a first relationship model of two or more of the one or more features, the candidate material categories, the one or more objects, and the scene; and generating a refined prediction of the material category of the patch based on the initial prediction of the material category and on the first relationship model.