1. In a computing environment, a method performed at least in part on at least one processor, comprising: generating, by an image understanding mechanism implemented on the at least one processor, a scene representation for a query image, including: searching scenes corresponding to a set of sample images to identify one or more objects from one or more of the set of sample images for comparing with a scene expressed by the query image; and transforming the query image and the one or more objects into semantic segments comprising at least a portion of the scene representation, wherein the semantic segments provide a set of relationships between the one or more objects and between the one or more objects and the scene expressed by the query image, including relative depth between the one or more objects and a depth layer or position at which at least one object is inserted into the scene expressed by the query image, wherein transforming the query image and the one or more objects further comprises: generating a synthesized image using the one or more objects, the depth layer of the at least one inserted object in the synthesized image determined by using the semantic segments and relative depth information to match a depth distribution of objects in layers above and below the at least one inserted object in a corresponding sample image to the depth distribution of objects in layers above and below the at least one inserted object in the synthesized image.