1. A computer-implemented method, comprising: receiving, at a server computer system, digital image data captured by a mobile device; performing image recognition analysis on an object within the digital image data to identify the object; receiving, at the server computer system, sensor data that describes a change in orientation executed by the user device; determining based on the identity of the object, the image analysis, and the sensor data, a gesture associated with the object; querying a semantic associations database based on the determined gesture and the object to determine a secondary meaning associated with the gesture and the object, wherein the secondary meaning is a non-literal meaning associated with the determined identity of the object based on the gesture, wherein the non-literal meaning is inferred from descriptions associated with a plurality of images located on a computer network that depict objects with the determined identity, wherein the descriptions each include one or more references to the non-literal meaning associated with the gesture; in response to the determination of the secondary meaning, requesting augmentation data from a remote service based on the secondary meaning; receiving the augmentation data from the remote service in response to the request, wherein the augmentation data is relevant to the secondary meaning associated with the identified object, and wherein the augmentation data is to be rendered over the identified object in the digital image data when displayed by the mobile device; and transmitting augmentation data to the mobile device that is semantically relevant to the object based on the secondary meaning.