1. A method for interacting with a virtual object in an augmented video stream, comprising: displaying an augmented video stream having a three-dimensional virtual object inserted in a given physical location of a video scene; receiving a request for interaction with the virtual object from a viewer; changing the appearance of the virtual object at the given physical location based on metadata associated with the virtual object and the request for interaction from the viewer.