1. A method for interacting with a virtual object in an augmented video stream, comprising: displaying an augmented video stream having a three-dimensional virtual object inserted in a given physical location of a video scene; receiving a request for interaction with the virtual object from a viewer; changing the appearance of the virtual object at the given physical location based on metadata associated with the virtual object and the request for interaction from the viewer, wherein the changing the appearance includes switching from a first predetermined metadata representing the virtual object to a second predetermined metadata representing a different virtual object.