1. A method for manipulating a first virtual object in a virtual space, the first virtual object being associated with a physical object in a first physical space, the physical object being separate from a first user, comprising: receiving first depth image data indicative of the first physical space; generating the first virtual object in the virtual space based on the physical object as represented in the first depth image data; in response to receiving second depth image data indicative of the first physical space where the physical object is being moved by the first user, displaying a corresponding moving of the first virtual object in the virtual space; and in response to receiving third depth image data indicative of a second user in a second physical space, the second physical space differing from the first physical space in which the physical object is located, and in response to determining that the third depth image data is indicative of the second user attempting to move a second virtual object in the virtual space and the first virtual object in the virtual space without the second user having moved the physical object in the first physical space, determining to display movement of the second virtual object and not to display movement of the first virtual object.