1. A method for providing a real-time three-dimensional interactive environment, comprising the operations of: obtaining two-dimensional data values for a plurality of pixels representing a physical scene; obtaining a depth value for each pixel of the plurality of pixels using a depth sensing device, each depth value indicating a distance from a physical object in the physical scene to the depth sensing device; inserting at least one virtual object into the scene, the virtual object being computer-generated; and detecting an interaction between a physical object in the scene and the virtual object based on coordinates of the virtual object and the obtained depth values.