1. A computer-implemented process for constructing shared virtual environments, comprising: applying a computer to perform process actions for: generating a real-time environmental model of a real-world environment in which a shared immersive virtual environment is being presented to two or more users; the real-time environmental model classifying geometry, positions, and motions of surfaces and objects in the real-world environment; generating a unified real-time tracking model for each of the two or more users; for each particular user, generating frames of the shared immersive virtual environment corresponding to a real-time field of view of the particular user; jointly constraining each frame of the shared immersive virtual environment for each particular user relative to both the real-time environmental model and each of the unified real-time tracking models; and mapping and rendering one or more virtual elements of the shared immersive virtual environment to real objects in the real-world environment that can be touched and manipulated by two or more of the users to provide shared tactile interaction.