1. A computer-implemented process for constructing shared virtual environments, comprising: applying a computer to perform process actions for: generating a real-time environmental model of a real-world environment in which a shared immersive virtual environment is being presented to two or more users; generating a unified real-time tracking model for each of the two or more users; for each particular user, applying the real-time environmental model and the real-time tracking model to generate frames of the shared immersive virtual environment corresponding to a real-time field of view of the particular user; rendering virtual representations of one or more non-participant persons into the shared immersive virtual environment in positions corresponding to a real-world position of each non-participant person; and presenting the virtual representation of one or more of the non-participant persons as a translucent rendering that increases in solidity whenever it is determined that the non-participant person is speaking to any user.