1. A method at a 3D environment modeling system comprising: at least one of accessing or generating a coarse 3D model of an environment, the coarse 3D model of the environment being generated in a 3D volume made up of a plurality of voxels and corresponding images stored on a memory device; receiving a first image of the environment captured, under specified lighting conditions, by a mobile environment capture device moved in an unplanned manner in the environment; obtaining at a processor, camera pose of at least one camera at the mobile environment capture device used to capture the received first image; determining whether a location in the memory device is available for the first image; based on a determination that the location is not available, determining whether the first image is higher quality than a second image stored in the memory device for one of the plurality of voxels; based on determining that the first image is higher quality than the second image stored in the memory device, replacing the second image stored in the memory device with the first image; calculating a representation of fine detail of surfaces in the environment, the representation of fine detail comprising at least one true color of the environment, the calculation being based at least in part on at least one surface normal of at least one element in the environment, the first image, camera pose, and lighting conditions; refining the coarse 3D model based at least in part on the representation of fine detail; and causing a projection of at least a portion of the refined 3D model into real space.