Projective texturing from many images
Apologies in advance for the poor quality of this answer, but this sounds a little like what an ex-colleague was doing as part of his PhD. The "Free-Viewpoint video" papers listed at the bottom of https://www.researchgate.net/profile/James_Imber might be useful, or at least a starting point to find related work.