Can you create a VR video that makes use of positional tracking?
You can use photogrammetry if you had enough different viewpoints to build a 3d representation of the scene and apply all necessary texturing and lighting.  The video would then play as an animated 3d scene rather than a traditional movie.  

The amount of processing involved is high but it is possible.  Microsoft's Hyperlapse video stabilization actually does some of this by determining a point cloud of the scene from some basic photogrammetry.

Software like Destinations on the Vive already use photogrammetry for reproducing landmarks in a fully explore-able 3D world.  There are also several animated projects which are composed of scenes that you can fully explore.  8i also has done some early work in video like you describe.  The quality level still leaves a lot to be desired, but work is being done in trying to accomplish what you are talking about.  

As an early Vive owner who has tried all these things first hand, I'd say "volumetric video" is at the interesting experimental phase, but not really that great yet.  I'm sure we'll see significant improvements in the future though as the technology is refined, particularly since it has implications outside of just VR and AR for things like computer vision.