Photogrammetry using pure rotation?
This is not a problem specific to Visual SFM, but a general technical problem. The software cannot accurately determine the depth of the pieces which approach perpendicularity with the camera lens. The software uses those varying measurements and estimated angles to calculate the geometry. Check out the law of cosines.