Good strategies for camera matching
What you are doing should be pretty easy, you should be using the fspy blender plugin BLAM.  Blenderguru did a video using a similar plugin to model a hallway:  https://www.youtube.com/watch?v=nb6rSMAooDs&amp;t=160s

Hopefully you just need to tweak your workflow a bit.