How to do camera tracking for a video shot in portrait mode?
Portrait

Under Solve tab check Refine &gt; Focal Length, Optical Center, Radial Distortion with those I got nice error only 0,10 px
Set Floor, Wall, Center, XY axis - with that I got camera correctly orientated

The only one thing that didn't match was camera Focal Length (and Shift).

If you select camera and go to Properties editor &gt; Object Data &gt; Camera &gt; Sensor Fit and switch to Horizontal &gt; Width 35 mm.

It does not fix the value itself, but empties and reconstructed scene match perfectly.

I still don't know why the given Focal Length is incorrect, but I'm not a tracking guru :) Alternatively you can adjust Focal Length and Shift manually according generated Empties.

Notes
How to determinate Lens Distortion
How to get better result ...