Rectangular Waveguide H-Plane Step Discontinuity - Equivalent Reactance for Change in Height
This method actually does not apply in this kind of discontinuity. See Pozar Problem 4.1, which directly shows that using this type of modal analysis results in a nonsensical result of having zero reflection due to the height change. Instead, a different kind of analysis needs to be done.

Refer to Marcuvitz v10, p308, section 5-26 - change in height of rectangular guide for approximate equations. Still no explanation for how the analysis is done, but good enough for me.