Different isolation when switching port calibration type
According to this Application Note from Keysight, the difference between those two calibrations is as follows...

TML:


  The TML port calibration (known as Single Mode in earlier versions
  of ADS) is used when a transmission line will be connected to the
  port. For this port calibration type, Momentum extends the
  transmission line (or any edge of the layout in contact with the
  port). A calibration standard (a transmission line of twice the
  length) is simulated to de-embed the effect introduced by the
  transmission line. This calibration technique reduces the impact of
  higher-order modes around the source and adds the mutual coupling
  effect of the transmission line current with the circuit. It is
  recommended when the circuit is fed through a long transmission
  line(s).


None:


  There is also the option to use uncalibrated ports in Momentum. These
  ports were previously called Internal ports in older versions of ADS,
  but are now simply called None. Uncalibrated ports can be placed
  anywhere within the circuit (not limited to boundaries between
  dielectrics and conductors) and may excite a single point, edge or
  area. Uncalibrated ports may be de ned using a single pin referenced
  to implicit ground (Gnd), or by using multiple positive pins and/or
  multiple negative pins. Sometimes, uncalibrated ports are the best
  choice for modeling how devices are connected to a circuit.


So it seems that each calibration will set different boundary conditions in the interfaces (ports) of your coupler, and that's why you're getting different results.

Just select the calibration type that better suits how the coupler will actually be integrated in the rest of your circuit. If you plan to access the ports of your coupler through transmission lines, the results of the TML calibration may be more accurate and correlate better with actual measurements when you build it.