How to terminate a coax connection into a receiver for best common-mode rejection?
The transformer is there to convert the input signal &mdash; regardless of whether it's balanced or unbalanced &mdash; into a true differential signal to drive the next stage. Note that the drawing is confusing: The two secondary windings need to be wired out-of-phase, not in-phase as a naive interpretation would imply.

This transformer also supplies the common-mode rejection that you are looking for. If you are using an unbalanced (e.g., BNC) antenna connection, you can simply connect the ANT RETURN signal to the shield of the connector and the chassis ground.

Note that the circuit ground (the one connected to the capacitor on the secondary side) should be connected to the chassis ground at only a single point, and preferably through a ferrite bead. This will keep any common-mode external RF currents out of the rest of the receiver circuitry.