How can I know the phase shift or phase delay of the SMA connector and RF port?
At 20 kHz, the phase shift of the SMA connector is negligible. The length of the signal path is maybe 5 mm. The dielectric constant of the PTFE (Teflon) material in the SMA connector is about 2.0. So the delay through the connector is about 23 ps. That's about 20 microradians of phase delay at 20 kHz.

Even at higher frequencies, the delay through the SMA connector is probably quite small compared to the delay through the cable that's connected to the connector, which you've said nothing about.

If you were working at a higher frequency where 20 ps made a difference, then you'd have a problem. Because the delay induced by a connector like this depends on the footprint on the PCB where you mount it as well as the construction of the connector itself. And because the two sides of the connector aren't the same type of waveguide. 

The best way to determine the characteristics of the connector would probably be to build two additional "test coupons" onto your pcb. Each coupon would be a length of trace with a connector on each end. The two traces would be different lengths. By measuring the S-parameters of the two coupons with a network analyzer, you'd have enough information to cancel out the affect of the traces and determine the characteristics of the connector.