Why connectors become smaller as frequency increases
The answer is actually modes. A coaxial connector is, electrically, essentially a short piece of coaxial waveguide.

Up to some cut-off frequency, signals are able to propagate along the waveguide in only one mode. Above the cut-off frequency, two or more modes can propagate. Since different modes propagate with different axial velocities, portions of the energy of a too-high frequency signal launched in one end of the waveguide will arrive at the other end at different times, resulting in distortion for signals containing more than one frequency.

A larger diameter would actually reduce the effect of skin effect, by providing a larger total conductive cross section (\$2 \pi r \delta\$, where \$r\$ is the radius of the center conductor, and \$\delta\$ is the skin depth at some transmission frequency), thus reducing resistive loss. You will in fact find that coaxial cable designed for higher frequencies (and typically used with 2.4 or 1.85-mm connectors) has higher loss per meter for this reason.