Why don't we have a circular USB port?
Rotatability while the USB connection is already established is a feature you don't want:
While for audio frequencies, the electric ringing caused by imperfect contact are benign enough to not matter, I wouldn't trust that to be true for 1 Mbd (low-speed USB1), and certainly not at 12 Mbd (Full-Speed USB1.1), 480 Mb (High Speed USB2), or the gigahertzes of bandwidth that USB3.0 and later generations use.
In fact, for the latter, you need pretty well-matched connectors that are designed to conduct an RF signal instead of a current. Building such connectors even in rotatable form is, if at all possible (never saw a rotatable connector that has more than one waveguide, e.g. coax connection), extremely expensive. Think of &gt; 20€ per connector, even in larger quantities, and, due to physics, probably not very compact.
I was surprised to find out there's actually USB3 slip rings! They look like this: