Pass USB 3.0 connection through D-SUB connector
I assume this is due to insufficient shielding to get a USB 3.0 link?


It's more likely that you simply introduce an impedance break so significant by separating the conductor pairs of the superspeed conductors that communication can't properly take place.


  What is the most likely cause for this failure, and how to avoid it in the future, if possible?


As said, you can't just separate the conductors of USB3 arbitrarily: the signal is carried as electromagnetic field between the conductors; because the signal frequencies of USB3 are solidly within the microwave range, your splitting of conductor pair essentially means you break the transport of energy.

You probably won't be able to solve this situation using your current D-SUB connectors at all. You'll need to replace these connectors with ones that at least approximately retain the nominal 90Ω impedance of USB3 SS connector pairs. It's pretty likely the easiest way to achieve that is through USB3 connectors themselves.