Parallel driving multiple pairs of LEDs in series from USB supply -- is it a terrible idea?
I would not design that way to just parallel the LED strings across the supply rails. As you already observed the current drawn by LEDs themselves increases exponentially with voltage supply change. Make sure to add an appropriate sized resistor in series with each LED string to limit the current to an amount that in compliance with the ratings of the LEDs.

Not directly inferred in your question but do not try to parallel the outputs of multiple USB chargers together.