Inductors for local power supply filtering
The obvious solution is to use a linear Low-Drop Out regulator (LDO) to feed both the Vdd opins on the op-amps and comparitors as well as any reference voltages needed.

You will continue to power your load directly from the Vusb supply but the op-amps &amp; comparitors should be nicely insulated from noise on the USB supply.

The total current that the LDO needs to supply will be very low so long as it is not required to power the load.