Raspberry PI powering a peripheral device while accessing it via USB causing PI brownout/short-circuit
Since RPi schematics are not released, this is a vague guess but might be worth a try.

It is possible (I'd even say likely) that the power delivered to the USB ports is coming from a power switch, sort of isolating the 5V pin of the RPi and the Bus voltage on the USB port. Personally I don't know if and how backpowering (see official documentation here) is possible in this case, but what I am proposing won't harm your devices in either case.

What I think is happening: Power is delivered from the 5V side to the output side of the power switch.

Simple fix: Add a diode to prevent current flow from your device to the USB cable (coming from the 5V pin).