Advice about microsd current consumption
From my experience, typical SD card current consumption is ~30mA in SPI mode and ~100mA in 4-bit SD mode. My advice would be to include the first figure with a 2x or 3x safety margin (that is, 60 or 90 mA) in your device power budget, and make sure that the rest of your circuit never consumes more than 500-60 = 440mA or 500-90 = 410mA respectively.

Actively limiting the current consumed by the SD card makes little sense, considering it consumes less that 10% of the overall power. Disabling the speaker when VBUS drops below 4.75V would make more sense. If you absolutely want to control SD card power consumption, send the IDLE command instead of cutting the power.