Directly powering a single 32x16 RGB LED matrix (without external PSU)
When powered through USB, the total current consumption is limited to 500mA by the USB interface. If you are using the power connector, the total current is limited by the on board 5V regulator to 1A.

The board will also require some current, so you can draw around 400mA, if powered through USB or 900mA, if powered through the connector; from the 5V pin.

While the matrix on average may not draw than 300 - 400mA, the voltage regulator can overheat and shut off temporarily, and you will never be able to reach 900mA. The regulator is linear, so any extra voltage you supply will result in waste heat.

It is best to get a buck converter or power the matrix externally. You won't be wasting as much power, and avoiding possible thermal shutoff.