Power LED Matrix With LiPo Battery & ESP32
The way you have it set up is fine if you aren't going to light up all the LEDs at full brightness. And if you do, you're not going to break your ESP32 Dev Board the way you have it set up in your diagram (since you are not running the power THROUGH the board). The LiPo battery probably has its own little regulator on it to prevent it from being damaged.
Check your dev board specifications on how much power you actually need to supply to the Vin pin. The ESP32 chip operates between 2.3 and 3.6 volts (3.3 recommended). Your dev board regulates the 5v to provide that amount to the chip.
Most likely, your dev board will run fine on the 3.7V provided by your LiPo without the buck-boost. I use a similar setup (without a buck-boost) to demo little projects to friends. I use a ESP32U-01 dev board from EZSBC, a 64 LED Matrix from Adafruit, and a 3.7v LiPo battery.
Depending on how many LEDs are lit (and what color), you shouldn't expect the battery to last very long. And the overall brightness of your led matrix will be limited.
If you are using the FastLED Animation Library, I would recommend setting a power limit using set_max_power_in_milliwatts function and limiting it to 1500mW for starters. Again, if you are not drawing power THROUGH the ESP32 board, you're fine.