Why do ESP boards break out the SPI flash pins?
Considering the very limited and difficult use of the SPI Flash pins, why do ESP boards break these out to the user? Why aren't they just kept internal, making the board a little smaller?


The ESP32 is not designed specifically for Arduino. There are other boot modes, such as SD card boot, whereby the internal flash is not directly used. In this situation you are free to use those pins as you wish (given that there is a flash chip attached to them).

The "ESP32 Development Board" was not primarily designed for Arduino. Instead it's a generic board designed by Espressif (and copied many times), and is usable with the Arduino core as one of many options.

In short: the pins aren't there for Arduino users, they're there for other users.



Addendum:

The SPI flash pins can also be used for (and are most commonly used for):


Adding more SPI flash chips which then get integrated into the normal memory space and with the internal caching system, and
Adding PSRAM to increase the RAM memory space.


A good example of the latter is the ESP32-CAM which has an SPI-connected PSRAM chip for framebuffering and image analysis connected to the SPI pins.

You can think of them not so much as SPI pins but as the EBI pins of the PIC32MZ or the EMB of the ATMega2560, etc. Used for expanding the memory rather than for adding random peripherals.