ESP32 output pin - internal pull-up/pull-down resistor
An output pin that is a push-pull output is a strong output - using an internal pull resistor (down or up) makes no sense.

Edit (due to new information): Since the ESP32 pin may not be an output when it boots up, it can be a floating input so the buffer output state is not known. Obviously to keep buffer output state stable and known during ESP32 boot, you need a pull-down (or pull-up) at the buffer input.