ESP32 with FTDI programmer
It's unclear from the image if you have set the FTDI adapter to the correct voltage 5.0V or 3.3V (it should be 3.3V).

I have experience programming bare ESP32 modules in this manner, but never a break-out board like this.

Looking at your log and reading through esptool.py is seems clear that you have communication in both directions, otherwise it should have failed earlier - BUT I would verify this by disconnecting one wire (TX, RX) at a time and try a download, just make sure that the download behavior does indeed change.

How are you powering the ESP32? It looks like you are taking VCC from the FTDI adapter. The ESP32 is a bit power hungrier than many other controllers, are you sure your FTDI adapter is able to power it adequately?

Removing the USB/TTL bridge should not be necessary as it should be passive with no USB connected, and most ESP32 breakout boards include a resistor on the bridge TX output to make sure you can "override" the signal externally, like you are doing now. However, if you keep being unable to program I would remove it or cut the relevant traces just to be sure.