12 V to 5 & 3.3 V regulator issue with Arduino and ESP8266
You connected your 3.3V regulator wrong.

Its pinout, as seen from the front, is Gnd/Vout/Vin:



You connected it as Vout/Gnd/Vin. Since its Vout is connected to ground, it's likely the regulator is passing its maximum current straight to ground. The same current also has to go through the 5V regulator, which has to drop at least 7V, resulting in a large power dissipation and the regulator getting too hot.

Also, neither your circuit diagram nor your wiring diagram show a ground connection between your power circuit and the rest of your circuit; your  esp8266 will never work that way.