74HC595 getting really hot on my pcb
Unless there is some kind of hidden short, which would not show up on the schematic, you’ve  probably got a damaged ‘595, which could have happened from ESD, a supply transient or something like that. Something caused latchup then the excessive supply current destroyed the chip.
You really MUST have bypass capacitors on this kind of circuit. One 1uF for each 595 and at least 100nF on the  ‘165, for example. Also, from the half of the layout we can see it’s pretty horrific in terms of potential ground bounce and supply transients. You potentially have hundreds of mA switching in nanoseconds. A mostly solid ground plane and bypass caps would greatly help.
I would also suggest adding 1K resistors in series with the logic connections to and from the Arduino.
And, as @Justme notes in a comment, all inputs whether for parts of the chip that are used or not must be tied to a valid logic level, either directly or through a resistor (the latter gives you a bit more flexibility).