Regulator/heatsink heating up even though load is only 50% of regulator Imax
The key things to figure out are:

Regulator current
Voltage drop

The first you've stated as 700mA.
The second we'll figure out. 24V AC will regulate to 24V/0.707 = 34V DC. With 24V output, we have a drop of as much as 10V (34V in - 24V out.)
10V at 700mA will dissipate... 7W in your part, worst-case. That's kind of a lot for the LM317 / 337 in TO-220.