24 VAC source unable to drive servo motor
24V AC when rectified and smoothed produces a DC level of about 32V. You are using the LM317 to produce 3.3V and then it seems that you are stepping this up to produce 5V for the motor. The regulator you are using is capable of producing over 1A.

You say your stepper motor consumes 200mA and this will require the LM317 to supply about 300mA into the booster.

300mA thru the LM317 whilst dropping about 29V gives a heat power dissipation of about 8.6 watts and you'll need a heatsink or the LM317 will just shut-down.

With a 24V DC supply the problem will be less but still the power dissipated by the LM317 could be as high as 6 watts.