Table jitter on 40 ft planer mill at certain low feedrates with new 40 hp 3-phase motor and 50 hp ABB drive
I would expect speed stability problems at lower speeds ( below 60% rated motor speed).
From your description the VFD is running open loop with no encoder feedback.  You can try different drive modes such as Constant Torque or Vector control - but without encoder speed feedback on the motor the speed and frequency control is open loop and low speed stability is limited.
It’s also likely that you are having vibration issues unrelated to the VFD and motor.