How can I get this 12v motor to run smoothly, and is it set up safely?
Judging from the motor picture, you have a fairly powerful motor RS-550 class, from some power tool/drill or other appliance. The motor start-up current (or load current) is likely above the capability of your power supply, and the supply goes into sporadic shutdown mode, and back. If your power supply is already giving out just 12 V instead of 18V, it already operates at the verge of shut-off, it could be thermal or overcurrent shutdown. This RS550 motor, for example, takes 1.4 A just for idle rotation at 12V. The motor will likely run at higher voltage, but the idle current will be proportionally higher.

Get more powerful supply, something like 12 V 5 A.