Can I use a 150w 12v 775 DC motor with an old 800w PC PSU?
Yes you can. But! With motors you may get power flowing back to PS, and this kind of PS probably will not like it much. And you will not know in advance, this is something that is hard to predict. So you need either a servo drive with a regeneration resistor, or a huge capacitor to absorb excess energy. 

But having all that said, i bet everything will be fine. Just give it a try.

Edit. Be sure you are not just plugging a brushed motor to a PS. For that you need either a servo drive or at list something to limit the current when the motor is not running (zero speed, not necessarily stuck- this is the stall condition).