Why do relay contacts open when connecting a dc motor on the switch contacts?
Your battery is not able to supply enough current to run the motor AND the relay AND everything else.  This may be caused by an inadequate battery, or it may be caused by inadequately-heavy-gauge wiring between the battery, the motor, and the relay contacts.

With the relay open, everything is fine.  When you key the transistor, current flows through the relay coil, the relay pulls in, closing the circuit to the motor.  The motor then draws a huge current, which the battery can't supply.  The battery voltage sags, current through the relay coil drops, and the relay drops out, cutting current to the motor.  Battery voltage pops back up, the transistor is still keyed, current is restored through the relay coil, and the cycle repeats.

Some years ago, I watched a similar phenomenon on a test rig.  We were using a very nice regulated power supply to control some solenoid-operated water jets.  With all the jets enabled at once, the total load was more than the supply could handle, and it started pulsing like crazy, as the overcurrent protector kicked in and out.  The fix was to energize no more than half of the jets simultaneously.