Why doesn't this motor turn?
The IO drivers of most common logic chips or microprocessors such as the ATmega on your Arduino are nowhere near strong enough to drive typical electromagnetic motors.

To drive a motor, you should use a higher current switching device - a standard solution for bidirectional control being an H bridge part such as a TB6612FNG, L293D, L298, etc.

These are commonly sold on "motor shields" which you can plug into your Arduino, if you want to experiment with the concept before building something (and if you do want to build something, reviewing such a working schematic first would be a good idea).

You can in theory put an L293D or its DIP-packaged work-alikes on your breadboard, but you will need some care to get it right, especially if you use one of the other parts which requires external catch diodes.

Finally, you should not be powering most motors from the Arduino's onboard regulator or a USB port, but rather should use an external power supply.

=============

Responding to your edit, by  changing connections to place the the motor between one pin and the 5v supply, you have done two things:

1) You have made it so that the voltage difference across the motor can be applied in one direction only, therefore it can only rotate in one direction.

2) You have made it so that only the N-channel MOSFET in an output driver is involved in powering the motor.  That change is almost a useful idea, as N-channel devices are in fact much more capable than P-channel devices of similar dimension (and practical capacitance, voltage rating, and other tradeoffs) due to the greater mobility of negative charger carriers.

But this is still not really a solution, as it is unlikely that your motor's current rating is within even the current sink rating of the N-channel side of the ATmega output.

If you want a simple, unidirectional motor control, you could pursue this idea using one external N-channel MOSFET (or even a biploar NPN power transistor, which will be less efficient but also less delicate) to connect the unpowered side of the motor to ground.

If you want bidirectional control, you will need either + and - power supplies, or four external switching devices taking the roles of the low current N-FET and P-FET totem poles inside each of the two on-chip IO pin drivers you started with.