Vin output voltage of an Arduino
The most likely thing that is wrong is you don't have enough current.

I would add another battery in series. (You said you had four in series now.)

If the current supplied by the power source is used up, and the motor is pulling more, the voltage will drop for wall adapters. I don't know for sure about batteries, but my guess is that will happen too.

Your batteries may not have enough current to be supplied at once. Batteries have internal resistance (that varies). What that means is there is limits to the instantaneous current it can supply. EX: You cannot draw 500A from one battery.