DC motor rpm change by under supplying voltage
If the 5V supply holds up and the motor load (torque) is not too heavy, the speed should be somewhat less than ~5/6 of the speed at 6V. The value of 'somewhat' will increase (speed will decrease) increasingly non linearly with increasing torque until the motor stalls. 

Unless the motor is very small, the charger may not like supplying the start-up surge for the motor and it may just sit there.