Dc Motor giving ac as well as dc voltage when used as generator
What waveform does the motor give when you rotate it?
If you know the waveform, you'll understand the multimeter readings.

For most (cheap) multimeters the DC measurement gives you the average value of the waveform, or the DC component and the AC measurement gives the RMS value of which the DC component is blocked, so, not the true RMS.

I made a setup up of 2 brushed DC motors, one driving the other.
I measured the back EMF of the driven brushed DC motor:

with the multimeter on DC: 4.0 V
with the multimeter on AC: 0.041 V
with the scope, mean value: 7.97/2 = 3.99 V
with the scope, RMS value: 7.97/2 = 3.99 V

(Somehow my DSO scope multiplies the readings with 2, no idea why.)

So, in my case the waveform hardly has an AC component and therefore the multimeter on AC measures no RMS as it blocks the DC component.

In your case: check the waveform with a scope and/or check how the multimeter is measuring the RMS value.