Is there a limit for voltage a PM generator gets to based on RPMs?
As discussed in the comments, you need to consider the winding inductance. As you increase the frequency the impedance will increase as given by the equation \$ Z=jωL = 2 \pi fL \$. That means that there will be an increase in the internal impedance of your coils and they will act as a potential divider with the load.

You can do make a reasonable measurement with your multimeter as follows:


Take an open circuit true RMS voltage measurement from your generator.
Connect up to your load and find what frequency causes the output to be half the open-circuit voltage.
Repeat step one at that frequency and check step two again to confirm that the AC reading is halved when on load.


Now you have the frequency where the impedance of the coil matches that of the load, R. From that we can say
$$ R = 2 \pi f L $$
so 
$$ L = \frac {R}{2 \pi f} $$

For a more thourough explanation see 
Is it possible to measure inductance without LCR Meter/OSC/FG.

With the coil's inductance and resistance measurements you can do all sorts of calculations and predict the output to a given load at various speeds.