Why my single-phase induction motor's velocity decreases after 100Hz?
The problem is likely the type of motor you are using - it is a single-phase AC motor with a capacitor. The capacitor feeds a secondary winding in the motor that ensures the direction of the motor is always maintained. The cap and the winding form a kind-of tuned circuit that create a phase angle (usually 90 degrees) and the winding is placed at 90 degrees mechanically (usually). What happens when you change the frequency from nominal: -

This is a picture of the phase response from an LCR filter but this will be exactly the same on the type of motor you have. Frequency = 1 (middle) can be taken to be your motor's nominal running frequency (i.e. 50 or 60 Hz) - note how the phase shift is +/- 90 degrees. What happens when F doubles - that phase shift moves by 45 degrees. It's worse on a light load because the losses are smaller and the graph is steeper: -

Ignore the picture on the right and concentrate on the picture on the left. On lighter loads (higher Q factor) the phase difference when doubling the frequency can almost move a further 90 degrees - you could find a situation that if you increased the speed further, the motor may even reverse direction.
That's your problem I believe.
EDIT
Running the main coil and auxiliary coil at 90 degrees is the right thing to do to avoid the effects noted above - this will allow you to run slower than the rated nominal frequency of the motor. The only concern here is that you reduce voltage with reductions in frequency - this is done because the reactances of stator windings and rotor winding also reduce linearly with frequency and can take too much current and burn. The force needed to move the rotor is proportional to current so reducing voltages with reducing frequencies makes sense.
On the other hand, increasing the frequency above nominal requires an increase in voltage in order to preserve the force (supplied by magnetism aka ampere-turns) but, it's going to be a bit worse than that - the force that moves the mechanical inertia of the rotor needs to increase at higher speeds to get it to &quot;shift&quot; 90 degrees in a shorter period of time. This is a slippery slope because to do that means even more voltage and the potential for electrical insulation breakdown not only due to the voltage but also the heat generated at higher frequencies - remember, this type of motor is a transformer and eddy current losses in the core will increase with frequency thus creating more heat.
In short, higher than nominal frequencies means more heat due to increased eddy current losses and a significantly bigger voltage supply to more-than-maintain the ampere-turns needed to move the rotor faster than it would at nominal frequency.