How does the AM modulated wave cancel out when the signal amplitude exceeds carrier wave amplitude?
That will only happen if the modulating signal is limited at the input, i.e. if its amplitude will never exceed the maximum modulation index m=1. If so, then you get what's in the pictured. Or if you consider the strict mathematical formula, as @ThePhoton says in his comment (just added). Else, the carrier will try to accomodate the excess by phase inversion:



The first (bottom) trace shows a modulation of 0.5, for the middle one m=1, and the top one m=1.5. Note the phase inversion that happens at ~0.4s. Fromhere on, the carrier will only change in amplitude, having the same inversion. Usually, the signal is limited to less than m=1, for good reasons.



Here's the FFT of each modulation:



The black trace is with m=0.5, it shows two peaks, symmetric over 10Hz (the carrier), 1Hz apart (input). The blue trace shows the same two peaks, twice as big, for m=1 (of course...).

Corection: For m=-1.5, there are even larger sidelobes. But the detection can not be done now because the peaks suggest the modulation input is folded where the negative m is.



In addition, heres how it looks when the input is limited to \$\pm\$1, but m=1.5: