Modulation Index of Conventional AM Signal An AM signal has the form

$$u(t)=100\left[1+\sin(2000 \pi t)+5\cos(4000\pi t)\right]\cos(2\pi f_ct)$$

Where $f_c$ is 800 kHz.

What is the modulation index?

(We can define modulation index as the ratio between the amplitude of message wave and the amplitude of carrier wave)
Modulation index for linear (continuous wave) amplitude modulation may be defined in different ways. But the one I found relevant to your definition is:

$$ \mu = \frac{ | \min \{ m(t) \} | } {A_c} $$

where $A_c$ is the carrier amplitude, $m(t)$ is the message signal and $\mu$ is the modulation index.

For simple demodulation of a conventional AM modulated signal $s(t)$ we look for $\mu &lt; 1$ so that the envelope of $s(t)$ never becomes less than zero.

$$s(t) = [ A_c + m(t) ] \cos(2 \pi f_0 t) $$

Your signal is:
$$ s(t) = [ 100 + 100 \left( ~~\sin(2000 \pi t)+5\cos(4000 \pi t)~~ \right) ] \cos(2 \pi f_c t) $$

Your message signal is:
$$ m(t) = 100 \left( ~~\sin(2000 \pi t)+5\cos(4000 \pi t)~~ \right) $$

whose min value is $-600$ and from which you deduce the modulation index as:

$$ \mu = \frac{ | \min \{ m(t) \} | } {A_c} = \frac{ | -600 | } {100}  = 6 $$

I'm sorry to say but this is severely overmodulated...