Why would I want to define a modulation index for each tone (DSB-FC)? So the exercise is basically a signal $f(t)$ that is going to modulate the carrier $A\cos(\omega_ct)$ using a modulation index of $m=1$. I have to find $A$ and the power of the modulated signal:
$$
f(t)=\cos(\omega_mt)+2\cos(2\omega_mt)
$$

The minimum amplitude of $f(t)$ is $-2$. Then $A = 2$. The power of the signal is, assuming that $R = 1 \ \Omega$:
$$
P = P_c+P_s=\frac{A^2}{2}+\frac{\overline{f^2(t)}}{2}
$$

Having in mind that:
$$
\overline{f^2(t)}=\frac{1^2}{2}+\frac{2^2}{2}=\frac{5}{2}
$$

The power is:
$$
P =\frac{A^2}{2}+\frac{\overline{f^2(t)}}{2}=\frac{2^2}{2}+\frac{5}{4}=3.25
$$

In the book the author uses an effective modulation index that is defined as $m_t = \sqrt{m_1^2+m_2^2}$ where $m_1=1/2$ and $m_2 = 2/2$. So the power is:
$$
P = P_c\left(1+\frac{m_t^2}{2}\right)=2\left(1+\frac{1.12^2}{2}\right)=3.25
$$

My question is, why would I want to define a modulation index for each tone? What do I get from that?

Another thing that I don't understand is that according to this guy the condition that ensures that there's no overmodulation, regardless of the tone frequencies, is $m_1+m_2\leq1$. Obviously in this case the condition is not met but I'm pretty sure there's no overmodulation with $A=2$.
I'm no expert on this, but until the big guys give an answer, here's my take:

For $f(t) = \cos(100t)+2\cos(2\cdot 100t)$, so $\omega_m = 100\ \rm Hz$ for example:



The amplitude's peak value$\ =3$ exceeds $A=2$, so there is overmodulation. This answer only addresses your last point of "I'm pretty sure there's no overmodulation".