SIMULINK: The spectrum of the modulated signal remains unchanged despite modification of the modulation index
According to Wikipedia, narrow band FM has a modulation index less than 0.3.
This set of slides also mentions a similar circuit (Fig. 4.4 in the link) for narrow band modulator. The modulation index is small there too.
Condition \$\beta\int m(t) &lt;&lt; 1\$ is not satisfied in your example because the bias added to the message when integrated, produces large values.
The equation for narrow band FM mentioned in your question is derived in the above links. The derivation assumes certain approximation. So it will generate only 2 side bands; one upper and lower. Increasing the modulation index will

Make the assumption made in the derivation invalid.
Have no effect on the narrow band approximation equation.

The &quot;FM&quot; output has varying amplitude since the input message signal has a bias in it. When integrated, the bias integrates to \$kt\$ which is a (linearly) increasing signal. When multiplied with the carrier, this linearly increasing signal results in the output also to increase in amplitude.
The narrowband equation should be used only if the result of the integration of the message signal has small values. Your message signal (with its bias) doesn't have satisfy that condition when it is integrated.
Try after removing the bias. You will still get only two side bands since it is still an approximate formula for FM.