Next steps for analyzing this PWM algorithm and choosing a low pass filtering technique?
The pulse generation algorithm sounds like it is more like Pulse Density Modulation (PDM), not Pulse Width Modulation (PWM), because PWM would pack the ones into the beginning of counting, and leave the zeroes in the end.

Indeed some values of N and M cause the output pattern to have a certain repetitive bit stream, for example the 100/200 means the output is 50% 0 and 50% 1, so the output is a bitstream of 1,0,1,0... which is easy to filter. If you set N and M to be exactly ratio of 1/3, the output pattern will also be repetitive, 0,0,1,0,0,1... So, in light of this this also explains the spectrogram of N=50 M=200, which is 25%, 0,0,0,1,0,0,0,1.

These patterns that have a short repetition rate show up as pure tone peaks in the spectrogram. Choosing the values so that there is no obvious short repeating patterns basically dithers the bitstream and widens the spectrum to have multiple lower peaks.

Depending on what is the usage of the output will determine how to filter it. For many cases a simple RC filter is enough, as long as the PDM bitstream output rate is high enough for the RC filter cutoff frequency.