How should I remove obvious anomalies from spectral data?
This seems to be a textbook case for a narrow median filter. Your $S$ is very broadband compared to your $N$, so a median filter will not affect your signal, as it can be designed quite narrow. And by using median instead of averaging (i.e. lowpass) you minimize the error introduced by the anomalies. I would give it a try.