Mechanical Switch Causes Transients from AC-DC Power Supply
What I think you are observing is the settling of your output unloaded.

You say the "noise" is occurring right after switch on of the AC input but on your oscilloscope we don't see the rise of the output voltage from 0 to 5V.

Thus making it difficult to help you...

But your problems looks a lot like switch bounce combined with the undampened beahviour of a unloaded supply.

What I would suggest you to do:


Do not put an EMI filter (this would help if other noisy equipment shared the AC voltage, but not for the switch)
A NTC is always a good idea following a switch on an AC line voltage (as you mentionned it will helps with the current inrush if you switch on the highest portion of the sine and if your redressing caps are completely deplated and if your wire has very little resistance)
Load your supply at least with a small load do not just put its output on a HiZ such has your oscilloscope, it will mostelikely not behave the same in your final setup (choose your resistor carefully not to set your house on fire, $$P=U^2/R &lt; P_{rating-of-your-resistor} $$)
Try a low pass filter at the output of your power supply (a simple capacitor in parrallel with the output might add some dampening of the regulation loop inside you PSU, note that this will worsten the dynamic behaviour of the voltage tracking of the supply, but it might helps with the stability a little bit)