Why is the regulation of my 5 volt power supply so poor and how to address it?
A 39000uF cap and a transformer winding is way too much for a filter.

I suspect your power supply is either defective, poorly-designed, or being operated way out of spec. The spikes are happening at a rate of ~85-90 kHz, which could be the switching frequency. The higher-frequency ringing afterward is clearly due to the spikes. If you can zoom in on the spikes with your scope, it might tell you (and us) more. A link to the Amazon page or a datasheet would also be helpful.

Regardless, your options are:


Try a more reasonable filter -- maybe a few microfarads ceramic capacitor.
Return or replace the supply.


The Photon's suggestion of trying a larger load is also a good one.