diode required for smoothing capacitor
A 1 kΩ thermal noise with a bandwidth of 10 MHz at 27 °C produces an RMS value of 12.87 μV. At 100 Hz, the value is 0.04 μV i.e. over 300 times smaller. I can't explain your scope noise pictures but the numbers speak for themselves.

also why a diode is needed in general in series with an AC signal for
capacitor to smooth out AC into DC signals

The diode is acting as a peak detector. Once the capacitor is charged up by the voltage peak, the subsequent falling voltage at the diode's anode reverse biases the diode and the capacitor remains charged to that peak.

Finally does that mean i should add a diode to LDO or buck regulators

No, you shouldn't.