Precise capacitor in parallel to power supply
If capacitors were ideal, then there wouldn't be much point in it.
Real capacitors have resistance and inductance, however.
In this case, it is the equivalent inductance that is the problem.
That 4.7µF capacitor will be a fairly large electrolytic capacitor that is made up of rolled foil.  Its inductance will be fairly high, and prevent the capacitor from responding to high frequencies.
The smaller capacitor will be a ceramic capacitor with low internal resistance and very low inductance.  It can respond to very fast changes in current that the electrolytic capacitor can't catch.

Large electrolytic capacitor to handle long(ish) incidents.
Small ceramic capacitor to catch fast transients.

The both together will do a better job of keeping the supply voltage clean than either alone would do.