Explanation for decoupling capacitors
Parasitic inductance is the major reason. A change in current causes a voltage drop across inductance proportional to the rate of change of current. This necessitates bypassing that is local to the chip. Otherwise we could just use one big capacitor for the entire power rail.

There can be and are current surges even without the LED due to the transistors charging internal and external capacitances and due to conduction through complementary pairs.

A voltage regulator also has an inductance-like behavior as you note, however usually it is swamped by a relatively large capacitor near the regulator output.

The capacitances and inductances of which I speak are parasitic- due to unwanted effects that are part of the construction- even a few cm of traces will have significant inductance and it's easy to get a few pF of capacitance. Much of the average current that a digital CMOS chip draws is just at the transitions, charging and discharging such capacitances, so it occurs in a spikey manner at the clock edges.

It is not straightforward to calculate the appropriate values because there is insufficient information, but following data sheet recommendations and rules of thumb (and good PCB layout) is typically more than sufficient. Sometimes datasheets eval boards and reference designs call out very large numbers of capacitors, perhaps 60 or 80 for a single IC with larger and smaller values paralleled for better performance. The Zynq 7000 eval board, for example, has around 150 bypass capacitors for the SOC chip.