How do LEDs slowly degrade in normal use?
Probably this article contains all you need to understand why high efficiency LEDs gradually fail:

Understanding the Cause of Fading in High-Brightness LEDs  (By Steven Keeping; Contributed By Electronic Products; 2012-02-21).

Indicator LEDs, on the other hand, are much less prone to failure, due to lower stress (less dissipated power), but the mechanism should be the same.

Some excerpts from that article follow:


  Primary cause of failure
  
  An LED is an electrical device, and as such there are many ways in which it can go wrong. [...] In practice however, LEDs are remarkably reliable and “failure” is most likely to be the result of light output falling below an acceptable threshold (typically 70 percent of the initial output [...]. The primary cause of that fading (or “lumen failure”) is triggered (for the most part) by the minute threading dislocations introduced to the chip during wafer manufacture.
  
  Threading dislocations act as nucleation sites for larger dislocations of the crystal. These form naturally due to heating during operation, thermal expansion and shrinkage when the LED is turned on and off, and mechanical stress such as vibration. As more and more dislocations occur over time, the number of sites for non-radiative recombination increases and quantum efficiency drops. (Some other factors, such as metal diffusion into the semiconductor from the connecting leads also contribute to lumen failure, but dislocations are the primary mechanism.)
  
  [...]
  
  Worse yet, the non-radiative recombinations that cause vibrations of the crystal lattice add to the overall temperature. In other words, as the chip ages, it will run hotter and hotter for a given forward voltage due to an increased number of phonons, accelerating the formation of dislocations and the device’s eventual demise.


Bottom line:


The PN junction fabrication cannot be perfect and this leads to imperfections in the crystal lattice.
These imperfections sport a different band-gap, so that electron-hole recombinations at those sites don't contribute to light (i.e. photons) emission, but causes emission of phonons (vibrational quanta).
The imperfections tend to act as centers where the lattice grows "irregular" ever more (this is called nucleation) due to vibrations, thermal shocks, etc...
Phonons tend to increase that nucleation effect, so the phenomenon has "positive feedback", and tends to worsen with time.
Adhering to manufacturer specs aids in keeping that problem under control.