Zener diode can vary current flow to maintain voltage drop, how does this magic effect work?
It just looks like magic doesn't it?


If you haven't taken a course in semiconductor device physics, I suppose it does look like magic.

PN junctions are not resistors.  Whereas the voltage across a resistor is proportional to the current through

\$v_R =R \cdot i_R  \$

the current through a diode is approximately:

\$i_D = I_S(e^{\frac{qv_D}{kT}}-1) \$

In words, the diode current goes up exponentially with the diode voltage.



The derivation of the diode equation isn't trivial but it isn't mysterious either; it's based on well understood device physics.

For zener and avalanche diodes, the physical explanation is a bit easier to grasp.  From Wiki:


  Under a high reverse-bias voltage, the p-n junction's depletion region
  expands, leading to a high strength electric field across the
  junction. A sufficiently strong electric field enables tunneling of
  electrons from the valence to the conduction band of a semiconductor
  leading to a large number of free charge carriers. This sudden
  generation of carriers rapidly increases the reverse current and gives
  rise to the high slope resistance of the Zener diode.
  
  The Zener effect is distinct from avalanche breakdown which involves
  minority carrier electrons in the transition region which are
  accelerated by the electric field to energies sufficient to free
  electron-hole pairs via collisions with bound electrons. Either the
  Zener or the avalanche effect may occur independently, or both may
  occur simultaneously. In general, diode junctions which break down
  below 5 V are caused by the Zener effect, while junctions which
  experience breakdown above 5 V are caused by the avalanche effect.