Why does the elliptical wing have the lowest drag?
Induced drag is caused by the backward inclination of the lift vector. Lift is defined as the aerodynamic force perpendicular to the flow direction, and since lift is created by deflecting this flow downwards, the resulting force is slightly inclined backwards.

The formula for the deflection angle (called downwash angle) $\alpha_w$: $$\alpha_w = arctan \left(\frac{2\cdot c_L}{\pi\cdot AR}\right)$$

To arrive at the drag, you now need to multiply the local lift with this angle. For small $\alpha_w$s the arcus tangens can be neglected, and we get this familiar looking equation for the backwards-pointing component of the aerodynamic force: $$c_{Di} = \frac{c_L^2}{\pi\cdot AR} = \frac{\alpha_w^2\cdot\pi\cdot AR}{4}$$

Now consider a wing which will do this unevenly over the wingspan. Some parts create a lot of lift and, therefore, a lot of downward deflection. Others produce less lift and less deflection. Due to the quadratic dependency of induced drag with downwash angle, the sum of all parts will always be higher when there are variations in the local downwash angle over span. Only the wing with a constant downwash angle will have the minimum induced drag for a given lift.

This is not necessarily an elliptic wing. By selecting the right twist (incidence variation over wingspan), any planform can be made to produce a constant downwash angle, albeit at only one angle of attack. Only the elliptic wing will combine the constant downwash angle over span with a constant lift coefficient over span such that variations in angle of attack still result in a constant downwash angle.

This is strictly true only in inviscid flow and when elastic deformation and the weight of the wing (which contributes to the need for lift and hence induced drag) are not considered. Also, an elliptic wing will produce nasty stall characteristics. Adding handling, friction and structural weight will shift the optimum to wings which produce more lift near the center. The exact optimum also will depend on the absolute size of the wing because the structural mass fraction goes up with size due to scaling laws.

Nomenclature:
    $AR \:\:$  aspect ratio of the wing
    $c_{Di} \:\:$  Induced drag coefficient
    $c_L \:\:\:$ lift coefficient
    $\pi \:\:\:\;$  3.14159$\dots$