Why does this simple circuit oscillate? (Ekasi oscillator)
First, the circuit as you present it is a bit too simple -- the resistor in series with the battery as shown on this page is necessary, and must be about the right value.

Second, the reason the capacitor will discharge (in the correct circuit) is because the LED-transistor string is consuming more current than the battery (through that resistor you left out) is supplying -- so the capacitor discharges.