Does voltage drop with discharge?
To answer your questions:


Correct, as you discharge a capacitor the voltage drops. This is due to the relationship of \$Q = VC\$ - the charge stored in a capacitor is proportional to the voltage for a given capacitance. As you discharge the capacitor, the charge on the capacitor is reduced, and so the voltage reduces.

However this has nothing to do with batteries.
A battery terminal voltage will drop as you discharge it, mainly because the chemical reactions slow down due to depletion. 

This is nothing to do with the principal of capacitance. 

At its simplest, you can think of a battery as an ideal voltage source, and a series resistor. As you discharge the battery, the reactions slow down, which increases the value of the representative series resistance. As a result for the same load, the terminal voltage will drop (see also: potential dividers). 

However in practice it is much more complex. For one there is no such thing as an ideal voltage source. The change in terminal voltage is also not linear. It depends on the chemistry of the battery, and also the current draw.
Here is an example Lithium Primary 9V PP3 battery, the discharge curve for which is shown below:


Notice how the voltage varies over time, and in a non-linear nor exponential fashion. This is very different to how a capacitor would discharge (exponential decay).
For a simple LED circuit, you would have a resistor in series with your LED. If you decrease the voltage across the circuit (regardless of whether you use a battery, a capacitor, or other power supply), you will reduce the current flow. Therefore you reduce brightness of the LED as the battery discharges.

If however your LED circuit used, say, a DC-DC converter rather than (or in addition to) a series resistor, you may find that the converter will keep the LED voltage constant as the input voltage varies. As a result the brightness stays constant even as the battery discharges. At some point the battery voltage would be too low for the DC-DC converter, and switch off, turning the LED off. Basically you end up with constant brightness until it turns off.