When I turn down the dimmer switch on my lights, do I actually use less electricity?
Yes. And here's why.
Rheostat dimmers
Old dimmers, used a variable resister to dim the light. Lets look at a simple example.

We can find total resistance (RT), by adding up all the resistance.
RT = R1 + R2 = 0 Ohms + 144 Ohms = 144 Ohms
Then we can find the total current (IT).
IT = ET / RT = 120V / 144 Ohms = .83A
We'll then calculate the voltage across each resistive load.
E1 = IT * R1 = .83A * 0 Ohms = 0V
E2 = IT * R2 = .83A * 144 Ohms = 120V
Finally, we'll calculate the total wattage (WT)
WT = V^2/R = 120V ^2 / 144 Ohms = 100 Watts
Lets see what happens when we increase the resistance of R1

RT = 200 Ohms + 144 Ohms = 344 Ohms
IT = 120V / 344 Ohms = .349A
E1 = .349A * 200 Ohms = 69.77V
E2 = .349A * 144 Ohms = 50.23V
WT = 120V ^2 / 344 = 41.86 Watts
As you can see, we've increased the resistance of R1 and effectively reduced the voltage across R2. And now we have a dim light.
Thyristor dimmer
Modern dimmers use a TRIAC, to reduce the amount of time the light is on. However, because of the circuitry in the dimmer, there is not a direct 1:1 energy savings. Dimming the light to 50%, will not equate to a 50% savings in electricity.
A typical waveform in an AC system would look like this.

A TRIAC prevents electricity from flowing every time voltage reaches 0, something like this.

So you end up with a waveform that looks like this.

With the TRIAC, the light is actually turning off and on 120 times per second. With every cycle, you're saving a small amount of power.  Is it enough to actually see on your electric bill? I guess it would depend on how long the lights are on, and what percentage they are dimmed.