Overclocking: permanent damage?
I've overclocked almost every computer (excluding laptops) i've ever owned purely for the cost savings and making matlab sims not take all day.

Overclocking, as in raising the clock speed or multiplier shouldn't damage modern CPU's.  The thermal shutdown in the CPU should trigger early enough to prevent damage.  Older CPU's didn't have as robust thermal protection.

If you're raising various voltages in an attempt to run even faster, you can inadvertently cause permanent damage to the CPU.  It's good to stay within the max voltage specifications given by the CPU manufacturer.

Depending on your usage model, overclocking can cause reduced life span.  This is really just a function of CPU temperature, the hotter it runs the shorter the life span.  If the CPU is running right on the edge of its TDP rating 24/7 i wouldn't expect it to last for 10 years.

You generally are not running the device outside its design specifications as long as you stay within the specified voltage levels.  As a design is fleshed out manufacturing yields get better and better and parts binned to 2.6GHz are very often capable of and tested for much higher speeds, they are just binned to the low end to fill the higher market demand for that segment.

Currently typing on a core i7 920 @ 4.1ghz with air cooling (granted it's one massive heatsink and 2 140mm fans).  D0 stepping, a newer stepping which is capable of much higher speeds than the older steppings.  I actually ran a 12hour prime95 test at 4.25ghz but anything higher started spewing errors and i didn't want to raise the supply voltages any more so I backed off a bit to get to 4.1 for some headroom.  You also have to make allowances for ambient temperature changes if your space isn't air conditioned.

EDIT for sheepsimulator:

The effect on the ram depends on the architecture your talking about and the features offered by the motherboard.

For example in the core i7 architecture:

In the Core i7 architecture you have 1 base clock that generates the clocks for the CPU core, the 'uncore', the QPI and the RAM via 4 different multipliers.  

In some CPU models these multipliers have limited ranges, but key to your question: when you overclock the system you normally crank the base clock up which does also increase the RAM clock.  But, you can reduce the RAM clock multiplier to get stock or very close to stock ram speeds if you wish.  The core i7 920 by default uses DDR3-1066 ram but DDR3-1600 is almost the same price so most people buy the faster ram and adjust the RAM multiplier to get to the 1600 rating.  You also have control over the ram voltage on good motherboards so you have the options of over volting/clocking the ram should you so wish.

In some older architectures there was limited or no control over the RAM clock multiplier which could mean that you need faster ram to achieve a particular CPU clock.