Using volatile in embedded C development
A definition of volatile
volatile tells the compiler that the variable's value may change without the compiler knowing. Hence the compiler cannot assume the value did not change just because the C program seems not to have changed it.
On the other hand, it means that the variable's value may be required (read) somewhere else the compiler does not know about, hence it must make sure that every assignment to the variable is actually carried out as a write operation.
Use cases
volatile is required when

representing hardware registers (or memory-mapped I/O) as variables - even if the register will never be read, the compiler must not just skip the write operation thinking &quot;Stupid programmer. Tries to store a value in a variable which he/she will never ever read back. He/she won't even notice if we omit the write.&quot; Conversly, even if the program never writes a value to the variable, its value may still be changed by hardware.
sharing variables between execution contexts (e.g. ISRs/main program) (see kkramo's answer)

Effects of volatile
When a variable is declared volatile the compiler must make sure that every assignment to it in program code is reflected in an actual write operation, and that every read in program code reads the value from (mmapped) memory.
For non-volatile variables, the compiler assumes it knows if/when the variable's value changes and can optimize code in different ways.
For one, the compiler can reduce the number of reads/writes to memory, by keeping the value in CPU registers.
Example:
void uint8_t compute(uint8_t input) {
  uint8_t result = input + 2;
  result = result * 2;
  if ( result &gt; 100 ) {
    result -= 100;
  }
  return result;
}

Here, the compiler will probably not even allocate RAM for the result variable, and will never store the intermediate values anywhere but in a CPU register.
If result was volatile, every occurrence of result in the C code would require the compiler to perform an access to RAM (or an I/O port), leading to a lower performance.
Secondly, the compiler may re-order operations on non-volatile variables for performance and/or code size. Simple example:
int a = 99;
int b = 1;
int c = 99;

could be re-ordered to
int a = 99;
int c = 99;
int b = 1;

which may save an assembler instruction because the value 99 won't have to be loaded twice.
If a, b and c were volatile the compiler would have to emit instructions which assign the values in the exact order as they are given in the program.
The other classic example is like this:
volatile uint8_t signal;

void waitForSignal() {
  while ( signal == 0 ) {
    // Do nothing.
  }
}

If, in this case, signal were not volatile, the compiler would 'think' that while( signal == 0 ) may be an infinite loop (because signal will never be changed by code inside the loop) and might generate the equivalent of
void waitForSignal() {
  if ( signal != 0 ) {
    return; 
  } else {
    while(true) { // &lt;-- Endless loop!
      // do nothing.
    }
  }
}

Considerate handling of volatile values
As stated above, a volatile variable can introduce a performance penalty when it is accessed more often than actually required. To mitigate this issue, you can &quot;un-volatile&quot; the value by assignment to a non-volatile variable, like
volatile uint32_t sysTickCount;

void doSysTick() {
  uint32_t ticks = sysTickCount; // A single read access to sysTickCount

  ticks = ticks + 1; 

  setLEDState( ticks &lt; 500000L );

  if ( ticks &gt;= 1000000L ) {
    ticks = 0;
  }
  sysTickCount = ticks; // A single write access to volatile sysTickCount
}

This may be especially beneficial in ISR's where you want to be as quick as possible not accessing the same hardware or memory multiple times when you know it is not needed because the value will not change while your ISR is running. This is common when the ISR is the 'producer' of values for the variable, like the sysTickCount in the above example. On an AVR it would be especially painful to have the function doSysTick() access the same four bytes in memory (four instructions = 8 CPU cycles per access to sysTickCount) five or six times instead of only twice, because the programmer does know that the value will be not be changed from some other code while his/her doSysTick() runs.
With this trick, you essentially do the exact same thing the compiler does for non-volatile variables, i.e. read them from memory only when it has to, keep the value in a register for some time and write back to memory only when it has to; but this time, you know better than the compiler if/when reads/writes must happen, so you relieve the compiler from this optimization task and do it yourself.
Limitations of volatile
Non-atomic access
volatile does not provide atomic access to multi-word variables. For those cases, you will need to provide mutual exclusion by other means, in addition to using volatile. On the AVR, you can use ATOMIC_BLOCK from &lt;util/atomic.h&gt; or simple cli(); ... sei(); calls. The respective macros act as a memory barrier too, which is important when it comes to the order of accesses:
Execution order
volatile imposes strict execution order only with respect to other volatile variables. This means that, for example
volatile int i;
volatile int j;
int a;

...

i = 1;
a = 99;
j = 2;

is guaranteed to first assign 1 to i and then assign 2 to j. However, it is not guaranteed that a will be assigned in between; the compiler may do that assignment before or after the code snippet, basically at any time up to the first (visible) read of a.
If it weren't for the memory barrier of the above mentioned macros, the compiler would be allowed to translate
uint32_t x;

cli();
x = volatileVar;
sei();

to
x = volatileVar;
cli();
sei();

or
cli();
sei();
x = volatileVar;

(For the sake of completeness I must say that memory barriers, like those implied by the sei/cli macros, may actually obviate the use of volatile, if all accesses are bracketed with these barriers.)