strange behavior for atmel 16
Real world switches do not contact cleanly - they bounce and open and close their contacts many times over a few milliseconds before settling to their close position. 

Almost certainly what is happening is that when you press the right hand button it closes, then opens again before closing finally.

The initial closure will cause an interrupt but the second closing will cause a pending interrupt to be latched into the processor.

When the ISR exits after toggling the LED there will be the second interrupt pending that will cause the ISR to be entered again.

 (From The Lab Book Pages)

This is a very common problem and one of the reasons that it is not good practice to connect a switch directly to an interrupt.

The problem can be solved in hardware or software. This site shows some ways Switch debouncing.

Also as a matter of software design it is considered bad to put a delay loop within an ISR.  Interrupt service routines should do as little as reasonable and exit quickly.  All the time the ISR is executing the processor cannot process other interrupts. In your example it doesn't matter but in a real system it would.