AVR strange toggle LED behaviour
Could you please state what type of AVR you have?

I would think that as long as you do not define the F_CPU, the delay routine will take the default value to calculate the delay. Now if you just tell the CPU to assume a faster clock (this is what you do with redefining F_CPU) it has to count much higher to reach the same delay. Since the physical clock is still slow, the delay will be off by some very nasty factor.

If you're using Atmel Studio 6(+) take a look at the ASF wizard which has a specific template for the sysclock. Then it's just a matter of correctly defining the board_sysclk.h (which gets generated automatically) and a call to sysclk_init();

What did you really connect to the external pin?

Also, care must be taken when exceeding a specific value (this differs for various parts). In case of a high clock frequency, you will need to adjust the flash waitstates before switching the clock source.