Arduino on breadboard, voltage regulator
If the output voltage of a linear regulator is more than 6-7V from the input, it may damage the pass element.

It's not uncommon to add a diode from the output to the input of a regulator to protect it in cases such as what you describe.  Some regulators include such protection, but probably yours does not (e.g. something like a 7805).  Take a look at Fig 27 of this datasheet for how that is wired.  

You might also want a diode to protect the 5V programming source.  You could use two diodes, one from the regulator and the other from the programming source, with both cathodes attached, feeding your Atmel.  Then neither source will be damaged by the other.

EDIT: Added schematic of this.  There would be a voltage drop from each diode so your circuit must be able to run on slightly less than 5V.







simulate this circuit &ndash; Schematic created using CircuitLab