Battery powered Arduino on a breadboard
Here's a photo of a minimal ATmega328 on solderless breadboard (this is equivalent to an Arduino Pro Mini or an Arduino UNO without the USB interface):



Note the small 0.1uF ceramic capacitors, with very short lead lengths, placed as close as possible to both of the VDD-GND supply pins of the ATmega328. Also note that after I cut the resistor and capacitor leads, I recycled the bare wire trimmings as short jumper connections elsewhere on the board. Longer wires add inductance, and these solderless breadboards already have a lot of unwanted inductance to begin with.

On your board, I don't see any 0.1uF ceramic bypass capacitors near the IC - only a large bulk Al electrolytic where power enters the board. Missing those 0.1uF bypass capacitors can cause a lot of weird problems on digital ICs. Whenever any of the internal transistors switch on or off, the current in both VDD and GND changes. This change in current, together with the inductance of long wires, causes the supply voltage to (very briefly) go out of spec.

Usually I try to avoid using solderless breadboard, due to the parasitic inductance and somewhat unreliable connections... in this case I had already prototyped a design using the Arduino Pro Mini board, and was designing a PCB around that circuit. But since my PCB was going to be custom built with blank ATmega328, I needed to verify that I had tools and equipment to initialize blank ATmega328 chips.