Standalone Atmega328p setup not working properly
A summary of some of the comments made, since I agree with all of them:


Do not use long wires for your crystal. The crystal is meant to
produce a waveform that is crucial to the microcontroller's ability
to generate a stable internal clock signal. This clock drives all
the logic inside the chip, enough deviation here can lead to some
very strange, undefined behavior. The parasitic resistance and
capacitance in these wires can greatly degrade the waveform from the
crystal.
Add some larger capacitors to your 5V supply lines. Start with
something on the order of 10-100uF at at the breadboard input, and
then 0.1uF closer to the microcontroller. It sounds like in one of
your comments that you've already tried powering through a 5V linear
regulator with a 12V (DC hopefully?) input with the right coupling
capacitors in place... I would actually stick with that even if it
didn't make a difference, I'd rather trust that than the 5V wall
supply.