Attiny84A and IR Sensor
It sounds like the microcontroller is browning out.

IR sensors are notorious for taking a lot of power when they operate; often they will tax a small power supply and cause the voltage to droop low enough to reset/brownout the microcontroller.

Try either putting a fairly beefy capacitor on the power bus, or getting a larger power supply.

Hope this helps!

(edit: I reverted another user's edit to this answer; "droop" is in fact the correct term for what is going on: http://en.wikipedia.org/wiki/Voltage_droop)