Problem with iMac Apple Remote
It is hard to say for sure without more information, but my first guess would be interference.

Do you have another piece of electrical equipment near your iMac that could be impacting the signal? Speakers? A wireless headset? A router?

In my new apartment whenever I get a phone call on my 2.4Ghz land-line telephone it drops my WiFi connection on my iMac so interference can definitely cause odd problems!