Why does a battery starts working on rubbing?
From your description, it sounds like some kind of dirt gets between the battery terminals and the holder terminals.  Batteries shouldn't be made of a material that oxidizes easily in air, and neither should the holder for exactly this reason.  However, I have seen holders made from just copper, which certainly can oxidize in air.  Good holders are usually nickle-plated steel, which keeps working for a long time.  Nickle is relatively inert in air.

Do you live someplace with high pollution or near the ocean?  Even just a little H2S over time can corrode things.