Do rubber bike pedals or rubber pedal covers exist?
A short length of foam pipe insulation should do the job.  It's cheap, will fit into your pocket, and you can get it at any hardware store or plumbing supply shop.  You could also slide it onto your top, down, or seat tube for storage.

It looks like this:



Choose the size based on the type of pedals you use.  Platform pedals will require a larger diameter and clipless pedals a smaller diameter.  You want it big enough so you can easily slide it over the pedal but small enough that it stays put.