What is the best removable insulation material?
You can use Backer Rod to fill in the crack if it is not too wide.  Backer rod is an easy to compress rope-like celled foam and should be easy to remove at a later date.  It is mainly used to fill deep cracks before caulking to give support to the caulk.

Backer rod comes in a variety of diameters and should be able to be found at your local building supply store.