insulating rim joist
In my house, a square piece of insulating board is up against the rim joist at the end of the bay and great stuff has been used around the edges of the insulating board.  (My house was built in late 1800s)

edit: if you don't use foam, there's really no insulating value - insulation really only happens when air movement is halted. 

2nd edit: foam is cheap.  You can use it, then in the spring tear a few off to check for termites and then refoam.  Use a sharpie to write the date on the foam the last time you checked it... just an idea.

I took a photo for you.