What is the proper way to insulate box sill joists in the basement?
The Family Handyman recommends using rigid foam, caulk, and spray foam to insulate the rim joist, in this article Insulate Basement Rim Joists.

Not sure if this is exactly what you're looking to do, but the idea should work in your situation I think.
Summary of steps:

Cut rigid foam to fit between joists.

Insert foam into gaps (against rim joist).

Caulk around the edge of the foam.

Fill any gaps in the rigid foam where pipes or vents pass through, with spray foam.