Insulating small rim joist spaces
Yep, spray foam is the ticket. The conventional stuff expands greatly, so go easy until you get a feel for final volume. You'll end up using way too much and having a mess otherwise. You can certainly do it in layers.
Be aware that open cans can't be stored for longer than a few hours or so. They cure up and quit working. Don't start a can unless you'll finish it, or at least most of it.
Also note that urethane does not come off skin, clothing, and other objects. Protect yourself and your property before starting to spray.