Is rigid foam on rim board in basement worth doing if walls aren't insulated?
Nothing below grade needs to be insulated in most regions.  If you are in Canada or very north part of the world insulating the whole wall is advised but not for those below the border.

I have done a lot of basements.  I insulate walls a foot below grade - to create a pure stop of airflow down.  

With no wall we stuff as much rockwool in the rim joists as it will hold and sometimes throw a strap or two in to help.  Two sheets of rockwool is the norm (depth-wise).  You will notice a dramatic difference within a day.  Spray foam will provide the same results (maybe slightly better) but if I need to move an electrical/plumbing/gas line it is very easy to move the rockwool comparatively.