How should I air-seal this gap by the floor along an exterior wall?
Expanding foam would do a fantastic job especially if you are already removing the baseboard anyway it should be no problem to fit the straw back there.