Does a buff remain in combat when the buffing creature dies to first strike damage?
No, you're right.


  510.4. If at least one attacking or blocking creature has first strike (see rule 702.7) or double strike (see rule 702.4) as the combat damage step begins, the only creatures that assign combat damage in that step are those with first strike or double strike.


After damage has been dealt, the active player gets priority (510.3) and state-based actions are checked; the Legion Lieutenant dies and its effect will stop (this is different from effects which say something like 'target creature will get +1/+1 until end of turn'). The Forerunner of the Legion is a 2/2 again.

Only then, a second combat damage step will be started, and damage done by the Forerunner of the Legion is calculated, which will only be 2 damage.