The Rail Accident Investigation Board (RAIB) report blamed the near miss at Hest Bank, Lancashire, last September on the lack of warning from a lookout.
Four of the group working at the bridge had to cling to its parapet to keep clear of the train, said the report.
Network Rail said: "Our railway is the safest in Europe."
"We are always looking at ways to improve safety for the benefit of those who work on the railway and those who travel on it," a spokesman added.
The RAIB report said the party, who were working on repairing a sleeper, should have received a warning from a lookout.
Instead the controller of site safety had to shout an urgent warning after spotting the train, travelling from Edinburgh to Manchester Airport, when it was just three seconds away.
"Staff took rapid evasive action, but some members of the workgroup were on the bridge and were unable to reach the designated position of safety, instead pressing themselves against the bridge parapet," said the report.
"The last member of the workgroup was not clear of the line until about one second before the train passed."
The report said the lookout might have pushed the wrong switch on the electronic system to warn colleagues, might have forgotten to send a warning or their "vigilance had degraded" because it was near the end of the recommended two-hour shift for a lookout.
Nine railway workers were within seconds of being hit by a train travelling at 98mph, a safety investigation has found.