This includes stadium safety expert Paul Scott.
The Cabinet Office report found relations behind the scenes were "broken", and it would be at least another year before another planning application could be submitted.
However, the authors said that a new stadium was still "achievable".
They said the overall business case "remains sound".
The report was commissioned after Mr Scott told a Stormont committee in April that the emergency-exiting arrangements in the proposed 38,000-capacity stadium were flawed.
However, the report has recommended that he be replaced as chair of the Safety Technical Group overseeing the Casement project.
The report has called for other personnel changes:
Sports Minister Carál Ní Chuilín has indicated she will implement all of the recommended changes.
She said a key finding that the stadium is still achievable was welcome and provided clarity.
The report stated: "Relationships between the key stakeholders are broken.
"We believe the use of an independent mediator will be needed to reset working relationships."
Work on the stadium was supposed to start this year but planning permission was overturned in December 2014.
The report stated: "The Casement Park project is behind schedule and has to address two key issues before it can make progress.
"It needs to find a way to resolve issues raised with the proposed emergency-exiting arrangements for the stadium.
"It also needs to reapply for and achieve planning permission."
The report also included a "delivery confidence assessment" that is based on traffic lights, with green being the highest rating.
Casement is rated "amber/red".
Although the report recommends that Mr Scott be replaced as chair of the safety technical group, that does not mean he cannot continue to sit on the group.
The report says his replacement will need mediation and facilitation skills "rather than technical knowledge".
The chairman of the Stormont committee which is investigating Mr Scott's allegations said the Cabinet Office report "is part of the resolution but only part".
Democratic Unionist Party MLA Nelson McCausland said the minister and her officials had more questions to answer.
"This coming week we'll be convening a special meeting of the committee at Stormont and we will be requesting that the minister and the permanent secretary appear before the committee to answer the questions that we will be putting to them," he said.
A report into the planned new stadium at Casement Park in Belfast has recommended the replacement of senior figures from key posts in the project.