Sir Bob Kerslake said one of the difficulties was that "easier" savings had already been made.
He said the "sense of urgency" would be reduced and the need for cuts would be hard to explain to public sector staff.
Sir Bob stepped down from his role this month and was replaced by former Cabinet Secretary Sir Jeremy Heywood.
He said the civil service had "excelled" during his time in charge.
Speaking at the Institute for Government think tank, he said it was clear that "under any government, we face up to a further five years of austerity in public sector spending".
"The first five years have been challenging but the second five years are likely to prove even harder for three reasons," he said.
"Firstly, the easier savings have already been made.
"Secondly, we are likely to be doing it against a background of a growing economy and greater competition for good staff.
"Thirdly, the sense of urgency that underpinned the first savings programme will be reduced.
"In reality, the task is not yet complete. But this will be hard to explain to those in the public sector, including our own staff, who are looking for some relief."
Sir Bob said the civil service had "excelled" in its challenge to "deliver big programme savings at the same time as undergoing huge change and reduction itself".
The roles of cabinet secretary and head of the civil service were split into two jobs when Sir Gus O'Donnell retired as both in 2012.
Sir Bob, who was already permanent secretary at the Department for Communities and Local Government (DCLG), took the role as head of the civil service at that time.
He will retire from his DCLG role early next year.
The outgoing head of the civil service has predicted another five years of UK government spending cuts - and he said making the cuts would be "even harder".