Board members are believed to have wanted bigger cost cuts and more focus on the investment bank's performance.
Chairman John McFarlane said the bank needed to become more efficient: "What we need is profit improvement. Barclays is not efficient. We are cumbersome."
In a statement, Barclays said a "new set of skills" was required at the top.
Mr Jenkins has been Barclays' chief executive since 2012. The bank said a search for his successor was under way.
Barclays' chairman John McFarlane has been named executive chairman until a new chief executive is appointed.
In a conference call, Mr McFarlane said the board had decided the firm needed to change its strategy in order to boost revenue growth.
Barclays needs to be "leaner and more agile" to improve the firm's capital performance, he said.
Investors welcomed the news of the change, sending shares in Barclays up more than 2% in London.
Mr McFarlane applauded Mr Jenkins' role in steering Barclays through the aftermath of the financial crisis, and through the fallout of Barclays' management shakeup three years ago.
But he also said: "There is no question that cultural change was urgently required."
Mr McFarlane told BBC business editor Kamal Ahmed that Mr Jenkins' skill set had been suitable when he took the top post, but that the firm's needs had changed.
When the BBC asked him about future job cuts, he did not rule them out.
Nor did he rule out the possibility of branch closures.
"Inevitably, banks are going to have fewer branches than they have now," Mr McFarlane said.
He also told the BBC that Barclays would not renew its sponsorship of the Premier League when it expires later this year.
Make no mistake, Antony Jenkins, the chief executive of Barclays, has been fired after falling out with the board over the size of the investment bank and the pace of cost cutting.
Mr Jenkins, who became chief executive in 2012, was told in the last few days that the board no longer had confidence that he was the right man to lead the organisation.
Senior sources have told me that Sir Mike Rake, the deputy chairman of Barclays, approached the new chairman, John McFarlane, to say that a number of board members were unhappy with the speed of change at the bank.
They wanted cost cutting to go further and more attention paid to the investment bank which is seen as under-performing.
It is thought that Sir Mike wants Barclays to retain its global presence as a major investment bank whilst Mr Jenkins felt it should be cut back.
It has been made clear to me that Mr Jenkins was seen as having done a good job steadying the bank after the previous chief executive, Bob Diamond, resigned.
But it is now time for a new person at the top.
Whilst that search continues, Mr McFarlane will become chairman and chief executive, a similar role to the one he held at the insurance giant Aviva.
In Barclays' statement, Mr McFarlane said the bank needed to boost returns to shareholders. "We therefore need to improve revenue, costs and capital performance."
The board said it recognised the contribution Mr Jenkins had made over the past three years, and was "extremely grateful to him for bringing the company to a much stronger position".
But the bank said its non-executive directors had "concluded that new leadership is required to accelerate the pace of execution going forward".
Mr Jenkins took over at Barclays in the summer of 2012 following the departure of Bob Diamond, who left in the wake of the Libor scandal.
The bank said Mr Jenkins had inherited a situation which "would have challenged anyone facing the same issues".
Mr Jenkins said that when he had taken over as group chief executive in 2012 it was a "particularly difficult time for Barclays".
"It is easy to forget just how bad things were three years ago both for our industry and even more so for us.  I am very proud of the significant progress we have made since then," he said.
Antony Jenkins, the chief executive of Barclays, has been fired after falling out with the board over the bank's cost cutting and profitability.