In a speech he said, given the effects of the referendum vote, the government had to be "realistic about achieving a surplus by the end of the decade".
The target had been the chancellor's most prized goal and had been driving austerity measures in previous budgets.
But he said the economy is showing "clear signs" of shock following the vote to leave the European Union.
Giving a speech in Manchester, Mr Osborne said: "The referendum is expected to produce a significant negative economic shock to our economy. How we respond will determine the impact on jobs and growth.
"We must provide fiscal credibility, continuing to be tough on the deficit while being realistic about achieving a surplus by the end of the decade."
Labour's shadow chancellor John McDonnell welcomed the move, saying: "Sadly the vote last Thursday for Brexit has only brought forward what was inevitable.
"The Chancellor had already dropped his other fiscal rules on welfare and debt at the Budget in March, and according to many economists he was expected to be forced to drop this one too."
BBC business editor Kamal Ahmed says that the target was heading for the "Treasury shredding machine" following a speech from Home Secretary Theresa May yesterday.
In a speech launching her bid to become prime minister she said "we should no longer seek to reach a budget surplus by the end of the parliament".
Kamal says that Mr Osborne has now followed suit.
Nevertheless, the Treasury insists that it still wants to balance the books and "fix the finances".
Mr Osborne first committed himself to returning the national finances to surplus by 2020 at the Conservative Party conference in 2013 and it became policy in the July budget of 2015.
But there have been questions ever since over whether that target could be met.
Even before the referendum vote he only had "a slightly better than 50/50 chance" of making the target said Carl Emmerson, deputy director of the Institute for Fiscal Studies.
But the uncertainty caused by the Leave vote could hold back the economy and make creating a surplus even more difficult.
"Having voted for Brexit last week, the economy is clearly going to go into a downswing, that might be a full-blown recession, that might just be very very low growth," said Paul Johnson the director of the IFS.
It is sometimes easy in these incredible political times to forget that for most people "it's the economy, stupid" still holds true.
For the UK economy, one of the most important passages of Theresa May's speech yesterday was when she signalled that George Osborne's "fiscal rule" (to produce a budget surplus by 2020) was for the Treasury shredding machine.
"While it is absolutely vital that the government continues with its intention to reduce public spending and cut the budget deficit, we should no longer seek to reach a budget surplus by the end of the parliament," Mrs May said.
Now the chancellor has said he agrees, arguing that the government must be "realistic" about its fiscal targets and that austerity policies could be eased.
My Treasury sources point out that the "rule" can be varied in "non-normal" times.
And these are pretty "non-normal" times.
The abandonment of the fiscal target suggests the government could borrow more, presumably for investment in infrastructure and to mitigate the need for tax rises and spending cuts, if the economy does take a turn for the worse as some predict.
More from Kamal: Brexit and the easing of austerity
The Tories have been in turmoil since David Cameron announced his intention to resign following the UK's vote in favour on leaving the European Union.
In a shock development on Thursday, Boris Johnson, widely seen as the frontrunner to become prime minister, announced that he would not be running.
The BBC has learned that Justice Secretary Michael Gove is now coming under growing pressure to abandon his bid to become Tory leader.
Sources have told the BBC government ministers are trying to persuade the justice secretary to give way so the party can "unite" around Home Secretary Theresa May.
Mr Gove opted to stand after switching his support from Boris Johnson.
In a speech, Mr Gove said his decision to stand to become Conservative leader is driven by "conviction" about what is right for the UK not personal ambition.
Three other candidates have put their names forward for the Conservative leadership: Energy Minister Andrea Leadsom, MP Liam Fox and Work and Pensions Secretary Stephen Crabb.
The government borrowed Â£74.9bn in the 2015/2016 fiscal year and for the financial year so far - covering April and May - borrowing reached Â£17.9bn, Â£0.2bn higher than the same period a year ago.
It is thought that the government will now raise borrowing, so it can avoid further cuts in spending or raising taxes. It could also borrow to invest in big building projects.
"The problem of course is, you can't borrow forever,"said Paul Johnson from the IFS.
"So we'll have a few more years of more borrowing, but my guess is this is not the end of austerity, actually this means austerity will just go on for longer because we'll probably have the spending cuts and tax rises right through the 2020s to pay for this."
Chancellor George Osborne has abandoned his target to restore government finances to a surplus by 2020.