A new personal savings allowance of £1,000 will be introduced in April next year, removing the first £1,000 of savings income from income tax.
From later this year, people will be able to move money in and out of Isas without infringing their tax free contribution limit.
And a new "Help to Buy" Isa will be launched to help people save for a deposit.
As announced a few days ago, the government will consult on letting people who have bought annuities with their pension pots sell them, in return for a lump sum.
Mr Osborne said the new personal savings allowance would come into effect in April 2016.
This would, he said, create tax-free banking for almost the entire population.
However, higher-rate taxpayers will benefit from a smaller personal savings allowance of only £500.
Users of the BBC News app tap here for the Budget Calculator.
And anyone earning more than £150,000 a year will not receive the benefit of the new savings allowance at all.
This change, along with more flexibility for Isas, will cost taxpayers more than £1bn in 2016-17 and more than half a billion pounds each year thereafter.
An important knock-on effect of these changes will be that from next April, banks and building societies will no longer deduct 20% income tax automatically from savings held outside an Isa.
The chancellor hailed this as a major piece of tax simplification.
"People have already paid tax once on their money when they earn it. They shouldn't have to pay tax a second time when they save it," he said.
"With our new personal savings allowance, 17 million people will see the tax on their savings not just cut, but abolished."
Richard Lloyd of the consumers' association Which? said: "The tax break and new flexibilities on savings will prove popular with the millions who have got a raw deal on their savings in recent years.
"But there are still many savers whose money is languishing in extremely poor paying accounts, so the financial industry must now play fair and help people get a better return."
On the new Isa rules for what he called a "fully flexible Isa", Mr Osborne explained that so long as Isa money is taken out and replaced during the same tax year, such a move would not count towards the annual Isa contribution limit, which is due to rise in any case to £15,240 this coming April.
"If you take that money out, you lose your tax-free entitlement, and so can't put it back in," he said.
"With the fully flexible Isa, people will have complete freedom to take money out, and put it back in later in the year, without losing any of their tax-free entitlement."
However, this new flexibility will apply to cash Isas only, and not to stocks and shares Isas.
Mr Osborne said the change would be scheduled for this autumn after consultation.
Iain McCluskey of accountants PwC said: "The very significant Isa allowance increase over this Parliament has meant that many savers already pay no income tax on their savings.
"The additional flexibility on Isa contributions is a further move to encourage saving through Isas and will be welcome for those who have to dip into Isa savings on those all too frequent rainy days."
HM Revenue & Customs confirmed that this new flexibility meant that someone could move out their entire Isa savings, accumulated over many years, and then reinstate them within the current tax year to resume their tax-free status.
As a further attempt to help first time buyers save a deposit to buy a home, a "Help to Buy" Isa will be created.
Within this, the government will add 25% to whatever is contributed by a saver.
The monthly maximum contribution from savers will be just £200, with the government adding just £50 a month at the most.
The accounts will run indefinitely once opened.  But if someone accumulates £12,000 over that time, a maximum of £3,000 will be added by the taxpayer.
"Savers will have access to their own money and will be able to withdraw funds from their account if they need them for another purpose, but the bonus will only be made available for home purchase," the government said.
The payment of the government bonus will only be triggered if the saver buys a home worth less than £450,000 in London, or less than £250,000 anywhere else.
Paul Smee of the Council of Mortgage Lenders said: "Such steps are welcome. But as saving for a deposit will never become easy, we still need a clear focus on the supply of new housing that will help deliver a sustainable, affordable housing market over the long term."
The cost of the tax subsidy to the Help to Buy Isa is estimated to rise from £230m in 2016-17 to £835m by 2019-20.
To help pay for some of these tax concessions, the chancellor announced plans to further restrict pension tax relief.
From 2016-17, the lifetime allowance - for pension savings that can be accumulated free of tax - will be cut from £1.25m to £1m.
That will save the taxman £300m in 2016-17, rising to nearly £600m in  2019-20.
From 2018, that allowance will be indexed to the consumer prices index (CPI) and will therefore start to rise again.
Steven Cameron of insurance firm Aegon was disappointed.
"A £1m pension pot may seem huge, but with improvements in health and life expectancy, people who retire at 60 may need to use their pension income to cover their costs for 30 years or more," he said.
"If you want your pension to continue to your partner and rise with inflation, £1m will buy you less than £30k a year. Many people aspire to more than that."
There will be no change to the annual allowance for pension savings, which stays at £40,000.
Mr Osborne said changing that would have involved "penalising moderately-paid, long-serving public servants, including police officers, teachers and nurses, and instead rewarding higher-paid graduates".
The change that will come into place will only affect the wealthiest pension savers.
Government statistics show that among people about to retire, only 4% have pension savings worth more than £1m.
It also justified its move by pointing out that in 2013-14, income tax relief for pensioners was worth a total of about £34bn, and of that about two-thirds went to people in the higher-rate or additional-rate tax bands.
Four big planned changes to encourage saving were outlined by the chancellor.