From April 2017, 600,000 small firms will not have to pay business rates, while 250,000 will pay lower rates.
To fund the giveaway, the chancellor capped debt interest payments used by larger firms to cut their corporation tax bills, to 30% of earnings.
George Osborne also said corporation tax would be cut to 17% by 2020.
Mr Osborne said his business tax reforms were part of a "Budget for small business".
The chancellor announced he was permanently raising the threshold for small business rate relief from £6,000 to a maximum of £15,000 and increasing higher rate relief from £18,000 to £51,000.
He said that the 600,000 small businesses relieved of paying business rates from next April would see an annual saving of nearly £6,000.
Mike Cherry, policy director at the Federation of Small Businesses, said: "The chancellor has listened to our calls for the tax system to be made simpler for small businesses and the self-employed and taken important action on business rates."
The chancellor also announced a number of reforms designed to help small businesses cope with what he called "the great unfairness" they faced when trying to compete with some suppliers selling goods online.
Mr Osborne said the government would take measures to stop overseas retailers storing goods in the UK and then selling the goods online without paying VAT.
He also introduced two new tax-free allowances worth £1,000 a year for so-called micro-entrepreneurs:  people who make money from occasional jobs or through renting out property they own.
The Treasury estimates that the debt interest rate relief cap, alongside other measures, will help raise £8bn over the next five years.
Among those other measures was the announcement that firms making an annual profit of over £5m would face the prospect of having the amount of past losses they could carry forward to offset their corporation tax bill capped to 50% of current profits.
Carolyn Fairbairn, director general of the CBI business lobby group, said: "The reduction in the headline corporation tax rate sends out a strong signal that the UK is open for global business investment, and reforms to interest deductibility are rightly in line with the international consensus.
But she added: "Changes to the tax treatment of losses will make it harder for larger scale-up firms and companies that have been through tough times to play their part in the recovery."
Small businesses have welcomed what they said were long overdue reforms to tax policy as the chancellor doubled business rate relief.