Many of the country's 32 local authorities had voiced concerns about a 2% fall in their income.
Mr Swinney told the local government and regeneration committee he did not underestimate the "on-going challenges".
However, he believed councils had been given a "very credible settlement".
I think some of the talk has been frankly over the top about the impact of the settlement for local government.
The minister said that in 2016/17, local authority spending would be cut by £350m, which equates to a 2% reduction.
Mr Swinney insisted that headline figure was not the full story.
He explained to MSPs: "When we take into account that £250m is going to be spent through the integrated joint boards to support expenditure on services which local authorities are key participants, and given the guidance that I have issued to local authorities as to what they can expect that fund to support, that result is a net reduction in the local authority budget of about £100m, out of £16bn, which equates to less than 1%."
For the last seven years local authorities have carried out a deal with the Scottish government whereby council tax bills have remained frozen.
However, this year Moray Council considered a move to increase council tax bills by 18% to ease the cuts burden.
It decided not to go ahead with the plan because it would have resulted in the Scottish government withholding £1.1m to offset the freeze.
Council leader Stewart Cree said: "We simply cannot proceed with the proposal as the extra penalties it would now attract would have a devastating effect on the services people in Moray tell us they want protected.
"So to make ends meet this year the administration have decided to defer certain works and spending, and to draw the remaining shortfall from reserves."
Responding to a question about Moray Council's proposal to drop the tax freeze, Mr Swinney said: "I think some of the talk we have heard has been frankly over the top about the impact of the settlement on local government.
"I don't underestimate the on-going challenges in delivering public services within a constrained financial environment."
He added: "In all, I think the settlement offered to local government is a very credible settlement, it certainly doesn't merit the type of description it has had from certain voices within local authorities and it most definitely does not merit an 18% increase in the council tax."
Separately, Scotland's first minister has called on councils to "do the right thing" and accept the funding deal being offered by the Scottish government.
Nicola Sturgeon insisted local authorities were being given a "good deal" as part of the budget, with a package containing resources that will help them pay the living wage to care workers.
She made the plea as the 9 February deadline loomed for local authorities to sign up to the financial package.
Scotland's Finance Secretary John Swinney has said talk about reduced council budgets impacting on local services was "frankly over the top".