The £433m motor racing track and leisure project aimed to create up to 6,000 jobs in a deprived area.
But Economy Secretary Ken Skates said job claims were "overstated" and there was too much financial risk.
The developers "strongly disagreed" with the decision and the rationale behind it.
Ministers said a £100m automotive business park would be built instead.
First Minister Carwyn Jones said backing the scheme would mean schools and homes would not be built.
The project, first unveiled in 2011, has been dogged by controversy.
Its backers - the Heads of the Valleys Development Company (HoVDC) - had wanted the Welsh Government to underwrite 80% of the cost, revised down to about 50% when ministers asked for a rethink.
Alongside the circuit itself, which would have hosted the British leg of the MotoGP for at least five years, there were also plans to bring BMX and mountain biking, indoor skiing and concerts.
The issue of the guarantee meant the Welsh Government would have received £99m over 33 years in exchange for underwriting the project once it was open for business.
But in the event of it failing, the public purse would have to pay back £210m over time, while backers Aviva would keep the circuit and the infrastructure.
But Mr Skates said on Tuesday the cost of the underwriting was still too high and would have had an impact on ministers' other spending plans.
He said after discussions with Treasury officials and the Office for National Statistics, it appeared there was a "very significant risk" that the full £373m debt of the project would be classified against Welsh Government capital spending.
Mr Skates added that the true figure on employment was about 100 direct jobs, with 500 indirect jobs and about 500 construction jobs while the track was being built.
Later, during First Minister's Questions, Mr Jones said providing the funding guarantees would have meant his government "would lose £373m of capital funding, £173m would have to be found this financial year".
He told AMs: "That is schools not being built. Hospitals not being modernised. Houses not being built."
Blaenau Gwent council leader Nigel Daniels said he was disappointed with the decision but understood the complex issues involved.
He welcomed the £100m investment in a new automotive technology  park.
But Mr Daniels said he would be "seeking urgent meetings with Welsh Government and other parties to explore other options that can be brought forward to mitigate the impact of this decision given the level of expectation which has grown in the community over the last few years".
UKIP assembly group leader Neil Hamilton said it was a "bean-counters convention".
"What we need to do here is change the accounting convention rather than to destroy the hopes of those who relying on regenerating the whole of south east Wales," he said.
Plaid Cymru finance spokesman Adam Price said it was a "shocking indictment" of ministers that they had "taken seven years and over £9m of public money" to turn down "the single biggest private investment proposal in the history of Wales".
"Is Wales open for business or open to ridicule?" Mr Price added.
Conservative AM Russell George said the decision contradicted comments by senior civil servant James Price on Monday that it was "good value for money".
He called it "deeply concerning" that "less than 24 hours later we learn that the minister takes a vastly different view and that the risk to Welsh Government would be too great".
But there was support for the Welsh Government's decision from Tory Monmouth MP David Davies.
Commenting on Twitter, he said: "Not often I praise Labour ministers but @KenSkatesAM made a good call based on proper scrutiny of business plan."
Analysis by Sarah Dickins, BBC Wales economics correspondent
The company behind Circuit of Wales has said it was "hugely disappointed" and disagrees with the reasons behind the decision. It had seemed confident it was going to be given a green light.
Reaction from the private sector has been divided.
Dr David Williams, from renewable energy firm eco2, described it as sad for inward investment in Wales "if you court big business for seven years move the goalposts and then say no", he tweeted.
But Cardiff lawyer Chris Nott tweeted that it would have been "mad" for Welsh Government to back the project.
There were of course many factors playing into the final decision.
One being the statement from Welsh Government that the project's debt could have had an impact on capital spending on the likes of hospitals and schools.
In the end it came down to perception of whether it would attract enough visitors.  If there had been more belief the project would work then the Welsh Government might have been less concerned about whether a guarantee would be called on.
Ministers were also concerned any companies interested in setting up on the the second phase of the project - a specialist automotive business park - would end up asking for more funding.
Instead the Welsh Government is now spending £100m over 10 years setting up its own automotive technology park in Ebbw Vale which it believes will attract 1,500 jobs. But its neighbouring enterprise zone - recently expanded - has not been successful so far.
Tim Williams, of the Wales Automotive Forum, said: "I can feel the disappointment of the wider industry, the HoVDC and the people of Ebbw Vale that it hasn't been given approval.
"However I respect the Welsh Government's decision on that."
But Mr Williams said the "Plan B" announced by Mr Skates was to be welcomed.
So far, the Aston Martin investment in the Vale of Glamorgan appears to be progressing smoothly while the Welsh Government is confident that companies will want to set up on the automotive technology park in Ebbw Vale .
Those who are desperate for local, stable jobs along the Heads of the Valleys will hope the Welsh Government is not under any illusion.
Martin Whitaker, chief executive of HoVDC, said he was "saddened" that the Welsh Government "failed to support... what would be a game changing development for Wales".
"We have always believed passionately, and continue to do so, in this project's ability to transform and provide opportunities and hope to one of the poorest parts of the UK, not just Wales," he said.
"The project is totally defined, finance is in place, and construction and hiring could start immediately."
A plea by developers of the Circuit of Wales in Ebbw Vale for £210m in taxpayer-funded guarantees has been rejected by the Welsh Government.