Under the adverse conditions, RBS's capital levels fell by 7.5 percentage points - the third biggest fall of the 51 banks tested.
However, RBS said the tests showed it had made "continued progress".
RBS was bailed out by the government in 2008 and the UK taxpayer continues to hold a 73% stake in the bank.
The health check of 51 lenders in the eurozone and the rest of the EU, including the UK, was carried out by the London-based European Banking Authority.
It examined the impact of GDP falling by about 7% between now and 2018.
Why are the banks being tested?
The Bank of England said the results for RBS and three other UK banks - Barclays, HSBC and Lloyds - were "consistent with those of previous Bank of England stress tests".
"They provide evidence that major UK banks have the resilience necessary to maintain lending to the real economy, even in a macroeconomic stress scenario," it added.
Under the test conditions, RBS was left with a capital buffer of just over 8%.
Ewen Stevenson, RBS chief financial officer, said: "The EBA stress test results demonstrate our continued progress towards transforming the balance sheet to being safe and sustainable.
"We are confident that in delivering our strategy, we will transform RBS into a low-risk, resilient bank."
Conservative former chancellor Lord Lamont urged people not to be overly concerned about RBS's position, saying it was still "well above the minimum capital" that would be required - and saying the real danger to the European economy came from continental institutions.
He told BBC's Today programme there was "work to be done" but attention should be focused on banks in Germany and Italy, as well as Portugal and Greece which were not tested because they were considered too small.
"That is where it is thought there is a lot of weakness in the European banking system," he said, warning the situation in Italy and other countries "could create a real political crisis" for Europe.
Under the test conditions, Barclays' capital buffers would fall by 4 percentage points in the event of a major economic shock, leaving it with a buffer of 7.3%.
Unlike in previous years, the EBA did not judge whether banks had passed or failed its latest tests.
In 2014, if banks had a capital buffer of 5.5% after the stress test, then they were considered healthy, and analysts use that as an informal benchmark.
Both RBS and Barclays surpassed that mark in the latest test.
Allied Irish Banks, which was bailed out by Irish taxpayers, showed a near 9 percentage points fall in capital levels in the test.
"AIB is well-capitalised and capital accretive," the bank said in a statement.
"The results published today are point-in-time projections based on prescribed stress assumptions and should not be treated as indicative of the future financial performance."
Italy's Monte dei Paschi di Siena was by far the worst performer, with the test forecasting that its capital would fall by 14 percentage points under the adverse conditions.
Shortly before the results of the stress test were released, Monte dei Paschi di Siena announced that it had secured the backing of a consortium of banks for a rescue plan.
The plan involves the sale of €9.2bn (£7.7bn) of bad loans and an injection of €5bn of fresh capital.
Founded in 1472, Monte dei Paschi is one of the world's oldest banks, but in recent years has been one of Europe's weakest, with €50bn of bad loans.
Analysts were also keen to see how German banks performed under the test conditions.
Deutsche Bank and Commerzbank were left with capital buffers of less than 8% at the end of the test.
"Commerzbank is robust and stress resistant," Commerzbank chief risk officer Marcus Chromik said in a statement.
"Even under the adverse conditions of the EBA stress scenario, the stability of the Bank would be guaranteed."
After the financial crisis of 2008, US banks took hefty charges to clean up their balance sheets, but European banks were much less aggressive, leaving them with billions of euros of poorly-performing loans.
"Whilst we recognise the extensive capital raising done so far, this is not a clean bill of health," EBA Chairman Andrea Enria said in a statement.
"There remains work to do."
RBS Group performed comparatively poorly in the latest European stress tests, which assess how the banks might perform in adverse economic conditions.