In total, 2.2 million patients were not seen within the target time in 2015-16 - more than double the one million figure in 2013-2014.
The Royal College for Emergency Medicine (RCEM) says there is a "large and systemic problem" caused by a lack of hospital beds.
NHS England said hospitals were under pressure but continuing to cope.
Some hospitals publish live waiting times and current patient levels to dissuade people who do not really need A&E from coming.
Live coverage from A&E unit
1,271
waits of 12 hours or more, 2015-16
260
waits of 12 hours or more, 2013-14
Dr Simon Howse, policy research manager at RCEM, said there was no evidence more people were wrongly using A&E compared with a decade ago.
"In any health system some people turn up who are not in great need but studies show the level of unnecessary visits in the UK is very, very low."
The problem in hospitals, he said, was "a large and systemic problem" in which "hospitals are being asked to do something they are not resourced to do.
"They are trying to treat a growing and more needy population with fewer and fewer beds.
"There has also been continuous growth in people over 75 years old with complex needs and they tend to take longer to treat than, say, a 25-year-old."
In terms of hospital beds per capita, he said, the NHS in England now has fewer than Chile, Estonia and Turkey.
Across England in 2015-2016, 85% of patients were seen within four hours. The lowest level was at The Hillingdon Hospitals in London at 68%.
But Dr Howse warned some hospitals were currently recording - in figures not yet published - rates in the 50s and 60s.
These rates matter, he said, because "people are being seriously put at risk" at rates of below 80%.
Neil Carvey, of Leatherhead in Surrey, was advised to go to accident and emergency by a GP after he awoke shivering and in pain following an operation on his spine last December.
His partner Ivanel Petrenkov told how they ended up waiting 16 hours at Epsom Hospital on Christmas Day despite there "being barely anybody there".
It has since emerged Mr Carvey suffered a spinal infection as a result of complications.  He is still in hospital.
Mr Petrenkov, who has friends who work in health care, said: "We have lost all faith in accident and emergency."
The hospital trust - which has seen a 97% rise over three years in patients waiting more than four hours, disputes the couple's 16-hour wait claim, saying: "During December, no patient waited in A&E for 16 hours to be seen."
A spokeswoman for the hospital said it had "an excellent track record" of meeting a 95% target for dealing with patients within four hours.
"That said, in recent weeks we have seen unprecedented levels of demand on our services and in some cases, that has meant some people have experienced longer waiting times," she said.
The largest increase percentage-wise in waits of four hours or more were seen at North Middlesex University Hospital, with a 380% rise, the Royal Free Hospital in London, with a 366% rise and Mid Yorkshire Hospitals at 351.5%.
Pennine Acute Hospitals has experienced the biggest rise in patients waiting more than four hours.  In 2013-2014, 13,867 people were not dealt with in four hours. In 2015-2016, that number rose 296% to 54,945.
The trust's medical director Prof Matt Makin said: "Our (three) accident and emergency departments have continued to face real pressures throughout the year and we know demand on our services further increases over winter.
"Like most trusts across the country, we are finding this a challenge due to the flow of patients in and out of hospitals and the large numbers of admissions of patients, particularly those who are elderly and with complex and chronic health conditions.
"We are sorry that some patients have to wait longer than we would like to be seen by a doctor and also those who are waiting to be admitted and taken to the ward."
In January, some patients at the Shrewsbury and Telford Hospital NHS Trust ended up waiting 20 hours or more to be dealt with.
The hospital said the situation arose out of "incredibly high levels of demand", but none of those waiting 20 hours were "serious" cases.
In an open letter, the hospital's chief executive Simon Wright said: "It's a fact that our acute emergency service has been frail for some time. It is no secret that health budgets - nationally and locally - are stretched."
NHS England did not respond to the RCEM's claims that the rise in delays was largely caused by a shortage of hospital beds and resources.
In a statement, a spokesman for NHS England blamed delays in being able to discharge patients as a result of pressures in social care.
"Hospitals report this affects their ability to quickly admit emergency A&E patients, so the NHS is working closely with local councils and community health services to enable older patients to get the support they need after a hospital stay, back at home," he said.
"Hospitals are coming under increasing pressure but in the main are continuing to cope."
The number of patients waiting four or more hours at A&E has risen more than 300% at some hospitals, it has emerged.