The government says GPs must try harder to stay open from 08:00 to 20:00, seven days a week, so fewer patients are forced to seek care at hospitals.
It warned GPs risk losing extra funding if they fail to meet their commitments to longer surgery opening times.
But the BMA and Labour said underfunding of the NHS was to blame.
Dr Chaand Nagpaul of the BMA told BBC Radio 4's Today programme: "We have got a very serious problem that we don't have the capacity in general practice.
"The crisis in the NHS won't be solved by scapegoating or deflecting blame on to GPs."
One in three GP practices were reporting unfilled vacancies, while eight in 10 said they were unable to provide safe care, he added.
Why bother with seven-day GP opening?
Patient films A&E corridor bed 'chaos'
Leak shows full extent of NHS crisis
Pressure on A&E was down to seriously-ill patients, he said, not access to GPs, and the government needed an emergency plan to tackle NHS under-resourcing.
Professor Helen Stokes-Lampard, chairwoman of the Royal College of GPs, said the "whole of general practice and primary care" was "close to the precipice" after being "under-funded and under-resourced for a decade".
32,628
GPs in England
5,000 more planned for 2020
1 in 3 considering retirement in next five years
13% of GP training places went unfilled last year
Surgeries are currently expected to open between the core hours of 08:00 and 18:30, Monday to Friday. Extra funding is available to those offering appointments outside those hours.
Downing Street said surgeries should do more to ensure they offered appointments in the evening and at weekends.
"Most GPs do a fantastic job, and have their patients' interests firmly at heart," it said.
"However, it is increasingly clear that a large number of surgeries are not providing the access that patients need - and that patients are suffering as a result because they are then forced to go to A&E to seek care.
"It's also bad for hospitals, who then face additional pressure on their services."
The government highlighted October 2015 figures from the National Audit Office (NAO), which showed that 46% of GP surgeries closed at some point during core hours, and 18% closed at or before 15:00 on at least one weekday.
Three-quarters of those that closed early were receiving extra funding in 2015-16 to provide access outside of core hours, the NAO said.
More than four in 10 hospitals in England declared a major alert in the first week of the new year.
The director of acute care for NHS England, Professor Keith Willett, has estimated that 30% of patients attending A&E would be better cared for elsewhere in the system, the government said.
Downing Street said the prime minister wanted to help reduce pressures on hospitals in a number of ways:
Dr Matt Owen, from Manchester, qualified as a GP three years ago.
"I would say demand for a seven-day week is actually not high.
"I do work extended hours on the weekend. On Sundays, it is not uncommon to be sat in my consultation room for 12 hours (08:00-20:00) with as few as five patients for the whole day.
"However, there can be a shortage of appointments and access to the GP Monday to Friday.
"The problem here is actually a shortage of GPs. No-one wants to be a GP because GPs are overworked, overused and over-regulated."
Health think tank the Nuffield Trust's chief executive Nigel Edwards said the problem was capacity, not timing.
"If you haven't got more GPs and more practice nurses to do it, you haven't increased the number of appointments, you've just put additional pressure on a service that's already under a very severe level of pressure," he said.
The government said a 14% increase in funding to general practice would ensure there would be about 5,000 more doctors by 2020.
Labour leader Jeremy Corbyn accused the prime minister of being in denial about the state of the NHS.
In a speech in central London, he said Theresa May was blaming the "crisis" on "hard-pressed and under-pressure GPs", not underfunding by the government.
Later, shadow health secretary Jon Ashworth told the BBC that the Conservatives had been promising a seven-day GP service for more than six years.
He said Mrs May's response to the "crisis" in the NHS was "increasingly incompetent and floundering", and urged her to "get a grip of the situation".
The doctors' union, the BMA, has accused the government of making GPs in England scapegoats for the pressure on A&E departments.