It comes as figures suggest a big spike in the number of GP practices being handed back to health boards to manage.
The British Medical Association (BMA) said GPs "can't simply carry on working under the stresses and strains they're facing".
A senior health official acknowledged a greater pace of change was needed.
Measures include trying to recruit more nurses, physiotherapists and pharmacists to become part of "multi-disciplinary teams" in primary care to relieve the pressure on GPs.
But the BMA has warned more family doctors could quit unless surgeries get more money.
The concerns are also echoed by a group representing grassroots GPs.
"Yes there is a GP crisis - there's a crisis of numbers, there's a crisis of funding and there's a crisis of demand," said Sophie Quinney, GP and representative of GP Survival Wales.
"Patients are struggling to see their GP because the workload is becoming unmanageable."
Meanwhile, the BMA said despite frequent warnings, changes designed to relieve the pressure were happening too slowly, with no improvement seen in three years.
Dr Charlotte Jones, chairwoman of the BMA's GPs committee in Wales, said it was hearing "week on week" of doctors being unable to carry on.
She said there was a "perfect storm" of GPs nearing retirement, recruitment difficulties to find younger replacements, longer working hours and more complex needs of patients.
Dr Jones said despite new strategies coming in to try to deal with the problem, the pace of change was not moving quickly enough and she had seen no improvement in the last three years.
"I would say things are getting worse and that's exactly what I'm hearing from my colleagues across Wales and that's being borne out now by the number of practices saying enough is enough," she said.
She said the health service was working hard at a national and local level to address issues "but I'm afraid change is not happening quickly enough".
Dr Jones said Wales was better placed than other parts of the UK, but added: "Health boards have to grab the nettle, put the investment in and start making the change, otherwise the situation is going to get significantly worse."
The BMA said patients in "large swathes of north Wales" now had directly-managed GP practices and they did not see the same doctors a lot of the time.
GP SURGERIES HANDED BACK
The concerns appear to be backed up by figures from health boards which show the numbers of GP surgeries across Wales having to be taken back into central control because doctors have given up.
A total of 33 GP practices were "handed back" to health boards across Wales over a five year period between 2010 and 2015.
This included 17 surgeries which closed. But in the past year, figures from health boards obtained by GP Survival Wales, show a spike in some areas.
In the south Wales valleys, there were four surgeries handed back in Cwm Taf in 2015/16 alone - twice the total in the previous five years.
Across Wales, 20 practices were handed back in the last year - but proportionately fewer of these - five - closed.
It suggests health boards are finding temporary solutions, including direct management of surgeries, rather than allowing communities to lose them.
Official figures show in 2015 the number of GP partnerships in Wales was just under 2,000 - 8.7% lower than in 2005.
Meanwhile, the number of GPs for every 10,000 people in Wales is 6.5 - lower than for the other UK nations.
Dr Jones said there was also a resources issue and more of the NHS budget needed to be spent on GP contracts to offer a "glimmer of hope" to carry on until longer-term changes could be introduced.
She said  the percentage spend of NHS Wales' budget on GP contracts had dropped this year to 7.6%.
A senior GP appointed by the Welsh Government last October to try to tackle the problems admits changes need to be "scaled up" and "happen at a "quicker pace".
Dr Richard Lewis, a former secretary of the BMA in Wales, is now national clinical lead for primary care in Wales.
He conceded the Welsh NHS would "need more GPs", adding there was "a huge amount still to do" to address the challenges.
In his first interview in his role, Dr Lewis said health systems were extremely complex so "making changes were not as fast as any of us would like".
He said Wales was not unique in facing challenges and primary and community care were being put "front and centre" in changes being brought in.
"Some areas are finding it far easier to recruit than others," he said.
"Some areas, particularly where we have high demand - from an ageing population and chronic and multiple disease - that workload and demand is very high, which makes it less attractive to certain groups of health professionals."
GPs in Rhondda - where up to half of family doctors are over 55 - have got together to create a new website aimed at tackling recruitment issues.
The area traditionally has one of the lowest numbers of GPs per head of the population in the UK.
The Rhondda Docs website and recruitment campaign was launched last week and has already had hits from as far afield as the Middle East.
Dr Ian O'Sullivan, based in Porth, has 8,000 patients on his list.
"There's been a surgery which has closed further up the valley and there are two surgeries under the management of the local health board so we need to change things to see what we could do to improve things for the patient," he said.
Surgeries are also working together as a cluster to improve services, including recruiting pharmacists to take pressure off doctors.
In his previous role at the BMA, Dr Lewis warned three years ago of an "impending crisis" in GP recruitment.
But he says now: "I don't think there's a crisis within health services in Wales currently. We have in place the appropriate plans to address the huge challenges facing all health services.
"We need increasing confidence in Wales. We need to involve and engage people more with the solutions."
The "crisis" facing GP services in parts of Wales is "getting worse" with more practices than ever facing an uncertain future, say doctors' leaders.