Private domiciliary care firms commissioned by local authorities say there is a recruitment crisis due to competition from hospitals and supermarkets.
The UK Homecare Association said there was a "real sense of desperation".
Public Health Minister Rebecca Evans said action would be taken.
Colin Angel - of the UK Homecare Association, the body which represents care companies - told the Week In Week Out programme many of his members were really struggling.
"What we hear in Wales is a real sense of desperation from some providers trying to work out how they can remain in business on the sorts of rates that they are being paid by local councils," he said.
"And I think in some parts of Wales, particularly rural areas, we may see care providers handing back work or going out of business even more quickly than in the rest of the country. And that's something that I hope the Welsh Government is going to take seriously."
Private domiciliary care firms provide 13 million hours of care every year in Wales.
One care company in North Wales "reluctantly" pulled out of providing care to Conwy council, saying it could not supply the care required for the money the council was prepared to pay.
Thirteen of the Wales' 22 local authorities said they had seen contracts handed back to them, when asked by BBC Wales.
The proportion with returned contracts - 59% - compares to a UK average of 48%.
Cymorth Llaw has provided domiciliary care in north Wales for 17 years and company boss Ken Hogg said it was getting more and more challenging.
While the company is not in danger of going bust, profits have been falling.
"There will be difficult decisions to be made. We've made difficult decisions in the past," Mr Hogg said.
"We have to stop this crisis in social care and we have to make these professional carers feel valued, not only valued members of society, but also the remuneration that they deserve."
Last year the company pulled out of providing care in Conwy - which initially paid £14.20 an hour for care. Conwy offered to raise that to £15.00, but the company decided they had no choice but to give up the contract.
"We didn't think that we could provide this level of service for that cost, for that amount of money that Conwy were offering. And we were very, very reluctant to leave, but we had to leave, it was as simple as that," Mr Hogg added.
Conwy council said it was committed to supporting vulnerable people in communities, despite facing financial challenges.
Care worker Amanda Hopewell, of Cymorth Llaw, is paid £7.55 per hour - just above the national living wage.
She is on a zero-hours contract, which means the hours she works are not guaranteed. She said she struggled to make ends meet.
"I did look into buying a house about four years ago, but because I didn't have a contract they wouldn't allow me to buy a house or anything like that," she said.
"It's hard. You see all your friends out every weekend, going for supper with each other, and you're like 'No, I can't come'."
Steve Thomas, from the Welsh Local Government Association, said councils needed more money to be able to pay carers a better wage.
"As it stands, unless more money comes into the system, we've got a real problem. Nobody's proud about the fact [of] how much money we pay people who work in the social care sector," said Mr Thomas.
Earlier this month, the UK government announced an extra £2bn of funding over three years for social care in England.
In Wales, there is an extra £200m, but the Welsh Government has yet to decide how to spend the money.
Rebecca Evans, Minister for Social Services and Public Health, said it was up to the Welsh Government's cabinet to decide how to spend the extra cash.
She said action was being taken to address recruitment problems in the care sector and that the introduction of a registration scheme for domiciliary care workers in 2020 would help.
"That will give the workforce the kind of kudos it deserves, because looking after our most vulnerable people - what more important job is there than that?" she said.
"And it will also give people going into that field of work the opportunity to think about career progression because we know that's not there for people at the moment as well."
The Minister also said the Welsh Government had invested in the sector to raise the terms and conditions of employees, including extra money for local authorities to meet an increase in the national living wage in April.
Firms caring for thousands of people living at home in Wales are facing tough financial pressures to stay open, a care companies body has said.