It said a 2014 Supreme Court ruling that widened the definition of who was subject to the Deprivation of Liberty Safeguards had led to a rise in cases.
Services cannot cope, deadlines were "routinely breached" and the system should be replaced, the body reported.
The government commissioned the report and said it would review the law.
The Law Commission, an independent body which reviews laws in England and Wales, made several recommendations to improve the system.
These included requiring all decision-makers to put greater weight on the person's wishes and feelings when making decisions under the Mental Capacity Act.
When a vulnerable person - usually suffering from a severe learning disability or dementia - in a care home or hospital has limits put on what they can do or where they can go for their own safety, the institution must apply to the local council to authorise the deprivation of liberty.
The deprivation of liberty safeguards (DoLS) should ensure that a care home, hospital or supported living arrangement only deprives someone of their liberty in a safe and correct way, and that this is only done when it is in the best interests of the person and there is no other way to look after them.
The system, under the Mental Capacity Act, requires that the care arrangements are independently checked.
A landmark ruling from the Supreme Court in 2014 - also referred to as the Cheshire West decision - widened the definition of who was subject to the Deprivation of Liberty Safeguards (DoLS).
This triggered an increase in the number of cases, which the commission said has left health and social care services "unable to cope".
Official figures show that hospitals and care homes in England made 195,840 DoLS applications in 2015-16 - more than 14 times the 13,700 in 2013-14.
The commission said the increased workload had resulted in a rising number of DoLS referrals that were not assessed and statutory timescales were "routinely breached".
In England, out of the 195,840 DoLS referrals during 2015-16, only 43% were completed in the year, the commission said.
Of those, only 29% were completed within the 21 day time-limit set in regulations.
It affects vulnerable people with dementia and learning disabilities and their families.
The Law Commission's report provided examples of cases where the current system did not work.
Steven Neary, a young man with autism and learning disabilities, lived with his father, who reported in December 2009 that he was having difficult coping.
The local authority arranged for Steven to stay in a residential support unit and his father agreed to an extended stay of a couple of weeks.
But Steven was detained there by the local authority for a year, including a period when he was subject to the DoLS regime, and the Court of Protection held that Steven had been unlawfully detained.
The court noted that the local authority did not properly discuss its concerns or its plans with Steven's father.
The Law Commission is calling for the current system to be scrapped and replaced "right away".
Law Commissioner Nicolas Paines QC said the Deprivation of Liberty Safeguards were designed at a time when fewer people were considered deprived of their liberty and now it was "failing" people it was set up to protect.
"It's not right that people with dementia and learning disabilities are being denied their freedoms unlawfully," she said.
"There are unnecessary costs and backlogs at every turn, and all too often family members are left without the support they need."
The commission wants to replacing the law with a new scheme called the Liberty Protection Safeguards. Its recommendations include:
A Department of Health spokesman said it was "committed" to protecting the rights of vulnerable people, which is why it commissioned the review.
"We thank the Law Commission for its detailed work, and will be responding to these constructive recommendations in due course."
The Department of Health said it provided Â£25m of additional funding for 2015/16 to local authorities to support them in managing the increase in DoLS applications.
People with dementia and learning difficulties are being detained in care without checks due to a 'failing' law, the Law Commission has said.