Five justices in the Supreme Court dismissed the challenge for an inquiry into the shootings at Batang Kali in 1948 by a four to one majority.
The families, who say the men were "massacred", had their case rejected by the UK Court of Appeal last year.
British forces at the time of the killings said the men were insurgents.
In the Supreme Court panel's judgement, one of the justices, Lord Kerr, described the case as "shocking" and said the "overwhelming preponderance of currently available evidence" showed "wholly innocent men were mercilessly murdered".
He said it was "with regret" that he dismissed the appeal.
"This is an instance where the law has proved itself unable to respond positively to the demand that there be redress for the historical wrong that the appellants so passionately believe has been perpetrated on them and their relatives," Lord Kerr said.
"That may reflect a deficiency in our system of law. It certainly does not represent any discredit on the honourable crusade that the appellants have pursued."
Another of the justices, Lady Hale, said she would have allowed the appeal and said the UK government "did not seriously consider the most cost-effective form which such an inquiry might take".
"They did not seriously consider the "bigger picture": the public interest in properly inquiring into an event of this magnitude; the private interests of the relatives and survivors in knowing the truth and seeing the reputations of their deceased relatives vindicated," she said.
The appellants in the case - Chong Nyok Keyu, Loh Ah Choi, Lim Kok and Wooi Kum Thai, two of whom were at Batang Kali as children - were supported by the action group Condemning The Batang Kali Massacre, a campaign in Malaysia that includes 568 civil society organisations.
Michael Fordham QC, representing the appellants, told the court that the families wished to "vindicate the legitimate interests of the deceased, in order to achieve justice, before they die themselves".
Relatives argued that Article 2 - the right to life - of the European Convention on Human Rights imposes a duty on the UK to commission an independent inquiry, despite the killings occurring before the convention was drafted and signed.
It was argued on behalf of the government that the Human Rights Act "does not have retrospective effect and does not impose an obligation to hold an inquiry into deaths occurring several decades before it came into force".
The account of the British authorities at the time was that the deceased were justifiably shot while they were attempting to escape from the patrol.
An official explanation was contained in a written parliamentary answer in Hansard, published on 26 January 1949.
On 11 and 12 December 1948 - when Malaya was still a British colony - 24 villagers were killed by a platoon of Scots Guards during a raid at Batang Kali.
The men were Chinese migrant workers suspected by the British of helping rebels during the Malayan Emergency - a conflict between communist guerrillas and British and Commonwealth forces, which lasted 12 years.
An investigation at the time cleared the soldiers of wrongdoing, but in 1970 some of the soldiers said the villagers had been executed.
In the 1990s, authorities in Malaysia opened an investigation, but it was halted before a conclusion was reached.
There have been numerous calls for a public inquiry - all of which have been rejected by the UK.
Relatives of 24 rubber plantation workers killed by British troops almost 70 years ago in Malaya have lost an appeal for an official investigation.