Patrick Joseph Connors, 59, his son Patrick Dean Connors, 39, and nephew William Connors, 36, denied requiring a person to perform forced or compulsory labour between 2010 and 2013.
They were convicted by a jury at Cardiff Crown Court on Tuesday.
Connors senior was jailed for 14 years, Patrick Dean for six-and-a-half years and William for four years.
Patrick Joseph Connors' son-in-law Lee Carbis, 34, who was cleared of forced labour, but found guilty of kidnap, was jailed for two-and-a-half years.
Catrin Evans, of the Crown Prosecution Service in Wales, said the men were guilty of "exploiting and controlling" their vulnerable victim in a "callous manner over a prolonged period of time".
'Modern slavery' victim 'emotionally broken'
The Connors kept Michael Hughes, 46, in "appalling conditions" in south Wales and paid him less than £10 a day for years.
He said: "I am over the moon at today's verdict. Both myself and my family can now move on and I am so happy to have my life back - a life that I never knew existed.
"Those people stole my life but the outcome of today's trial means that they will now face the justice that they deserve."
Judge Neil Bidder QC said the three Connors, who operated a tarmacking business based at a farm in Rumney, treated their victims as little more than objects.
He said: "All the offences which you, Patrick Joseph Connors, have been convicted of are examples of conduct designed to keep two vulnerable men in what can only be described as modern-day slavery - in the case of Michael John Hughes, for 21 years.
"By the end of that time he had been completely conditioned to being used by you as a serf."
The court heard over the course of the trial, Patrick Joseph Connors forced Mr Hughes to work for him for long hours, paying him tiny sums or just with alcohol and tobacco.
Mr Hughes was "hunted down" when he escaped and beaten "all the time", he said in evidence.
He was made to live in "appalling conditions", including in a 1.2m (4ft)-wide garden shed with no heating or running water for two years.
The court heard the only respite came when Mr Hughes was sent back to Scotland for unpaid fines and jailed. He said prison was "like a holiday camp" compared to his ordeal in Wales.
Connors senior was also convicted of eight counts of assault occasioning actual bodily harm (ABH), four of kidnap and one charge of conspiracy to kidnap, while Patrick Dean Connors was also found guilty of kidnap.
The jury was told to return not guilty verdicts on a charge of conspiracy to kidnap against Patrick Dean Connors and another of assault against William Connors.
Gwent Police's Ch Supt Paul Griffiths, who led the investigation, said: "These types of crimes have a lasting effect on the victims - it is not just the physical degradation, it's psychological too. It causes deep and long-lasting trauma."
Three family members have been jailed for forcing a man to do heavy labour for tiny amounts in Cardiff.