At 10 years old, Williams' family moved to Adelaide so that he and his two sisters could attend school there.
After his marriage broke down in the 1950s, Williams purchased 55 hectares of land at the rear of Yatala Labour Prison South Australia.