Havers was the second son of High Court Judge Sir Cecil Havers and Enid Flo Havers (née Snelling) and was the brother of Baroness Butler-Sloss (born 1933) who in 1988 became the first woman named to the Court of Appeal and later President of the Family Division.
He was educated at Westminster School and Corpus Christi College, Cambridge, where he read Law.