He was the son of Eliza Berkeley née Frinsham, and her husband the Rev. George Berkeley, a son of George Berkeley the philosopher, born on 8 February 1763 at Bray, Berkshire.
In poor health, Berkeley went for the benefit of the sea breezes to Dover.