An uncle of the 3rd Earl, George Henry Lee II, he was a son of Edward Lee, 1st Earl of Lichfield and his wife Charlotte Fitzroy, an illegitimate daughter of Charles II.
Lee held the sinecure position of Custos Brevium of the Court of Common Pleas, in the royal gift.