The son of William Chaytor (MP) of Croft and Spenningthorne (1732 - 1819), , Member of Parliament for Hedon (UK Parliament constituency), Recorder of Richmond and Vice-Lieutenant for the North Riding of Yorkshire, and great-grandson of Sir William Chaytor (bart.
He was succeeded by his eldest son, the second Baronet, who represented Durham in the House of Commons.