 He was the son of Robert James Livingston (1725–1771) and Susanna (née Smith) Livingston (1729–1791), daughter of Chief Justice William Smith (1728–1793).His brothers were Col. William Smith Livingston (1755–1795) and Judge Maturin Livingston (1769–1847).They were among the many great-grandchildren of Robert Livingston the Younger (1663–1725), through their grandfather, James Livingston (1701–1763), Younger's eldest son.