Bryan Fairfax was the son of Col. William Fairfax (1691-1757) of Belvoir and Deborah Clarke (1708-1746).
In 1793, when his cousin, Robert Fairfax, 7th Lord Fairfax of Cameron, died in 1793, he initially ignored the title.