 Brown had four siblings; Minnie, John (who died at sixteen), Lula, and Grover.Brown married Annie Love Sitgreaves, daughter of Edwin McBurney and Centellia Martin Sitgreaves, on December 30, 1913 and together they had one child, Emily McBurney Brown.Brown was also survived by a grandson, Richard Allan Jefferies, and a granddaughter, Emily McBurney Jefferies.