Roberts was born in Bristol, Virginia (known as Goodson until renamed after the English town in 1890) to Henry B. Roberts (1846 - 1913) and his wife Mary Catherine Hortenstine Roberts (1847 - 1906).
He married Eliza Clapp Roberts (1880 - 1967), and they had three sons who survived their parents--Bradley Roberts (1908 - 1976), Randolph Roberts (1911 - 1992), and James Clapp Roberts (1917 - 1995).