 Brown was born near Shiloh Springs in Aiken County to Augustus Abraham Brown and Elizabeth Howard Brown.He, accompanied by his oldest sibling, Minnie, attended the China Springs School close to Edgefield Road.He was also educated at The Graniteville Academy which was maintained by the Graniteville Company.