 Hillyer was born in Athens, Georgia, one of eight children of Judge Junius Hillyer, a United States Congressman and solicitor of the U.S. Treasury, and Jane Selina Watkins.He graduated from Mercer University in July 1854, "securing honors in all branches of study".He then studied law under the supervision of his father, Judge Hillyer, in Monroe, Georgia and in 1855 was admitted to the state bar.