Graves was born in Hope Hull, Alabama, son of David and Mattie Bibb Graves and a descendant of Alabama's first governor William Wyatt Bibb.
As adjutant general of the Alabama National Guard, he helped organize the 1st Alabama Cavalry and served on the Mexican border in 1916.