Garrard was born at his paternal grandfather's home in Bourbon County, Kentucky, while his mother was visiting there.
In 1855, Garrard was transferred to the 2nd U.S. Cavalry as an adjutant to Colonel Albert Sidney Johnston and Lieutenant Colonel Robert E. Lee, both future generals in the Confederate States Army.