Yager was born and raised in Page County, Virginia, a son of Nicholas Wesley Yager and Christina Williams Overall Yager.
In 1848, he matriculated at the Virginia Military Institute, graduating four years later, fifth in his class, with fellow Page countians Simeon Beauford Gibbons and Hiram Jackson Strickler, as well as later Confederate notables, Thomas T. Munford, James A. Walker, Joseph C. Mayo, and George Smith Patton, Sr.