Avery was the grandson of Waightstill Avery (1741-1821), a fiery American Revolutionary War hero who served as the first attorney general of North Carolina and who had once been challenged to a duel by Andrew Jackson.
Avery later formed a partnership with Charles F. Fisher and Samuel McDowell Tate to act as contractors in the building of the Western North Carolina Railroad in the mid-1850s.