Holladay was born October 14, 1819, in Nicholas County, Kentucky. His father, William Holladay (born in what is now Spotsylvania County, Virginia) was a third-generation American, descended from John The Ranger Holladay. William migrated to Bourbon County, Kentucky, where he was a guide for wagon trains through the Cumberland Gap. Benjamin's mother was Margaret Peggy Hughes. Benjamin Holladay learned the freight business at an early age and left home in his late teens for a road trip to Santa Fe in what was then Mexico.