Gantt was born to a slave-owning plantation family in Calvert County, Maryland at the outbreak of the American Civil War.
He graduated from McDonogh School in 1878 and from Johns Hopkins University in 1880, then returned to the McDonogh School to teach for three years.