Fleming was born in Rogersville, Tennessee, the son of David Fleming, a Methodist minister, and Mary (Miller) Fleming.
In 1857, Fleming quarreled with Irish Patriot John Mitchel, who spent time in Knoxville while in exile and befriended the city's Democrats.