Fortier was born in St. James Parish, on the plantation of his grandfather Valcour Aime, Petit Versailles Plantation.
Fortier's life was marked by the interruption of the American Civil War, and his family's abrupt changes in fortune following the war.