Born into a farming family in the unincorporated community of Deer River in Lewis County, Swinburne lost his father at the age of twelve and had to work to support himself as well as assume responsibility for his mother and sisters.
While on a trip to Europe in July 1870, following retirement from the port, he arrived in France at the outbreak of the Franco-Prussian War.