 Larrabee was born in Ledyard, Connecticut, into a family of French Huguenot extraction.His father, Adam Larrabee (1787–1869), was a West Point graduate and an accomplished soldier, who served with distinction in the War of 1812.His mother was Hannah (née Lester).