He was commissioned adjutant of the 4th Vermont Infantry on August 1, 1861 with the rank of Captain, and mustered into military service on September 21.
Stoughton led the regiment at the Battle of Fredericksburg, on December 13, 1862, where the 4th Vermont suffered 56 casualties.