With the outbreak of the Civil War, Gillem became a captain on May 14, 1861, initially serving under George H. Thomas.
From June 1, 1863, until the close of the war, with rank of brigadier general of volunteers, he was active in Tennessee, where he was adjutant general.