The island served as a prison for Union soldiers during the American Civil War.
In April 1864, Peter DeWitt, Assistant Surgeon at Jarvis Hospital, Baltimore, received a number of prisoners recently released from the Prisoner of War camp at Belle Isle.