Dry Tortugas National Park, Florida (CNN) -- Weather and time have inflicted more damage to Fort Jefferson than hostile cannon fire ever did. 

The crumbling citadel rises 40 feet from the turquoise water of the Gulf of Mexico, an outpost of a bygone era. Built on an island in Florida's Dry Tortugas, 70 miles past Key West, the fort is being restored by a squad of craftsmen who spend three-week shifts there between hurricane seasons. 

A cocoon of mesh and scaffolding shrouds one section of its 8-foot-thick walls as they work. It's a grueling job, but it's one Dennis Wood, a mason from Massachusetts, says he's glad to have. 

"Things are slow today, you know," Wood said. "Got a job offer to come down here for the winter, so we took it." 

Efforts to restore Fort Jefferson have been under way for about 30 years. The current phase received a boost from the economic stimulus bill that passed Congress in 2009, which devoted $7 million to the project. But planners estimate another $13 million is needed, and that money has yet to be allocated. 

The six-sided fort, roughly the size of Yankee Stadium, was designed to protect shipping lanes through the Gulf of Mexico. Construction began in 1846, but advancements in the science of artillery rendered it obsolete before it was finished. During the Civil War, the Confederacy never bothered to try to seize it from Union forces. 

The Union kept a garrison of about 2,000 men there and used it mostly as a prison for Confederate captives and deserters. It wasn't until after the war that it received its most famous inmate -- Samuel Mudd, the Maryland doctor convicted of aiding Abraham Lincoln's assassin, John Wilkes Booth, by setting his broken leg. 
How many were garrisoned there?
The Union kept a garrison of about 2,000 men