Charles Howard, 11th Duke of Norfolk, was known for his restoration work and improvements to the castle beginning in 1787 and continuing for a number of years, as he desired to live there and entertain his visitors there.
The Folly that still stands on the hill above Swanbourne Lake was commissioned by and built for the Duke by Francis Hiorne to prove his abilities as an architect and builder.