The Key Monument Association erected a memorial in 1898 and the remains of both Francis Scott Key and his wife, Mary Tayloe Lloyd, were placed in a crypt in the base of the monument.
In1806, Key's sister, Anne Phoebe Charlton Key, married Roger B. Taney, who would later become Chief Justice of the United States.