Holt House is intimately linked to the early history of both Washington and the nation through its association with such prominent figures as Thomas Johnson (first governor of Maryland), Thomas Jefferson, Benjamin Stoddert (first Secretary of the Navy), Benjamin Mackall (Georgetown merchant), John Adams, John Quincy Adams, and Andrew Jackson.
The land on which Holt House sits was originally part of a tract of land acquired from the Beall family in 1793 by Benjamin Stoddert (builder of Halcyon House in Georgetown).