 The Privy Garden was originally created as a private royal pleasure garden and continued to serve a similar purpose during the Interregnum ( 1649 – 1660 ) , when the English Council of State put considerable effort and money into repairing and improving the garden . They appear to have reserved it exclusively for their own use , with their own individual keys for access . By Samuel Pepys ' time , after the Restoration of the monarchy , it had become " a through @-@ passage , and common . " The wall that enclosed the garden was often used by ballad @-@ sellers to display their wares to passers @-@ by , and courtiers used it to air their laundry . Pepys recorded his titillation at the sight of the underwear of Charles II 's mistress , Lady Castlemaine , hanging out to dry in the Privy Garden . A century later , James Boswell wrote in his diary that he had taken a prostitute into the garden and " indulged sensuality " , but he was shocked to find when he got home that she had stolen his handkerchief . 
