 The garden remained in Crown ownership but it became neglected and filthy with the departure of the monarchy from Whitehall . In 1733 the Duke of Richmond and other residents of the surviving properties adjoining the garden petitioned that they be allowed to lease the " void ground " of the garden . The Crown agreed , and in 1734 most of the garden was leased to the Dukes of Richmond and Montagu , the Earl of Loudon and Sir Conyers D 'Arcy . 
