 From the start of the 18th century onwards , the garden went through major changes as it fell into neglect . It was painted in 1747 by Canaletto during a period of transition , as Westminster was being transformed by the construction of new buildings and roads . By the start of the 19th century it had been redeveloped as the site for a row of townhouses , some of which were occupied by prime ministers seeking homes near the government buildings nearby . The last remnants of the Privy Garden were destroyed in 1938 during the construction of government offices which now house the Ministry of Defence 's headquarters building . 
