 When Canaletto painted a view of the garden looking north from the Duke of Richmond 's dining room in Richmond House in 1747 , it was a last view of a prospect that was soon to disappear with the demolition of the old palace 's Holbein Gate adjoining the garden . Parliament Street was driven through the western side of the garden in 1750 to connect Whitehall to the Palace of Westminster . At the start of the 19th century the garden 's " decayed wall , long fringed by pamphlets , ballads , and ragged advertisements " was removed and replaced by an iron railing and newly planted trees . Its north side remained " most confused and unpleasant " , terminating in a maze of " fifty narrow passages , formed by sheds , blank walls , the residences of the nobility , and the workshops of the tradesmen . " 
