 After the destruction by fire of the Palace of Whitehall in 1698 , the surroundings of the Privy Garden changed dramatically . Most of the palace buildings had been burned down in the 1698 fire ; others were torn down as the last vestiges of the old Tudor and Stuart Palace were removed . With the demise of the Privy Gallery , the Privy Garden was extended north to include the Pebble Court behind the Banqueting House . 
