 The earliest surviving depiction of the garden , from 1670 , shows a very different layout . By this time the garden had been redesigned in a grid pattern with sixteen squares of grass separated by paths . The relentless expansion of the Palace of Whitehall , which was by now a sprawling jumble of structures , had hemmed in the Privy Garden behind walls and buildings on all sides . A high wall to the west separated it from The Street , the main thoroughfare at the south end of Whitehall that bisected the palace in a north @-@ south direction . To the north , a range of buildings occupied by high @-@ ranking courtiers separated it from the Pebble Court that lay behind the Banqueting House , while to the east the Stone Gallery and state apartments , used by the king 's closest courtiers , blocked it off from the River Thames . The royal apartments were off the Stone Gallery and had a view of the Privy Garden , with a screen in place to prevent passers @-@ by from seeing the naked king in his 7 by 7 feet ( 2 @.@ 1 by 2 @.@ 1 m ) bathtub . A row of trees on the south side screened it from the Bowling Green , which had been an orchard in Henry VIII 's time but was converted for leisure use after the Restoration . A terrace also separated the Privy Garden from the Bowling Green , but this was removed in 1673 – 4 and part of the Bowling Green was added to the garden . 
