 The Privy Garden originated in the 16th century as part of the estate of York Place ; Cardinal Wolsey 's London residence . The estate already had a privy , or private , garden that was located behind what is now the Banqueting House . An orchard , which was part of the estate , adjoined it to the south . When Henry VIII seized York Place , he bought more land to the south of the orchard to expand the estate . The old privy garden was cobbled over and later became known as the Pebble Court , while the orchard was converted into a new and much larger Privy Garden , known at first as the " great garden " . At the time , Westminster was not heavily built up as it is now , and York Place – later renamed Whitehall Palace – lay within a suburban area dominated by parks and gardens . St. James 's Park , across the other side of Whitehall , was a royal hunting ground . 
