 The area was ripe for redevelopment by the landholding elite , who wished to have suitably grand townhouses to occupy while attending Parliament and the court . The vestiges of the Privy Garden became the site of a new street . In 1808 a row of houses called Whitehall Gardens was constructed on the site . Behind each house , long grassy gardens planted with rows of trees led directly down to the river , until they were cut off by the construction of the Victoria Embankment along the river bank between 1865 and 1870 . Prime Minister Benjamin Disraeli lived in one of the houses between 1873 and 1875 , while Sir Robert Peel lived in another ( and died there in 1850 after falling from his horse on Constitution Hill ) . The trees outside the front entrance to Whitehall Gardens were the last survivors of the original Privy Garden , and stood until as recently as the late 1930s . 
