This private communal garden was provided for residents of Elm Park Gardens, which were laid out in 1875 by George Godwin. The site was formerly part of the grounds of Chelsea Park and a single mulberry tree survived in the centre of the garden until 1906. The garden consists of a number of strips of planting between old and new buildings, with lawn, mature trees and paved areas.
By early C17th, grand houses, among them early properties of Henry VIII and Sir Thomas More (Beaufort House), began to be established around the old village of Chelsea. In subsequent decades more building took place both along the river as well as further inland particularly up Church Lane, now Old Church Street, which developed northwards to the Queen's Elm. This is where Queen Elizabeth I is supposed to have sheltered from the rain with Lord Burghley; the 'Queen's Tree' was recorded as early as 1586 and an arbour was built around it. In the early C16th an area between Duke's Walk and Lovers' Walk called 'Sand Hill' was part of Sir Thomas More's estate. In 1664 it was known as Lord Wharton's Park after its then owner; after Wharton sold his property in 1714 it became known as Chelsea Park. By c.1718 it had been purchased for a mulberry garden with the purpose of manufacturing raw silk but the venture failed, and Chelsea Park was gradually built over for housing.