The name derives from an old connection to the Grey family, descendants of the Norman knight Anchetil de Greye.
James Paul bought the house in 1688 and it passed via his son William's daughter's dowry to Sir William Stapleton, 4th Baronet in 1724.