The earliest known use of the name "Summer Hill" was in 1876, for a land subdivision adjacent to the present-day St Andrew's Anglican Church.
Summer Hill's largest mansion, Carleton (now the Grosvenor Hospital's main building), was built in the early 1880s on Liverpool Road for Charles Carleton Skarrat.