The house then remained empty until 1876 when it was bought by Sir Mountstuart Elphinstone Grant Duff (1829-1906), a Scottish MP, junior minister in Gladstone's first government, and from 1881 to 1887 Governor of Madras.
York House was acquired by Twickenham Urban District Council and after major alterations became the council's offices.