Kilvert was born on December 3rd 1840 at The Rectory, Hardenhuish Lane, near Chippenham, Wiltshire, to the Rev. Robert Kilvert, rector of Langley Burrell, Wiltshire, and Thermuthis, daughter of Walter Coleman and Thermuthis Ashe.
In August 1879 he married Elizabeth Ann Rowland (1846-1911), whom he had met on a visit to Paris, but he died a few days after returning from his honeymoon in Scotland from peritonitis, aged 38.