Kilmer was born 15 December 1851 to Charles Kilmer and Mary Anne (née Langdon) in Chaplinville, Connecticut [REF].
Kilmer married Annis Eliza 'Annie' Kilburn [REF] on 25 December 1874 at Sunbury, Pennsylvania, with whom he had four children, namely Anda Frederick (b.1873 d.1899), Ellen Annie (b.1875 d.1876), Charles Willoughby (b.1880 d.1880) and writer and poet Alfred Joyce (b.1886 d.1918) [REF].