 He married firstly Mary Ashton, daughter of Samuel Ashton at the Unitarian Chapel of Gee Cross on 5 February 1846.She died in 1885 and Potter married secondly Helena Hicks, daughter of John Hicks at St Paul's Church, Lambeth, Surrey on 10 March 1887.Potter had four sons and a daughter by his first wife.