Born Harriet E. "Hattie" Adams in Milford, New Hampshire, she was the mixed-race daughter of Margaret Ann (or Adams) Smith, a washerwoman of Irish ancestry, and Joshua Green, an African-American "hooper of barrels."
Adams married Thomas Wilson in Milford on October 6, 1851.