Alfred Milnes was born in Bradford, England, in 1844. He and his parents and several siblings emigrated to the United States, landing at New Orleans in 1854. After a few years of wandering, during which time his mother and two of his little brothers died, Milnes settled in Coldwater, Mich.
Milnes attended public school, served in a Michigan infantry unit throughout the Civil War, and went into business in Coldwater, becoming a banker, postmaster and dealer in real estate.
He also was active in Republican politics, later claiming to have cast his first vote for Abraham Lincoln in 1864 while on a Civil War battlefield (he would have had to lie about his age by a few months in order to have voted that year). He served his county as a delegate to numerous state Republican conventions.