He was the eldest child of John Cain Andrews (born in Norwalk, Ohio) and Sarah Wright (born in Barlestone, England).
The Andrews family moved to Evansville, Wisconsin for the purpose of Byron and his four siblings to attend the Evansville Seminary of Wisconsin that was established by the local Methodist church to educate students for entrance into college.