Wagner grew up in the East Riverton area, and after graduating from high school in 1934, attended Duke University where he played baseball under former major leaguer Jack Coombs.
Wagner then played for three different minor league teams - in 1937 he was with the Portsmouth Cubs of the Class B Piedmont League, in 1938 the Spartanburg Spartans of the Class B South Atlantic League, and in 1939 the Newark Bears of the Class AA International League.