Walker's coaching career began in 1926 at Atlantic Christian College, today known as Barton College, in Wilson, North Carolina, where he also played professional baseball for the Wilson Bugs of the Virginia League.
After quitting Wake Forest, Walker sought a higher paying job and joined longtime friend and former assistant Herman Hickman at Yale University.