Robinson then took an offer from his old friend and pastor Rev. Karl Downs to be the athletic director at Sam Huston College in Austin . The school was part of the Southwestern Athletic Conference at the time . Robinson then accepted an offer from his old friend and pastor Rev. Karl Downs to be the athletic director at Sam Huston College in Austin , then of the Southwestern Athletic Conference .