 As a child, Williams' heroes were Pepper Martin of the St. Louis Cardinals and Bill Terry of the New York Giants.Williams graduated from Herbert Hoover High School in San Diego, where he played baseball as a pitcher and was the star of the team.During this time, he also played American Legion Baseball, later being named the 1960 American Legion Baseball Graduate of the Year.