Cole met his first wife, Nadine Robinson, while they were on tour for the all-black Broadway musical Shuffle Along.
In 1956, Cole was assaulted on stage during a concert in Birmingham, Alabama, with the Ted Heath Band (while singing the song "Little Girl").