Kelly was born in 1926 in Montrose, Kansas to attorney Herman Albertus (H.A.) "Bert" Kelly, a member of the Dudley-Winthrop Family of Massachusetts fame, and Theodora Viola Pacaková a Czech immigrant from the Austro-Hungarian Empire.
He attended elementary and high school in Montrose.