Kelly was born in Jersey City, New Jersey on July 11, 1926, to an Irish father and an Italian mother.
He worked as an accountant for the firm of Peat, Marwick, and Mitchell (now part of KPMG), where one of his responsibilities was auditing Nutley Savings and Loan.