Martin married his first wife, Muriel Weiss, in 1941.
In 1967, Martin married Olavee Lucile Parsons (a successful model and documentary director) and adopted her two children, Rebecca (Martin) Schacht and George Martin.