Flom was born in Baltimore, Maryland on December 21, 1923, the son of Itzak (Isadore) Flom, a labor organizer in the Manhattan garment district, and the former Fannie Hirsch.
After graduating from Townsend Harris High School, he worked as an office boy in a law firm during the day, while attending City College of New York on a pre-law major at night.