Schwartz was born in Passaic, New Jersey to a Jewish family [REF].
On 23 December 1941 he married his wife of 69 years Mildred and together they had four children (3 sons and 1 daughter): Donald, who became an ophthalmologist, Lloyd, who worked with his father and in show business, Ross, who became an attorney, and Hope Juber, a writer/producer.