Schwarz is the great-grandson of Frederick August Otto Schwarz, the founder of the Fifth Avenue toy store, F.A.O.
He graduated from Harvard University in 1957 and received a law degree from Harvard Law School in 1960.