Hazen was born in Rockville Centre, New York, on November 1, 1948. His parents were Peggy Hazen (née Dorothy Ellen Chapin; 1918–2002) and Dan Hazen (né Daniel Francis Hazen, Jr.; 1918–2016). He spent his early childhood in Cleveland, near a fossil quarry where he collected his first trilobite at the age of about 9.
The Hazen family moved to New Jersey, where Roberts eight-grade teacher, Bill Welsh, observed Roberts interest in his collection of minerals. Hazen later recalled "He gave me a starter collection of 100 specimens, mineral field guides, and mimeographed directions to Paterson and Franklin, New Jersey." Hazen also had an early interest in music, starting with the piano at age 5, the violin at 6 and the trumpet at age 9.