How isotopes atomic mass was defined?
Historically, the relative masses of the isotopes were measured relative to O=16.  

There is a good historical explanation in A New Mass-Spectrograph and the Whole Number Rule (1927). 

At this time (1927), though it was understood that many elements had more than one isotope, only one isotope of oxygen was known.