RF Harmonics and Spectrum Analysers
A spectrum analyzer will automatically add the attenuator setting to the measured absolute power.

The units you need to know are


dB -- power relative to some reference value
dBm -- power relative to 1mW
dBc -- power relative to carrier power


The problem with using the RTL-SDR is that these devices are not designed for absolute power measurements, but rather for extracting digital data. As such, they do not have any compensation for their own frequency response, which is overlaid with the measurement data you have.

The main issue, however is that you are measuring in dBc, by normalizing your carrier power to 0dB and then looking at the harmonics. dBc give you a good estimate of the quality of your amplifier, but for regulatory purposes, only absolute power in dBm is relevant, which requires you to use a device that has been calibrated.

The attenuation setting on the spectrum analyzer needs to be chosen so the entire RF range is overload free, as overloads in the RF frontend would produce harmonics in the first mixer stage, before the selection filter.

Some analyzers include a high pass filter option, where higher harmonics are measured by replacing the attenuator with a high pass filter, allowing for better resolution while still remaining overload free; also, analyzers often provide a dedicated harmonics measurement that automatically searches the base frequency, adjusts the attenuator and reports back absolute and relative values with error margins.