Accurately measure relative intensity noise (RIN) using RF spectral analyzer?
However, dBc/Hz is the power referenced to the carrier and I'm not sure what that is in this case.


I suspect the carrier in this case is the average optical power, which they may be thinking of as a many-terahertz carrier.


  some authors present system noise floor measurements in units of dBc/Hz. Is this wrong since in this case there's no carrier?


It's not clear to me why somebody would choose those units for a noise floor. It may be wrong, but I'd want to see the context where you read it to say for sure.


  I find that the trace on the RF spectrum analyzer shows harmonics as a series of peaks. The levels between peaks is at the same level as the background level (i.e. when there is no signal input). Can we therefore infer that the RIN at these points (i.e. if we integrate from 10 Hz, say, up to the 1st harmonic) is equal to or less than the system RIN?


In the RIN measurements I've seen, there are no measurable harmonics, just a single peak related to the laser's intrinsic relaxation oscillation frequency. Are you testing with a modulation signal applied to the laser? Most RIN measurement's I've seen were done with the laser operated CW, and I'd think the results are easier to interpret for a CW optical signal.

In general spectrum analyzers have a noise floor, but I wouldn't call it "RIN", because it is not "relative intensity" --- it doesn't change in proportion to the optical power. The measurement system noise is a fixed "floor" and you can't measure power spectral density below that floor. So whenever the trace is down at the noise floor, you're not measuring anything about the device you are testing, just the capabilities of the analyzer.

General comment

The RIN measurement is fairly difficult to do. Unless the laser has very bad performance you need a very low-noise detector, very low-noise preamplifier, and a very sensitive spectrum analyzer (with a low noise floor). You will want to test the noise floor of your whole receiver system (detector, preamplifier, spectrum analyzer) before measuring your laser to be sure you know when you're measuring the laser behavior and when you're just seeing instrument noise. 

Edit

To follow up your questions in comments:

Sorry I'm not familiar with RIN measurements on pulsed lasers. But the units of dBc/Hz make a lot more sense now --- they're just talking about the fundamental of the pulse signal as the carrier.

The measurements I'm familiar with, you're most interested in the peak frequency in the RIN spectrum. I don't think you could do this with a pulsed laser because you'd have to pulse at a higher frequency than the RIN peak, which would also be beyond the modulation capabilities of the laser. But maybe there are tricks I'm not aware of.

I will suggest that for a pulsed RIN measurement, you don't need the bias tee, though you might want a blocking capacitor for the sake of your SA input. The peak of the fundamental of the pulse signal gives you the laser signal power that you'd be measuring the noise relative to.


  is it fair to say then that the laser has equal or better noise performance?


I'd say it this way: if the laser noise is too small to measure on your detector/SA system, then the measurement system is not adequate to measure the noise of that laser.


  how would you recommend characterising the system noise floor?


Typically, you turn on the photodetector and pre-amp, but don't apply any laser signal. Then take a sweep on the spectrum analyzer, using the exact settings you'll use for your measurement. This gives the combined floor for the detector plus the SA.

You should be able to display this for comparison to your laser RIN measurements by just using the save-trace features of the SA, without any need for calculations.