Converting raw I/Q to dB
Basics

The amplitude of an IQ signal is just the vector magnitude, $\sqrt{I^2 + Q^2}$.

The power of an IQ signal is the squared magnitude, $I^2 + Q^2$.

When you see a logarithmic (dB) meter, it is usually measuring the log of the power, i.e. $10 \log_{10}(I^2 + Q^2)$. (This can also be calculated as $20 \log_{10}$ of the amplitude, but unless you already have the amplitude that wastes a square root operation.)

Units

Remember, dB is a relative figure. If you just take $10 \log_{10}(I^2 + Q^2)$, then 0 dB corresponds to an amplitude of exactly 1. If your hardware driver takes the usual floating-point convention of the absolute extreme sample values being from &minus;1 to +1, then you can say that your dB power values are dBFS — decibels relative to full scale. Any signals stronger than that level will be clipped, distorting the signal.

dBm, decibels relative to one milliwatt of power, just uses a different reference level. You can convert to dBm just by adding or subtracting the proper calibration value from the dBFS value — but you need to know that calibration for your hardware at the frequency of interest, such as by measuring it (using a signal source of known power output); it is impossible to perform that calibration purely in software since the digital samples are just numbers with no inherent units.

(A mistake I've seen is to refer to the sample values, or parameters which scale according to them such as an amplitude threshold, as being in “volts”; this is complete nonsense unless your ADC (and other hardware) is actually calibrated to one volt. This is unreasonably large for a radio receiver.)

Practical application

If you are just looking to ignore signals that aren't sufficiently strong (this is known as carrier squelch or power squelch), it doesn't matter what units you use, or even if they're logarithmic or linear, because you're just doing a greater-than comparison. The only critical component is that you start with $I^2 + Q^2$ (as opposed to, say, $I + Q$, which would be just plain wrong).

Note on bandwidth you probably don't need to read

It may also be relevant to note that if you filter a signal, you are by definition removing some of the signal power, so the measurement will be smaller.

In particular, a FFT (such as is the primary visual display in tools like SDR#) can loosely be thought of as a large collection of extremely sharp filters; each output “bin” collects some fraction of the input power. Accordingly, the power in each bin depends on the width of the bin. If you divide by the width of the bin in hertz (that value being $\text{sample rate}/\text{FFT length}$) before taking the logarithm, then instead of dB power, you measure dB power spectral density, which has the advantage of being independent of the FFT bin width if the features you care about are wider than one bin (e.g. a wideband modulated signal); if they are narrower (e.g. pure tones) then the power value is more useful.