What is the difference between complex FFT of baseband fft(I+jY) vs. FFT of absolute fft(abs|I+jY|)?
Well, taking the absolute value is a nonlinear operation you do on your signal.
That means it introduces frequencies into your signal (harmonics!) that weren't there in the original, and you have no way of figuring out whether the things you observe are from your original signal or from your nonlinear operation.
So, you basically make your whole thing useless for spectral estimation. Don't do that!
It's easy to imagine how that goes wrong. Take the simplest to analyze signal we can think of: a simple real-valued cosine $\cos(2\pi t)$. We know the spectrum of that – a line spectrum, with components at $\pm 1$, each having power $\frac 12$.
If you take the absolute of that, that's (just for this specific signal!) equivalent with multiplying with a $\pm 1$ square wave that is scaled and shifted exactly so that it becomes negative whenever the cosine is negative. That way, the product is always the absolute value.
Multiplication of two signals means in spectrum a convolution of the spectra. The cosine spectrum we know, $\frac12(\delta(f-1)+\delta(f+1))$, and the spectrum of a square wave is an infinite sum of such deltas, at every odd multiple of the fundamental frequency $f$.
So, your &quot;innocent&quot; absolute just converted your pure and easy cosine to something with infinitely many spectral components. No good.