Using FFT for spectral analysis: How to convert from power to dBm?
10 Log10 of the magnitude squared yields dB, but not dBm.  dB is a relative measure.  Since dB are logarithmic, you can add some constant to a pile of dB numbers and the pile still has the same meaning because separate dB numbers are only relative to each other.

However, dBm defines 0 as representing 1 mW.  Now there is a absolute reference, so individual dBm numbers are no longer just ratios but actually represent fixed power levels.  For example, 42 dBm = 15.8 W.

The problem you have is that you don't apparently know what the input values to your FFT represent.  Is it a voltage signal, a current signal, something else?  Power is proportional to both the voltage squared and the current squared, but you can't know absolute power without the proportionality constant.  In the case of voltage squared, the proportionality constant is 1/resistance.  In the case of current squared, it is resistance.

So the real problem is that you don't know what is coming out of your FFT because you don't know what you stuck in.  This is often a problem when people get sloppy with units, or ignore them altogether as in this case.

Let's say for sake of example that what you put into the FFT was a EMF signal in units of volts.  That means the magnitude squared you are referring to is in units of volts squared.  That still doesn't tell you power though.  If you knew, for example, that this voltage was driving a load of 600 &Omega;, then you can compute power.  A FFT output value of 1 V&sup2; would then imply a power of 1.67 mW, which means 2.22 dBm.  In this example, you could multiply the V&sup2; values by 1.67 first and then take 10 Log10 to get dBm, or equivalently take 10 Log10 of the V&sup2; values then add 2.22 to get dBm.  Of course the actual factor you use to adjust 10 Log10(V&sup2;) to dBm depends on the resistance the original voltage is driving, which you haven't told us.

Pay attention to units.