Why do I need inverse FFT?
I'm still not sure that I understand what your question is, but this specific one is easy;


  My question is why do I need the inverse of the integral of that window?


The issue is that you'll apply a window to the data $x$:
$$
x_2(t) = x(t) \cdot w(t)
$$
that means that the sum of $x_2$ is going to be different from the sum of the (rectangular windowed) $x$.  All the inverse of the integral is doing is rescaling things to attempt to make the sums equal.

As far as your other question goes:


  Are my units going to be nT?


That's all down to calibration. You have millvolts.  Your expression for $H(\omega)$ has a calibration factor $0.8V/nT$. To me, that means that 0.8V (or 800 mV) is 1 nT.

Tesla are one kilogram per second squared per ampere in base SI units.  Usually, with any measurements calibration is what equates the voltage (or current, usually) measured to the units of you're actually interested in.  

All $H(\omega)$ seems to be doing is applying a bandpass filter to your data.