Why do I need inverse FFT? I have magnetometer time-series measurements in mV. I am trying to understand what is theory behind software that the company has provided.

Let $x$ be your time series vector, which is an array of double, and $W$ your
window length. $x$ is sliced into $L$ windows, each window is detrended and
(if not rectangular window function is chosen) a Hanning window is
applied. Then the FFTW is used to obtain an array of complex numbers $X$.
From $W$ real input values we get $P = W/2$ complex numbers in $X$. $X$ is calibrated with

$$
    X'= \frac{X}{\sqrt{f_s*P}}
$$
Transfer function is given by
$$
   H(\omega) =0.8V/nT*F_m(\omega)*F_t(\omega)*V
$$
where
$$
F_m(\omega)=\frac{j\omega}{j\omega+2\pi 4}\\
F_t(\omega)=\frac{2\pi 1591}{2 \pi 1591 + j\omega}
$$

If a transfer function $H$ is given we obtain

$$
    X'' = X'/H
$$

Finally if you use a windowing function the result has to be multiplied with the inverse of the integral of that window.

My question is why do I need the inverse of the integral of that window?

Are my units going to be $nT$?

I am giving part of the calibration file

 FREQUENCY    MAGNITUDE      PHASE       
    Hz                     deg          
Chopper Off                             
+4.0000E-01  +1.0714E-02  +1.4808E+02   
+5.5580E-01  +1.3325E-02  +1.3911E+02    
+7.7230E-01  +1.5775E-02  +1.2956E+02    
+1.0731E+00  +1.7732E-02  +1.2025E+02    
+1.4910E+00  +1.9094E-02  +1.1174E+02    
+2.0718E+00  +1.9931E-02  +1.0442E+02    
+2.8787E+00  +2.0392E-02  +9.8155E+01    
+4.0000E+00  +2.0603E-02  +9.2520E+01    
+5.5580E+00  +2.0611E-02  +8.7089E+01    
+7.7230E+00  +2.0408E-02  +8.1384E+01    
+1.0731E+01  +1.9949E-02  +7.4971E+01    
+1.4910E+01  +1.9006E-02  +6.7315E+01    
+2.0718E+01  +1.7761E-02  +5.8714E+01    
+2.8788E+01  +1.5629E-02  +4.9042E+01    
+4.0000E+01  +1.3116E-02  +3.9247E+01    
+5.5580E+01  +1.0350E-02  +3.0040E+01    
+7.7230E+01  +8.0188E-03  +2.2525E+01    
+1.0731E+02  +5.9868E-03  +1.6408E+01


It seems that people do not understand my questions.Manuel can dowloaded from here

https://geo-metronix.de/ocl/public.php?service=files&amp;t=9b54b82730e4b8b869e083574923d99a


You can go to tsmp and download tsmp.pdf.Calibration is on page 40.
I'm still not sure that I understand what your question is, but this specific one is easy;


  My question is why do I need the inverse of the integral of that window?


The issue is that you'll apply a window to the data $x$:
$$
x_2(t) = x(t) \cdot w(t)
$$
that means that the sum of $x_2$ is going to be different from the sum of the (rectangular windowed) $x$.  All the inverse of the integral is doing is rescaling things to attempt to make the sums equal.

As far as your other question goes:


  Are my units going to be nT?


That's all down to calibration. You have millvolts.  Your expression for $H(\omega)$ has a calibration factor $0.8V/nT$. To me, that means that 0.8V (or 800 mV) is 1 nT.

Tesla are one kilogram per second squared per ampere in base SI units.  Usually, with any measurements calibration is what equates the voltage (or current, usually) measured to the units of you're actually interested in.  

All $H(\omega)$ seems to be doing is applying a bandpass filter to your data.