How to get a non-equally spaced FFT back into the time domain
The short answer is that you can solve a least squares problem with the input signal as the decision variables.
About Nonequispaced FFT
There is a related question in stack overflow.
If you are not using python, you can take a look into the nfft github.
Here will find some information. Section 4.6 covers the inversion of such transforms.