What assumptions should be used to invert spectrum into time domain data?
If you know anything at all about the time-domain signal, for example that it's piecewise linear, or should never be negative, or should look like a series of spikes with exponentially decaying tails, you could define a cost function that measures how well a proposed signal meets those criterea.  How to do that is hard to describe in a generic way.  Then feed that cost function into your favorite multi-variable optimizer.  The phases of the Fourier amplitudes are the variables to be optimized.   Try a variety of initial values, beware local minima, and all the other usual cautions for optimization apply.