IFFT Synthesis from an empty spectrum
I believe that if you will keep the phases and not convert it into magnitude (power spectrum) the IFFT will work. When you use the center frequency, the phase is 0, in other words, the real part is also the magnitude. As you move from the center frequency the error in magnitude growth as the phase growth. As result your error in recovery by means of IFFT growth.