Inverse FFT - Synch the Phase
If you were attempting to modify the contents of a DFT before the inverse DFT, say for filtering, then you need to do overlap-add or overlap-save fast convolution instead of just an Inverse DFT.  You will also need to use a longer DFT and inverse DFT.  If you can determine the length of the impulse response of your filter, then the length of the new DFT/IDFT needs to be extended and zero-padded by the length of that impulse response, so that the filter convolution is equivalent to linear, not circular.

If filtering, the noise you hear is likely the result of doing a circular convolution, instead of zero-padding and overlapping.

If, instead you are synthesizing audio, look up phase vocoder methods.