How to do smoothing without affecting phase
Any smoothing in the frequency domain is similar to convolution filtering.  And convolution filtering (with anything other an impulse or zero) will spread the width of a signal.

If you want a real result, you may need to check to make sure the input to your IFFT is conjugate symmetric.

If you don't want any phase shift, you could try copying the phase from before your filter into the result after your filtering and before the IFFT.