Why remove energy at Nyquist before ifft?
you don't have to set $X\left(\frac{N}{2} \right)=0$ if you don't want to.  it will correspond to this component:

$$ X(k)\frac{1}{N}e^{j 2 \pi \frac{nk}{N}}\bigg|_{k=\frac{N}{2}} = X\left(\frac{N}{2} \right)\frac{1}{N}(-1)^n $$

but when you sample some $x[n]$, FFT it and find that $X\left(\frac{N}{2} \right) \ne 0$, you do not know the phase of that Nyquist component before it was sampled.  the Nyquist component actually aliases with itself in such a way that the quadrature part of the negative frequency cancels the quadrature part of the positive Nyquist frequency component.  such would look like:

$$ = \frac{X\left(\frac{N}{2} \right)}{N}(-1)^n = \frac{X\left(\frac{N}{2} \right)}{N}\left( \cos(\pi n) + A \sin(\pi n) \right) $$

$A$ could be anything and you wouldn't know it because $\sin(\pi n)$ is always zero for any integer $n$.

so if you're doing audio, i would likely set both the Nyquist and the DC components to zero and also insure the complex-conjugate symmetry with the other bins before iFFT.  just to make sure that, within rounding error, the imaginary part of your result is zero (the result is purely real) and that you have an unambiguous magnitude and phase for every non-zero component.