Get data ready for IFFT
The first component of the FFT is the DC component. The other NFFT-1 should be symmetric.
If NFFT is even, that means that 

response(NFFT/2-k)=conj(response(NFFT/2+k))


for $k=1,\ldots,NFFT/2-1$ and response(NFFT/2) is arbitrary.
If NFFT is odd then 

response((NFFT+1)/2-k)=conj(response((NFFT+1)/2+k+1))


for $k=0,\ldots,(NFFT-1)/2-1$.

However, MATLAB allows you to ignore all that by using the 'symmetric' argument of ifft:

response_tot = [response; zeros(NFFT-1,1)]; % assuming NFFT is odd
time_sig = ifft(response_tot,'symmetric');