Inverse fourier transform of Hermitian function, getting an imaginary part in the cartoon below 


It shows that if we take the inverse Fourier transform of a Hermitian function, real part even and imaginary part is odd we should get a purely real function in the time domain.

I tried to replicate this by taking a frequency response I have, zero padding it making it even for the real and odd in the imaginary. Shown below

Although my ifft in Matlab I still have a significant imaginary component, the real is shown left and imaginary on the right.

I also wrote my own script for DFT/IDFT and tested it on a sine function to make sure I get the same as Matlab. Maybe I have missed something?
A (near) sinusoid in one domain (zoom in off center, and you might see it) can indicate a shift away from symmetry in the other domain.  So your initial imaginary signal may be 1 or 1/2 samples off from exact circular symmetry around element 0 (maybe around element 1 in MatLab or Fortran weird array indexing)