Why we add imaginary part in inverse fourier transform since the time domain signal has only real value?
then the imaginary part must go to zero

No. That's a fallacy. There's really not much to explain here other than this intuition is wrong.
The imaginary part of a Fourier transform is only zero if the transformed signal has conjugate symmetry (which is the same as symmetry for real signals) around t=0.
Think about it. You decomposed your $e^{-ix}=\cos x + i\sin x$, and then used an integral to project your signal onto these two things. (since the integral of a product between a signal and another (conjugated) signal is the inner product of the signal vector space, and hence the projection, if you're after the mathematical formalism.)
The part of your signal that is symmetrical to t=0 projects onto the cosine, and the part that is antisymmetrical to t=0 to the sine. The sine has an imaginary unit in front.
This is really among the first things that are typically illustrated in textbooks when they introduce the Fourier transform, so this might be a great point to go back to that!