How do you relate imaginary numbers with phase shift? How to imagine this?
Paint a red dot on the blue complex spiral at t=0, where the value is 1.0.  Now roll the spiral upside-down until the red dot is at -1.0, still at t=0.  This roll would be equivalent to a phase shift of Pi.  Note that the real part no longer corresponds to a cosine wave that starts at t=0.  

But the real part of the rolled spiral does correspond to a cosine wave that starts (with value 1.0) at +Pi or -Pi, which is a different starting time, contradicting one of your assumptions above.

Thus phase is related to starting time.