Does "Common Mode Noise" only affect single-ended signals?
If my understanding is correct; does that mean if one talks about
  "common mode noise", he is talking about a single-ended system?


A differential receiver has to "manage" the common mode noise and although a perfect receiver will certainly achieve this, the real world receivers don't. For instance (just an example), an op-amp may be specified as having 100 dB common mode rejection but the fine detail in the data sheet tells you that this might only be at 100 Hz and at (say) 10 kHz, this has degraded to 60 dB and maybe 40 dB at 100 kHz etc..

So no, common mode noise is a big issue for both types of system.