How to achieve a smoothed frequency phase response with no outliers
If your signals are not perfectly integer periodic in the FFT aperture (width), the phase measurements of those narrow band signals will toggle between FFT result bins.  This phase toggling will look like phase noise, but is actually a windowing artifact.  It is due to any window edge discontinuity from any non-exactly-integer-in-aperture periodicity in the signal (plus the fact that all the basis vectors of a DFT are perfectly circular in aperture).

To stop this phase toggling (apparent phase noise), perform an FFTShift before the FFT.  The fftshift will rotate the edge discontinuity away from sample 0, and reference the DFT phase estimate to the center of your data window, where it is likely continuous (instead of to a phase discontinuity between the last sample and first sample of the data).

Added: 

There are (at least) 2 ways to perform a time-domain fftshift.  The first is to circularly rotate the data vector 50% after windowing but before performing the FFT.  The 2nd way is to complex conjugate every other result bin (odd ones usually) after the FFT.  Due to the DFT shift property, these two methods produce identical results.  Note that these are in the opposite order from doing a frequency domain fftshift.