What is the meaning of a Nyquist diagram in a vibration signal
The "Nyquist plot" you have is really just the Fourier transform of the vibration data.  That the plot is "circular" doesn't mean much: it just means that the phase of the signal is continually increasing and that the magnitude is (roughly) constant.

I wouldn't call such a plot a Nyquist plot.  Nyquist plots are plots of the transfer function of a system. You may be able to use your measured signal and the reference to attempt to generate a Nyquist plot, but it's probably not going to be just a matter of multiplying / dividing Fourier transforms of the signals.

Generally, plotting the Fourier transform of a signal on the complex plane doesn't show much useful information.  More useful is to plot the magnitude against frequency, as this shows the dominant frequencies in the signal.

The reason the phase plots are "chaotic" (sic) is that the signal is noisy.  Taking the argument of a noisy complex signal will nearly always generate discontinuous phase plots.

I believe an Argand diagram is the same as your "Nyquist plot".