is root mean square possible to calculate similarity of FFT result?
The Discrete Fourier Transform is a unitary transform (maybe scaled).  As such, it preserves the Frobenius norm of the signal.  That is essentially the same as the Root Mean Square calculation.

In other words:

 rms( x - y ) == C * norm( fft(x) - fft(y) )


for an appropriate constant C depending on your fft implementation.

In Matlab or octave, C=1/nfft:

 n=8192;
 x=randn(n,2)*[1;j];
 y=randn(n,2)*[1;j];
 sqrt( mean( abs(x-y).^2 ) ) - norm(fft(x) - fft(y)) / n

 ans = -5.3291e-15