How to measure quantization noise?
First of all, quantization noise does not exist.

When you quantize a signal in amplitude you commit a quantization error because your ADC has a finite number of levels. If you are quantizing a voltage signal then we can call the amplitude of a level of your ADC \$\Delta v\$. E.g., if your ADC has an input range from \$-1\$ V to \$+1\$ V and is a 10 bit converter, then the total number of levels is \$2^{10}=1024\$ and \$\Delta v = \frac{+1\ \mathrm{V} - (-1\ \mathrm{V})}{1024}=19.53\ \mathrm{mV}\$.

If your current sample is at \$0.6\ \mathrm{V}\$ and you have a perfect ADC, its output would be code 819, which corresponds to an input voltage of \$599.61\ \mathrm{mV}\$. You are committing an absolute error of \$390\ \mu\mathrm{V}\$, and this is quantization error.

Since engineers are good at working with AWGN, i.e. Additive White Gaussian Noise, they try to model this error, which is a stochastic process, as if it was AWGN.

If your input signal respects some hypothesis then this model is sort of valid and the power of this quantization noise can be calculated as:
$$
P_n = \frac{\Delta v^2}{12}\ [\mathrm{V^2}]
$$
This is a fundamental formula when speaking of quantization noise, try to remember it.

In your case \$\Delta v = 305\ \mu\mathrm{V}\$, thus \$P_n=7.76\ \mathrm{nV^2}\$. If you take the square root you get \$\sqrt{P_n}=88.1\ \mu\mathrm{V}\$, so far so good.

What you are trying to do is to feed a certain signal to your ADC, compute the ADC output, calculate the quantization error and finally calculate the rms value of the error, to see if it approaches the ideal minimum value of 
\$\frac{\Delta v^2}{12}\$.

Now we have to differentiate: it is clear that your system is "amplitude quantized", but is it also time quantized? This is very important because the formula to calculate the rms value of a sequence is different from the formula that works for a continuous function.

Anyway, if you feed your ADC with a continuous time signal \$x(t)\$ your output will either be \$y(t)\$ or \$y_n\$, depending if your system also samples or not.

The error can be calculated as the difference between the input and the output:
$$
\epsilon(t) = x(t) - y(t)\\
\epsilon_n = x(nT) - y_n
$$
where T is the sampling period, if present.

Finally, for a continuous function:
$$
\epsilon_{\mathrm{rms}} = \sqrt {{1 \over {T_2-T_1}} {\int_{T_1}^{T_2} {[\epsilon(t)]}^2\, dt}}
$$
where \$T_1\$ and \$T_2\$ are the starting and ending of the interval in which you want to calculate the rms.

For a sequence:
$$
\epsilon_{\mathrm{rms}} =
\sqrt{ \frac{1}{N} \left( \sum_i \epsilon_i^2 \right) }
$$
where N is the total number of samples and the sum is intended to be over all the samples.

Please note that in a real system you do not get only quantization noise. There are many other sources of noise that will be present in your ADC readings, ranging from electrical noise to nonlinearities of the ADC. Quantization noise is something specifically meant to characterize only one aspect of ADCs and does not take in account other noise sources in the ADC, nor the rest of the conversion chain.

The \$P_n\$ calculated early on is just a "you can't do better than this" value, so expect to get greater numbers from a real system. Due to the fact that the hypothesis for the formula to be valid are rarely completely respected, even an ideal system will have a greater noise power for many "nice" input signals.