How to evaluate chi squared result? I've been recently working on a (supposedly) TRNG. I'm still at the beginning of the project, so it is certainly not cryptographically secure yet, for now I'm just playing around.  In fact, I've already discussed the project and the chi square results here in question: Random number generator issue.

At this early stage I'm using ent to see if there is any gross issue, later I'll move to a more accurate test suite. 

One of the ent measured parameters draw my attention: the chi-squared. After doing some research I've understood that this value basically says how much, in a set of values, their frequencies match with the theoretical expected frequency. 
$$\chi^2=\sum_{i=1}^n\frac{(o_i-e_i)^2}{e_i}$$
where $o_i$ is the actual frequency and $e_i$ is the theoretical frequency.

And that's fine, but how do I understand which would be my optimal value?  For my particular case (values ranging from 0 to 255, uniform distribution) I'm using 254.3 as a reference "optimal value" from ent's manual. (I could be wrong on this, ent manual explicitly states only that for true randomness i should expect my percentage value to be between 10 and 90%, which is centered to 50%, which gives a value of 254.3)

Let's say everything is perfect, I'd have $o_i$ which is equal to  $e_i$, so the difference would be 0 for every $i$, they should add up to 0 and my optimal chi-squared would be 0.

But according to ent my optimal chi-squared is the above mentioned 254.3.

How I should calculate my optimal chi-squared for a different distribution (e.g. the normal distribution) or for a different range of values (e.g. 0 to 65535) ?
Testing the frequency of bytes produced by an ideal (uniform) random generator per question's method with $1200$ samples or more gives a $\chi^2$ that varies typically by few dozens around the average, $255$. It should be below $254.334$ for 50% of experiments, and the most likely values those closest to $253$.



For a good uniform random sequence of bytes, and "enough" samples, the distribution of $\chi^2$ computed as in the question has a well-studied distribution: the $\chi^2$ distribution with $2^8-1=255$ degrees of freedom, with this Probability Density Function:



The mean is $255$ (that's what you obtain by averaging the $\chi^2$ of many experiments). The median is $\approx254.334$ (the $\chi^2$ is less than this for 50% of experiments). The mode is $253$ (that's the most likely value, maximizing the PDF).

The standard deviation is $\sigma=\sqrt{510}\approx22.58$. Because the PDF is close to a normal, the 68–95–99.7 rule applies: that gives the probability (in percent) to be one, two and three $\sigma$ from the mean. Again that's with "enough" samples. A rule of thumb tells that about $5$ samples per expected value (thus like $1200$ samples) starts to be enough when reasoning at up to $2\sigma$. More sample usually help (I wish I knew how to more rigorously choose the minimum number of samples).

The Cumulative Density Function $\displaystyle\operatorname{CDF}(x)=\int_0^x\operatorname{PDF}(u)\ du$ is:



That tells the expected proportion of experiments (for a uniform random sequence) where the computed $\chi^2$ is less than a certain value.

We have a 0.05% chance that the $\chi^2$ is less than $187.171$, and 0.05% chance that it is above $335.917$. Thus in only about one experiment out of 1000 will the $\chi^2$ will be out of that range, for a truly uniform random sequence, and "enough" samples.

In particular, a generator that consistently gives low $\chi^2$ is not random!



When we generalize to different tests with a random variable taking one out of $n\ge2$ different outcomes with expected frequencies $e_i$ and $\displaystyle1=\sum e_i$, and the number of samples is at least about $5/\min(e_i)$ (enough for a minimum of $5$ expected samples for each value), then the distribution of $\chi^2$ is the $\chi^2$ distribution with $k=n-1$ degree(s) of liberty.

The shape of the Probability Density Function vary with $k$.



The mean is $k$, the median is about $k(1-2/9k)^3$, the mode is $\max(0, k-2)$.The variance is $\sigma=\sqrt{2k}$, but for low $k$ computations or tables (rather than estimations based on $\sigma$ and a normal approximation) are necessary.

The shape of the Cumulative Density Function vary with $k$: