NIST randomness test p values I have a binary sequence of length $2^{20}$. I am using the NIST statistical test, assess.


What should be taken for ./assess and  How many bitstreams? in this test?
In the final Analysis report, what p-values should be there to pass this test?
In general, if you draw a sample at random from a distribution $D$, and you apply a statistical test for the null hypothesis $D$ yielding a $p$-value, and you print the $p$-value, rounded to, say, a multiple of 0.05, it is as if you rolled a fair d20 to pick among the possible results.  (For finer granularity, roll a die with more sides.)  That's all that a $p$-value is.

One might hope that the test was designed so that if you sampled from a different distribution $D'$ instead of $D$, an alternative hypothesis, it is as if you rolled a d20 with a weight on one of the faces so that it comes up &lt;0.05 most of the time.  Then if you raise an alarm whenever $p &lt; 0.05$, there's a 5% false alarm rate in your test of the null hypothesis, and a high true alarm rate when there's a problem and you're actually sampling from $D'$.

In the statistics literature, the false alarm rate for a test like this is confusingly called the ‘significance level’; the true alarm rate is called the ‘statistical power’.  The false alarm rate depends only on the null hypothesis $D$, not on $D'$, while the statistical power depends only on the alternative hypothesis $D'$.  A poorly designed test might behave identically on $D$ and $D'$, in which case an alarm means nothing at all about $D'$ even if your ‘significance level’ is 0.05.  (An exceptionally poorly designed test might even have a lower alarm rate under $D'$.)

Typical ‘randomness tests’ like dieharder and the NIST SP 800-22 suite* are collections of statistical tests for a null hypothesis of the uniform random distribution $D$ on bit strings.  They have high statistical power to detect very simple-minded alternative hypotheses like strings of IID bits with 1/4 chance of 0 and 3/4 chance of 1.  They do not have high statistical power to detect alternative hypotheses like strings of $\operatorname{AES256}_k(0) \mathbin\| \operatorname{AES256}_k(1) \mathbin\| \cdots$ for uniform random key $k$.

What really matters in cryptography is how you got your sample.  There are two options:


You got your sample by measuring a physical process like counting ionizing events in a Geiger–Müller tube.  In this case, you should have a specific probabilistic physical model in mind with fancy words like Poisson process: to measure the cryptographic value of this process you need to study the physics and engineering of the process to determine a lower bound on its min-entropy by finding the best way to reliably predict what the output is with the help of teams of trained physicists and engineers.

The NIST tests cannot help you here: they don't know anything about Geiger–Müller tubes.
You got your sample by computing a difficult-to-invert function on a small random input.  In this case, the difficult-to-invert function, like AES-256, should be well-studied by teams of trained cryptographers around the world publishing papers with fancy words like differential cryptanalysis to get confidence that it really is difficult to invert.

The NIST tests cannot help you here: they don't know anything about cryptanalysis techniques.


So what can the NIST tests do?  Not much, really.  They might give you hints about particular nonuniformities in the distribution that lead you to ideas about (1) or (2), that's all.

But, if you want cookie-cutter answers to questions…


  
  What should be taken for ./assess and How many bitstreams? in this test?
  


Pass the number of bits in your stream to ./assess.  Specify 1 bit stream.


  
  In the final Analysis report, what p-values should be there to pass this test?
  


Always use p &lt; 0.05, unless it's a two-tailed test in which case use $p &lt; 0.025$ or $p &gt; 0.975$.  Beware of green jelly beans.



* Other tools like ent and the NIST SP 800-90B suite fit parameters to model families and compute or estimate the entropy of the resulting models.  Many of the concepts are the same, and likewise these tools are too dumb to know anything about your physical process in (1) or your difficult-to-invert function in (2).