Verifying randomness and uniqueness of TRNG output We need to verify two claims about a set of numbers:


They are random
They are unique in the set (the set is about 200 billion (2TB bin storage))


Supposedly, the numbers were generated by a TRNG (based on a radiation detector). The TRNG itself is a PCI Express card (which we do not have yet) and is probably a TRNG feeding a CSPRNG.

What ways are there to verify the claims? For randomness there are the NIST and other specs and tools.

But how can one tie a verification of the first claim to verifying the second (uniqueness in the set)? Keep in mind that the output of the TRNG has almost certainly been filtered from duplicates. That is most probably going to break the randomness tests.
There is no such thing as a random number and as such it is a category error to attempt to verify whether a particular set of numbers is ‘random’.  There are random generators of numbers; the relevant question is whether you—or, more importantly, a well-funded adversary—can predict the output when you don't know it a priori.  To answer that, you must:


Study the physics of mechanism inside the device.  Hire the world's best theoretical physicists to find the best model with humanity's current understanding of physics for how the mechanism behaves.


What verification can you do here?  You, or the experimental physicists you hired, can devise experiments to rule out alternative hypotheses for how physics might work.  But there's nothing at this stage that you can meaningfully do to the output of your RNG device.

Study the engineering of the device.  Hire competent engineers to determine whether there are ways for the device to fail, and if so, model how it will behave when it does fail.


What verification can you do here?  You, or the statisticians you hired too, can devise a hypothesis test to distinguish a failed component from a working component based on the output of your RNG device.
Make sure that computing the hypothesis test doesn't leak secrets through side channels.

If there is a CSPRNG involved, study the cryptography of the CSPRNG.  Hire the smartest cryptanalysts in the world to find distinguishers for the CSPRNG.


What verification can you do here?  You, or the cryptography engineers you hired too, can write automatic tests using known-answer test vectors to confirm that the CSPRNG implements the same design that has been subjected to decades of scrutiny and vetted by cryptanalysts publishing in serious peer-reviewed academic conferences around the world.
Note that the CSPRNG is essentially guaranteed to prevent hardware health tests, which is why you should apply those first to the hardware RNG output before conditioning it with a CSPRNG.



Alternatively, if you don't like these answers, I hear you can buy a book of certified random numbers from RAND Corporation, so named presumably because they're experts on the matter.



As for detecting duplicates, it depends on what units the data are in.  Duplicate bits?  Well, there's only two possible bits, so you probably have duplicates unless you've found new kinds of bits hitherto unknown to humankind!  Duplicate terabit-length bit strings?  You probably have only one of those even if you have several terabytes of data.  But maybe you meant something like 32-bit or 256-bit strings.  One way to detect duplicates in a large list is to sort it—costing $O(n \log n)$ time in the naive view of things, or $O(n \sqrt n)$ in the more realistic area-time metric which may become relevant in large volumes of data—and then scan through it for duplicates.  But this seems to be a general non-cryptographic algorithm question.