What is FIPS 140-2 for? I've just used the FIPS 140-2 tests as implemented in rngtools, the rngtest.  I ran it on a 12.3MB file of pak8 compressed ASCII text.  It passed most of the tests, with a 0.2% failure rate.  Monobit and Continuous runs tests actually complete without any failures.

It seems very lax at detecting randomness.  What's the point of this test if it (effectively) passes off a highly structured zip like file as good quality random numbers? 

Here is the detail... 

cat /tmp/raw.txt.fp8 | rngtest

rngtest: starting FIPS tests...
rngtest: entropy source drained
rngtest: bits received from input: 98327312
rngtest: FIPS 140-2 successes: 4906
rngtest: FIPS 140-2 failures: 10
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 1
rngtest: FIPS 140-2(2001-10-10) Runs: 1
rngtest: FIPS 140-2(2001-10-10) Long run: 8
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
First, in a strict sense, detecting randomness by statistical tests is impossible. One can only detect absence of randomness. 

Second, a perfect compression algorithm should generate an output which is indistinguishable to random numbers. A hand-waving argument goes as follows:
If the output would not be indistinguishable from random numbers, then it would contain some remaining structure. And this structure could be used to compress the input even more.  

This means, the only thing you can probably do is to throw many good statistical tests to your random numbers. It cannot prove that your source is true random but it can give you some evidence that it is not broken.  

A third point to mention is that NIST has already superseeded the simple test of Fips 140-2 by the far more comprehensive tests of the Nist SP 800-90B publication.