estimating entropy/randomness as fail-safe mechanism Real-world encryption systems are often not broken algorithmically but because the implementation has a bug. Sometimes the bug only happens in circumstances that could not have been foreseen, such as weak random numbers, or an error in a library causing e.g. predictable IVs.

So I wonder, are there workable methods of estimating randomness or entropy that could be helpful to detect (a subset of) actual failures in cryptographic systems?

As a simple example, such an estimator could be fed with consecutive CTR IVs and alert higher protocol layers when there is an apparent failure, because all IVs are identical for example. Or an estimator could be fed with encrypted DES output to detect a bug in the implemention.
Such practices are commonly applied to results of some random number generators, but less often to other applications.

There is one very common such fail-safe test

NIST's FIPS 140-2 requires very simple contiguous test for random number generation: store previous generated block and compare. Only if the block is different it is assumed that the RNG did not fail.

From FIPS 140-2 4.9.2:


  Continuous random number generator test. If a cryptographic module employs Approved or non- Approved RNGs in an Approved mode of operation, the module shall perform the following continuous random number generator test on each RNG that tests for failure to a constant value.


More entropy tests

The standard (FIPS 140-2) used to require more tests (mono bit test, poker test, runs test, long runs test). These tests give fairly good estimate of if input appears random. These were decommissioned, because they work better with NDRBG than RNGs required by the standard (see FIPS 140-2 Annex C if you need details), and because for some cases it is excessive requirement to produce 20000 bits for analysis (it may be that you just needed one 128-bit key). AIS31 standard used to evaluate true random number generators uses some very similar tests.

Used in practice

These tests are useful against some hardware random bit generators and for this reason, Linux rngd program is running the tests (previously part of FIPS 140-2) to ensure quality of entropy gotten (for example from /dev/hwrng) and to ignore inputs which appear non-random.

You may use Linux rngtest program to apply these tests to data of your choice, for instance, examine output of (possibly) buggy encryption software.