How to compute the dataset size required by dieharder tests? I am trying to use the dieharder random number test suite.
However, this suite requires massive amount of data (this paper claims 228GB for no rewinding for every test) for most of the tests. I'd like to understand how rewinding is implemented and to know how much data is required for each test. It seems related to the tsamples value but I cannot find an exact correlation.
For example, if I generate 400MiB of data using dd if=/dev/urandom of=random.dat bs=1M count=400, this produces the following (cropped) output:
$ dieharder -a -g 201 -f random.dat
#=============================================================================#
#            dieharder version 3.31.1 Copyright 2003 Robert G. Brown          #
#=============================================================================#
   rng_name    |           filename             |rands/second|
 file_input_raw|                      random.dat|  6.38e+07  |
#=============================================================================#
        test_name   |ntup| tsamples |psamples|  p-value |Assessment
#=============================================================================#
   diehard_birthdays|   0|       100|     100|0.70238534|  PASSED  
# The file file_input_raw was rewound 1 times
      diehard_operm5|   0|   1000000|     100|0.74169241|  PASSED  
# The file file_input_raw was rewound 2 times
  diehard_rank_32x32|   0|     40000|     100|0.03164866|  PASSED  
# The file file_input_raw was rewound 2 times
    diehard_rank_6x8|   0|    100000|     100|0.18168084|  PASSED  
# The file file_input_raw was rewound 3 times
   diehard_bitstream|   0|   2097152|     100|0.59056568|  PASSED  
# The file file_input_raw was rewound 5 times
        diehard_opso|   0|   2097152|     100|0.19175240|  PASSED  
# The file file_input_raw was rewound 6 times
        diehard_oqso|   0|   2097152|     100|0.67542036|  PASSED  
# The file file_input_raw was rewound 7 times
         diehard_dna|   0|   2097152|     100|0.04906220|  PASSED  
# The file file_input_raw was rewound 7 times
diehard_count_1s_str|   0|    256000|     100|0.80349572|  PASSED  
# The file file_input_raw was rewound 8 times
diehard_count_1s_byt|   0|    256000|     100|0.43003125|  PASSED  
# The file file_input_raw was rewound 8 times
 diehard_parking_lot|   0|     12000|     100|0.07270933|  PASSED  
# The file file_input_raw was rewound 8 times
    diehard_2dsphere|   2|      8000|     100|0.92548415|  PASSED  

I assume that dieharder just loops over the file, keeping its pointer position when jumping to the next test and that The file file_input_raw was rewound N times is just the total count that the pointer reached the end of the file.
I'll be so bold as to say that no one knows for sure how much data you really need. There is no guidance on the homepage, the manpage and this thread from Duke is equally unspecific. Anecdotal evidence suggests that a 'few' GB will return a sane PASS/FAIL answer, but there will still be rewinds. For a 4 GB /dev/urandom test:-
...snip
# The file file_input_raw was rewound 43 times
      rgb_lagged_sum|  27|   1000000|     100|0.75608263|  PASSED  
# The file file_input_raw was rewound 46 times
      rgb_lagged_sum|  28|   1000000|     100|0.88574962|  PASSED  
# The file file_input_raw was rewound 49 times
      rgb_lagged_sum|  29|   1000000|     100|0.69532845|  PASSED  
# The file file_input_raw was rewound 52 times
      rgb_lagged_sum|  30|   1000000|     100|0.93874399|  PASSED  
# The file file_input_raw was rewound 55 times
      rgb_lagged_sum|  31|   1000000|     100|0.59282678|  PASSED  
# The file file_input_raw was rewound 58 times
      rgb_lagged_sum|  32|   1000000|     100|0.88988310|  PASSED  

58 times!
In your case (input file), a rewind is just that; start again from the beginning. Eight rewinds is good. I've seen 1849 rewinds. Honestly I can't understand the reasoning behind that, as all it does is retest what's been tested. All we know for certain is that Dieharder uses a lot. Your paper and my research shows that a free run of Dieharder (with no rewinds) needs approximately 225 - 250 GB! Notice that $58 \times 4 \text{GB}= 232 \text{GB}$.
There are other tests though. A more common test in the TRNG scientific literature is NIST's Statistical Test Suite, (SP 800-22). It's sample requirements are much more predictable.
And now a warning. You have to resist manipulating any randomness tests in order to get a PASS. Tweaking test parameters or selectively choosing individual tests to produce your expected result is dangerous, as a random sample is random no matter how you test it. From the manpage:-

...where a file that is too small will &quot;rewind&quot; and render the test results where a rewind occurs suspect.


My version 3.31.1 will not allow the number of p samples to be changed via the -p switch.