Why Pandas can load a gzip file in chunks while Dask cannot?
Specifically, you cannot random access a gzipped file: to get to a particular point in the decompressed data, you need to read through the whole file to that point. Thus, you can load chunk-by-chunk sequentially, maintaining the file position.
However, dask wants to be able to access some arbitrary offset in the file, so that each partition can be loaded independently and in parallel. For gzip, this would mean each task spooling through the file to the offset they need, which is a really bad idea.
I would like to know why Pandas can open in chunks a gzip-compressed file while Dask has to load to memory the whole file before processing it.
Concretely, I have a gzip-compressed csv file of size ~10GB (and ~50GB when not compressed). My computer has 16GB of RAM.
I thought gzip files cannot be operated in chunks because they need to be decompressed before they can be partitioned. However, I am able to use the chunk method of Pandas to process the gzip file with no issue.
Additionaly, since I want to use Dask in the future, can you recommend me any compressed format that can easily be read in chunks by Dask ?