Theoretical limit of human genome compression How small can a compressed file containing the human genome be?
I'm aware that this question cannot actually be answered, since it is asking for the Kolmogorov complexity of the human genome, which is not computable. So reformulating:
What is the largest known lower bound that we can put on the size of a file containing the human genome?
An example answer could be: on average we know that the human genome contain at least this amount X of randomness, therefore it is not possible to do better than Y bytes.
I'm agnostic on which human genome, but I would say without polymorphisms to make it simpler.
The compressed sequence would need to be stand alone, without using a reference (I asked a question on that topic here: Compressing the human genome to few megabytes).
There should be a program able to compress and decompress the genome in a lossless way. Apart from that, any kind of modelling and coding is allowed.
TLDR; For the most modern CHM13 assembly (v1.1; Complete T2T reconstruction of a human genome; https://github.com/marbl/CHM13), the best compression without any use of a reference would be between 624 MB and 641 MB.
Upper estimates of genome compressibility can be made using standard compressors (see, e.g., this comparison). In my experience, xz can achieve great compression ratios on genomic sequences, especially when used with the maximum compression level (-9) and a single thread (-T1). Importantly, the original FASTA file needs to be cleaned – linebreaks hamper compression as they act as noise; the reformatting can be done by seqtk seq -U (which also converts all nucleotides to the upper case). It the case of the CHM13 assembly, xz compresses the human genome to 641,369,600 bytes.
Lower estimates are trickier. The most straightforward approach is through the number of distinct k-mers. Since k-mer sets contain less information than the original assemblies, we can use them for determining the lower bounds of genome compressibility. The lower bound on the compressibility of k-mer sets is 2 bits per distinct k-mer (under certain assumptions that I believe are satisfied in our case). The CHM13 human genome assembly contains 2,497,988,809 distinct canonical 31-mers, which corresponds to 624,497,203 bytes.
Overall, xz does a great job at compressing single human genomes, and the resulting size is very close to the optimum. The full analysis from this post can be found on http://github.com/karel-brinda/human-genome-compression. The resulting files are here: https://zenodo.org/record/4776580.