title: gzipped file is smaller than Shannon limit?
body: <p>I have a text file that's 664KB in size (using ls -l). To my understanding, a file cannot be compressed into anything smaller than the <a href="https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem" rel="nofollow noreferrer">Shannon Source Coding limit</a> without incurring information loss.
  I used a program here to calculate the average shannon entropy of the textfile (4.36) and multiplied it by its number of characters. I get <strong>371KB</strong>.</p>
  <p>Then, I used bzip2 which to my understanding is lossless, and found that it compressed the file to <strong>171K</strong>. From what i understand, nothing can be compressed smaller than the shannon limit without losing information, so how can bzip compress a file losslessly that is smaller than it? Am I missing some information as to how the os is encoding the file perhaps?</p>
  <p><a href="http://classics.mit.edu/Plato/republic.html" rel="nofollow noreferrer">The text file I used for this experiment is MIT Classic's The Republic by Plato</a>.</p>
  <p><a href="https://kennethghartman.com/calculate-file-entropy/" rel="nofollow noreferrer">The program I used to calculate shannon entropy is this one</a>. It gave me the same result as another program I used to cross-check it.</p>