title: Why is there a difference in filesize between using standalone bzip2 and pandas to_csv(, compression='bz2') function?
body: <p>I write out a csv using pandas, and apply bzip2 compression as follows:</p>
  
  <pre class="lang-py prettyprint-override"><code>df.to_csv('/home/user/file.bz2', index=False, mode=writemode, header=header)
  </code></pre>
  
  <p>According to the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html#pandas-dataframe-to-csv" rel="nofollow noreferrer">documentation</a>, <code>to_csv</code> infers from the filename that it needs to compress it using the <code>bzip2</code> method.</p>
  
  <p>This ensures my ~100 MB CSV becomes ~ 23 MB.</p>
  
  <p>However, if I decompress that <code>bz2</code> file, and run the resulting csv file through <code>bzip2</code> on my Mac with:</p>
  
  <pre class="lang-sh prettyprint-override"><code>bzip2 /home/user/file
  </code></pre>
  
  <p>I get a file of ~7 MB! I get the same result if I run <code>bzip2</code> on Debian.</p>
  
  <p>What can cause this difference?</p>