title: tail in a Docker container: Cannot allocate memory
body: <p>I'm banging my head against the wall with this issue. We're running many containers in parallel, they're running simple filesystem operations or simple linux commands and some of them under certain circumstances fail with memory allocation issues, the Docker container get OOMKiled. </p>
  
  <p>I believe it is not related to the specific command. <code>tail</code> is not the only command that fails, we have also encountered <code>cp</code> or <code>gzip</code>. </p>
  
  <p>We have narrowed down the issue and created a script, that will fail almost certainly when the parameters are adjusted accordingly to the underlying system. </p>
  
  <p><a href="https://github.com/keboola/processor-oom-test" rel="noreferrer">https://github.com/keboola/processor-oom-test</a></p>
  
  <p>The script with the default settings generates a random CSV with 100M rows (~2.5GB), copies it 20 times and then runs 20 containers running <code>tail -n +2 ...</code>. On a <code>m5.2xlarge</code> AWS EC2 instance with 1TB SSD some of the containers are OOMKilled (and some end with different errors). The processes are terminated with various errors:</p>
  
  <pre><code>/code/tail.sh: line 2:    10 Killed                  tail -n +2 '/data/source.csv' &gt; '/data/destination.csv'
  tail: error reading '/data/source.csv': Cannot allocate memory
  tail: write error
  </code></pre>
  
  <p>(the last one is not OOMKilled)</p>
  
  <p>I am not aware that <code>tail</code> should consume any memory at all. If the number of concurrently working containers is low enough, it can easily survive with 64MB of memory. With larger number of containers even 256MB is not enough mempry. I have been watching <code>htop</code> and <code>docker stats</code> and havent seen any spikes in memory consumption.</p>
  
  <p>Things we have already tried</p>
  
  <ul>
  <li>different Docker imges (alpine, centos, ubuntu)</li>
  <li>different file systems (ext3, xfs)</li>
  <li>different os distros (centos, ubuntu)</li>
  <li>different instance providers (Digital Ocean, AWS)</li>
  <li>different types of instances and block devices  </li>
  <li>filesystem swap/swappiness</li>
  <li>Docker memory swap and swappiness</li>
  </ul>
  
  <p>Some of that helped only partially. Adjusting memory limit or number of containers made it crash again every time. We had a container with 1GB memory running simple <code>tail</code> on a large file crash with OOMKilled.</p>
  
  <p>Further on what I have tried several months ago - <a href="https://500.keboola.com/cp-in-docker-cannot-allocate-memory-1a5f57113dc4" rel="noreferrer">https://500.keboola.com/cp-in-docker-cannot-allocate-memory-1a5f57113dc4</a>. And the <code>--memory-swap</code> turned out only to be only a partial help.</p>
  
  <p>Any suggestions? I am no Linux expert, so I may be missing something important. Any help or advice is greatly appreciated.</p>