title: AWS ECS docker containers crash randomly with Error code 137 for netcoreapp2.0
body: <p>We have recently migrated our .net4.6 WEB API to netcoreapp2.0
  We are using AWS ECS docker containers for deployment of our services.</p>
  
  <p>Short time load test works fine.
  But long running load test shows that docker containers recyles with error code 137.</p>
  
  <p>During the entire load test, memory and CPU utilization is normal ~30 % both.</p>
  
  <p>As Error 137 is memory related following fixes have been tried.</p>
  
  <ol>
  <li>Changed garbage collection mode :</li>
  </ol>
  
  <blockquote>
    <p>&lt; ServerGarbageCollection>true&lt; /ServerGarbageCollection>
         &lt; ConcurrentGarbageCollection>true&lt; /ConcurrentGarbageCollection></p>
  </blockquote>
  
  <ol start="2">
  <li>Migrated to netcore 2.0.3 , as it has some fixes for memory management.</li>
  </ol>
  
  
  
  <blockquote>
    <p>FROM microsoft/dotnet:2.0.3-runtime</p>
  </blockquote>
  
  <ol start="3">
  <li>Configured cgroup, as below were some error in docker logs</li>
  </ol>
  
  <blockquote>
    <p>cgroup: docker-runc (3365) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
    [   23.104548] cgroup: "memory" requires setting use_hierarchy to 1 on the root</p>
  </blockquote>
  
  <p>Our ECS Task configurations are below :</p>
  
  <ol>
  <li>Number of running Tasks : 2 on 2 C4.xlarge EC2 behind ECS.</li>
  <li>Memory Soft limit : 2 Gb</li>
  <li>Have also validated our Healthcheck endpoint, which does not have any issues and responds fast. Even tried hard coding the healhcheck with 200 Ok</li>
  </ol>
  
  <p>Some Docker logs : (Notice OOM killed is false, even there are no kernel level logs.)</p>
  
  <pre><code> "State": {
          "Status": "exited",
          "Running": false,
          "Paused": false,
          "Restarting": false,
          "OOMKilled": false,
          "Dead": false,
          "Pid": 0,
          "ExitCode": 137,
          "Error": "",
          "StartedAt": "2018-02-12T06:15:00.481719209Z",
          "FinishedAt": "2018-02-12T07:13:02.962733905Z"
      },
  </code></pre>
  
  <p><strong>Some weird observations, if we run load test directly on docker container ip and port. They work just fine. If we run them through ALB, crash behavior is observed.</strong></p>
  
  <p>Please let me know any other linux command which can give me actual reason for process termination or any possible fixes for above case.</p>