title: K8 container orchestration: JVM not releasing memory
body: <p>We have java based microservices.  When these microservices are not serving any requests, each pod of the microservice is using about 270MB.   We have kept the memory limit as 512Mi for each microservice.</p>
  <p>I did some load testing on some of these microservices. For one microservice, I made 20 API requests per second. For another , 30 requests per second.<br />
  A consistent pattern that I have observed is that: the memory utilization goes up as I increase the load and memory does not get released after the load testing (unless of course that container gets terminated).<br />
  Once I complete the load testing, I wait for an hour or so.   Yet, the memory utilization does not come down. The pod continues occupying same amount of memory as before.<br />
  I also got a shell inside the container of such a pod, I can see only the <code>java process</code> and the <code>sh process</code> running. And indeed, I see that memory usage of the pod has increased to about 360MB.</p>
  <p>So, a pod which usually takes 270MB (in absence of any load testing), takes 360MB under load test (of 30 API requests per second), and continues to use 360MB long after the load testing has completed.</p>
  <p>I do not understand why the pod usage is not coming down.</p>
  <p>Could it be:</p>
  <ol>
  <li>JVM heap usage problem, probably garbage collection not happening?</li>
  <li>Something related to Kubernetes pod requests and limits settings?</li>
  </ol>
  <p>Here are my requests and limits for Kubernetes pods:</p>
  <pre><code>  resources:
      limits:
        memory: 512Mi
        cpu: '0.5'
      requests:
        memory: 256Mi
        cpu: '0.1'
  </code></pre>
  <p>By the way, I am using <code>Apache JMeter</code> for API load testing.
  The Docker image being used is : <code> openjdk:8-alpine</code><br />
  I also downloaded the heap dump and opened it in <code>Eclipse Memory analyzer tool</code>.<br />
  But, I do not see any application classes (from our code) using a lot of heap.<br />
  I see classes related to JDK and Spring framework only, and they do not show any memory leak.</p>
  <p>So, the mystery for me is : why does memory usage not come down even 2-3 hours after load testing?<br />
  Is it the Kubernetes config of requests and limits OR is it the JVM GC settings tuning?<br />
  Is it possible that JVM GC settings are not playing nicely with Kubernetes requests and memory limits?</p>
  <p>adding JVM details for reference:</p>
  <pre><code>openjdk version &quot;1.8.0_212&quot;
  OpenJDK Runtime Environment (IcedTea 3.12.0) (Alpine 8.212.04-r0)
  OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)
  </code></pre>