title: Java using much more memory than heap size (or size correctly Docker memory limit)
body: <p>For my application, the memory used by the Java process is much more than the heap size.</p>
  
  <p>The system where the containers are running starts to have memory problem because the container is taking much more memory than the heap size.</p>
  
  <p>The heap size is set to 128 MB (<code>-Xmx128m -Xms128m</code>) while the container takes up to 1GB of memory. Under normal condition, it needs 500MB. If the docker container has a limit below (e.g. <code>mem_limit=mem_limit=400MB</code>) the process gets killed by the out of memory killer of the OS.</p>
  
  <p><strong>Could you explain why the Java process is using much more memory than the heap? How to size correctly the Docker memory limit? Is there a way to reduce the off-heap memory footprint of the Java process?</strong></p>
  
  <hr>
  
  <p>I gather some details about the issue using command from <a href="https://gist.github.com/prasanthj/48e7063cac88eb396bc9961fb3149b58" rel="noreferrer">Native memory tracking in JVM</a>.</p>
  
  <p>From the host system, I get the memory used by the container.</p>
  
  <pre><code>$ docker stats --no-stream 9afcb62a26c8
  CONTAINER ID        NAME                                                                                        CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
  9afcb62a26c8        xx-xxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.0acbb46bb6fe3ae1b1c99aff3a6073bb7b7ecf85   0.93%               461MiB / 9.744GiB   4.62%               286MB / 7.92MB      157MB / 2.66GB      57
  </code></pre>
  
  <p>From inside the container, I get the memory used by the process.</p>
  
  <pre><code>$ ps -p 71 -o pcpu,rss,size,vsize
  %CPU   RSS  SIZE    VSZ
  11.2 486040 580860 3814600
  </code></pre>
  
  <hr>
  
  <pre><code>$ jcmd 71 VM.native_memory
  71:
  
  Native Memory Tracking:
  
  Total: reserved=1631932KB, committed=367400KB
  -                 Java Heap (reserved=131072KB, committed=131072KB)
                              (mmap: reserved=131072KB, committed=131072KB) 
  
  -                     Class (reserved=1120142KB, committed=79830KB)
                              (classes #15267)
                              (  instance classes #14230, array classes #1037)
                              (malloc=1934KB #32977) 
                              (mmap: reserved=1118208KB, committed=77896KB) 
                              (  Metadata:   )
                              (    reserved=69632KB, committed=68272KB)
                              (    used=66725KB)
                              (    free=1547KB)
                              (    waste=0KB =0.00%)
                              (  Class space:)
                              (    reserved=1048576KB, committed=9624KB)
                              (    used=8939KB)
                              (    free=685KB)
                              (    waste=0KB =0.00%)
  
  -                    Thread (reserved=24786KB, committed=5294KB)
                              (thread #56)
                              (stack: reserved=24500KB, committed=5008KB)
                              (malloc=198KB #293) 
                              (arena=88KB #110)
  
  -                      Code (reserved=250635KB, committed=45907KB)
                              (malloc=2947KB #13459) 
                              (mmap: reserved=247688KB, committed=42960KB) 
  
  -                        GC (reserved=48091KB, committed=48091KB)
                              (malloc=10439KB #18634) 
                              (mmap: reserved=37652KB, committed=37652KB) 
  
  -                  Compiler (reserved=358KB, committed=358KB)
                              (malloc=249KB #1450) 
                              (arena=109KB #5)
  
  -                  Internal (reserved=1165KB, committed=1165KB)
                              (malloc=1125KB #3363) 
                              (mmap: reserved=40KB, committed=40KB) 
  
  -                     Other (reserved=16696KB, committed=16696KB)
                              (malloc=16696KB #35) 
  
  -                    Symbol (reserved=15277KB, committed=15277KB)
                              (malloc=13543KB #180850) 
                              (arena=1734KB #1)
  
  -    Native Memory Tracking (reserved=4436KB, committed=4436KB)
                              (malloc=378KB #5359) 
                              (tracking overhead=4058KB)
  
  -        Shared class space (reserved=17144KB, committed=17144KB)
                              (mmap: reserved=17144KB, committed=17144KB) 
  
  -               Arena Chunk (reserved=1850KB, committed=1850KB)
                              (malloc=1850KB) 
  
  -                   Logging (reserved=4KB, committed=4KB)
                              (malloc=4KB #179) 
  
  -                 Arguments (reserved=19KB, committed=19KB)
                              (malloc=19KB #512) 
  
  -                    Module (reserved=258KB, committed=258KB)
                              (malloc=258KB #2356) 
  
  $ cat /proc/71/smaps | grep Rss | cut -d: -f2 | tr -d " " | cut -f1 -dk | sort -n | awk '{ sum += $1 } END { print sum }'
  491080
  </code></pre>
  
  <p>The application is a web server using Jetty/Jersey/CDI bundled inside a fat far of 36 MB.</p>
  
  <p>The following version of OS and Java are used (inside the container). The Docker image is based on <code>openjdk:11-jre-slim</code>.</p>
  
  <pre><code>$ java -version
  openjdk version "11" 2018-09-25
  OpenJDK Runtime Environment (build 11+28-Debian-1)
  OpenJDK 64-Bit Server VM (build 11+28-Debian-1, mixed mode, sharing)
  $ uname -a
  Linux service1 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 GNU/Linux
  </code></pre>
  
  <p><a href="https://gist.github.com/prasanthj/48e7063cac88eb396bc9961fb3149b58" rel="noreferrer">https://gist.github.com/prasanthj/48e7063cac88eb396bc9961fb3149b58</a></p>