Memory Fully utilized by Java ConcurrentHashMap (under Tomcat)
<p>This is a memory stack (serves as a cache) that consist of nothing but a static ConcurrentHashMap (CHM). </p>

<p>All incoming HTTP request data are store in this ConcurrentHashMap. And there is a asynch scheduler process that takes the data from the same ConcurrentHashMap and remove the key.value after storing them into the Database.</p>

<p>This system runs fine and smooth but just discover under following criteria, the memory was fully utilized (2.5GB) and all CPU time was taken to perform GC:</p>

<p>-concurrent http hit of 1000/s</p>

<p>-maintain the same concurrent hit for a period of 15 minutes</p>

<p>The asynch process log the remaining size of the CHM everytime it writes to database. The CHM.size() maintain at around Min:300 to Max:3500 </p>

<p>I thought there is a Memory Leak on this application. so i used Eclipse MAT to look at the Heap Dump. After running the Suspect Report, i got these comments from MAT: </p>

<blockquote>
  <p>One instance of "org.apache.catalina.session.StandardManager" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".</p>
</blockquote>

<pre><code>3,646,166 instances of java.util.concurrent.ConcurrentHashMap$Segment retain &gt;= 2,135,429,456 bytes.
</code></pre>

<p>and</p>

<pre><code>Length    # Objects      Shallow Heap      Retained Heap 
0         3,646,166      482,015,968       &gt;= 2,135,429,456 
</code></pre>

<p>The length 0 above i translate it as empty length record inside the CHM (each time i call CHM.remove() method). It is consistent to the number of record inside the database, 3,646,166 records was inside the database when this dump was created</p>

<p><strong>The strange scenario is</strong>: if i pause the stress test, the utilization in Heap Memory will gradually release down to 25MB.This takes about 30-45 minutes. i have re-simulate this application and the curves looks similar to the VisualVM Graph below:
<img src="http://i.stack.imgur.com/tT3ZE.png" alt="alt text"></p>

<p>Heres the questions:</p>

<p>1) Does this looks like a Memory Leak? </p>

<p>2) Each remove call <code>remove(Object key, Object value)</code> to remove a <code>&lt;key:value&gt;</code> from CHM, does that removed object get GC?</p>

<p>3) Is this something to do with the GC settings? i have added the following GC parameters but without help: </p>

<pre><code>-XX:+UseParallelGC

-XX:+UseParallelOldGC

-XX:GCTimeRatio=19

-XX:+PrintGCTimeStamps

-XX:ParallelGCThreads=6

-verbose:gc
</code></pre>

<p>4) Any idea to resolve this is very much appreciated! :)</p>

<p><strong>NEW</strong> 5) Could it be possible because all my reference are hard reference? My understanding is as long as the HTTP session is ended, all those variables that is not static are now available for GC. </p>

<p><strong>NEW</strong> Note I tried replace the CHM with ehcache 2.2.0, but i get the same OutOfMemoryException problem. i suppose ehcache is also using ConcurrentHashMap. </p>

<p>Server Spec:</p>

<p>-Xeon Quad core, 8 threads. </p>

<p>-4GB Memory</p>

<p>-Windows 2008 R2</p>

<p>-Tomcat 6.0.29</p>
