title: jvm hang and kill -3 & jmap failed
body: <p>I have a tomcat process that is:</p>
  
  <ol>
  <li>all the threads is not running, all incoming connections is stuck on SYNC_RECV status</li>
  <li>not responding to kill -3</li>
  <li><p>jmap and jstack failed to attach to it </p>
  
  <pre><code>sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap 19938
  Attaching to process ID 19938, please wait...
  Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
  
  sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap -F 19938
  
  Attaching to process ID 19938, please wait...
  Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
  </code></pre></li>
  <li><p>the output of gc stat is not changing, even the <strong>timestamp</strong></p>
  
  <pre><code>sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jstat -gc -t 19938 1000 5
  
  Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
         370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
         370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
         370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
         370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
         370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
  </code></pre></li>
  </ol>
  
  <p>Environment information;</p>
  
  <ol>
  <li><p>linux</p>
  
  <pre><code>Linux xxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre></li>
  <li><p>jdk</p>
  
  <pre><code>java version "1.6.0_45"
  Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
  Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
  </code></pre></li>
  <li><p>JAVA_OPTS</p>
  
  <pre><code>-server -Xms9g -Xmx9g -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=24 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/tomcatdump  
  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/tomcatlog.log -XX:NewSize=3g -XX:MaxNewSize=3g -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
  </code></pre></li>
  </ol>
  
  <p><strong>UPDATES</strong></p>
  
  <p>The <code>SYN_RECV</code> status of connections is a problem of LVS, I think LVS detected the abnormal of tomcat and switch the service to other server, causing the last <code>ACK</code> delivered to another server, then the connections on this server will stuck on <code>SYN_RECV</code> status, but I don't see how that relevant to the jvm hang, huge thanks to @Stephen C</p>
  
  <p><strong>UPDATES2</strong></p>
  
  <p>This process has been stuck on this status for over a week, and the CPU usage is very low</p>