title: Multiple Java webapps total thread limit on linux
body: <p>Recently I observed the "java.lang.OutOfMemoryError: unable to create new native thread" for two standalone Java webapps on the same machine when the total number of threads for both of them reached <strong>1024</strong>.</p>
  
  <p>The command I used to show the number of threads for a process is: <code>ps huH p $pid | wc -l</code></p>
  
  <pre><code>[root@vm119 ~]# ps huH p 11294 | wc -l
  378
  [root@vm119 ~]# ps huH p 11052 | wc -l
  646
  </code></pre>
  
  <p>The java webapps in my case are actually Java daemons spawned from two copies of the same jar file.</p>
  
  <p>At the time of this instance, there were still plenty of free RAM shown by <code>vmstat</code>. I also started another java code(keeps creating new threads until it gets OutOfMemoryError and print out the total amount of threads it creates) to see how many threads it can create. <strong>Expectedly, it said it still can create 31051 threads.</strong> This means that the OS still have native resources required to create native threads at that time.</p>
  
  <p>Both of the Java webapps are started with the following JVM options: <code>-Xmx4096m -Xms512m -Xss256k</code></p>
  
  <p>The <code>ulimit -a</code> on the machine:</p>
  
  <pre><code>[root@vm119 ~]# ulimit -a
  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 62810
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 100000
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 10240
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) 62810
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  </code></pre>
  
  <p>I researched and followed normal investigation routine for "java.lang.OutOfMemoryError: unable to create new native thread" problem, but could not really find anything useful.</p>
  
  <p>The total number of 1024 threads for both webapps seems very suspicious...Was it an accident that they added up to 1024? Or there is some limitation on the OS or JVM that I missed?</p>