title: Infinite 100% CPU usage at java.io.FileInputStream.readBytes(Native Method)
body: <p>I'm right now debugging a program which has two threads per one external process, and those two threads keep on reading Process.getErrorStream() and Process.getInputStream() using a <code>while ((i = in.read(buf, 0, buf.length)) &gt;= 0)</code> loop.</p>
  
  <p>Sometimes when the external process crashes due to a JVM crash (see <a href="https://gist.github.com/2405004" rel="noreferrer">these hs_err_pid.log files</a>), those threads which read the stdout/stderr of that external process begin consuming 100% CPU and never exit. <em>The loop body is not being executed</em> (I've added a logging statement there), so the infinite loop appears to be inside the native method <code>java.io.FileInputStream.readBytes</code>.</p>
  
  <p>I've reproduced this on both Windows 7 64-bit (jdk1.6.0_30 64-bit, jdk1.7.0_03 64-bit), and Linux 2.6.18 (jdk1.6.0_21 32-bit). The code in question is <a href="http://code.google.com/p/pitestrunner/source/browse/pitest/src/main/java/org/pitest/util/StreamMonitor.java?r=9210fd33a879c17af2f94b5d94b72ddbaf345a02#38" rel="noreferrer">here</a> and it is used <a href="http://code.google.com/p/pitestrunner/source/browse/pitest/src/main/java/org/pitest/util/JavaProcess.java?r=9210fd33a879c17af2f94b5d94b72ddbaf345a02#43" rel="noreferrer">like this</a>. See those links for the full code - here are the interesting bits:</p>
  
  <pre><code>private final byte[]              buf = new byte[256];
  private final InputStream         in;
  ...    
  
  int i;
  while ((i = this.in.read(this.buf, 0, this.buf.length)) &gt;= 0) {
      ...
  }
  </code></pre>
  
  <p>The stack traces look like</p>
  
  <pre><code>"PIT Stream Monitor" daemon prio=6 tid=0x0000000008869800 nid=0x1f70 runnable [0x000000000d7ff000]
     java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:220)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      - locked &lt;0x00000007c89d6d90&gt; (a java.io.BufferedInputStream)
      at org.pitest.util.StreamMonitor.readFromStream(StreamMonitor.java:38)
      at org.pitest.util.StreamMonitor.process(StreamMonitor.java:32)
      at org.pitest.util.AbstractMonitor.run(AbstractMonitor.java:19)
     Locked ownable synchronizers:
      - None
  </code></pre>
  
  <p>or </p>
  
  <pre><code>"PIT Stream Monitor" daemon prio=6 tid=0x0000000008873000 nid=0x1cb8 runnable [0x000000000e3ff000]
     java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:220)
      at org.pitest.util.StreamMonitor.readFromStream(StreamMonitor.java:38)
      at org.pitest.util.StreamMonitor.process(StreamMonitor.java:32)
      at org.pitest.util.AbstractMonitor.run(AbstractMonitor.java:19)
     Locked ownable synchronizers:
      - None
  </code></pre>
  
  <p>With the Sysinternals Process Explorer I was able to get native stack traces of those threads. Most often, over 80% of the time, the stack trace looks like this:</p>
  
  <pre><code>ntdll.dll!NtReadFile+0xa
  KERNELBASE.dll!ReadFile+0x7a
  kernel32.dll!ReadFile+0x59
  java.dll!handleRead+0x2c
  java.dll!VerifyClassCodesForMajorVersion+0x1d1
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  </code></pre>
  
  <p>This also happens quite often:</p>
  
  <pre><code>ntdll.dll!RtlNtStatusToDosErrorNoTeb+0x52
  ntdll.dll!RtlNtStatusToDosError+0x23
  KERNELBASE.dll!GetCurrentThreadId+0x2c
  KERNELBASE.dll!CreatePipe+0x21a
  kernel32.dll!ReadFile+0x59
  java.dll!handleRead+0x2c
  java.dll!VerifyClassCodesForMajorVersion+0x1d1
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  ntdll.dll!RtlNtStatusToDosErrorNoTeb+0x42
  ntdll.dll!RtlNtStatusToDosError+0x23
  KERNELBASE.dll!GetCurrentThreadId+0x2c
  KERNELBASE.dll!CreatePipe+0x21a
  kernel32.dll!ReadFile+0x59
  java.dll!handleRead+0x2c
  java.dll!VerifyClassCodesForMajorVersion+0x1d1
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  </code></pre>
  
  <p>And sometimes it's executing this part of the code:</p>
  
  <pre><code>java.dll!VerifyClassCodesForMajorVersion+0xc3
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  java.dll!Java_sun_io_Win32ErrorMode_setErrorMode+0x847c
  java.dll!VerifyClassCodesForMajorVersion+0xd7
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  jvm.dll!JNI_GetCreatedJavaVMs+0x1829f
  java.dll!VerifyClassCodesForMajorVersion+0x128
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  jvm.dll+0x88c1
  jvm.dll!JNI_GetCreatedJavaVMs+0x182a7
  java.dll!VerifyClassCodesForMajorVersion+0x128
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  java.dll!VerifyClassCodesForMajorVersion+0x10b
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  jvm.dll!JNI_CreateJavaVM+0x1423
  java.dll!VerifyClassCodesForMajorVersion+0x190
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  jvm.dll+0x88bf
  jvm.dll!JNI_CreateJavaVM+0x147d
  java.dll!VerifyClassCodesForMajorVersion+0x190
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  java.dll!VerifyClassCodesForMajorVersion+0x1aa
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  java.dll!VerifyClassCodesForMajorVersion+0x1c3
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  
  java.dll!VerifyClassCodesForMajorVersion+0x224
  java.dll!Java_java_io_FileInputStream_readBytes+0x1d
  </code></pre>
  
  <p>Any ideas how to solve this problem? Is this a known problem with the JVM? Is there a workaround?</p>