title: Getting true UTF-8 characters in Java JNI
body: <p>Is there an easy way to convert a Java string to a true UTF-8 byte array in JNI code?</p>
  
  <p>Unfortunately GetStringUTFChars() <em>almost</em> does what's required but not quite, it returns a "modified" UTF-8 byte sequence. The main difference is that a modified UTF-8 doesn't contain any null characters (so you can treat is an ANSI C null terminated string) but another difference seems to be how Unicode supplementary characters such as emoji are treated.</p>
  
  <p>A character such as U+1F604 "SMILING FACE WITH OPEN MOUTH AND SMILING EYES" is stored as a surrogate pair (two UTF-16 characters U+D83D U+DE04) and has a 4-byte UTF-8 equivalent of F0 9F 98 84, and that is the byte sequence that I get if I convert the string to UTF-8 in Java:</p>
  
  <pre><code>    char[] c = Character.toChars(0x1F604);
      String s = new String(c);
      System.out.println(s);
      for (int i=0; i&lt;c.length; ++i)
          System.out.println("c["+i+"] = 0x"+Integer.toHexString(c[i]));
      byte[] b = s.getBytes("UTF-8");
      for (int i=0; i&lt;b.length; ++i)
          System.out.println("b["+i+"] = 0x"+Integer.toHexString(b[i] &amp; 0xFF));
  </code></pre>
  
  <p>The code above prints the following:</p>
  
  <blockquote>
    <p>
    c[0] = 0xd83d
    c[1] = 0xde04
    b[0] = 0xf0
    b[1] = 0x9f
    b[2] = 0x98
    b[3] = 0x84</p>
  </blockquote>
  
  <p>However, if I pass 's' into a native JNI method and call GetStringUTFChars() I get 6 bytes. Each of the surrogate pair characters is being converted to a 3-byte sequence independently:</p>
  
  <pre><code>JNIEXPORT void JNICALL Java_EmojiTest_nativeTest(JNIEnv *env, jclass cls, jstring _s)
  {
      const char* sBytes = env-&gt;GetStringUTFChars(_s, NULL);
      for (int i=0; sBytes[i]!=0; ++i)
          fprintf(stderr, "%d: %02x\n", i, sBytes[i]);
      env-&gt;ReleaseStringUTFChars(_s, sBytes);
      return result;
  }
  </code></pre>
  
  <blockquote>
    <p>0: ed
    1: a0
    2: bd
    3: ed
    4: b8
    5: 84</p>
  </blockquote>
  
  <p>The <a href="https://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8">Wikipedia UTF-8 article</a> suggests that GetStringUTFChars() actually returns CESU-8 rather than UTF-8. That in turn causes my native Mac code to crash because it's not a valid UTF-8 sequence:</p>
  
  <pre><code>CFStringRef str = CFStringCreateWithCString(NULL, path, kCFStringEncodingUTF8);
  CFURLRef url = CFURLCreateWithFileSystemPath(NULL, str, kCFURLPOSIXPathStyle, false);
  </code></pre>
  
  <p>I suppose I could change all my JNI methods to take a byte[] rather than a String and do the UTF-8 conversion in Java but that seems a bit ugly, is there a better solution?</p>