title: Japanese Character Encoding in Base64
body: <p>I have been asked to fix a bug in our email processing software.
  When a message whose subject is encoded in <a href="http://tools.ietf.org/pdf/rfc2047.pdf" rel="nofollow">RFC 2047</a> like this:</p>
  
  <pre><code>=?ISO-2022-JP?B?GyRCR1s/LiVGJTklSC1qRnxLXDhsGyhC?=
  </code></pre>
  
  <p>is received, it is incorrectly decoded - one of the Japanese characters is not rendered properly. It is rendered like this: 配信テスト？日本語 when it should be 配信テスト㈱日本語
  (I do not understand Japanese) - clearly one of the characters, the one which looks its in brackets, has not been rendered.</p>
  
  <p>The decoding is carried out by <code>javax.mail.internet.MimeUtility.decodeText()</code> </p>
  
  <p>If I try it with an on-line decoder (the only one I've found is <a href="http://dogmamix.com/MimeHeadersDecoder/" rel="nofollow">here</a>) it seems to work OK, so I was suspecting a bug in <code>MimeUtility</code>.</p>
  
  <p>So I tried some experiments, in the form of this little program:</p>
  
  <pre><code>public class Encoding {
  
    private static final Charset CHARSET = Charset.forName("ISO-2022-JP");
  
    public static void main(String[] args) throws UnsupportedEncodingException {
  
          String control = "繋がって";
          String subject= "配信テスト㈱日本語";            
  
          String controlBase64 = japaneseToBase64(control);
          System.out.println(controlBase64);
          System.out.println(base64ToJapanese(controlBase64));
  
          String subjectBase64 = japaneseToBase64(subject);
          System.out.println(subjectBase64);
          System.out.println(base64ToJapanese(subjectBase64));
  
    }
  
    private static String japaneseToBase64(String in) {
          return Base64.encodeBase64String(in.getBytes(CHARSET));
    }
  
    private static String base64ToJapanese(String in) {
          return new String(Base64.decodeBase64(in), CHARSET);
    }
  
  }
  </code></pre>
  
  <p>(The <code>Base64</code> and <code>Hex</code> classes are in <code>org.apache.commons.codec</code>)</p>
  
  <p>When I run it, here's the output:</p>
  
  <pre><code>GyRCN1IkLCRDJEYbKEI=
  繋がって
  GyRCR1s/LiVGJTklSCEpRnxLXDhsGyhC
  配信テスト？日本語
  </code></pre>
  
  <p>The first, shorter Japanese string is a control, and this returns the same as the input, having been converted into Base64 and back again, using Charset ISO-2022-JP. All OK there.</p>
  
  <p>The second Japanese string is the one with the dodgy character. As you see, it returns with a ? instead of the character. The Base64 encoding output is also different from the original subject encoding. </p>
  
  <p>Sorry if this is long, I wanted to be thorough. What's going on, and how can I decode this character correctly?</p>