title: Problem with decoding UTF-8 JSON in perl
body: <p>UTF-8 characters are destroyed when processed with the JSON library (maybe this is similar to <a href="https://stackoverflow.com/questions/5536796/problem-with-decoding-unicode-json-in-perl">Problem with decoding unicode JSON in perl</a>, however setting binmode only creates another problem).</p>
  
  <p>I have reduced the problem down to the following example:</p>
  
  <pre><code>(hlovdal) localhost:/tmp/my_test&gt;cat my_test.pl
  #!/usr/bin/perl -w
  use strict;
  use warnings;
  use JSON;
  use File::Slurp;
  use Getopt::Long;
  use Encode;
  
  my $set_binmode = 0;
  GetOptions("set-binmode" =&gt; \$set_binmode);
  
  if ($set_binmode) {
          binmode(STDIN,  ":encoding(UTF-8)");
          binmode(STDOUT, ":encoding(UTF-8)");
          binmode(STDERR, ":encoding(UTF-8)");
  }
  
  sub check {
          my $text = shift;
          return "is_utf8(): " . (Encode::is_utf8($text) ? "1" : "0") . ", is_utf8(1): " . (Encode::is_utf8($text, 1) ? "1" : "0"). ". ";
  }
  
  my $my_test = "hei på deg";
  my $json_text = read_file('my_test.json');
  my $hash_ref = JSON-&gt;new-&gt;utf8-&gt;decode($json_text);
  
  print check($my_test), "\$my_test = $my_test\n";
  print check($json_text), "\$json_text = $json_text";
  print check($$hash_ref{'my_test'}), "\$\$hash_ref{'my_test'} = " . $$hash_ref{'my_test'} .  "\n";
  
  (hlovdal) localhost:/tmp/my_test&gt;
  </code></pre>
  
  <p>When running  testing the text is for some reason crippeled into iso-8859-1. Setting binmode sort of solves it but then causes double encoding of other strings.</p>
  
  <pre><code>(hlovdal) localhost:/tmp/my_test&gt;cat my_test.json 
  { "my_test" : "hei på deg" }
  (hlovdal) localhost:/tmp/my_test&gt;file my_test.json 
  my_test.json: UTF-8 Unicode text
  (hlovdal) localhost:/tmp/my_test&gt;hexdump -c my_test.json 
  0000000   {       "   m   y   _   t   e   s   t   "       :       "   h
  0000010   e   i       p 303 245       d   e   g   "       }  \n        
  000001e
  (hlovdal) localhost:/tmp/my_test&gt;
  (hlovdal) localhost:/tmp/my_test&gt;perl my_test.pl
  is_utf8(): 0, is_utf8(1): 0. $my_test = hei på deg
  is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
  is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
  (hlovdal) localhost:/tmp/my_test&gt;perl my_test.pl --set-binmode
  is_utf8(): 0, is_utf8(1): 0. $my_test = hei pÃ¥ deg
  is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei pÃ¥ deg" }
  is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
  (hlovdal) localhost:/tmp/my_test&gt;
  </code></pre>
  
  <p><strong>What is causing this and how to solve?</strong></p>
  
  <hr>
  
  <p>This is on a newly installed and up to date Fedora 15 system.</p>
  
  <pre><code>(hlovdal) localhost:/tmp/my_test&gt;perl --version | grep version
  This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-thread-multi
  (hlovdal) localhost:/tmp/my_test&gt;rpm -q perl-JSON
  perl-JSON-2.51-1.fc15.noarch
  (hlovdal) localhost:/tmp/my_test&gt;locale
  LANG=en_US.UTF-8
  LC_CTYPE="en_US.UTF-8"
  LC_NUMERIC="en_US.UTF-8"
  LC_TIME="en_US.UTF-8"
  LC_COLLATE="en_US.UTF-8"
  LC_MONETARY="en_US.UTF-8"
  LC_MESSAGES="en_US.UTF-8"
  LC_PAPER="en_US.UTF-8"
  LC_NAME="en_US.UTF-8"
  LC_ADDRESS="en_US.UTF-8"
  LC_TELEPHONE="en_US.UTF-8"
  LC_MEASUREMENT="en_US.UTF-8"
  LC_IDENTIFICATION="en_US.UTF-8"
  LC_ALL=
  (hlovdal) localhost:/tmp/my_test&gt;
  </code></pre>
  
  <hr>
  
  <p>Update: Adding <code>use utf8</code> does not solve it, characters are still not processed right (although slightly different from before):</p>
  
  <pre><code>(hlovdal) localhost:/tmp/my_test&gt;perl  my_test.pl
  is_utf8(): 1, is_utf8(1): 1. $my_test = hei p� deg
  is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
  is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
  (hlovdal) localhost:/tmp/my_test&gt;perl  my_test.pl --set-binmode
  is_utf8(): 1, is_utf8(1): 1. $my_test = hei på deg
  is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei pÃ¥ deg" }
  is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
  (hlovdal) localhost:/tmp/my_test&gt;
  </code></pre>
  
  <p>As noted by <a href="http://perldoc.perl.org/perlunifaq.html#Can-I-use-Unicode-in-my-Perl-sources?" rel="nofollow noreferrer">perlunifaq</a></p>
  
  <blockquote>
    <p>Can I use Unicode in my Perl sources?</p>
    
    <p>Yes, you can! If your sources are
    UTF-8 encoded, you can indicate that
    with the use utf8 pragma.</p>
  
  <pre><code>use utf8;
  </code></pre>
    
    <p>This doesn't do anything to your
    input, or to your output. It only
    influences the way your sources are
    read. You can use Unicode in string
    literals, in identifiers (but they
    still have to be "word characters"
    according to \w ), and even in custom
    delimiters.</p>
  </blockquote>