title: How do I get perl to read the values from my html form as unicode?
body: <p>I have an html form that sends data to .cgi page.  Here is the html:</p>
  
  <pre><code>&lt;HTML&gt;
  
  &lt;BODY BGCOLOR="#FFFFFF"&gt;
  
      &lt;FORM METHOD="post" ACTION="test.cgi"&gt;
  
  
          &lt;B&gt;Write to me below:&lt;/B&gt;&lt;P&gt;
          &lt;TEXTAREA NAME="feedback" ROWS=10 COLS=50&gt;&lt;/TEXTAREA&gt;&lt;P&gt;
  
          &lt;CENTER&gt;
              &lt;INPUT TYPE=submit VALUE="SEND"&gt;
              &lt;INPUT TYPE=reset VALUE="CLEAR"&gt;
          &lt;/CENTER&gt;
  
      &lt;/FORM&gt;
  
  &lt;/BODY&gt;
  &lt;/HTML&gt; 
  </code></pre>
  
  <p>Here is the perl script for test.cgi:</p>
  
  <pre><code>#!/usr/bin/perl
  
  use utf8;
  use encoding('utf8');
  require Encode;
  require CGI;
  
  
  # The following accepts the data from the form and puts it in %FORM
  
  if ($ENV{'REQUEST_METHOD'} eq 'POST') {
      read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
  
      @pairs = split(/&amp;/, $buffer);
  
      foreach $pair (@pairs) {
          ($name, $value) = split(/=/, $pair);
          $value =~ tr/+/ /;
          $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
  
      $FORM{$name} = $value;
      }
  
  # The following generates the html for the page
  
      print "Content-type: text/html\n\n";
      print "&lt;HTML&gt;\n";
      print "&lt;HEAD&gt;\n";
      print "&lt;TITLE&gt;Thank You!&lt;/TITLE&gt;\n";
      print "&lt;/HEAD&gt;\n";
      print "&lt;BODY BGCOLOR=#FFFFCC TEXT=#000000&gt;\n";  
      print "&lt;H1&gt;Thank You!&lt;/H1&gt;\n";
      print "&lt;P&gt;\n";
      print "&lt;H3&gt;Your feedback is greatly appreciated.&lt;/h3&gt;&lt;BR&gt;\n";
      print "&lt;P&gt;\n&lt;P&gt;\n";
      print "The user wrote:\n\n";
      print "&lt;P&gt;\n";
  
  # This is print statement A
      print "$FORM{'feedback'}&lt;br&gt;\n";
  
      $FORM{'feedback'}=~s/(\w)/ $1/g;
  
  # This is print statement B
      print "$FORM{'feedback'}\n";
  
      print "&lt;/BODY&gt;\n";
      print "&lt;/HTML&gt;\n";
      exit(0);
  }
  </code></pre>
  
  <p>This all works the way it's supposed to if the user enters English text.  However, this will eventually be used in a product where the user will enter Chinese text.  So here's an example of the problem.  If the user enters "中文" into the form, then Print Statement A prints "中文." However, Print Statement B (which prints $value after the regex has been run) prints "&amp;# 2 0 0 1 3;&amp;# 2 5 9 9 1; ".  What I want it to print however is "中 文".  If you want to see this, go to <a href="http://thedeandp.com/chinese/input.html" rel="nofollow">http://thedeandp.com/chinese/input.html</a> and try it yourself.</p>
  
  <p>So basically, what I've figured out is that when perl reads in the form, it's just treating each byte as a character, so the regex adds a space between each byte. Chinese characters use unicode, so it's multiple bytes to a character.  That means the regex breaks up the unicode with a space between the bytes, and that is what produces the output seen in Print Statement B.  I've tried methods like $value = Encode::decode_utf8($value) to get perl to treat it as unicode, but nothing has worked so far.</p>