title: Perl LWP::UserAgent mishandling UTF-8 response
body: <p>When I use LWP::UserAgent to retrieve content encoded in UTF-8 it seems LWP::UserAgent doesn't handle the encoding correctly.</p>
  
  <p>Here's the output after setting the Command Prompt window to Unicode by the command <code>chcp 65001</code> Note that this initially gives the appearance that all is well, but I think it's just the shell reassembling bytes and decoding UTF-8, From the other output you can see that perl itself is not handling wide characters correctly.</p>
  
  <pre>
  C:\>perl getutf8.pl
  ======================================================================
  HTTP/1.1 200 OK
  Connection: close
  Date: Fri, 31 Dec 2010 19:24:04 GMT
  Accept-Ranges: bytes
  Server: Apache/2.2.8 (Win32) PHP/5.2.6
  Content-Length: 75
  Content-Type: application/xml; charset=utf-8
  Last-Modified: Fri, 31 Dec 2010 19:20:18 GMT
  Client-Date: Fri, 31 Dec 2010 19:24:04 GMT
  Client-Peer: 127.0.0.1:80
  Client-Response-Num: 1
  
  &lt;?xml version="1.0" encoding="UTF-8"?>
  &lt;name>Budějovický Budvar&lt;/name>
  
  ======================================================================
  response content length is 33
  
  ....v....1....v....2....v....3....v....4
  &lt;name>Budějovický Budvar&lt;/name>
  
  . . . . v . . . . 1 . . . . v . . . . 2 . . . . v . . . . 3 . . . .
  3c6e616d653e427564c49b6a6f7669636bc3bd204275647661723c2f6e616d653e
  &lt; n a m e > B u d � � j o v i c k � �   B u d v a r &lt; / n a m e >
  </pre>
  
  <p>Above you can see the payload length is 31 characters but Perl thinks it is 33.
  For confirmation, in the hex, we can see that the UTF-8 sequences c49b and c3bd are being interpreted as four separate characters and not as two Unicode characters.</p>
  
  <p>Here's the code</p>
  
  <pre>
  #!perl
  use strict;
  use warnings;
  use LWP::UserAgent;
  
  my $ua = LWP::UserAgent->new();
  my $response = $ua->get('http://localhost/Bud.xml');
  if (! $response->is_success) { die $response->status_line; }
  
  print '='x70,"\n",$response->as_string(), '='x70,"\n";
  
  my $r = $response->decoded_content((charset => 'UTF-8')); 
  $/ = "\x0d\x0a"; # seems to be \x0a otherwise!
  chomp($r);
  
  # Remove any xml prologue
  $r =~ s/^&lt;\?.*\?>\x0d\x0a//;
  
  print "Response content length is ", length($r), "\n\n";
  print "....v....1....v....2....v....3....v....4\n";
  print $r,"\n";
  
  print ". . . . v . . . . 1 . . . . v . . . . 2 . . . . v . . . . 3 . . . . \n";
  print unpack("H*", $r), "\n";
  print join(" ", split("", $r)), "\n";
  
  </pre>
  
  <p>Note that Bud.xml is UTF-8 encoded without a BOM.</p>
  
  <p><strong>How can I persuade LWP::UserAgent to do the right thing?</strong></p>
  
  <p><em>P.S. Ultimately I want to translate the Unicode data into an ASCII encoding, even if it means replacing each non-ASCII character with <strong>one</strong> question mark or other marker.</em></p>
  
  <hr>
  
  <h3>Update 1</h3>
  
  <p>I have accepted Ysth's "upgrade" answer - because I know it is the right thing to do when possible. However there is a work around to fix up the data into a well formed Perl Unicode string.</p>
  
  <pre><code>$r = decode("utf8", $r);
  </code></pre>
  
  <hr>
  
  <h3>Update 2</h3>
  
  <p>My data gets fed to a non-Perl application that displays the data using Code Page 437 to Putty/Reflection/Teraterm terminals at many locations. The app is currently displaying something like:</p>
  
  <pre>Bud├ä┬øjovick├â┬¢ Budvar</pre>
  
  <p>I am going to use <code>($r = decode("UTF-8", $r)) =~ s/[\x80-\x{FFFF}]/\xFE/g;</code> to get the app to display:</p>
  
  <pre>Bud■jovick■ Budvar</pre>
  
  <p>Moving away from CP437 would be a major job, so that is not going to happen in the short to medium term.</p>
  
  <hr>
  
  <h3>Update 3</h3>
  
  <p>CPAN has some interesting Unicode modules such as:</p>
  
  <ul>
  <li>Text::Unidecode</li>
  <li>Unicode::Map8</li>
  <li>Unicode::Map</li>
  <li>Unicode::Escape</li>
  <li>Unicode::Transliterate</li>
  </ul>
  
  <p>Text::Unidecode translated "Budějovický Budvar" into "Budejovicky Budvar" - which didn't seem to me a particularly impressive attempt at a phonetic transliteration but then I don't speak Czech. English speakers might prefer it to "Bud■jovick■ Budvar" though.</p>