title: Character encoding problem - PHP output, read by .NET, via HttpWebRequest
body: <p>I have a PHP script (running on a Linux server) that ouputs the names of some files on the server. It outputs these file names in a simple text-only format.</p>
  
  <p>This output is read from a VB.NET program by using HttpWebRequest, HttpWebResponse, and a StreamReader.</p>
  
  <p>The problem is that some of the file names being output contain... unusual characters. Specifically, the "section" symbol (§). </p>
  
  <p>If I view the output of the PHP script in a web browser, the symbol appears fine.</p>
  
  <p>But when I read the output of the PHP script into my .NET program, the symbol doesn't appear correctly (it appears as a generic "block" symbol). </p>
  
  <p>I've tried all the different character encoding options that you can use when reading the response stream (from the HttpWebResponse). I've tried outputting the stream directly to a text file (no good), displaying it in a TextBox (no good), and even when viewing the results directly in the Visual Studio debugger, the character appears as a block instead of as the "section" symbol. </p>
  
  <p>I've examined the output in a hex editor (as suggested by a related question, "<a href="https://stackoverflow.com/questions/29499/how-do-you-troubleshoot-character-encoding-problems">how do you troubleshoot character encoding problems</a>." </p>
  
  <p>When I write out the section symbol (§) from .NET itself, the hex bytes I see representing it are "c2 a7" (makes sense if it's unicode, right? requires two bytes?). When I write out the output from the PHP script directly to a file and examine that with a hex editor, the symbol shows up as "ef bf bd" - three bytes instead of two?</p>
  
  <p>I'm at a loss as to what to do - if I need to specify some other character encoding, or if I'm missing something obvious about this. </p>
  
  <p>Here's the code that's used to get the output of the PHP script (VB-style comments modified so they appear correctly on this site):</p>
  
  <pre><code>
  Dim myRequest As HttpWebRequest = WebRequest.Create("http://www.example.com/sample.php")
  
  Dim myResponse As HttpWebResponse = myRequest.GetResponse()
  
  // read the response stream
  Dim myReader As New StreamReader(myResponse.GetResponseStream())
  
  // read the entire output in one block (just as an example)
  Dim theOutput as String = myReader.ReadToEnd()
  </code></pre>
  
  <p>Any ideas? </p>
  
  <ul>
  <li>Am I using the wrong kind of StreamReader? (I've tried passing the character encoding in the call to create the new StreamReader - I've tried all the ones that are in System.Text.Encoding - UTF-8, UTF-7, ASCII, UTF-32, Unicode, etc.)</li>
  <li>Should I be using a different method for reading the output of the PHP script? </li>
  <li>Is there something I should be doing different on the PHP-side when outputting the text?</li>
  </ul>
  
  <p><strong>UPDATED INFO:</strong></p>
  
  <ul>
  <li>The output from PHP is specifically encoded UTF-8 by calling: <code>utf8_encode($file);</code></li>
  <li>When I wrote out the symbol from .NET, I copied and pasted the symbol from the Character Map app in Windows. I also copied &amp; pasted it directly from the file's name (in Windows) and from this web page itself - all gave the same hex value when written out (c2 a7).</li>
  <li>Yes, the "section symbol" I'm talking about is U+00A7 (ALT+0167 on Windows, according to Character Map).</li>
  <li>The content-type is set explicitly via <code>header('Content-Type: text/html; charset=utf-8');</code> right at the beginning of the PHP script.</li>
  </ul>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>Figured it out myself, but I couldn't have done it without the help from the people who answered. Thank you!</p>