title: Converting UnicodeString to AnsiString
body: <p>In the olden times, i had a function that would convert a <code>WideString</code> to an <code>AnsiString</code> of the specified code-page:</p>
  
  <pre><code>function WideStringToString(const Source: WideString; CodePage: UINT): AnsiString;
  ...
  begin
     ...
      // Convert source UTF-16 string (WideString) to the destination using the code-page
      strLen := WideCharToMultiByte(CodePage, 0,
          PWideChar(Source), Length(Source), //Source
          PAnsiChar(cpStr), strLen, //Destination
          nil, nil);
      ...
  end;
  </code></pre>
  
  <p>And everything worked. I passed the function a <em>unicode</em> string (i.e. UTF-16 encoded data) and converted it to an <code>AnsiString</code>, with the understanding that the bytes in the <code>AnsiString</code> represented characters from the specified code-page.</p>
  
  <p>For example:</p>
  
  <pre><code>TUnicodeHelper.WideStringToString('Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ', 1252);
  </code></pre>
  
  <p>would return the <code>Windows-1252</code> encoded string:</p>
  
  <pre><code>The qùíçk brown fôx jumped ovêr the lázÿ dog
  </code></pre>
  
  <blockquote>
    <p>Note: Information was of course lost during the conversion from the full Unicode character set to the limited confines of the Windows-1252 code page:</p>
    
    <ul>
    <li><code>Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ</code> <em>(before)</em></li>
    <li><code>The qùíçk brown fôx jumped ovêr the lázÿ dog</code> <em>(after)</em></li>
    </ul>
  </blockquote>
  
  <p>But the Windows <code>WideChartoMultiByte</code> does a pretty good job of best-fit mapping; as it is designed to do.</p>
  
  <h2>Now the after times</h2>
  
  <p>Now we are in the after times. <code>WideString</code> is now a pariah, with <code>UnicodeString</code> being the goodness. It's an inconsequential change; as the Windows function only needed a <em>pointer</em> to a series of <code>WideChar</code> anyway (which a <code>UnicodeString</code> also is). So we change the declaration to use <code>UnicodeString</code> instead:</p>
  
  <pre><code>funtion WideStringToString(const Source: UnicodeString; CodePage: UINT): AnsiString;
  begin
     ...
  end;
  </code></pre>
  
  <p>Now we come to the return value. i have an <code>AnsiString</code> that contains the bytes:</p>
  
  <pre><code>54 68 65 20 71 F9 ED E7  The qùíç
  6B 20 62 72 6F 77 6E 20  k brown 
  66 F4 78 20 6A 75 6D 70  fôx jump
  65 64 20 6F 76 EA 72 20  ed ovêr 
  74 68 65 20 6C E1 7A FF  the lázÿ
  20 64 6F 67               dog
  </code></pre>
  
  <p>In the olden times that was fine. I kept track of what code-page the <code>AnsiString</code> actually contained; i had to <em>remember</em> that the returned <code>AnsiString</code> was not encoded using the computer's locale (e.g. Windows 1258), but instead is encoded using another code-page (the <code>CodePage</code> code page).</p>
  
  <p>But in Delphi XE6 an <code>AnsiString</code> also secretly contains the codepage:</p>
  
  <ul>
  <li><strong>codePage:</strong> 1258</li>
  <li><strong>length:</strong>   44</li>
  <li><strong>value:</strong> <em><code>The qùíçk brown fôx jumped ovêr the lázÿ dog</code></em></li>
  </ul>
  
  <p>This code-page is wrong. Delphi is specifying the code-page of my computer, rather than the code-page that the string is. Technically this is not a problem, i always understood that the <code>AnsiString</code> was in a particular code-page, i just had to be sure to pass that information along.</p>
  
  <p>So when i wanted to decode the string, i had to pass along the code-page with it:</p>
  
  <pre><code>s := TUnicodeHeper.StringToWideString(s, 1252);
  </code></pre>
  
  <p>with</p>
  
  <pre><code>function StringToWideString(s: AnsiString; CodePage: UINT): UnicodeString;
  begin
     ...
     MultiByteToWideChar(...);
     ...
  end;
  </code></pre>
  
  <h2>Then one person screws everything up</h2>
  
  <p>The problem was that in the olden times i declared a type called <code>Utf8String</code>:</p>
  
  <pre><code>type
     Utf8String = type AnsiString;
  </code></pre>
  
  <p>Because it was common enough to have:</p>
  
  <pre><code>function TUnicodeHelper.WideStringToUtf8(const s: UnicodeString): Utf8String;
  begin
     Result := WideStringToString(s, CP_UTF8);
  end;
  </code></pre>
  
  <p>and the reverse:</p>
  
  <pre><code>function TUnicodeHelper.Utf8ToWideString(const s: Utf8String): UnicodeString;
  begin
     Result := StringToWideString(s, CP_UTF8);
  end;
  </code></pre>
  
  <p>Now in XE6 i have a function that <strong>takes</strong> a <code>Utf8String</code>. If some existing code somewhere were take a UTF-8 encoded <code>AnsiString</code>, and try to convert it to UnicodeString using <code>Utf8ToWideString</code> it would fail:</p>
  
  <pre><code>s: AnsiString;
  s := UnicodeStringToString('Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ', CP_UTF8);
  
  ...
  
   ws: UnicodeString;
   ws := Utf8ToWideString(s); //Delphi will treat s an CP1252, and convert it to UTF8
  </code></pre>
  
  <p>Or worse, is the breadth of existing code that does:</p>
  
  <pre><code>s: Utf8String;
  s := UnicodeStringToString('Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ', CP_UTF8);
  </code></pre>
  
  <p>The returned string will become totally mangled:</p>
  
  <ul>
  <li>the function returns <code>AnsiString(1252)</code> (<code>AnsiString</code> tagged as encoded using the current codepage)</li>
  <li>the return result is being stored in an <code>AnsiString(65001)</code> string (<code>Utf8String</code>)</li>
  <li>Delphi converts the UTF-8 encoded string into UTF-8 as though it was 1252.</li>
  </ul>
  
  <h1>How to move forward</h1>
  
  <p>Ideally my <code>UnicodeStringToString(string, codePage)</code> function (which returns an <code>AnsiString</code>) could set the <code>CodePage</code> inside the string to match the actual code-page using something like <a href="http://docwiki.embarcadero.com/Libraries/XE2/en/System.SetCodePage"><code>SetCodePage</code></a>:</p>
  
  <pre><code>function UnicodeStringToString(s: UnicodeString; CodePage: UINT): AnsiString;
  begin
     ...
     WideCharToMultiByte(...);
     ...
  
     //Adjust the codepage contained in the AnsiString to match reality
     //SetCodePage(Result, CodePage, False); SetCodePage only works on RawByteString
     if Length(Result) &gt; 0 then
        PStrRec(PByte(Result) - SizeOf(StrRec)).codePage := CodePage;
  end;
  </code></pre>
  
  <p>Except that manually mucking around with the internal structure of an <code>AnsiString</code> is horribly dangerous.</p>
  
  <h2>So what about returning <code>RawByteString</code>?</h2>
  
  <p>It has been said, over an over, by a lot of people who aren't me that <code>RawByteString</code> is meant to be the <em>universal recipient</em>; it wasn't meant to be as a return parameter:</p>
  
  <pre><code>function UnicodeStringToString(s: UnicodeString; CodePage: UINT): RawByteString;
  begin
     ...
     WideCharToMultiByte(...);
     ...
  
     //Adjust the codepage contained in the AnsiString to match reality
     SetCodePage(Result, CodePage, False); SetCodePage only works on RawByteString
  end;
  </code></pre>
  
  <p>This has the virtue of being able to use the supported and documented <code>SetCodePage</code>.</p>
  
  <p>But if we're going to cross a line, and start returning <code>RawByteString</code>, surely Delphi already has a function that can convert a <code>UnicodeString</code> to a <code>RawByteString</code> string and vice versa:</p>
  
  <pre><code>function WideStringToString(const s: UnicodeString; CodePage: UINT): RawByteString;
  begin
     Result := SysUtils.Something(s, CodePage);
  end;
  
  function StringToWideString(const s: RawByteString; CodePage: UINT): UnicodeString;
  begin
     Result := SysUtils.SomethingElse(s, CodePage);       
  end;
  </code></pre>
  
  <p>But what is it?</p>
  
  <h1>Or what else should i do?</h1>
  
  <p>This was a long-winded set of background for a trivial question. The <em>real</em> question is, of course, what should i be doing instead? There is a lot of code out there that depends on the <code>UnicodeStringToString</code> and the reverse.</p>
  
  <h1>tl;dr:</h1>
  
  <p>I can convert a <code>UnicodeString</code> to UTF by doing:</p>
  
  <pre><code>Utf8Encode('Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ');
  </code></pre>
  
  <p>and i can convert a <code>UnicodeString</code> to the current code-page by using:</p>
  
  <pre><code>AnsiString('Ŧĥε ｑùíçķ ƀřǭŵņ fôｘ ǰűmpεď ōvêŗ ţħě łáƶÿ ďơǥ');
  </code></pre>
  
  <p>But how do i convert a <code>UnicodeString</code> to an arbitrary (unspecified) code-page?</p>
  
  <p>My feeling is that since everything really is an <code>AnsiString</code>:</p>
  
  <pre><code>Utf8String = AnsiString(65001);
  RawByteString = AnsiString(65535);
  </code></pre>
  
  <p>i should bite the bullet, bust open the <code>AnsiString</code> structure, and poke the correct code-page into it:</p>
  
  <pre><code>function StringToAnsi(const s: UnicodeString; CodePage: UINT): AnsiString;
  begin
     LocaleCharsFromUnicode(CodePage, ..., s, ...);
  
     ...
  
     if Length(Result) &gt; 0 then
        PStrRec(PByte(Result) - SizeOf(StrRec)).codePage := CodePage;
  end;
  </code></pre>
  
  <p>Then the rest of the VCL will fall in line.</p>