title: Unicode problems with Base64 strings and Streams
body: <p>I have been working on a project in Lazarus and have decided to move it to Delphi XE for the time being (due to some limitations).</p>
  
  <p>A brief overview of what is going on:</p>
  
  <p>At runtime I am loading external files and adding them to streams. The streams belong to several different classes that descend from one main object (TObject). These classes are added to a TList from the main object, basically each class has its own stream property and the class is child to the main object.</p>
  
  <p>In this main object I have a save and load procedure:</p>
  
  <p>When saving the object it also saves all the stream data from the other classes to file by using string to stream. The output string here must be base64 encoded as I am saving to XML.</p>
  
  <p>When opening the file, the idea is to decode the base64 string and move it back into the streams just as if it were the original file before it was base64 encoded.</p>
  
  <p>In Lazarus it works, and here is the important code (note, some of it was not written by me).</p>
  
  <pre><code>const
    Keys64 = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/';
  
  function Encode64String(S: string): string;
  function Decode64String(S: string): string;
  function Encode64StringToStream(const Input: TStream; var Output: string): Boolean;
  procedure Decode64StringToStream(const Input: string; Output: TStream);
  procedure StringToStream(Stream: TStream; const S: string);
  function StreamToString(MS: TMemoryStream): string;
  
  implementation
  
  function Encode64String(S: string): string;
  var
    i: Integer;
    a: Integer;
    x: Integer;
    b: Integer;
  begin
    Result := '';
    a := 0;
    b := 0;
    for i := 1 to Length(s) do
    begin
      x := Ord(s[i]);
      b := b * 256 + x;
      a := a + 8;
      while a &gt;= 6 do
      begin
        a := a - 6;
        x := b div (1 shl a);
        b := b mod (1 shl a);
        Result := Result + Keys64[x + 1];
      end;
    end;
    if a &gt; 0 then
    begin
      x := b shl (6 - a);
      Result := Result + Keys64[x + 1];
    end;
  end;
  
  function Decode64String(S: string): string;
  var
    i: Integer;
    a: Integer;
    x: Integer;
    b: Integer;
  begin
    Result := '';
    a := 0;
    b := 0;
    for i := 1 to Length(s) do
    begin
      x := Pos(s[i], Keys64) - 1;
      if x &gt;= 0 then
      begin
        b := b * 64 + x;
        a := a + 6;
        if a &gt;= 8 then
        begin
          a := a - 8;
          x := b shr a;
          b := b mod (1 shl a);
          x := x mod 256;
          Result := Result + chr(x);
        end;
      end
      else
        Exit;
    end;
  end;
  
  function Encode64StringToStream(const Input: TStream; var Output: string): Boolean;
  var
    MS: TMemoryStream;
  begin
    Result := False;
  
    MS := TMemoryStream.Create;
    try
      Input.Seek(0, soFromBeginning);
      MS.CopyFrom(Input, Input.Size);
      MS.Seek(0, soFromBeginning);
      Output := Encode64String(StreamToString(MS));
    finally
      MS.Free;
    end;
  
    Result := True;
  end;    
  
  procedure Decode64StringToStream(const Input: string; Output: TStream);
  var
    MS: TMemoryStream;
  begin
    try
      MS := TMemoryStream.Create;
      try
        StringToStream(MS, Decode64String(Input));
  
        MS.Seek(0, soFromBeginning);
        Output.CopyFrom(MS, MS.Size);
        Output.Position := 0;
      finally
        MS.Free;
      end;
  
    except on E: Exception do
      raise Exception.Create('stream decode error - ' + E.Message);
    end;
  end;
  
  procedure StringToStream(Stream: TStream; const S: string);
  begin
    Stream.Write(Pointer(S)^, Length(S));
  end;
  
  function StreamToString(MS: TMemoryStream): string;
  begin
    SetString(Result, PChar(MS.Memory), MS.Size div SizeOf(Char));
  end;
  </code></pre>
  
  <p>I am 99% sure the problem here is going to be unicode related. It's a shame because I believe Lazarus/Freepascal has always been unicode but not Delphi and so uses different string types making it almost impossible for the less professional users like myself to solve!</p>
  
  <p>To be honest I think all the code above is a bit of a mess, and it feels like I am just trying to guess what to change the strings to without really knowing what I am doing.</p>
  
  <p>My first thought was to change everything from <code>String</code> to <code>AnsiString</code>. This nearly worked one time but when trying to use <code>Decode64StringToStream</code> I got zero data back. Other times the data was not properly saving as base64 encoded format, and sometimes I even got errors like TStream.Seek not implemented or something.</p>
  
  <p>PS, I have read the guides and there is plenty around such as the whitepapers etc on how to migrate old Delphi projects to newer unicode versions and to be honest I am still at a loss with it. I thought replacing <code>string</code> to <code>AnsiString</code> would have been enough, but it seems it isn't.</p>
  
  <p>Any tips, pointers or general advice or clues would be greatly appreciated thanks.</p>