title: TRestClient/TRestRequest incorrectly decodes gzip response
body: <p>I tried to read a REST API, which is gzip encoded. To be exact, I tried to read the StackExchange API.</p>
  
  <p>I already found the question <a href="https://stackoverflow.com/questions/26610376/automatically-decode-gzip-in-trestresponse">Automatically Decode GZIP In TRESTResponse?</a>, but that answer doesn't solve my issue for some reason.</p>
  
  <p><strong>Test setup</strong></p>
  
  <p>In XE5, I added a TRestClient, a TRestRequest and a TRestResponse with the following relevant properties. I set the BaseURL of the client, the resource and parameters of the request, and I set <code>AcceptEncoding</code> of the request to <code>gzip, deflate</code>, which should make it automatically decode gzipped responses.</p>
  
  <pre><code>  object RESTClient1: TRESTClient
      BaseURL = 'https://api.stackexchange.com/2.2'
    end
    object RESTRequest1: TRESTRequest
      AcceptEncoding = 'gzip, deflate'
      Client = RESTClient1
      Params = &lt;
        item
          Kind = pkURLSEGMENT
          name = 'id'
          Options = [poAutoCreated]
          Value = '511529'
        end
        item
          name = 'site'
          Value = 'stackoverflow'
        end&gt;
      Resource = 'users/{id}'
      Response = RESTResponse1
    end
    object RESTResponse1: TRESTResponse
    end
  </code></pre>
  
  <p>This results in the url:</p>
  
  <blockquote>
    <p><a href="https://api.stackexchange.com/2.2/users/511529?site=stackoverflow">https://api.stackexchange.com/2.2/users/511529?site=stackoverflow</a></p>
  </blockquote>
  
  <p>I invoke the request like this, with two message boxes to show the url and the outcome of the request:</p>
  
  <pre><code>ShowMessage(RESTRequest1.GetFullRequestURL());
  RESTRequest1.Execute; // Actual call
  ShowMessage(RESTResponse1.Content);
  </code></pre>
  
  <p>If I call that url in a browser, I get a proper result, which is a json object with some of my user information in it.</p>
  
  <p><strong>Problem</strong></p>
  
  <p>However, in Delphi, I don't get the JSON response. In fact, I get a bunch of bytes which <em>seems</em> to be a mangled gzip response. I tried to decompress it with <code>TIdCompressorZlib.DecompressGZipStream()</code>, but it fails with a <code>ZLib Error (-3)</code>. When I inspect the bytes of the response myself, I see it starts with #1F#3F#08. This is especially weird, since the gzip header should be #1F#8B#08, so #8B is transformed into #3F, which is a question mark. </p>
  
  <p>So it seems to me like the RESTClient has attempted to decode the gzip stream as if it was a UTF-8 response, and has replaced invalid sequences (#8B is in itself not a valid UTF-8 character) with a question mark. </p>
  
  <p><strong>Attempts (superficial)</strong></p>
  
  <p>I've done quite some experimenting, like</p>
  
  <ul>
  <li>Use RESTResponse.RawBytes and try to decode it. I noticed the bytes in this byte array are already invalid. Comments in the source of TRESTResponse taught me that 'RawBytes' is already decoded, so that makes sense.</li>
  <li>Saved RESTResponse.RawBytes in a file and tried to decompress it with 7zip and a couple of online gzip decompressors. They all failed, of course, since even the gzip header is incorrect.</li>
  <li>Assigned the value 'gzip, deflate' to TRESTClient.AcceptEncoding, TRESTResponse.AcceptEncoding and a combination of those. Also tried to append it to the pre-filled Accept property of each of those components.</li>
  <li>Switched from authenticated to an unauthenticated request. I had the whole oAuth part working, but I though that would make the question too complex. The anonymous API which I used in this question has the same issue, though.</li>
  </ul>
  
  <p>Unfortunately it still doesn't work and I still get a mangled response.</p>
  
  <p><strong>Attemps (digging into the VCL)</strong></p>
  
  <p>Eventually, I dug a little deeper, and dove into TRestRequest.Execute. I won't paste all the code here, but eventually it performs the request by calling</p>
  
  <pre><code>FClient.HTTPClient.Get(LURL, LResponseStream);
  </code></pre>
  
  <p>FClient is the TRESTClient that is linked to the request and LResponseStream is a TMemoryStream. I added <code>LResponseStream.SaveToFile('...')</code> to the watches, so it would save this unprocessed result, et voilá, it gave me a valid gz file, which I could decompress to get my JSON.</p>
  
  <p><strong>A bug in the work-around?</strong></p>
  
  <p>But then, a couple of lines down, I see this piece of code:</p>
  
  <pre><code>  if FClient.HTTPClient.Response.CharSet &gt; '' then
    begin
      LResponseStream.Position := 0;
      S := FClient.HTTPClient.ReadStringAsCharset(LResponseStream, FClient.HTTPClient.Response.CharSet);
      LResponseStream.Free;
      LResponseStream := TStringStream.Create(S);
    end;
  </code></pre>
  
  <p>According to the comment above this block, this is done because the contents of the memory stream are "NOT encoded accordingly to a possibly present Encoding or Content-Type Charset parameter", which is considered a bug in Indy by the writer of this VCL code.</p>
  
  <p>So basically, what happens here: the raw response is treated as a string and converted to the 'right' encoding. FClient.HTTPClient.Response.CharSet is 'UTF-8', which is indeed the encoding of the JSON, but unfortunately, this conversion should only be done after decompressing the stream, which isn't done yet. So this is considered a bug by me. ;)</p>
  
  <p>I tried to dig deeper, but I couldn't find the place where this decompression should have taken place. The actual request is performed by an IIPHTTP instance, which is IPPeerAPI.dcu of which I don't have the source.</p>
  
  <p><strong>So...</strong></p>
  
  <p>So my question is twofold:</p>
  
  <ol>
  <li>Why does this happen? TRestClient should automatically decode the gzip stream when you set AcceptEncoding to 'gzip, deflate'. What setting did I miss? Or isn't this supported yet in XE5?</li>
  <li>How do I prevent this incorrect translation of the gzip stream? I don't mind decoding the response myself, as long as it works, although ideally the REST components should do it automatically.</li>
  </ol>
  
  <p>My setup: VCL Forms application, Windows 8.1, Delphi XE5 professional Update 2.</p>
  
  <p><strong>Update</strong></p>
  
  <ul>
  <li>Work-around was found (see my answer)</li>
  <li>Bug report <a href="https://quality.embarcadero.com/browse/RSP-9855" rel="nofollow noreferrer">RSP-9855</a> filed in quality central </li>
  <li>It's supposedly fixed in Delphi 10.1 (Berlin), but I have yet to test this.</li>
  </ul>