title: HTTPClient Buffer Exceeded 2G; Cannot write more bytes to the buffer
body: <p>The latest in what is becoming the saga of the Walmart API. I'm submitting a call to get a list of clearance items using HttpClient. Things work fine with other requests, but this one is so large it busts the HTTPRequest buffer. <em>Odd, too, that it's a REQUEST buffer error and not a RESPONSE error, since the request is only the URL.</em> </p>
  
  <p><strong>Exception Information:</strong></p>
  
  <pre><code> Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. (System.Net.Http)
  </code></pre>
  
  <blockquote>
    <p>at System.Net.Http.HttpContent.LimitMemoryStream.CheckSize(Int32 countToAdd)\r\n   at System.Net.Http.HttpContent.LimitMemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.Net.Http.StreamToStreamCopy.TryStartWriteSync(Int32 bytesRead)\r\n   at System.Net.Http.StreamToStreamCopy.StartRead()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter<code>1.GetResult()\r\n   at Wal_Mart_Crawler.NET_IO.&lt;walMart_Special_Feed_Lookup&gt;d__4.MoveNext() in c:\\users\\user\\documents\\visual studio 2015\\Projects\\Wal-Mart_Inventory_Tracker\\Wal-Mart_Crawler\\NET_IO.cs:line 65\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter</code>1.GetResult()\r\n   at Wal_Mart_Crawler.Special_Feeds.d__2.MoveNext() in c:\users\user\documents\visual studio 2015\Projects\Wal-Mart_Inventory_Tracker\Wal-Mart_Crawler\Special_Feeds.cs:line 22</p>
  </blockquote>
  
  <p>(for the curious, Line 22 is):</p>
  
  <pre><code> API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Wal_Mart_Crawler.Properties.Resources.API_Key_Walmart));
   //which uses HttpClient to call the API
  </code></pre>
  
  <p>At first, my eyes twinkled a bit when I saw "configured," as that means I can change it, right? I'm running in x64 after all - might be able to bust 2G <em>can't figure out how.</em></p>
  
  <p>Read up on SO, found that I should disable streaming. Tried:</p>
  
  <pre><code> var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
  </code></pre>
  
  <p>... <em>no Bueno.</em></p>
  
  <p>Found an <em>AllowReadStreamBuffering</em> setting for WebRequest - can't seem to find one for HttpClient.</p>
  
  <p>I can't control how much data I'm getting -- that's on Walmart. I'm also limited by my shrunken head on what to do with the response stream because it's going straight to deserialize:</p>
  
  <pre><code>   var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
     return JsonConvert.DeserializeObject&lt;API_Json_Special_Feeds.RootObject&gt;(result);
  </code></pre>
  
  <p>So even if I <em>could</em> break up the response, I'd be stuck because I don't think I can feed partial data to deserialize.</p>
  
  <p><strong>Question:</strong>  How can I either increase the buffer beyond 2G or otherwise avoid the buffer size exceeded exception?</p>
  
  <p>I'm hoping it's another easy fix my weary, ignorant brain just can't figure out. As always - a sincere THANK YOU for your time and in advance for any help you can provide.</p>
  
  <p><strong>Response Headers:</strong></p>
  
  <pre><code> StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
  {
   X-Mashery-Responder: prod-j-worker-us-west-1c-63.mashery.com
   transfer-encoding: chunked
   Connection: keep-alive
   Date: Sat, 10 Sep 2016 00:57:38 GMT
   Server: Mashery
   Server: Proxy
   Content-MD5: BvJMDJiZPUvmAxxmwKGSog==
   Content-Type: application/json; charset=utf-8
   Last-Modified: Fri Sep 09 15:31:08 PDT 2016
  }
  </code></pre>