title: Is it better to use HttpClient or WebRequest in ASP.NET Core to read the content of a file line by line asynchronously remotely?
body: <p>I plan to read a remote file line by line asynchronously using <a href="https://github.com/Dasync/AsyncEnumerable" rel="nofollow noreferrer">https://github.com/Dasync/AsyncEnumerable</a> (since there is not yet Async Streams [C# 8 maybe]: <a href="https://github.com/dotnet/csharplang/blob/master/proposals/async-streams.md" rel="nofollow noreferrer">https://github.com/dotnet/csharplang/blob/master/proposals/async-streams.md</a>):</p>
  
  <pre><code>public static class StringExtensions
  {
      public static AsyncEnumerable&lt;string&gt; ReadLinesAsyncViaHttpClient(this string uri)
      {
          return new AsyncEnumerable&lt;string&gt;(async yield =&gt;
          {
              using (var httpClient = new HttpClient())
              {
                  using (var responseStream = await httpClient.GetStreamAsync(uri))
                  {
                      using (var streamReader = new StreamReader(responseStream))
                      {
                          while(true)
                          {
                              var line = await streamReader.ReadLineAsync();
  
                              if (line != null)
                              {
                                  await yield.ReturnAsync(line);
                              }
                              else
                              {
                                  return;
                              }
                          } 
                      }
                  }
              }
          });
      }
      public static AsyncEnumerable&lt;string&gt; ReadLinesAsyncViaWebRequest(this string uri)
      {
          return new AsyncEnumerable&lt;string&gt;(async yield =&gt;
          {
              var request = WebRequest.Create(uri);
              using (var response = request.GetResponse())
              {
                  using (var responseStream = response.GetResponseStream())
                  {
                      using (var streamReader = new StreamReader(responseStream))
                      {
                          while(true)
                          {
                              var line = await streamReader.ReadLineAsync();
  
                              if (line != null)
                              {
                                  await yield.ReturnAsync(line);
                              }
                              else
                              {
                                  return;
                              }
                          } 
                      }
                  }
              }
          });
      }
  }
  </code></pre>
  
  <p>It seems that they both run just fine in a simple Console application like below:</p>
  
  <pre><code>public class Program
  {
      public static async Task Main(string[] args)
      {
          // Or any other remote file
          const string url = @"https://gist.githubusercontent.com/dgrtwo/a30d99baa9b7bfc9f2440b355ddd1f75/raw/700ab5bb0b5f8f5a14377f5103dbe921d4238216/by_tag_year.csv";
  
          await url.ReadLinesAsyncViaWebRequest().ForEachAsync(line =&gt;
          {
              Console.WriteLine(line, Color.GreenYellow);
          });
          await url.ReadLinesAsyncViaHttpClient().ForEachAsync(line =&gt;
          {
              Console.WriteLine(line, Color.Purple);
          });
      }
  }
  </code></pre>
  
  <p>... but I have some concerns if it is used as part of an ASP.NET Core WebAPI to process the lines and then push them using PushStreamContent: </p>
  
  <ul>
  <li><a href="https://docs.microsoft.com/en-us/previous-versions/aspnet/hh995285(v=vs.108)" rel="nofollow noreferrer">https://docs.microsoft.com/en-us/previous-versions/aspnet/hh995285(v=vs.108)</a></li>
  <li><a href="https://blog.stephencleary.com/2016/10/async-pushstreamcontent.html" rel="nofollow noreferrer">https://blog.stephencleary.com/2016/10/async-pushstreamcontent.html</a></li>
  </ul>
  
  <p>The idea would be to have a pipeline of data which leverages <code>async</code> / <code>await</code> so that the number of threads in use is as low as possible and also to avoid an increase in memory (which leverage the enumerable-like feature of AsyncEnumerable).</p>
  
  <p>I read several articles but it seems it's all non .NET Core versions and I don't really know if there would be some potential performance issues / caveats in regard to what I would like to achieve?</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/8209781/difference-between-httprequest-httpwebrequest-and-webrequest">Difference between HttpRequest, HttpWebRequest and WebRequest</a></li>
  <li><a href="http://www.diogonunes.com/blog/webclient-vs-httpclient-vs-httpwebrequest/" rel="nofollow noreferrer">http://www.diogonunes.com/blog/webclient-vs-httpclient-vs-httpwebrequest/</a></li>
  </ul>
  
  <p>An example of "business" case would be:</p>
  
  <pre><code>using System;
  using System.Collections.Async;
  using System.IO;
  using System.Threading.Tasks;
  using Microsoft.AspNetCore.Mvc;
  
  namespace WebApplicationTest.Controllers
  {
      [Route("api/[controller]")]
      [ApiController]
      public class DumbValuesController : ControllerBase
      {
          private static readonly Random Random = new Random();
  
          // GET api/values
          [HttpGet]
          public async Task&lt;IActionResult&gt; DumbGetAsync([FromQuery] string fileUri)
          {
              using (var streamWriter = new StreamWriter(HttpContext.Response.Body))
              {
                  await fileUri.ReadLinesAsyncViaHttpClient().ForEachAsync(async line =&gt;
                  {
                      // Some dumb process on each (maybe big line)
                      line += Random.Next(0, 100 + 1);
                      await streamWriter.WriteLineAsync(line);
                  });
              }
  
              return Ok();
          }
      }
  }
  </code></pre>