title: Reading very large text files, should I be incorporating async?
body: <p>I have been challenged with producing a method that will read in very large text files into a program these files can range from 2gb to 100gb.</p>
  
  <p>The idea so far has been to read say a couple of 1000 lines of text into the method.</p>
  
  <p>At the moment the program is setup using a stream reader reading a file line by line and processing the necessary areas of data found on that line.</p>
  
  <pre><code>using (StreamReader reader = new StreamReader("FileName"))
  {
      string nextline = reader.ReadLine();
      string textline = null;
  
      while (nextline != null)
      {
          textline = nextline;
          Row rw = new Row();
          var property = from matchID in xmldata
                         from matching in matchID.MyProperty
                         where matchID.ID == textline.Substring(0, 3).TrimEnd()
                         select matching;
  
          string IDD = textline.Substring(0, 3).TrimEnd();
  
          foreach (var field in property)
          {
              Field fl = new Field();
  
              fl.Name = field.name;
              fl.Data = textline.Substring(field.startByte - 1, field.length).TrimEnd();
              fl.Order = order;
              fl.Show = true;
  
              order++;
  
              rw.ID = IDD;
              rw.AddField(fl);
          }
          rec.Rows.Add(rw);
          nextline = reader.ReadLine();
  
          if ((nextline == null) || (NewPack == nextline.Substring(0, 3).TrimEnd()))
          {
              d.ID = IDs.ToString();
              d.Records.Add(rec);
              IDs++;
              DataList.Add(d.ID, d);
              rec = new Record();
  
              d = new Data();
          }
      }
  }
  </code></pre>
  
  <p>The program goes on further and populates a class. ( just decided not to post the rest) </p>
  
  <p>I know that once the program is shown an extremely large file, memory exception errors will occur.</p>
  
  <p>so that is my current problem and so far i have been googling several approaches with many people just answering use a stream reader and reader.readtoend, i know readtoend wont work for me as i will get those memory errors.</p>
  
  <p>Finally i have been looking into async as a way of creating a method that will read a certain amount of lines and wait for a call before processing the next amount of lines.</p>
  
  <p>This brings me to my problem i am struggling to understand async and i can't seem to find any material that will help me learn and was hoping someone here can help me out with a way to understand async.</p>
  
  <p>Of course if anyone knows of a better way to solve this problem I am all ears. </p>
  
  <p><strong>EDIT</strong> Added the remainder of the code to put a end to any confusion. </p>