title: Processing large text files in .NET 3.5 c#
body: <p>I'm tasked with reading a  large text file (around 150 MB), parsing it and displaying the records in a data grid. The file is delimited by parenthesis. </p>
  
  <p>I'm accomplishing this by -- in a separate thread -- reading the entire file into memory, storing the information in a dataset and then binding the dataset to the data grid which sits on the main form in the original thread. </p>
  
  <p>I have two questions/issues:</p>
  
  <ol>
  <li><p>Is this the best way to do it? Is reading a 150MB file into memory to large? What is the best practice when doing this type of work?</p></li>
  <li><p>The amount of memory that gets allocated for the process is HUGE.. which is understandable because I'm reading such a large file. But, the problem is it doesn't get deallocated. So if I want to do process two files, more and more memory will get allocated. Until at some point the program will just crash. I'm guessing the dataset object is being referenced by something that's preventing the memory from being allocated... is there anyway to determine what that object is? Is there a tool or a method I can use for this purpose? </p></li>
  </ol>
  
  <p>Any help on this will be greatly appreciated. I've never in my coding career ever had to worry about memory management. Thanks. </p>