title: C# OutOfMemory Issue when dealing with large data
body: <p>In our application we are generating reports using <em>Windows Service</em>. The data for reports is fetched from <em>SQL Server</em> using a <em>stored procedure</em>. In some scenario the result set returned contains 250,000 records (We can not help on this part and we need this data in one go as we need to do some calculations on this).</p>
  
  <p><strong>Problem</strong></p>
  
  <p>Our application is getting this data in reader and we are converting this dataset in our custom collection of custom objects. As the data is huge it is not able to store the complete data in the custom object and hence throwing out of memory. When we see the task manager for the process usage while executing the record, it goes very high and even the CPU utilization.</p>
  
  <p>I am not sure what should be do in this case.</p>
  
  <blockquote>
    <ol>
    <li>Can we increase the size of the memory allocated to a single process running under CLR?</li>
    <li>Any other workarounds?</li>
    </ol>
  </blockquote>
  
  <p>Any help would be really appreciated</p>
  
  <blockquote>
    <ol>
    <li>Why do I need all data at once : We need to do calculations on complete resultset</li>
    <li>We are using ADO.NET and transforming the data set in to our custom object (collection)</li>
    <li>Our system is 32 bit</li>
    <li>We can not page the data</li>
    <li>Can not move the computation to sql server</li>
    </ol>
  </blockquote>
  
  <p>This stack trace might help:</p>
  
  <blockquote>
    <p>Exception of type 'System.OutOfMemoryException' was thrown. Server
    stack trace: at
    System.Collections.Generic.Dictionary<code>2.ValueCollection.System.Collections.Generic.IEnumerable&lt;TValue&gt;.GetEnumerator()
    at System.Linq.Enumerable.WhereEnumerableIterator</code>1.MoveNext() at
    System.Collections.Generic.List<code>1.InsertRange(Int32 index,
    IEnumerable</code>1 collection) at
    System.Collections.Generic.List<code>1.AddRange(IEnumerable</code>1 collection)
    at MyProject.Common.Data.DataProperty.GetPropertiesForType(Type t) in
    C:\Ashish-Stuff\Projects\HCPA\Dev
    Branch\Common\Benefits.Common\Data\DataProperty.shared.cs:line 60 at
    MyProject.Common.Data.Extensions.GetProperties[T](T target) in
    C:\Ashish-Stuff\Projects\HCPA\Dev
    Branch\Common\Benefits.Common\Data\Extensions.shared.cs:line 30 at
    MyProject.Common.Data.Factories.SqlServerDataFactoryContract<code>1.GetData(String
    procedureName, IDictionary</code>2 parameters, Nullable<code>1 languageId,
    Nullable</code>1 pageNumber, Nullable`1 pageSize)</p>
  </blockquote>
  
  <p>Thanks,
  Ashish</p>