title: Garbage Collection and Parallel.ForEach Issue After VS2015 Upgrade
body: <p>I have some code to process several million data rows in my own R-like C# DataFrame class.  There's a number of Parallel.ForEach calls for iterating over the data rows in parallel.  This code has been running for over a year using VS2013 and .NET 4.5 without issues.</p>
  
  <p>I have two dev machines (A and B) and recently upgraded machine A to VS2015.  I started noticing a strange intermittent freeze in my code about half the time.  Letting it run for a long time, it turns out that the code does eventually finish.  It just takes 15-120 minutes instead of 1-2 minutes.</p>
  
  <p>Attempts to Break All using the VS2015 debugger keep failing for some reason.  So I inserted a bunch of log statements.  It turns out that this freeze occurs when there is a Gen2 collection during a Parallel.ForEach loop (comparing the collection count before and after each Parallel.ForEach loop). The entire extra 13-118 minutes is spent inside whichever Parallel.ForEach loop call happens to overlap with a Gen2 collection (if any).  If there are no Gen2 collections during any Parallel.ForEach loops (about 50% of the time when I run it), then everything finishes fine in 1-2 minutes.</p>
  
  <p>When I run the same code in VS2013 on Machine A, I get the same freezes.  When I run the code in VS2013 on Machine B (which was never upgraded), it works perfectly.  It ran dozens of time overnight with no freezing.</p>
  
  <p>Some things I've noticed / tried:</p>
  
  <ul>
  <li>The freezes happen with or without the debugger attached on Machine A (I figured it was something with the VS2015 debugger at first)</li>
  <li>The freezes happen whether I build in Debug or Release mode</li>
  <li>The freezes happen if I target .NET 4.5 or .NET 4.6</li>
  <li>I tried disabling RyuJIT but that did not affect the freezes</li>
  </ul>
  
  <p>I'm not changing the default GC settings at all.  According to GCSettings, all runs are happening with LatencyMode Interactive and IsServerGC as false.</p>
  
  <p>I could just switch to LowLatency before every call to Parallel.ForEach, but I'd really prefer to understand what's going on.</p>
  
  <p>Has anyone else seen strange freezes in Parallel.ForEach after the VS2015 upgrade?  Any ideas on what a good next step would be?</p>
  
  <p><strong>UPDATE 1: Adding some sample code to the nebulous explanation above...</strong></p>
  
  <p>Here is some sample code that I hope will demonstrate this issue.  This code runs in 10-12 seconds on B machine, consistently.  It encounters a number of Gen2 collections, but they take almost no time at all.  If I uncomment the two GC settings lines, I can force it to have no Gen2 collections.  It's somewhat slower then at 30-50 seconds.</p>
  
  <p>Now on my A machine, the code takes a random amount of time.  Seems to be between 5 and 30 minutes.  And it seems to get worse, the more Gen2 collections it encounters.  If I uncomment the two GC settings lines, it takes 30-50 seconds on Machine A also (same as Machine B).</p>
  
  <p>It might take some tweaking in terms of the number of rows and array size for this to show up on another machine.</p>
  
  <pre><code>using System;
  using System.Collections;
  using System.Collections.Generic;
  using System.IO;
  using System.Diagnostics;
  using System.Threading;
  using System.Threading.Tasks;
  using System.Linq;
  using System.Runtime;    
  
  public class MyDataRow
  {
      public int Id { get; set; }
      public double Value { get; set; }
      public double DerivedValuesSum { get; set; }
      public double[] DerivedValues { get; set; }
  }
  
  class Program
  {
      static void Example()
      {
          const int numRows = 2000000;
          const int tempArraySize = 250;
  
          var r = new Random();
          var dataFrame = new List&lt;MyDataRow&gt;(numRows);
  
          for (int i = 0; i &lt; numRows; i++) dataFrame.Add(new MyDataRow { Id = i, Value = r.NextDouble() });
  
          Stopwatch stw = Stopwatch.StartNew();
  
          int gcs0Initial = GC.CollectionCount(0);
          int gcs1Initial = GC.CollectionCount(1);
          int gcs2Initial = GC.CollectionCount(2);
  
          //GCSettings.LatencyMode = GCLatencyMode.LowLatency;
  
          Parallel.ForEach(dataFrame, dr =&gt;
          {
              double[] tempArray = new double[tempArraySize];
              for (int j = 0; j &lt; tempArraySize; j++) tempArray[j] = Math.Pow(dr.Value, j);
              dr.DerivedValuesSum = tempArray.Sum();
              dr.DerivedValues = tempArray.ToArray();
          });
  
          int gcs0Final = GC.CollectionCount(0);
          int gcs1Final = GC.CollectionCount(1);
          int gcs2Final = GC.CollectionCount(2);
  
          stw.Stop();
  
          //GCSettings.LatencyMode = GCLatencyMode.Interactive;
  
          Console.Out.WriteLine("ElapsedTime = {0} Seconds ({1} Minutes)", stw.Elapsed.TotalSeconds, stw.Elapsed.TotalMinutes);
  
          Console.Out.WriteLine("Gcs0 = {0} = {1} - {2}", gcs0Final - gcs0Initial, gcs0Final, gcs0Initial);
          Console.Out.WriteLine("Gcs1 = {0} = {1} - {2}", gcs1Final - gcs1Initial, gcs1Final, gcs1Initial);
          Console.Out.WriteLine("Gcs2 = {0} = {1} - {2}", gcs2Final - gcs2Initial, gcs2Final, gcs2Initial);
  
          Console.Out.WriteLine("Press Any Key To Exit...");
          Console.In.ReadLine();
      }
  
      static void Main(string[] args)
      {
          Example();
      }
  }
  </code></pre>
  
  <p><strong>UPDATE 2: Just to move things out of the comments for future readers...</strong></p>
  
  <p>This hotfix: <a href="https://support.microsoft.com/en-us/kb/3088957">https://support.microsoft.com/en-us/kb/3088957</a> totally fixes the issue.  I'm not seeing any slowness issues at all after applying.</p>
  
  <p>It turned out not to have anything to do with Parallel.ForEach I believe based on this: <a href="http://blogs.msdn.com/b/maoni/archive/2015/08/12/gen2-free-list-changes-in-clr-4-6-gc.aspx">http://blogs.msdn.com/b/maoni/archive/2015/08/12/gen2-free-list-changes-in-clr-4-6-gc.aspx</a>  though the hotfix does mention Parallel.ForEach for some reason.</p>