title: Apache Spark: nondeterministic StackOverflowError when training with GBTRegressor
body: <p>Context:</p>
  
  <p>Training with the GBTRegressor on Spark 1.6.3 + Scala on a feature set of about 2000 and with ~1 million rows.</p>
  
  <p>Using 10 EC2 instances with 32 cores each, and we end up with 30 executors each with 10 cores.</p>
  
  <p>HyperParameters/Parameters- Depth:3, Iterations:100, StepSize: 0.1, CacheNodeIds: true, CheckpointInterval:10</p>
  
  <p>Partition size is 5000</p>
  
  <p>Problem: </p>
  
  <p>About 50% of the time we get a stackoverflow error always around stage 1000, other times the trainer finishes fine. Our GBTRegressor is the first <code>PipelineStage</code> in the <code>Pipeline</code>.</p>
  
  <p>Sample exceptions (though the error has been occuring at varying depths inside <code>TreeNode</code>):</p>
  
  <pre><code>[Stage 1014:=======================&gt;                       (2514 + 2486) / 5000]
  [Stage 1014:=============================&gt;                 (3088 + 1912) / 5000]
  [Stage 1014:==================================&gt;            (3666 + 1334) / 5000]
  [Stage 1014:========================================&gt;       (4242 + 758) / 5000]
  [Stage 1014:==============================================&gt; (4819 + 181) / 5000]
  
  Exception in thread "main" java.lang.StackOverflowError
      at org.apache.spark.sql.catalyst.trees.TreeNode.containsChild$lzycompute(TreeNode.scala:86)
      at org.apache.spark.sql.catalyst.trees.TreeNode.containsChild(TreeNode.scala:86)
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:280)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
      at scala.collection.Iterator$class.foreach(Iterator.scala:742)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
      at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
      at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
      at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
      at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:308)
      at scala.collection.AbstractIterator.to(Iterator.scala:1194)
      at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:300)
      at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1194)
      at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:287)
      at scala.collection.AbstractIterator.toArray(Iterator.scala:1194)
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:321)
  </code></pre>
  
  <p>Another:</p>
  
  <pre><code>[Stage 1044:=========================&gt;                     (2692 + 2308) / 5000]
  [Stage 1044:==============================&gt;                (3275 + 1725) / 5000]
  [Stage 1044:====================================&gt;          (3851 + 1149) / 5000]
  [Stage 1044:==========================================&gt;     (4430 + 570) / 5000]
  [Stage 1044:=================================================&gt;(4996 + 4) / 5000]
   Exception in thread "main" java.lang.StackOverflowError
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
      at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
      at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:86)
      at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:85)
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:259)
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:259)
  </code></pre>
  
  <p>We submit with (also tried upping the thread stack size with Xss but to no avail)</p>
  
  <pre><code>spark-submit --conf "spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
  </code></pre>