title: Apache spark job failed immediately without retry, setting maxFailures doesn't work
body: <p>I was testing a web crawling/scrapping program on Apache Spark locally on my computer.</p>
  
  <p>the program use a few RDD transformations that takes a volatile function that sporadically fails. (The function's purpose is to transform URL links into web pages, sometimes the headless browser it invoked just blackout or got overloaded - I can't avoid that)</p>
  
  <p>I heard that Apache Spark has powerful failover and retrying feature, any unsuccessful transformation or lost data can be recalculated from scratch from whatever resource it can find (sounds like magic right?) so I didn't put any failover or try-catch in my code.</p>
  
  <p>This is my spark configuration:</p>
  
  <pre><code>val conf = new SparkConf().setAppName("MoreLinkedIn")
  conf.setMaster("local[*]")
  conf.setSparkHome(System.getenv("SPARK_HOME"))
  conf.setJars(SparkContext.jarOfClass(this.getClass).toList)
  conf.set("spark.task.maxFailures","40") //definitely enough
  </code></pre>
  
  <p>Unfortunately the job failed after the majority of stages and individual tasks succeeded. The latest log in console shows:</p>
  
  <pre><code>Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1.0:7 failed 1 times, most recent failure: Exception failure in TID 23 on host localhost: org.openqa.selenium.TimeoutException: Timed out after 50 seconds waiting for...
  </code></pre>
  
  <p>Looks like Spark just give up cowardly after failed once. How do I configure it properly to make it more tenacious?</p>
  
  <p>(my program can be downloaded from <a href="https://github.com/tribbloid/spookystuff" rel="nofollow noreferrer">https://github.com/tribbloid/spookystuff</a>, sorry for the scarce and disorganized code/documentation, I just start it for a few days)</p>
  
  <p>ADD: if you want to try it yourself, The following code can demonstrate this problem:</p>
  
  <pre><code>def main(args: Array[String]) {
  val conf = new SparkConf().setAppName("Spark Pi")
  conf.setMaster("local[*]")
  conf.setSparkHome(System.getenv("SPARK_HOME"))
  conf.setJars(SparkContext.jarOfClass(this.getClass).toList)
  conf.set("spark.task.maxFailures","400000")
  val sc = new SparkContext(conf)
  val slices = if (args.length &gt; 0) args(0).toInt else 8
  val n = 100000 * slices
  val count = sc.parallelize(1 to n, slices).map { i =&gt;
    val x = java.lang.Math.random()
    if (x &gt; 0.9) throw new IllegalStateException("the map has a chance of 10% to fail")
    x
  }.reduce(_ + _)
  sc.stop()
  println("finished")
  }
  </code></pre>
  
  <p>It should be noted that the same IllegalStateException got retried for 32 times in this post:
  <a href="https://stackoverflow.com/questions/23710032/apache-spark-throws-java-lang-illegalstateexception-unread-block-data">Apache Spark Throws java.lang.IllegalStateException: unread block data</a></p>