title: An error occurred while calling o2864.collect
body: <p>I have a RDD object(bid) that get from map, and then I just do a simple bid.count(). Then I received the following error. Can someone tell me what this error might be? I try to google what it means but got no answer.</p>
  
  <pre><code>---------------------------------------------------------------------------
  Py4JJavaError                             Traceback (most recent call last)
  &lt;ipython-input-10-0283df3b1938&gt; in &lt;module&gt;()
  ----&gt; 1 bid.count()
  
  /home/username/spark-1.0.0-bin-hadoop2/python/pyspark/rdd.pyc in count(self)
      706         3
      707         """
  --&gt; 708         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
      709 
      710     def stats(self):
  
  /home/username/spark-1.0.0-bin-hadoop2/python/pyspark/rdd.pyc in sum(self)
      697         6.0
      698         """
  --&gt; 699         return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)
      700 
      701     def count(self):
  
  /home/username/spark-1.0.0-bin-hadoop2/python/pyspark/rdd.pyc in reduce(self, f)
      617             if acc is not None:
      618                 yield acc
  --&gt; 619         vals = self.mapPartitions(func).collect()
      620         return reduce(f, vals)
      621 
  
  /home/username/spark-1.0.0-bin-hadoop2/python/pyspark/rdd.pyc in collect(self)
      581         """
      582         with _JavaStackTrace(self.context) as st:
  --&gt; 583           bytesInJava = self._jrdd.collect().iterator()
      584         return list(self._collect_iterator_through_file(bytesInJava))
      585 
  
  /home/username/spark-1.0.0-bin-hadoop2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
      535         answer = self.gateway_client.send_command(command)
      536         return_value = get_return_value(answer, self.gateway_client,
  --&gt; 537                 self.target_id, self.name)
      538 
      539         for temp_arg in temp_args:
  
  /home/username/spark-1.0.0-bin-hadoop2/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
      298                 raise Py4JJavaError(
      299                     'An error occurred while calling {0}{1}{2}.\n'.
  --&gt; 300                     format(target_id, '.', name), value)
      301             else:
      302                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling o148.collect.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1.0:9 failed 4 times, most recent failure: TID 7 on host 08.bm-hadoope-datanode.dev.lax1 failed for unknown reason
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
      at akka.actor.ActorCell.invoke(ActorCell.scala:456)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
      at akka.dispatch.Mailbox.run(Mailbox.scala:219)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  </code></pre>
  
  <p>And then I join the bid table with another table, then get a new RDD. It seems going smooth, but when I do first() for this new RDD. I received the following error:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o274.collectPartitions.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 9.0:6 failed 4 times, most recent failure: Exception failure in TID 32 on host 05.bm-hadoope-datanode.dev.lax1: java.io.IOException: Filesystem closed
          org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:629)
          org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:735)
          org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
          java.io.DataInputStream.readFully(DataInputStream.java:195)
          java.io.DataInputStream.readFully(DataInputStream.java:169)
          parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:599)
          parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
          parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:100)
          parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:172)
          parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)
          org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:122)
          org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
          scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
          scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
          scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
          org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:293)
          org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply$mcV$sp(PythonRDD.scala:200)
          org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply(PythonRDD.scala:175)
          org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$1.apply(PythonRDD.scala:175)
          org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1160)
          org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:174)
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
      at akka.actor.ActorCell.invoke(ActorCell.scala:456)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
      at akka.dispatch.Mailbox.run(Mailbox.scala:219)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  </code></pre>