title: Pyspark: TaskMemoryManager: Failed to allocate a page: Need help in Error Analysis
body: <p>I am facing these errors while running a spark job in standalone cluster mode.</p>
  
  <p>My spark job aims at:</p>
  
  <ul>
  <li>Running some <code>groupby</code>,</li>
  <li><code>count</code>,</li>
  <li>and <code>joins</code> to get a final <code>df</code> and then <code>df.toPandas().to_csv()</code>. </li>
  </ul>
  
  <p>Input dataset is of 524 Mb. Error I get:</p>
  
  <blockquote>
    <p>WARN TaskMemoryManager: Failed to allocate a page (33554432 bytes), try again.</p>
  </blockquote>
  
  <p>After multiple times repeating the above , again new error</p>
  
  <ol>
  <li><p>WARN NettyRpcEnv: Ignored failure: java.util.concurrent.TimeoutException: Cannot receive any reply in 10 seconds</p></li>
  <li><p>org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval</p></li>
  <li><p>at org.apache.spark.rpc.RpcTimeout. org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException</p></li>
  <li><p>ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 158295 ms</p></li>
  <li><p>Exception happened during processing of request from ('127.0.0.1', 49128)
  Traceback (most recent call last):</p>
  
  <p>File "/home/stp/spark-2.0.0-bin-hadoop2.7/python/pyspark/accumulators.py", line 235, in handle
  num_updates = read_int(self.rfile)
  File "/home/stp/spark-2.0.0-bin-hadoop2.7/python/pyspark/serializers.py", line 545, in read_int
  raise EOFError
  EOFError</p></li>
  <li><p>At last ###********##</p>
  
  <p>py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:38073)</p></li>
  </ol>
  
  <p>On first thought , i assumed, the error might be due to memory error(TaskMemoryManager) and from Total 16gb, the process was consuming max 6 gb, leaving 9+gb free. Also I had set the driver memory as 10G. so Pass.</p>
  
  <p>But, when I do a count() or show() on my final dataframe, it was a success op. But while doing toCsv, it is throwing the above errors/Warnings.</p>
  
  <p>Don't actually understand/guess what might be causing the issue.</p>
  
  <p>Please help me analyzing the above errors. Any help/comment is welcome. Thanks.</p>