title: Py4JJavaError: An error occurred while calling
body: <p>I am new to PySpark. I have been writing my code with a test sample. Once I run the code on the larger file(3gb compressed). My code is only doing some filtering and joins. I keep getting errors regarding py4J. </p>
  
  <p>Any help would be useful, and appreciated.  </p>
  
  <pre><code>from pyspark.sql import SparkSession
  from pyspark.conf import SparkConf
  
  ss = SparkSession \
        .builder \
        .appName("Example") \
        .getOrCreate()
  
  ss.conf.set("spark.sql.execution.arrow.enabled", 'true')
  
  df = ss.read.csv(directory + '/' + filename, header=True, sep=",")
  # Some filtering and groupbys...
  df.show()
  </code></pre>
  
  <p>Return</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o88.showString.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in 
  stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 
  1, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
  ...
  Caused by: java.lang.OutOfMemoryError: Java heap space
  </code></pre>
  
  <p>UPDATE:
  I was using py4j 10.7 and just updated to 10.8</p>
  
  <p>UPDATE(1): Adding spark.driver.memory:</p>
  
  <pre><code> ss = SparkSession \
    .builder \
    .appName("Example") \
    .config("spark.driver.memory", "16g")\
    .getOrCreate()
  </code></pre>
  
  <p>Summarized Return Error:</p>
  
  <pre><code>ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:38004)
  
  py4j.protocol.Py4JNetworkError: Answer from Java side is empty
  "Error while receiving", e, proto.ERROR_ON_RECEIVE)
  py4j.protocol.Py4JNetworkError: Error while receiving
  
  Py4JError
  Py4JError: An error occurred while calling o94.showString
  </code></pre>
  
  <p>UPDATE(2) : I tried this, by changing the spark-defaults.conf file. Still getting error <a href="https://stackoverflow.com/questions/32336915/pyspark-java-lang-outofmemoryerror-java-heap-space">PySpark: java.lang.OutofMemoryError: Java heap space</a></p>
  
  <p>SEMI-SOLVED : This seemed to be a general memory problem. I started a 2xlarge instance with 32g of memory. The program runs with no errors. </p>
  
  <p>Knowing this, is there something else, a conf option that could help so I don't have to run an expensive instance? </p>
  
  <p>Thanks Everyone.</p>