title: Spark out of memory no matter what i do
body: <p>I'm trying to use spark to filter a large dataframe. As a pandas dataframe it would be somewhere around 70GB in memory.</p>
  
  <p>I am able to load and filter this data using pandas, however really slowly because I have to swap to disk etc.</p>
  
  <p>However when i try doing this using PySpark I run into memory errors from Java.</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql import SparkSession
  spark = SparkSession.builder.appName("Master").getOrCreate()
  master = spark.read.csv(master_path, inferSchema=True, schema=schema, header=True)
  
  master_desember = spark.sql("SELECT * FROM master_filter_ready WHERE born_month='12'")
  master_desember = master_desember.toPandas()
  </code></pre>
  
  <p><strong>Edit:</strong> So my question is how do I set memory on a spark session before I create it? Also How can I determine what values to use? At the moment I'm just guessing.</p>