title: PySpark serialization EOFError
body: <p>I am reading in a CSV as a Spark DataFrame and performing machine learning operations upon it. I keep getting a Python serialization EOFError - any idea why? I thought it might be a memory issue - i.e. file exceeding available RAM - but drastically reducing the size of the DataFrame didn't prevent the EOF error. </p>
  
  <p>Toy code and error below.</p>
  
  <pre><code>#set spark context
  conf = SparkConf().setMaster("local").setAppName("MyApp")
  sc = SparkContext(conf = conf)
  sqlContext = SQLContext(sc)
  
  #read in 500mb csv as DataFrame
  df = sqlContext.read.format('com.databricks.spark.csv').options(header='true',
       inferschema='true').load('myfile.csv')
  
  #get dataframe into machine learning format
  r_formula = RFormula(formula = "outcome ~ .")
  mldf = r_formula.fit(df).transform(df)
  
  #fit random forest model
  rf = RandomForestClassifier(numTrees = 3, maxDepth = 2)
  model = rf.fit(mldf)
  result = model.transform(mldf).head()
  </code></pre>
  
  <p>Running the above code with <code>spark-submit</code> on a single node repeatedly throws the following error, even if the size of the DataFrame is reduced prior to fitting the model (e.g. <code>tinydf = df.sample(False, 0.00001)</code>:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/hduser/spark1.6/python/lib/pyspark.zip/pyspark/daemon.py", line 157, 
       in manager
    File "/home/hduser/spark1.6/python/lib/pyspark.zip/pyspark/daemon.py", line 61, 
       in worker
    File "/home/hduser/spark1.6/python/lib/pyspark.zip/pyspark/worker.py", line 136, 
       in main if read_int(infile) == SpecialLengths.END_OF_STREAM:
    File "/home/hduser/spark1.6/python/lib/pyspark.zip/pyspark/serializers.py", line 545, 
       in read_int
      raise EOFError
    EOFError
  </code></pre>