title: How to resolve pickle error in pyspark?
body: <p>I am iterating through files to gather information about the values in their columns and rows in a dictionary.  I have the following code which works locally:</p>
  
  <pre><code>def search_nulls(file_name):
      separator = ','
      nulls_dict = {}
      fp = open(file_name,'r')
      null_cols = {}
      lines = fp.readlines()
  
      for n,line in enumerate(lines):
          line = line.split(separator)
          for m,data in enumerate(line):
              data = data.strip('\n').strip('\r')
              if str(m) not in null_cols:
                  null_cols[str(m)] = defaultdict(lambda: 0)
              if len(data) &lt;= 4:
                  null_cols[str(m)][str(data)] = null_cols[str(m)][str(data)] + 1
  
      return null_cols
  
  
  files_to_process = ['tempfile.csv']
  results = map(lambda file: search_nulls(file), files_to_process)
  </code></pre>
  
  <p>The above code works fine without spark.
  I comment the last two lines above, and I try with spark, since this is a prototype of something that will need to run distributed:</p>
  
  <pre><code>os.environ['SPARK_HOME'] = &lt;path_to_spark_folder&gt;
  conf = SparkConf().setAppName("search_files").setMaster('local')
  
  sc = SparkContext(conf=conf)
  
  objects = sc.parallelize(files_to_process)
  resulting_object = \
      objects.map(lambda file_object: find_nulls(file_object))
  
  result = resulting_object.collect()
  </code></pre>
  
  <p>When using spark, though, this results in the following error:</p>
  
  <pre><code>File "&lt;path-to-spark&gt;/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
      process()
    File "&lt;path-to-spark&gt;/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
      serializer.dump_stream(func(split_index, iterator), outfile)
    File "&lt;path-to-spark&gt;/python/lib/pyspark.zip/pyspark/serializers.py", line 267, in dump_stream
      bytes = self.serializer.dumps(vs)
    File "&lt;path-to-spark&gt;/python/lib/pyspark.zip/pyspark/serializers.py", line 415, in dumps
      return pickle.dumps(obj, protocol)
  TypeError: expected string or Unicode object, NoneType found​
  </code></pre>
  
  <p>I've been unable to find any obvious reason why this would fail, since it runs perfectly locally, and I am not sharing any files across worker nodes.  In fact, I'm only running this on my local machine anyway.  </p>
  
  <p>Does anyone know of a good reason why this might be failing?  </p>