title: How to serialize a pyspark Pipeline object?
body: <p>I'm trying to serialize a PySpark <code>Pipeline</code> object so that it can be saved and retrieved later. Tried using the Python pickle library as well as the PySpark's <code>PickleSerializer</code>, the <code>dumps()</code> call itself is failing.</p>
  
  <p>Providing the code snippet while using native <code>pickle</code> library.</p>
  
  <pre><code>pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
  with open ('myfile', 'wb') as f:
     pickle.dump(pipeline,f,2)
  with open ('myfile', 'rb') as f:
     pipeline1 = pickle.load(f)
  </code></pre>
  
  <p>Getting the below error while running:</p>
  
  <pre><code>py4j.protocol.Py4JError: An error occurred while calling o32.__getnewargs__. Trace:
  py4j.Py4JException: Method __getnewargs__([]) does not exist
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:335)
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:344)
      at py4j.Gateway.invoke(Gateway.java:252)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:209)
      at java.lang.Thread.run(Thread.java:785)
  </code></pre>
  
  <p>Is it possible to serialize PySpark <code>Pipeline</code> objects ?</p>