Unable to serialize PySpark UDF
The library used by pyspark is cpickle which stands for cloud pickle and serializing libraries written in C is not supported by it yet.
If you want to parse xml use databricks XML parser instead and it'll be fast as well.
I am trying to create a UDF in PySpark. The function takes in an input string which is an xml. It then uses lxml to parse it and returns a list of dictionaries with the attributes. I created the function parse_xml, but when I try the line spark.udf.register("parse_xml", parse_xml) but this gives the error: PicklingError: Could not serialize object: TypeError: can't pickle lxml.etree.XMLParser objects. 
It seems as if lxml objects are not serializable, but the input is a string and the output is a list/dictionary -- is there any way to create a UDF like this?