can't define a udf inside pyspark project
Question: I have a python project that uses pyspark and i am trying to define a udf
function inside the spark project (not in my python project) specifically in
spark\python\pyspark\ml\tuning.py but i get pickling problems. it can't load
the udf. The code:

    
    
    from pyspark.sql.functions import udf, log
    test_udf = udf(lambda x : -x[1], returnType=FloatType())
    d = data.withColumn("new_col", test_udf(data["x"]))
    d.show()
    

when i try d.show() i am getting exception of unknown attribute test_udf

In my python project i defined many udf and it worked fine.


Answer: add the following to your code. It isn't recognizing the datatype.

    
    
    from pyspark.sql.types import *
    

Let me know if this helps. Thanks.

