Found duplicate column when trying to query with Spark SQL
I've got the solution now. What I needed to use was:
Add from pyspark.sql.functions import * at the file header
Simply use col()'s alias function like so:
filtered_df2 = filtered_df.select(col("li"),col("result.li").alias("result_li"), col("fw")).orderBy("fw")
I want to do a filter on a dataframe like so:
filtered_df2 = filtered_df.select("li", "result.li", "fw").orderBy("fw")
However, the nested column, result.li has the same name as li and this poses a problem. I get the following error:
AnalysisException: 'Found duplicate column(s) when inserting into hdfs://...: `li`;'
How can I filter both fields successfully ?