title: How to resolve duplicate column names while joining two dataframes in PySpark?
body: <p>I have a file A and B which are exactly the same. I am trying to perform inner and outer joins on these two dataframes. Since I have all the columns as duplicate columns, the existing answers were of no help.
  The other questions that I have gone through contain a col or two as duplicate, my issue is that the whole files are duplicates of each other: both in data and in column names.</p>
  
  <p>My code:</p>
  
  <pre><code>import sys
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkContext
  from awsglue.context import GlueContext
  from awsglue.job import Job
  from pyspark.sql import DataFrameReader, DataFrameWriter
  from datetime import datetime
  
  import time
  
  # @params: [JOB_NAME]
  args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  
  sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args['JOB_NAME'], args)
  
  print("All imports were successful.")
  
  df = spark.read.orc(
      's3://****'
  )
  print("First dataframe read with headers set to True")
  df2 = spark.read.orc(
      's3://****'
  )
  print("Second dataframe read with headers set to True")
  
  # df3 = df.join(df2, ['c_0'], "outer")
  
  # df3 = df.join(
  #     df2,
  #     df["column_test_1"] == df2["column_1"],
  #     "outer"
  # )
  
  df3 = df.alias('l').join(df2.alias('r'), on='c_0') #.collect()
  
  print("Dataframes have been joined successfully.")
  output_file_path = 's3://****'
  )
  
  df3.write.orc(
      output_file_path
  )
  print("Dataframe has been written to csv.")
  job.commit()
  </code></pre>
  
  <p>The error that I am facing is:</p>
  
  <pre><code>pyspark.sql.utils.AnalysisException: u'Duplicate column(s): "c_4", "c_38", "c_13", "c_27", "c_50", "c_16", "c_23", "c_24", "c_1", "c_35", "c_30", "c_56", "c_34", "c_7", "c_46", "c_49", "c_57", "c_45", "c_31", "c_53", "c_19", "c_25", "c_10", "c_8", "c_14", "c_42", "c_20", "c_47", "c_36", "c_29", "c_15", "c_43", "c_32", "c_5", "c_37", "c_18", "c_54", "c_3", "__created_at__", "c_51", "c_48", "c_9", "c_21", "c_26", "c_44", "c_55", "c_2", "c_17", "c_40", "c_28", "c_33", "c_41", "c_22", "c_11", "c_12", "c_52", "c_6", "c_39" found, cannot save to file.;'
  End of LogType:stdout
  </code></pre>