title: Joining Dataframes with same coumn name in pyspark
body: <p>I have two dataframe which has been readed from two csv files.</p>
  
  <pre><code>+---+----------+-----------------+
  | ID|  NUMBER  |  RECHARGE_AMOUNT|
  +---+----------+-----------------+
  |  1|9090909092|               30|
  |  2|9090909093|               30|
  |  3|9090909090|               30|
  |  4|9090909094|               30|
  +---+----------+-----------------+
  </code></pre>
  
  <p>and </p>
  
  <pre><code>+---+----------+-----------------+
  | ID|  NUMBER  |  RECHARGE_AMOUNT|
  +---+----------+-----------------+
  |  1|9090909092|               40|
  |  2|9090909093|               50|
  |  3|9090909090|               60|
  |  4|9090909094|               70|
  +---+----------+-----------------+
  </code></pre>
  
  <p>I am triying to join this two data from using NUMBER coumn using the pyspark code <code>dfFinal = dfFinal.join(df2, on=['NUMBER'], how='inner')</code> and new dataframe is generated as follows. </p>
  
  <pre><code>+----------+---+-----------------+---+-----------------+
  |  NUMBER  | ID|  RECHARGE_AMOUNT| ID|  RECHARGE_AMOUNT|
  +----------+---+-----------------+---+-----------------+
  |9090909092|  1|               30|  1|               40|
  |9090909093|  2|               30|  2|               50|
  |9090909090|  3|               30|  3|               60|
  |9090909094|  4|               30|  4|               70|
  +----------+---+-----------------+---+-----------------+
  </code></pre>
  
  <p>But i am not able to write this dataframe into a file since the dataframe after joining is having duplicate column. I am using the following code. <code>dfFinal.coalesce(1).write.format('com.databricks.spark.csv').save('/home/user/output',header = 'true')</code> Is there any way to avoid duplicate column after joining in spark. Given below is my pyspark code.</p>
  
  <pre><code>from pyspark.sql import SparkSession
  from pyspark.sql.functions import col
  spark = SparkSession.builder.appName("test1").getOrCreate()
  files = ["/home/user/test1.txt", "/home/user/test2.txt"]
  dfFinal = spark.read.load(files[0],format="csv", sep=",", inferSchema="false", header="true", mode="DROPMALFORMED")
  dfFinal.show()
  for i in range(1,len(files)):
      df2 = spark.read.load(files[i],format="csv", sep=",", inferSchema="false", header="true", mode="DROPMALFORMED")
      df2.show()
      dfFinal = dfFinal.join(df2, on=['NUMBER'], how='inner')
  dfFinal.show()
  dfFinal.coalesce(1).write.format('com.databricks.spark.csv').save('/home/user/output',header = 'true')
  </code></pre>
  
  <p>I need to generate unique column name.ie: if i gave two files in files array with same coumn it should generate as follows.</p>
  
  <pre><code>+----------+----+-------------------+-----+-------------------+
  |  NUMBER  |IDx |  RECHARGE_AMOUNTx | IDy |  RECHARGE_AMOUNTy |
  +----------+----+-------------------+-----+-------------------+
  |9090909092|  1 |               30  |  1  |               40  |
  |9090909093|  2 |               30  |  2  |               50  |
  |9090909090|  3 |               30  |  3  |               60  |
  |9090909094|  4 |               30  |  4  |               70  |
  +----------+---+-----------------+---+------------------------+
  </code></pre>
  
  <p>In panda i can use <code>suffixes</code> argument as show below <code>dfFinal = dfFinal.merge(df2,left_on='NUMBER',right_on='NUMBER',how='inner',suffixes=('x', 'y'),sort=True)</code> which will generate the above dataframe. Is there any way i can replicate this on pyspark.</p>