title: Pyspark remove duplicate columns in a dataframe
body: <p>I found many solutions are related with <code>join</code> situation. My question is if the duplicates exist in the dataframe itself, how to detect and remove them? The following example is just showing how I create a data frame with duplicate columns.</p>
  
  <pre><code>df = spark.createDataFrame([
  (1, "A", "X1"),
  (2, "B", "X2"),
  (3, "B", "X3"),
  ], ["ID", "TYPE", "CODE"])
  
  df1 = df.withColumn("TYPE1", df["TYPE"]).withColumn("TYPE2", df["TYPE"])
  
  +---+----+----+
  | ID|TYPE|CODE|
  +---+----+----+
  |  1|   A|  X1|
  |  2|   B|  X2|
  |  3|   B|  X3|
  +---+----+----+
  
  +---+----+----+-----+-----+
  | ID|TYPE|CODE|TYPE1|TYPE2|
  +---+----+----+-----+-----+
  |  1|   A|  X1|    A|    A|
  |  2|   B|  X2|    B|    B|
  |  3|   B|  X3|    B|    B|
  +---+----+----+-----+-----+
  </code></pre>
  
  <p>Suppose I am just given <code>df1</code>, how can I remove duplicate columns to get <code>df</code>? Thanks!</p>