title: pyspark withColumnRenamed, drop functions, u'Reference ambiguous error
body: <p>I have a function which changes the column headers of a DF with a new set of headers in a list.</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>def updateHeaders(dataFrame, newHeader):
      oldColumns = dataFrame.schema.names
      dfNewCol =  reduce(lambda dataFrame, idx: dataFrame.withColumnRenamed(oldColumns[idx], newHeader[idx]), xrange(len(oldColumns)), dataFrame)
      return dfNewCol
  </code></pre>
  
  <p>I capture the <code>newHeader</code> list from another function. The first header in the list is named as <code>Action</code>. Later I apply a filter function in which I drop the <code>Action</code> column and create a new DF</p>
  
  <pre class="lang-python prettyprint-override"><code>def willBeInserted(dataFrame):
      insertData = ["I"] # Some rows of 'Action' column include value "I"
      insertDF = dataFrame.filter(dataFrame.Action.isin(insertData)).drop('Action')
      return insertDF
  </code></pre>
  
  <p>Later I call the functions</p>
  
  <pre class="lang-python prettyprint-override"><code>DF1 = updateHeaders(someDF, headerList) #Update the headers
  DF2 = willBeInserted(DF1) #Drop 'Action' column and create new DF
  </code></pre>
  
  <p>The result is the following error:</p>
  
  <pre class="lang-python prettyprint-override"><code>pyspark.sql.utils.AnalysisException: u'Reference 'Action' is ambiguous, could be: Action#29, Action#221.;"
  </code></pre>
  
  <p>I tried the solution approaches in <a href="https://stackoverflow.com/questions/33778664/spark-dataframe-distinguish-columns-with-duplicated-name">this link</a> and in other similar questions, no change so far. Any ideas? </p>