title: How do you remove an ambiguous column in pyspark?
body: <p>There are many questions similar to this that are asking a different question with regard to avoid duplicate columns <strong>in a join</strong>; that is not what I am asking here.</p>
  <p>Given that I <em>already</em> have a DataFrame with ambiguous columns, how do I remove a <em>specific</em> column?</p>
  <p>For example, given:</p>
  <pre><code>df = spark.createDataFrame(
      spark.sparkContext.parallelize([
          [1, 0.0, &quot;ext-0.0&quot;],
          [1, 1.0, &quot;ext-1.0&quot;],
          [2, 1.0, &quot;ext-2.0&quot;],
          [3, 2.0, &quot;ext-3.0&quot;],
          [4, 3.0, &quot;ext-4.0&quot;],
      ]),
      StructType([
          StructField(&quot;id&quot;, IntegerType(), True),
          StructField(&quot;shared&quot;, DoubleType(), True),
          StructField(&quot;shared&quot;, StringType(), True),
      ])
  )
  </code></pre>
  <p>I wish to retain only the numeric columns.</p>
  <p>However, attempting to do something like <code>df.select(&quot;id&quot;, &quot;shared&quot;).show()</code> results in:</p>
  <pre><code>raise AnalysisException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.AnalysisException: &quot;Reference 'shared' is ambiguous, could be: shared, shared.;&quot;
  </code></pre>
  <p>Many related solution to this problem are simply 'avoid ever getting into this situation', eg. by using <code>['joinkey']</code> instead of <code>a.joinkey = b.joinkey</code> on the join. I reiterate that this is not the situation here; this relates to a dataframe that has already been converted into this form.</p>
  <p>The metadata from the DF disambiguates these columns:</p>
  <pre><code>$ df.dtypes
  [('id', 'int'), ('shared', 'double'), ('shared', 'string')]
  
  $ df.schema
  StructType(List(StructField(id,IntegerType,true),StructField(shared,DoubleType,true),StructField(shared,StringType,true)))
  </code></pre>
  <p>So the data <em>is</em> retained internally... I just can't see how to use it.</p>
  <p>How do I pick one column over the other?</p>
  <p>I expected to be able to use, eg. <code>col('shared#11')</code> or similar... but there is nothing like that I can see?</p>
  <p>Is this simply not possible in spark?</p>
  <p>To answer this question, I would ask, please post either a) a working code snippet that solves the problem above, or b) link to something official from the spark developers that this simply isn't supported?</p>