title: Spark for Python - can't cast a string column to decimal/double
body: <p>In all the posted questions for this action, I couldn't find something that works.</p>
  <p>I was trying several versions, in all of them I have this <code>DataFrame</code>:</p>
     
  <pre class="lang-python prettyprint-override"><code>dataFrame = spark.read.format(&quot;com.mongodb.spark.sql&quot;).load()
  </code></pre>
  <p>The printout of <code>dataFrame.printSchema()</code> is</p>
  <pre class="lang-python prettyprint-override"><code>root
   |-- SensorId: string (nullable = true)
   |-- _id: struct (nullable = true)
   |    |-- oid: string (nullable = true)
   |-- _type: string (nullable = true)
   |-- device: string (nullable = true)
   |-- deviceType: string (nullable = true)
   |-- event_id: string (nullable = true)
   |-- gen_val: string (nullable = true)
   |-- lane_id: string (nullable = true)
   |-- system_id: string (nullable = true)
   |-- time: string (nullable = true)
  </code></pre>
  <p>After the DataFrame is created, I want to cast the column <code>'gen_val'</code>(that is stored in the variable <code>results.inputColumns</code>) from <code>String</code> type to <code>Double</code> type. Different versions led to different errors.</p>
  <p><strong>Version #1</strong></p>
  <p>Code:</p>
  <pre class="lang-python prettyprint-override"><code>dataFrame = dataFrame.withColumn(results.inputColumns, dataFrame[results.inputColumns].cast('double'))
  </code></pre>
  <p>using <code>cast(DoubleType())</code> instead, will generate the same error</p>
  <p>Error:</p>
  <pre class="lang-python prettyprint-override"><code>AttributeError: 'DataFrame' object has no attribute 'cast'
  </code></pre>
  <p><strong>Version #2</strong></p>
  <p>Code:</p>
  <pre class="lang-python prettyprint-override"><code>dataFrame = dataFrame.withColumn(results.inputColumns, dataFrame['gen_val'].cast('double'))
  </code></pre>
  <p>even though this option is not really releveant, because the parameter cannot be hard-coded...</p>
  <p>Error:</p>
  <pre class="lang-python prettyprint-override"><code>dataFrame = dataFrame.withColumn(results.inputColumns, dataFrame['gen_val'].cast('double'))
    File &quot;/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py&quot;, line 1502, in withColumn
    File &quot;/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py&quot;, line 1133, in __call__
    File &quot;/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py&quot;, line 63, in deco
    File &quot;/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py&quot;, line 323, in get_return_value
  py4j.protocol.Py4JError: An error occurred while calling o31.withColumn. Trace:
  py4j.Py4JException: Method withColumn([class java.util.ArrayList, class org.apache.spark.sql.Column]) does not exist
          at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
          at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
          at py4j.Gateway.invoke(Gateway.java:272)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:214)
          at java.lang.Thread.run(Thread.java:748)
  </code></pre>