title: Spark cast column to sql type stored in string
body: <p>The simple request is I need help adding a column to a dataframe but, the column has to be empty, its type is from ...spark.sql.types and the type has to be defined from a string.</p>
  
  <p>I can probably do this with ifs or case but I'm looking for something more elegant. Something that does not require writing a case for every type in org.apache.spark.sql.types</p>
  
  <p>If I do this for example:</p>
  
  <pre><code>df = df.withColumn("col_name", lit(null).cast(org.apache.spark.sql.types.StringType))
  </code></pre>
  
  <p>It works as intended, but I have the type stored as a string, </p>
  
  <pre><code>var the_type = "StringType"
  </code></pre>
  
  <p>or 
      var the_type = "org.apache.spark.sql.types.StringType"</p>
  
  <p>and I can't get it to work by defining the type from the string.</p>
  
  <p>For those interested here are some more details: I have a set containing tuples (col_name, col_type) both as strings and I need to add columns with the correct types for a future union between 2 dataframes.</p>
  
  <p>I currently have this:</p>
  
  <pre><code>for (i &lt;- set_of_col_type_tuples) yield {
      val tip = Class.forName("org.apache.spark.sql.types."+i._2)
      df = df.withColumn(i._1, lit(null).cast(the_type))
      df }
  </code></pre>
  
  <p>if I use</p>
  
  <pre><code>val the_type = Class.forName("org.apache.spark.sql.types."+i._2)
  </code></pre>
  
  <p>I get</p>
  
  <pre><code>error: overloaded method value cast with alternatives:   (to: String)org.apache.spark.sql.Column &lt;and&gt;   (to: org.apache.spark.sql.types.DataType)org.apache.spark.sql.Column  cannot be applied to (Class[?0])
  </code></pre>
  
  <p>if I use</p>
  
  <pre><code>val the_type = Class.forName("org.apache.spark.sql.types."+i._2).getName()
  </code></pre>
  
  <p>It's a string so I get:</p>
  
  <pre><code>org.apache.spark.sql.catalyst.parser.ParseException: mismatched input '.' expecting {&lt;EOF&gt;, '('}(line 1, pos 3)
  == SQL == org.apache.spark.sql.types.StringType
  ---^^^
  </code></pre>
  
  <p>EDIT: So, just to be clear, the set contains tuples like this ("col1","IntegerType"), ("col2","StringType") not ("col1","int"), ("col2","string"). A simple cast(i._2) does not work.</p>
  
  <p>Thank you.</p>