title: Adding StringType column to existing Spark DataFrame and then applying default values
body: <p>Scala 2.10 here using Spark 1.6.2. I have a <em>similar</em> (but not the same) question as <a href="https://stackoverflow.com/questions/29483498/append-a-column-to-data-frame-in-apache-spark-1-3">this one</a>, however, the accepted answer is not an <a href="http://sscce.org" rel="nofollow noreferrer">SSCCE</a> and assumes a certain amount of "upfront knowledge" about Spark; and therefore I can't reproduce it or make sense of it. <strong>More importantly</strong>, that question is also just limited to adding a new column to an existing dataframe, whereas I need to add a column <em>as well as</em> a value for all existing rows in the dataframe.</p>
  
  <hr>
  
  <p>So I want to add a column to an existing Spark DataFrame, and then apply an initial ('default') value for that new column to all rows.</p>
  
  <pre><code>val json : String = """{ "x": true, "y": "not true" }"""
  val rdd = sparkContext.parallelize(Seq(json))
  val jsonDF = sqlContext.read.json(rdd)
  
  jsonDF.show()
  </code></pre>
  
  <p>When I run that I get this following as output (via <code>.show()</code>):</p>
  
  <pre><code>+----+--------+
  |   x|       y|
  +----+--------+
  |true|not true|
  +----+--------+
  </code></pre>
  
  <p>Now I want to add a new field to <code>jsonDF</code>, after it's created and without modifying the <code>json</code> string, such that the resultant DF would look like this:</p>
  
  <pre><code>+----+--------+----+
  |   x|       y|   z|
  +----+--------+----+
  |true|not true| red|
  +----+--------+----+
  </code></pre>
  
  <p>Meaning, I want to add a new "<code>z</code>" column to the DF, of type <code>StringType</code>, and then default all rows to contain a <code>z</code>-value of <code>"red"</code>.</p>
  
  <p>From that other question I have pieced the following pseudo-code together:</p>
  
  <pre><code>val json : String = """{ "x": true, "y": "not true" }"""
  val rdd = sparkContext.parallelize(Seq(json))
  val jsonDF = sqlContext.read.json(rdd)
  
  //jsonDF.show()
  
  val newDF = jsonDF.withColumn("z", jsonDF("col") + 1)
  
  newDF.show()
  </code></pre>
  
  <p>But when I run this, I get a compiler error on that <code>.withColumn(...)</code> method:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Cannot resolve column name "col" among (x, y);
      at org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:152)
      at org.apache.spark.sql.DataFrame$$anonfun$resolve$1.apply(DataFrame.scala:152)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.sql.DataFrame.resolve(DataFrame.scala:151)
      at org.apache.spark.sql.DataFrame.col(DataFrame.scala:664)
      at org.apache.spark.sql.DataFrame.apply(DataFrame.scala:652)
  </code></pre>
  
  <p>I also don't see any API methods that would allow me to set <code>"red"</code> as the default value. Any ideas as to where I'm going awry?</p>