title: Programmatically generate the schema AND the data for a dataframe in Apache Spark
body: <p>I would like to dynamically generate a dataframe containing a header record for a report, so creating a dataframe from the value of the string below:</p>
  
  <pre><code>val headerDescs : String = "Name,Age,Location"
  
  val headerSchema = StructType(headerDescs.split(",").map(fieldName =&gt; StructField(fieldName, StringType, true)))
  </code></pre>
  
  <p>However now I want to do the same for the data (which is in effect the same data i.e. the metadata).</p>
  
  <p>I create an RDD :</p>
  
  <pre><code>val headerRDD = sc.parallelize(headerDescs.split(","))
  </code></pre>
  
  <p>I then intended to use createDataFrame to create it:</p>
  
  <pre><code>val headerDf = sqlContext.createDataFrame(headerRDD, headerSchema)
  </code></pre>
  
  <p>however that fails because <code>createDataframe</code> is expecting a <code>RDD[Row]</code>, however my RDD is an array of strings - I can't find a way of converting my RDD to a Row RDD and then mapping the fields dynamically. Examples I've seen assume you know the number of columns beforehand, however I want the ability eventually to be able to change the columns without changing the code - having the columns in a file for example.</p>
  
  <p>Code excerpt based on first answer:</p>
  
  <pre><code>val headerDescs : String = "Name,Age,Location"
  
  // create the schema from a string, splitting by delimiter
  val headerSchema = StructType(headerDescs.split(",").map(fieldName =&gt; StructField(fieldName, StringType, true)))
  
  // create a row from a string, splitting by delimiter
  val headerRDDRows = sc.parallelize(headerDescs.split(",")).map( a =&gt; Row(a))
  
  val headerDf = sqlContext.createDataFrame(headerRDDRows, headerSchema)
  headerDf.show()
  </code></pre>
  
  <p>Executing this Results in:</p>
  
  <pre><code>+--------+---+--------+
  
  |    Name|Age|Location|
  
  +--------+---+--------+
  
  |    Name|
  
  |     Age|
  
  |Location|
  
  +--------+---+-------
  </code></pre>