title: Spark SQL: automatic schema from csv
body: <p>does spark sql provide any way to automatically load csv data?
  I found the following Jira: <a href="https://issues.apache.org/jira/browse/SPARK-2360">https://issues.apache.org/jira/browse/SPARK-2360</a> but it was closed....</p>
  
  <p>Currently I would load a csv file as follows:</p>
  
  <pre><code>case class Record(id: String, val1: String, val2: String, ....)
  
   sc.textFile("Data.csv")
  .map(_.split(",")) 
  .map { r =&gt;                  
     Record(r(0),r(1), .....)
  }.registerAsTable("table1")
  </code></pre>
  
  <p>Any hints on the automatic schema deduction from csv files? In particular a) how can I generate a class representing the schema and  b) how can I automatically fill it (i.e. Record(r(0),r(1), .....))?</p>
  
  <p>Update:
  I found a partial answer to the schema generation here:
  <a href="http://spark.apache.org/docs/1.1.0/sql-programming-guide.html#data-sources">http://spark.apache.org/docs/1.1.0/sql-programming-guide.html#data-sources</a></p>
  
  <pre><code>// The schema is encoded in a string
  val schemaString = "name age"
  // Generate the schema based on the string of schema
  val schema =
   StructType(
  schemaString.split(" ").map(fieldName =&gt; StructField(fieldName, StringType, true)))
  // Convert records of the RDD (people) to Rows.
  val rowRDD = people.map(_.split(",")).map(p =&gt; Row(p(0), p(1).trim))
  // Apply the schema to the RDD.
  val peopleSchemaRDD = sqlContext.applySchema(rowRDD, schema)
  </code></pre>
  
  <p>So the only question left would be how to do the step
    <code>map(p =&gt; Row(p(0), p(1).trim))</code> dynamically for the given number of attributes?</p>
  
  <p>Thanks for your support!
  Joerg</p>