title: Building a StructType from a dataframe in pyspark
body: <p>I am new spark and python and facing this difficulty of building a schema from a metadata file that can be applied to my data file.
  Scenario: Metadata File for the Data file(csv format), contains the columns and their types: for example:</p>
  
  <pre><code>id,int,10,"","",id,"","",TRUE,"",0
  created_at,timestamp,"","","",created_at,"","",FALSE,"",0
  </code></pre>
  
  <p>I have successfully converted this to a dataframe that looks like:</p>
  
  <pre><code>+--------------------+---------------+
  |                name|           type|
  +--------------------+---------------+
  |                  id|  IntegerType()|
  |          created_at|TimestampType()|
  |          updated_at|   StringType()|
  </code></pre>
  
  <p>But when I try to convert this to a StructField format using this </p>
  
  <pre><code>fields = schemaLoansNew.map(lambda l:([StructField(l.name, l.type, 'true')]))
  </code></pre>
  
  <p>OR</p>
  
  <pre><code>schemaList = schemaLoansNew.map(lambda l: ("StructField(" + l.name + "," + l.type + ",true)")).collect()
  </code></pre>
  
  <p>And then later convert it to StructType, using</p>
  
  <pre><code>schemaFinal = StructType(schemaList)
  </code></pre>
  
  <p>I get the following error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "/opt/mapr/spark/spark-1.4.1/python/pyspark/sql/types.py", line 372, in __init__
  assert all(isinstance(f, DataType) for f in fields), "fields should be a list of DataType"
  AssertionError: fields should be a list of DataType
  </code></pre>
  
  <p>I am stuck on this due to my lack of knowledge on Data Frames, can you please advise, how to proceed on this. once I have schema ready I want to use createDataFrame to apply to my data File. This process has to be done for many tables so I do not want to hardcode the types rather use the metadata file to build the schema and then apply to the RDD.</p>
  
  <p>Thanks in advance.</p>