title: Unable to fix a Pyspark: Error length of fields (%d)" % (len(obj), len(dataType.fields))
body: <p>I am relatively new to Pyspark. I am trying to figure one particular type of error which is bugging me. 
  </p>
  
  <pre class="lang-py prettyprint-override"><code>lines = sc.textFile('train.csv')
  from pyspark.sql.types import *
  </code></pre>
  
  <p>The train.csv is stored <a href="https://drive.google.com/file/d/0By0bAi7hOBAFcFgtUEdpLUFDQkk/view" rel="nofollow noreferrer">here</a>: It is a bit huge. </p>
  
  <p>The first row contains the column information. For setting the schema from the first row of the data</p>
  
  <pre class="lang-py prettyprint-override"><code>fields = [StructField(field_name, StringType(), True) for field_name in lines.first().split(',')] # I am setting the schema here
  schema = StructType(fields)
  mstr_header = lines.filter(lambda l: "Country" in l) #Ihave seen the first row of the data, I want to remove it. Only the first row contains 'Country'
  linesNoHeader = lines.subtract(mstr_header)
  
  lines_df = linesNoHeader.map(lambda x: x.split(",")).toDF(schema) #make a dataframe
  </code></pre>
  
  <p>When I run <code>lines_df.count()</code>, PySpark throws an error saying, </p>
  
  <blockquote>
    <p>length of fields (%d)" % (len(obj), len(dataType.fields)))ValueError: Length of object (18) does not match with length of fields (17)</p>
  </blockquote>
  
  <p>I am unable to figure out where I am going wrong. Apologies for the large datafile. </p>