title: Index out of range while converting from rdd to dataframe
body: <p>I am trying to convert spark RDD to dataframe. While RDD is fine when I convert it to dataframe I get index out of range error. </p>
  
  <pre><code>alarms = sc.textFile("hdfs://nanalyticsedge.com:8020/hdp/oneday.csv")
  alarms = alarms.map(lambda line: line.split(","))
  header = alarms.first()
  alarms = alarms.filter(lambda line:line != header)
  alarms = alarms.filter(lambda line: len(line)&gt;1)
  alarms_df = alarms.map(lambda line: Row(IDENTIFIER=line[0],SERIAL=line[1],NODE=line[2],NODEALIAS=line[3],MANAGER=line[4],AGENT=line[5],ALERTGROUP=line[6],ALERTKEY=line[7],SEVERITY=line[8],SUMMARY=line[9])).toDF()
  alarms_df.take(100)
  </code></pre>
  
  <p><em>Here alarms.count() works fine whereas alarms_df.count() gives index out of range. <strong>It is data export from oracle</em></strong></p>
  
  <p>From @Dikei's answer I found that:</p>
  
  <pre><code>alarms = alarms.filter(lambda line: len(line) == 10)
  </code></pre>
  
  <p>gives me proper Dataframe but why do dataframe get lost when it is database export and how do I prevent it?</p>