title: org.apache.spark.SparkException: Task not serializable -- Scala
body: <p>I am reading a text file and it is fixed width file which I need to convert to csv. My program works fine in local machine but when I run it on cluster, it throws "Task not serializable" exception.</p>
  
  <p>I tried to solve same problem with map and mapPartition.</p>
  
  <p>It works fine by using toLocalIterator on RDD. But it doesm't work with large file(I have files of 8GB)</p>
  
  <p>Below is code by using mapPartition which I recently tried</p>
  
  <p>//reading source file and creating RDD</p>
  
  <pre><code>def main(){
      var inpData = sc.textFile(s3File)
      LOG.info(s"\n inpData &gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt; [${inpData.count()}]")
  
      val rowRDD = inpData.mapPartitions(iter=&gt;{
      var listOfRow = new ListBuffer[Row]
      while(iter.hasNext){
         var line = iter.next()
         if(line.length() &gt;= maxIndex){
            listOfRow += getRow(line,indexList)
          }else{
            counter+=1
          }
       }
      listOfRow.toIterator
      })
  
      rowRDD .foreach(println)
  }
  
  case class StartEnd(startingPosition: Int, endingPosition: Int) extends Serializable
  
  def getRow(x: String, inst: List[StartEnd]): Row = {
      val columnArray = new Array[String](inst.size)
      for (f &lt;- 0 to inst.size - 1) {
        columnArray(f) = x.substring(inst(f).startingPosition, inst(f).endingPosition)
      }
      Row.fromSeq(columnArray)
  }
  </code></pre>
  
  <p>//Note : for your refernce, indexList I have created using StartEnd case class, which looks like below after creation</p>
  
  <pre><code>[List(StartEnd(0,4), StartEnd(4,10), StartEnd(7,12), StartEnd(10,14))]
  </code></pre>
  
  <p>This program works fine in my local machine. But when I put on cluster(AWS) it throws exception as shown below.</p>
  
  <pre><code>&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;Map(ResultantDF -&gt; [], ExceptionString -&gt; 
  Exception occurred while applying the FileConversion transformation and the exception Message is :Task not serializable
  Exception occurred while applying the FileConversion transformation and the exception Message is :Task not serializable)
  [Driver] TRACE reflection.ReflectionInvoker$.invokeDTMethod - Exit
  </code></pre>
  
  <p>I am not able to understand what's wrong here and what is not serializable, why it is throwing exception.</p>
  
  <p>Any help is appreciated.
  Thanks in advance!</p>