title: Spark Scala: Task Not serializable error
body: <p>I am using IntelliJ Community Edition with Scala Plugin and spark libraries. I am still learning Spark and am using Scala Worksheet. </p>
  
  <p>I have written the below code which removes punctuation marks in a String:</p>
  
  <pre><code>def removePunctuation(text: String): String = {
    val punctPattern = "[^a-zA-Z0-9\\s]".r
    punctPattern.replaceAllIn(text, "").toLowerCase
  }
  </code></pre>
  
  <p>Then I read a text file and try to remove punctuation:</p>
  
  <pre><code>val myfile = sc.textFile("/home/ubuntu/data.txt",4).map(removePunctuation)
  </code></pre>
  
  <p>This gives error as below, any help would be appreciated:</p>
  
  <blockquote>
    <p>org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(/home/ubuntu/src/main/scala/Test.sc:294)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(/home/ubuntu/src/main/scala/Test.sc:284)
        at org.apache.spark.util.ClosureCleaner$.clean(/home/ubuntu/src/main/scala/Test.sc:104)
        at org.apache.spark.SparkContext.clean(/home/ubuntu/src/main/scala/Test.sc:2090)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(/home/ubuntu/src/main/scala/Test.sc:366)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(/home/ubuntu/src/main/scala/Test.sc:365)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(/home/ubuntu/src/main/scala/Test.sc:147)
        at #worksheet#.#worksheet#(/home/ubuntu/src/main/scala/Test.sc:108)
    Caused by: java.io.NotSerializableException: A$A21$A$A21
    Serialization stack:
        - object not serializable (class: A$A21$A$A21, value: A$A21$A$A21@62db3891)
        - field (class: A$A21$A$A21$$anonfun$words$1, name: $outer, type: class A$A21$A$A21)
        - object (class A$A21$A$A21$$anonfun$words$1, )
        at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2094)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:370)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:369)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
        at org.apache.spark.rdd.RDD.map(RDD.scala:369)
        at A$A21$A$A21.words$lzycompute(Test.sc:27)
        at A$A21$A$A21.words(Test.sc:27)
        at A$A21$A$A21.get$$instance$$words(Test.sc:27)
        at A$A21$.main(Test.sc:73)
        at A$A21.main(Test.sc)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jetbrains.plugins.scala.worksheet.MyWorksheetRunner.main(MyWorksheetRunner.java:22)</p>
  </blockquote>