title: org.apache.avro.UnresolvedUnionException: Not in union ["long","null"]
body: <p>I have written this code to save a Spark dataframe into an avro file using Spark (1.3.0), Scala (2.10.4), and Spark Avro (1.0.0)</p>
  
  <pre><code>  def getMatchingLine(line: String) : Option[Row] = {
      val regex = "^.*&amp;50=(\\w+)&amp;.*&amp;62-\\d=8&amp;63-\\d=(\\w+)&amp;.*timestamp=(\\d+).*$".r
      line match {
        case regex(a, b, c) =&gt; Some(Row(a, b, c))
        case _ =&gt; None
      }
    }
  
      val schema = StructType(List(StructField("a", StringType, true), StructField("b", StringType, true), StructField("c", LongType, true)))
      val rdd = sc.textFile(inputPath).cache()
      val rdd = rdd.map(getMatchingLine).filter(_.isDefined).flatMap(x =&gt; x)
      val df = sqlSc.createDataFrame(rdd, schema)
      df.save("/user/foo/", "com.databricks.spark.avro")
  </code></pre>
  
  <p>The stack trace of the error message is </p>
  
  <pre><code>ion: org.apache.avro.UnresolvedUnionException: Not in union ["long","null"]: 1426647603695
          at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
          at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:169)
          at org.apache.avro.mapred.AvroOutputFormat$1.write(AvroOutputFormat.java:166)
          at org.apache.spark.SparkHadoopWriter.write(SparkHadoopWriter.scala:96)
          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1073)
          at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
          at org.apache.spark.scheduler.Task.run(Task.scala:64)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
  </code></pre>