title: How can I use Spark Dataset with Thrift
body: <p>My data format is defined with apache thrift, with code generated by scrooge. I store it in spark using parquet, very much like what is explained in this <a href="http://engineering.ooyala.com/blog/using-parquet-and-scrooge-spark" rel="nofollow">blog</a>.</p>
  
  <p>I can read that data back into a Dataframe very easily, simply doing:</p>
  
  <pre><code>val df = sqlContext.read.parquet("/path/to/data")
  </code></pre>
  
  <p>And I can read it in RDD with a bit more of gymnastics:</p>
  
  <pre><code>def loadRdd[V &lt;: TBase[_, _]](inputDirectory: String, vClass: Class[V]): RDD[V] = {
      implicit val ctagV: ClassTag[V] = ClassTag(vClass)
      ParquetInputFormat.setReadSupportClass(jobConf, classOf[ThriftReadSupport[V]])
      ParquetThriftInputFormat.setThriftClass(jobConf, vClass)
      val rdd = sc.newAPIHadoopFile(
        inputDirectory, classOf[ParquetThriftInputFormat[V]], classOf[Void], vClass, jobConf)
      rdd.asInstanceOf[NewHadoopRDD[Void, V]].values
    }
  loadRdd("/path/to/data", classOf[MyThriftClass])
  </code></pre>
  
  <p>My question is: how can I access that data in the new Dataset api released with spark 1.6? The reason I want that are the benefits of the dataset api: type safety with the same speed of dataframe.</p>
  
  <p>I understand some sort of Encoder is needed, and those are already provided for primitive types and case classes, but what I have is the thrift generated code (the java or scala one, either one fits the bill), which does look a lot like a case class, but it is not really one.</p>
  
  <p>I tried the obvious options, which did not work:</p>
  
  <pre><code>val df = sqlContext.read.parquet("/path/to/data")
  
  df.as[MyJavaThriftClass]
  
  &lt;console&gt;:25: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._  Support for serializing other types will be added in future releases.
  
  df.as[MyScalaThriftClass]
  
  scala.ScalaReflectionException: &lt;none&gt; is not a term
    at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:199)
    at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:84)
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:492)
    at org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:394)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
    at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
    ... 48 elided
  
  
  df.as[MyScalaThriftClass.Immutable]
  
  java.lang.UnsupportedOperationException: No Encoder found for org.apache.thrift.protocol.TField
  - field (class: "org.apache.thrift.protocol.TField", name: "field")
  - array element class: "com.twitter.scrooge.TFieldBlob"
  - field (class: "scala.collection.immutable.Map", name: "_passthroughFields")
  - root class: "com.worldsense.scalathrift.ThriftRange.Immutable"
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:597)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor$1.apply(ScalaReflection.scala:509)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor$1.apply(ScalaReflection.scala:502)
    at scala.collection.immutable.List.flatMap(List.scala:327)
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:502)
    at org.apache.spark.sql.catalyst.ScalaReflection$.toCatalystArray$1(ScalaReflection.scala:419)
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:537)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor$1.apply(ScalaReflection.scala:509)
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor$1.apply(ScalaReflection.scala:502)
    at scala.collection.immutable.List.flatMap(List.scala:327)
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:502)
    at org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:394)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54)
    at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41)
    ... 48 elided
  </code></pre>
  
  <p>It seems that shapeless <a href="https://github.com/milessabin/shapeless/issues/371" rel="nofollow">works fine</a> with Thrift generated code, and I was wondering if maybe I can use it to generate something that the current encoders api will accept.</p>
  
  <p>Any hints?</p>