title: How to create a custom Encoder in Spark 2.X Datasets?
body: <p>Spark Datasets move away from Row's to <code>Encoder</code>'s for Pojo's/primitives. The <code>Catalyst</code> engine uses an <code>ExpressionEncoder</code> to convert columns in a SQL expression.  However there do not appear to be other subclasses of <code>Encoder</code> available to use as a template for our own implementations. </p>
  
  <p>Here is an example of code that is happy in Spark 1.X / DataFrames that does not compile in the new regime:</p>
  
  <pre class="lang-scala prettyprint-override"><code>//mapping each row to RDD tuple
  df.map(row =&gt; {
      var id: String = if (!has_id) "" else row.getAs[String]("id")
      var label: String = row.getAs[String]("label")
      val channels  : Int = if (!has_channels) 0 else row.getAs[Int]("channels")
      val height  : Int = if (!has_height) 0 else row.getAs[Int]("height")
      val width : Int = if (!has_width) 0 else row.getAs[Int]("width")
      val data : Array[Byte] = row.getAs[Any]("data") match {
        case str: String =&gt; str.getBytes
        case arr: Array[Byte@unchecked] =&gt; arr
        case _ =&gt; {
          log.error("Unsupport value type")
          null
        }
      }
      (id, label, channels, height, width, data)
    }).persist(StorageLevel.DISK_ONLY)
  </code></pre>
  
  <p>}</p>
  
  <p>We get a compiler error of </p>
  
  <pre><code>Error:(56, 11) Unable to find encoder for type stored in a Dataset.
  Primitive types (Int, String, etc) and Product types (case classes) are supported 
  by importing spark.implicits._  Support for serializing other types will be added in future releases.
      df.map(row =&gt; {
            ^
  </code></pre>
  
  <p>So then somehow/somewhere there should be a means to </p>
  
  <ul>
  <li>Define/implement our custom Encoder</li>
  <li>Apply it when performing a mapping on the <code>DataFrame</code> (which is now a Dataset of type <code>Row</code>)</li>
  <li>Register the Encoder for use by other custom code</li>
  </ul>
  
  <p>I am looking for code that successfully performs these steps.</p>