title: Do Spark encoders respect Java's rules of inheritance?
body: <p><strong>My understanding</strong>: If I have a model class that extends a second model class, I shouldn't be able to access the private members of the parent class in the child class (unless I use reflection).</p>
  <p>Extending this, I <em>expect</em> that when a Spark dataframe is encoded as a dataset of the child model class, it shouldn't have columns that include private members of the parent model class. (But this is not what I observe.)</p>
  <p>More concretely, my parent class:</p>
  <pre><code>public class Foo {
      private int one;
      protected String two;
      protected double three;
  }
  </code></pre>
  <p>The child class:</p>
  <pre><code>public class Bar extends Foo {
      private int four;
      protected String five;
  }
  </code></pre>
  <p>I have a couple of <code>Bar</code> objects that I use to create a Spark dataframe i.e., <code>Dataset&lt;Row&gt;</code> like so:</p>
  <pre><code>Dataset&lt;Row&gt; barDF = session.createDataframe(barList, Bar.class);
  </code></pre>
  <p>When, at a later point, I want to encode this as a dataset,</p>
  <pre><code>Dataset&lt;Bar&gt; barDS = barDF.as(Encoders.bean(Bar.class));
  </code></pre>
  <p>I expect <code>barDS</code> to have four columns (excluding <code>one</code>, the private member of <code>Foo</code>). But the result of <code>barDS.show()</code> is instead:</p>
  <pre><code>+------+------+-----+-------+-----+
  | five | four | one | three | two |
  +------+------+-----+-------+-----+
  | 9    | 9    | 0   | 3.0   | 3   |
  | 16   | 16   | 0   | 4.0   | 4   |
  +------+------+-----+-------+-----+
  </code></pre>
  <p>What am I missing in expecting <code>one</code> not to be present in the dataset? Also, what encoding can I use instead of bean encoding so that Java's rules of inheritance are obeyed?</p>