title: Spark Join Returning Null Values in Columns
body: <p>I'm pulling my hair out trying to solve what I feel is an extremely simple problem, but I'm not sure if there's some spark voodoo occurring as well.</p>
  
  <p>I have two tables, which are both very small. <code>Table A</code> has about 90K rows and <code>Table B</code> has about 2K rows.</p>
  
  <p>Table A</p>
  
  <pre><code>A  B  C  D
  ===========
  a1 b1 c1 d1
  a1 b1 c2 d2
  a1 b1 c3 d3
  a2 b2 c1 d1
  a2 b2 c2 d2
  .
  .
  .
  </code></pre>
  
  <p>Table B</p>
  
  <pre><code>A  B  E  F
  ===========
  a1 b1 e1 f1
  a2 b2 e2 f2
  </code></pre>
  
  <p>I want a table that looks like</p>
  
  <p>Result Table</p>
  
  <pre><code>A  B  C  D  E  F
  =================
  a1 b1 c1 d1 e1 f1
  a1 b1 c2 d2 e1 f1
  a2 b2 c1 d1 e2 f2
  .
  .
  .
  </code></pre>
  
  <p>I was a little loose, but the idea is I want to join the table with fewer rows on the table with more rows and it's okay to have multiple associated values in the final table.</p>
  
  <p>This should be really simple:</p>
  
  <p><code>table_a.join(table_b, table_a.a == table_b.a, table_a.b == table_b.b).select(..stuff..)</code></p>
  
  <p>HOWEVER, for <strong>almost all</strong> of the resulting values in the <code>Result Table</code> (which should have about 90K rows since <code>Table A</code> has about 90K rows), I get <code>null</code> values in columns <code>E</code> and <code>F</code>.</p>
  
  <blockquote>
    <p>When I save the result of just <code>Table B</code>, I see all the columns and values.
    When I save the result of just <code>Table A</code>, I see all the columns and values.
    (i.e I could do a paper and pencil join)</p>
  </blockquote>
  
  <p>The weird thing is that even though ~89K rows have <code>null</code> values in columns <code>E</code> and <code>F</code> in the <code>Result Table</code>, there are a few values that do randomly join.</p>
  
  <p>Does anyone know what's going on or how I can diagnose this?</p>