title: Spark refusing explicit broadcast join
body: <p>In my pyspark code (v2.4) I join two dataframes: one very large, the other tiny (just a few entries). The small DF, however, is obtained as a summary of another large DF. 
  My broadcast join maxsize thresholds are generous enough for a broadcast join to take place, but it just doesn't. Even when I explicitly call for it:
  <code>df_large.join(broadcast(df_small), ...)</code>
  physical plan still indicates SortMergeJoin. I get BroadcastJoin only if I save the small DF to a file and re-read it prior to the join command.</p>
  
  <p>What other means do I have to enforce broadcast join?
  As a background info: </p>
  
  <ol>
  <li>Broadcast join is needed because otherwise there is an enormous skew</li>
  <li>replacing a join with UDF seems like a bad alternative because this involves an early collect (though maybe broadcast join does it as well?)</li>
  </ol>
  
  <p>Thanks</p>