title: How improve performance when join pyspark Dataframes
body: <p>I have 2 pyspark Dataframess, the first one contain <code>~500.000 rows</code> and the second contain <code>~300.000 rows</code>. I did 2 join, in the second join will take cell by cell from the second dataframe (300.000 rows) and compare it with all the cells in the first dataframe (500.000 rows).</p>
  
  <p>So, there's is very slow join. I broadcasted the dataframes before join.</p>
  
  <p>Test 1:</p>
  
  <pre><code>df_join = df1.join(F.broadcast(df2), df1.String.contains(df2["search.subString"]), "left") 
  </code></pre>
  
  <p>The job took many hours to finish.</p>
  
  <p>Test 2:</p>
  
  <pre><code>df_join = F.broadcast(df1).join(F.broadcast(df2), df1.String.contains(df2["search.subString"]), "left")
  </code></pre>
  
  <p>The running is very slow than the first code above, so the performance is very bad.</p>
  
  <p>I tried <code>to cache the dataframes</code> before join.</p>
  
  <p>I used:</p>
  
  <p><code>df.cache()</code> for each dataframe. But, the performance always not good.</p>
  
  <p>I tried to use <code>persist in memory_only</code>:</p>
  
  <pre><code>df.persist(MEMORY_ONLY) ==&gt; NameError: global name 'MEMORY_ONLY' is not defined
  df.persist(StorageLevel.MEMORY_ONLY) ==&gt; NameError: global name 'StorageLevel' is not defined
  </code></pre>
  
  <p>How can I persist the Dataframe in memory ?</p>
  
  <p>Can you please suggest me a solution to improve the performance ?</p>
  
  <p>Thanks in advance. </p>