title: Tips on decreasing network shuffle in spark
body: <p>I have this use case where I am joining two dataframes in Spark, <code>A</code> and <code>B</code>.</p>
  
  <pre><code>A -&gt; Huge dataframe approx size: 100 TB
  B -&gt; Smaller dataframe approx size: 100 MB
  </code></pre>
  
  <p>Two questions:</p>
  
  <ol>
  <li>How to reduce network shuffle as spark UI showed shuffle read of about 30gb.</li>
  <li>The number of tasks is also huge approx 1,000,000. Any tips to reduce them?</li>
  </ol>
  
  <p>I have tried caching dataframe <code>A</code>, but surprisingly it only made the job slower.
  Any help would be appreciated.</p>