title: How to use salting technique for joining data frames having skewed data
body: <p>I am new to spark and trying to understand how to deal with skewed data in spark. I have created two tables employee and department. Employee has skewed data for one of the department.</p>
  <p>One of the solution is to broadcast the department table and that works fine. But I want to understand how could I use salting technique in below code to improve performance.</p>
  <pre><code>from pyspark.sql import SparkSession
  import pyspark.sql.functions as f
  spark = SparkSession.builder.appName(&quot;skewTestSpark&quot;).config(&quot;spark.sql.warehouse.dir&quot;,
                                         '/user/hive/warehouse').enableHiveSupport().getOrCreate()
  df1 = spark.sql(&quot;select * from spark.employee&quot;)
  df2 = spark.sql(&quot;select id as dept_id, name as dept_name from spark.department&quot;)
  res = df1.join(df2, df1.department==df2.dept_id)
  res.write.parquet(&quot;hdfs://&lt;host&gt;:&lt;port&gt;/user/result/employee&quot;)
  </code></pre>
  <p>Distribution for above code:
  <a href="https://i.stack.imgur.com/1ZWl5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1ZWl5.png" alt="enter image description here" /></a></p>