title: Spark GroupBy operation performance Improvement
body: <p>I have a DataFrame like below around 1 billion records with approximately 100 columns.</p>
  
  <pre><code>+---------+---------+------+------+------+
  |  Col1   |  Col2   | Col3 | col4 | col5 |
  +---------+---------+------+------+------+
  | Value 1 | Value 2 |  123 | 10.0 | sae  |
  | Value 1 | Value 2 |  123 | 10.0 | ser  |
  | Value 1 | Value 2 |  123 | 10.0 | wer  |
  +---------+---------+------+------+------+
  </code></pre>
  
  <p>I have to apply groupBy operation on 5 columns out of 100 columns like below.</p>
  
  <pre><code>df.groupBy("col1").count()
  df.groupBy("col2").count()
  df.groupBy("col3").count()
  df.groupBy("col4").count()
  df.groupBy("col5").count()
  </code></pre>
  
  <p>As it is an expensive operation with too much shuffling it's taking more time. For applying groupBy operations for 5 times itself, it is taking more time.</p>
  
  <p>But in my case I have to apply this 5 groupBy operations for every Filter.
  like If there are 5 filters i have to apply groupBy operations by 5*5 time's</p>
  
  <p>I am looking for a solution to resolve this issue with caching the whole data with partitions of those 5 columns. I'm thinking if we do the partitions the data will be local to that executor (Too much shuffle wont happen).</p>
  
  <p>CodeWise Flow will be:</p>
  
  <pre><code>val df = spark.read.format("parquet").load("file://location")
  
  val 1stdf = df.filter("$col1" == "a")
  df.groupBy("col1").count()
  df.groupBy("col2").count()
  df.groupBy("col3").count()
  df.groupBy("col4").count()
  df.groupBy("col5").count()
  
  val 2nddf = df.filter("$col2" == "abc")
  df.groupBy("col1").count()
  df.groupBy("col2").count()
  df.groupBy("col3").count()
  df.groupBy("col4").count()
  df.groupBy("col5").count()
  
  val 3rddf = df.filter("$col3" == "abdc")
  df.groupBy("col1").count()
  df.groupBy("col2").count()
  df.groupBy("col3").count()
  df.groupBy("col4").count()
  df.groupBy("col5").count()
  
  
  
  </code></pre>