title: Pyspark groupBy on large dataframe
body: <p>We have a very large Pyspark Dataframe, on which we need to perform a groupBy operation.</p>
  <p>We've tried with</p>
  <pre><code>df_gp=df.groupBy('some_column').count()
  </code></pre>
  <p>and it's taking a really long time (it's been running for more than 17hrs with no result).</p>
  <p>I also tried with</p>
  <pre><code>df_gp=df.groupBy('some_column').agg(count)
  </code></pre>
  <p>but as far as I can tell the behaviour is the same.</p>
  <p>For more context :</p>
  <ul>
  <li>we are running this operation on Zeppelin (Version 0.8.0), using %spark2.pyspark interpreter</li>
  <li>Zeppelin is running on a Yarn client</li>
  <li>Data is stored on Hive (Hive 3.1.0.3.1.0.0-78)</li>
  <li>Initial Dataframe is created by querying Hive with llap :</li>
  </ul>
  <pre><code>from pyspark_llap import HiveWarehouseSession
  hive = HiveWarehouseSession.session(spark).build()
  
  req=&quot;&quot;&quot; SELECT *
          FROM table
          where isodate='2020-07-27'
      &quot;&quot;&quot;
  
  df = hive.executeQuery(req)
  </code></pre>
  <ul>
  <li>Dataframe size is ~60 millions rows, 9 columns</li>
  <li>other operations performed on the same Dataframe on the same environment, such as <code>count()</code> or <code>cache()</code> work in under a minute</li>
  </ul>
  <p>I've been reading about Spark's <code>groupBy</code> on different sources, but from what I gathered <a href="https://stackoverflow.com/questions/48870399/should-the-dataframe-function-groupby-be-avoided">here</a>, Dataframe API doesn't need to load or shuffle keys in memory, so it should not be a problem even on large Dataframes.</p>
  <p>I get that a <code>groupBy</code> on such a large volume of data can take some time, but this is really too much. I guess there are some memory parameters that maybe need tuning, or is there maybe something wrong with the way we're executing the groupBy operation?</p>
  <p>[EDIT] I forgot to mention there are some UDFs being processed on the Dataframe before <code>groupBy</code>. I've tried :</p>
  <ul>
  <li><code>groupBy</code> on a large Dataframe, without UDFs : gives result in less than a minute</li>
  <li><code>groupBy</code> on a sample of the processed Dataframe : same problem as before</li>
  </ul>
  <p>So we're thinking the UDFs are the actual cause of the problem, not <code>groupBy</code></p>