title: Pyspark: Serialized task exceeds max allowed. Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values
body: <p>I'm doing calculations on a cluster and at the end when I ask summary statistics on my Spark dataframe with df.describe().show() I get an error:</p>
  <p>Serialized task 15:0 was 137500581 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values</p>
  <p>In my Spark configuration I already tried to increase the aforementioned parameter:</p>
  <pre><code>spark = (SparkSession
           .builder
           .appName(&quot;TV segmentation - dataprep for scoring&quot;)
           .config(&quot;spark.executor.memory&quot;, &quot;25G&quot;)
           .config(&quot;spark.driver.memory&quot;, &quot;40G&quot;)
           .config(&quot;spark.dynamicAllocation.enabled&quot;, &quot;true&quot;)
           .config(&quot;spark.dynamicAllocation.maxExecutors&quot;, &quot;12&quot;)
           .config(&quot;spark.driver.maxResultSize&quot;, &quot;3g&quot;)
           .config(&quot;spark.kryoserializer.buffer.max.mb&quot;, &quot;2047mb&quot;)
           .config(&quot;spark.rpc.message.maxSize&quot;, &quot;1000mb&quot;)
           .getOrCreate())
  </code></pre>
  <p>I also tried to repartition my dataframe using:</p>
  <pre class="lang-py prettyprint-override"><code>dfscoring=dfscoring.repartition(100)
  </code></pre>
  <p>but still I keep on getting the same error.</p>
  <p>My environment: Python 3.5, Anaconda 5.0, Spark 2</p>
  <p>How can I avoid this error ?</p>