title: What is spark.driver.maxResultSize?
body: <p>The <a href="http://spark.apache.org/docs/latest/configuration.html" rel="noreferrer">ref</a> says:</p>
  
  <blockquote>
    <p>Limit of total size of serialized results of all partitions for each
    Spark action (e.g. collect). Should be at least 1M, or 0 for
    unlimited. Jobs will be aborted if the total size is above this limit.
    Having a high limit may cause out-of-memory errors in driver (depends
    on spark.driver.memory and memory overhead of objects in JVM). Setting
    a proper limit can protect the driver from out-of-memory errors.</p>
  </blockquote>
  
  <p>What does this attribute do exactly? I mean at first (since I am not battling with a job that fails due to out of memory errors) I thought I should increase that.</p>
  
  <p>On second thought, it seems that this attribute defines the max size of the result a worker can send to the driver, so leaving it at the default (1G) would be the best approach to protect the driver..</p>
  
  <p>But will happen on this case, the worker will have to send more messages, so the overhead will be just that the job will be slower?</p>
  
  <hr>
  
  <p>If I understand correctly, assuming that a worker wants to send 4G of data to the driver, then having <code>spark.driver.maxResultSize=1G</code>, will cause the worker to send 4 messages (instead of 1 with unlimited <code>spark.driver.maxResultSize</code>). If so, then increasing that attribute to protect my driver from being assassinated from Yarn should be wrong.</p>
  
  <p>But still the question above remains..I mean what if I set it to 1M (the minimum), will it be the most protective approach?</p>