title: Spark driver pod getting killed with 'OOMKilled' status
body: <p>We are running a Spark Streaming application on a Kubernetes cluster using spark 2.4.5.
  The application is receiving massive amounts of data through a Kafka topic (one message each 3ms). 4 executors and 4 kafka partitions are being used.</p>
  <p>While running, the memory of the driver pod keeps increasing until it is getting killed by K8s with an 'OOMKilled' status. The memory of executors is not facing any issues.</p>
  <p>When checking the driver pod resources using this command :</p>
  <pre><code>kubectl top pod podName
  </code></pre>
  <p>We can see that the memory increases until it reaches 1.4GB, and the pod is getting killed.</p>
  <p>However, when checking the storage memory of the driver on Spark UI, we can see that the storage memory is not fully used (50.3 KB / 434 MB). Is there any difference between the <strong>storage memory of the driver</strong>, and the <strong>memory of the pod containing the driver</strong> ?</p>
  <p>Has anyone had experience with a similar issue before?</p>
  <p>Any help would be appreciated.</p>
  <p>Here are few more details about the app :</p>
  <ul>
  <li>Kubernetes version : 1.18</li>
  <li>Spark version : 2.4.5</li>
  <li>Batch interval of spark streaming context : 5 sec</li>
  <li>Rate of input data : 1 kafka message each 3 ms</li>
  <li>Scala language</li>
  </ul>