title: How to optimize number of executor instances in spark structured streaming app?
body: <h3>Runtime</h3>
  
  <p>YARN cluster mode</p>
  
  <h3>Application</h3>
  
  <ul>
  <li>Spark structured streaming</li>
  <li>Read data from Kafka topic</li>
  </ul>
  
  <h3>About Kafka topic</h3>
  
  <ul>
  <li>1 topic with 4 partitions -for now. (number of partitions can be changed)</li>
  <li>Added 2000 records maximum in topic per 1 second.</li>
  </ul>
  
  <p>I've found out that the number of Kafka topic partitions is matched with the number of spark executors (1:1).
  <br>
  So, in my case, what I know until now, 4 spark executors is the solution I think.
  <br>
  But I'm worried about data throughput - can be ensured 2000 rec/sec?
  <br><br>
  Is there any guidance or recommendation about setting proper configuration in spark structured streaming?
  <br>
  Especially <code>spark.executor.cores</code>, <code>spark.executor.instances</code> or something about executor.</p>