title: spark-submit configuration to handle event load of 60K events per second
body: <p>I'm setting up spark Submit job to handle more than 60k events per seconds. What should be my batch internal and driver, executor, nodes, core setting to handle that load. </p>
  
  <p>I have tried batch interval from 1 minute to 10 minutes. 
  With executor memory from 4GB to 30G, with number of core 10 to 60. </p>
  
  <p>spark-submit --conf "spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2" --master spark://masterURL:7077 --deploy-mode cluster --driver-memory 30g --num-executors 60 --executor-cores 10   --executor-memory 30g --conf "spark.scheduler.mode=FAIR" --class “MainClass” SampleJar.jar</p>
  
  <p>The scheduled batch interval should get completed before next schedule</p>