title: DSE Spark Streaming: Long active batches queue
body: <p>I have the following code:</p>
  
  <pre><code>val conf = new SparkConf()
    .setAppName("KafkaReceiver")
    .set("spark.cassandra.connection.host", "192.168.0.78")
    .set("spark.cassandra.connection.keep_alive_ms", "20000")
    .set("spark.executor.memory", "2g")
    .set("spark.driver.memory", "4g")
    .set("spark.submit.deployMode", "cluster")
    .set("spark.executor.instances", "3")
    .set("spark.executor.cores", "3")
    .set("spark.shuffle.service.enabled", "false")
    .set("spark.dynamicAllocation.enabled", "false")
    .set("spark.io.compression.codec", "snappy")
    .set("spark.rdd.compress", "true")
    .set("spark.streaming.backpressure.enabled", "true")
    .set("spark.streaming.backpressure.initialRate", "200")
    .set("spark.streaming.receiver.maxRate", "500")
  
  val sc = SparkContext.getOrCreate(conf)
  val ssc = new StreamingContext(sc, Seconds(10))
  val sqlContext = new SQLContext(sc)
  val kafkaParams = Map[String, String](
    "bootstrap.servers" -&gt; "192.168.0.113:9092",
    "group.id" -&gt; "test-group-aditya",
    "auto.offset.reset" -&gt; "largest")
  
  val topics = Set("random")
  val kafkaStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topics)
  </code></pre>
  
  <p>I'm running the code through <code>spark-submit</code> with the following command:</p>
  
  <pre><code>dse&gt; bin/dse spark-submit --class test.kafkatesting /home/aditya/test.jar
  </code></pre>
  
  <p>I have a three-node Cassandra DSE cluster installed on different machines. Whenever I run the application, it takes so much data and starts creating a queue of active batches, which in turn creates a backlog and a long scheduling delay. How can I increase the performance and control the queue such that it receives a new batch only after it finishes executing the current batch?</p>