title: Dynamically update batch size of input for spark kafka consumer
body: <p>I use <code>createDirectStream</code> in my spark streaming application. I set the batch interval to 7 seconds and most of the time the batch job can finish within about 5 seconds. However, in very rare cases, the batch job need cost 60 seconds and this will delay some batch of jobs.
  To cut down the total delay time, I hope I can process more streaming data which spread over the delayed jobs at one time. This will help the streaming return to normal as soon as possible. </p>
  
  <p>So, I want to know there is some method to dynamically update/merge batch size of input for spark and kafka when delay appears.</p>