title: spark kafka 10 consumer (DirectStream) Hangs
body: <p>it seems that we are having same issue as described here:
  <a href="https://issues.apache.org/jira/browse/SPARK-20780" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-20780</a></p>
  
  <p>I am already aware that its a Kafka issue rather then spark, but still would like to get some advice on how to act until that issue would be resolved by the Kafka community.</p>
  
  <p>Increasing the request.timeout is not much helping, because then we could get large queue. For example if the micro-batch is 10 seconds and the Kafka <code>request.timeout.ms</code> is 20 seconds, each time that the issue occurs, it would create a delay of around 20 seconds which would lead to 2 micro-batches queuing up. Obviously the more it happens, the more delays it causes which eventually leads to quite a large queue.</p>
  
  <p>Any best practices / workarounds / tips for how to overcome that issue until it would be resolved ?</p>