title: Spark Streaming Kafka job stuck in 'processing' stage
body: <p>I have a streaming job that reads from Kafka (@1min batch) and after some operations POSTs it to a HTTP endpoint. Every few hours it's getting stuck in 'processing' stage and starts queueing jobs thereafter:</p>
  
  <p><a href="https://i.stack.imgur.com/VlqZG.png" rel="nofollow"><img src="https://i.stack.imgur.com/VlqZG.png" alt=""></a></p>
  
  <p>After examining the running 'Executors' (in app-UI page) I found that only 
  1 out of 6 executors was showing 2 'Active Tasks'. </p>
  
  <p><a href="https://i.stack.imgur.com/Kmq3s.png" rel="nofollow"><img src="https://i.stack.imgur.com/Kmq3s.png" alt="enter image description here"></a></p>
  
  <p>Upon doing thread dump for that, it showed 2 threads for "Executor task launch worker" threadpool (<a href="https://github.com/ueshin/apache-spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L85" rel="nofollow">source</a>). These threads were all stuck at the same error:</p>
  
  <p><a href="https://i.stack.imgur.com/hAya0.png" rel="nofollow"><img src="https://i.stack.imgur.com/hAya0.png" alt="enter image description here"></a></p>
  
  <p>Full readable error:</p>
  
  <pre><code>java.lang.Object.wait(Native Method)
  java.lang.Object.wait(Object.java:502)
  java.net.InetAddress.checkLookupTable(InetAddress.java:1393)
  java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1310)
  java.net.InetAddress.getAllByName0(InetAddress.java:1276)
  java.net.InetAddress.getAllByName(InetAddress.java:1192)
  java.net.InetAddress.getAllByName(InetAddress.java:1126)
  java.net.InetAddress.getByName(InetAddress.java:1076)
  java.net.InetSocketAddress.&lt;init&gt;(InetSocketAddress.java:220)
  kafka.network.BlockingChannel.liftedTree1$1(BlockingChannel.scala:59)
  kafka.network.BlockingChannel.connect(BlockingChannel.scala:49)
  kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
  kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:151)
  kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:69)
  kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:112)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:112)
  kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:111)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111)
  kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:111)
  kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
  kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:110)
  org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:193)
  org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:209)
  org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
  scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
  scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
  org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:161)
  org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
  org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
  org.apache.spark.scheduler.Task.run(Task.scala:85)
  org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>This seems to be a <a href="http://bugs.java.com/view_bug.do?bug_id=7012768" rel="nofollow">JDK bug</a> that must have been fixed in JDK 7 - I made sure that I am using '1.8.0_101 (Oracle Corporation)'. I tried adding following on the command line (as suggested <a href="https://wiki.zimbra.com/wiki/Configuring_for_IPv4" rel="nofollow">here</a>), but it didn't fix the issue:</p>
  
  <pre><code>-Djava.net.preferIPv4Stack=true -Dnetworkaddress.cache.ttl=60
  </code></pre>
  
  <p>Does anyone have any ideas on an approach to debug/fix this?</p>
  
  <p>*Edit: Renaming the question to remove confusing JDK reason</p>