<p dir="auto">We just ran into this in production: <a href="https://issues.apache.org/jira/browse/CURATOR-358" rel="nofollow">https://issues.apache.org/jira/browse/CURATOR-358</a></p>
<p dir="auto">Long story short, zookeeper was down, druid came up, zookeeper came up, but no master ever took over.</p>
<p dir="auto">Might want to bump the curator version in the pom.xml.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2017-03-14T20:50:58,718 WARN [qtp997219303-85] org.eclipse.jetty.servlet.ServletHandler - /druid/coordinator/v1/datasources
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:347) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.http.CoordinatorRedirectInfo.getRedirectURL(CoordinatorRedirectInfo.java:49) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHO
T]
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.Server.handle(Server.java:497) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1650) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:215) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:339) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        ... 22 more"><pre class="notranslate"><code class="notranslate">2017-03-14T20:50:58,718 WARN [qtp997219303-85] org.eclipse.jetty.servlet.ServletHandler - /druid/coordinator/v1/datasources
java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:347) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.http.CoordinatorRedirectInfo.getRedirectURL(CoordinatorRedirectInfo.java:49) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHO
T]
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.Server.handle(Server.java:497) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druid/prod/coordinator/_COORDINATOR
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1650) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:215) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.locks.LockInternals.getSortedChildren(LockInternals.java:151) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.locks.LockInternals.getParticipantNodes(LockInternals.java:133) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at org.apache.curator.framework.recipes.leader.LeaderLatch.getLeader(LeaderLatch.java:453) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        at io.druid.server.coordinator.DruidCoordinator.getCurrentLeader(DruidCoordinator.java:339) ~[druid-services-0.9.2.1-SNAPSHOT-selfcontained.jar:0.9.2.1-SNAPSHOT]
        ... 22 more
</code></pre></div>
<p dir="auto">When zookeeper has a blip, the overlord can get into a condition we call "split brain" where leadership election is all screwed up. This causes the state of submitted requests to be non deterministic, sometimes "succeeding" or sometimes registering that it succeeding but not actually returning a success.</p>
<p dir="auto">We have found two indicators for such a scenario. One is the following error in the logs of the overlord:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:251)
        at io.druid.indexing.overlord.http.OverlordRedirectInfo.getRedirectURL(OverlordRedirectInfo.java:52)
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:304)
        at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:293)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:290)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:281)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:42)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:375)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:346)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:339)
        at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:243)
        ... 22 more"><pre class="notranslate"><code class="notranslate">org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:251)
        at io.druid.indexing.overlord.http.OverlordRedirectInfo.getRedirectURL(OverlordRedirectInfo.java:52)
        at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /DRUID_PATH/indexer/leaderLatchPath/_c_3c32cd34-4370-4e26-922d-a9b24afa4a91-lock-0000102217
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:304)
        at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:293)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:290)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:281)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:42)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.participantForPath(LeaderSelector.java:375)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:346)
        at org.apache.curator.framework.recipes.leader.LeaderSelector.getLeader(LeaderSelector.java:339)
        at io.druid.indexing.overlord.TaskMaster.getLeader(TaskMaster.java:243)
        ... 22 more
</code></pre></div>
<p dir="auto">( This looks like <a href="https://issues.apache.org/jira/browse/CURATOR-358" rel="nofollow">https://issues.apache.org/jira/browse/CURATOR-358</a> )</p>
<p dir="auto">And the other is increased CPU on both active overlords.</p>
<p dir="auto">The solution is to restart the overlords and let them clean up their state.</p>
<p dir="auto">In rare scenarios, a middle manager can fail to submit its stuff to the overlord after a peon completes, and will never complete because it can get in a state where the peon already gave up its task lock, but never properly finished its segment insertion at the middle manager level, so the middle manager will retry indefinitely. The fix here is to restart the middle manager, but this looses any outstanding stuff not yet submitted to the overlord.</p>