title: How to avoid double connection close upon regaining connectivity?
body: <p>Under these circumstances:</p>
  
  <ol>
  <li>The client loses network connectivity to zk.</li>
  <li>One minute passes.</li>
  <li>The client regains network connectivity to zk.</li>
  </ol>
  
  <p>I'm getting the following panic:</p>
  
  <pre><code>panic: close of closed channel
  
  goroutine 2849 [running]:
  github.com/samuel/go-zookeeper/zk.(*Conn).Close(0xc420795180)
    github.com/samuel/go-zookeeper/zk/conn.go:253 47
  github.com/curator-go/curator.(*handleHolder).internalClose(0xc4203058f0, 0xc420302470, 0x0)
    github.com/curator-go/curator/state.go:136 +0x8d
  github.com/curator-go/curator.(*handleHolder).closeAndReset(0xc4203058f0, 0xc42587cd00, 0x1e)
    github.com/curator-go/curator/state.go:122 +0x2f
  github.com/curator-go/curator.(*connectionState).reset(0xc420302420, 0x1b71d87, 0xf)
    github.com/curator-go/curator/state.go:234 +0x55
  github.com/curator-go/curator.(*connectionState).handleExpiredSession(0xc420302420)
    github.com/curator-go/curator/state.go:351 +0xd9
  github.com/curator-go/curator.(*connectionState).checkState(0xc420302420, 0xffffff90, 0x0, 0x0, 0xc425ed2600, 0xed0e5250a)
    github.com/curator-go/curator/state.go:318 +0x9c
  github.com/curator-go/curator.(*connectionState).process(0xc420302420, 0xc425ed2680)
    github.com/curator-go/curator/state.go:299 +0x16d
  created by github.com/curator-go/curator.(*Watchers).Fire
    github.com/curator-go/curator/watcher.go:64 +0x96
  </code></pre>
  
  <p>This is the detailed sequence of events:</p>
  
  <ol>
  <li>The client loses network connectivity to zk.</li>
  <li>One minute passes.</li>
  <li>The client regains network connectivity to zk.</li>
  <li>goroutine A calls <code>s.ReregisterAll()</code> -> <code>Conn()</code> -> <code>checkTimeout()</code> -> <code>reset</code> (bc 1 minute has elapsed) -> <code>closeAndReset()</code> -> <code>conn.Close()</code> <strong>which can block for a second</strong></li>
  <li>goroutine B handles <code>zk.StateExpired</code> (zk cluster sends this bc it considers this client as dead since it didn't ping during 2.) -> <code>reset</code> -> <code>closeAndReset()</code> -> <code>conn.Close()</code> which causes a panic because <code>conn.Close()</code> already closed the connection's <code>c.shouldQuit</code> channel AND <code>s.zooKeeper.getZookeeperConnection</code> was never called by goroutine A because it was blocking for the second so there's no new connection.</li>
  </ol>
  
  <p>A naive solution I tried is to just use a mutex on <code>reset</code>, but now I'm getting <code>helper.GetConnectionString()</code> equal to empty string. What's the best way to avoid this crash and try to get into a good state when the client loses and then regains network connectivity? Should the fix be in <a href="http://github.com/samuel/go-zookeeper" rel="noreferrer">github.com/samuel/go-zookeeper</a>'s implementation of not letting you close an already closed connection?</p>
  
  <p>(I've filed this issue <a href="https://github.com/curator-go/curator/issues/4" rel="noreferrer">here</a>, but the project seems to be lacking in terms of discussion so I'm asking on SO.)</p>