title: Memcached for PHP and failover
body: <p>We are deploying memcached for our application, and I would like to make it as resistant as I can.</p>
  
  <p>We plan to use the newer <a href="http://ar2.php.net/manual/en/book.memcached.php" rel="noreferrer">memcacheD</a> extension.</p>
  
  <p>One thing that I haven not fully figured out is, what happens if one of the servers dies. At the very least it seems like the memcached client just 'gives up' on that server, and doesn't store anything in it.</p>
  
  <p>This behavior I'm fine with. We can deal with a bunch of cache-misses. However, it would be nice that after one of the servers is deemed 'failed' the subsequent sets and gets get re-distributed to the servers that are left over.</p>
  
  <p>Since this does not seem to happen automatically; I guess the only way to approach this issue is to have an external system do health-checks on memcached systems, and update the list of servers appropriately.</p>
  
  <p>But if there's a list of 10 servers, and lets say, the 5th dies.. even with the Ketama-hashing it would seem that this would trigger a big redistribution of keys (this is just based on common sense).</p>
  
  <p>So ideally, I would simply like the PHP extension to figure out a server is down, mark it for down for a specified amount of time (10 minutes) and during those 10 minutes fall back to other servers (nicely distributed) for sets and gets.</p>
  
  <p>How do other people solve this?</p>
  
  <p><strong>Edit</strong> : clarifying my libketama point.</p>
  
  <p>Say we have 10 servers:</p>
  
  <pre><code>1,2,3,4,5,6,7,8,9,10
  </code></pre>
  
  <p>One of them dies. Libketama will then provide a very high likelyhood that the hits to the missing server get equally distributed to the remaining servers:</p>
  
  <pre><code>1,2,3,4,inactive,6,7,8,9,10
  </code></pre>
  
  <p>BUT: if we provide and manage this list manually, this is not the case:</p>
  
  <pre><code>1,2,3,4,6,7,8,9,10 // There are now 9 servers!
  </code></pre>
  
  <p>6 will get now 5's previous keys, 7 will get 6's. 8 will get 7's, 9 will get 8's and 10 will get 9's. All the hits the 10th servers used to get, will not be evenly distributed among the remaining ones. Resulting a high likelyhood of almost 50% of all keys being sent to new servers.</p>