title: Is there a right way to implement distributed caching (resistent to concurrent writing, netsplits, etc)?
body: <p>Lets say we have a database (PostgreSQL). Some requests are too expensive or slow, so we decided to cache some data, say, in Memcached. At first everything seems to be OK, but in fact there is a number of corner cases:</p>
  
  <ul>
  <li>Some Memcached servers could be unavailable (server is down, netsplits)</li>
  <li>Writes by one key can occur concurrently by 2+ backends</li>
  <li>Memcached server was added/removed, some backends are already aware of this, some are not</li>
  <li>etc</li>
  </ul>
  
  <p>Is there some well known solution to avoid inconsistency between caches and database under these circumstances? I can think of a few really simple solutions, but they lead to stale reads.</p>