title: Implement a parallel batch processing queue in Neo4J Cypher
body: <p>When I worked with Informix SQL it was possible to use transaction isolation of simple updates to create a processing queue with something like this:</p>
  
  <pre><code>UPDATE Queue
  SET processed_by = ?
  WHERE processed_by IS NULL
  ORDER BY inserted_at
  LIMIT 3
  </code></pre>
  
  <p>this marks a batch of at most 3 items (queue may be empty) to be processed by the processor in the first param (<code>?</code>, e.g. server name and thread number). After this <code>UPDATE</code> finished on each parallel processor (separate threads or machines), the queue was sharded:</p>
  
  <pre><code>+-----+------------------+---------------+
  | _id |   inserted_at    | processed_by  |
  +-----+------------------+---------------+
  |   1 | 2017-03-07 01:15 | host2:thread3 |
  |   2 | 2017-03-07 01:16 | host2:thread3 |
  |   3 | 2017-03-07 01:17 | host2:thread3 |
  |   4 | 2017-03-07 01:18 | host1:thread1 |
  |   5 | 2017-03-07 01:19 | host1:thread1 |
  |   6 | 2017-03-07 01:20 | host1:thread1 |
  |   7 | 2017-03-07 01:21 | NULL          |
  +-----+------------------+---------------+
  </code></pre>
  
  <p>and each processor can work on its own on a batch in isolation:</p>
  
  <pre><code>SELECT * from Queue WHERE processed_by = ?
  </code></pre>
  
  <p>while they could be sure no-one else touches their items to process. As a side effect, if a processor died, it was easy for them to pick up where they left off.</p>
  
  <hr>
  
  <p>I'm trying to achieve the same thing with Cypher, but I'm running into problems when parallel reads observe the same state and hence update the same items:</p>
  
  <pre><code>MATCH (n:Queue)
  WHERE NOT exists(n.processed_by)
  SET n.processed_by = $processor
  RETURN n
  LIMIT 3
  </code></pre>
  
  <p>Building on this I was able to <a href="http://neo4j.com/docs/stable/transactions-isolation.html" rel="nofollow noreferrer">lock items</a> and do the processing with this:</p>
  
  <pre><code>MATCH (n:Queue)
  SET n._LOCK_ = true
  WITH n
  WHERE NOT exists(n.processed_by)
  SET n.processed_by = $processor
  REMOVE n._LOCK_
  RETURN n
  LIMIT 3
  </code></pre>
  
  <p>The problem is that the the <code>_LOCK_</code> is applied to all Queue nodes, not just 3 of them, which could result in a performance hit if there are many of them. It wouldn't be a problem if sometimes a processor would get back 0 locked items due to other processors locking all available items.</p>
  
  <p><strong>What is the correct way to express this with Cypher?</strong><br>
  If possible, I would like to also include ordering (see <code>inserted_at</code> in SQL).<br>
  I would be also happy to make it work without a limit, so just marking a single item without conflicts.</p>
  
  <p><em>Note: I'm hoping this is possible without using any server-side extensions.</em></p>