title: LockRows plan node taking long time
body: <p>I have the following query in Postgres (<a href="https://www.2ndquadrant.com/en/blog/what-is-select-skip-locked-for-in-postgresql-9-5/" rel="nofollow noreferrer">emulating a work queue</a>):</p>
  <pre><code>DELETE FROM work_queue
  WHERE id IN ( SELECT l.id
                FROM work_queue l
                WHERE l.delivered = 'f' and l.error = 'f' and l.archived = 'f'
                ORDER BY created_at
                LIMIT 5000
                FOR UPDATE SKIP LOCKED );
  </code></pre>
  <p>While running the above concurrently (4 processes per second) along with a concurrent ingest at the rate of 10K records/second into <code>work_queue</code>, the query effectively bottlenecks on <code>LockRow</code> node.</p>
  <p>Query plan output:</p>
  <pre><code> Delete on work_queue  (cost=478.39..39609.09 rows=5000 width=67) (actual time=38734.995..38734.998 rows=0 loops=1)
     -&gt;  Nested Loop  (cost=478.39..39609.09 rows=5000 width=67) (actual time=36654.711..38507.393 rows=5000 loops=1)
           -&gt;  HashAggregate  (cost=477.96..527.96 rows=5000 width=98) (actual time=36654.690..36658.495 rows=5000 loops=1)
                 Group Key: &quot;ANY_subquery&quot;.id
                 -&gt;  Subquery Scan on &quot;ANY_subquery&quot;  (cost=0.43..465.46 rows=5000 width=98) (actual time=36600.963..36638.250 rows=5000 loops=1)
                       -&gt;  Limit  (cost=0.43..415.46 rows=5000 width=51) (actual time=36600.958..36635.886 rows=5000 loops=1)
                             -&gt;  LockRows  (cost=0.43..111701.83 rows=1345680 width=51) (actual time=36600.956..36635.039 rows=5000 loops=1)
                                   -&gt;  Index Scan using work_queue_created_at_idx on work_queue l  (cost=0.43..98245.03 rows=1345680 width=51) (actual time=779.706..2690.340 rows=250692 loops=1)
                                         Filter: ((NOT delivered) AND (NOT error) AND (NOT archived))
           -&gt;  Index Scan using work_queue_pkey on work_queue  (cost=0.43..7.84 rows=1 width=43) (actual time=0.364..0.364 rows=1 loops=5000)
                 Index Cond: (id = &quot;ANY_subquery&quot;.id)
   Planning Time: 8.424 ms
   Trigger for constraint work_queue_logs_work_queue_id_fkey: time=5490.925 calls=5000
   Trigger work_queue_locked_trigger: time=2119.540 calls=1
   Execution Time: 46346.471 ms
  </code></pre>
  <p>(corresponding visualization: <a href="https://explain.dalibo.com/plan/ZaZ" rel="nofollow noreferrer">https://explain.dalibo.com/plan/ZaZ</a>)</p>
  <p>Any ideas on improving this? Why should locking rows take so long in the presence of concurrent inserts? Note that if I do not have concurrent inserts into the <code>work_queue</code> table, the query is super fast.</p>