mysql, performance issue with FOR UPDATE query
My Mantra:  "Don't queue it, just do it."  Although I don't fully understand your design, I suspect that the problems would decrease if you simply processed the data as it comes in.

A Rule of Thumb:  If an item can be processed in less time than it takes to enqueue and dequeue it, then the Queue is wasting time.

To better discuss this, I need to understand all the dataflow.  For starters, fill in the blanks:


Item(s) coming in ___ times a second.
__ arrive in each batch.
The client threads are limited to ____ threads by ______ (Apache/Tomcat/...).
The threading uses _____ (PHP/Java/C#/...).
max_connections = ____
Processing one item takes, on average ___ ms.
The table domains is current ____ rows; it is purged ______ (never/daily/immediately/...)  And provide SHOW CREATE TABLE domains.


More

Thanks for the update!

I gather that something else is changing test_stage, thereby effectively re-queueing rows?

Get rid of KEY test_stage (test_stage), since it is redundant with two other indexes.

Change KEY test_stage_id (test_stage, id) to KEY tid (test_stage, id, domain_name) so that it is "covering".

One thought is to get rid of id and promote domain_name to be the PRIMARY KEY.  This avoids the need to deal with two unique keys.  Note that that would change the query and optimal index to

SELECT domain_name  FROM domains  WHERE test_stage=1 
    ORDER BY domain_name  LIMIT 200  FOR UPDATE
INDEX(test_stage, domain_name)


Another thought is to assign the "worker" threads to contiguous chunks of the table, based on the PRIMARY KEY.  That way, they should never (or at least rarely) collide over which rows to grab.  (I suspect this is the core of the issue.)

I assume your buffer_pool is big enough.  But if you would like a critique of the VARIABLES, please provide SHOW GLOBAL STATUS and SHOW GLOBAL VARIABLES.