MySQL replication slave VERY slow I am working on a replication problem that completely baffles me! This client has two MySQL replication clusters on HUGE bare metal HW. See Environment below.
The IO_Tread of the slave is WAY behind, several hours or more. Yes, the IO_tread, not the SQL_tread. Why is it so hard to download not all that large binlog records and write them to disk. I tried finding a resource bottleneck, but given the massive HW none can be found.
The only strange observation is that the slave has 8x the IO OPS than the master. But even this does not really overload the SSD disk. Packet traces show the SLAVE often setting the TCP window to zero. Why, there are plenty resources?
Anyone with ideas what could be causing this strange behaviour? Why do I have more IO on the slave? What can cause the IO_tread to slow down?
Environment:
Both machines:
Bare metal DELL,   MySQL 5.6.30,   12CPUs,   128GB mem,
datadir on SSD, Net I/F:  Emulex 10Gb,  ROW based binlog FMT
Symptoms:
MASTER:
CPU:  67%  1 processor lightly used,   MEM:  70% used, 30% free,
IO OPS: ~2500 tps,  30% util on SSD,   slave client tread: Send binlog to slave.
SLAVE:
CPU:  40%  1 processor lightly used,   MEM:  70% used, 30% free,   IO OPS:   ~16000 tps,  70% util on SSD,   Error counters on net I/F are 0 (zero),
TCP window is often set to 0 on IO_tread,
Slave IO_tread is VERY slow. Lags more than an hour!
Another slave on SAME master has no trouble at all! This slave has much lower HW spec!
Trouble downloading master binlog.
Why this insanely high IO rate?
Stopping the slave also stops the IO OPS. (As expected, the OPS are from MySQL)
Copying large amounts of data from master to slave over the network (using ncat) show performance as expected.
Other observations:
When reversing the roles the problem stays the same.
Another replication cluster with same HW has no trouble. IO OPS in this cluster on slave are slightly less than on master. This cluster uses STATEMENT based binlog
Sorry, I need a 50 rep to comment, so...  Here's a couple of things to start.


Are you by chance using ZFS for your filesystem (it can throttle the disk unexpectedly)?
Are your config files (nearly) identical (with exception of replication master/slave settings)?  The config should be the same as the other slave except maybe memory buffers (sounded like the server HW was different).  https://www.percona.com/doc/percona-toolkit/2.2/pt-config-diff.html
(Run on the slave, and post) SHOW SLAVE STATUS\G
(Run on the master, and post) SHOW MASTER STATUS\G
Check the Slave Status a few times (maybe a minute apart). Are both Exec_Master_Log_Pos and Read_Master_Log_Pos (or Retrieved_Gtid_Set and Executed_Gtid_Set), moving (ideal is if they are catching up)?
Are the relay logs being stored as tables or files?


Edit
After reading Rick's post, I went back and noticed (thought at first both were SBR) your cluster that runs faster is running SBR (statement based replication), and the slow one is RBR (row based).  Rick's question about the type of queries and how many binlogs you generate in say an hour is important.

Sometimes RBR is faster, and sometimes SBR wins.  Always important to test and benchmark all scenarios.