Why is my replication lagging all of a sudden? We run a db.m4.10xlarge instance on AWS for our web app(s) and I've recently set up a read replica as a db.m4.2xlarge instance. We've been experimenting with smaller instances for a read replica but they couldn't keep up with the master. The db.m4.2xlarge had no trouble catching up with the master server within an hour or so - ready to go.

Then last night, something odd happened. Write activity on the replica dropped right down and now I can see the replica has drifted 2.5hrs or so from the master (we have constant write activity on one table with timestamps so it's easy to reliably check).

Does anyone know what the cause of this could be? I can't see anything at all in the logs - it's almost as though we've lost a write thread or something like that.. very weird.

Write IOPS last 24hrs:


Replica lag:


If anyone can provide any pointers that'd be helpful. Both the master and replica are MySQL 5.6.34. No changes have been made since spinning up the replica.
Ok, so after much googling I discovered the issue was quite simply we had run out of IOPS to catch up with the master database instance.

The replica DB had 120GB of storage, and the baseline IOPS for RDS is 3x storage space, so in this case it was 360 IOPS.

Cloudwatch can tell you the burst credits you have remaining, and it turned out they were all used up right at the time the replica lag started. This was great as it confirmed my suspicion that we'd run out of IOPS (in addition to @Rick_James suggesting it..)

I've now upgraded the replica to 250GB which gives 750 IOPS as a baseline - based off IOPS when everything was working fine this should be more than enough to keep up with the master server. Also, it's not too expensive ;)

Replica lag after EBS resize to 250GB:


IOPS chart after resize: