Why is my replication lagging all of a sudden?
Ok, so after much googling I discovered the issue was quite simply we had run out of IOPS to catch up with the master database instance.

The replica DB had 120GB of storage, and the baseline IOPS for RDS is 3x storage space, so in this case it was 360 IOPS.

Cloudwatch can tell you the burst credits you have remaining, and it turned out they were all used up right at the time the replica lag started. This was great as it confirmed my suspicion that we'd run out of IOPS (in addition to @Rick_James suggesting it..)

I've now upgraded the replica to 250GB which gives 750 IOPS as a baseline - based off IOPS when everything was working fine this should be more than enough to keep up with the master server. Also, it's not too expensive ;)

Replica lag after EBS resize to 250GB:


IOPS chart after resize: