Materialized Subquery Not Optimized
I have now found the source of the issue. It is because my ri_email_messages table was using the Aria engine. I had done this because InnoDB did not support full text search at the time. In MariaDB > 10.0.5, you can now do fulltext search in InnoDB, so I have switched back.

The way to tell in the EXPLAIN statement is that the PRIMARY query was using type=index. This should be type=eq_ref, since it is constrained upon the primary key, but this wasn't happening with Aria. After switching to InnoDB, this correctly shows as type=eq_ref, and the row match went from 126156 to 1.