Why are the system tables in MySQL and MariaDB utf8 and not utf8mb4? I know the general advice is never to use utf8, but instead to use utf8mb4. If that's the advice, then why do the system tables themselves use utf8?

You can see this in the MariaDB test suite, and and the MySQL test suite; or, by using SHOW CREATE TABLE mysql.db;
By Definitions as per MySQL Reference Manual:
  
  utf8mb4: A UTF-8 encoding of the Unicode character set using one to
  four bytes per character.
  
  utf8mb3 : A UTF-8 encoding of the Unicode character set using one to
  three bytes per character.


Since, utf8mb3 (also known as UTF8) can not store all languages like Chinese or Emoticons, it is recommended to use utf8mb4 when it comes to store application data.  

For simple example, You want to extend your existing business into China.  if you have used utf8mb4, your application is always ready to adopt your business extension while in case of utf8mb3(UTF-8), you need an character set upgrade to your application.

While storing system's data MySQL does not store information (supplementary character) which requires utf8mb4, this could be the reason, why MySQL itself is not using utf8mb4.