Send test mail works, CiviMail doesn't
Apparently, CiviCRM adds a lot of things before the @yourdomain.com in the From-address. Things like a job id, event queue id, and a hash. Google didn't like that because that email address didn't match an existing mailbox. Luckily, in the Google SMTP relay settings, you can specify to accept any address from your domain. That did it!