Using mariabackup/xtrabackup vs block level snapshots
To answer my own question, GCP (and AWS) snapshots are mostly consistent, but in a very write heavy database, you would want to lock the tables for the duration of the snapshot backup. I believe there is actually an system level option to lock the drive itself for the snapshot, but that is not necessary for my use-case.

I ended up using both methods to backup. I scheduled an incremental snapshot to happen every 2 hours. Then I also do a mysql dump every night. For the mysqldump, it is done with the --single-transaction flag which will give you a consistent copy of the database. There is a momentary lock on all tables while a binary log point is found. But this is usually very short and not noticeable.

Binary logs are turned on and also included in the mysql dump. So this allows both methods to be used to do a point in time recovery. It is definitely possible that the snapshot image could create inconsistent copies, but it seems to not be an issue and innodb is smart enough to recover from most minor inconsistency issues.

The snapshots take very little storage space because they are only storing the delta with each snap. The mysqldump files do not, but if they become a problem, there are other backup options to store only a delta between each mysql dump.