Is it safe to remove ~/Library/Mail folders
By safe, I assume you mean "will not delete message on the server?" That's correct, you can go ahead and delete those - think of the as a "cache" of your emails. As long as you aren't deleting and emptying the trash from within whatever mail client you are using, they will remain on the server.

Do consider a backup before deleting ANYTHING from ~/Library (or any other Libraries on your Mac).

In this specific case, I would explicitly rebuild the mailbox indexes just in case the system will clean them up once you're done deleting things...