Disk encryption format for external drives that works on both macos and linux
Veracrypt+exFat works on Mac/Linux/Windows. The problems with mounting on MacOS were caused by configuring veracrypt encryption for the entire disk rather than a single partition. As a result the partition table was encrypted too. While this worked fine on linux it caused problems on MacOS. Creating a normal unencrypted partition table with an exFat partition and then encrypting that partition with veracrypt works fine cross-platform.

The downside is that Apple's exFat driver does not always cleanly unmount the partition which in turn unconditionally triggers exfatfsck on the next mount attempt and that can take hours to finish for large size disks.