A fast encryption for external hard drive
Mac OS Journaled Extended, Encrypted

I have external Time Machine drives formatted as Mac OS Journaled Extended, Encrypted. The encryption is fast enough not to be significant.

With this set-up, I have not experienced any problems or unacceptable back-up times. I too disconnect drives between back-ups.

Formatting and Mounting

Format the external drive using an encrypted format. This will ensure the whole disk is encrypted. Time Machine thus is no longer involved in the encryption process.

On connecting the drive, macOS will ask for the drive's passphrase. You can choose to remember the passphrase in your Keychain.

Encrypting

If Time Machine is handling the encryption, you will see a Sparse Disk Image Bundle in your external drive. This disk image file will be named after your computer.

If FileVault is handling the encryption, you will see a folder called Backups.backupdb in your external drive. This folder will contain a folder with the name of your computer.

You want the second, FileVault to transparently handle the encryption.