Unable to encrypt my Note 5 anymore (data partition is corrupt)
Flashing complete stock ROM including stock recovery will fix encryption, if you factory reset from stock recovery.  

To make your ROM boot along with TWRP you need to flash no-verity-opt-encrypt or universal disable dm-verity.  

Due some bug in TWRP, decryption dialog fails to open because EFS is not mounted at the time that TWRP checks for device encryption. TWRP is working along with stock ROM encryption if you swap entries in /etc/recovery.fstab and one of these flags for USERDATA:  

flags=encryptable=footer
flags=encryptable=footer,length=-16384
flags=encryptable=footer,length=-20480
flags=encryptable=/efs/metadata
flags=encryptable=/dev/block/bootdevice/by-name/metadata
flags=encryptable=/dev/block/platform/15570000.ufs/by-name/EFS/metadata
flags=fileencryption=aes-256-xts:aes-256-cts
flags=fileencryption=ice:aes-256-cts
flags=fileencryption=ice:aes-256-cts;keydirectory=/metadata/vold/metadata_encryption


source: source.android.com/security/encryption