What kind of encryption is required for LineageOS on Xiaomi Redmi Note 5 (whyred) to boot?
Format data using TWRP, unmount the data partition, then resize the partition with adb shell :
resize2fs /dev/block/mmcblk0p50 51G

(my original size of filesystem /dev/block/mmcblk0p50 was 52.1G (you can find it with the command "df -h /dev/block/mmcblk0p50")
The encryption process need to take some place at the end of the partition or it will fail - this thread has helped me : github.com/CyanogenMod/android_device_motorola_xt907/issues/3
You can debug also the problem with the adb logcat command.