Encryption Unsuccessful even after system backup restore
Found out the way to fix it:


Download stock firmware (*.tar.md5)
Flash it with Odin (it removes TWRP and replaces it with a standard recovery)
Start Android - still "Unsuccessful encryption" error
Boot into recovery mode (now it's standard recovery instead of TWRP)
Run factory reset
Start Android - now it finally starts - problem solved
Flash again TWRP with Odin
Restore the old good backup with TWRP
Done


So basically the factory reset form the original recovery mode fixes it.

Also I found out why the tablet bricked. When it performs encryption process it looks like it's off (no lights, sounds or anything). Do not disturb it, after 20-30 minutes it finishes the encryption and turns itself on. Just leave it be until the screen turns on.