TWRP 3.3.0 cannot decrypt data partition (of Android 9)
Well, sorry for bothering. The solution was simple: Don't use the wipe function of TWRP but the Format Data button in the wipe menu.

I was able to factory reset and install a different custom rom.

BTW: Don't use TWRP for Android >= 9 (Pie). As of Version 3.3.0 TWRP lacks the vendor partition feature. Pitchblack seems to be the standard now.

Edit: I think the problem is: Android 9 introduced the quota feature of the ext4 filesystem. TWRP doesn't know about it yet.