(Update Needed?) after PRAM reset, File Vault seems bricked
Entering my User-Password or the FileVault-Password didn't help. Recovery disk wouldn't work either.
After a lot of fiddling around I managed to access my internal TimeMachine drive. It's encrypted as well but here my backup key for the FileVault worked and I have another restore partition on this drive.

Once in restore mode I did still not manage to unlock my boot drive, and I finally had some textbox to check my typing. No idea what went wrong :( When I set up FileVault I checked all my backup codes and they worked all.

Currently I'm reinstalling 10.9 from my TM Recovery Drive, so in the end not a lot of harm done but still very frustrating.

Solution (sort of): Do backups and put Recovery Drives on those backup disks.