Mounting encrypted HFS partition after messed up partition table
I managed to get my data back.

After copying the harddisk to another one with dd to not cause any more damage I started to try to get it to work again. The solution turned out to be:


change mbr back to gpt (used disk genius for that again, had to delete the first 200mb partition to be able to do that)
change the guid type of disk2s2 to corestorage using diskutil (set id=53746F72-6167-11AA-AA11-00306543ECAC)
boot again in macos and the volume was back


It took a lot longer because I expected that disk2s2 was a HFS+ encrypted volume but viewing it in a hex editor I didn't find the markers for that, so I thought my data somehow has been overwritten or that the partitions were not aligned properly anymore. Only after recreating my previous setup I found out that there was a CoreStorage layer in between and then it was an easy fix.

Still not entirely sure what went wrong in the first place, because I tried to recreate my scenario on another harddisk and it worked there. My guess is my old harddisk had an mbr and gpt (hybrid) and thats what confused disk genius, but I don't know for sure.