Upgrade from Postgres 11 to Postgres 12 made some queries 300x slower, probably due to new CTE handling
As pointed out in the answer to my other, more specific question, the reason is Just-in-time compilation.

SET jit = false; solves all performance problems in my case.