What happens if transaction rollback fails?
I think your question hits the core of the database reliability properties; the ACID properties. They are there in order to guarantee the database's reliability. The transaction log plays a vital role here. You probably know the ACID properties, so Iam not diving into that.
If a rollback fails, then you would have a serious problem. The reliability of the database cannot be guaranteed. In other words; you probably have some sort of corruption in your transaction log and will end up with an inconsistent database. I am not sure how it works with PostgreSQL, but for MS SQL Server it will set the DB offline and restarts it, which will trigger a recovery action and it will try to rollback the transaction again. If that doesn't work you will probably have to restore the database from an earlier backup (log, differential or full).