Using relatedTo, and/or and :empty/:notempty in queries
Please don't laugh, but I thought to myself, let's give this a go in Twig and see where it takes me.
It very soon began to become clear, that this will end up in lots of criteria models, db queries and a huge pile of Twig code.

So my better answer should be, don't do it in Twig. This is something you should better do in PHP with
custom database queries (→ Yii query builder). But now that I've done it, here's a gist of that Twig code.