Get entries where relation matches or is empty
For anyone that comes across this. Craft CMS has added this as a feature for next release.
https://github.com/craftcms/cms/issues/7256#issuecomment-744644568