How to profile the PostgreSQL query planner diagnose what is causing it to take up large amounts of memory and time
The first stop to investigating this be would the system tool "perf".  You can get the pid of the running backend (by using "top", for example--make sure you get the pid of the backend, not the client program) then do sudo perf top -p &lt;pid&gt;.  You might get more useful output by first installing the debug symbols. How you do that is going to depend on your OS and how you installed it.

But once you identify the bottleneck, I wouldn't hold out much hope that anyone will do anything about it, given the far-out nature of your query.  If you are willing to dig into the source code to propose our own solution, then I would suggest you start by compiling your own server with version 13dev.  You aren't going to get very far in your investigations using only packaged versions, and especially not older packaged versions.  (Also, the problem might already be fixed in 12 or 13dev).