In PostgreSQL, what area in memory does HashSetOp use, work_mem or shared_buffer?
work_mem is not an &quot;area&quot;, it is a just value.
HashSetOp is pretty poor at memory estimation and usage.  It has no provision for spilling to disk. If it ends up using more memory than work_mem is set to, then it just goes ahead and uses more memory than it was supposed to.  There is no external enforcement mechanism (until the kernel steps in), and in the case of HashSetOp no internal one either.
If the planner thinks ahead of time that HashSetOp would overrun work_mem, then it just doesn't get chosen, and you will probably get some sort-based method instead.  But again, the estimation is poor, it will often use several times more memory than the planner thought it would, and will happily blow way past the setting of work_mem.
HashSetOp could really use some improvements, but it is so seldom used that no one is very excited about doing the work.