How to limit work_mem on PostgreSQL
According to the PostgreSQL Wiki this sort of thing isn't possible.


  PostgreSQL has no facilities to limit what resources a particular
  user, query, or database consumes, or correspondingly to set
  priorities such that one user/query/database gets more resources than
  others. It's necessary to use operating system facilities to achieve
  what limited prioritization is possible.


As suggested in the text above, you could try using operating system facilities to achieve limits such as that. For example, ulimit can be used to create process-specific resource limits.