Is there some way to specify memory limits on specific apps in MacOS?
Even though you can limit memory for specific apps on macOS, you will not achieve the result you're looking for:

The problem is that when you limit the amount of memory an application can have, this means in practical sense that programs are going to crash when they reach that limit. 

So you essentially have the option of having crashing programs or pausing them when they reach the limit. Pausing them is something you have to implement a program to do yourself (not very complex, but you would need some scripting experience or programming experience).

In simple terms, memory limits are not going to help you any more than simply quitting one or two of those hogs when you need to do something else.

UPDATE: In the comments you wrote that you wouldn't mind crashing applications, and that you are an accomplished software engineer, so you would be able to do scripting/programming yourself.

In that case you can implement a memory limit in a variety of ways:

Simplest version

The simplest would be to create a small script that periodically runs ps x -o rss -p $PID | tail -n1 to observe the memory usage of the indicated process ID. This grabs the resident set size, but you might for some odd reason want to use vsz instead of rss to get the vsize instead.

If the observed memory usage is above a threshold you specify, you can then terminate the process. The best way to do this would be to run a command like kill -15 $PID and then after a time run kill -9 $PID. The first is SIGTERM, which tells the process to start closing down by itself. The latter is SIGKILL, which kills the process if it refuses to close down by itself.

Advanced version

A slightly more advanced version is to create a small wrapper program that calls setrlimit() to set a limit on the process, such as for example RLIMIT_DATA (maximum data segment size) or RLIMIT_RSS (maximum resident set size). Afterwards it needs to call execl() or similar to start the application for which you want to limit memory usage.

Note that there's also a possibility of using ulimit instead of rlimit, however my experience is that these limits are not observed on current macOS.