How do I limit the resources that pg_dump uses?
My assumption is that RAM isn't your problem. But that you're so IO-bogged down that you can't write to a file as a quick as you can read from the database.

There is no solution here, pg_dump is made it work as fast as it can. If it can read data faster than it can write it, your ram usage may go up. Until the table is flushed to disk. You may want to enable compression, see man psql and look for -Z,

   -Z 0..9
   --compress=0..9
       Specify the compression level to use. Zero means no compression. For the custom archive format, this specifies compression of individual table-data
       segments, and the default is to compress at a moderate level. For plain text output, setting a nonzero compression level causes the entire output file
       to be compressed, as though it had been fed through gzip; but the default is not to compress. The tar archive format currently does not support
       compression at all.


As far as I know, processes don't know how much ram a system has or how much they should take. Normally, setting a limit just does what happened here and kills the process. If it didn't kill the process, they process may just keep drinking ram until it starts paging, and then beyond.