python child process exit unexpectedly with exit code -9
I have a PyTorch script with 16 processes. Following is a code snippet from the main process:

procs = [mp.Process(target=self.worker_wrapper, args=(i, )) for i in range(self.n_workers)]
for p in procs: p.start()
while True:
    time.sleep(60)
    for i, p in enumerate(procs):
        self.logger.info('Check: id %d, exitcode %s, alive %s' % (
            i, str(p.exitcode), str(p.is_alive())))


the worker_wrapper is this:

def worker_wrapper(self, id):
    try:
        self.worker(id)
    except Exception as e:
        self.logger.info(e)
        self.logger.error(traceback.format_exc())


The worker will do some stuff and should never stop. It works well at beginning, but usually after 1 days running, some child processes (not all) exit unexpectedly, I get the output like this:

[2017-06-15 08:45:21,540] Check: id 13, exitcode -9, alive False


And it didn't catch any exception. I tried to search exitcode -9 but didn't get anything useful. Does anyone meet this before? Or do you have any idea about how to debug this? I'm sorry I can't provide a minimal reproducible code snippet, because it usually takes one day to reproduce. And I'm using python2.7

Exit code -9 means the process was killed via SIGKILL. 

It's probably due to your machine running out of memory and the OS kernel was triggering the OOM killer. 

To verify that, check the kernel logs via dmesg you should notice the OOM killer targeting your worker processes.

You need to reduce the memory footprint of your application (or increase the memory of your server/workstation). Also verify your application is not leaking memory.
