How do I diagnose my Python program being killed due to an out-of-memory error?
Here are some ways to check for issues but I fear they may not be a complete or even the correct solution without more peeking or poking:

In another shell after you start your python process (or if you screen / tmux it):


df /
top -l 1 -S | head -12
vm_stat and vm_stat 600
sudo du -sm /var/vm/*


Once you have a good baseline, you can watch things over time to see how the neural net is behaving each hour for a while. If you think things are about stop, you can run sysdiagnose python (or use the process # if you have more than one python process running). Also, if you don't want to wait the day for things to bulk up, you can inflict memory_pressure on the system before or after starting the neural net in python. See this answer for how to monitor the Activity Monitor when you run this process:


https://apple.stackexchange.com/a/107181/5472