Tensorflow Hanging on Google Compute Engine Using nohup
Try to use absolute paths in your execution instead of relative paths
I am trying to run a TensorFlow model that I estimate will take roughly 11 hours. As such, I would like to use nohup so I can exit my terminal keep the process running. I use the following command to do so:
nohup python3 trainModel.py > log.txt &
My model appears to be running as normal, but gets hung up with the last message outputted being:
2020-01-22 19:06:24.669183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
It is normal for my model to output this, however when I am not using nohup, the rest of the code still executes. What do I need to do to have this command run as it does when I am not using nohup?