Does Colab's GPU predict '0's after using the resources over a long period of time?
It's likely that some sort of training instability, for example caused by invalid data, an edge case in your data ingestion code or loss function, or exploding gradients, has caused some of your model weights to become NaN.
I'm trying to train a model and I've changed the runtime to include GPU (all my work is on Colab). I'm printing the predicted output of my model to make sure it's working properly, at first it was predicting the output just fine, however after the runtime disconnected once, it started predicting '0's and has been ever since. I've tried changing accounts, using VPNs, changing runtime types but without an accelerator it predicts the output once then proceeds with 'nan'. Am I missing some sort of restriction to Colab's GPU usages besides the 12 hour limit?