Validation loss differs on GPU vs CPU I am consistently seeing higher validation loss when I train &amp; evaluate a model on AWS GPU vs local CPU.

I am using the exact same train/eval datasets and the exact same Tensorflow code and configuration for both runs. Has anyone else experienced this before and know how to explain?

See screenshots below comparing GPU vs CPU loss for equivalent steps.
Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.