Do numerical inaccuracies play any role in training neural networks?
Are there publications which mention numerical problems in neural network optimization?


Of course, there has been a lot of research on vanishing gradients, which is entirely a numerical problem. There is also a fair amount of research of training with low precision operations, but the result is surprising: reduced floating point precision doesn't seem to affect neural network training. This means that precision loss is pretty unlikely to be the cause of this phenomenon.

Still, the environment can affect the computation (as suggested in the comments):


Most obviously, random-number generator. Use a seed in your script and try to make a reproducible result at least on a single machine. After that you can compute the summary of activations and gradients (e.g. via tf.summary in tensorflow) and compare the tensors across the machines. Clearly, basic operations such as matrix multiplication or piece-wise exponent should give very close if not identical result, no matter what hardware is used. You should be able to see if the tensors diverge immediately (which means there is another source of randomness) or gradually.
python interpreter, cuda, cudnn driver and key libraries versions (numpy, tensorflow, etc). You can go as far the same linux kernel and libc version, but I think you should expect reproducibility even without it. cudnn version is important, because the convolution is likely to be natively optimized. tensorflow is also very important, because Google rewrites the core all the time.
Environment variables, e.g. PATH, LD_LIBRARY_PATH, etc., linux configuration parameters, e.g. limits.conf, permissions.


Extra precautions:


Explicitly specify the type of each variable, don't rely on defaults.
Double check that the training / test data is identical and is read in the same order. 
Does your computation use any pre-trained models? Any networking involved? Check that as well.


I would suspect hardware differences the last: it's an extraordinary case if the high-level computation without explicit concurrency leads to different results (except floating point precision differencies) depending on the number of cores or cache size.