Numerical accuracy
In modern computers, floating point numbers are represented using IEEE
754 standard. For more details on floating point arithmetics and IEEE
754 standard, please see Floating point arithmetic In particular, note
that floating point provides limited accuracy (about 7 decimal digits
for single precision floating point numbers, about 16 decimal digits
for double precision floating point numbers) and that floating point
addition and multiplication are not associative, so the order of the
operations affects the results. Because of this, PyTorch is not
guaranteed to produce bitwise identical results for floating point
computations that are mathematically identical. Similarly, bitwise
identical results are not guaranteed across PyTorch releases,
individual commits, or different platforms. In particular, CPU and GPU
results can be different even for bitwise-identical inputs and even
after controlling for the sources of randomness.
Batched computations or slice computations