incorrect results (and their gradients may be infinite) when their
inputs have singular values that are close to each other. This is
because the algorithms used to compute these decompositions struggle
to converge for these inputs.
Running the computation in "float64" (as NumPy does by default) often
helps, but it does not solve these issues in all cases. Analyzing the
spectrum of the inputs via "torch.linalg.svdvals()" or their condition
number via "torch.linalg.cond()" may help to detect these issues.
TensorFloat-32(TF32) on Nvidia Ampere devices
On Ampere Nvidia GPUs, PyTorch can use TensorFloat32 (TF32) to speed
up mathematically intensive operations, in particular matrix
multiplications and convolutions. When an operation is performed using
TF32 tensor cores, only the first 10 bits of the input mantissa are
read. This may reduce accuracy and produce surprising results (e.g.,
multiplying a matrix by the identity matrix may produce results that