Unit tests failed on some Intel Sapphire Rapids CPUs due to wrong reference results from numpy 1.23.5
Since the issue depends mainly on Numpy, let's wait till the Numpy issue to get resolved. @sachinprasadhs It seems that numpy is not going to make new releases for 1.23.* (https://github.com/numpy/numpy/issues/24903#issuecomment-1779586475). Could we move to use 1.24.0 in TF so that we won't be impacted by the bug? @wenchenvincent , Does 1.24.0 has the fix related to this issue? > @wenchenvincent , Does 1.24.0 has the fix related to this issue?

@sachinprasadhs Yes. Numpy 1.24.0 fixed this issue. Thanks for confirming, but for `TensorFlow` to include `Numpy` 1.24.0 it has to take account of compatibility of all the dependency packages and bump the version, in the upcoming version 2.15.0 `TensorFlow` still uses `Numpy >=1.23.5` and same with the ongoing nightly version as well.
@MichaelHudgins , Is there any plan to bump the `Numpy` version? To my knowledge there was no immediate plan to bump the Numpy version.  Let me check with some others on this
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.15

### Custom code

No

### OS platform and distribution

_No response_

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

This is not an issue with using Tensorflow but an issue of false test failure when running some unit tests (e.g. //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_gpu) due to issues with numpy 1.23.5 on some Intel Sapphire Rapids CPUs, including and not limited to:
Intel(R) Xeon(R) Platinum 8480C
Intel(R) Xeon(R) Platinum 8468

I have filed an issue with numpy, which has more details on the numpy bug (https://github.com/numpy/numpy/issues/24903).

Currently, Tensorflow uses numpy 1.23.5 when running unit tests via bazel (https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/requirements_lock_3_10.txt#L287). For some tests like //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_gpu, it uses the results from numpy as reference. When running on systems with the above described Intel CPUs, the tests would falsely fail.



### Standalone code to reproduce the issue

```shell
## This is modified from tensorflow/tools/ci_build/linux/gpu/run_py3_core.sh 
set -e
set -x

N_JOBS=$(grep -c ^processor /proc/cpuinfo)

echo ""
echo "Bazel will use ${N_JOBS} concurrent job(s)."
echo ""

# Run configure.
export PYTHON_BIN_PATH=`which python3`
export CC_OPT_FLAGS='-mavx'

export TF_NEED_ROCM=0
export TF_NEED_CUDA=1
export TF_CUDA_COMPUTE_CAPABILITIES=9.0 ## Replace with the compute capability of your GPUs
export TF_CUDA_CLANG=0

yes "" | $PYTHON_BIN_PATH configure.py

# Run bazel test command. Double test timeouts to avoid flakes.
bazel test --config=cuda --test_tag_filters=-no_oss,-oss_excluded,-oss_serial,-no_gpu,-benchmark-test -k \
    --test_lang_filters=py --jobs=${N_JOBS} --test_timeout 300,450,1200,3600 \
    --build_tests_only --test_output=errors --local_test_jobs=8 --config=opt \
    --test_size_filters=small,medium \
    --run_under=//tensorflow/tools/ci_build/gpu_build:parallel_gpu_execute -- \
    //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_gpu
```


### Relevant log output

```shell
[ RUN      ] TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False
INFO:tensorflow:Running test_tensordot_scalar_axes in GRAPH mode.
I1012 05:21:54.708423 139914706186240 test_util.py:1521] Running test_tensordot_scalar_axes in GRAPH mode.
2023-10-12 05:21:54.715617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 78659 MB memory:  -> device: 0, name: NVIDIA H100 80GB HBM3, pci bus id: 0000:5d:00.0, compute capability: 9.0
INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False): 5.42s
I1012 05:22:00.128815 139914706186240 test_util.py:2574] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False): 5.42s
INFO:tensorflow:Running test_tensordot_scalar_axes in EAGER mode.
I1012 05:22:00.129978 139914706186240 test_util.py:1540] Running test_tensordot_scalar_axes in EAGER mode.
INFO:tensorflow:time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False): 5.11s
I1012 05:22:05.238928 139914706186240 test_util.py:2574] time(__main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False): 5.11s
No pending test case: __main__.TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False
======================================================================
FAIL: test_test_tensordot_scalar_axes_float64_5_5_5_False (__main__.TensordotTest) [graph_mode]
TensordotTest.test_test_tensordot_scalar_axes_float64_5_5_5_False
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1535, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 2436, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/kernel_tests/math_ops/tensordot_op_test.py", line 226, in test_tensordot_scalar_axes
    self.assertAllClose(tf_ans, np_ans, rtol=tol, atol=tol)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 1657, in decorated
    return f(*args, **kwds)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3293, in assertAllClose
    self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3249, in _assertAllCloseRecursive
    self._assertArrayLikeAllClose(
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_util.py", line 3186, in _assertArrayLikeAllClose
    np.testing.assert_allclose(
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/pypi_numpy/site-packages/numpy/testing/_private/utils.py", line 1527, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/kernel_tests/math_ops/tensordot_op_test_gpu.runfiles/pypi_numpy/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-12, atol=1e-12
Mismatched value: a is different from b.
not close where = (array([0, 0, 0, ..., 4, 4, 4]), array([0, 0, 0, ..., 4, 4, 4]), array([0, 0, 0, ..., 4, 4, 4]), array([0, 0, 0, ..., 4, 4, 4]), array([0, 0, 0, ..., 4, 4, 4]), array([0, 0, 0, ..., 4, 4, 4]), array([2, 2, 2, ..., 2, 2, 3]), array([0, 1, 2, ..., 3, 4, 0]))
not close lhs = [ 1.62769716 -0.26834483  0.47169531 ...  0.99900873  0.74603632
 -1.2264429 ]
not close rhs = [-0.02489534  1.05437826 -1.4554834  ...  0.14947069  1.62145362
 -0.23293471]
not close dif = [1.65259251 1.32272308 1.92717871 ... 0.84953805 0.8754173  0.99350819]
not close tol = [1.02489534e-12 2.05437826e-12 2.45548340e-12 ... 1.14947069e-12
 2.62145362e-12 1.23293471e-12]
dtype = float64, shape = (5, 5, 5, 5, 5, 5, 5, 5)
Mismatched elements: 315000 / 390625 (80.6%)
Max absolute difference: 4.74676846
Max relative difference: 795311.53094219
 x: array([[[[[[[[ 9.517157e-01, -4.421175e-01, -5.290973e-01,
              -9.733468e-01, -9.410919e-02],
             [-3.183883e-01, -1.084177e+00, -6.588139e-01,...
 y: array([[[[[[[[ 9.517157e-01, -4.421175e-01, -5.290973e-01,
              -9.733468e-01, -9.410919e-02],
             [-3.183883e-01, -1.084177e+00, -6.588139e-01,...
```
