tf.math.cumsum weird behaviour 
@sermakarevich,
As mentioned the provided code was executed with the different results in tensorflow [v2.13](https://colab.research.google.com/gist/tilakrayal/8b2044bbe3672aa4c4cca41eb33ae1b9/untitled1377.ipynb). But whereas when I tried to execute the same code on tf-nightly(2.15.0-dev20230926), the **manual cumsum diff is 0.0**. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/df65231d360551bacbc068728fbcd0d1/untitled1378.ipynb). Thank you!  @tilakrayal thank you. confirmed that on latest nightly it works well. 

however it does not work on nightly of version 2.14.0*  even early versions of 2.15.0 still have the same problems
Is this some kind of a known bug and there is a commit with the fix I can check ? Unfortunately bumping tf version is not an option for me atm.  @sermakarevich,
As mentioned, it was the issue/bug with the 2.14 nightly versions which has been resolved with the latest nightly release(2.15.0-dev20230926). You can try to use the latest tf-nightly for immediate use-case and it will be made available from TF 2.15. Thank you! @tilakrayal got it. 

Is there a chance to backport the fix into 2.11 and 2.12 tf versions ?  @sermakarevich,
It's unlikely for older versions to receive any bug fixes except when we have security patches. We recommend the community to use the latest versions if the bug is resolved. Thank you! @tilakrayal roger. Can you please point me to the commit with the fix ? I tried to scroll down through commits over last three days but did not found anything that looked related to cumsum fix @sermakarevich,
I will do a deep dive into the same requirement and try to provide the information if it is available. Meanwhile, you can try to install the tf-nightly from the master branch, where it resolved the issue mentioned above. 
https://www.tensorflow.org/install
https://github.com/tensorflow/tensorflow
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61975">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61975">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.13

### Custom code

Yes

### OS platform and distribution

Ubuntu 22.04.2 LTS

### Mobile device

_No response_

### Python version

ython 3.10.12

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

12.0

### GPU model and memory

T4

### Current behavior?

tf.math.cumsum produce weird results on float32 and float64 when device is GPU:
- results are different than on CPU
- results do not match manual cumsum calculation 
- cumsum value can change when adding 0 (last two values in log output). 
- float64 has similar problems as well
- tested on multiple GPUs: T4, GTX1080Ti and tensorflow versions: 2.13, 2.8.4

### Standalone code to reproduce the issue

```shell
with tf.device('/GPU'):
  arr = tf.constant(
        [  0.  ,     0.  ,     0.  ,     0.  ,  9759.35,  9759.35,
          9759.35,  9759.35,  9759.35,  9759.35,  9762.03,  9700.78,
          9700.78,  9700.78,  9700.78,  9700.78,  9700.78,  9660.83,
          9600.46,  9600.46,  9600.46,  9600.46,  9600.46,  9600.46,
          9715.65,  9742.31,  9742.31,  9742.31,  9742.31,  9742.31,
          9742.31,  9774.32,  9750.2 ,  9750.2 ,  9750.2 ,  9750.2 ,
          9750.2 ,  9750.2 ,  9796.23,  9824.72,  9824.72,  9824.72,
          9824.72,  9824.72,  9824.72, 11737.25,  9759.23,  9759.23,
          9759.23,  9759.23,  9759.23,  9759.23,  9551.41,  9551.47,
          9551.47,  9551.47,  9551.47,  9551.47,  9551.47,  9723.25,
          9693.61,  9693.61,  9693.61,  9693.61,  9693.61,  9693.61,
          9629.34,  9658.78,  9658.78,  9658.78,  9658.78,  9658.78,
          9658.78,  9986.66, 10005.04, 10005.04, 10005.04, 10005.04,
          10005.04, 10005.04,  9977.68,  9955.32,  9955.32,  9955.32,
          9955.32,  9955.32,  9955.32,  9875.19,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.  ,
              0.  ,     0.  ,     0.  ,     0.  ,     0.  ,     0.], dtype=tf.float32)

  manual_cumsum = []
  for val in arr:
    if len(manual_cumsum) == 0:
      manual_cumsum.append(val)
    else:
       manual_cumsum.append(manual_cumsum[-1] + val)

  tf_cumsum = tf.math.cumsum(arr)

  for index in range(arr.shape[0]):
    print("arr_value:", arr[index].numpy(), "tensor - manual cumsum diff:", (tf_cumsum[index] - manual_cumsum[index]).numpy())
```
```


### Relevant log output

```shell
arr_value: 0.0 tensor - manual cumsum diff: 0.0
arr_value: 0.0 tensor - manual cumsum diff: 0.0
arr_value: 0.0 tensor - manual cumsum diff: 0.0
arr_value: 0.0 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9759.35 tensor - manual cumsum diff: 0.0
arr_value: 9762.03 tensor - manual cumsum diff: 0.0
arr_value: 9700.78 tensor - manual cumsum diff: 0.0
arr_value: 9700.78 tensor - manual cumsum diff: -0.0078125
arr_value: 9700.78 tensor - manual cumsum diff: -0.0078125
arr_value: 9700.78 tensor - manual cumsum diff: -0.0078125
arr_value: 9700.78 tensor - manual cumsum diff: -0.0078125
arr_value: 9700.78 tensor - manual cumsum diff: -0.0078125
arr_value: 9660.83 tensor - manual cumsum diff: -0.015625
arr_value: 9600.46 tensor - manual cumsum diff: -0.015625
arr_value: 9600.46 tensor - manual cumsum diff: -0.015625
arr_value: 9600.46 tensor - manual cumsum diff: 0.0
arr_value: 9600.46 tensor - manual cumsum diff: 0.0
arr_value: 9600.46 tensor - manual cumsum diff: 0.0
arr_value: 9600.46 tensor - manual cumsum diff: 0.0
arr_value: 9715.65 tensor - manual cumsum diff: 0.03125
arr_value: 9742.31 tensor - manual cumsum diff: 0.03125
arr_value: 9742.31 tensor - manual cumsum diff: 0.03125
arr_value: 9742.31 tensor - manual cumsum diff: 0.03125
arr_value: 9742.31 tensor - manual cumsum diff: 0.015625
arr_value: 9742.31 tensor - manual cumsum diff: 0.015625
arr_value: 9742.31 tensor - manual cumsum diff: 0.015625
arr_value: 9774.32 tensor - manual cumsum diff: 0.03125
arr_value: 9750.2 tensor - manual cumsum diff: 0.03125
arr_value: 9750.2 tensor - manual cumsum diff: 0.03125
arr_value: 9750.2 tensor - manual cumsum diff: 0.03125
arr_value: 9750.2 tensor - manual cumsum diff: 0.03125
arr_value: 9750.2 tensor - manual cumsum diff: 0.09375
arr_value: 9750.2 tensor - manual cumsum diff: 0.09375
arr_value: 9796.23 tensor - manual cumsum diff: 0.09375
arr_value: 9824.72 tensor - manual cumsum diff: 0.09375
arr_value: 9824.72 tensor - manual cumsum diff: 0.125
arr_value: 9824.72 tensor - manual cumsum diff: 0.125
arr_value: 9824.72 tensor - manual cumsum diff: 0.125
arr_value: 9824.72 tensor - manual cumsum diff: 0.125
arr_value: 9824.72 tensor - manual cumsum diff: 0.125
arr_value: 11737.25 tensor - manual cumsum diff: 0.125
arr_value: 9759.23 tensor - manual cumsum diff: 0.125
arr_value: 9759.23 tensor - manual cumsum diff: 0.125
arr_value: 9759.23 tensor - manual cumsum diff: 0.15625
arr_value: 9759.23 tensor - manual cumsum diff: 0.15625
arr_value: 9759.23 tensor - manual cumsum diff: 0.15625
arr_value: 9759.23 tensor - manual cumsum diff: 0.15625
arr_value: 9551.41 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9551.47 tensor - manual cumsum diff: 0.1875
arr_value: 9723.25 tensor - manual cumsum diff: 0.1875
arr_value: 9693.61 tensor - manual cumsum diff: 0.1875
arr_value: 9693.61 tensor - manual cumsum diff: 0.1875
arr_value: 9693.61 tensor - manual cumsum diff: 0.1875
arr_value: 9693.61 tensor - manual cumsum diff: 0.1875
arr_value: 9693.61 tensor - manual cumsum diff: 0.125
arr_value: 9693.61 tensor - manual cumsum diff: 0.125
arr_value: 9629.34 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.125
arr_value: 9658.78 tensor - manual cumsum diff: 0.25
arr_value: 9986.66 tensor - manual cumsum diff: 0.25
arr_value: 10005.04 tensor - manual cumsum diff: 0.25
arr_value: 10005.04 tensor - manual cumsum diff: 0.25
arr_value: 10005.04 tensor - manual cumsum diff: 0.1875
arr_value: 10005.04 tensor - manual cumsum diff: 0.1875
arr_value: 10005.04 tensor - manual cumsum diff: 0.1875
arr_value: 10005.04 tensor - manual cumsum diff: 0.1875
arr_value: 9977.68 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9955.32 tensor - manual cumsum diff: 0.1875
arr_value: 9875.19 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.1875
arr_value: 0.0 tensor - manual cumsum diff: 0.25
arr_value: 0.0 tensor - manual cumsum diff: 0.25
```
```
