Tersorflow deterministic training not working for different workflows even when GPU and envs are same
Hi @ N-Harish,

In Tensorflow there are 2 random seeds one is at global level set by `tf.random.set_seed(seed)` and another at **Op** level for eg. say` tf.random.uniform(..., seed=seed2)`. Together with both these seeds we can able to get deterministic results. If we don't set seed at Op level each process will generate its own random seed which may result in difference of outputs.

I am not sure whether your code has both seeds set for all workflows. Please confirm whether your code has any other Op which needs to set the seed? Please check and confirm the problem with reproducible code. 

Thanks! We are using tf dataset map method and shuffle method so do we need to set separate seed for that as well ? @N-Harish ,

If you are using tf.data.Dataset.[shuffle](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle) method it has `seed` argument for which you can pass a constant value to enable determinism.

Also for [map](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map) method you may need to pass `deterministic = True` to preserve the order.But this will have some performance cost.Please go through attached documentation for more details.

Thanks! we have added seed for randomflip, shuffle and set deterministic=true for map but still there are some differences in the models. Also tf dataset has num_parallel_calls set eqaul to tf.data.AUTOTUNE Hi @N-Harish ,

Could you please confirm the difference in the results in terms of percentages rather than absolute difference? Also TF2.5v is quiet older and not having active support  now, Could you able to test with latest versions and confirm the results. 

Also please submit some the code snippet that can reproduce the issue so that we may check it on different environments.
It's hard to get at a solution without reproducible code snippet.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61831">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61831">No</a>

### System information

- Ubuntu 20.04
- Tensorflow version :-  2.5.0
- python version :- 3.7
- GPU model and memory :- Nvidia RTX 4090, 24 GB GPU memory


### Describe the problem
We recently started setting environmental variable for enabling deterministic training on tensorflow version 2.5. We have 5 to 6 models, most of which are regression models and one of them is a classification model. The following seed values were set :- 
   ```
    os.environ["PYTHONHASHSEED"] = str(seed)
    
    import tensorflow as tf
    import random
    import numpy as np
    
    # set seed for random, numpy and tensorflow
    random.seed(seed)
    np.random.seed(seed)
    tf.random.set_seed(seed)

    # set env for determinism
    os.environ["TF_DETERMINISTIC_OPS"] = "1"
    os.environ["TF_CUDNN_DETERMINISTIC"] = "1"
   
    # to ensure determinism set threads to one
    tf.config.threading.set_inter_op_parallelism_threads(1)
    tf.config.threading.set_intra_op_parallelism_threads(1)
``````

 We created multiple workflows by keeping the same GPU version and all of the above environmental variables and seed values. But, for different workflows we get different results even though the initial weights for all models on both workflows are same. On the other hand, on training in the same workflow, we get similar results. kindly let us know what other seed values do we need to set and let us know a solution so that we can train the models in a deterministic way


### Source code / logs

Here is the code to compare initialised weights of model in different workflow but when performing inference after training for 70 epochs the results vary a lot

```
import tensorflow as tf
import numpy as np


tf_det_ini = tf.keras.models.load_model("./tf_det/nobrainer_initial.h5")
wb_diff_ini = tf.keras.models.load_model("./wb_diff/nobrainer_initial.h5")

diff_arr = []

cnt = 0
for tf_m, wb in zip(tf_det_ini.layers[1:], wb_diff_ini.layers[1:]):
    if tf_m.name == 'MobilenetV3large':
        tf_m_mob = tf_m.layers
        wb_mob = wb.layers

        for tf_m_mobl, wb_mob_l in zip(tf_m_mob[2:], wb_mob[2:]):
            try:
                diff = np.unique(tf_m_mobl.weights[0].numpy() - wb_mob_l.weights[0].numpy())
                diff_arr.extend(diff)
            except:
                continue
            # print(tf_m_mobl.weights[0].numpy())

    try:
        print(tf_m.name)
        diff = tf_m.weights[0].numpy() - wb.weights[0].numpy()
        diff_arr.extend(np.unique(diff))
    except:
        continue

print(max(diff_arr))
print(min(diff_arr))
```

output :- 

```
MobilenetV3large
global_average_pooling2d
extradata
concatenate
dense
0.0
0.0
```