Non deterministic training of LSTM with recurrent_dropout>0
@ftesser,
I was facing a different assertion error while executing the mentioned code. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/740c36db1b70b0946d60e7984a61f254/untitled1059.ipynb). Thank you! > @ftesser, I was facing a different assertion error while executing the mentioned code. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/740c36db1b70b0946d60e7984a61f254/untitled1059.ipynb). Thank you!

The assertion "assert history.history['loss'] == history_0.history['loss']" is precisely the assertion that I inserted to verify the determinism, i.e. I check that the history of the new training is equal to the history of training "0".

So the behavior I described in the issue was reproduced.
If you try to set recurrent_dropout=0 the assertion does not fails.


 @ftesser,
Thank you for opening this issue. Development of keras moved to another [repository](https://github.com/keras-team/keras/issues). 

Could you please post this issue on keras-team/keras [repo](https://github.com/keras-team/keras/issues).
To know more please refer:
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thank you!
 > @ftesser, Thank you for opening this issue. Development of keras moved to another [repository](https://github.com/keras-team/keras/issues).
> 
> Could you please post this issue on keras-team/keras [repo](https://github.com/keras-team/keras/issues). To know more please refer: https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999 Thank you!

thanks @tilakrayal (I opened the issue https://github.com/keras-team/tf-keras/issues/49) @ftesser,
Could you please feel free to move this issue to closed status, so that we can track in Keras repo. Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60170">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60170">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tf 2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 22.04.2 LTS

### Mobile device

_No response_

### Python version

3.10.6

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?
The training of a LSTM model with recurrent_droput>0 is not deterministic: setting the op_determinism configuration  (https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism) different runs of creation and train of the same model produce different model (different training history).
In contrast, deterministic training is obtained with recurrent_dropout=0 and also also with the  dropout>0.

The code below has been used to reproduce the bug.
Note that the  `det_session` function is used each time before the creation of the model.
This function contains the suggested setting from https://www.tensorflow.org/api_docs/python/tf/config/experimental/enable_op_determinism, but I have tested this code, with the same results, also with the following version in which suggested determinism sets from various issues or sites  have been added: 

```python
def det_session():
    os.environ['PYTHONHASHSEED'] = str(1)
    rn.seed(1)
    np.random.seed(1)
    tf.random.set_seed(1)
    tf.keras.utils.set_random_seed(1)
    tf.config.experimental.enable_op_determinism()
```


### Standalone code to reproduce the issue

```python
import tensorflow as tf
from tensorflow.keras.losses import mean_squared_error
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import RMSprop


def det_session():
    tf.keras.utils.set_random_seed(1)
    tf.config.experimental.enable_op_determinism()


def create_model(inputs_shape):
    # Define the model
    model = Sequential()
    model.add(tf.keras.layers.LSTM(4, input_shape=(inputs_shape[1], inputs_shape[2]), dropout=0.0, recurrent_dropout=0.1))
    # Compile the model
    model.compile(optimizer=RMSprop(1e-3), loss=mean_squared_error)
    # Give a summary
    model.summary()
    return model


if __name__ == '__main__':
    inputs = tf.random.normal([32, 10, 8])
    outputs = tf.random.normal([32, 4])

    # Create and train the model the first time
    det_session()
    model = create_model(inputs.shape)
    history_0 = model.fit(inputs, outputs, epochs=10)
    # Create and train the model more times and check the loss history
    for _ in range(10):
        # Create and train the model again and check the history loss
        det_session()
        model = create_model(inputs.shape)
        history = model.fit(inputs, outputs, epochs=10)
        assert history.history['loss'] == history_0.history['loss'], 'Losses history does not corresponds'
```


### Relevant log output

_No response_</details>