Deterministic behavior gets extremely slow with CNNs in TF > 2.13
Hi @erickrf ,

I have replicated the reported problem and attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/f627915b4bb167190faa73aa179736cd/62100.ipynb) here. With Tf2.9v I can see execution time around 63 sec whereas with TF2.14v estimated time Going to 5 Hours. With tf-nightly its taken 724 seconds.

Without `tf.config.experimental.enable_op_determinism()` it seems fine with around 70 sec in all versions.

This needs to be checked. Thanks for reporting.


### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.14.0

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

Tesla T4 15 GB

### Current behavior?

When setting up deterministic behavior with `tf.config.experimental.enable_op_determinism()`, a small model with convolution layers gets prohibitively slow.

The same code runs in about 80 seconds without the flag, but over 10 minutes with it. This behavior was not observed in TF 2.8.

The model code is as follows:

```
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Embedding, GlobalMaxPooling1D, Conv1D, BatchNormalization, Activation, concatenate, Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.initializers import Constant
import numpy as np

vocab_size = 200_000
vocab_dim = 100
length1 = 50
length2 = 100
length = 100
num_samples = 100_000
conv_dim = 128
window_sizes = [2, 3, 4, 5]
num_classes = 2000

input_features = [Input(shape=(length1,), dtype="int32"), Input(shape=(length2,), dtype="int32")]
hidden_tensors = []
    
for feature in input_features:
    embeddings = Embedding(vocab_size, vocab_dim)(feature)

    for window_size in window_sizes:
        convoluted = Conv1D(conv_dim, window_size)(embeddings)
        normed = BatchNormalization()(convoluted)
        activated = Activation("relu")(normed)
        pooled = GlobalMaxPooling1D()(activated)
        hidden_tensors.append(pooled)
    
hidden = Dropout(0.5)(concatenate(hidden_tensors))
outputs = Dense(num_classes, activation='softmax')(hidden)

model = Model(inputs=input_features, outputs=outputs)
model.compile(loss="sparse_categorical_crossentropy", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

features1 = np.random.randint(0, vocab_size, size=(num_samples, length1), dtype=np.int32)
features2 = np.random.randint(0, vocab_size, size=(num_samples, length2), dtype=np.int32)
labels = np.random.randint(0, num_classes, size=num_samples)

model.fit([features1, features2], labels, epochs=1, batch_size=256)
```

And I triggered determinism with:
```
tf.keras.utils.set_random_seed(1234)
tf.config.experimental.enable_op_determinism()
```

### Standalone code to reproduce the issue

[Colab notebook](https://colab.research.google.com/drive/16B1zAX24EYLkty6j3FmbecBx-t9jNoFe?usp=sharing)


### Relevant log output

_No response_