conv2d op produces output without nan with nan inputs
Hi @pzy2000 ,

Could you help me with a minimal reproducible code snippet? Given code is lengthy and takes time. Sorry, but it is already a minimal reproducible code snippet, at least to my knowledge:)
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tf 2.10.0

### Custom code

Yes

### OS platform and distribution

Linux Ubuntu 20.04.4

### Mobile device

_No response_

### Python version

3.7.6

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

12.2

### GPU model and memory

_No response_

### Current behavior?

![image](https://github.com/tensorflow/tensorflow/assets/77827720/36360026-c000-4b66-a046-5b59908fafc9)
we expect layer 5 output to be also nan, since layer 5 input is nan
[lenet5-fashion-mnist_origin-SpecialI5-LMerg34-MDtype57.zip](https://github.com/tensorflow/tensorflow/files/14554923/lenet5-fashion-mnist_origin-SpecialI5-LMerg34-MDtype57.zip)



### Standalone code to reproduce the issue

```shell
import keras
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model


def _init_input(input_shape):
    input_shape = list(input_shape)
    input_shape[0] = 10
    input_shape = tuple(input_shape)
    input = np.random.rand(*input_shape)
    return input


class CustomCastLayer(keras.layers.Layer):
    def __init__(self, target_dtype, **kwargs):
        self.target_dtype = target_dtype
        super().__init__(**kwargs)

    def build(self, input_shape):
        """
        Do Nothing
        """
        super().build(input_shape)

    def call(self, inputs, **kwargs):
        output = tf.cast(inputs, self.target_dtype)
        return output

    def get_config(self):
        config = super().get_config()
        config.update({"target_dtype": self.target_dtype})
        return config

    def compute_output_shape(self, input_shape):
        """
        Do Nothing
        """
        return input_shape


class CustomPadLayer(keras.layers.Layer):
    def __init__(self, padding=[[0, 0]], constant_values=0, **kwargs):
        self.padding = padding  # add [0,0] to padding so we will not change the shape of batch dimension
        self.constant_values = constant_values
        super().__init__(**kwargs)

    def build(self, input_shape):
        super().build(input_shape)

    def call(self, inputs, **kwargs):
        output = tf.pad(inputs, paddings=tf.constant([[0, 0]] + self.padding), mode="CONSTANT",
                        constant_values=self.constant_values)
        return output

    def get_config(self):
        config = super().get_config()
        config.update({"padding": self.padding, "constant_values": self.constant_values})
        return config

    def compute_output_shape(self, input_shape):
        """
        Formula to calculate the output shape
        Suppose the input_shape is (None, N, W, C), the `paddings` is [[a, b], [c, d]].
        The output_shape is (None, N+a+b, W+c+d, C).
        """
        input_shape_list = list(input_shape)
        padding = [[0, 0]] + self.padding
        assert len(input_shape_list) == len(padding)  # Two dimensions should match.
        output_shape = [None]
        for i, pad in zip(input_shape_list[1:], padding[1:]):
            output_shape.append(i + pad[0] + pad[1])
        return tuple(output_shape)


class CustomCropLayer(keras.layers.Layer):
    def __init__(self, cropping, **kwargs):
        self.cropping = cropping
        super().__init__(**kwargs)

    def build(self, input_shape):
        super().build(input_shape)

    def call(self, inputs, **kwargs):
        input_shape = inputs.shape.as_list()
        indices = [slice(None)]
        cropping = [[0, 0]] + self.cropping  # add [0,0] to padding so we will not change the shape of batch dimension
        for shape, crop in zip(input_shape[1:], cropping[1:]):
            indices.append(slice(0 + crop[0], shape - crop[1]))
        return inputs[indices]

    def get_config(self):
        config = super().get_config()
        config.update({"cropping": self.cropping})
        return config

    def compute_output_shape(self, input_shape):
        """
        Formula to calculate the output shape
        Suppose the input_shape is (None, N, W, C), the `cropping` is [[a, b], [c, d]].
        The output_shape is (None, N-a-b, W-c-d, C).
        """
        input_shape_list = list(input_shape)
        cropping = [[0, 0]] + self.cropping
        assert len(input_shape_list) == len(cropping)  # Two dimensions should match.
        output_shape = [None]
        for i, crop in zip(input_shape[1:], cropping[1:]):
            output_shape.append(i - crop[0] - crop[1])
        return tuple(output_shape)


class CustomExpandLayer(keras.layers.Layer):
    def __init__(self, axis=1, **kwargs):
        self.axis = axis
        super().__init__(**kwargs)

    def build(self, input_shape):
        super().build(input_shape)

    def call(self, inputs, **kwargs):
        return tf.expand_dims(inputs, self.axis)

    def get_config(self):
        config = super().get_config()
        config.update({"axis": self.axis})
        return config

    def compute_output_shape(self, input_shape):
        """
        Formula to calculate the output shape
        Suppose the input_shape is [None, N, W, C]:
        axis=0:
            output_shape: [1, None, N, W, C]
        axis=1 (default):
            output_shape: [None, 1, N, W, C]
        axis=2:
            output_shape: [None, N, 1, W, C]
        axis=3:
            output_shape: [None, N, W, 1, C]
        axis=4:
            output_shape: [None, N, W, C, 1]
        axis=5:
            raise Exception
        """
        input_shape_list = list(input_shape)
        if self.axis > len(input_shape_list):
            raise ValueError(f"axis {self.axis} should be smaller than input_shape + 1: {len(input_shape_list) + 1}")
        output_shape = input_shape_list[0:self.axis] + [1] + input_shape_list[self.axis:]
        return tuple(output_shape)  # we should use tuple!!! not list !!!


class CustomDropDimLayer(keras.layers.Layer):
    def __init__(self, axis=1, **kwargs):
        self.axis = axis
        super().__init__(**kwargs)

    def build(self, input_shape):
        super().build(input_shape)

    def call(self, inputs, **kwargs):
        """
        Something magic to automatically generate indices for array slicing.
        To determine a specific axis, we can use slice(None) to replace `:`
        """
        dim = len(inputs.shape)
        if self.axis > dim - 1 or self.axis < 1:
            raise ValueError(f"axis: {self.axis} should be within the range: [1, {dim - 1}] for {dim}D tensor")
        indices = [slice(None) for i in range(dim)]
        indices[self.axis] = 0
        return inputs[indices]

    def get_config(self):
        config = super().get_config()
        config.update({"axis": self.axis})
        return config

    def compute_output_shape(self, input_shape):
        """
        Formula to calculate the output shape
        Suppose the input_shape is [None, N, W, C]:
        axis=0:  # Although it is feasible, we don't allow this to happen
            Raise Exception
        axis=1 (default):
            output_shape: [None, W, C]
        axis=2:
            output_shape: [None, N, C]
        axis=3:
            output_shape: [None, N, W]
        axis=4:
            Raise Exception
        """
        input_shape_list = list(input_shape)
        output_shape = input_shape_list[0:self.axis] + input_shape_list[self.axis + 1:]
        return tuple(output_shape)


def custom_objects(mode="custom"):
    def no_activation(x):
        return x

    def leakyrelu(x):
        import keras.backend as K
        return K.relu(x, alpha=0.01)

    # objects = {}
    objects = {'no_activation': no_activation, 'leakyrelu': leakyrelu}
    if mode == "custom":
        objects['CustomPadLayer'] = CustomPadLayer
        objects['CustomCropLayer'] = CustomCropLayer
        objects['CustomDropDimLayer'] = CustomDropDimLayer
        objects['CustomExpandLayer'] = CustomExpandLayer
        objects['CustomCastLayer'] = CustomCastLayer

    return objects


# 加载预训练模型
model = keras.models.load_model(
    "lenet5-fashion-mnist_origin-SpecialI5-LMerg34-MDtype57.h5",
    custom_objects=custom_objects())

input_layer = model.input
layer_outputs = [layer.output for layer in model.layers]
print(layer_outputs)
print("===================================")

new_model = Model(inputs=input_layer, outputs=layer_outputs)
# 创建一个具有特定形状的随机tensor，例如形状为(1, 3, 32, 32)
with tf.device('cpu'):
    input_tensor = _init_input(model.input_shape)
    # Use this model to predict
    all_layer_outputs = new_model.predict(input_tensor)
    # print(all_layer_outputs)
    # print(type(all_layer_outputs))
    # print(all_layer_outputs[0])
    print("===================================")
    for i, layer_output in enumerate(all_layer_outputs):
        # print(f"Output shape of layer {i}: {layer_output.shape}")
        print(f"Name of layer {i}: {model.layers[i].name}")
        print(f"Output if nan of layer {i}: {np.isnan(layer_output).any()}")
        print()
```


### Relevant log output

```shell
Name of layer 0: conv2d_1_input
Output if nan of layer 0: False

Name of layer 1: conv2d_1_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 1: False

Name of layer 2: max_pooling2d_1_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 2: False

Name of layer 3: lambda_copy_LMerg_copy_MDtype
Output if nan of layer 3: True

Name of layer 4: dropout_1_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 4: True

Name of layer 5: conv2d_2_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 5: False

Name of layer 6: max_pooling2d_2_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 6: False

Name of layer 7: dropout_2_copy_SpecialI_copy_LMerg_merge1_copy_MDtype
Output if nan of layer 7: False

Name of layer 8: subtract_copy_ML_copy_MDtype
Output if nan of layer 8: False

Name of layer 9: custom_cast_layer
Output if nan of layer 9: False

Name of layer 10: flatten_1_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 10: False

Name of layer 11: custom_cast_layer_1
Output if nan of layer 11: False

Name of layer 12: dense_1_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 12: False

Name of layer 13: dense_2_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 13: False

Name of layer 14: dense_3_copy_SpecialI_copy_LMerg_copy_MDtype
Output if nan of layer 14: False

Name of layer 15: dense_insert
Output if nan of layer 15: False
```
