tf.keras.layers.ConvLSTM2D throws error when backprop
@sachinprasadhs 
I was able to replicate the reported behaviour with TF2.13, TF2.14 and tf-nightly. Kindly find the [gist](https://colab.research.google.com/gist/tilakrayal/264de69376578a37cec37af078d8cee2/untitled1522.ipynb) for reference.

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf 2.14.0

### Custom code

Yes

### OS platform and distribution

windows colab

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

API crashes in backprop.

### Standalone code to reproduce the issue

```shell
filters = 2
kernel_size_0 = 4
kernel_size_1 = 2
kernel_size = [kernel_size_0,kernel_size_1,]
strides_0 = 1
strides_1 = 1
strides = [strides_0,strides_1,]
padding = "valid"
data_format = "channels_first"
dilation_rate_0 = 1
dilation_rate_1 = 1
dilation_rate = [dilation_rate_0,dilation_rate_1,]
activation = "tanh"
recurrent_activation = "hard_sigmoid"
use_bias = False
kernel_initializer = None
recurrent_initializer = None
bias_initializer = "zeros"
unit_forget_bias = True
kernel_regularizer = None
recurrent_regularizer = None
bias_regularizer = None
activity_regularizer = None
kernel_constraint = None
recurrent_constraint = None
bias_constraint = None
return_sequences = False
return_state = False
go_backwards = True
stateful = False
dropout = 0.0
recurrent_dropout = 0.0
__input___0_tensor = tf.random.uniform([2, 2, 2, 5, 1], minval=1.7518786460769893, maxval=2.8945805380363145, dtype=tf.float32)
__input___0 = tf.identity(__input___0_tensor)
ConvLSTM2D_class = tf.keras.layers.ConvLSTM2D(filters, kernel_size, strides=strides, padding=padding, data_format=data_format, dilation_rate=dilation_rate, activation=activation, recurrent_activation=recurrent_activation, use_bias=use_bias, kernel_initializer=kernel_initializer, recurrent_initializer=recurrent_initializer, bias_initializer=bias_initializer, unit_forget_bias=unit_forget_bias, kernel_regularizer=kernel_regularizer, recurrent_regularizer=recurrent_regularizer, bias_regularizer=bias_regularizer, activity_regularizer=activity_regularizer, kernel_constraint=kernel_constraint, recurrent_constraint=recurrent_constraint, bias_constraint=bias_constraint, return_sequences=return_sequences, return_state=return_state, go_backwards=go_backwards, stateful=stateful, dropout=dropout, recurrent_dropout=recurrent_dropout)

layer = ConvLSTM2D_class
inputs = __input___0

with tf.GradientTape() as g:
    g.watch(inputs)
    res = layer(inputs)
print(res.shape)
grad = g.jacobian(res, inputs)  # Error
```


### Relevant log output

```shell
(2, 2, 2, 0)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-75a544b673ad> in <cell line: 57>()
     55     res = layer(inputs)
     56 print(res.shape)
---> 57 grad = g.jacobian(res, inputs)  # Error
     58 # print(grad)

1 frames
/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py in set_shape(self, shape)
    534   def set_shape(self, shape) -> None:
    535     if not self.shape.is_compatible_with(shape):
--> 536       raise ValueError(f"Tensor's shape {self.shape} is not compatible "
    537                        f"with supplied shape {shape}.")
    538 

ValueError: Tensor's shape (2, 2, 2, 0, 1, 2, 2, 5, 1) is not compatible with supplied shape (2, 2, 2, 0, 2, 2, 2, 5, 1).
```
