SimpleRNN doesn't appear to use its recurrent machinery
Hi, thanks for taking the time to providing complete working code to reproduce the issue.

The problem here isn't the RNN, it's the shape of your input data:

https://colab.research.google.com/drive/1pkFuU_nMyJnVHF3TvCSQ49e-IORFu3PQ?authuser=1#scrollTo=lIYdn1woOS1n

Keras, expects RNN inputs to have shape `(batch, time, features)`, your data with shape `(10k, 1, 1)`

It only ever sees one time step, so it never has a chance to learn how to handle the previous time-step.

Try this tool for a fix: https://www.tensorflow.org/api_docs/python/tf/keras/utils/timeseries_dataset_from_array Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59757">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59757">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.7.0

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

Python 3.9.15

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
code generating the issue is reported here:
https://stackoverflow.com/questions/75496612/keras-simplernncell-appears-to-fail-to-distribute-learning-among-all-its-weights.

basically I expect that all weights in my model change when changing the epochs number, while what I observe is that the state weight (in my simple code the state matrix reduces to only one weight) stays stuck to its initialization.
```


### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow_addons.layers import ESN
from tensorflow_addons.rnn import ESNCell
from tensorflow.keras.layers import RNN
from tensorflow.keras.layers import SimpleRNN, SimpleRNNCell
from sklearn.preprocessing import MinMaxScaler
from tensorflow import random as rnd


# Fix the seed
rnd.set_seed(0)


# The data can be downloaded from https://mantas.info/wp/wp-content/uploads/simple_esn/MackeyGlass_t17.txt
data = np.loadtxt('MackeyGlass_t17.txt')

# Normalize
scaler = MinMaxScaler(feature_range=(0, 1))
scaled = scaler.fit_transform(data.reshape(-1, 1))

# Split Dataset in Train and Test
train, test = scaled[0:-100], scaled[-100:]

# Split into input and output 
train_X, train_y = train[:-1], train[1:]
test_X, test_y = test[:-1], test[1:] 

# Reshaping 
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))

# Batch and epochs
batch_size = 20
epochs = 3


# Design and run the model
model = Sequential()
model.add(RNN(SimpleRNNCell(1)))
#model.add(ESN(units = 12, spectral_radius = spectral_radius, leaky=0.75, connectivity = 0.9)) # this line works exactly like the next one
#model.add(RNN(ESNCell(12, spectral_radius = spectral_radius, leaky=0.75, connectivity = 0.9)))
model.add(Dense(train_y.shape[1]))
model.compile(loss='huber', optimizer='adam')

model.fit(train_X, train_y, epochs=epochs, batch_size=batch_size, validation_data=(test_X, test_y), verbose=0, shuffle=False)

# Print the weights of the dense layer
#print(model.layers[1].get_weights())
#for layer in model.layers: print(layer.get_config(), layer.get_weights())
for layer in model.layers: print(layer.get_weights())
```


### Relevant log output

```shell
f I run this code with 2 epochs I receive the following output:

    [array([[-0.8942287]], dtype=float32), array([[1.]], dtype=float32), array([0.05435111], dtype=float32)] [array([[-1.272426]], dtype=float32), array([0.04711587], dtype=float32)]

If I run this code with 3 epochs I receive the following output:

    [array([[-0.89395165]], dtype=float32), array([[1.]], dtype=float32), array([0.06734365], dtype=float32)] [array([[-1.2927996]], dtype=float32), array([0.05247825], dtype=float32)]

so the state weight (array([[1.]]) is the only one not changing.
```
</details>