Neural network always outputs the average of target values no matter the input
Also, I checked the weights and biases of a recently trained model. Almost in all layers weights look like this:
```
-4.1277455e-33  1.9695687e-33  4.1031190e-23 ... -1.5578026e-20
  -8.8301270e-33 -1.5537877e-20
```
  
And biases look like this:

 ```
 -4.0955161e-05  4.0955088e-05 -4.0955132e-05 -4.0955088e-05
 -4.0955088e-05 -4.0955132e-05 -4.0955132e-05 -4.0955088e-05
``` Hi @flexorcist ,

This seems to be a support issue not a bug. FOr support issues please post the issue at tensorflow [forum](https://discuss.tensorflow.org/) for help.

As you confirmed most of your Y_train data having range of 0.4-0.6 which means your data seems biased.Data Augmentation might help for your case.

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62804">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62804">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

2.10

### Custom code

Yes

### OS platform and distribution

Windows 11

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I have a linear regression neural network made with Keras with a numerical input and a text input converted to encoded integers. It consistently outputs the same value no matter the input. The reason I think it outputs average of the entire Y-train array, which consists of normalized values mostly between 0.4-0.6, is that my model's loss is decreasing during training to really small values like 0.02 though the output is always the same. Training accuracy always stays the same and model is able to accurately predict exactly one entry. I checked which value does it predict correctly every epoch and it is always quite similar to average. Also the output of a trained model is almost identical to the average.
Firstly I tried training the model with 3 different optimizers and 2 loss functions (6 models in total), each model produced its own output, they were different, but in terms of 1 model it was again, same no matter the input. I was searching for a solution for a long time, and I found plenty of methods that should've helped. I implemented weight regularization and initialization, early stopping and dropout (against overfitting), I scaled all the inputs and outputs, but still it didn't help. Then I found a question describing similar problem, and people were telling that the model was just not complex enough and that it was increasing its biases and decreasing its weights in the training phase, which sounded reasonable. I checked my model and actually my biases were something like 0.05 while weights were about 1-e10. So I coded in Zero bias initializers, quite strong regularizers for them, changed activation fucntions to LeakyReLU (against dead neurons), made an increased complexity model, all which resulted in the code below. I tried training it several times while tweaking learning rates from values like 0.003 to 1e-6 during each epoch and still output stays the same. Loss was higher during most of the training because of bias regularization I think, but the result was the same - model with 0.02-0.03 loss which outputs the same average value of Y_train no matter the input. Also, I consider my training data as good as it can be because the amount of time I spent searching for mistakes in it, improving preprocessing and quadruple-checking it all is insane.

### Standalone code to reproduce the issue

```shell
def regDense(a):
    return layers.Dense(a, activation=LeakyReLU(), kernel_initializer=initializers_v2.HeNormal(),
                       kernel_regularizer=l2(0.001), bias_initializer=initializers_v2.Zeros(),
                       bias_regularizer=l1_l2(0.003, 0.02))


def regLSTM(a):
    return layers.LSTM(a, kernel_regularizer=l1_l2(0.0001, 0.0003),
                kernel_initializer=initializers_v2.GlorotNormal(),
                bias_initializer=initializers_v2.Zeros(),
                return_sequences=True,
                bias_regularizer=l1_l2(0.0002, 0.002))


num_inp = keras.Input(shape=(30, 3, 1), name='nums')
text_inp = keras.Input(shape=(30, 7, 3208), name='text')

embed = layers.Embedding(vocabsize, output_dim=152)(text_inp)
tlstm1 = layers.TimeDistributed(layers.TimeDistributed(regLSTM(256)))(embed)
tdrop1 = layers.Dropout(0.2)(tlstm1)
nlstm1 = layers.TimeDistributed(regLSTM(256))(num_inp)
ndrop1 = layers.Dropout(0.2)(nlstm1)


tlstm2 = layers.TimeDistributed(layers.TimeDistributed(regLSTM(256)))(tdrop1)
tdrop2 = layers.Dropout(0.2)(tlstm2)
nlstm2 = layers.TimeDistributed(regLSTM(256))(ndrop1)
ndrop2 = layers.Dropout(0.2)(nlstm2)


tlstm3 = layers.TimeDistributed(layers.TimeDistributed(regLSTM(256)))(tdrop2)
tdrop3 = layers.Dropout(0.2)(tlstm3)
ndense1 = regDense(213)(ndrop2)
ndrop3 = layers.Dropout(0.5)(ndense1)


tdense1 = regDense(200)(tdrop3)
tdrop4 = layers.Dropout(0.5)(tdense1)
ndense2 = regDense(170)(ndrop3)
ndrop4 = layers.Dropout(0.5)(ndense2)


tdense2 = regDense(144)(tdrop4)
tpool2 = layers.MaxPooling3D((1, 1, 401), padding='same')(tdense2)
trsp1 = layers.Reshape((30, 1152, 7))(tpool2)
tpool3 = layers.MaxPooling2D((1, 3), padding='same')(trsp1)
trsp2 = layers.Reshape((2688, 30))(tpool3)
tpool4 = layers.MaxPooling1D(7, padding='same')(trsp2)
trsp3 = layers.Reshape((30, 384))(tpool4)
tdrop5 = layers.Dropout(0.5)(trsp3)
ndense3 = regDense(128)(ndrop4)
nrsp = layers.Reshape((30, 384))(ndense3)
ndrop5 = layers.Dropout(0.5)(nrsp)


concat = layers.concatenate([tdrop5, ndrop5])
prc1 = regDense(384)(concat)
pdrop1 = layers.Dropout(0.5)(prc1)
prc2 = regDense(192)(pdrop1)
pdrop2 = layers.Dropout(0.5)(prc2)
prc3 = regDense(96)(pdrop2)
pdrop3 = layers.Dropout(0.5)(prc3)
prc4 = regDense(48)(pdrop3)
pdrop4 = layers.Dropout(0.5)(prc4)
prc5 = regDense(24)(pdrop4)
pdrop5 = layers.Dropout(0.3)(prc5)
prc6 = regDense(12)(pdrop5)
lastpool = layers.GlobalAveragePooling1D()(prc6)
last = layers.Dense(1, activation='relu', kernel_regularizer=l2(0.0008),
                    kernel_initializer=initializers_v2.HeNormal(),
                    bias_initializer=initializers_v2.Zeros(),
                    bias_regularizer=l1_l2(0.003, 0.02),
                    name='output')(lastpool)
model = keras.Model(inputs=[num_inp, text_inp], outputs=last)

model.compile(optimizer='adam', loss=losses.MeanSquaredError(), metrics=['accuracy'])

model.fit(
    {"nums": X1_train, "text": X2_train},
    {"output": y_train}, validation_data=({'nums':X1_test, 'text':X2_test}, y_test),
    epochs=9, callbacks=[callbacks.EarlyStopping(monitor='val_loss', patience=1, restore_best_weights=True), callbacks.LearningRateScheduler(schedule=scheduler, verbose=1)],
    batch_size=1)
```


### Relevant log output

_No response_