Why does the output of the neural network is the same for all samples
I'm using Pytorch to make a regression model (neural network). Just train and test it on the same sample (because i'm just learning how to build a neural network).
The data is the fish market dataset from kaggle:
https://www.kaggle.com/aungpyaeap/fish-market
Small sample from the training Tensor:
tensor([[ 0.0000, 23.2000, 25.4000, 30.0000, 11.5200,  4.0200],
        [ 0.0000, 24.0000, 26.3000, 31.2000, 12.4800,  4.3056],
        [ 0.0000, 23.9000, 26.5000, 31.1000, 12.3778,  4.6961],
        [ 0.0000, 26.3000, 29.0000, 33.5000, 12.7300,  4.4555],
        [ 0.0000, 26.5000, 29.0000, 34.0000, 12.4440,  5.1340],
        [ 0.0000, 26.8000, 29.7000, 34.7000, 13.6024,  4.9274],
        [ 0.0000, 26.8000, 29.7000, 34.5000, 14.1795,  5.2785],
        [ 0.0000, 27.6000, 30.0000, 35.0000, 12.6700,  4.6900],
        [ 0.0000, 27.6000, 30.0000, 35.1000, 14.0049,  4.8438],
        [ 0.0000, 28.5000, 30.7000, 36.2000, 14.2266,  4.9594],
        [ 0.0000, 28.4000, 31.0000, 36.2000, 14.2628,  5.1042],
        [ 0.0000, 28.7000, 31.0000, 36.2000, 14.3714,  4.8146],
        [ 0.0000, 29.1000, 31.5000, 36.4000, 13.7592,  4.3680],

Small Sample from the target tensor:
tensor([[ 242.0000],
        [ 290.0000],
        [ 340.0000],
        [ 363.0000],
        [ 430.0000],
        [ 450.0000],
        [ 500.0000],
        [ 390.0000],
        [ 450.0000],
        [ 500.0000],
        [ 475.0000],
        [ 500.0000],
        [ 500.0000],
        [ 340.0000],
        [ 600.0000],
        [ 600.0000],
        [ 700.0000],
        [ 700.0000],

The neural network:
class NeuralNetFish(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(6, 10)
        self.tanh1 = nn.Tanh()
        self.fc2 = nn.Linear(10, 10)
        self.tanh2 = nn.Tanh()
        self.fc3 = nn.Linear(10, 1)
    
    def forward(self, x):
        out = self.fc1(x)
        out = self.tanh1(out)
        out = self.fc2(out)
        out = self.tanh2(out)
        out = self.fc3(out)
        return out


model = NeuralNetFish()
criteron = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.02)

for i in range(10000):
    output = model(fish_tensor)
    loss = criteron(output, weight_tensor)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if i % 1000 == 0:
        print(&quot;Epoch:&quot;, i, &quot; ---  Loss:&quot;, loss.item())

OUTPUT:
Epoch: 0  ---  Loss: 286071.78125
Epoch: 1000  ---  Loss: 127342.515625
Epoch: 2000  ---  Loss: 127342.515625
Epoch: 3000  ---  Loss: 127342.515625
Epoch: 4000  ---  Loss: 127342.515625
Epoch: 5000  ---  Loss: 127342.515625
Epoch: 6000  ---  Loss: 127342.515625
Epoch: 7000  ---  Loss: 127342.515625
Epoch: 8000  ---  Loss: 127342.515625
Epoch: 9000  ---  Loss: 127342.515625

And i get the same output for all the samples.

Scale your inputs.
Your inputs are roughly in range [0, 40], you take linear combinations of these numbers, and then apply the tanh function. Most likely your outputs will be at the &quot;saturated&quot; region of the tanh thus the gradients are roughly zero. Your neurons are dead.
To overcome this issue:

Scale your inputs: compute the mean and std over the training set (mean and std are 6-vectors in your case). Subtract the mean and divide by the std to have your inputs distributed roughly as N(0, 1).
Replace the tanh activations with nn.PReLU.
You should tune your learning-rate to be in accordance with the batch size (in your case batch size = entire dataset).

