Loss Analysis of Deep Learning
I'm new to deep learning, and I have built a graph convolution network. I used 5 fold cross-validations. After plotting the mean train_loss (blue) and validate_loss (orange) together, I got this baby.
MSE loss
As you can see, from the curvilinear trend of validate_loss, it seems that my networks learn few things. (I guess data? GCN frameworks? learning rate?)
Could you guys specifically help me to figure out the bugs?
I would appreciate that! If you don't get my point, please let me know.
class Scorer(nn.Module):
    &quot;&quot;&quot;
    Three conv_layers and two fc_layers with Dropout
    &quot;&quot;&quot;
    def __init__(self):
        super(Scorer, self).__init__()
        self.conv_layer1 = GraphConvNet(5, 64)
        self.conv_layer2 = GraphConvNet(64, 128)
        self.conv_layer3 = GraphConvNet(128, 256) # (I have tried delete conv_layer3)
        self.fc_layer1 = nn.Linear(256, 128)
        self.drop_layer1 = nn.Dropout(0.5)
        self.fc_layer2 = nn.Linear(128, 64)
        self.drop_layer2 = nn.Dropout(0.5)
        self.out_layer = nn.Linear(64, 1)


    def forward(self, NormLap, feat):
        h = self.conv_layer1(NormLap, feat)
        h = F.leaky_relu(h)
        h = self.conv_layer2(NormLap, h)
        h = F.leaky_relu(h)
        h = self.conv_layer3(NormLap, h)
        h = F.leaky_relu(h)
        h = self.fc_layer1(h)
        h = self.drop_layer1(h)
        h = F.leaky_relu(h)
        h = self.fc_layer2(h)
        h = self.drop_layer2(h)
        h = F.leaky_relu(h)
        h = self.out_layer(h)
        h = F.leaky_relu(h)


Below is my networks and parameters:

    # parameter setting
    learning_rate = 0.001 # (I have tried 1e-1, 1e-2)
    weight_decay = 1e-3  # (I have tried 1e-4)
    epochs = 500
    batch_size = 50  # (I have tried 30)

    model = Scorer()
    loss_func = nn.MSELoss()
    optimizer = th.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)


This is pretty much what train and validation loss is supposed to do.  Loss goes down over time; that's what an optimizer is trying to do.  train_loss keeps going down after valid_loss levels out or plateaus, indicating the model starts overfitting after epoch ~100 or so.  Whether MSE 0.3 is good or bad for your application depends entirely on the application, but yes, the optimizer is optimizing just fine.
Please have a look at this resource for how to interpret loss curves: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/
&quot;from the curvilinear trend of validate_loss, it seems that my networks learn few things&quot; - It would help if you explain (in a lot of detail :) WHY you think that, in order to get better answers. What do you expect to see other than what you see? I look at the same graph and it seems to me your network is learning to model the data and predict whatever you're trying to predict.
