Regression loss functions incorrect
I'm trying a basic averaging example, but the validation and loss don't match and the network fails to converge if I increase the training time. I'm training a network with 2 hidden layers, each 500 units wide on three integers from the range [0,9] with a learning rate of 1e-1, Adam, batch size of 1, and dropout for 3000 iterations and validate every 100 iterations. If the absolute difference between the label and the hypothesis is less than a threshold, here I set the threshold to 1, I consider that correct. Could someone let me know if this is an issue with the choice of loss function, something wrong with Pytorch, or something I'm doing. Below are some plots:

val_diff = 1
acc_diff = torch.FloatTensor([val_diff]).expand(self.batch_size)


Loop 100 times to during validation:

num_correct += torch.sum(torch.abs(val_h - val_y) &lt; acc_diff)


Append after each validation phase:

validate.append(num_correct / total_val)


Here are some examples of the (hypothesis, and labels):

[...(-0.7043088674545288, 6.0), (-0.15691305696964264, 2.6666667461395264),
 (0.2827358841896057, 3.3333332538604736)]


I tried six of the loss functions in the API that are typically used for regression:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.L1Loss(size_average=False)


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.L1Loss()


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.MSELoss(size_average=False)


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.MSELoss()


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.SmoothL1Loss(size_average=False)


&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;torch.nn.SmoothL1Loss()
 

Thanks.

Network code:

class Feedforward(nn.Module):
    def __init__(self, topology):
        super(Feedforward, self).__init__()
        self.input_dim     = topology['features']
        self.num_hidden    = topology['hidden_layers']
        self.hidden_dim    = topology['hidden_dim']
        self.output_dim    = topology['output_dim']
        self.input_layer   = nn.Linear(self.input_dim, self.hidden_dim)
        self.hidden_layer  = nn.Linear(self.hidden_dim, self.hidden_dim)
        self.output_layer  = nn.Linear(self.hidden_dim, self.output_dim)
        self.dropout_layer = nn.Dropout(p=0.2)


    def forward(self, x):
        batch_size = x.size()[0]
        feat_size  = x.size()[1]
        input_size = batch_size * feat_size

        self.input_layer = nn.Linear(input_size, self.hidden_dim).cuda()
        hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)

        for _ in range(self.num_hidden):
            hidden = self.dropout_layer(F.relu(self.hidden_layer(hidden)))

        output_size = batch_size * self.output_dim
        self.output_layer = nn.Linear(self.hidden_dim, output_size).cuda()
        return self.output_layer(hidden).view(output_size)


Training code:

def train(self):
    if self.cuda:
        self.network.cuda()

    dh        = DataHandler(self.data)
    # loss_fn = nn.L1Loss(size_average=False)
    # loss_fn = nn.L1Loss()
    # loss_fn = nn.SmoothL1Loss(size_average=False)
    # loss_fn = nn.SmoothL1Loss()
    # loss_fn = nn.MSELoss(size_average=False)
    loss_fn   = torch.nn.MSELoss()
    losses    = []
    validate  = []
    hypos     = []
    labels    = []
    val_size  = 100
    val_diff  = 1
    total_val = float(val_size * self.batch_size)

    for i in range(self.iterations):
        x, y = dh.get_batch(self.batch_size)
        x = self.tensor_to_Variable(x)
        y = self.tensor_to_Variable(y)

        self.optimizer.zero_grad()
        loss = loss_fn(self.network(x), y)
        loss.backward()
        self.optimizer.step()


It looks like you've misunderstood how layers in pytorch works, here are a few tips:


In your forward when you do nn.Linear(...) you are definining new layers instead of using those you pre-defined in your network __init__. Therefore, it cannot learn anything as weights are constantly reinitalized.
You shouldn't need to call .cuda() inside net.forward(...) since you've already copied the network on gpu in your train by calling self.network.cuda()
Ideally the net.forward(...) input should directly have the shape of the first layer so you won't have to modify it. Here you should have x.size() &lt;=&gt; Linear -- &gt; (Batch_size,  Features).


Your forward should look close to this:

def forward(self, x):
    x = F.relu(self.input_layer(x))
    x = F.dropout(F.relu(self.hidden_layer(x)),training=self.training)
    x = self.output_layer(x)
    return x

