AND-gate with Pytorch
I'm new to PyTorch and deep learning generally.
The code I wrote can be seen longer down.
I'm trying to learn the simple 'And' problem, which is linearby separable.
The problem is, that I'm getting poor results. Only around 2/10 times it gets to the correct answer.
Sometimes the loss.item() values is stuck at 0.250.

Just to clear up    


Why does it only work 2/10 times?


.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.autograd as autog

data_x = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
data_y = np.array([[0, 0, 0, 1]]).T
data_x = autog.Variable(torch.FloatTensor(data_x))
data_y = autog.Variable(torch.FloatTensor(data_y), requires_grad=False)

in_dim = 2
out_dim = 1
epochs = 15000
epoch_print = epochs / 5
l_rate = 0.001

class NeuralNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(NeuralNet, self).__init__()
        self.lin1 = nn.Linear(input_size, output_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = x
        out = self.lin1(out)
        out = self.relu(out)
        return out

model = NeuralNet(in_dim, out_dim)
criterion = nn.L1Loss()
optimizer = optim.Adam(model.parameters(), lr=l_rate)

for epoch in range(epochs):
    pred = model(data_x)
    loss = criterion(pred, data_y)
    loss.backward()
    optimizer.step()
    if (epoch + 1) % epoch_print == 0:
        print("Epoch %d  Loss %.3f" %(epoch + 1, loss.item()))


for x, y in zip(data_x, data_y):
    pred = model(x)
    print("Input", list(map(int, x)), "Pred", int(pred), "Output", int(y))


1. Using zero_grad with optimizer

You are not using optimizer.zero_grad() to clear the gradient. Your learning loop should look like this:

for epoch in range(epochs):
    optimizer.zero_grad()
    pred = model(data_x)
    loss = criterion(pred, data_y)
    loss.backward()
    optimizer.step()
    if (epoch + 1) % epoch_print == 0:
        print("Epoch %d  Loss %.3f" %(epoch + 1, loss.item()))


In this particular case it will not have any detrimental effect, the gradient is accumulating, but as you have the same dataset looped over and over it makes barely any difference (you should get into this habit though, as you will use it throughout your deep learning journey).

2. Cost Function

You are using Mean Absolute Error which is regression loss function, not a classification one (what you do is binary classification).

Accordingly, you should use BCELoss and sigmoid activation or (I prefer it that way), return logits from the network and use BCEWithLogitsLoss, both of them calculate binary cross entropy (simplified version of cross-entropy).

See below:

class NeuralNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(NeuralNet, self).__init__()
        self.lin1 = nn.Linear(input_size, output_size)

    def forward(self, x):
        # You may want to use torch.nn.functional.sigmoid activation
        return self.lin1(x)

...
# Change your criterion to nn.BCELoss() if using sigmoid
criterion = nn.BCEWithLogitsLoss()
...


3. Predictions

If you used the logits version, classifier learns to assign negative values to 0 label and positive to indicate 1. Your display function has to be modified to incorporate this knowledge:

for x, y in zip(data_x, data_y):
    pred = model(x)
    # See int(pred &gt; 0), that's the only change
    print("Input", list(map(int, x)), "Pred", int(pred &gt; 0), "Output", int(y))


This step does not apply if your forward applies sigmoid to the output. Oh, and it's better to use torch.round instead of casting to int.
