Instruction: how to fix capsule training problem for a single class of MNIST dataset?

Input:
I am training a Capsule Network with both encoder and decoder part. It works perfectly fine with all the classes (10 classes) of the MNIST data set. But when I am extracting a single class say (class 0 or class 5) and then training the capsule network, the reconstruction of the image is very poor.

Where do I need to change the network setting, or do I have an error in my data preparation?

I tried:

I changed the total class from 10 (for ten digits to 1 for 1 digit and even for 2 for 2 digits).
When I am using the default MNIST dataset, I am getting no error or tensor size, but when I am extracting a particular class and then passing it into the network, I am facing issues like a) Dimensional Issues b) Float tensor warning.

I fixed these things but manually adding a dimension and converting the data to data.float().cuda() tensor. I did this for both the case i.e when I am using the 10 Digit Capsules and when I am using the 1 Digit Capsules for training a single class digit.

But after this, the network is running fine, but I am getting really blurred and poor reconstructions. While when I am training the whole MNIST dataset without extracting any class and passing it to the network, it doesn't throw any error and the reconstruction works really fine.

I would love to share the more detail and other parts of the code -

import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.optim import Adam
from torchvision import datasets, transforms

USE_CUDA = True

### **Here we prepare the data for the complete 10 class digit training**###
class Mnist:
    def __init__(self, batch_size):
        dataset_transform = transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])

        train_dataset = datasets.MNIST('../data', train=True, download=True, transform=dataset_transform)
        test_dataset = datasets.MNIST('../data', train=False, download=True, transform=dataset_transform)

        self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
        self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

## **Here is my code for extracting a single class digit extraction**##
class Mnist:
    def __init__(self,batch_size):

        dataset_transform = transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Normalize((0.1307,), (0.3081,))
        ])

        train_mnist = datasets.MNIST("../data", train=True)
        test_mnist = datasets.MNIST("../data", train= False)
        train_image, train_label = train_mnist.train_data, train_mnist.train_labels
        test_image, test_label = test_mnist.test_data, test_mnist.test_labels

        train_0, test_0 = [train_image[key] for (key, label) in enumerate(train_label) if int(label) == 5],[test_image[key] for (key, label) in enumerate(test_label) if int(label) == 5]
        train_label_0, test_label_0 = zero__train = [train_label[key] for (key, label) in enumerate(train_label) if int(label) == 5],[test_label[key] for (key, label) in enumerate(test_label) if int(label) == 5]

        train_dataset = tuple(zip(train_0, train_label_0))
        test_dataset = tuple(zip(test_0, test_label_0))

        self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
        self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

# Here is the main code for the capsule training.

''' The below code is used for training the 1 class but using the 10 Digit capsules
'''
class ConvLayer(nn.Module):
    def __init__(self, in_channels=1, out_channels=256, kernel_size=9):
        super(ConvLayer, self).__init__()

        self.conv = nn.Conv2d(in_channels=in_channels,
                               out_channels=out_channels,
                               kernel_size=kernel_size,
                               stride=1
                             )

    def forward(self, x):
        return F.relu(self.conv(x))
class PrimaryCaps(nn.Module):
    def __init__(self, num_capsules=8, in_channels=256, out_channels=32, kernel_size=9):
        super(PrimaryCaps, self).__init__()

        self.capsules = nn.ModuleList([
            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=2, padding=0)
                          for _ in range(num_capsules)])

    def forward(self, x):
        u = [capsule(x) for capsule in self.capsules]
        u = torch.stack(u, dim=1)
        u = u.view(x.size(0), 32 * 6 * 6, -1)
        return self.squash(u)

    def squash(self, input_tensor):
        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)
        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))
        return output_tensor
class DigitCaps(nn.Module):
    def __init__(self, num_capsules=10, num_routes=32 * 6 * 6, in_channels=8, out_channels=16):
        super(DigitCaps, self).__init__()

        self.in_channels = in_channels
        self.num_routes = num_routes
        self.num_capsules = num_capsules

        self.W = nn.Parameter(torch.randn(1, num_routes, num_capsules, out_channels, in_channels))

    def forward(self, x):
        batch_size = x.size(0)
        x = torch.stack([x] * self.num_capsules, dim=2).unsqueeze(4)
#         print(f"x at epoch {epoch} is equal to : {x}")

        W = torch.cat([self.W] * batch_size, dim=0)
#         print(f"W at epoch {epoch} is equal to : {W}")
        u_hat = torch.matmul(W, x)
#         print(f"u_hatat epoch {epoch} is equal to : {u_hat}")

        b_ij = Variable(torch.zeros(1, self.num_routes, self.num_capsules, 1))
        if USE_CUDA:
            b_ij = b_ij.cuda()
#             print(f"b_ij at epoch {epoch} is equal to : {b_ij}")

        num_iterations = 3
        for iteration in range(num_iterations):
            c_ij = F.softmax(b_ij, dim =1)
            c_ij = torch.cat([c_ij] * batch_size, dim=0).unsqueeze(4)

            s_j = (c_ij * u_hat).sum(dim=1, keepdim=True)
            v_j = self.squash(s_j)
#             print(f"b_ij at iteration {iteration} is equal to : {b_ij}")
            if iteration < num_iterations - 1:
                a_ij = torch.matmul(u_hat.transpose(3, 4), torch.cat([v_j] * self.num_routes, dim=1))
                b_ij = b_ij + a_ij.squeeze(4).mean(dim=0, keepdim=True)

        return v_j.squeeze(1)

    def squash(self, input_tensor):
        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)
        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))
        return output_tensor
class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.reconstraction_layers = nn.Sequential(
            nn.Linear(16 * 10, 512),
            nn.ReLU(inplace=True),
            nn.Linear(512, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 784),
            nn.Sigmoid()
        )

    def forward(self, x, data):
        classes = torch.sqrt((x ** 2).sum(2))
        classes = F.softmax(classes, dim =1)

        _, max_length_indices = classes.max(dim=1)
        masked = Variable(torch.sparse.torch.eye(10))
        if USE_CUDA:
            masked = masked.cuda()
        masked = masked.index_select(dim=0, index=max_length_indices.squeeze(1).data)

        reconstructions = self.reconstraction_layers((x * masked[:, :, None, None]).view(x.size(0), -1))
        reconstructions = reconstructions.view(-1, 1, 28, 28)

        return reconstructions, masked
class CapsNet(nn.Module):
    def __init__(self):
        super(CapsNet, self).__init__()
        self.conv_layer = ConvLayer()
        self.primary_capsules = PrimaryCaps()
        self.digit_capsules = DigitCaps()
        self.decoder = Decoder()

        self.mse_loss = nn.MSELoss()

    def forward(self, data):
        output = self.digit_capsules(self.primary_capsules(self.conv_layer(data)))
        reconstructions, masked = self.decoder(output, data)
        return output, reconstructions, masked

    def loss(self, data, x, target, reconstructions):
        return self.margin_loss(x, target) + self.reconstruction_loss(data, reconstructions)
#         return self.reconstruction_loss(data, reconstructions)

    def margin_loss(self, x, labels, size_average=True):
        batch_size = x.size(0)

        v_c = torch.sqrt((x**2).sum(dim=2, keepdim=True))

        left = F.relu(0.9 - v_c).view(batch_size, -1)
        right = F.relu(v_c - 0.1).view(batch_size, -1)
#         print(f"shape of labels, left and right respectively - {labels.size(), left.size(), right.size()}")

        loss = labels * left + 0.5 * (1.0 - labels) * right
        loss = loss.sum(dim=1).mean()

        return loss

    def reconstruction_loss(self, data, reconstructions):
        loss = self.mse_loss(reconstructions.view(reconstructions.size(0), -1), data.view(reconstructions.size(0), -1))
        return loss*0.0005

capsule_net = CapsNet()
if USE_CUDA:
    capsule_net = capsule_net.cuda()
optimizer = Adam(capsule_net.parameters())
capsule_net

##### Here is the problem while training####
batch_size = 100
mnist = Mnist(batch_size)

n_epochs = 5

for epoch in range(n_epochs):
    capsule_net.train()
    train_loss = 0
    for batch_id, (data, target) in enumerate(mnist.train_loader):

        target = torch.eye(10).index_select(dim=0, index=target)
        data, target = Variable(data), Variable(target)

        if USE_CUDA:
            data, target = data.cuda(), target.cuda()
            data, target = data.float().cuda(), target.float().cuda() # Here I changed the data to float and it's required only when I am using my extracted dataset for a single class
            data = data[:,:,:] # Use this when 1st MNist data is used
#             data = data[:,None,:,:] # Use this when I am using my extracted single class digits

        optimizer.zero_grad()
        output, reconstructions, masked = capsule_net(data)
        loss = capsule_net.loss(data, output, target, reconstructions)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()

#         if batch_id % 100 == 0:
#             print ("train accuracy:", sum(np.argmax(masked.data.cpu().numpy(), 1) ==
#                                    np.argmax(target.data.cpu().numpy(), 1)) / float(batch_size))

    print (train_loss / len(mnist.train_loader))

I used this to see the main data as image and the reconstructed image

import matplotlib
import matplotlib.pyplot as plt

def plot_images_separately(images):
    "Plot the six MNIST images separately."
    fig = plt.figure()
    for j in range(1, 10):
        ax = fig.add_subplot(1, 10, j)
        ax.matshow(images[j-1], cmap = matplotlib.cm.binary)
        plt.xticks(np.array([]))
        plt.yticks(np.array([]))
    plt.show()
plot_images_separately(data[:10,0].data.cpu().numpy())
plot_images_separately(reconstructions[:10,0].data.cpu().numpy())

Output:
I checked the normal performing code and then the problematic one, I found that the dataset passed into the network was of not same nature. The problems were -

The MNIST data extracted for a single class was not transformed into tensor and no normalization was applied, although I tried passing it through the transformation.

This is what I did to fix it -

I created transformation objections and tensor objection and then passed by list comprehension elements to it. Below are the codes and the final output of my network -

Preparing class 0 dataset (dataset for the digit 5)

class Mnist:
    trans = transforms.ToTensor()
    normalize = transforms.Normalize((0.1307,), (0.3081,))
    def init(self,batch_size):

    dataset_transform = transforms.Compose([
                        transforms.ToTensor(),
                        transforms.Normalize((0.1307,), (0.3081,))
    ])

    trans = transforms.ToTensor()
    normalize = transforms.Normalize((0.1307,), (0.3081,))
    train_mnist = datasets.MNIST("../data", train=True, transform=dataset_transform)
    test_mnist = datasets.MNIST("../data", train= False, transform=dataset_transform)
    train_image, train_label = train_mnist.train_data, train_mnist.train_labels
    test_image, test_label = test_mnist.test_data, test_mnist.test_labels

    train_0, test_0 = [normalize(trans(train_image[key].unsqueeze(2).numpy())) for (key, label) in enumerate(train_label) if int(label) == 5],[test_image[key] for (key, label) in enumerate(test_label) if int(label) == 5]
    train_label_0, test_label_0 = zero__train = [train_label[key] for (key, label) in enumerate(train_label) if int(label) == 5],[test_label[key] for (key, label) in enumerate(test_label) if int(label) == 5]

    train_dataset = tuple(zip(train_0, train_label_0))
    test_dataset = tuple(zip(test_0, test_label_0))

    self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

enter image description here


Instruction: pytorch : unable to understand model.forward function

Input:
I am learning deep learning and am trying to understand the pytorch code given below. I'm struggling to understand how the probability calculation works. Can somehow break it down in lay-man terms. Thanks a ton.

  ps = model.forward(images[0,:])

# Hyperparameters for our network
input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
print(images.shape)
ps = model.forward(images[0,:])

Output:
I'm a layman so I'll help you with the layman's terms :)

input_size = 784
hidden_sizes = [128, 64]
output_size = 10

These are parameters for the layers in your network. Each neural network consists of layers, and each layer has an input and an output shape.

Specifically input_size deals with the input shape of the first layer. This is the input_size of the entire network. Each sample that is input into the network will be a 1 dimension vector that  is length 784 (array that is 784 long).

hidden_size deals with the shapes inside the network. We will cover this a little later.

output_size deals with the output shape of the last layer. This means that our network will output a 1 dimensional vector that is length 10 for each sample.

Now to break up model definition line by line:

model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),

The nn.Sequential part simply defines a network, each argument that is input defines a new layer in that network in that order.

nn.Linear(input_size, hidden_sizes[0]) is an example of such a layer. It is the first layer of our network takes in an input of size input_size, and outputs a vector of size hidden_sizes[0]. The size of the output is considered "hidden" in that it is not the input or the output of the whole network. It "hidden" because it's located inside of the network far from the input and output ends of the network that you interact with when you actually use it.

This is called Linear because it applies a linear transformation by multiplying the input by its weights matrix and adding its bias matrix to the result. (Y = Ax + b, Y = output, x = input, A = weights, b = bias).

nn.ReLU(),

ReLU is an example of an activation function. What this function does is apply some sort of transformation to the output of the last layer (the layer discussed above), and outputs the result of that transformation. In this case the function being used is the ReLU function, which is defined as ReLU(x) = max(x, 0). Activation functions are used in neural networks because they create non-linearities. This allows your model to model non-linear relationships.

nn.Linear(hidden_sizes[0], hidden_sizes[1]),

From what we discussed above, this is a another example of a layer. It takes an input of hidden_sizes[0] (same shape as the output of the last layer) and outputs a 1D vector of length hidden_sizes[1].

nn.ReLU(),

Apples the ReLU function again.

nn.Linear(hidden_sizes[1], output_size)

Same as the above two layers, but our output shape is the output_size this time.

nn.Softmax(dim=1))

Another activation function. This activation function turns the logits outputted by nn.Linear into an actual probability distribution. This lets the model output the probability for each class. At this point our model is built.

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
print(images.shape)

These are simply just preprocessing training data and putting it into the correct format

ps = model.forward(images[0,:])

This passes the images through the model (forward pass) and applies the operations previously discussed in layer. You get the resultant output.


Instruction: What is the difference between MLP implementation from scratch and in PyTorch?

Input:
Following up the question from How to update the learning rate in a two layered multi-layered perceptron?

Given the XOR problem:

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

And a simple

two layered Multi-Layered Perceptron (MLP) with
sigmoid activations between them and
Mean Square Error (MSE) as the loss function/optimization criterion

If we train the model from scratch as such:

from itertools import chain
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(0)

def sigmoid(x): # Returns values that sums to one.
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(sx):
    # See https://math.stackexchange.com/a/1225116
    return sx * (1 - sx)

# Cost functions.
def mse(predicted, truth):
    return 0.5 * np.mean(np.square(predicted - truth))

def mse_derivative(predicted, truth):
    return predicted - truth

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

# Define the shape of the weight vector.
num_data, input_dim = X.shape
# Lets set the dimensions for the intermediate layer.
hidden_dim = 5
# Initialize weights between the input layers and the hidden layer.
W1 = np.random.random((input_dim, hidden_dim))

# Define the shape of the output vector.
output_dim = len(Y.T)
# Initialize weights between the hidden layers and the output layer.
W2 = np.random.random((hidden_dim, output_dim))

# Initialize weigh
num_epochs = 5000
learning_rate = 0.3

losses = []

for epoch_n in range(num_epochs):
    layer0 = X
    # Forward propagation.

    # Inside the perceptron, Step 2.
    layer1 = sigmoid(np.dot(layer0, W1))
    layer2 = sigmoid(np.dot(layer1, W2))

    # Back propagation (Y -> layer2)

    # How much did we miss in the predictions?
    cost_error = mse(layer2, Y)
    cost_delta = mse_derivative(layer2, Y)

    #print(layer2_error)
    # In what direction is the target value?
    # Were we really close? If so, don't change too much.
    layer2_error = np.dot(cost_delta, cost_error)
    layer2_delta = cost_delta *  sigmoid_derivative(layer2)

    # Back propagation (layer2 -> layer1)
    # How much did each layer1 value contribute to the layer2 error (according to the weights)?
    layer1_error = np.dot(layer2_delta, W2.T)
    layer1_delta = layer1_error * sigmoid_derivative(layer1)

    # update weights
    W2 += - learning_rate * np.dot(layer1.T, layer2_delta)
    W1 += - learning_rate * np.dot(layer0.T, layer1_delta)
    #print(np.dot(layer0.T, layer1_delta))
    #print(epoch_n, list((layer2)))

    # Log the loss value as we proceed through the epochs.
    losses.append(layer2_error.mean())
    #print(cost_delta)

# Visualize the losses
plt.plot(losses)
plt.show()

We get a sharp dive in the loss from epoch 0 and then saturates quickly:

But if we train a similar model with pytorch, the training curve has a gradual drop in losses before saturating:

What is the difference between the MLP from scratch and the PyTorch code?

Why is it achieving convergence at different point?

Other than the weights initialization, np.random.rand() in the code from scratch and the default torch initialization, I can't seem to see a difference in the model.

Code for PyTorch:

from tqdm import tqdm
import numpy as np

import torch
from torch import nn
from torch import tensor
from torch import optim

import matplotlib.pyplot as plt

torch.manual_seed(0)
device = 'gpu' if torch.cuda.is_available() else 'cpu'

# XOR gate inputs and outputs.
X = xor_input = tensor([[0,0], [0,1], [1,0], [1,1]]).float().to(device)
Y = xor_output = tensor([[0],[1],[1],[0]]).float().to(device)

# Use tensor.shape to get the shape of the matrix/tensor.
num_data, input_dim = X.shape
print('Inputs Dim:', input_dim) # i.e. n=2

num_data, output_dim = Y.shape
print('Output Dim:', output_dim)
print('No. of Data:', num_data) # i.e. n=4

# Step 1: Initialization.

# Initialize the model.
# Set the hidden dimension size.
hidden_dim = 5
# Use Sequential to define a simple feed-forward network.
model = nn.Sequential(
            # Use nn.Linear to get our simple perceptron.
            nn.Linear(input_dim, hidden_dim),
            # Use nn.Sigmoid to get our sigmoid non-linearity.
            nn.Sigmoid(),
            # Second layer neurons.
            nn.Linear(hidden_dim, output_dim),
            nn.Sigmoid()
        )
model

# Initialize the optimizer
learning_rate = 0.3
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

# Initialize the loss function.
criterion = nn.MSELoss()

# Initialize the stopping criteria
# For simplicity, just stop training after certain no. of epochs.
num_epochs = 5000

losses = [] # Keeps track of the loses.

# Step 2-4 of training routine.

for _e in tqdm(range(num_epochs)):
    # Reset the gradient after every epoch.
    optimizer.zero_grad()
    # Step 2: Foward Propagation
    predictions = model(X)

    # Step 3: Back Propagation
    # Calculate the cost between the predictions and the truth.
    loss = criterion(predictions, Y)
    # Remember to back propagate the loss you've computed above.
    loss.backward()

    # Step 4: Optimizer take a step and update the weights.
    optimizer.step()

    # Log the loss value as we proceed through the epochs.
    losses.append(loss.data.item())

plt.plot(losses)

Output:
List of differences between hand-rolled code and PyTorch code

Turns out there are a lot of differences between what your hand-rolled code and the PyTorch code are doing. Here's what I uncovered, listed roughly in order of most to least impact on the output:

Your code and the PyTorch code use two different functions to report the loss.
Your code and the PyTorch code set up the initial weights very differently. You mention this in your question, but it turns out to have a pretty significant impact on the results.
By default, the torch.nn.Linear layers add an extra bunch of "bias" weights to the model. Thus, the 1st layer of the Pytorch model effectively has 3x5 weights and the second layer has 6x1 weights. The layers in the hand-rolled code have 2x5 and 5x1 weights, respectively.

The bias seems to help the model to learn and adapt somewhat faster. If you turn the bias off, it takes roughly twice as many training epochs for the Pytorch model to reach near 0 loss.

Curiously, it seems like the Pytorch model is using a learning rate that is effectively half of what you specify. Alternatively, it may be that there's a stray factor of 2 that's found its way into your hand-rolled math/code somewhere.

How to get identical results from the hand-rolled and Pytorch code

By carefully accounting for the above 4 factors, it is possible to achieve complete parity between the hand-rolled and the Pytorch code. With the correct tweaks and settings, the two snippets will produce identical results:

The most important tweak - make the loss reporting functions match

The critical difference is that you end up using two completely different functions to measure the loss in the two code snippets:

In the hand rolled code, you measure the loss as layer2_error.mean(). If you unpack the variable, you can see that layer2_error.mean() is a somewhat screwy and meaningless value:

layer2_error.mean()
== np.dot(cost_delta, cost_error).mean()
== np.dot(mse_derivative(layer2, Y), mse(layer2, Y)).mean()
== np.sum(.5 * (layer2 - Y) * ((layer2 - Y)**2).mean()).mean()

On the other hand, in the PyTorch code the loss is measured in terms of the traditional definition of the mse, ie as the equivalent of np.mean((layer2 - Y)**2). You can prove this to yourself by modifying your PyTorch loop like so:

def mse(x, y):
    return np.mean((x - y)**2)

torch_losses = [] # Keeps track of the loses.
torch_losses_manual = [] # for comparison

# Step 2-4 of training routine.

for _e in tqdm(range(num_epochs)):
    # Reset the gradient after every epoch.
    optimizer.zero_grad()
    # Step 2: Foward Propagation
    predictions = model(X)

    # Step 3: Back Propagation
    # Calculate the cost between the predictions and the truth.
    loss = criterion(predictions, Y)
    # Remember to back propagate the loss you've computed above.
    loss.backward()

    # Step 4: Optimizer take a step and update the weights.
    optimizer.step()

    # Log the loss value as we proceed through the epochs.
    torch_losses.append(loss.data.item())
    torch_losses_manual.append(mse(predictions.detach().numpy(), Y.detach().numpy()))

plt.plot(torch_losses, lw=5, label='torch_losses')
plt.plot(torch_losses_manual, lw=2, label='torch_losses_manual')
plt.legend()

Output:

Also important - use the same initial weights

PyTorch uses it's own special routine for setting the initial weights which produces very different results from np.random.rand. I haven't been able to exactly replicate it yet, but for the next best thing we can just hijack Pytorch. Here's a function that will get the same initial weights that the Pytorch model uses:

import torch
from torch import nn
torch.manual_seed(0)

def torch_weights(nodes_in, nodes_hidden, nodes_out, bias=None):
    model = nn.Sequential(
        nn.Linear(nodes_in, nodes_hidden, bias=bias),
        nn.Sigmoid(),
        nn.Linear(nodes_hidden, nodes_out, bias=bias),
        nn.Sigmoid()
    )

    return [t.detach().numpy() for t in model.parameters()]

Finally - in Pytorch, turn off all bias weights and double the learning rate

Eventually you may want to implement the bias weights in your own code. For now, we'll just turn the bias off in the Pytorch model and compare the results of the hand-rolled model to those of the unbiased Pytorch model.

Also, in order to make the results match you need to double the Pytorch model's learning rate. This effectively scales the results along the x axis (ie doubling the rate means that it takes half as many epochs to reach some particular feature on the loss curve).

Putting it together

In order to reproduce the hand_rolled_losses data from the plot at the start of my post, all you need to do is take your hand-rolled code and replace the mse function with:

def mse(predicted, truth):
    return np.mean(np.square(predicted - truth))

the lines that initialize the weights with:

W1,W2 = [w.T for w in torch_weights(input_dim, hidden_dim, output_dim)]

and the line that tracks the losses with:

losses.append(cost_error)

and you should be good to go.

In order to reproduce the torch_losses data from the plot we'll also need to turn the bias weights off in the Pytorch model. To do that, you just need to change the lines defining the Pytorch model like so:

model = nn.Sequential(
    # Use nn.Linear to get our simple perceptron.
    nn.Linear(input_dim, hidden_dim, bias=None),
    # Use nn.Sigmoid to get our sigmoid non-linearity.
    nn.Sigmoid(),
    # Second layer neurons.
    nn.Linear(hidden_dim, output_dim, bias=None),
    nn.Sigmoid()
)

You also need to change the line defining the learning_rate:

learning_rate = 0.3 * 2

Complete code listing

The hand-rolled code

Here's a complete listing of my version of the hand-rolled neural net code, to help with reproducing my results:

from itertools import chain
import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
import scipy.stats
import torch
from torch import nn

np.random.seed(0)
torch.manual_seed(0)

def torch_weights(nodes_in, nodes_hidden, nodes_out, bias=None):
    model = nn.Sequential(
        nn.Linear(nodes_in, nodes_hidden, bias=bias),
        nn.Sigmoid(),
        nn.Linear(nodes_hidden, nodes_out, bias=bias),
        nn.Sigmoid()
    )

    return [t.detach().numpy() for t in model.parameters()]

def sigmoid(x): # Returns values that sums to one.
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(sx):
    # See https://math.stackexchange.com/a/1225116
    return sx * (1 - sx)

# Cost functions.
def mse(predicted, truth):
    return np.mean(np.square(predicted - truth))

def mse_derivative(predicted, truth):
    return predicted - truth

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

# Define the shape of the weight vector.
num_data, input_dim = X.shape
# Lets set the dimensions for the intermediate layer.
hidden_dim = 5
# Define the shape of the output vector.
output_dim = len(Y.T)

W1,W2 = [w.T for w in torch_weights(input_dim, hidden_dim, output_dim)]

num_epochs = 5000
learning_rate = 0.3
losses = []

for epoch_n in range(num_epochs):
    layer0 = X
    # Forward propagation.

    # Inside the perceptron, Step 2.
    layer1 = sigmoid(np.dot(layer0, W1))
    layer2 = sigmoid(np.dot(layer1, W2))

    # Back propagation (Y -> layer2)

    # In what direction is the target value?
    # Were we really close? If so, don't change too much.
    cost_delta = mse_derivative(layer2, Y)
    layer2_delta = cost_delta *  sigmoid_derivative(layer2)

    # Back propagation (layer2 -> layer1)
    # How much did each layer1 value contribute to the layer2 error (according to the weights)?
    layer1_error = np.dot(layer2_delta, W2.T)
    layer1_delta = layer1_error * sigmoid_derivative(layer1)

    # update weights
    W2 += - learning_rate * np.dot(layer1.T, layer2_delta)
    W1 += - learning_rate * np.dot(layer0.T, layer1_delta)

    # Log the loss value as we proceed through the epochs.
    losses.append(mse(layer2, Y))

# Visualize the losses
plt.plot(losses)
plt.show()

The Pytorch code

import matplotlib.pyplot as plt
from tqdm import tqdm
import numpy as np

import torch
from torch import nn
from torch import tensor
from torch import optim

torch.manual_seed(0)
device = 'gpu' if torch.cuda.is_available() else 'cpu'

num_epochs = 5000
learning_rate = 0.3 * 2

# XOR gate inputs and outputs.
X = tensor([[0,0], [0,1], [1,0], [1,1]]).float().to(device)
Y = tensor([[0],[1],[1],[0]]).float().to(device)

# Use tensor.shape to get the shape of the matrix/tensor.
num_data, input_dim = X.shape
num_data, output_dim = Y.shape

# Step 1: Initialization.

# Initialize the model.
# Set the hidden dimension size.
hidden_dim = 5
# Use Sequential to define a simple feed-forward network.
model = nn.Sequential(
    # Use nn.Linear to get our simple perceptron.
    nn.Linear(input_dim, hidden_dim, bias=None),
    # Use nn.Sigmoid to get our sigmoid non-linearity.
    nn.Sigmoid(),
    # Second layer neurons.
    nn.Linear(hidden_dim, output_dim, bias=None),
    nn.Sigmoid()
)

# Initialize the optimizer
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

# Initialize the loss function.
criterion = nn.MSELoss()

def mse(x, y):
    return np.mean((x - y)**2)

torch_losses = [] # Keeps track of the loses.
torch_losses_manual = [] # for comparison

# Step 2-4 of training routine.

for _e in tqdm(range(num_epochs)):
    # Reset the gradient after every epoch.
    optimizer.zero_grad()
    # Step 2: Foward Propagation
    predictions = model(X)

    # Step 3: Back Propagation
    # Calculate the cost between the predictions and the truth.
    loss = criterion(predictions, Y)
    # Remember to back propagate the loss you've computed above.
    loss.backward()

    # Step 4: Optimizer take a step and update the weights.
    optimizer.step()

    # Log the loss value as we proceed through the epochs.
    torch_losses.append(loss.data.item())
    torch_losses_manual.append(mse(predictions.detach().numpy(), Y.detach().numpy()))

plt.plot(torch_losses, lw=5, c='C1', label='torch_losses')
plt.plot(torch_losses_manual, lw=2, c='C2', label='torch_losses_manual')
plt.legend()

Notes

Bias weights

You can find some very instructive examples that show what the bias weights are and how to implement them in this tutorial. They list a bunch of pure-Python implementations of neural nets very similar to your hand-rolled one, so it's likely you could adapt some of their code to make your own implementation of the bias.

Functions to produce an initial guess of the weights

Here's a function I adapted from that same tutorial that can produce reasonable initial values for the weights. I think the algorithm Pytorch uses internally is somewhat different, but this produces similar results:

import scipy as sp
import scipy.stats

def tnorm_weights(nodes_in, nodes_out, bias_node=0):
    # see https://www.python-course.eu/neural_network_mnist.php
    wshape = (nodes_out, nodes_in + bias_node)
    bound = 1 / np.sqrt(nodes_in)
    X = sp.stats.truncnorm(-bound, bound)
    return X.rvs(np.prod(wshape)).reshape(wshape)


Instruction: Model parameter initialization

Input:
I am new to PyTorch. I am writing a simple program for linear regression and i want to compare the results by using different methods (SGD,momentum,ADAM,etc). The problem I have is that I want every time a loop ends for the model parameters to be reinitialized to the same value that the previous model started with  so the comparison is valid.

This is what i have so far, this is my training data:

      x1=np.arange(0,10,1).reshape(10,1)
      y1=2*x1+1+np.random.normal(0,1,(10,1))
      x=torch.from_numpy(x1)
      y=torch.from_numpy(y1)

Here i train the data

      from torch.utils.data import TensorDataset,DataLoader
      train=TensorDataset(xdata,ydata)
      size_batch=10
      dl=DataLoader(train,size_batch,shuffle=True)

Define the model and opt

     model=nn.Linear(1,1)
     opt=torch.optim.SGD(model.parameters(), lr=1e-4)
     import torch.nn.functional as F
     loss1=F.mse_loss
     loss=loss1(model(x),y)

Function

     def fitmodel(nepochs, model, loss1, opt):
         for epoch in range(nepochs):
             for xm,ym in dl:
                 predict = model(xm)
                 loss = loss1(predict, ym)
                 loss.backward()
                 opt.step()
                 opt.zero_grad()

Call the function

     fitmodel(1000,model,loss1,opt)

Now i want to rerun the above but for different optimization algorithms. If I just rerun fitmodel it will use some of the parameters it already has calculated. I want to start from the same 'initial conditions' as i started the previous run. Anyone has any ideas how to do that?

Edit
Before i run the fitmodel I copy the initial bias and weight

   w1=model.weight
   b1=model.bias
   fitmodel(1000,model,loss1,opt)
   model.weight=w1
   model.bias=b1
   loss=[]

But i get this error:
TypeError: cannot assign 'list' as parameter 'bias' (torch.nn.Parameter or None expected)

Output:
The parameters of a linear layer are stored in model.weight and model.bias. You need to copy those before training, and then restore afterwards. This is a bit more involved than what you're doing in your code. Example below

# clone and detach so that we have an actual backup copy,
# not merely a reference to the parameters
w1=model.weight.clone().detach()
b1=model.bias.clone().detach()

for i in range(3): # as many experiments as you wish to run

    # since we have detached, w1 and b1 are no longer nn.Parameter -
    # we have to rewrap them. We keep copying so that the tensors used
    # in the computation are separate from the backup copies
    model.weight=nn.Parameter(w1.clone())
    model.bias=nn.Parameter(b1.clone())

    # we reinitialize the optimizer because it looks at model.parameters()
    # if not for this line, it would try to optimize the values from
    # the previous experiment!
    opt=torch.optim.SGD(model.parameters(), lr=1e-4)

    fitmodel(1000,model,loss1,opt)


Instruction: how to multiply all rows by column of single matrix in pytorch in a vectorized way

Input:
I need to multiply all rows of a matrix by column, i think with an example it will be clearer:

matrix is:

1,2,3
4,5,6
7,8,9

An i need an operation that returns:

28,80,162

But i can't find anything in the documentation and blogs and other SO question  only are related to matrix multiplication and dot product, which is not what i need in this case,how can it be achieved in a vectorized fashion(instead of for loop based) ?

For example this is easy to achieve for the case of sum like:

the_matrix.sum(dim=0)

But there's not something like:

the_matrix.mul(dim=0)

Output:
I found the solution, there's no :

the_matrix.mul(dim=0)

But there's:

he_matrix.prod(dim=0)

Which does exactly what is needed.


Instruction: Pytorch mask missing values when calculating rmse

Input:
I'm trying to calculate the rmse error of two torch tensors. I would like to ignore/mask the rows where the labels are 0 (missing values). How could I modify this line to take that restriction into account?

torch.sqrt(((preds.detach() - labels) ** 2).mean()).item()

Thank you in advance.

Output:
This can be solved by defining a custom MSE loss function* that masks out the missing values, 0 in your case, from both the input and target tensors:
def mse_loss_with_nans(input, target):

    # Missing data are nan's
    # mask = torch.isnan(target)

    # Missing data are 0's
    mask = target == 0

    out = (input[~mask]-target[~mask])**2
    loss = out.mean()

    return loss

(*) Computing MSE is equivalent to RMSE from an optimisation point of view -- with the advantage of being computationally faster.


Instruction: How Weight update in Dynamic Computation Graph of pytorch works?

Input:
How does the Weight Update works in Pytorch code of Dynamic Computation Graph when Weights are shard (=reused multiple times)

https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#sphx-glr-beginner-examples-nn-dynamic-net-py

import random
import torch

class DynamicNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
    """
    In the constructor we construct three nn.Linear instances that we will use
    in the forward pass.
    """
    super(DynamicNet, self).__init__()
    self.input_linear = torch.nn.Linear(D_in, H)
    self.middle_linear = torch.nn.Linear(H, H)
    self.output_linear = torch.nn.Linear(H, D_out)

def forward(self, x):
    """
    For the forward pass of the model, we randomly choose either 0, 1, 2, or 3
    and reuse the middle_linear Module that many times to compute hidden layer
    representations.

    Since each forward pass builds a dynamic computation graph, we can use normal
    Python control-flow operators like loops or conditional statements when
    defining the forward pass of the model.

    Here we also see that it is perfectly safe to reuse the same Module many
    times when defining a computational graph. This is a big improvement from Lua
    Torch, where each Module could be used only once.
    """
    h_relu = self.input_linear(x).clamp(min=0)
    for _ in range(random.randint(0, 3)):
        h_relu = self.middle_linear(h_relu).clamp(min=0)
    y_pred = self.output_linear(h_relu)
    return y_pred

I want to know what happens to middle_linear weight at each backward which is used multiple times at a step

Output:
When you call backward (either as the function or a method on a tensor) the gradients of operands with requires_grad == True are calculated with respect to the tensor you called backward on. These gradients are accumulated in the .grad property of these operands. If the same operand A appears multiple times in the expression, you can conceptually treat them as separate entities A1, A2... for the backpropagation algorithm and just at the end sum their gradients so that A.grad = A1.grad + A2.grad + ....

Now, strictly speaking, the answer to your question

  I want to know what happens to middle_linear weight at each backward

is: nothing. backward does not change weights, only calculates the gradient. To change the weights you have to do an optimization step, perhaps using one of the optimizers in torch.optim. The weights are then updated according to their .grad property, so if your operand was used multiple times, it will be updated accordingly to the sum of the gradients in each of its uses.

In other words, if your matrix element x has positive gradient when first applied and negative when used the second time, it may be that the net effects will cancel out and it will stay as it is (or change just a bit). If both applications call for x to be higher, it will raise more than if it was used just once, etc.


Instruction: Pytorch errors: "received an invalid combination of arguments" in Jupyter Notebook

Input:
I'm trying to learn Pytorch, but whenever I seem to try any online tutorial (https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py), I get errors when trying to run certain functions, but only in Jupyter Notebook.

When running

x = torch.empty(5, 3)

I get an error:

module 'torch' has no attribute 'empty'

Furthermore, when running

x = torch.zeros(5, 3, dtype=torch.long)

I get the error:

module 'torch' has no attribute 'long'

Some other functions work fine like:

x = torch.rand(5, 3)

But generally, most code I try to run seems to run into an error really quickly. I couldn't find any resolution online.

When I go into my docker container and simply run python in the shell, I can run these lines just fine with no errors.

I'm running pytorch in a Docker image that I extended from a fastai image, as it already included things like jupyter notebook and pytorch. I used anaconda to update everything, and committed it to a new image for myself.

I have absolutely no idea what the issue could be. I've tried updating packages through anaconda, pip, aptitude in my docker container, and making sure to commit my changes, but nothing seems to work. I also tried creating a new kernel with python 3.7 as I noticed that my Jupyter Notebook only runs in 3.6.4, and when I run python in the shell it is at 3.7.

I've also tried getting different docker images and extending them with what I need, but all images that I've tried have had errors with anaconda where it gets stuck on "Solving environment" step.

Output:
Ok, so the fix for me was to either update pytorch through conda using the following command

conda update pytorch

If it's not installed yet, I've gotten it to work in other environments by simply installing it through conda

conda install pytorch

Kind of stupid that I didn't try this earlier, but I was confused on the difference between conda and pip.


Instruction: How to look at the parameters of a pytorch model?

Input:
I have a simple pytorch neural net that I copied from openai, and I modified it to some extent (mostly the input).

When I run my code, the output of the network remains the same on every episode, as if no training occurs.

I want to see if any training happens, or if some other reason causes the results to be the same.

How can I make sure any movement happens to the weights?

Thanks

Output:
Depends on what you are doing, but the easiest would be to check the weights of your model.

You can do this (and compare with the ones from previous iteration) using the following code:

for parameter in model.parameters():
    print(parameter.data)

If the weights are changing, the neural network is being optimized (which doesn't necessarily mean it learns anything useful in particular).


Instruction: How does the parameter 'dim' in torch.unique() work?

Input:
I am trying to extract the unique values in each row of a matrix and returning them into the same matrix (with repeated values set to say, 0) For example, I would like to transform

torch.Tensor(([1, 2, 3, 4, 3, 3, 4],
              [1, 6, 3, 5, 3, 5, 4]])

to

torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 0, 0, 4]])

or

torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 4, 0, 0]])

I.e. the order does not matter in the rows. I have tried using pytorch.unique() and in the documentation it is mentioned that the dimension to take the unique values can be specified with the parameter dim. However, It doesn't seem to work for this case.

I've tried:

output= torch.unique(torch.Tensor([[4,2,52,2,2],[5,2,6,6,5]]), dim = 1)

output

Which gives

tensor([[ 2.,  2.,  2.,  4., 52.],
        [ 2.,  5.,  6.,  5.,  6.]])

Does anyone have a particular fix for this? If possible, I'm trying to avoid for loops.

Output:
One must admit the unique function can sometimes be very confusing without given proper examples and explanations.

The dim parameter specifies which dimension on the matrix tensor you want to apply on.

For instance, in a 2D matrix, dim=0 will let operation perform vertically where dim=1 means horizontally.

Example, let's consider a 4x4 matrix with dim=1. As you can see from my code below, the unique operation is applied row by row.

You notice the double occurrence of the number 11 in the first and last row. Numpy and Torch does this to preserve the shape of the final matrix.

However, if you do not specify any dimension, torch will automatically flatten your matrix and then apply unique to it and you will get a 1D array that contains unique data.

import torch

m = torch.Tensor([
    [11, 11, 12,11],
    [13, 11, 12,11],
    [16, 11, 12, 11],
    [11, 11, 12, 11]
])

output, indices = torch.unique(m, sorted=True, return_inverse=True, dim=1)
print("Ori \n{}".format(m.numpy()))
print("Sorted \n{}".format(output.numpy()))
print("Indices \n{}".format(indices.numpy()))

# without specifying dimension
output, indices = torch.unique(m, sorted=True, return_inverse=True)
print("Sorted (no dim) \n{}".format(output.numpy()))

Result (dim=1)

Ori
[[11. 11. 12. 11.]
 [13. 11. 12. 11.]
 [16. 11. 12. 11.]
 [11. 11. 12. 11.]]
Sorted
[[11. 11. 12.]
 [11. 13. 12.]
 [11. 16. 12.]
 [11. 11. 12.]]
Indices
[1 0 2 0]

Result (no dimension)

Sorted (no dim)
[11. 12. 13. 16.]


Instruction: How to convert a pytorch tensor into a numpy array?

Input: How do I convert a torch tensor to numpy?

Output:
copied from pytorch doc:
a = torch.ones(5)
print(a)

tensor([1., 1., 1., 1., 1.])

b = a.numpy()
print(b)

[1. 1. 1. 1. 1.]

Following from the below discussion with @John:
In case the tensor is (or can be) on GPU, or in case it (or it can) require grad, one can use
t.detach().cpu().numpy()

I recommend to uglify your code only as much as required.


Instruction: Strange behavior of linear regression in PyTorch

Input:
I am facing a peculiar problem and I was wondering if there is an explanation. I am trying to run a linear regression problem and test different optimization methods and two of them have a strange outcome when comparing to each other. I build a data set that satisfies y=2x+5 and I add a random noise to that.

 xtrain=np.range(0,50,1).reshape(50,1)
 ytrain=2*train+5+np.random.normal(0,2,(50,1))
 opt1=torch.optim.SGD(model.parameters(),lr=1e-5,momentum=0.8))
 opt2=torch.optim.Rprop(model.parameters(),lr=1e-5)
 F_loss=F.mse_loss
 from torch.utils.data import TensorDataset,DataLoader
 train_d=TensorDataset(xtrain,ytrain)
 train=DataLoader(train_d,50,shuffle=True)
 model1=nn.Linear(1,1)
 loss=F_loss(model1(xtrain),ytrain)

 def fit(nepoch, model1, F_loss, opt):
     for epoch in range(nepoch):
         for i,j in train:
         predict = model1(i)
         loss = F_loss(predict, j)
         loss.backward()
         opt.step()
         opt.zero_grad()

When i compare the results of the following commands:

        fit(500000, model1, F_loss, opt1)
        fit(500000, model1, F_loss, opt2)

In the last epoch for opt1:loss=2.86,weight=2.02,bias=4.46
 In the last epoch for opt2:loss=3.47,weight=2.02,bias=4.68

These results do not make sense to me, shouldn't opt2 have a smaller loss than opt1 since the weight and bias it finds is closer to the real value? opt2's method finds weights and biases to be closer to the real value (they are respectively 2 and 5). Am i doing something wrong?

Output:
This has to do with the fact that you are drawing the training samples themselves from a random distribution.

By doing so, you inherently randomized the ground truth to some extent. Sure, you will get values that are inherently distributed around 2x+5, but you do not guarantee that 2x+5 will also be the best fit to this data distribution.
It could thus happen that you accidentally end up with values that deviate quite significantly from the original function, and, since you use a mean squared error, these values get weighted quite significantly.

In expectation (i.e., for the number of samples going towards infinity), you will likely get closer and closer to the expected parameters.

A way to verify this would be to plot your training samples against the parameter set, as well as the (ideal) underlying function.
Also note that Linear Regression does have a direct solution - something that is very uncommon in Machine Learning - meaning you can directly calculate an optimal solution, e.g., with sklearn's function


Instruction: RuntimeError: PyTorch does not currently provide packages for PyPI

Input:
I'm trying to run  this https://github.com/shariqiqbal2810/MAAC repository and it has a module called torch
import torch as McLawrence
from torch.autograd import Variable

I'm using python version 3.7.1 and I downgraded to 3.6.5 on win10
I tried to use
pip install torch
pip install pytorch
pip install torchvision

I change torch to pytorch
I enter their website https://pytorch.org/ and I tried all suggestion that uses pip such as
pip3 install https://download.pytorch.org/whl/cu80/torch-1.0.0-cp36-cp36m-win_amd64.whl
pip3 install torchvision

I read every comment relate to this issue in
https://github.com/pytorch/pytorch/issues/566
I read most of the answer here
https://stackoverflow.com/search?q=module+named+torch
But I still get the same message:
Collecting torch
  Using cached https://files.pythonhosted.org/packages/5f/e9/bac4204fe9cb1a002ec6140b47f51affda1655379fe302a1caef421f9846/torch-0.1.2.post1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\iibra\AppData\Local\Temp\pip-install-yupbt_qk\torch\setup.py", line 11, in <module>
        raise RuntimeError(README)
    RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

    Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

pip install pytorch
Collecting pytorch
  Using cached https://files.pythonhosted.org/packages/a9/41/4487bc23e3ac4d674943176f5aa309427b011e00607eb98899e9d951f67b/pytorch-0.1.2.tar.gz
Building wheels for collected packages: pytorch
  Running setup.py bdist_wheel for pytorch ... error
  Complete output from command c:\users\iibra\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\iibra\\AppData\\Local\\Temp\\pip-install-7qwe1571\\pytorch\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\iibra\AppData\Local\Temp\pip-wheel-_mjh_olk --python-tag cp36:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\iibra\AppData\Local\Temp\pip-install-7qwe1571\pytorch\setup.py", line 17, in <module>
      raise Exception(message)
  Exception: You should install pytorch from http://pytorch.org

pip install torchvision
Collecting torchvision
  Using cached https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: six in c:\users\iibra\appdata\local\programs\python\python36\lib\site-packages (from torchvision) (1.11.0)
Requirement already satisfied: pillow>=4.1.1 in c:\users\iibra\appdata\local\programs\python\python36\lib\site-packages (from torchvision) (5.1.0)
Collecting torch (from torchvision)
  Using cached https://files.pythonhosted.org/packages/5f/e9/bac4204fe9cb1a002ec6140b47f51affda1655379fe302a1caef421f9846/torch-0.1.2.post1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\iibra\AppData\Local\Temp\pip-install-k8dl2vhz\torch\setup.py", line 11, in <module>
        raise RuntimeError(README)
    RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

    Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

Output:
Installing from the PyTorch wheel should have worked. But, the problem turns out to be that pip is using the cached pytorch to install it as mentioned on GitHub here.

Collecting pytorch
  Using cached https://files.pythonhosted.org/packages...

Either removing the pip's cache from %LocalAppData%\pip\Cache on Windows or disabling it by using --no-cache-dir would solve the issue as follows:

pip3 --no-cache-dir install https://download.pytorch.org/whl/cu80/torch-1.0.0-cp36-cp36m-win_amd64.whl


Instruction: Applying any optimization causes values to be NaN

Input:
I am working on a classifier model for this data set:
https://archive.ics.uci.edu/ml/datasets/ILPD+%28Indian+Liver+Patient+Dataset%29
and I have come up with this code in pytorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np

ILPDataset = pd.read_csv('preprocessed-ilpd.csv')
ILPDataset["Age"] = pd.to_numeric(ILPDataset["Age"])
LabelEncoder = preprocessing.LabelEncoder()
LabelEncoder.fit(ILPDataset["Gender"])
ILPDataset["Gender"] = LabelEncoder.transform(ILPDataset["Gender"])
print(ILPDataset["Gender"].describe())
ILPDataset["AAP"] = preprocessing.scale(ILPDataset["AAP"])
ILPDataset["SgAlAm"] = preprocessing.scale(ILPDataset["SgAlAm"])
ILPDataset["SgApAm"] = preprocessing.scale(ILPDataset["SgApAm"])
Features = ["Age","Gender","TB","DB","AAP","SgAlAm","SgApAm","TP","ALB","A/G"]
ILPDFeatures = ILPDataset[Features]
ILPDTarget = ILPDataset["Selector"]

X_Train, X_Test, Y_Train, Y_Test = train_test_split(ILPDFeatures,ILPDTarget,test_size=0.2,random_state=0)

print(X_Train.shape)
print(Y_Train.shape)
print(X_Test.shape)
print(Y_Test.shape)
torch.set_default_tensor_type(torch.DoubleTensor)
TrainX = torch.from_numpy(X_Train.values).double()
TestX = torch.from_numpy(X_Test.values).double()

TrainY = torch.from_numpy(Y_Train.values).long().view(1,-1)[0]
TestY = torch.from_numpy(Y_Test.values).long().view(1,-1)[0]
TrainY.reshape(TrainY.shape[0],1)
TestY.reshape(TestY.shape[0],1)
print(X_Train.shape[1])
input_layers = X_Train.shape[1]
output_layers = 2
hidden = 100

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(input_layers,hidden)
        self.fc2 = nn.Linear(hidden,hidden)
        self.fc3 = nn.Linear(hidden,output_layers)

    def forward(self,x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        x = self.fc3(x)
        return F.softmax(x,dim=-1)

model = Net()
optimizer = torch.optim.SGD(model.parameters(),lr = 0.0001,momentum = 0.9)
loss_fn = nn.NLLLoss()
epochs_data = []
epochs = 601
print(TrainX.shape)
print(TestY.shape)
for epoch in range(1,3):
    optimizer.zero_grad()
    Ypred = model(TrainX)
    loss = loss_fn(Ypred,TrainY)
    loss.backward()
    optimizer.step()
    print(Ypred)
    if epoch%100==0:
        print("Epoch - %d, (%d%%) "%(epoch,epoch/epochs*100))

YPred_Test = model(TestX)
Loss_Test = loss_fn(YPred_Test,TestY)
print(Loss_Test.item())
print(YPred_Test)
print(YPred_Test)
print(TestY)

I have used different optimizers with different LRs and Momentum but on applying each one of them after loss.backward(), it turns the values in the tensor to NaN. I looked up the different answers on SO but even after trying them out I couldn't solve the bug.

Output: The problem is caused by not normalizing every column in the dataset. I have no idea why but normalizing column will solve the problem.


Instruction: What's the difference between torch.stack() and torch.cat() functions?

Input:
OpenAI's REINFORCE and actor-critic example for reinforcement learning has the following code:
REINFORCE:
policy_loss = torch.cat(policy_loss).sum()

actor-critic:
loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()

One is using torch.cat, the other uses torch.stack, for similar use cases.
As far as my understanding goes, the doc doesn't give any clear distinction between them.
I would be happy to know the differences between the functions.

Output:
stack

Concatenates sequence of tensors along a new dimension.

cat

Concatenates the given sequence of seq tensors in the given dimension.

So if A and B are of shape (3, 4):

torch.cat([A, B], dim=0) will be of shape (6, 4)
torch.stack([A, B], dim=0) will be of shape (2, 3, 4)


Instruction: Pytorch median - is it bug or am I using it wrong

Input:
I am trying to get median of each row of 2D torch.tensor. But the result is not what I expect when compared to working with standard array or numpy

import torch
import numpy as np
from statistics import median

print(torch.__version__)
>>> 0.4.1

y = [[1, 2, 3, 5, 9, 1],[1, 2, 3, 5, 9, 1]]
median(y[0])
>>> 2.5

np.median(y,axis=1)
>>> array([2.5, 2.5])

yt = torch.tensor(y,dtype=torch.float32)
yt.median(1)[0]
>>> tensor([2., 2.])

Output:
Looks like this is the intended behaviour of Torch as mentioned in this issue

https://github.com/pytorch/pytorch/issues/1837
https://github.com/torch/torch7/pull/182

The reasoning as mentioned in the link above

  Median returns 'middle' element in case of odd-many elements, otherwise one-before-middle element (could also do the other convention to take mean of the two around-the-middle elements, but that would be twice more expensive, so I decided for this one).


Instruction: Update step in PyTorch implementation of Newton's method

Input:
I'm trying to get some insight into how PyTorch works by implementing Newton's method for solving x = cos(x). Here's a version that works:

x =  Variable(DoubleTensor([1]), requires_grad=True)

for i in range(5):
    y = x - torch.cos(x)
    y.backward()
    x = Variable(x.data - y.data/x.grad.data, requires_grad=True)

print(x.data) # tensor([0.7390851332151607], dtype=torch.float64) (correct)

This code seems inelegant (inefficient?) to me since it's recreating the entire computational graph during each step of the for loop (right?). I tried to avoid this by simply updating the data held by each of the variables instead of recreating them:

x =  Variable(DoubleTensor([1]), requires_grad=True)
y = x - torch.cos(x)
y.backward(retain_graph=True)

for i in range(5):
    x.data = x.data - y.data/x.grad.data
    y.data = x.data - torch.cos(x.data)
    y.backward(retain_graph=True)

print(x.data) # tensor([0.7417889255761136], dtype=torch.float64) (wrong)

Seems like, with DoubleTensors, I'm carrying enough digits of precision to rule out round-off error. So where's the error coming from?

Possibly related: The above snippet breaks without the retain_graph=True flag set at every step if the for loop. The error message I get if I omit it within the loop --- but retain it on line 3 --- is:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. This seems like evidence that I'm misunderstanding something...

Output:
I think your first version of code is optimal, meaning that it is not creating a computation graph on every run.

# initial guess
guess = torch.tensor([1], dtype=torch.float64, requires_grad = True)

# function to optimize
def my_func(x):
    return x - torch.cos(x)

def newton(func, guess, runs=5):
    for _ in range(runs):
        # evaluate our function with current value of `guess`
        value = my_func(guess)
        value.backward()
        # update our `guess` based on the gradient
        guess.data -= (value / guess.grad).data
        # zero out current gradient to hold new gradients in next iteration
        guess.grad.data.zero_()
    return guess.data # return our final `guess` after 5 updates

# call starts
result = newton(my_func, guess)

# output of `result`
tensor([0.7391], dtype=torch.float64)

In each run, our function my_func(), which defines the computation graph, is evaluated with the current guess value. Once that returns the result, we compute the gradient (with value.backward() call). With this gradient, we now update our guess and zero-out our gradients so that it will be afresh for holding the gradients the next time we call value.backward() (i.e. it stops accumulating the gradient; without zeroing out the gradient, it would by default starts accumulating the gradients. But, we want to avoid that behaviour here).


Instruction: Size mismatch error with PyTorch toturial

Input:
I am learning the tutorial called Deep Learning with PyTorch: A 60 Minute Blitz on PyTorch website. My codes are the same as those of it, but there is a size mismatch error as shown below. Could anyone tell me why and how to solve it? Thank you:)

  RuntimeError: size mismatch, m1: [80 x 5], m2: [400 x 120] at
  c:\a\w\1\s\tmp_conda_3.7_110509\conda\conda-bld\pytorch_1544094576194\work\aten\src\th\generic/THTensorMath.cpp:940

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
def __init__(self):
    super(Net,self).__init__()
    self.conv1=nn.Conv2d(1,6,5)
    self.conv2=nn.Conv2d(6,16,5)
    self.fc1=nn.Linear(16*5*5,120)
    self.fc2=nn.Linear(120,84)
    self.fc3=nn.Linear(84,10)

def forward(self,x):
    x=F.max_pool2d(F.relu(self.conv1(x)),(2,2))
    x=F.max_pool2d(F.relu(self.conv2(x)),2)
    x.view(-1,self.num_flat_features(x))
    x=F.relu(self.fc1(x))
    x=F.relu(self.fc2(x))
    x=self.fc3(x)
    return x

def num_flat_features(self,x):
    size=x.size()[1:]
    num_features=1
    for s in size:
        num_features*=s
    return num_features

net=Net()
input=torch.randn(1,1,32,32)
out=net(input)
print(out)

Output: Sorry for disturbance, I find the mistake. I missed x= in x=x.view(-1,self.num_flat_features(x))...


Instruction: PIL image resize change the value of pixel?

Input:
I want to resize an PIL image without changing the pixel value range.
I have tried the Image.resize() but it changes my pixel value range from [0,255] to [79,179]

I'm using Python and PyTorch, in PyTorch, the transforms.resize() will implement Image.resize()

Here is the test code I used

a = torch.randint(0,255,(500,500), dtype=torch.uint8)
print(a.size())
print(torch.max(a))
a = torch.unsqueeze(a, dim =0)
print(a.size())
compose = transforms.Compose([transforms.ToPILImage(),transforms.Resize((128,128))])
a_trans = compose(a)
print(a_trans.size)
print(a_trans.getextrema())

The output of the code is:

torch.Size([500, 500])
tensor(254, dtype=torch.uint8)
torch.Size([1, 500, 500])
(128, 128)
(79, 179)

So someone could explain why the output range is [79,179]?
If I want to do the resize without changing the value range, what should I do?
Thank you in advance

Output: Try resize with Nearest neighbor algorithm implemented in pil. It doesn't change pixels


Instruction: How to fill in the blank using bidirectional RNN and pytorch?

Input:
I am trying to fill in the blank using a bidirectional RNN and pytorch.

The input will be like: The dog is _____, but we are happy he is okay.

The output will be like:

1. hyper (Perplexity score here)
2. sad (Perplexity score here)
3. scared (Perplexity score here)

I discovered this idea here: https://medium.com/@plusepsilon/the-bidirectional-language-model-1f3961d1fb27

import torch, torch.nn as nn
from torch.autograd import Variable

text = ['BOS', 'How', 'are', 'you', 'EOS']
seq_len = len(text)
batch_size = 1
embedding_size = 1
hidden_size = 1
output_size = 1

random_input = Variable(
    torch.FloatTensor(seq_len, batch_size, embedding_size).normal_(), requires_grad=False)

bi_rnn = torch.nn.RNN(
    input_size=embedding_size, hidden_size=hidden_size, num_layers=1, batch_first=False, bidirectional=True)

bi_output, bi_hidden = bi_rnn(random_input)

# stagger
forward_output, backward_output = bi_output[:-2, :, :hidden_size], bi_output[2:, :, hidden_size:]
staggered_output = torch.cat((forward_output, backward_output), dim=-1)

linear = nn.Linear(hidden_size * 2, output_size)

# only predict on words
labels = random_input[1:-1]

# for language models, use cross-entropy :)
loss = nn.MSELoss()
output = loss(linear(staggered_output), labels)

I am trying to reimplement the code above found at the bottom of the blog post. I am new to pytorch and nlp, and can't understand what the input and output to the code is.

Question about the input: I am guessing the input are the few words that are given. Why does one need beginning of sentence and end of sentence tags in this case? Why don't I see the input being a corpus on which the model is trained like other classic NLP problems? I would like to use the Enron email corpus to train the RNN.

Question about the output: I see the output is a tensor. My understanding is the tensor is a vector, so maybe a word vector in this case. How can you use the tensor to output the words themselves?

Output:
As this question is rather open-ended I will start from the last parts, moving towards the more general answer to the main question posed in the title.
Quick note: as pointed in the comments by @Qusai Alothman, you should find a better resource on the topic, this one is rather sparse when it comes to necessary informations.
Additional note: full code for the process described in the last section would take way too much space to provide as an exact answer, it would be more of a blog post. I will highlight possible steps one should take to create such a network with helpful links as we go along.
Final note: If there is anything dumb down there below (or you would like to expand the answer in any way or form, please do correct me/add info by posting a comment below).
Question about the input
Input here is generated from the random normal distribution and has no connection to the actual words. It is supposed to represent word embeddings, e.g. representation of words as numbers carrying semantic (this is important!) meaning (sometimes depending on the context as well (see one of the current State Of The Art approaches, e.g. BERT)).
Shape of the input
In your example it is provided as:
seq_len, batch_size, embedding_size,
where

seq_len - means length of a single sentence (varies across your
dataset), we will get to it later.
batch_size - how many sentences
should be processed in one step of forward pass (in case of
PyTorch it is the forward method of class inheriting from
torch.nn.Module)
embedding_size - vector with which one word is represented (it
might range from the usual 100/300 using word2vec up to 4096 or
so using the more recent approaches like the BERT mentioned
above)

In this case it's all hard-coded of size one, which is not really useful for a newcomer, it only outlines the idea that way.
Why does one need beginning of sentence and end of sentence tags in this case?
Correct me if I'm wrong, but you don't need it if your input is separated into sentences. It is used if you provide multiple sentences to the model, and want to indicate unambiguously the beginning and end of each (used with models which depend on the previous/next sentences, it seems to not be the case here). Those are encoded by special tokens (the ones which are not present in the entire corpus), so neural network "could learn" they represent end and beginning of sentence (one special token for this approach would be enough).
If you were to use serious dataset, I would advise to split your text using libraries like spaCy or nltk (the first one is a pleasure to use IMO), they do a really good job for this task.
You dataset might be already splitted into sentences, in those cases you are kind of ready to go.
Why don't I see the input being a corpus on which the model is trained like other classic NLP problems?
I don't recall models being trained on the corpuses as is, e.g. using strings. Usually those are represented by floating-points numbers using:

Simple approaches, e.g. Bag Of
Words or
TF-IDF
More sophisticated ones, which provide some information about word
relationships (e.g. king is more semantically related to queen
than to a, say, banana). Those were already linked above, some
other noticeable might be
GloVe or
ELMo and tons of other creative
approaches.

Question about the output
One should output indices into embeddings, which in turn correspond to words represented by a vector (more sophisticated approach mentioned above).
Each row in such embedding represents a unique word and it's respective columns are their unique representations (in PyTorch, first index might be reserved for the words for which a representation is unknown [if using pretrained embeddings], you may also delete those words, or represent them as aj average of sentence/document, there are some other viable approaches as well).
Loss provided in the example
# for language models, use cross-entropy :)
loss = nn.MSELoss()

For this task it makes no sense, as Mean Squared Error is a regression metric, not a classification one.
We want to use one for classification, so softmax should be used for multiclass case (we should be outputting numbers spanning [0, N], where N is the number of unique words in our corpus).
PyTorch's CrossEntropyLoss already takes logits (output of last layer without activation like softmax) and returns loss value for each example. I would advise this approach as it's numerically stable (and I like it as the most minimal one).
I am trying to fill in the blank using a bidirectional RNN and pytorch
This is a long one, I will only highlight steps I would undertake in order to create a model whose idea represents the one outlined in the post.
Basic preparation of dataset
You may use the one you mentioned above or start with something easier like 20 newsgroups from scikit-learn.
First steps should be roughly this:

scrape the metadata (if any) from your dataset (those might be HTML tags, some headers etc.)
split your text into sentences using a pre-made library (mentioned above)

Next, you would like to create your target (e.g. words to be filled) in each sentence.
Each word should be replaced by a special token (say <target-token>) and moved to target.
Example:

sentence: Neural networks can do some stuff.

would give us the following sentences and it's respective targets:

sentence: <target-token> networks can do some stuff. target: Neural
sentence: Neural <target-token> can do some stuff. target: networks
sentence: Neural networks <target-token> do some stuff. target: can
sentence: Neural networks can <target-token> some stuff. target: do
sentence: Neural networks can do <target-token> stuff. target: some
sentence: Neural networks can do some <target-token>. target: some
sentence: Neural networks can do some stuff <target-token> target: .

You should adjust this approach to the problem at hand by correcting typos if there are any, tokenizing, lemmatizing and others, experiment!
Embeddings
Each word in each sentence should be replaced by an integer, which in turn points to it embedding.
I would advise you to use a pre-trained one. spaCy provides word vectors, but another interesting approach I would highly recommend is in the open source library flair.
You may train your own, but it would take a lot of time + a lot of data for unsupervised training, and I think it is way beyond the scope of this question.
Data batching
One should use PyTorch's torch.utils.data.Dataset and torch.utils.data.DataLoader.
In my case, a good idea is was to provide custom collate_fn to DataLoader, which is responsible for creating padded batches of data (or represented as torch.nn.utils.rnn.PackedSequence already).
Important: currently, you have to sort the batch by length (word-wise) and keep the indices able to "unsort" the batch into it's original form, you should remember that during implementation. You may use torch.sort for that task. In future versions of PyTorch, there is a chance, one might not have to do that, see this issue.
Oh, and remember to shuffle your dataset using DataLoader, while we're at it.
Model
You should create a proper model by inheriting from torch.nn.Module. I would advise you to create a more general model, where you can provide PyTorch's cells (like GRU, LSTM or RNN), multilayered and bidirectional (as is described in the post).
Something along those lines when it comes to model construction:
import torch

class Filler(torch.nn.Module):
    def __init__(self, cell, embedding_words_count: int):
        self.cell = cell
        # We want to output vector of N
        self.linear = torch.nn.Linear(self.cell.hidden_size, embedding_words_count)

    def forward(self, batch):
        # Assuming batch was properly prepared before passing into the network
        output, _ = self.cell(batch)
        # Batch shape[0] is the length of longest already padded sequence
        # Batch shape[1] is the length of batch, e.g. 32
        # Here we create a view, which allows us to concatenate bidirectional layers in general manner
        output = output.view(
            batch.shape[0],
            batch.shape[1],
            2 if self.cell.bidirectional else 1,
            self.cell.hidden_size,
        )

        # Here outputs of bidirectional RNNs are summed, you may concatenate it
        # It makes up for an easier implementation, and is another often used approach
        summed_bidirectional_output = output.sum(dim=2)
        # Linear layer needs batch first, we have to permute it.
        # You may also try with batch_first=True in self.cell and prepare your batch that way
        # In such case no need to permute dimensions
        linear_input = summed_bidirectional_output.permute(1, 0, 2)
        return self.linear(embedding_words_count)

As you can see, information about shapes can be obtained in a general fashion. Such approach will allow you to create a model with how many layers you want, bidirectional or not (batch_first argument is problematic, but you can get around it too in a general way, left it out for improved clarity), see below:
model = Filler(
    torch.nn.GRU(
        # Size of your embeddings, for BERT it could be 4096, for spaCy's word2vec 300
        input_size=300,
        hidden_size=100,
        num_layers=3,
        batch_first=False,
        dropout=0.4,
        bidirectional=True,
    ),
    # How many unique words are there in your dataset
    embedding_words_count=10000,
)

You may pass torch.nn.Embedding into your model (if pretrained and already filled), create it from numpy matrix or plethora of other approaches, it's highly dependent how your structure your code exactly. Still, please, make your code more general, do not hardcode shapes unless it's totally necessary (usually it's not).
Remember it's only a showcase, you will have to tune and fix it on your own.
This implementation returns logits and no softmax layer is used. If you wish to calculate perplexity, you may have to add it in order to obtain a correct probability distribution across all possible vectors.
BTW: Here is some info on concatenation of bidirectional output of RNN.
Model training
I would highly recommend PyTorch ignite as it's quite customizable, you can log a lot of info using it, perform validation and abstract cluttering parts like for loops in training.
Oh, and split your model, training and others into separate modules, don't put everything into one unreadable file.
Final notes
This is the outline of how I would approach this problem, you may have more fun using attention networks instead of merely using the last output layer as in this example, though you shouldn't start with that.
And please check PyTorch's 1.0 documentation and do not follow blindly tutorials or blog posts you see online as they might be out of date really fast and quality of the code varies enormously. For example torch.autograd.Variable is deprecated as can be seen in the link.


Instruction: Test set accuracy is very high after very few epochs on mnist dataset

Input:
With very few epochs this model learns to classify beween 1 and 0 extremely quickly which leads me to consider something is wrong.

Below code downloads mnist dataset, extracts the mnist images that contain 1 or 0 only. A random sample of size 200 is selected from this subset of mnist images. This random sample is the dataset the model is trained on. With just 2 epochs the model achieves 90%+ test set accuracy, is this expected behaviour ? I expected many more epochs would be required in order to train the model to achieve this level of test set accuracy.

Model code :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader_subset):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % print_progress_every_n_epochs == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

predicted_test = []
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
probs_l = []

predicted_values = []
actual_values = []
labels_l = []

with torch.no_grad():
    for images, labels in test_loader_subset:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        predicted_test.append(predicted.cpu().numpy())

        sm = torch.nn.Softmax()
        probabilities = sm(outputs)
        probs_l.append(probabilities)
        labels_l.append(labels.cpu().numpy())

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

if (epoch) % 1 == 0:
    print('test accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

Output of model (12665 & 2115 represents the training and test set sizes) :

12665
2115
Epoch [1/2], Step [50/198], Loss: 0.1256
Epoch [2/2], Step [50/198], Loss: 0.0151
test accuracy :  99.76359338061465

/anaconda3/envs/pytorch/lib/python3.7/site-packages/ipykernel_launcher.py:143: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.

Output:
Here's my 2 cents on your binary experiment.

It would seem like you have severely reduce the complexity of your dataset and with the high number of neurons in your intermediate layers, your model is expected to converge very quickly.

Note that MNIST dataset has channel of 1 and this makes the task very simple.

You may try to play with CIFAR10 and see if you are still getting high accuracy in just 2 epochs.


Instruction: Impact of using data shuffling in Pytorch dataloader

Input:
I implemented an image classification network to classify a dataset of 100 classes by using Alexnet as a pretrained model and changing the final output layers.
I noticed when I was loading my data like

trainloader = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=False)

, I was getting accuracy on validation dataset around 2-3 % for around 10 epochs but when I just changed shuffle=True and retrained the network, the accuracy jumped to 70% in the first epoch itself.

I was wondering if it happened because in the first case the network was being shown one example after the other continuously for just one class for few instances resulting in network making poor generalizations during training or is there some other reason behind it?

But, I did not expect that to have such a drastic impact.

P.S: All the code and parameters were exactly the same for both the cases except changing the shuffle option.

Output:
Yes it totally can affect the result! Shuffling the order of the data that we use to fit the classifier is so important, as the batches between epochs do not look alike.
Checking the Data Loader Documentation it says:
"shuffle (bool, optional) – set to True to have the data reshuffled at every epoch"
In any case, it will make the model more robust and avoid over/underfitting.
In your case this heavy increase of accuracy (from the lack of awareness of the dataset) probably is due to how the dataset is "organised" as maybe, as an example, each category goes to a different batch, and in every epoch, a batch contains the same category, which derives to a very bad accuracy when you are testing.


Instruction: Unable to get an image in the output when using MNIST Database using Pytorch

Input:
iter_test = 0

for images, labels in test_loader:

iter_test += 1

images = images.view(-1, 28*28)

outputs = model(images)
_, predicted = torch.max(outputs.data, 1)

if iter_test == 1:
    print('PREDICTION')
    print(predicted[0])

    print('LABEL SIZE')
    print(labels.size())

    print('LABEL FOR IMAGE 0')
    print(labels[0])

I get the output of this code as follows:

    PREDICTION

    tensor(7)

    LABEL SIZE

    torch.Size([100])

    LABEL FOR IMAGE 0

    tensor(7)

The code works perfectly. I was wondering if i could get the "MNIST" like image as output along with prediction?

Output:
Yes :).

You can use pyplot and show the image loaded by test_loader.

Check https://www.oreilly.com/learning/not-another-mnist-tutorial-with-tensorflow .

Hope this helps!


Instruction: Is there any other reason why we make sequence length the same using padding?

Input: Is there any other reason why we make sequence length the same length using padding? Other than in order to do matrix multiplication (therefore doing parallel computation).

Output:
It may depend on the specific situation you are dealing with. But in general, the only reason I would do zero padding or any kind of padding to RNN would be to make batch-wise computations work. Also, padding should be done in a way that it doesn't affect the results. So, it should not contribute to computing hidden state computation that you would be using for downstream tasks. For example, you may pad the end of the particular sequences from {t+1:T}, but then for further task or processing we should use only h{0:t}

However, if you are doing anything different than simple RNN (for eg. bidirectional-RNN), it can be complicated to do padding. For example: for the forward direction you would pad in the end and for the reverse direction, you would want to pad the front part of sequences.

Even for batching or doing parallel computations pytorch has packed sequences which should be faster than padding IMO.


Instruction: Pytorch: Why is the memory occupied by the `tensor` variable so small?

Input:
In Pytorch 1.0.0, I found that a tensor variable occupies very small memory. I wonder how it stores so much data.
Here's the code.

a = np.random.randn(1, 1, 128, 256)
b = torch.tensor(a, device=torch.device('cpu'))

a_size = sys.getsizeof(a)
b_size = sys.getsizeof(b)

a_size is 262288. b_size is 72.

Output:
The answer is in two parts. From the documentation of sys.getsizeof, firstly

All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific.

so it could be that for tensors __sizeof__ is undefined or defined differently than you would expect - this function is not something you can rely on. Secondly

Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.

which means that if the torch.Tensor object merely holds a reference to the actual memory, this won't show in sys.getsizeof. This is indeed the case, if you check the size of the underlying storage instead, you will see the expected number
import torch, sys
b = torch.randn(1, 1, 128, 256, dtype=torch.float64)
sys.getsizeof(b)
>> 72
sys.getsizeof(b.storage())
>> 262208

Note: I am setting dtype to float64 explicitly, because that is the default dtype in numpy, whereas torch uses float32 by default.


Instruction: os.mknod returns [error38] function not implemented in google colab

Input:
I am trying to run the following piece of code on google colab.

dir_path = '/content/drive/My Drive/Colab Notebooks'
log_loss_path =os.path.join(dir_path, 'log_loss.txt')
if not os.path.isfile(log_loss_path):
    os.mknod(log_loss_path)

but i get the error [Errno 38] Function not implemented

    OSError                                   Traceback (most recent call last)
<ipython-input-15-bd3880e6bb8b> in <module>()
      2 log_loss_path = os.path.join(dir_path, 'log_loss.txt')
      3 if not os.path.isfile(log_loss_path):
----> 4     os.mknod(log_loss_path)

OSError: [Errno 38] Function not implemented

can anyone help to solve it?

Output:
/content/drive is a FUSE filesystem which doesn't support this operation.

If you are just trying to create a file, use instead open(log_loss_path, 'w').


Instruction: By picture that is displayed in Jupyter notebook isn't shown when running the same code in IDE?

Input:
I was trying to run code from an online tutorial on my local machine by copying code from Jupiter notebook to my IDE (pycharm).

This part

import numpy as np
import matplotlib.pyplot as plt
import matplotlib
from PIL import Image
from torchvision import transforms
import matplotlib.pyplot as plt

pig_img = Image.open("pig.jpg")
preprocess = transforms.Compose([
   transforms.Resize(224),
   transforms.ToTensor(),
])
pig_tensor = preprocess(pig_img)[None,:,:,:]

plt.imshow(pig_tensor[0].numpy().transpose(1,2,0))

While Jupiter notebook shows the imagine - I cannot get it displayed when running in terminal or IDE.

Any ideas why?

Thank you

Output:
You need to call .show() explicitly to show the image in terminal i.e.
Add this to the end of the code
plt.show()

From the documentation:

Display a figure. When running in ipython with its pylab mode, display all figures and return to the ipython prompt.
In non-interactive mode, display all figures and block until the figures have been closed; in interactive mode it has no effect unless figures were created prior to a change from non-interactive to interactive mode (not recommended). In that case it displays the figures but does not block.


Instruction: What kinds of optimization are used in PyTorch methods?

Input: I'm using PyTorch to implement an intense sequence of matrix operations, using methods such as torch.mm or torch.dot. I was wondering if PyTorch uses multithreading or other optimization mechanisms to speed up the process. I am not utilizing a GPU. I appreciate if you could inform me of how fast these methods are and whether I need to take any actions to help the process.

Output:
PyTorch uses an efficient BLAS implementation and multithreading (openMP, if I'm not wrong) to parallelize such operations with multiple cores. Some performance loss comes from the Python itself - since this is an interpreted language, no significant compiler-like optimization can be done. You can use the jit module to speed up the "wrapper" code around the matrix multiplies, but for anything more than very small matrices this cost is probably negligible.

One big improvement you may be able to get manually, but which PyTorch doesn't apply automatically, is to properly order the matrix multiplies. As you probably know, depending on matrix shapes, a multiplication ABCD may have different performance computed as A(B(CD)) than if computed as (AB)(CD), etc.


Instruction: Pytorch RuntimeError: Invalid index in gather

Input:
I'm new to Pytorch and I encounter this error:

  x.gather(1, c)

  RuntimeError: Invalid index in gather at
  /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:457

Here is some informations about the tensors:

print(x.size())
print(c.size())
print(type(x))
print(type(c))

torch.Size([128, 2])
torch.Size([128, 1])
<class 'torch.Tensor'>
<class 'torch.Tensor'>

x is filled with float values and c with integers, could it be the problem?

Output:
This simply means your index tensor c has invalid indices.
For example, the following index tensor is valid:

        x = torch.tensor([
        [5, 9, 1],
        [3, 2, 8],
        [7, 4, 0]
    ])
    c = torch.tensor([
        [0, 0, 0],
        [1, 2, 0],
        [2, 2, 1]
    ])
    x.gather(1, c)
>>>tensor([[5, 5, 5],
        [2, 8, 3],
        [0, 0, 4]])

However, the following index tensor is invalid:

c = torch.tensor([
    [0, 0, 0],
    [1, 2, 0],
    [2, 2, 3]
])

And it gives the exception you mention

  RuntimeError: Invalid index in gather


Instruction: Forward Jacobian Of Neural Network in Pytorch is Slow

Input:
I am computing the forward jacobian (derivative of outputs with respect to inputs) of a 2 layer feedforward neural network in pytorch, and my results are correct but relatively slow. Given the nature of the calculation I would expect it to be approximately as fast as a forward pass through the network (or maybe 2-3x as long), but it takes ~12x as long to run an optimization step on this routine (in my test example I just want the jacobian=1 at all points) vs the standard mean squared error so I assume I am doing something in an un-optimal manner. I'm just wondering if anyone knew a faster way to code this. My test network has 2 input nodes, followed by 2 hidden layers of 5 nodes each and an output layer of 2 nodes, and uses tanh activation functions on the hidden layers, with a linear output layer.

The Jacobian calculations are based on the paper The Limitations of Deep Learning in Adversarial Settings which gives a basic recursive definition of the forward derivative (basically you end up multiplying the derivative of your activation functions with the weights and previous partial derivatives of each layer). This is very similar to forward propagation, which is why I would expect it to be faster than it is. Then the determinant of the 2x2 jacobian at the end is pretty straightforward.

Below is the code for the network and the jacobian

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.h_1_1 = torch.nn.Linear(input_1, hidden_1)
        self.h_1_2 = torch.nn.Linear(hidden_1, hidden_2)
        self.out = torch.nn.Linear(hidden_2, out_1)

    def forward(self, x):
        x = F.tanh(self.h_1_1(x))
        x = F.tanh(self.h_1_2(x))
        x = (self.out(x))
        return x

    def jacobian(self, x):
        a = self.h_1_1.weight
        x = F.tanh(self.h_1_1(x))
        tanh_deriv_tensor = 1 - (x ** 2)
        expanded_deriv = tanh_deriv_tensor.unsqueeze(-1).expand(-1, -1, input_1)
        partials = expanded_deriv * a.expand_as(expanded_deriv)

        a = torch.matmul(self.h_1_2.weight, partials)
        x = F.tanh(self.h_1_2(x))
        tanh_deriv_tensor = 1 - (x ** 2)
        expanded_deriv = tanh_deriv_tensor.unsqueeze(-1).expand(-1, -1, out_1)
        partials = expanded_deriv*a

        partials = torch.matmul(self.out.weight, partials)

        determinant = partials[:, 0, 0] * partials[:, 1, 1] - partials[:, 0, 1] * partials[:, 1, 0]
        return determinant

and here are the two error functions being compared. Note that the first one requires an extra forward call through the network, to get the output values (labeled action) while the second function does not since it works on the input values.

def actor_loss_fcn1(action, target):
    loss = ((action-target)**2).mean()
    return loss

def actor_loss_fcn2(input): # 12x slower
    jacob = model.jacobian(input)
    loss = ((jacob-1)**2).mean()
    return loss

Any insight on this would be greatly appreciated

Output:
The second calculation of 'a' takes the most time on my machine (cpu).

# Here you increase the size of the matrix with a factor of "input_1"
expanded_deriv = tanh_deriv_tensor.unsqueeze(-1).expand(-1, -1, input_1)
partials = expanded_deriv * a.expand_as(expanded_deriv)

# Here your torch.matmul() needs to handle "input_1" times more computations than in a normal forward call
a = torch.matmul(self.h_1_2.weight, partials)

On my machine the time of computing the Jacobian is roughly the time it takes torch to compute

a = torch.rand(hidden_1, hidden_2)
b = torch.rand(n_inputs, hidden_1, input_1)
%timeit torch.matmul(a,b)

I don't think it is possible to speed this up, computationally wise. Unless you can move from CPU to GPU, because GPU get better on larges matrices.


Instruction: How to fix ' ImportError: cannot import name 'numpy_type_map' ' in Python?

Input:
I've followed the instructions in Detectron and I've configured it several times: the code compiles as it should. When it comes to run the code, I get this error:

    Traceback (most recent call last):
    File "tools/train_net_step.py", line 21, in <module>
    import nn as mynn
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/__init__.py", line 2, in <module>
    from .parallel import DataParallel
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/__init__.py", line 3, in <module>
    from .data_parallel import DataParallel, data_parallel
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/data_parallel.py", line 4, in <module>
    from .scatter_gather import scatter_kwargs, gather
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/scatter_gather.py", line 8, in <module>
    from torch.utils.data.dataloader import numpy_type_map
    ImportError: cannot import name 'numpy_type_map'

I've also tried to google it many times, but I can't find a way to solve it. What can I do? I'm using PyTorch 0.4.1 and pytorch nightly 1.0.0-dev.

EDIT: Thanks to sancelot, I managed to solve that error (PyTorch 0.4.0 did the thing). Anyway, now I've got another error:

    Traceback (most recent call last):
    File "tools/train_net_step.py", line 27, in <module>
    from modeling.model_builder import Generalized_RCNN
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/modeling/model_builder.py", line 11, in <module>
    from model.roi_pooling.functions.roi_pool import RoIPoolFunction
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/functions/roi_pool.py", line 3, in <module>
    from .._ext import roi_pooling
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/_ext/roi_pooling/__init__.py", line 3, in <module>
    from ._roi_pooling import lib as _lib, ffi as _ffi
    ImportError: /home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/_ext/roi_pooling/_roi_pooling.so: undefined symbol: PyInt_FromLong

What I can't get this time is: is this an error given by an external library? I'm using an anaconda environment previously made by my professor, who has used it for Detectron... so I can't guess why I get this.

Output:
I suppose there is a version mismatch between detectron and the needed pytorch release you are using.
if you look at latest pytorch source code, there is no numpy_type_map component.
https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py