Instruction: Distrubuted PyTorch code halts on multiple nodes when using MPI backend

Input:
I am trying to run Pytorch code on three nodes using openMPI but the code just halts without any errors or output. Eventually my purpose is to distribute a Pytorch graph on these nodes.

Three of my nodes are connected in same LAN and have SSH access to each other without password and have similar specifications:

Ubuntu 18.04
Cuda 10.0
OpenMPI built and installed from source
PyTorch built and installed from source

The code shown below works on single node - multiple processes, as:

> mpirun -np 3 -H 192.168.100.101:3 python3 run.py

With following output:

INIT 0 of 3 Init env://
INIT 1 of 3 Init env://
INIT 2 of 3 Init env://
RUN 0 of 3 with tensor([0., 0., 0.])
RUN 1 of 3 with tensor([0., 0., 0.])
RUN 2 of 3 with tensor([0., 0., 0.])
Rank  1  has data  tensor(1.)
Rank  0  has data  tensor(1.)
Rank  2  has data  tensor(1.)

But when I placed the code on three nodes and run following command on each node separately, it does nothing:

> mpirun -np 3 -H 192.168.100.101:1,192.168.100.102:1,192.168.100.103:1 python3 run.py

Please give some idea about any modifications in code or configurations for MPI to run given Pytorch code on multiple nodes?

#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    tensor = torch.zeros(size)
    print(f"RUN {rank} of {size} with {tensor}")

    # incrementing the old tensor
    tensor += 1

    # sending tensor to next rank
    if rank == size-1:
       dist.send(tensor=tensor, dst=0)
    else:
       dist.send(tensor=tensor, dst=rank+1)

    # receiving tensor from previous rank
    if rank == 0:
        dist.recv(tensor=tensor, src=size-1)
    else:
        dist.recv(tensor=tensor, src=rank-1)

    print('Rank ', rank, ' has data ', tensor[0])

def init_processes(rank, size, fn, backend, init):
    print(f"INIT {rank} of {size} Init {init}")
    dist.init_process_group(backend, init, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    os.environ['MASTER_ADDR'] = '192.168.100.101'
    os.environ['BACKEND'] = 'mpi'
    os.environ['INIT_METHOD'] = 'env://'
    world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
    world_rank = int(os.environ['OMPI_COMM_WORLD_RANK'])

    init_processes(world_rank, world_size, run, os.environ['BACKEND'], os.environ['INIT_METHOD'])

N.B. NCCL is not an option for me due to arm64-based hardware.

Output:
Apologies for replying late to this, but I could solve the issue by adding --mca btl_tcp_if_include eth1 flag to mpirun command.

The reason for halt was that openMPI, by default, tries to locate and communicate with other nodes over local loopback network interface e.g. lo. We have to explicitly specify which interface(s) should be included (or excluded) to locate other other nodes.

I hope it would save someone's day :)


Instruction: Size mismatch for fc.bias and fc.weight in PyTorch

Input:
I used the transfer learning approach to train a model and saved the best-detected weights. In another script, I tried to use the saved weights for prediction. But I am getting errors as follows. I have used ResNet for finetuning the network and have 4 classes.

RuntimeError: Error(s) in loading state_dict for ResNet:
size mismatch for fc.bias: copying a param of torch.Size([1000]) from
checkpoint, where the shape is torch.Size([4]) in current model.
size mismatch for fc.weight: copying a param of torch.Size([1000,
512]) from checkpoint, where the shape is torch.Size([4, 512]) in
current model.

I am using the following code for prediction of output:

checkpoint = torch.load("./models/custom_model13.model")
model = resnet18(pretrained=True)

model.load_state_dict(checkpoint)
model.eval()

def predict_image(image_path):
    transformation = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
    image_tensor = transformation(image).float()
    image_tensor = image_tensor.unsqueeze_(0)

    if torch.cuda.is_available():
        image_tensor.cuda()

    input = Variable(image_tensor)
    output = model(input)

    index = output.data.numpy().argmax()
    return index

if __name__ == "main":
    imagefile = "image.png"
    imagepath = os.path.join(os.getcwd(),imagefile)
    prediction = predict_image(imagepath)
    print("Predicted Class: ",prediction)

And the following code to train and save the model:

Data_dir = 'Dataset'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print (device)

def save_models(epochs, model):
    torch.save(model.state_dict(), "custom_model{}.model".format(epochs))
    print("Checkpoint Saved")

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'train' and epoch_acc > best_acc:
                save_models(epoch,model)
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 4)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

Output:
Cause:

You trained a model derived from resnet18 in this way:

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 4)

That is, you changed the last nn.Linear layer to output 4 dim prediction instead of the default 1000.
When you try and load the model for prediction, your code is:

model = resnet18(pretrained=True)
model.load_state_dict(checkpoint)

You did not apply the same change of the last nn.Linear layer to model therefore the checkpoint you are trying to load does not fit.

Fix:

(1)  Apply the same change before loading the checkpoint:

model = resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 4)  # make the change
model.load_state_dict(checkpoint)  # load

(2) Even better, use num_classes argument to construct resnet with the desired number of outputs to begin with:

model = resnet18(pretrained=True, num_classes=4)
model.load_state_dict(checkpoint)  # load


Instruction: Copy GpuMat to CUDA Tensor

Input:
I am trying to run model inference in C++.
I succesfully traced model in Python with torch.jit.trace.
I am able to load model in C++ using torch::jit::load().
I was able to perform inference both on cpu and gpu, however the starting point was always torch::from_blob method which seems to be creating cpu-side tensor.
For efficiency, I would like to cast/copy cv::cuda::GpuMat directly to CUDA Tensor. I have been digging through pytorch tests and docs in search of convinient example, but was unable to find one.

Question:
How to create CUDA Tensor from cv::cuda::GpuMat?

Output:
Here is an example:

//define the deleter ...
void deleter(void* arg) {};

//your convert function

cuda::GpuMat gImage;

//build or load your image here ...

std::vector<int64_t> sizes = {1, static_cast<int64_t>(gImage.channels()),
                          static_cast<int64_t>(gImage.rows),
                          static_cast<int64_t>(gImage.cols)};

long long step = gImage.step / sizeof(float);

std::vector<int64_t> strides = {1, 1, step, static_cast<int64_t>(gImage.channels())};

auto tensor_image = torch::from_blob(gImage.data, sizes, strides, deleter,  torch::kCUDA);
std::cout << "output tensor image : " << tensor_image << std::endl;


Instruction: Auto-encoder for vector encodings

Input:
Here is an autoencoder I wrote to encode two vectors :

[1,2,3] & [1,2,3]

The vectors are created in : features = torch.tensor(np.array([ [1,2,3],[1,2,3] ]))

This works as per code :

%reset -f

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3] ]))

print(features)

batch = 2
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=True)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

        encoded_images.append(encoder(images))

But when I add a new vector :

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[1,2,3] ]))

I receive error :

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-223-3ca45519e975> in <module>
     32 encoded_images = []
     33 for i in range(epochs):
---> 34     for j, (images, _) in enumerate(data_loader):
     35     #     images = images.view(images.size(0), -1)
     36         images = Variable(images).type(FloatTensor)

ValueError: not enough values to unpack (expected 2, got 1)

Have I setup my data loader correctly ?

Output: your Dataset (inside DataLoader) returns only image per item without label. When you iterate and expecting each item to be (image, _) you are trying to unpack a feature without a label into image and _ and this is not possible. This is why you get "not enough values to unpack" error


Instruction: PyTorch optimizer.step() function doesn't update weights

Input:
The code can be seen below.
The problem is, that the optimizer.step() part doesn't work. I'm printing model.parameters() before and after the training, and the weights don't change.

I'm trying to make a perceptron that can solve the AND-problem. I've been successful in doing this with my own tiny library, where I've implemented a perceptron with the two functions predict() and train().

Just to clarify, I've just started learning deep learning using PyTorch, so it's probably a very newbie problem. I've tried searching for a solution, but without luck. I've also compared my code with other codes that work, but I don't know what I'm doing wrong.

import torch
from torch import nn, optim
from random import randint

class NeuralNet(nn.Module):
  def __init__(self):
    super(NeuralNet, self).__init__()
    self.layer1 = nn.Linear(2, 1)

  def forward(self, input):
    out = input
    out = self.layer1(out)
    out = torch.sign(out)
    out = torch.clamp(out, 0, 1) # 0=false, 1=true
    return out

data = torch.Tensor([[0, 0], [0, 1], [1, 0], [1, 1]])
target = torch.Tensor([0, 0, 0, 1])
model = NeuralNet()
epochs = 1000
lr = 0.01

print(list(model.parameters()))
print() # Print parameters before training
loss_func = nn.L1Loss()
optimizer = optim.Rprop(model.parameters(), lr)
for epoch in range(epochs + 1):
  optimizer.zero_grad()
  rand_int = randint(0, len(data) - 1)
  x = data[rand_int]
  y = target[rand_int]

  pred = model(x)
  loss = loss_func(pred, y)

  loss.backward()
  optimizer.step()

# Print parameters again
# But they haven't changed
print(list(model.parameters()))

Output:
Welcome to stackoverflow!

The issue here is you are trying to perform back-propagation through a non-differentiable function. Non-differentiable means that no gradients can flow back through them, implying that all trainable weights applied before them will not be updated by your optimizer. Such functions are easy to spot; they are discrete, sharp operations that resemble 'if' statements. In your case it is the sign() function.

Unfortunately, PyTorch does not do any hand-holding in this regard and will not point you to the issue. What you could do to alleviate the issue would be to transform the range of your output to [-1,1] and apply a Tanh() non-linearity instead of the sign() and clamp() operators.


Instruction: How do I display a single image in PyTorch?

Input:
How do I display a PyTorch Tensor of shape (3, 224, 224) representing a 224x224 RGB image?
Using plt.imshow(image) gives the error:

TypeError: Invalid dimensions for image data

Output:
Given a Tensor representing the image, use .permute() to put the channels as the last dimension:
plt.imshow(  tensor_image.permute(1, 2, 0)  )

Note: permute does not copy or allocate memory, and  from_numpy() doesn't either.


Instruction: Updating pre-trained Deep Learning model with respect to new data points

Input:
Considering the example of Image classification on ImageNet, How to update the pre-trained model using the new data points.
I have loaded the pre-trained model. I have a new data point that is quite different from the distribution of the original data on which the model was previously trained. So, I would like to update/fine-tune the model with the help of new data point. How to go about doing it? Can anyone help me out in doing it? I am using pytorch 0.4.0 for implementation, running on GPU Tesla K40C.

Output:
If you don't want to change the output of the classifier (i.e. the number of classes), then you can simply continue training the model with new example images, assuming that they are reshaped to the same shape that the pretrained model accepts.

On the other hand, if you want to change the number of classes in a pre-trained model, then you can replace the last fully connected layer with a new one and train only this specific layer on new samples. Here's a sample code for this case from PyTorch's autograd mechanics notes:

model = torchvision.models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
# Replace the last fully-connected layer
# Parameters of newly constructed modules have requires_grad=True by default
model.fc = nn.Linear(512, 100)

# Optimize only the classifier
optimizer = optim.SGD(model.fc.parameters(), lr=1e-2, momentum=0.9)


Instruction: Differences between `torch.Tensor` and `torch.cuda.Tensor`

Input:
We can allocate a tensor on GPU using torch.Tensor([1., 2.], device='cuda'). Are there any differences using that way rather than torch.cuda.Tensor([1., 2.]), except we can pass in a specific CUDA device to the former one?

Or in other words, in which scenario is torch.cuda.Tensor() necessary?

Output:
So generally both torch.Tensor and torch.cuda.Tensor are equivalent. You can do everything you like with them both.

The key difference is just that torch.Tensor occupies CPU memory while torch.cuda.Tensor occupies GPU memory. Of course operations on a CPU Tensor are computed with CPU while operations for the GPU / CUDA Tensor are computed on GPU.

The reason you need these two tensor types is that the underlying hardware interface is completely different. Apart from the point it doesn't make sense computationally, you will get an error as soon as you try to do computations between torch.Tensor and torch.cuda.Tensor:

import torch

# device will be 'cuda' if a GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# creating a CPU tensor
cpu_tensor = torch.rand(10)
# moving same tensor to GPU
gpu_tensor = cpu_tensor.to(device)

print(cpu_tensor, cpu_tensor.dtype, type(cpu_tensor), cpu_tensor.type())
print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())

print(cpu_tensor*gpu_tensor)

Output:

tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619]) torch.float32 <class 'torch.Tensor'> torch.FloatTensor
tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619], device='cuda:0') torch.float32 <class 'torch.Tensor'> torch.cuda.FloatTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ac794171c178> in <module>()
     12 print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())
     13
---> 14 print(cpu_tensor*gpu_tensor)

RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'other'

As the underlying hardware interface is completely different, CPU Tensors are just compatible with CPU Tensor and verse visa GPU Tensors are just compatible to GPU Tensors.

Edit:

As you can see here that a tensor which is moved to GPU is actually a tensor of type: torch.cuda.*Tensor i.e. torch.cuda.FloatTensor.

So cpu_tensor.to(device) or torch.Tensor([1., 2.], device='cuda') will actually return a tensor of type torch.cuda.FloatTensor.

In which scenario is torch.cuda.Tensor() necessary?

When you want to use GPU acceleration (which is much faster in most cases) for your program, you need to use torch.cuda.Tensor, but you have to make sure that ALL tensors you are using are CUDA Tensors, mixing is not possible here.


Instruction: Why does this error pop up while working with Deep Q learning?

Input:
I have been working with Deep Q Learning on Windows 10 Machine. I have version 0.4.1 of pytorch with NVIDA graphics card.

def select_action(self, state):
    probs = F.softmax(self.model(Variable(state, volatile = True))*7)
    action = probs.multinomial()
    return action.data[0,0]

From this section of the code, I keep getting this error:

TypeError: multinomial() missing 1 required positional arguments: "num_samples"

If any other information is needed, It will be very quickly provided.

Output:
Based on the documentation you didn't specify the num_samples of multinomial function to draw your multinomial distribution.

  torch.multinomial(input, num_samples, replacement=False, out=None)

    Returns a tensor where each row contains num_samples
    indices sampled from the multinomial probability distribution located
    in the corresponding row of tensor input.

Change the code as below:

def select_action(self, state):
    probs = F.softmax(self.model(Variable(state, volatile = True))*7)
    action = probs.multinomial(1) # 1 is the number of samples to draw
    return action.data[0,0]


Instruction: Unexpected increase in validation error in MNIST Pytorch

Input:
I'm a bit new to the whole field and thus decided to work on the MNIST dataset. I pretty much adapted the whole code from https://github.com/pytorch/examples/blob/master/mnist/main.py, with only one significant change: Data Loading. I didn't want to use the pre-loaded dataset within Torchvision. So I used MNIST in CSV.

I loaded the data from CSV file by inheriting from Dataset and making a new dataloader.
Here's the relevant code:

mean = 33.318421449829934
sd = 78.56749081851163
# mean = 0.1307
# sd = 0.3081
import numpy as np
from torch.utils.data import Dataset, DataLoader

class dataset(Dataset):
    def __init__(self, csv, transform=None):
        data = pd.read_csv(csv, header=None)
        self.X = np.array(data.iloc[:, 1:]).reshape(-1, 28, 28, 1).astype('float32')
        self.Y = np.array(data.iloc[:, 0])

        del data
        self.transform = transform

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        item = self.X[idx]
        label = self.Y[idx]

        if self.transform:
            item = self.transform(item)

        return (item, label)

import torchvision.transforms as transforms
trainData = dataset('mnist_train.csv', transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (sd,))
]))
testData = dataset('mnist_test.csv', transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (sd,))
]))

train_loader = DataLoader(dataset=trainData,
                         batch_size=10,
                         shuffle=True,
                         )
test_loader = DataLoader(dataset=testData,
                        batch_size=10,
                        shuffle=True,
                        )

However this code gives me the absolutely weird training error graph that you see in the picture, and a final validation error of 11% because it classifies everything as a '7'.

I managed to track the problem down to how I normalize the data and if I use the values given in the example code (0.1307, and 0.3081) for transforms.Normalize, along with reading the data as type 'uint8' it works perfectly.
Note that there is very minimal difference in the data which is provided in these two cases. Normalizing by 0.1307 and 0.3081 on values from 0 to 1 has the same effect as normalizing by 33.31 and 78.56 on values from 0 to 255. The values are even mostly the same (A black pixel corresponds to -0.4241 in the first case and -0.4242 in the second).

If you would like to see a IPython Notebook where this problem is seen clearly, please check out https://colab.research.google.com/drive/1W1qx7IADpnn5e5w97IcxVvmZAaMK9vL3

I am unable to understand what has caused such a huge difference in behaviour within these two slightly different ways of loading data. Any help would be massively appreciated.

Output:
Long story short: you need to change item = self.X[idx] to item = self.X[idx].copy().

Long story long: T.ToTensor() runs torch.from_numpy, which returns a tensor which aliases the memory of your numpy array dataset.X. And T.Normalize() works inplace, so each time the sample is drawn it has mean subtracted and is divided by std, leading to degradation of your dataset.

Edit: regarding why it works in the original MNIST loader, the rabbit hole is even deeper. The key line in MNIST is that the image is transformed into a PIL.Image instance. The operation claims to only copy in case the buffer is not contiguous (it is in our case), but under the hood it checks whether it's strided instead (which it is), and thus copies it. So by luck, the default torchvision pipeline involves a copy and thus in-place operation of T.Normalize() does not corrupt the in-memory self.data of our MNIST instance.


Instruction: Why was the method of a class called without mentioning the method?

Input:
I am currently going through this pytorch tutorial but I think this problem is one regarding Python classes in general: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py
Specifically, a class called Net was created and we created an object called net=Net(). In the Net class there is a method forward(self,X). However, later forward was used just by writing net(X). Shouldn't it be net.forward(X)?
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

input = torch.randn(1, 1, 32, 32)
out = net(input)
print(out)

Output:
If you check the source code of nn.Module you will see that it implements __call__, which makes its instances (and instances of its subclasses) callable.

def __call__(self, *input, **kwargs):
    for hook in self._forward_pre_hooks.values():
        hook(self, input)
    if torch.jit._tracing:
        result = self._slow_forward(*input, **kwargs)
    else:
        result = self.forward(*input, **kwargs)
    for hook in self._forward_hooks.values():
        hook_result = hook(self, input, result)
        if hook_result is not None:
            raise RuntimeError(
                "forward hooks should never return any values, but '{}'"
                "didn't return None".format(hook))
    if len(self._backward_hooks) > 0:
        var = result
        while not isinstance(var, torch.Tensor):
            if isinstance(var, dict):
                var = next((v for v in var.values() if isinstance(v, torch.Tensor)))
            else:
                var = var[0]
        grad_fn = var.grad_fn
        if grad_fn is not None:
            for hook in self._backward_hooks.values():
                wrapper = functools.partial(hook, self)
                functools.update_wrapper(wrapper, hook)
                grad_fn.register_hook(wrapper)
    return result

That's why

net = Net()
input = torch.randn(1, 1, 32, 32)
out = net(input)

Is a totally valid code. net(input) executes net.__call__(input).


Instruction: VSCode bug with PyTorch DataLoader?

Input:
The following code example works in Python, but fails in VSCode in Linux (but not VSCode in Windows).  I am wondering if there is something wrong with my code, or if there is something wrong with VSCode under Linux?

#Test of PyTorch DataLoader and Visual Studio Code
from torch.utils.data import Dataset, DataLoader

class SimpleData(Dataset):
    """Very simple dataset"""
    def __init__(self):
        self.data = range(20)
    def __len__(self):
        return len(self.data)
    def __getitem__(self, idx):
        return self.data[idx]

if __name__ == '__main__':

    #Initialize DataLoader with above Dataset:
    dataloader = DataLoader(SimpleData(), batch_size=4, num_workers=1)

    print('Using DataLoader to show data in batches: ')
    for i, sample_batch in enumerate(dataloader):    #This fails in VSCode in Linux
        print('batch ', i, ':', sample_batch)

    print("--- Done ---")

The expected output is:

Using DataLoader to show data in batches:
batch  0 : tensor([0, 1, 2, 3])
batch  1 : tensor([4, 5, 6, 7])
batch  2 : tensor([ 8,  9, 10, 11])
batch  3 : tensor([12, 13, 14, 15])
batch  4 : tensor([16, 17, 18, 19])
--- Done ---

But with VSCode in Linux it hangs after printing the first line.

Output:
Have you tried with num_workers=0?

May be VS Code is not able to spawn a new process properly on linux.


Instruction: Implementing a many-to-many regression task

Input:
Sorry if I present my problem not clearly, English is not my first language

Problem

Short description:

I want to train a model which map input x (with shape of [n_sample, timestamp, feature]) to an output y (with exact same shape). It's like mapping 2 space

Longer version:

I have 2 float ndarrays of shape [n_sample, timestamp, feature], representing MFCC feature of n_sample audio file. These 2 ndarray are 2 speakers' speech of the same corpus, which was aligned by DTW.  Lets name these 2 arrays x and y. I want to train a model, which predict y[k] given x[k]. It's like mapping from space x to space y, and the output must be exact same shape as the input

What I've tried

It's time-series problem so I decide to use RNN approach. Here is my code in PyTorch (I put comment along the code. I removed the calculation of average loss for simplicity). Note that I've tried many option for learning rate, the behavior still the same

Class define

class Net(nn.Module):
    def __init__(self, in_size, hidden_size, out_size, nb_lstm_layers):
        super().__init__()
        self.in_size = in_size
        self.hidden_size = hidden_size
        self.out_size = out_size
        self.nb_lstm_layers = nb_lstm_layers

        # self.fc1 = nn.Linear()
        self.lstm = nn.LSTM(input_size=self.in_size, hidden_size=self.hidden_size, num_layers=self.nb_lstm_layers, batch_first=True, bias=True)
        # self.fc = nn.Linear(self.hidden_size, self.out_size)
        self.fc1 = nn.Linear(self.hidden_size, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, self.out_size)

    def forward(self, x, h_state):
        out, h_state = self.lstm(x, h_state)
        output_fc = []

        for frame in out:
            output_fc.append(self.fc3(torch.tanh(self.fc1(frame)))) # I added fully connected layer to each frame, to make an output with same shape as input

        return torch.stack(output_fc), h_state

    def hidden_init(self):
        if use_cuda:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)]).cuda()
        else:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)])

        return h_state

Training step:

net = Net(20, 20, 20, nb_lstm_layers)
optimizer = optim.Adam(net.parameters(), lr=0.0001, weight_decay=0.0001)
criterion = nn.MSELoss()

for epoch in range(nb_epoch):
    count = 0
    loss_sum = 0

    batch_x = None
    for i in (range(len(data))):
    # data is my entire data, which contain A and B i specify above.
        temp_x = torch.tensor(data[i][0])
        temp_y = torch.tensor(data[i][1])

        for ii in range(0, data[i][0].shape[0] - nb_frame_in_batch*2 + 1): # Create batches
            batch_x, batch_y = get_batches(temp_x, temp_y, ii, batch_size, nb_frame_in_batch)
            # this will return 2 tensor of shape (batch_size, nb_frame_in_batch, 20),
            # with `batch_size` is the number of sample each time I feed to the net,
            # nb_frame_in_batch is the number of frame in each sample
            optimizer.zero_grad()

            h_state = net.hidden_init()

            prediction, h_state = net(batch_x.float(), h_state)
            loss = criterion(prediction.float(), batch_y.float())

            h_state = (h_state[0].detach(), h_state[1].detach())

            loss.backward()
            optimizer.step()

Problem is, the loss seems not to decrease but fluctuate a lot, without a clear behaviour

Please help me. Any suggestion will be greatly appreciated. If somebody can inspect my code and provide some comment, that would be so kind.
Thanks in advance!

Output:
It seems the network learning nothing from your data, hence the loss fluctuation (since weights depends on random initialization only). There are something you can try:

Try to normalize the data (this suggestion is quite broad, but I can't give you more details since I don't have your data, but normalize it to a specific range like [0, 1], or to a mean and std value is worth trying)
One very typical problem of LSTM in pytorch is its input dimension is quite different to other type of neural network. You must feed into your network a tensor with shape (seq_len, batch, input_size). You should go here, LSTM section for better details
One more thing: try to tune your hyperparameters. LSTM is harder to train compare to FC or CNN (to my experience).

Tell me if you have improvement. Debugging a neural network is always hard and full of potential coding mistake


Instruction: in-place shuffle torch.Tensor in the order of a numpy.ndarray

Input:
I want to change the order of elements of a torch.Tensor from default to a numpy.ndarray. In other words, I want to shuffle it so that the order of its elements be specified with a numpy array; the important thing about this problem is that I don't want any third object to be created (because of memory limits)
Is there something like below code in python 2.7?

torch_tensor.shuffle(order)

Output:
Edit: This should be an in-place version:

import torch
import numpy as np

t = torch.rand(10)
print('Original Tensor:', t)

order = np.array(range(10))
np.random.shuffle(order)
print('Order:', order)

# in-place changing of values
t[np.array(range(10))] = t[order]
print('New Tensor:', t)

Output:

Original Tensor: tensor([ 0.3380,  0.3450,  0.2253,  0.0279,  0.3945,  0.6055,  0.1489,
         0.7676,  0.4213,  0.2683])
Order: [7 1 3 6 2 9 0 5 4 8]
New Tensor: tensor([ 0.7676,  0.3450,  0.0279,  0.1489,  0.2253,  0.2683,  0.3380,
         0.6055,  0.3945,  0.4213])

I hope this is roughly what you were looking for!


Instruction: Can pytorch's autograd handle torch.cat?

Input:
I'm trying to implement a simple neural network that is supposed to learn an grayscale image. The input consist of the 2d indices of a pixel, the output should be the value of that pixel.

The net is constructed as follows: Each neuron is connected to the input (i.e. the indices of the pixel) as well as to the output of each of the previous neurons. The output is just the output of the last neuron in this sequence.

This kind of network has been very successfull in learning images, as demonstrated e.g. here.

The Problem:
In my implementation the loss function stays between 0.2 and 0.4 depending on the number of neurons, the learning rate and the number of iterations used, which is pretty bad. Also if you compare the output to what what we've trained it for there it just looks like noise. But this is the first time I used torch.cat within the network, so I'm not sure whether this is the culprit. Can anyone see what I'm doing wrong?

from typing import List
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.nn import Linear

class My_Net(nn.Module):
    lin: List[Linear]

    def __init__(self):
        super(My_Net, self).__init__()
        self.num_neurons = 10
        self.lin = nn.ModuleList([nn.Linear(k+2, 1) for k in range(self.num_neurons)])

    def forward(self, x):
        v = x
        recent = torch.Tensor(0)
        for k in range(self.num_neurons):
            recent = F.relu(self.lin[k](v))
            v = torch.cat([v, recent], dim=1)
        return recent

    def num_flat_features(self, x):
        size = x.size()[1:]
        num = 1
        for i in size():
            num *= i
        return num

my_net = My_Net()
print(my_net)

#define a small 3x3 image that the net is supposed to learn
my_image = [[1.0, 1.0, 1.0], [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]] #represents a T-shape
my_image_flat = []    #output of the net is the value of a pixel
my_image_indices = [] #input to the net is are the 2d indices of a pixel
for i in range(len(my_image)):
    for j in range(len(my_image[i])):
        my_image_flat.append(my_image[i][j])
        my_image_indices.append([i, j])

#optimization loop
for i in range(100):
    inp = torch.Tensor(my_image_indices)

    out = my_net(inp)

    target = torch.Tensor(my_image_flat)
    criterion = nn.MSELoss()
    loss = criterion(out.view(-1), target)
    print(loss)

    my_net.zero_grad()
    loss.backward()
    optimizer = optim.SGD(my_net.parameters(), lr=0.001)
    optimizer.step()

print("output of current image")
print([[my_net(torch.Tensor([[i,j]])).item() for i in range(3)] for j in range(3)])
print("output of original image")
print(my_image)

Output:
Yes, torch.cat is backprob-able. So you use it without problems for this.

What's the problem here is that you define a new optimizer at every iteration. Instead you should define it once after you defined your model.

So having this changed the code works fine and loss is decreasing continuously. I  also added a print out every 5000 iterations to show the progress.

from typing import List
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.nn import Linear

class My_Net(nn.Module):
    lin: List[Linear]

    def __init__(self):
        super(My_Net, self).__init__()
        self.num_neurons = 10
        self.lin = nn.ModuleList([nn.Linear(k+2, 1) for k in range(self.num_neurons)])

    def forward(self, x):
        v = x
        recent = torch.Tensor(0)
        for k in range(self.num_neurons):
            recent = F.relu(self.lin[k](v))
            v = torch.cat([v, recent], dim=1)
        return recent

    def num_flat_features(self, x):
        size = x.size()[1:]
        num = 1
        for i in size():
            num *= i
        return num

my_net = My_Net()
print(my_net)

optimizer = optim.SGD(my_net.parameters(), lr=0.001)

#define a small 3x3 image that the net is supposed to learn
my_image = [[1.0, 1.0, 1.0], [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]] #represents a T-shape
my_image_flat = []    #output of the net is the value of a pixel
my_image_indices = [] #input to the net is are the 2d indices of a pixel
for i in range(len(my_image)):
    for j in range(len(my_image[i])):
        my_image_flat.append(my_image[i][j])
        my_image_indices.append([i, j])

#optimization loop
for i in range(50000):
    inp = torch.Tensor(my_image_indices)

    out = my_net(inp)

    target = torch.Tensor(my_image_flat)
    criterion = nn.MSELoss()
    loss = criterion(out.view(-1), target)
    if i % 5000 == 0:
        print('Iteration:', i, 'Loss:', loss)

    my_net.zero_grad()
    loss.backward()
    optimizer.step()
print('Iteration:', i, 'Loss:', loss)

print("output of current image")
print([[my_net(torch.Tensor([[i,j]])).item() for i in range(3)] for j in range(3)])
print("output of original image")
print(my_image)

Loss output:

Iteration: 0 Loss: tensor(0.4070)
Iteration: 5000 Loss: tensor(0.1315)
Iteration: 10000 Loss: tensor(1.00000e-02 *
       8.8275)
Iteration: 15000 Loss: tensor(1.00000e-02 *
       5.6190)
Iteration: 20000 Loss: tensor(1.00000e-02 *
       3.2540)
Iteration: 25000 Loss: tensor(1.00000e-02 *
       1.3628)
Iteration: 30000 Loss: tensor(1.00000e-03 *
       4.4690)
Iteration: 35000 Loss: tensor(1.00000e-03 *
       1.3582)
Iteration: 40000 Loss: tensor(1.00000e-04 *
       3.4776)
Iteration: 45000 Loss: tensor(1.00000e-05 *
       7.9518)
Iteration: 49999 Loss: tensor(1.00000e-05 *
       1.7160)

So the loss goes down to 0.000017 in this case. I have to admit that your error surface is really ragged. Depending on the on the initial weights it may also converge to a minimum of 0.17, 0.10 .. etc. The local minimum where it converges can be very different. So you might try initializing your weights within a smaller range.

Btw. here is the output without changing the location of defining the optimizer:

Iteration: 0 Loss: tensor(0.5574)
Iteration: 5000 Loss: tensor(0.5556)
Iteration: 10000 Loss: tensor(0.5556)
Iteration: 15000 Loss: tensor(0.5556)
Iteration: 20000 Loss: tensor(0.5556)
Iteration: 25000 Loss: tensor(0.5556)
Iteration: 30000 Loss: tensor(0.5556)
Iteration: 35000 Loss: tensor(0.5556)
Iteration: 40000 Loss: tensor(0.5556)
Iteration: 45000 Loss: tensor(0.5556)


Instruction: Appending a recurrent layer to PyTorch LSTM model with different hidden size

Input:
I'm developing a BI-LSTM model for sequence analysis using PyTorch. For which I am using torch.nn.LSTM. Using that module, you can have several layers with just passing a parameter num_layers to be the number of layers (e.g., num_layers=2). However all of them will have the same hidden_size which is partially fine for me, I just want to have all of them the same hidden_size but the last layer  with a different size. Basic example follows:

rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
inp = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)
output, (hn, cn) = rnn(inp, (h0, c0))

The output dim is (5, 3,20)

One solution (But unfavorable to me) is implementing extra model that outputs the dimension I need and takes the input from the first model, e.g.,:

rnn_two = nn.LSTM(input_size=20, hidden_size=2)
output2, _ = rnn_two(output)

However, I do not wanna do this because I parallelize the model using DataParallel, so I need all to be one package. I was hoping to find something similar to keras, e.g.,:

rnn.add(LSTM, hidden_size=2)

I have checked the LSTM source code but couldn't find what I need.

Any suggestions?

Output:
If I'm not mistaken this can be done like this:

import torch.nn as nn
import torch.nn.functional as F

class RnnWith2HiddenSizesModel(nn.Module):
    def __init__(self):
        super(RnnWith2HiddenSizesModel, self).__init__()
        self.rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
        self.rnn_two = nn.LSTM(input_size=20, hidden_size=2)

    def forward(self, inp, hc):
        output, _ = self.rnn(inp, hc)
        output2, _ = self.rnn_two(output)
        return output2

inp = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
c0 = torch.randn(2, 3, 20)

rnn = RnnWith2HiddenSizesModel()

output = RnnWith2HiddenSizesModel()(inp, (h0, c0))

tensor([[[-0.0305,  0.0327],
     [-0.1534, -0.1193],
     [-0.1393,  0.0474]],

    [[-0.0370,  0.0519],
     [-0.2081, -0.0693],
     [-0.1809,  0.0826]],

    [[-0.0561,  0.0731],
     [-0.2307, -0.0229],
     [-0.1780,  0.0901]],

    [[-0.0612,  0.0891],
     [-0.2378,  0.0164],
     [-0.1760,  0.0929]],

    [[-0.0660,  0.1023],
     [-0.2176,  0.0508],
     [-0.1611,  0.1053]]], grad_fn=<CatBackward>)


Instruction: can't find the inplace operation: one of the variables needed for gradient computation has been modified by an inplace operation

Input:
I am trying to compute a loss on the jacobian of the network (i.e. to perform double backprop), and I get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I can't find the inplace operation in my code, so I don't know which line to fix.

*The error occurs in the last line:
            loss3.backward()

            inputs_reg = Variable(data, requires_grad=True)
            output_reg = self.model.forward(inputs_reg)

            num_classes = output.size()[1]
            jacobian_list = []
            grad_output = torch.zeros(*output_reg.size())

            if inputs_reg.is_cuda:
                grad_output = grad_output.cuda()
                jacobian_list = jacobian.cuda()

            for i in range(10):

                zero_gradients(inputs_reg)
                grad_output.zero_()
                grad_output[:, i] = 1
                jacobian_list.append(torch.autograd.grad(outputs=output_reg,
                                                  inputs=inputs_reg,
                                                  grad_outputs=grad_output,
                                                  only_inputs=True,
                                                  retain_graph=True,
                                                  create_graph=True)[0])

            jacobian = torch.stack(jacobian_list, dim=0)
            loss3 = jacobian.norm()
            loss3.backward()

Output:
grad_output.zero_() is in-place and so is grad_output[:, i-1] = 0. In-place means "modify a tensor instead of returning a new one, which has the modifications applied". An example solution which is not in-place is torch.where. An example use to zero out the 1st column

import torch
t = torch.randn(3, 3)
ixs = torch.arange(3, dtype=torch.int64)
zeroed = torch.where(ixs[None, :] == 1, torch.tensor(0.), t)

zeroed
tensor([[-0.6616,  0.0000,  0.7329],
        [ 0.8961,  0.0000, -0.1978],
        [ 0.0798,  0.0000, -1.2041]])

t
tensor([[-0.6616, -1.6422,  0.7329],
        [ 0.8961, -0.9623, -0.1978],
        [ 0.0798, -0.7733, -1.2041]])

Notice how t retains the values it had before and zeroed has the values you want.


Instruction: PyTorch - loading images without sub folders

Input:
First of all I wanted to say that I am new to PyTorch, so i apologize in advice if the level of my questions is not that high.
I was wondering if you can help me with something (I actually have 2 questions). The story behind them: I am working on image classification. My test data is divided into sub folders based on their labels and I am loading them via DataLoader.

First question:
1) Is it true that if you have trained your model with specific batch sized, testing it with other sizes will affect the accuracy?
2) Is there a way to load and use the model with test data located in a single folder (without sub folders). DataLoader needs sub folders as far as i know.

Thank you in advance!

Output:
It depends if you're using operations that depend on other items in the batch. If you're using things like batch normalization it may, but in general if your network processes batch items separately, it doesn't.
If you check the documentation of torch.utils.data.Dataset, you will see that a dataset essentially only requires the __len__ and __getitem__ methods, where the former says how many items the dataset contains and the latter gets the ith item - be it an image and a label, an image and its segmentation mask, or other things. There is nothing stopping you from writing a custom Dataset. I suggest you take a look at source code of DatasetFolder and modify it according to your needs.


Instruction: torch.utils.data.dataloader outputs TypeError: 'module' object is not callable

Input:
So im trying to learn pytorch and i got this code from a tutorial and its just there to import a mnist dataset but it outputs "TypeError: 'module' object is not callable"
In the tutorial     "dataloader" was written as     "Dataloader" but when i run it like that it outputs "AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'"

The data downloaded inside a file mnist but i dont know if it is complete

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optom
from torchvision import datasets, transforms
from torch.autograd import Variable
kwargs={}
train=torch.utils.data.dataloader(datasets.MNIST("mnist",train=True,download=True,transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307),(0.3081,) )] ) ),batch_size=64, shuffle=True, **kwargs)

Output:
It's neither dataloader nor Dataloader but DataLoader :)

Side-note: if you're new to PyTorch, consider using the newest version 1.0. torch.autograd.Variable is deprecated as of PyTorch 0.4.1 (I believe) so you're either using an older version of PyTorch or an outdated tutorial.


Instruction: Why we need image.to('CUDA') when we have model.to('CUDA')

Input:
I'm taking a course on PyTorch.
And I'm wondering why we need to separately tell to torch.utils.data.DataLoader output on what device it's running on. If the model is already on CUDA why doesn't it automatically change the inputs accordingly? This pattern seems funny to me:

model.to(device)

for ii, (inputs, labels) in enumerate(trainloader):

    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

Is there a use case where I'd like to have model running on GPU but my inputs to be on CPU mode or vice versa?

Output:
When you call model.to(device) (assuming device is a GPU) your model parameters will be moved to your GPU. Regarding to your comment: they are moved from CPU memory to GPU memory then.

By default newly created tensors are created on CPU, if not specified otherwise. So this applies also for your inputs and labels.

The problem here is that all operands of an operation need to be on the same device! If you leave out the to and use CPU tensors as input you will get an error message.

Here is an short example for illustration:

import torch

# device will be 'cuda' if a GPU is available
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# creating a CPU tensor
cpu_tensor = torch.rand(10)
# moving same tensor to GPU
gpu_tensor = cpu_tensor.to(device)

print(cpu_tensor, cpu_tensor.dtype, type(cpu_tensor), cpu_tensor.type())
print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())

print(cpu_tensor*gpu_tensor)

Output:

tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619]) torch.float32 <class 'torch.Tensor'> torch.FloatTensor
tensor([0.8571, 0.9171, 0.6626, 0.8086, 0.6440, 0.3682, 0.9920, 0.4298, 0.0172,
        0.1619], device='cuda:0') torch.float32 <class 'torch.Tensor'> torch.cuda.FloatTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ac794171c178> in <module>()
     12 print(gpu_tensor, gpu_tensor.dtype, type(gpu_tensor), gpu_tensor.type())
     13
---> 14 print(cpu_tensor*gpu_tensor)

RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'other'


Instruction: Issues importing pytorch with conda

Input:
My system is:

 x86_64
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=17.1
DISTRIB_CODENAME=rebecca
DISTRIB_DESCRIPTION="Linux Mint 17.1 Rebecca"
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
cat: /etc/upstream-release: Is a directory

I have recently installed pytorch since it is required for the use of the Python causal discovery toolbox, and I have installed it using Anaconda without the use of CUDA, like so:

conda install pytorch-cpu torchvision-cpu -c pytorch

This installation was successful, but when I try to run

import torch

in IDLE, it is not able to find the module. I realize that this question may be a redundancy, but I have searched through similar issues without finding anything that worked, and I am sort on time. Also, I am unfamiliar with Anaconda, so I am not sure how to get packages installed through it to work with the rest of Python. Thank you in advance for any help.

Output:
If you would like to use PyTorch, install it in your local environment using

conda create -n pytorch_env  python=3
source activate pytorch_env
conda install pytorch-cpu torchvision -c pytorch

Go to python shell and import using the command

import torch


Instruction: pytorch equivalent tf.gather

Input:
I'm having some trouble porting some code over from tensorflow to pytorch.

So I have a matrix with dimensions 10x30 representing 10 examples each with 30 features. Then I have another matrix with dimensions 10x5 containing indices of the the 5 closest examples for each examples in the first matrix. I want to 'gather' using the indices contained in the second matrix the 5 closet examples for each example in the first matrix leaving me with a 3d tensor of shape 10x5x30.

In tensorflow this is done with tf.gather(matrix1, matrix2). Does anyone know how i could do this in pytorch?

Output:
How about this?

matrix1 = torch.randn(10, 30)
matrix2 = torch.randint(high=10, size=(10, 5))
gathered = matrix1[matrix2]

It uses the trick of indexing with an array of integers.


Instruction: How can I compute the tensor in Pytorch efficiently?

Input:
I have a tensor x and x.shape=(batch_size,10), now I want to take

 x[i][0] = x[i][0]*x[i][1]*...*x[i][9] for i in range(batch_size)

Here is my code:

for i in range(batch_size):
    for k in range(1, 10):
        x[i][0] = x[i][0] * x[i][k]

But when I implement this in forward() and call loss.backward(), the speed of back-propagation is very slow. Why is it slow and is there any way to implement it efficiently?

Output:
It is slow because you use two for loops.

You can use .prod See: https://pytorch.org/docs/stable/torch.html#torch.prod

In your case,

x = torch.prod(x, dim=1) or x = x.prod(dim=1)

should work


Instruction: How to slice Torch images as numpy images

Input:
I am working on a problem in which I have the coordinates to slice the image like

  X cordinate, Y coordinate, Height, width of the region to crop

So if if I have torch image obtained using

img = Variable(img.cuda())

how can we slice this image to get that specific area of image [y:y+height, x:x+width] .
Thanks

Output:
If I understand your question correctly, then you can do it just the same way as in numpy.

Here is a short example:

import torch
t = torch.rand(5, 5)
# original matrix
print(t)
h = 2
w = 2
x = 1
y = 1
# cropped out matrix
print(t[x:x+h, y:y+w])

Output:

tensor([[ 0.5402,  0.4106,  0.9904,  0.9556,  0.2217],
        [ 0.4533,  0.6300,  0.5352,  0.2710,  0.4307],
        [ 0.6389,  0.5660,  0.1582,  0.5701,  0.1614],
        [ 0.1717,  0.4071,  0.4960,  0.2127,  0.5587],
        [ 0.9529,  0.2865,  0.6667,  0.7401,  0.3372]])
tensor([[ 0.6300,  0.5352],
        [ 0.5660,  0.1582]])

As you can see a 2x2 matrix is cropped out of t.


Instruction: PyTorch - applying attention efficiently

Input:
I have build a RNN language model with attention and I am creating context vector for every element of the input by attending all the previous hidden states (only one direction).

The most straight forward solution in my opinion is using a for-loop over the RNN output, such that each context vector is computed one after another.

import torch
import torch.nn as nn
import torch.nn.functional as F

class RNN_LM(nn.Module):
    def __init__(self, hidden_size, vocab_size, embedding_dim=None, droprate=0.5):
        super().__init__()
        if not embedding_dim:
            embedding_dim = hidden_size
        self.embedding_matrix = nn.Embedding(vocab_size, embedding_dim)

        self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_size, batch_first=False)
        self.attn = nn.Linear(hidden_size, hidden_size)
        self.vocab_dist = nn.Linear(hidden_size, vocab_size)
        self.dropout = nn.Dropout(droprate)

    def forward(self, x):
        x = self.dropout(self.embedding_matrix(x.view(-1, 1)))
        x, states = self.lstm(x)
        #print(x.size())
        x = x.squeeze()
        content_vectors = [x[0].view(1, -1)]
        # for-loop over hidden states and attention
        for i in range(1, x.size(0)):
            prev_states = x[:i]
            current_state = x[i].view(1, -1)

            attn_prod = torch.mm(self.attn(current_state), prev_states.t())
            attn_weights = F.softmax(attn_prod, dim=1)
            context = torch.mm(attn_weights, prev_states)
            content_vectors.append(context)

        return self.vocab_dist(self.dropout(torch.cat(content_vectors)))

Note: The forward method here is only used for training.

However this solution is not very efficient as the code is not well parallelizable with computing each context vector sequently. But since the context vectors are not dependent on each other, I wonder if there is a non-sequential way of calculating them.

So is there is a way to compute the context vectors without for-loop so that more of computation can be parallelized?

Output:
Ok, for clarity: I assume we only really care about vectorizing the for loop. What is the shape of x? Assuming x is 2-dimensional, I have the following code, where v1 executes your loop and v2 is a vectorized version:

import torch
import torch.nn.functional as F

torch.manual_seed(0)

x = torch.randn(3, 6)

def v1():
    for i in range(1, x.size(0)):
        prev = x[:i]
        curr = x[i].view(1, -1)

        prod = torch.mm(curr, prev.t())
        attn = prod # same shape
        context = torch.mm(attn, prev)
        print(context)

def v2():
    # we're going to unroll the loop by vectorizing over the new,
    # 0-th dimension of `x`. We repeat it as many times as there
    # are iterations in the for loop
    repeated = x.unsqueeze(0).repeat(x.size(0), 1, 1)

    # we're looking to build a `prevs` tensor such that
    # prevs[i, x, y] == prev[x, y] at i-th iteration of the loop in v1,
    # up to 0-padding necessary to make them all the same size.
    # We need to build a higher-dimensional equivalent of torch.triu
    xs = torch.arange(x.size(0)).reshape(1, -1, 1)
    zs = torch.arange(x.size(0)).reshape(-1, 1, 1)
    prevs = torch.where(zs < xs, torch.tensor(0.), repeated)

    # this is an equivalent of the above iteration starting at 1
    prevs = prevs[:-1]
    currs = x[1:]

    # a batched matrix multiplication
    prod = torch.matmul(currs, prevs.transpose(1, 2))
    attn = prod # same shape
    context = torch.matmul(attn, prevs)
    # equivalent of a higher dimensional torch.diagonal
    contexts = torch.einsum('iij->ij', (context))
    print(contexts)

print(x)

print('\n------ v1 -------\n')
v1()
print('\n------ v2 -------\n')
v2()

which vectorizes your loop, with some caveats. First, I assume x is 2-dimensional. Secondly, I skip taking the softmax claiming it doesn't change the size of the input and thus doesn't affect vectorization. That's a true, but unfortunately softmax of a 0-padded vector v is not equal to a 0-padded softmax of unpadded v. This can be fixed with renormalization though. Please let me know if my assumptions are correct and whether this is a good enough starting point for your work.


Instruction: pytorch vgg model test on one image

Input:
I've trained a vgg model, this is how I transformed the test data

test_transform_2= transforms.Compose([transforms.RandomResizedCrop(224),
                                     transforms.ToTensor()])

test_data = datasets.ImageFolder(test_dir, transform=test_transform_2)

the model's finished training now I want to test it on a single image

from scipy import misc

test_image = misc.imread('flower_data/valid/1/image_06739.jpg')
vgg16(torch.from_numpy(test_image))

Error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-60-b83587325fea> in <module>
----> 1 vgg16(torch.from_numpy(test_image))

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torchvision\models\vgg.py in forward(self, x)
     40
     41     def forward(self, x):
---> 42         x = self.features(x)
     43         x = x.view(x.size(0), -1)
     44         x = self.classifier(x)

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     89     def forward(self, input):
     90         for module in self._modules.values():
---> 91             input = module(input)
     92         return input
     93

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

c:\users\sam\mydocu~1\code\envs\data-science\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    299     def forward(self, input):
    300         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 301                         self.padding, self.dilation, self.groups)
    302
    303

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 3, 3], but got input of size [628, 500, 3] instead

I can tell I need to shape the input, however I don't know how to based on the way it seems to expect the input to be inform of a batch.

Output:
Your image is [h, w, 3] where 3 means the rgb channel, and pytorch expects [b, 3, h, w] where b is batch size. So you can reshape it by calling do that by calling reshaped = img.permute(2, 0, 1).unsqueeze(0). I think there is also a utility function for that somewhere, but I can't find it right now.

So in your case

tensor = torch.from_numpy(test_image)
reshaped = tensor.permute(2, 0 1).unsqueeze(0)
your_result = vgg16(reshaped)


Instruction: Can I input a Byte Tensor to my RNN/LSTM model?

Input:
I am developing an RNN/LSTM model to which I want to encode the sequence in a ByteTensor to save memory as I am limited to a very tight memory. However, when I do so, the model returns the following error:

  Expected object of scalar type Byte but got scalar type Float for argument #2 'mat2'

So, there seems to be something else that is need to be Byte tensor as well, but I do not know what is it since the console only shows an error at the line:

output = model(predictor)

Output: It means that inside the model there are float tensors which are being used to operate on your byte tensor (most likely operands in matrix multiplications, additions, etc). I believe you can technically cast them to byte by executing model.type(torch.uint8), but your approach will sooner or later fail anyway - since integers are discrete there is no way to used them in gradient calculations necessary for backpropagation. uint8 values can be used in deep learning to improve performance and memory footprint of inference in a network which is already trained, but this is an advanced technique. For this task your best bet are the regular float32s. If your GPU supports it, you could also use float16 aka half, though it introduces additional complexity and I wouldn't suggest it for beginners.


Instruction: Will pytorch performs correctly with python calculate codes in net?

Input:
Take the fake codes below as an example:

class():
    def forward(input):
        x = some_torch_layers(input)
        x = some_torch_layers(x)
        ...
        x = sum(x) # or numpy or other operations
        x = some_torch_layers(x)
        return x

Will the pytorch net operates well? Especially, while the sum(x) performs well in the backward process.

Output:
TL;DR
No.

In order for PyTorch to "perform well" it needs to propagate gradients through the net. PyTorch doesn't (and can't) know how to differentiate an arbtrary numpy code, it can only propagate gradients through PyTorch tensor operations.
In your examples the gradients will stop at the numpy sum so only the top-most torch layers will be trained (layers between numpy operation and the criterion), The other layers (between input and numpy operation) will have zero gradient and therefore their parameters will remain fixed throughout the training.


Instruction: PRelue is not supperted with mmdnn?

Input:
I converted my caffe model to IR successfully, error happened when I tried convert IR to pytorch:

Pytorch Emitter has not supported operator [PRelu]

How should I deal with that please?

Output:
Yes MMdnn support supports LeakyRelu. Check the link below for pytorch_emitter.py implementation from MMdnn.

pytorch_emitter.py

If you check the implementation you will find all the supported operations and it doesn't include PRelu.


Instruction: torch in-place operations to save memory (softmax)

Input:
Some operations in torch are executed in-place. Shorthand operators like += for example.

Is it possible to get in-place execution for other operations, such as softmax?

I'm currently working with language processing. The model produces a long sequence of probability distributions over a large vocabulary. This final output tensor is responsible for ca 60% of allocated memory. Which is a huge problem, since I need to calculate a softmax over it and that doubles the required memory.

Here is an example of the problem. I am not interested in the tensor t, only in its softmax:

import numpy as np
import torch
import torch.nn.functional as F

t = torch.tensor(np.zeros((30000,30000))).cuda()  #allocates 6.71 GB of GPU
softmax = F.softmax(t, 1)  #out of memory error
del t  #too late, program crashed

Even the following doesn't work:

F.softmax(torch.tensor(np.zeros((30000,30000))).cuda(), 1)

Output:
I have created an in-place version of softmax:

import numpy as np
import torch
import torch.nn.functional as F

# in-place version
t = torch.tensor(np.ones((100,200)))
torch.exp(t, out=t)
summed = torch.sum(t, dim=1, keepdim=True)
t /= summed

# original version
t2 = torch.tensor(np.ones((100,200)))
softmax = F.softmax(t2, 1)

assert torch.allclose(t, softmax)

To answer my question: If you want in-place functions, you have to create them yourself by plugging together low-level operations:

many functions such as torch.exp can be given an optional out parameter.
assignments t[idx] = something are in-place
shorthand operators /=, *=, +=, -= are in-place

This requires careful debugging and can be non-intuitive:

t = t / summed  #not in-place
t /= summed  #in-place

I've read that in-place operations can produce problems with gradients. I'll do some more testing with this code.


Instruction: PyTorch - Incorrect labeling using torchvision.datasets.ImageFolder

Input:
I have structured my dataset in the following way:

dataset/train/0/456.jpg
dataset/train/1/456456.jpg
dataset/train/2/456.jpg
dataset/train/...

dataset/val/0/878.jpg
dataset/val/1/234.jpg
dataset/val/2/34554.jpg
dataset/val/...

So I used torchvision.datasets.ImageFolder to import my dataset to PyTorch. However, it seems like it is not giving the right label to the right image. I've added my code below:

data_transforms = {
    'train': transforms.Compose(
        [transforms.Resize((176,176)),
         transforms.RandomRotation((0,360)),
         transforms.RandomHorizontalFlip(),
         transforms.RandomVerticalFlip(),
         transforms.CenterCrop(128),
         transforms.Grayscale(),
         transforms.ToTensor(),
         transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
    ]),
    'val': transforms.Compose(
        [transforms.Resize((128,128)),
         transforms.Grayscale(),
         transforms.ToTensor(),
         transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
    ]),
}

data_dir = 'dataset'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

I found out that the labels are wrong using the following function:

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

dataiter = iter(dataloaders['val'])
images, labels = dataiter.next()

imshow(torchvision.utils.make_grid(images))
print(labels)

Using the shown images and the labels, I manually checked whether they are correct. Unfortunately, the labels do not correspond to the images. Can someone tell me what I'm doing wrong?

Output: Someone helped me out with this. ImageFolder creates its own internal labels. By printing image_datasets['train'].class_to_idx you can see what label is paired to what internal label. Using this dictionary, you can trace back the original label.


Instruction: Setting dimensions of layers in a convolutional neural network

Input:
Say I have 3x100x100 images in batches of 4 as input and I'm trying to make my first convolutional neural networks with pytorch. I'm really not sure if I'm getting convolutional neural networks right because when I train my input through the following arrangement I run into the error:

Expected input batch_size (1) to match target batch_size (4).

The following is my forward nnet:

Then If I were to pass it through:

nn.Conv2d(3, 6, 5)

I would get 6 layers of maps each with dimensions (100-5+1).

Then If I were to pass it through:

nn.MaxPool2d(2, 2)

I would get 6 layers of maps each with dimensions (96/2)

Then if I were to pass it through:

nn.Conv2d(6, 16, 5)

I would get 16 layers of maps each with dimensions (48-5+1)

Then If I were to pass it through:

self.fc1 = nn.Linear(44*44*16, 120)

I would get 120 neurons

Then If I were to pass it through:

self.fc2 = nn.Linear(120, 84)

I would get 84 neurons

Then If I were to pass it through:

self.fc3 = nn.Linear(84, 3)

I would get 3 outputs which would be perfect because I have 3 classes of labels. But as I said before, this leads to an error which is really surprising because this makes a lot of sense to me.

Full neural network code:

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(44*44*16, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 3)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 *44*44)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
net.to(device)

Output:
Your understanding is correct and very detailed.

However, you have used two pooling layers (see relevant code below). So output after the second step will be 16 maps with 44/2=22 dimension.

x = self.pool(F.relu(self.conv1(x)))
x = self.pool(F.relu(self.conv2(x)))

To fix this either not pool or change the dimension of the fully-connected layer to 22*22*16.

To fix by not pooling modify you forward function as below.

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = F.relu(self.conv2(x))
    x = x.view(-1, 16 *44*44)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

To fix by changing the dimension of the fully-connected layer, change the declaration of networks as below.

def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1 = nn.Linear(22*22*16, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)


Instruction: How to profiling layer-by-layer in Pytroch?

Input:
I have tried to profile layer-by-layer of DenseNet in Pytorch as caffe-time tool.

First trial : using autograd.profiler like below

...
model = models.__dict__['densenet121'](pretrained=True)
model.to(device)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model.eval()
print(prof)
...

But the any results are shown except for this message :

<unfinished torch.autograd.profile>

Ultimately, I want to profile network archtiectures(i.g.DenseNet) to check where bottlenecks happen.

Could anyone do this?

Output:
To run profiler you have do some operations, you have to input some tensor into your model.
Change your code as following.
import torch
import torchvision.models as models

model = models.densenet121(pretrained=True)
x = torch.randn((1, 3, 224, 224), requires_grad=True)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model(x)
print(prof)

This is the sample of the output I got:
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                        CPU time        CUDA time            Calls        CPU total       CUDA total
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
conv2d                                    9976.544us       9972.736us                1       9976.544us       9972.736us
convolution                               9958.778us       9958.400us                1       9958.778us       9958.400us
_convolution                              9946.712us       9947.136us                1       9946.712us       9947.136us
contiguous                                   6.692us          6.976us                1          6.692us          6.976us
empty                                       11.927us         12.032us                1         11.927us         12.032us
mkldnn_convolution                        9880.452us       9889.792us                1       9880.452us       9889.792us
batch_norm                                1214.791us       1213.440us                1       1214.791us       1213.440us
native_batch_norm                         1190.496us       1193.056us                1       1190.496us       1193.056us
threshold_                                 158.258us        159.584us                1        158.258us        159.584us
max_pool2d_with_indices                  28837.682us      28836.834us                1      28837.682us      28836.834us
max_pool2d_with_indices_forward          28813.804us      28822.530us                1      28813.804us      28822.530us
batch_norm                                1780.373us       1778.690us                1       1780.373us       1778.690us
native_batch_norm                         1756.774us       1759.327us                1       1756.774us       1759.327us
threshold_                                  64.665us         66.368us                1         64.665us         66.368us
conv2d                                    6103.544us       6102.142us                1       6103.544us       6102.142us
convolution                               6089.946us       6089.600us                1       6089.946us       6089.600us
_convolution                              6076.506us       6076.416us                1       6076.506us       6076.416us
contiguous                                   7.306us          7.938us                1          7.306us          7.938us
empty                                        9.037us          8.194us                1          9.037us          8.194us
mkldnn_convolution                        6015.653us       6021.408us                1       6015.653us       6021.408us
batch_norm                                 700.129us        699.394us                1        700.129us        699.394us

There are many rows below this.
I have used (1,3,224,224) tensor as densenet only accepts 224x224 images. In the future change tensor size according to the network.


Instruction: KeyError: "filename 'storages' not found"

Input:
I'm trying to load pre-trained model weights using this line :

state_dict = torch.load('models/seq_to_txt_state_7.tar')

and I'm getting:

       KeyError                                  Traceback (most recent call last)
<ipython-input-30-3f7b5be8fc72> in <module>()
----> 1 state_dict = torch.load('models/seq_to_txt_state_7.tar')

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in load(f, map_location, pickle_module)
    365         f = open(f, 'rb')
    366     try:
--> 367         return _load(f, map_location, pickle_module)
    368     finally:
    369         if new_fd:

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in _load(f, map_location, pickle_module)
    521         # only if offset is zero we can attempt the legacy tar file loader
    522         try:
--> 523             return legacy_load(f)
    524         except tarfile.TarError:
    525             # if not a tarfile, reset file offset and proceed

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in legacy_load(f)
    448                 mkdtemp() as tmpdir:
    449
--> 450             tar.extract('storages', path=tmpdir)
    451             with open(os.path.join(tmpdir, 'storages'), 'rb', 0) as f:
    452                 num_storages = pickle_module.load(f)

/usr/lib/python2.7/tarfile.pyc in extract(self, member, path)
   2107
   2108         if isinstance(member, basestring):
-> 2109             tarinfo = self.getmember(member)
   2110         else:
   2111             tarinfo = member

/usr/lib/python2.7/tarfile.pyc in getmember(self, name)
   1827         tarinfo = self._getmember(name)
   1828         if tarinfo is None:
-> 1829             raise KeyError("filename %r not found" % name)
   1830         return tarinfo
   1831

KeyError: "filename 'storages' not found"

I'm using python 2.7 on Ubuntu 18.

In addition the model is saved using this function in first place:

def save_state(enc, dec, enc_optim, dec_optim, dec_idx_to_word, dec_word_to_idx, epoch):
state = {'enc':enc.state_dict(), 'dec':dec.state_dict(),
         'enc_optim':enc_optim.state_dict(), 'dec_optim':dec_optim.state_dict(),
        'dec_idx_to_word':dec_idx_to_word, 'dec_word_to_idx':dec_word_to_idx}
torch.save(state, epoch_to_save_path(epoch))

Output: @reportgunner is right. The model file was corrupted. End of the message!


Instruction: Are there any computational efficiency differences between nn.functional() Vs nn.sequential() in PyTorch

Input:
The following is a Feed-forward network using the nn.functional() module in PyTorch

import torch.nn as nn
import torch.nn.functional as F

class newNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64,10)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x))
        return x

model = newNetwork()
model

The following is the same Feed-forward using nn.sequential() module to essentially build the same thing. What is the difference between the two and when would i use one instead of the other?

input_size = 784
hidden_sizes = [128, 64]
output_size = 10

Build a feed-forward network

 model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
    print(model)

Output: There is no difference between the two. The latter is arguably more concise and easier to write and the reason for "objective" versions of pure (ie non-stateful) functions like ReLU and Sigmoid is to allow their use in constructs like nn.Sequential.


Instruction: pytorch runs in anaconda prompt but not in python idle

Input:
I know this question might be stupid, but I couldn't find any help on the internet. Recently I installed anaconda in my computer, it runs Windows 10 x64. Then I used anaconda prompt to download and install pytorch for 3.6 python:

conda install pytorch torchvision cuda100 -c pytorch

After the installation I verified in anaconda's prompt that pytorch is installed:

>>> Python
>>> Import torch
>>> torch.cuda.is_available()
True

I also checked conda list and indeed pytorch is installed in my machine.
However, I write Python code in python 3.6.7 IDLE, not in anaconda prompt, so, whenever I try to import pytorch I get the message:

Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import pytorch
ModuleNotFoundError: No module named 'pytorch'

For some reason, Anacoda prompt recognizes pytorch, but not IDLE. Is there any solution for this? Is there any way to import pytorch module to IDLE?
Thanks in advance.

Output:
It seems like the python used by idle is not the one from anaconda. In python it's very common to have multiple environments, and you always need to be aware of which environment is activated. To see what environment is activated, you can do something in this in anaconda and idle

>>> import sys
>>> print(sys.executable)

If they have different paths, you need to first figure out how to stay in a specific environment in idle.


Instruction: how to calculate cross entropy in 3d image pytorch?

Input:
See the figure here

left thing is (2, 480, 640) and it's softmax value

right thing is (2, 480, 640) and it's one-hot encoding value

how to get cross entropy loss in all element?

Output:
Exactly the same way as with any other image. Use binary_cross_entropy(left, right). Note that

Both have to be of torch.float32 dtype so you may need to first convert right using right.to(torch.float32).
If your left tensor contains logits instead of probabilities it is better to call binary_cross_entropy_with_logits(left, right) than to call binary_cross_entropy(torch.sigmoid(left), right)


Instruction: What does it mean if a deeper conv layer converges first?

Input:
I am training a 3-layer convnet to classify images - a very standard problem, I know. I first tried 3 convolutional layers with ReLU, and got this:

weights from layer 1 with ReLU - looks like edge detection

weights from layer 3 with ReLU - looks like feature detection

The first layer (16 filters) is learning edges, as expected, and the third layer (64 filters) is learning features, as expected. Then, I just wanted to try a different non-linear term so I tried sELU instead.

Oddly, the third layer seems to be now learning features, and the first layer seems to not converge at all? What does it mean for a third layer to learn edges, does it mean I need more layers? I don't see why the first layer would fail to learn edges.

weights from layer 1 with SELU - looks unconverged?

weights from layer 3 with SELU - looks like edge detection?

I don't think the architecture is super important, but I have a 180x180 black-and-white image, and the filters are all 10 x 10 with stride 2 (16 filters for layer 1, 32 for layer 2, 64 for layer 3).

Output:
First off, you're confusing terminology.

The notion of convergence applies to an optimization algorithm and whether it arrives at some fixed location in the parameter space or not. If not, it may keep going forever, either improving at infinitesimally slow rate and never arriving at an optimum, oscillating around it or straight up diverging due to numerical precision/gradient explosion issues. In other words, you can speak of your network optimization having converged but not particular filters. You do that by inspecting the training loss plot, not your kernels.
A feature, in deep learning parlance, is a general notion for, well, features - that is any pattern of interest in the data. So edges would definitely be considered features as well. Perhaps you meant texture when mentioning features?

With that covered, you're unfortunately too optimistic about the state of theory regarding neural networks. Interpretation of convolution kernels is very difficult and a great research problem. Nobody can responsibly make a general statement about which course of action you should take given the kernels you observe - there are way too many variables, from the dataset, through network architecture, to hyperparameters like learning rate. From my own experience, networks with all their kernels looking like this "noise" above can achieve very good results, at least on segmentation tasks with which I am working.

If you are beginning with deep learning (which it looks like you are), I would suggest you also look at feature maps, that is inspect the tensors of intermediate values during the forward propagation of your network - you will see how they react to different parts of your picture and may lead you to more insight. You need to remember that aside from the first layer, the further kernels look at already transformed representations of the image so inspecting them without relation to the input featuremaps will not tell you much. A more advanced technique for understanding your kernels is deep visualization. That being said, I encourage you keep doing this kind of experiments and visualizations, since they will help you develop experience and an intuitive sense for what kernels may look like, how they interact and what is to be expected and what is not.


Instruction: inputing numpy array images into pytorch neural net

Input:
I have a numpy array representation of an image and I want to turn it into a tensor so I can feed it through my pytorch neural network.

I understand that the neural networks take in transformed tensors which are not arranged in [100,100,3] but [3,100,100] and the pixels are rescaled and the images must be in batches.

So I did the following:

import cv2
my_img = cv2.imread('testset/img0.png')
my_img.shape #reuturns [100,100,3] a 3 channel image with 100x100 resolution
my_img = np.transpose(my_img,(2,0,1))
my_img.shape #returns [3,100,100]
#convert the numpy array to tensor
my_img_tensor = torch.from_numpy(my_img)
#rescale to be [0,1] like the data it was trained on by default
my_img_tensor *= (1/255)
#turn the tensor into a batch of size 1
my_img_tensor = my_img_tensor.unsqueeze(0)
#send image to gpu
my_img_tensor.to(device)
#put forward through my neural network.
net(my_img_tensor)

However this returns the error:

RuntimeError: _thnn_conv2d_forward is not implemented for type torch.ByteTensor

Output:
The problem is that the input you give to your network is of type ByteTensor while only float operations are implemented for conv like operations. Try the following

my_img_tensor = my_img_tensor.type('torch.DoubleTensor')
# for converting to double tensor

Source PyTorch Discussion Forum

Thanks to AlbanD


Instruction: How do I install PyTorch v1.0.0+ on Google Colab?

Input:
PyTorch v1.0.0 stable was released on 8 December 2018 after being announced 7 months earlier.

I want get a version optimised for the hardware that my IPython kernel is running on.

How do I get this version on Google Colab?

Output:
try the following code snippet (it works equally for the runtime with or without gpu)

!pip install -q torch==1.0.0 torchvision

to check the version

import torch
print(torch.__version__)

here you have the version 1.0.0

UPDATE

!pip install torch

Works fine now, as the most stable version is 1.0.0


Instruction: How are the pytorch dimensions for linear layers calculated?

Input:
In the PyTorch tutorial, the constructed network is

Net(
  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

And used to process images with dimensions 1x32x32. They mention, that the network cannot be used with images with a different size.

The two convolutional layers seem to allow for an arbitrary number of features, so the linear layers seem to be related to getting the 32x32 into into 10 final features.

I do not really understand, how the numbers 120 and 84 are chosen there and why the result matches with the input dimensions.

And when I try to construct a similar network, I actually get the problem with the dimension of the data.

When I for example use a simpler network:

Net(
  (conv1): Conv2d(3, 8, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(8, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=3, bias=True)
)

for an input of the size 3x1200x800, I get the error message:

RuntimeError: size mismatch, m1: [1 x 936144], m2: [400 x 3] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:940

Where does the number 936144 come from and how do I need to design the network, such that the dimensions are matching?

Output:
The key step is between the last convolution and the first Linear block. Conv2d outputs a tensor of shape [batch_size, n_features_conv, height, width] whereas Linear expects [batch_size, n_features_lin]. To make the two align you need to "stack" the 3 dimensions [n_features_conv, height, width] into one [n_features_lin]. As follows, it must be that n_features_lin == n_features_conv * height * width. In the original code this "stacking" is achieved by

x = x.view(-1, self.num_flat_features(x))

and if you inspect num_flat_features it just computes this n_features_conv * height * width product. In other words, your first conv must have num_flat_features(x) input features, where x is the tensor retrieved from the preceding convolution. But we need to calculate this value ahead of time, so that we can initialize the network in the first place...

The calculation follows from inspecting the operations one by one.

input is 32x32
we do a 5x5 convolution without padding, so we lose 2 pixels at each side, we drop down to 28x28
we do maxpooling with receptive field of 2x2, we cut each dimension by half, down to 14x14
we do another 5x5 convolution without padding, we drop down to 10x10
we do another maxpooling, we drop down to 5x5

and this 5x5 is why in the tutorial you see self.fc1 = nn.Linear(16 * 5 * 5, 120). It's n_features_conv * height * width, when starting from a 32x32 image. If you want to have a different input size, you have to redo the above calculation and adjust your first Linear layer accordingly.

For the further operations, it's just a chain of matrix multiplications (that's what Linear does). So the only rule is that the n_features_out of previous Linear matches n_features_in of the next one. Values 120 and 84 are entirely arbitrary, though they were probably chosen by the author such that the resulting network performs well.


Instruction: saving and loading pytorch neural nets

Input:
So I created a neural net and I would like to save it and load it whenever I want. Specifically, I want to take pictures and do real time processing. I am using the neural net created here

I read that the standard way is to create the net then use torch.save(net,'mynet') to save it and then load it with torch.load('mynet').

However if I open a new python3 terminal and use:

>>import torch
>>torch.load('mynet')

It gives me the error:

  File "<stdin>", line 1, in <module>
  File "/home/tim/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 367, in load
    return _load(f, map_location, pickle_module)
  File "/home/tim/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 538, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'Net' on <module '__main__' (built-in)>

I think this is from not having the Net class defined. Adding

import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 15, 3)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(15, 15, 5)
        self.conv3 = nn.Conv2d(15, 10, 3)
        self.fc1 = nn.Linear(10*4*4, 100)
        self.fc2 = nn.Linear(100, 24)
        self.fc3 = nn.Linear(24, 4)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(-1, 10*4*4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

is what you need to do , but why do we need to define the neural net class? What If I load a neural net with a different architecture to the one i specify in the class will the architecture defined in the class get overwritten? surely the object im loading have all the architecture and class information incapsulated in it?

Update:
Actually it doesn't even work when i define the Net class.

Output:
Please refer to the docs on serialization semantics, which first describes the suggested approach and then the one  you used as "serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors."

In other words, you need to save/load net.state_dict(), not the net itself.