Instruction: pip - Installing specific package version does not work

Input:
I was trying to install a library (allennlp) via pip3. But it complained about the PyTorch version. While allennlp requires torch=0.4.0 I have torch=0.4.1:

...
Collecting torch==0.4.0 (from allennlp)
  Could not find a version that satisfies the requirement torch==0.4.0 (from allennlp) (from versions: 0.1.2, 0.1.2.post1, 0.4.1)
No matching distribution found for torch==0.4.0 (from allennlp)

Also manually install:

pip3 install torch==0.4.0

Doesn't work either:

  Could not find a version that satisfies the requirement torch==0.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.4.1)
No matching distribution found for torch==0.4.0

Same for other versions.

Python is version Python 3.7.0 installed via brew on Mac OS.

I remember that some time ago I was able to switch between version 0.4.0 and 0.3.1 by using pip3 install torch==0.X.X.

How do I solve this?

Output:
allennlp master branch specifies torch>=0.4.0,<0.5.0.

The latest release is v0.6.0 - released only about 3 hours ago - and also specifies this range: https://github.com/allenai/allennlp/blob/v0.6.0/setup.py#L104

It's possible you are using an older release (probably v0.51) which pinned torch==0.4.0: https://github.com/allenai/allennlp/blob/v0.5.1/setup.py#L104

torch have not yet released a v0.4.0 distribution supporting 3.7 to PyPI: there are wheels for CPython 2.7, 3.5, and 3.6.  No source distributions.

allennlp==0.6.0 and torch==0.4.1post2 should work on Python 3.7. I was able to resolve the (considerably large) dependency tree on a linux/Python 3.7.0 runtime using my project johnnydep:

$ johnnydep allennlp --fields name version_latest_in_spec
name                                         version_latest_in_spec
-------------------------------------------  ------------------------
allennlp                                     0.6.0
├── awscli>=1.11.91                          1.15.78
│   ├── PyYAML<=3.13,>=3.10                  3.13
│   ├── botocore==1.10.77                    1.10.77
│   │   ├── docutils>=0.10                   0.14
│   │   ├── jmespath<1.0.0,>=0.7.1           0.9.3
│   │   └── python-dateutil<3.0.0,>=2.1      2.7.3
│   │       └── six>=1.5                     1.11.0
│   ├── colorama<=0.3.9,>=0.2.5              0.3.9
│   ├── docutils>=0.10                       0.14
│   ├── rsa<=3.5.0,>=3.1.2                   3.4.2
│   │   └── pyasn1>=0.1.3                    0.4.4
│   └── s3transfer<0.2.0,>=0.1.12            0.1.13
│       └── botocore<2.0.0,>=1.3.0           1.10.77
│           ├── docutils>=0.10               0.14
│           ├── jmespath<1.0.0,>=0.7.1       0.9.3
│           └── python-dateutil<3.0.0,>=2.1  2.7.3
│               └── six>=1.5                 1.11.0
├── cffi==1.11.2                             1.11.2
│   └── pycparser                            2.18
├── conllu==0.11                             0.11
├── editdistance                             0.4
├── flaky                                    3.4.0
├── flask-cors==3.0.3                        3.0.3
│   ├── Flask>=0.9                           1.0.2
│   │   ├── Jinja2>=2.10                     2.10
│   │   │   └── MarkupSafe>=0.23             1.0
│   │   ├── Werkzeug>=0.14                   0.14.1
│   │   ├── click>=5.1                       6.7
│   │   └── itsdangerous>=0.24               0.24
│   └── Six                                  1.11.0
├── flask==0.12.1                            0.12.1
│   ├── Jinja2>=2.4                          2.10
│   │   └── MarkupSafe>=0.23                 1.0
│   ├── Werkzeug>=0.7                        0.14.1
│   ├── click>=2.0                           6.7
│   └── itsdangerous>=0.21                   0.24
├── gevent==1.3.5                            1.3.5
│   └── greenlet>=0.4.13                     0.4.14
├── h5py                                     2.8.0
│   ├── numpy>=1.7                           1.15.0
│   └── six                                  1.11.0
├── jsonnet==0.10.0                          0.10.0
├── nltk                                     3.3.0
│   └── six                                  1.11.0
├── numpy                                    1.15.0
├── numpydoc==0.8.0                          0.8.0
│   ├── Jinja2>=2.3                          2.10
│   │   └── MarkupSafe>=0.23                 1.0
│   └── sphinx>=1.2.3                        1.7.6
│       ├── Jinja2>=2.3                      2.10
│       │   └── MarkupSafe>=0.23             1.0
│       ├── Pygments>=2.0                    2.2.0
│       ├── alabaster<0.8,>=0.7              0.7.11
│       ├── babel!=2.0,>=1.3                 2.6.0
│       │   └── pytz>=0a                     2018.5
│       ├── docutils>=0.11                   0.14
│       ├── imagesize                        1.0.0
│       ├── packaging                        17.1
│       │   ├── pyparsing>=2.0.2             2.2.0
│       │   └── six                          1.11.0
│       ├── requests>=2.0.0                  2.19.1
│       │   ├── certifi>=2017.4.17           2018.8.13
│       │   ├── chardet<3.1.0,>=3.0.2        3.0.4
│       │   ├── idna<2.8,>=2.5               2.7
│       │   └── urllib3<1.24,>=1.21.1        1.23
│       ├── setuptools                       40.0.0
│       ├── six>=1.5                         1.11.0
│       ├── snowballstemmer>=1.1             1.2.1
│       └── sphinxcontrib-websupport         1.1.0
├── overrides                                1.9
├── parsimonious==0.8.0                      0.8.0
│   └── six>=1.9.0                           1.11.0
├── pytest                                   3.7.1
│   ├── atomicwrites>=1.0                    1.1.5
│   ├── attrs>=17.4.0                        18.1.0
│   ├── more-itertools>=4.0.0                4.3.0
│   │   └── six<2.0.0,>=1.0.0                1.11.0
│   ├── pluggy>=0.7                          0.7.1
│   ├── py>=1.5.0                            1.5.4
│   ├── setuptools                           40.0.0
│   └── six>=1.10.0                          1.11.0
├── pytz==2017.3                             2017.3
├── requests>=2.18                           2.19.1
│   ├── certifi>=2017.4.17                   2018.8.13
│   ├── chardet<3.1.0,>=3.0.2                3.0.4
│   ├── idna<2.8,>=2.5                       2.7
│   └── urllib3<1.24,>=1.21.1                1.23
├── responses>=0.7                           0.9.0
│   ├── cookies                              2.2.1
│   ├── requests>=2.0                        2.19.1
│   │   ├── certifi>=2017.4.17               2018.8.13
│   │   ├── chardet<3.1.0,>=3.0.2            3.0.4
│   │   ├── idna<2.8,>=2.5                   2.7
│   │   └── urllib3<1.24,>=1.21.1            1.23
│   └── six                                  1.11.0
├── scikit-learn                             0.19.2
├── scipy                                    1.1.0
│   └── numpy>=1.8.2                         1.15.0
├── spacy<2.1,>=2.0                          2.0.12
│   ├── cymem<1.32,>=1.30                    1.31.2
│   ├── dill<0.3,>=0.2                       0.2.8.2
│   ├── murmurhash<0.29,>=0.28               0.28.0
│   ├── numpy>=1.7                           1.15.0
│   ├── plac<1.0.0,>=0.9.6                   0.9.6
│   ├── preshed<2.0.0,>=1.0.0                1.0.1
│   │   └── cymem<1.32.0,>=1.30              1.31.2
│   ├── regex==2017.4.5                      2017.4.5
│   ├── requests<3.0.0,>=2.13.0              2.19.1
│   │   ├── certifi>=2017.4.17               2018.8.13
│   │   ├── chardet<3.1.0,>=3.0.2            3.0.4
│   │   ├── idna<2.8,>=2.5                   2.7
│   │   └── urllib3<1.24,>=1.21.1            1.23
│   ├── thinc<6.11.0,>=6.10.3                6.10.3
│   │   ├── cymem<1.32.0,>=1.30.0            1.31.2
│   │   ├── cytoolz<0.10,>=0.9.0             0.9.0.1
│   │   │   └── toolz>=0.8.0                 0.9.0
│   │   ├── dill<0.3.0,>=0.2.7               0.2.8.2
│   │   ├── msgpack-numpy<1.0.0,>=0.4.1      0.4.3.1
│   │   │   ├── msgpack>=0.3.0               0.5.6
│   │   │   └── numpy>=1.9.0                 1.15.0
│   │   ├── msgpack<1.0.0,>=0.5.6            0.5.6
│   │   ├── murmurhash<0.29.0,>=0.28.0       0.28.0
│   │   ├── numpy>=1.7.0                     1.15.0
│   │   ├── plac<1.0.0,>=0.9.6               0.9.6
│   │   ├── preshed<2.0.0,>=1.0.0            1.0.1
│   │   │   └── cymem<1.32.0,>=1.30          1.31.2
│   │   ├── six<2.0.0,>=1.10.0               1.11.0
│   │   ├── tqdm<5.0.0,>=4.10.0              4.24.0
│   │   └── wrapt<1.11.0,>=1.10.0            1.10.11
│   └── ujson>=1.35                          1.35
├── tensorboardX==1.2                        1.2
│   ├── numpy                                1.15.0
│   ├── protobuf>=0.3.2                      3.6.1
│   │   ├── setuptools                       40.0.0
│   │   └── six>=1.9                         1.11.0
│   └── six                                  1.11.0
├── torch<0.5.0,>=0.4.0                      0.4.1.post2
├── tqdm>=4.19                               4.24.0
├── typing                                   3.6.4
└── unidecode                                1.0.22


Instruction: pytorch (numpy) calculation about the closest pixels to points

Input:
I am trying to solve a complicated problem.

For example, I have a batch of 2D predicted images (softmax output, value between 0 and 1) with size: Batch x H x W and ground truth Batch x H x W

The light gray color pixels are the background with value 0, and the dark gray color pixels are the foreground with value 1. I try to compute the mass center coordinates using scipy.ndimage.center_of_mass on each ground truth image. Then I get the center location point C (red color) for each ground truth. The C points set is Batch x 1.

Now, for each pixel A (yellow color) in the predicted images, I want to get three pixels B1, B2, B3 (blue color) which are the closest to A on the line AC (here C is corresponding location of mass center in ground truth).

I used following code to get the three closest points B1, B2, B3.

def connect(ends, m=3):
    d0, d1 = np.abs(np.diff(ends, axis=0))[0]
    if d0 > d1:
        return np.c_[np.linspace(ends[0, 0], ends[1, 0], m + 1, dtype=np.int32),
                 np.round(np.linspace(ends[0, 1], ends[1, 1], m + 1))
                     .astype(np.int32)]
    else:
        return np.c_[np.round(np.linspace(ends[0, 0], ends[1, 0], m + 1))
                     .astype(np.int32),
                 np.linspace(ends[0, 1], ends[1, 1], m + 1, dtype=np.int32)]

So the B points set is Batch x 3 x H x W.

Then, I want to compute like this: |Value(A)-Value(B1)|+|Value(A)-Value(B2)|+|Value(A)-Value(B3)|. The size of the result should be Batch x H x W.

Is there any numpy vectorization tricks that can be used to update the value of each pixel in predicted images? Or can this be solved using pytorch functions? I need to find a method to update the whole image. The predicted image is the softmax output. I cannot use for loop to compute each single value since it will become non-differentiable. Thanks a lot.

Output:
As suggested by @Matin, you could consider Bresenham's algorithm to get your points on the AC line.

A simplistic PyTorch implementation could be as follows (directly adapted from the pseudo-code here ; could be optimized):

import torch

def get_points_from_low(x0, y0, x1, y1, num_points=3):
    dx = x1 - x0
    dy = y1 - y0
    xi = torch.sign(dx)
    yi = torch.sign(dy)
    dy = dy * yi
    D = 2 * dy - dx

    y = y0
    x = x0

    points = []
    for n in range(num_points):
        x = x + xi
        is_D_gt_0 = (D > 0).long()
        y = y + is_D_gt_0 * yi
        D = D + 2 * dy - is_D_gt_0 * 2 * dx

        points.append(torch.stack((x, y), dim=-1))

    return torch.stack(points, dim=len(x0.shape))

def get_points_from_high(x0, y0, x1, y1, num_points=3):
    dx = x1 - x0
    dy = y1 - y0
    xi = torch.sign(dx)
    yi = torch.sign(dy)
    dx = dx * xi
    D = 2 * dx - dy

    y = y0
    x = x0

    points = []
    for n in range(num_points):
        y = y + yi
        is_D_gt_0 = (D > 0).long()
        x = x + is_D_gt_0 * xi
        D = D + 2 * dx - is_D_gt_0 * 2 * dy

        points.append(torch.stack((x, y), dim=-1))

    return torch.stack(points, dim=len(x0.shape))

def get_points_from(x0, y0, x1, y1, num_points=3):
    is_dy_lt_dx = (torch.abs(y1 - y0) < torch.abs(x1 - x0)).long()
    is_x0_gt_x1 = (x0 > x1).long()
    is_y0_gt_y1 = (y0 > y1).long()

    sign = 1 - 2 * is_x0_gt_x1
    x0_comp, x1_comp, y0_comp, y1_comp = x0 * sign, x1 * sign, y0 * sign, y1 * sign
    points_low = get_points_from_low(x0_comp, y0_comp, x1_comp, y1_comp, num_points=num_points)
    points_low *= sign.view(-1, 1, 1).expand_as(points_low)

    sign = 1 - 2 * is_y0_gt_y1
    x0_comp, x1_comp, y0_comp, y1_comp = x0 * sign, x1 * sign, y0 * sign, y1 * sign
    points_high = get_points_from_high(x0_comp, y0_comp, x1_comp, y1_comp, num_points=num_points) * sign
    points_high *= sign.view(-1, 1, 1).expand_as(points_high)

    is_dy_lt_dx = is_dy_lt_dx.view(-1, 1, 1).expand(-1, num_points, 2)
    points = points_low * is_dy_lt_dx + points_high * (1 - is_dy_lt_dx)

    return points

# Inputs:
# (@todo: extend A to cover all points in maps):
A = torch.LongTensor([[0, 1], [8, 6]])
C = torch.LongTensor([[6, 4], [2, 3]])
num_points = 3

# Getting points between A and C:
# (@todo: what if there's less than `num_points` between A-C?)
Bs = get_points_from(A[:, 0], A[:, 1], C[:, 0], C[:, 1], num_points=num_points)
print(Bs)
# tensor([[[1, 1],
#          [2, 2],
#          [3, 2]],
#         [[7, 6],
#          [6, 5],
#          [5, 5]]])

Once you have your points, you could retrieve their "values" (Value(A), Value(B1), etc.) using torch.index_select() (note that as of now, this method only accept 1D indices, so you need to unravel your data). All things put together, this would look like something such as the following (extending A from shape (Batch, 2) to (Batch, H, W, 2) is left for exercise...)

# Inputs:
# (@todo: extend A to cover all points in maps):
A = torch.LongTensor([[0, 1], [8, 6]])
C = torch.LongTensor([[6, 4], [2, 3]])
batch_size = A.shape[0]
num_points = 3
map_size = (9, 9)
map_num_elements = map_size[0] * map_size[1]
map_values = torch.stack((torch.arange(0, map_num_elements).view(*map_size),
                          torch.arange(0, -map_num_elements, -1).view(*map_size)))

# Getting points between A and C:
# (@todo: what if there's less than `num_points` between A-C?)
Bs = get_points_from(A[:, 0], A[:, 1], C[:, 0], C[:, 1], num_points=num_points)

# Get map values in positions A:
A_unravel = torch.arange(0, batch_size) * map_num_elements
A_unravel = A_unravel + A[:, 0] * map_size[1] + A[:, 1]
values_A = torch.index_select(map_values.view(-1), dim=0, index=A_unravel)
print(values_A)
# tensor([ 1, -4])

# Get map values in positions A:
A_unravel = torch.arange(0, batch_size) * map_num_elements
A_unravel = A_unravel + A[:, 0] * map_size[1] + A[:, 1]
values_A = torch.index_select(map_values.view(-1), dim=0, index=A_unravel)
print(values_A)
# tensor([  1, -78])

# Get map values in positions B:
Bs_flatten = Bs.view(-1, 2)
Bs_unravel = (torch.arange(0, batch_size)
              .unsqueeze(1)
              .repeat(1, num_points)
              .view(num_points * batch_size) * map_num_elements)
Bs_unravel = Bs_unravel + Bs_flatten[:, 0] * map_size[1] + Bs_flatten[:, 1]
values_B = torch.index_select(map_values.view(-1), dim=0, index=Bs_unravel)
values_B = values_B.view(batch_size, num_points)
print(values_B)
# tensor([[ 10,  20,  29],
#         [-69, -59, -50]])

# Compute result:
res = torch.abs(values_A.unsqueeze(-1).expand_as(values_B) - values_B)
print(res)
# tensor([[ 9, 19, 28],
#         [ 9, 19, 28]])
res = torch.sum(res, dim=1)
print(res)
# tensor([56, 56])


Instruction: Expected parameters of Conv2d

Input:
Below code :

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 20
batch_size_value = 10
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)
labels = [1 for i in range(num_instances)]
x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()
my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Hyper parameters
num_epochs = 5
num_classes = 1
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=1):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

returns error :

RuntimeError: size mismatch, m1: [10 x 1600], m2: [1568 x 1] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249

Reading the documentation for conv2d,  I tried to change the first parameter to 10X100 to match

  input – input tensor of shape (minibatch×in_channels×iH×iW)

from https://pytorch.org/docs/stable/nn.html#torch.nn.functional.conv2d

but then received the error :

RuntimeError: Given groups=1, weight[16, 1000, 5, 5], so expected input[10, 1, 100, 10] to have 1000 channels, but got 1 channels instead

So I'm not sure if I've corrected the original error or just caused a new one?

How should Conv2d be set in order to match image shape of (10,100) ?

Output:
The error comes from your final fully-connected layer self.fc = nn.Linear(7*7*32, num_classes), not your convolution ones.

Given your input dimensions ((10, 100)), the shape of out = self.layer2(out) is (batch_size, 32, 25, 2), and thus the shape of out = out.reshape(out.size(0), -1) is (batch_size, 32*25*2) = (batch_size, 1600).

On the other hand, your fully-connected layer is defined for inputs of shape (batch_size, 32*7*7) = (batch_size, 1568).

This mismatch between the shape of your 2nd convolution output and the expected shape for your fully-connected layer is causing the error (notice how the shapes mentioned in the trace correspond to the aforementioned ones).


Instruction: Custom NN architecture using Pytorch

Input:
I am trying to make a custom CNN architecture using Pytorch. I want to have about the same control as what I would get if I make the architecture using numpy only.

I am new to Pytorch and would like to see some code samples of CNNs implemented without the nn.module class, if possible.

Output:
You have to implement backward() function in your custom class.

However from your question it is not clear whether

your need just a new series of CNN block (so you better use nn.module and something like
nn.Sequential(nn.Conv2d( ...)  )
you just need gradient descent  https://github.com/jcjohnson/pytorch-examples#pytorch-autograd , so computation backward on your own.


Instruction: Pytorch LSTM - Training for Q&A classification

Input:
I'm trying to train a model to classify if an answer answers the question given using this dataset.

I'm training in batches and using GloVe word embeddings. I train in batches of 1000 except the last one. The method I'm trying to use is to first giving the first sentence (question), and then the second sentence (answer) to LSTM and have it give me a number between 0 and 1 by using sigmoid function.

The problem is, loss always repeats itself after epoch 1. It never converges to the correct result which is if the answer belongs to the question, 1, 0 otherwise.

My code is as below:

class QandA(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(QandA, self).__init__()

        self.hidden_size = hidden_size
        self.num_layers = 1
        self.bidirectional = True

        self.lstm = nn.LSTM(input_size, self.hidden_size, num_layers = self.num_layers, bidirectional = self.bidirectional)
        self.lstm.to(device)
        self.hidden2class = nn.Linear(self.hidden_size * 2, 1)
        self.hidden2class.to(device)

    def forward(self, glove_vec, glove_vec2):
        # glove_vec.shape = (sentence_len, batch_size, 300)
        output, hidden = self.lstm(glove_vec)
        output, _ = self.lstm(glove_vec2, hidden)
        # output.shape = (sentence_len, batch_size, hidden_size * 2)
        output = self.hidden2class(output[-1,:,:])
        # output.shape = (batch_size, 1)
        return F.sigmoid(output)
model = QandA(300, 60).to(device)
loss_function = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.1)

Is my approach so wrong that it can't work in practice? Or is there any other problem that I'm overseeing?

edit: Extra code regarding the training;

batch_size = 1000
# load_dataset loads the data from the file.
questions, answers, outputs = load_dataset()
N = len(outputs)
losses = []
for epoch in range(10):
    for batch in range(math.ceil(N / batch_size)):
        model.zero_grad()

        # get_data gets the data from the dataset (size batch_size, sequence batch)
        input1, input2, targets = get_data(batch, batch_size)

        class_pred = model(input1, input2)
        loss = loss_function(class_pred, targets)
        loss.backward()
        optimizer.step()

Output:
I would suggest to encode question and answer independently and put a classifier on top of it. For example, you can encode with biLSTM question and answer, concatenate their representations and feed to the classifier. The code could be something like this (not tested, but hope you got the idea):

class QandA(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(QandA, self).__init__()

        self.hidden_size = hidden_size
        self.num_layers = 1
        self.bidirectional = True

        self.lstm_question = nn.LSTM(input_size, self.hidden_size, num_layers = self.num_layers, bidirectional = self.bidirectional)
        self.lstm_question.to(device)
        self.lstm_answer = nn.LSTM(input_size, self.hidden_size, num_layers = self.num_layers, bidirectional = self.bidirectional)
        self.lstm_answer.to(device)
        self.fc = nn.Linear(self.hidden_size * 4, 1)
        self.fc.to(device)

    def forward(self, glove_question, glove_answer):
        # glove.shape = (sentence_len, batch_size, 300)
        question_last_hidden, _ = self.lstm_question(glove_question)
        # question_last_hidden.shape = (question_len, batch_size, hidden_size * 2)
        answer_last_hidden, _ = self.lstm_answer(glove_answer)
        # answer_last_hidden.shape = (answer_len, batch_size, hidden_size * 2)

        # flatten output of the lstm, if you have multiple lstm layers you need to take only the last layers backward/forward hidden states
        question_last_hidden = question_last_hidden[-1,:,:]
        answer_last_hidden = answer_last_hidden[-1,:,:]
        representation = torch.cat([question_last_hidden, answer_last_hidden], -1) # check here to concatenate over feature size
        # representation.shape = (hidden_size * 4, batch_size)
        output = self.fc(representation)
        # output.shape = (batch_size, 1)
        return F.sigmoid(output)


Instruction: Get values of tensors in loss function

Input:
I would like to get the values of the y_pred and y_true tensors of this keras backend function. I need this to be able to perform some custom calculations and change the loss, these calculations are just possible with the real array values.

def mean_squared_error(y_true, y_pred):
    #some code here
    return K.mean(K.square(y_pred - y_true), axis=-1)

There is a way to do this in keras? Or in any other ML framework (tf, pytorch, theano)?

Output:
No, in general you can't compute the loss that way, because Keras is based on frameworks that do automatic differentiation (like Theano, TensorFlow) and they need to know which operations you are doing in between in order to compute the gradients of the loss.

You need to implement your loss computations using keras.backend functions, else there is no way to compute gradients and optimization won't be possible.


Instruction: Pytorch: Image label

Input:
I am working on an image classifier with 31 classes(Office dataset). There is one folder for each of the classes.  I have a python script written using PyTorch that loads the dataset using datasets.ImageFolder and assigns a label to each image and then trains. Here is my code snippet for loading data:

from torchvision import datasets, transforms
import torch

def load_training(root_path, dir, batch_size, kwargs):
    transform = transforms.Compose(
        [transforms.Resize([256, 256]),
         transforms.RandomCrop(224),
         transforms.RandomHorizontalFlip(),
         transforms.ToTensor()])
    data = datasets.ImageFolder(root=root_path + dir, transform=transform)
    train_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=True, drop_last=True, **kwargs)
    return train_loader

The code takes each folder, assigns the same label to all images in that folder. Is there any way to find which label is assigned to which image/image folder?

Output:
The class ImageFolder has an attribute class_to_idx which is a dictionary mapping the name of the class to the index (label). So, you can access the classes with data.classes and for each class get the label with data.class_to_idx.

For reference: https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py


Instruction: Is there a mistake in pytorch tutorial?

Input:
The official pytorch tutorial (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients) indicates that out.backward() and out.backward(torch.tensor(1)) are equivalent. But this does not seem to be the case.

import torch

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()

# option 1
out.backward()

# option 2. Replace! do not leave one after the other
# out.backward(torch.tensor(1))

print(x.grad)

Using option 2 (commented out) results in an error.

Note: do not leave two backward calls. Replace option 1 with 2.

Is the tutorial out of date? What is the purpose of the argument?

Update
If I use out.backward(torch.tensor(1)) as the tutorial says, I get:

E       RuntimeError: invalid gradient at index 0 - expected type torch.FloatTensor but got torch.LongTensor

../../../anaconda3/envs/phd/lib/python3.6/site-packages/torch/autograd/__init__.py:90: RuntimeError

I tried also using out.backward(torch.Tensor(1)) and I get instead:

E       RuntimeError: invalid gradient at index 0 - expected shape [] but got [1]

../../../anaconda3/envs/phd/lib/python3.6/site-packages/torch/autograd/__init__.py:90: RuntimeError

Output:
You need to use dtype=torch.float:

import torch

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()

# option 1
out.backward()
print(x.grad)

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()

#option 2. Replace! do not leave one after the other
out.backward(torch.tensor(1, dtype=torch.float))

print(x.grad)

Output:

tensor([[ 4.5000,  4.5000],
        [ 4.5000,  4.5000]])
tensor([[ 4.5000,  4.5000],
        [ 4.5000,  4.5000]])


Instruction: PyTorch: Image dimension issue

Input:
I am working on an image classifier dataset. There are 31 classes in my dataset and there is a folder for each class. For training, I am loading data in the following manner:

from torchvision import datasets, transforms
import torch

def load_training(root_path, dir, batch_size, kwargs):
    transform = transforms.Compose(
        [transforms.Resize([256, 256]),
         transforms.RandomCrop(224),
         transforms.RandomHorizontalFlip(),
         transforms.ToTensor()])
    data = datasets.ImageFolder(root=root_path + dir, transform=transform)
    train_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=False, drop_last=True, **kwargs)
    return train_loader

Now for a batch size 32, each batch dim are : [32,3,224,224]. I know Pytorch uses PIL for loading images. But when I'm trying to load images from a list named images of size 32 using PIL using the following code snippet:

for i in images:
    im = Image.open(root_dir + dir + i)
    im = im.convert('RGB')
    im.load()
    im = im.resize((224, 224))
    im = np.asarray( im, dtype="int32" )
    print(im.shape)
    temp_im_2 = torch.from_numpy(np.array(temp_im)).float()

It says im.shape is (32,224,224,3). What am I doing wrong? What should I do to get the batches to be of the same shape in both cases say [32,3,224,224]?

Output: How about just swapping axes? Like im.transpose(0, 3, 1, 2) if im has four dimensions. However, im.shape should return (224, 224, 3) as you've loaded only one image, so that im.transpose(2, 0, 1) should give you the image with the channels in the first dimension which you can then stack together to form a batch.


Instruction: What is the difference between torch.tensor and torch.Tensor?

Input:
Since version 0.4.0, it is possible to use torch.tensor and torch.Tensor

What is the difference? What was the reasoning for providing these two very similar and confusing alternatives?

Output:
In PyTorch torch.Tensor is the main tensor class. So all tensors are just instances of torch.Tensor.
When you call torch.Tensor() you will get an empty tensor without any data.
In contrast torch.tensor is a function which returns a tensor. In the documentation it says:

torch.tensor(data, dtype=None, device=None, requires_grad=False) → Tensor

Constructs a tensor with data.

This also explains why it is no problem creating an empty tensor instance of `torch.Tensor` without `data` by calling:
tensor_without_data = torch.Tensor()

But on the other side:
tensor_without_data = torch.tensor()

Will lead to an error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-ebc3ceaa76d2> in <module>()
----> 1 torch.tensor()

TypeError: tensor() missing 1 required positional arguments: "data"

But in general there is no reason to choose `torch.Tensor` over `torch.tensor`. Also `torch.Tensor` lacks a docstring.
Similar behaviour for creating a tensor without data like with: torch.Tensor() can be achieved using:
torch.tensor(())

Output:
tensor([])


Instruction: Multiple matrix multiplication loses weight updates

Input:
When in forward method I only do one set of torch.add(torch.bmm(x, exp_w), self.b) then my model is back propagating correctly. When I add another layer - torch.add(torch.bmm(out, exp_w2), self.b2) - then the gradients are not updated and the model isn't learning. If I change the activation function from nn.Sigmoid to nn.ReLU then it works with two layers.

Been thinking about this a day now, and not figuring out why it's not working with nn.Sigmoid.

I've tried different learning rates, Loss functions and optimization functions, but no combination seems to work. When I add the weights together before and after training they are the same.

Code:

class MyModel(nn.Module):

    def __init__(self, input_dim, output_dim):
        torch.manual_seed(1)
        super(MyModel, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        hidden_1_dimentsions = 20
        self.w = torch.nn.Parameter(torch.empty(input_dim, hidden_1_dimentsions).uniform_(0, 1))
        self.b = torch.nn.Parameter(torch.empty(hidden_1_dimentsions).uniform_(0, 1))

        self.w2 = torch.nn.Parameter(torch.empty(hidden_1_dimentsions, output_dim).uniform_(0, 1))
        self.b2 = torch.nn.Parameter(torch.empty(output_dim).uniform_(0, 1))

    def activation(self):
        return torch.nn.Sigmoid()

    def forward(self, x):
        x = x.view((x.shape[0], 1, self.input_dim))

        exp_w = self.w.expand(x.shape[0], self.w.size(0), self.w.size(1))
        out = torch.add(torch.bmm(x, exp_w), self.b)
        exp_w2 = self.w2.expand(out.shape[0], self.w2.size(0), self.w2.size(1))
        out = torch.add(torch.bmm(out, exp_w2), self.b2)
        out = self.activation()(out)
        return out.view(x.shape[0])

Output:
Besides loss functions, activation functions and learning rates, your parameter initialisation is also important. I suggest you to take a look at Xavier initialisation: https://pytorch.org/docs/stable/nn.html#torch.nn.init.xavier_uniform_

Furthermore, for a wide range of problems and network architectures Batch Normalization, which ensures that your activations have zero mean and standard deviation, helps: https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm1d

If you are interested to know more about the reason for this, it's mostly due to the vanishing gradient problem, which means that your gradients get so small that your weights don't get updated. It's so common that it has its own page on Wikipedia: https://en.wikipedia.org/wiki/Vanishing_gradient_problem


Instruction: Correct way to create Pytorch dataset that returns sequence of data for RNN?

Input:
I am attempting to train an RNN on time series data, and while there are plenty of tutorials out there on how to build a RNN model I am having some trouble with building the dataloader object for this task. The data is all going to be the same length, so no need for padding as well. The approach I have taken so far is to return a range of data in the getitem function on the dataset class and define the length as

len(data) - seq_len + 1

, however I feel that this is a bit "hacky" and that there should be a more proper way to do this. This method seems confusing and I feel that this would cause problems if collaborating with a group. More specifically, I think that somehow overriding the sampler function in the Pytorch Dataset constructor is the correct way, but I am having trouble understanding how to implement that. Below is the current dataset class I have built, can anyone point me in the right direction with how to fix it? Thank you in advance.

class CustomDataset(Dataset):
  def __init__(self, df, cats, y, seq_l):
    self.n, self.seq_l = len(df), seq_l
    self.cats  = np.array(np.stack([c.values for n,c in df[cats].items()], 1).astype(np.int64))
    self.conts = np.array(np.stack([c.values for n,c in df[[i for i in df.columns if i not in cats]].items()], 1).astype(np.float32))
    self.y = np.array(y)

  def __len__(self): return len(self.y) - self.seq_l + 1

  def __getitem__(self, idx):
    return [
      (torch.from_numpy(self.cats[idx:idx+self.seq_l]),
      torch.from_numpy(self.conts[idx:idx+self.seq_l])),
      self.y[idx+self.seq_l-1]
    ]

Output:
If I understood correctly you have time series data and you want to crate batches of data with the same length by sampling from it?
I think you can use Dataset for returning just one sample of data as it was initially intended by the PyTorch developers. You can stack them in the batch with your own _collate_fn function and pass it to the DataLoader class (_collate_fn is a callable which takes a list of samples and returns a batch, usually, for example, padding is done there). So you would not have a dependency of length (=batch size in your dataset class). I assume you want to preserve sequential order of your samples when you form a batch (given that you work with time series), you can write your own Sampler class (or use SequentialSampler already available in PyTorch).
As a result, you will decouple your sample representation, forming them in a batch (_collate_fn in DataLoader) and sampling (Sampler class). Hope this helps.


Instruction: cifar100 dataset consists of repeated image?

Input:
I randomly browsed some images in cifar100, and found many images like this:
.

Anything went wrong? Or cifar100 indeed consist of such images?

Output:
I think you did not load in the images correctly. Take a look at loading an image from cifar-10 dataset to see that others also have those problems. The correct way to reshape one of those cifar images is as follows:

single_img_reshaped = np.transpose(np.reshape(single_img,(3, 32,32)), (1,2,0))


Instruction: What is this attribute syntax in Python?

Input:
I am current following a tutorial in Pytorch and there is this expression:

grad_h[h < 0] = 0

How does this syntax work and what does it do?

Output:
It means replace with zeros all the values in grad_h where its corresponding h is negative.

So it is implementing some kind of mask, to keep the gradient values only when h is negative

suppose that grad_h and h have the same shape.

grad_h.shape == h.shape

when you do h < 0 you obtain an array of booleans of the same shape that is set to True if h[i] < 0 for each i.

So then you apply this mask to do slicing on grad_h and finally you set all the sliced elements to zero


Instruction: How can I get data for a quiver plot in torch?

Input:
I have some function z(x, y) and I would like to generate a quiver plot (a 2D plot of the gradients). Something like this:

In order to do it, I have to run gradient over a linear mesh and adjust data to the format that matplotlib.quiver does.

A naive way is to iterate forward and backward in a loop:

for i in range(10):
    for j in range(10):
        x = torch.tensor(1. * i, requires_grad=True)
        y = torch.tensor(1. * j, requires_grad=True)
        z = x ** 2 + y ** 2
        z.backward()
        print(x.grad, y.grad)

This is obviously very inefficient. There are some examples on how to generate a linear mesh from x, y but I would need later change the mesh back to the format of the forward formula, get vectors of gradient and put them back, etc..

A simple example in numpy would be:

import matplotlib.pyplot as plt
n = 25
x_range = np.linspace(-25, 25, n)
y_range = np.linspace(-25, 25, n)
X, Y = np.meshgrid(x_range, y_range)
Z = X**2 + Y**2
U, V = 2*X, 2*Y
plt.quiver(X, Y, U, V, Z, alpha=.9)

What would be the standard way of doing this with pytorch? Are there some simple examples available?

Output:
You can compute gradients of non-scalars by passing torch.Tensors of ones.

import matplotlib.pyplot as plt
import torch

# create meshgrid
n = 25
a = torch.linspace(-25, 25,  n)
b = torch.linspace(-25, 25, n)
x = a.repeat(n)
y = b.repeat(n, 1).t().contiguous().view(-1)

x.requires_grad = True
y.requires_grad=True
z = x**2 + y**2
# this line will compute the gradients
torch.autograd.backward([z], [torch.ones(x.size()), torch.ones(y.size())])

# detach to plot
plt.quiver(x.detach(), y.detach(), x.grad, y.grad, z.detach(), alpha=.9)
plt.show()

If you need to do this repeatedly you need to zero the gradients (set x.grad = y.grad = None).


Instruction: Making a prediction from a trained convolution network

Input:
Here is my convolution net that creates training data , then trains on this data using a single convolution with relu activation :

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 10

for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)

mu, sigma = 100, 0.80 # mean and standard deviation
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)

labels_1 = [1 for i in range(num_instances)]
labels_0 = [0 for i in range(num_instances)]

labels = labels_1 + labels_0

print(labels)

x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()

my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
# device = 'cpu'

# Hyper parameters
num_epochs = 50
num_classes = 2
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=1):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(32*25*2, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i % 10) == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

To make a single prediction I use :

model(x2[10].unsqueeze_(0).cuda())

Which outputs :

tensor([[ 4.4880, -4.3128]], device='cuda:0')

Should this not return an image tensor of shape (100,10) of the prediction ?

Update : In order to perform a prediction I use :

torch.argmax(model(x2[2].unsqueeze_(0).cuda()), dim=1)

src : https://discuss.pytorch.org/t/argmax-with-pytorch/1528/11

torch.argmax in this context returns the position of the value that maximises the prediction.

Output:
As noted by Koustav your net is not "fully convolutional": although you have two nn.Conv2d layers, you still have a "fully-connected" (aka nn.Linear) layer on top, which outputs only 2 dimensional (num_classes) output tensor.

More specifically, your net expects a 1x100x10 input (single channel, 100 by 10 pixels image).
After self.layer1 you have a 16x50x5 tensor (16 channels from the convolution, spatial dimensions reduced by max pooling layer).
After self.layer2 you have a 32x25x2 tensor (32 channels from the convolution, spatial dimensions reduced by another max pooling layer).
Finally, your fully connected self.fc nn.Linear layer takes the entire 32*25*2 dimensional input tensor and produces a num_classes output from the entire input.


Instruction: Implementing Curriculum Dropout in Pytorch

Input:
Is there anyone who can help me implement Curriculum Dropout by Pytorch. Thanks in advance, and any kind of help will be appreciable.

I want to do some experiments of Curriculum Dropout in Pytorch. Curriculum Dropout tries to use a time scheduling for adjusting the dropout rate in the neural networks. The related paper can be downloaded from here.
The source code in python can be found here

Output:
I took a quick look at the paper and it seems like the main idea is to use a scheduled dropout rate instead of a fixed dropout rate.

Torch already has a Dropout module: torch.nn.modules.dropout.Dropout.

For your custom neural net using a Dropout module dropout, you can schedule the dropout rate simply by modifying dropout.p between optimization steps.


Instruction: TypeError: Cannot handle the data type in PIL Image

Input:
I have a Pytorch tensor of size (4,3,224,224). When I am trying to convert the first tensor into an Image object, it says:

TypeError: Cannot handle this data type

I ran the following command:

img = Image.fromarray(data[0][i].numpy().astype(np.uint8))

where data is the Pytorch tensor

I tried other solutions but couldn't find any solution.

Please suggest !!

Output:
You are trying to convert 3x224x224 np.array into an image, but PIL.Image expects its images to be of shape 224x224x3, threfore you get an error.
If you transpose your tensor so that the channel dimension will be the last (rather than the first), you should have no problem

img = Image.fromarray(data[0][i].transpose(0,2).numpy().astype(np.uint8))


Instruction: PyTorch element-wise filter layer

Input:
Hi, I want to add element-wise multiplication layer to duplicate the input to multi-channels like this figure. (So, the input size M x N and multiplication filter size M x N is same), as illustrated in this figure

I want to add custom initialization value to filter, and also want them to get gradient while training. However, I can't find element-wise filter layer in PyTorch. Can I make it? Or is it just impossible in PyTorch?

Output:
In pytorch you can always implement your own layers, by making them subclasses of nn.Module. You can also have trainable parameters in your layer, by using nn.Parameter.
Possible implementation of such layer might look like

import torch
from torch import nn

class TrainableEltwiseLayer(nn.Module)
  def __init__(self, n, h, w):
    super(TrainableEltwiseLayer, self).__init__()
    self.weights = nn.Parameter(torch.Tensor(1, n, h, w))  # define the trainable parameter

  def forward(self, x):
    # assuming x is of size b-1-h-w
    return x * self.weights  # element-wise multiplication

You still need to worry about initializing the weights. look into nn.init on ways to init weights. Usually one init the weights of all the net prior to training and prior to loading any stored model (so partially trained models can override random init). Something like

model = mymodel(*args, **kwargs)  # instantiate a model
for m in model.modules():
  if isinstance(m, nn.Conv2d):
     nn.init.normal_(m.weights.data)  # init for conv layers
  if isinstance(m, TrainableEltwiseLayer):
     nn.init.constant_(m.weights.data, 1)  # init your weights here...


Instruction: Calculating Euclidian Norm in Pytorch.. Trouble understanding an implementation

Input:
I've seen another StackOverflow thread talking about the various implementations for calculating the Euclidian norm and I'm having trouble seeing why/how a particular implementation works.

The code is found in an implementation of the MMD metric: https://github.com/josipd/torch-two-sample/blob/master/torch_two_sample/statistics_diff.py

Here is some beginning boilerplate:

import torch
sample_1, sample_2 = torch.ones((10,2)), torch.zeros((10,2))

Then the next part is where we pick up from the code above.. I'm unsure why the samples are being concatenated together..

sample_12 = torch.cat((sample_1, sample_2), 0)
distances = pdist(sample_12, sample_12, norm=2)

and are then passed to the pdist function:

def pdist(sample_1, sample_2, norm=2, eps=1e-5):
    r"""Compute the matrix of all squared pairwise distances.
    Arguments
    ---------
    sample_1 : torch.Tensor or Variable
        The first sample, should be of shape ``(n_1, d)``.
    sample_2 : torch.Tensor or Variable
        The second sample, should be of shape ``(n_2, d)``.
    norm : float
        The l_p norm to be used.
    Returns
    -------
    torch.Tensor or Variable
        Matrix of shape (n_1, n_2). The [i, j]-th entry is equal to
        ``|| sample_1[i, :] - sample_2[j, :] ||_p``."""

here we get to the meat of the calculation

    n_1, n_2 = sample_1.size(0), sample_2.size(0)
    norm = float(norm)
    if norm == 2.:
        norms_1 = torch.sum(sample_1**2, dim=1, keepdim=True)
        norms_2 = torch.sum(sample_2**2, dim=1, keepdim=True)
        norms = (norms_1.expand(n_1, n_2) +
             norms_2.transpose(0, 1).expand(n_1, n_2))
        distances_squared = norms - 2 * sample_1.mm(sample_2.t())
        return torch.sqrt(eps + torch.abs(distances_squared))

I am at a loss for why the euclidian norm would be calculated this way. Any insight would be greatly appreciated

Output:
Let's walk through this block of code step by step. The definition of Euclidean distance, i.e., L2 norm is

Let's consider the simplest case. We have two samples,

Sample a has two vectors [a00, a01] and [a10, a11]. Same for sample b. Let first calculate the norm

n1, n2 = a.size(0), b.size(0)  # here both n1 and n2 have the value 2
norm1 = torch.sum(a**2, dim=1)
norm2 = torch.sum(b**2, dim=1)

Now we get

Next, we have norms_1.expand(n_1, n_2) and norms_2.transpose(0, 1).expand(n_1, n_2)

Note that b is transposed. The sum of the two gives norm

sample_1.mm(sample_2.t()), that's the multiplication of the two matrix.

Therefore, after the operation

distances_squared = norms - 2 * sample_1.mm(sample_2.t())

you get

In the end, the last step is taking the square root of every element in the matrix.


Instruction: Pytorch DataLoader memory is not released

Input:
I'd like to implement SRGAN on pythorch on google collaboratory, but memory of DataLoader seems to be released, so if you turn epoch, memory error will occur.
It would be greatly appreciated if you tell me how to do it in order to free up memory per batch.
This is the github link of the code
https://github.com/pacifinapacific/Hello-World/blob/master/Untitled0.ipynb

It turned 48 and a memory error occurred on 1 echoch,
If you set the batch size to 1/6 of 8, you will get an error at about 6 epoch.

I am reading high resolution and low resolution images with the following code. Extend ImageFolder

but For example, even if an error occurs when learning is executed, the memory of the GPU is not released

class DownSizePairImageFolder(ImageFolder):
def __init__(self, root, transform=None, large_size=256, small_size=64, **kwds):
    super().__init__(root, transform=transform, **kwds)
    self.large_resizer = transforms.Scale(large_size)
    self.small_resizer = transforms.Scale(small_size)

def __getitem__(self, index):
    path, _ = self.imgs[index]
    img = self.loader(path)
    large_img = self.large_resizer(img)
    small_img = self.small_resizer(img)
    if self.transform is not None:
        large_img = self.transform(large_img)
        small_img = self.transform(small_img)
    return small_img, large_img

train_data = DownSizePairImageFolder('./lfw-deepfunneled/train',   transform=transforms.ToTensor())
test_data = DownSizePairImageFolder('./lfw-deepfunneled/test',    transform=transforms.ToTensor())
batch_size = 8
train_loader = DataLoader(train_data, batch_size, shuffle=True)
test_loader = DataLoader(test_data, batch_size, shuffle=False)

Output:
Pytorch builds a computational graph each time you propagate through your model. This graph is normally retained until the output variable G_loss is out of scope, e.g. when a new iteration through the loop starts.

However, you append this loss to a list. Hence, the variable is still known to python and the graph not freed. You can use .detach() to detach the variable from the current graph (which is better than .clone() which I proposed before as it will also copy the data of the tensor).

As a little side node: In your train() function, you return D_loss,G_loss in the for loop, not after it; so you always only use the first batch.


Instruction: Can nvidia-docker be run without a GPU?

Input: The official PyTorch Docker image is based on nvidia/cuda, which is able to run on Docker CE, without any GPU. It can also run on nvidia-docker, I presume with CUDA support enabled. Is it possible to run nvidia-docker itself on an x86 CPU, without any GPU? Is there a way to build a single Docker image that takes advantage of CUDA support when it is available (e.g. when running inside nvidia-docker) and uses the CPU otherwise? What happens when you use torch.cuda from inside Docker CE? What exactly is the difference between Docker CE and why can't nvidia-docker be merged into Docker CE?

Output:
nvidia-docker is a shortcut for docker --runtime nvidia. I do hope they merge it one day, but for now it's a 3rd party runtime. They explain what it is and what it does on their GitHub page.

  A modified version of runc adding a custom pre-start hook to all containers.
  If environment variable NVIDIA_VISIBLE_DEVICES is set in the OCI spec, the hook will configure GPU access for the container by leveraging nvidia-container-cli from project libnvidia-container.

Nothing stops you from running images meant for nvidia-docker with normal docker. They work just fine but if you run something in them that requires the GPU, that will fail.

I don't think you can run nvidia-docker on a machine without a GPU. It won't be able to find the CUDA files it's looking for and will error out.

To create an image that can run on both docker and nvidia-docker, your program inside it needs to be able to know where it's running. I am not sure if there's an official way, but you can try one of the following:

Check if nvidia-smi is available
Check if the directory specified in $CUDA_LIB_PATH exists
Check if your program can load the CUDA libraries successfully, and if it can't just fallback


Instruction: How to calculate kernel dimensions from original image dimensions?

Input:
https://github.com/kuangliu/pytorch-cifar/blob/master/models/resnet.py

From reading https://www.cs.toronto.edu/~kriz/cifar.html the cifar dataset consists of images each with 32x32 dimension.

My understanding of code :

self.conv1 = nn.Conv2d(3, 6, 5)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1   = nn.Linear(16*5*5, 120)

Is :

self.conv1 = nn.Conv2d(3, 6, 5) # 3 channels in, 6 channels out ,  kernel size of 5
self.conv2 = nn.Conv2d(6, 16, 5) # 6 channels in, 16 channels out ,  kernel size of 5
self.fc1   = nn.Linear(16*5*5, 120) # 16*5*5 in features , 120 ouot feature

From resnet.py the following :

self.fc1   = nn.Linear(16*5*5, 120)

From http://cs231n.github.io/convolutional-networks/ the following is stated :

  Summary. To summarize, the Conv Layer:

  Accepts a volume of size W1×H1×D1 Requires four hyperparameters:
  Number of filters K, their spatial extent F, the stride S, the amount
  of zero padding P. Produces a volume of size W2×H2×D2 where:
  W2=(W1−F+2P)/S+1 H2=(H1−F+2P)/S+1 (i.e. width and height are computed
  equally by symmetry) D2=K With parameter sharing, it introduces F⋅F⋅D1
  weights per filter, for a total of (F⋅F⋅D1)⋅K weights and K biases. In
  the output volume, the d-th depth slice (of size W2×H2) is the result
  of performing a valid convolution of the d-th filter over the input
  volume with a stride of S, and then offset by d-th bias.

From this I'm attempting to understand how the training image dimension 32x32 (1024 pixels) is transformed to feature map (16*5*5 -> 400) as aprt of nn.Linear(16*5*5, 120)

From https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d can see default stride is 1 and padding is 0.

What are steps to arrive at 16*5*5 from image dimension of 32*32 and can 16*5*5 be derived from above steps ?

From above steps how to calculate spatial extent ?

Update :

Source code :

'''LeNet in PyTorch.'''
import torch.nn as nn
import torch.nn.functional as F

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out

Taken from https://github.com/kuangliu/pytorch-cifar/blob/master/models/lenet.py

My understanding is that the convolution operation is applied to the image data per kernel. So if 5 kernels are set then 5 convolutions are applied to the data which generates a 5 dimensional image representation.

Output:
You do not provide enough information in your question (see my comment).

However, if I have to guess then you have two pooling layers (with stride 2) in between your convolution layers:

input size 32x32 (3 channels)
conv1 output size 28x28 (6 channels): conv with no padding and kernel size 5, reduces input size by 4.
Pooling layer with stride 2, output size 14x14 (6 channels).
conv2 output size 10x10 (16 channels)
Another pooling layer with stride 2, output size 5x5 (16 channels)
A fully connected layer (nn.Linear) connecting all 5x5x16 inputs to all 120 outputs.

A more thorough guide for estimating the receptive field can be found here.


Instruction: 'None' gradients in pytorch

Input:
I am trying to implement a simple MDN that predicts the parameters of a distribution over a target variable instead of a point value, and then assigns probabilities to discrete bins of the point value. Narrowing down the issue, the code from which the 'None' springs is:

import torch

# params
tte_bins = np.linspace(
    start=0,
    stop=399,
    num=400,
    dtype='float32'
).reshape(1, 1, -1)
bins = torch.tensor(tte_bins, dtype=torch.float32)
x_train = np.random.randn(1, 1024, 3)
y_labels = np.random.randint(low=0, high=399, size=(1, 1024))
y_train = np.eye(400)[y_labels]

# data
in_train = torch.tensor(x_train[0:1, :, :], dtype=torch.float)
in_train = (in_train - torch.mean(in_train)) / torch.std(in_train)
out_train = torch.tensor(y_train[0:1, :, :], dtype=torch.float)

# model
linear = torch.nn.Linear(in_features=3, out_features=2)
lin = linear(in_train)
preds = torch.exp(lin)

# intermediate values
alpha = torch.clamp(preds[0:1, :, 0:1], 0, 500)
beta = torch.clamp(preds[0:1, :, 1:2], 0, 100)

# probs
p1 = torch.exp(-torch.pow(bins / alpha, beta))
p2 = torch.exp(-torch.pow((bins + 1.0) / alpha, beta))
probs = p1 - p2

# loss
loss = torch.mean(torch.pow(out_train - probs, 2))

# gradients
loss.backward()
for p in linear.parameters():
    print(p.grad, 'gradient')

in_train has shape: [1, 1024, 3], out_train has shape: [1, 1024, 400], bins has shape: [1, 1, 400]. All the broadcasting etc.. appears find, the resulting matrices (like alpha/beta/loss) are the right shape and have the right values - there's simply no gradients

edit: added loss.backward() and x_train/y_train, now I have nans

Output:
You simply forgot to compute the gradients. While you calculate the loss, you never tell pytorch with respect to which function it should calculate the gradients.

Simply adding

loss.backward()

to your code should fix the problem.

Additionally, in your code some intermediate results like alpha are sometimes zero but are in a denominator when computing the gradient. This will lead to the nan results you observed.


Instruction: How can I only update some specific tensors in network with pytorch?

Input:
For instance, I want to only update all cnn weights in Resnet in the first 10 epochs and freeze the others.
And from 11th epoch, I wanna change to update the whole model.
How can I achieve the goal?

Output:
You can set the learning rate (and some other meta-parameters) per parameters group. You only need to group your parameters according to your needs.
For example, setting different learning rate for conv layers:

import torch
import itertools
from torch import nn

conv_params = itertools.chain.from_iterable([m.parameters() for m in model.children()
                                             if isinstance(m, nn.Conv2d)])
other_params = itertools.chain.from_iterable([m.parameters() for m in model.children()
                                              if not isinstance(m, nn.Conv2d)])
optimizer = torch.optim.SGD([{'params': other_params},
                             {'params': conv_params, 'lr': 0}],  # set init lr to 0
                            lr=lr_for_model)

You can later access the optimizer param_groups and modify the learning rate.

See per-parameter options for more information.


Instruction: Cannot convert list to array: ValueError: only one element tensors can be converted to Python scalars

Input:
I'm currently working with the PyTorch framework and trying to understand foreign code. I got an indices issue and wanted to print the shape of a list.
The only way of doing so (as far as Google tells me) is to convert the list into a numpy array and then getting the shape with numpy.ndarray.shape().
But trying to convert my list into an array, I got a ValueError: only one element tensors can be converted to Python scalars.
My List is a converted PyTorch Tensor (list(pytorchTensor)) and looks somewhat like this:
[
tensor([[-0.2781, -0.2567, -0.2353,  ..., -0.9640, -0.9855, -1.0069],
        [-0.2781, -0.2567, -0.2353,  ..., -1.0069, -1.0283, -1.0927],
        [-0.2567, -0.2567, -0.2138,  ..., -1.0712, -1.1141, -1.1784],
        ...,
        [-0.6640, -0.6425, -0.6211,  ..., -1.0712, -1.1141, -1.0927],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9426, -0.9640, -0.9640],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9640, -0.9426, -0.9426]]),

tensor([[-0.0769, -0.0980, -0.0769,  ..., -0.9388, -0.9598, -0.9808],
        [-0.0559, -0.0769, -0.0980,  ..., -0.9598, -1.0018, -1.0228],
        [-0.0559, -0.0769, -0.0769,  ..., -1.0228, -1.0439, -1.0859],
        ...,
        [-0.4973, -0.4973, -0.4973,  ..., -1.0018, -1.0439, -1.0228],
        [-0.4973, -0.4973, -0.4973,  ..., -0.8757, -0.9177, -0.9177],
        [-0.4973, -0.4973, -0.4973,  ..., -0.9177, -0.8967, -0.8967]]),

tensor([[-0.1313, -0.1313, -0.1100,  ..., -0.8115, -0.8328, -0.8753],
        [-0.1313, -0.1525, -0.1313,  ..., -0.8541, -0.8966, -0.9391],
        [-0.1100, -0.1313, -0.1100,  ..., -0.9391, -0.9816, -1.0666],
        ...,
        [-0.4502, -0.4714, -0.4502,  ..., -0.8966, -0.8966, -0.8966],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.8115, -0.7903],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.7690, -0.7690]]),
]

Is there a way of getting the shape of that list without converting it into a numpy array?

Output:
It seems like you have a list of tensors. For each tensor you can see its size() (no need to convert to list/numpy). If you insist, you can convert a tensor to numpy array using numpy():
Return a list of tensor shapes:
>> [t.size() for t in my_list_of_tensors]

Returns a list of numpy arrays:
>> [t.numpy() for t in my_list_of_tensors]

In terms of performance, it is always best to avoid casting of tensors into numpy arrays, as it may incur sync of device/host memory. If you only need to check the shape of a tensor, use size() function.


Instruction: pytorch - use device inside 'with statement'

Input:
Is there a way of running pytorch inside the context of a specific (GPU) device (without having to specify the device for each new tensor, such as the .to option)?

Something like an equivalent of the tensorflow with tf.device('/device:GPU:0'):..

It seems that the default device is the cpu (unless I'm doing it wrong):

with torch.cuda.device('0'):
   a = torch.zeros(1)
   print(a.device)

>>> cpu

Output:
Unfortunately in the current implementation the with-device statement doesn't work this way, it can just be used to switch between cuda devices.

You still will have to use the device parameter to specify which device is used (or .cuda() to move the tensor to the specified GPU), with a terminology like this when:

# allocates a tensor on GPU 1
a = torch.tensor([1., 2.], device=cuda)

So to access cuda:1:

cuda = torch.device('cuda')

with torch.cuda.device(1):
    # allocates a tensor on GPU 1
    a = torch.tensor([1., 2.], device=cuda)

And to access cuda:2:

cuda = torch.device('cuda')

with torch.cuda.device(2):
    # allocates a tensor on GPU 2
    a = torch.tensor([1., 2.], device=cuda)

However tensors without the device parameter will still be CPU tensors:

cuda = torch.device('cuda')

with torch.cuda.device(1):
    # allocates a tensor on CPU
    a = torch.tensor([1., 2.])

To sum it up:

  No - unfortunately it is in the current implementation of the with-device
  statement not possible to use in a way you described in your
  question.

Here are some more examples from the documentation:

cuda = torch.device('cuda')     # Default CUDA device
cuda0 = torch.device('cuda:0')
cuda2 = torch.device('cuda:2')  # GPU 2 (these are 0-indexed)

x = torch.tensor([1., 2.], device=cuda0)
# x.device is device(type='cuda', index=0)
y = torch.tensor([1., 2.]).cuda()
# y.device is device(type='cuda', index=0)

with torch.cuda.device(1):
    # allocates a tensor on GPU 1
    a = torch.tensor([1., 2.], device=cuda)

    # transfers a tensor from CPU to GPU 1
    b = torch.tensor([1., 2.]).cuda()
    # a.device and b.device are device(type='cuda', index=1)

    # You can also use ``Tensor.to`` to transfer a tensor:
    b2 = torch.tensor([1., 2.]).to(device=cuda)
    # b.device and b2.device are device(type='cuda', index=1)

    c = a + b
    # c.device is device(type='cuda', index=1)

    z = x + y
    # z.device is device(type='cuda', index=0)

    # even within a context, you can specify the device
    # (or give a GPU index to the .cuda call)
    d = torch.randn(2, device=cuda2)
    e = torch.randn(2).to(cuda2)
    f = torch.randn(2).cuda(cuda2)
    # d.device, e.device, and f.device are all device(type='cuda', index=2)


Instruction: Indexing a multi-dimensional tensor with a tensor in PyTorch

Input:
I have the following code:

a = torch.randint(0,10,[3,3,3,3])
b = torch.LongTensor([1,1,1,1])

I have a multi-dimensional index b and want to use it to select a single cell in a. If b wasn't a tensor, I could do:

a[1,1,1,1]

Which returns the correct cell, but:

a[b]

Doesn't work, because it just selects a[1] four times.

How can I do this? Thanks

Output:
A more elegant (and simpler) solution might be to simply cast b as a tuple:

a[tuple(b)]
Out[10]: tensor(5.)

I was curious to see how this works with "regular" numpy, and found a related article explaining this quite well here.


Instruction: Torchtext BucketIterator wrapper from tutorial produces SyntaxError

Input:
I am following and implementing code from this short tutorial on Torchtext, which is surprisingly clear given the poor documentation of Torchtext.

When the Iterator has been created (the batch generator) he proposes to create a wrapper to produce more reusable code. (See step 5 in the tutorial).

The code contains a surprisingly long and weird line, which I don't understand and which raises a SyntaxError: invalid syntax. Does anyone have a clue of what is going on?

(The problematic line is the one that starts with: if self.y_vars is &amp;amp;amp;lt;g [...])

class BatchWrapper:
  def __init__(self, dl, x_var, y_vars):
        self.dl, self.x_var, self.y_vars = dl, x_var, y_vars # we pass in the list of attributes for x &amp;amp;amp;amp;lt;g class="gr_ gr_3178 gr-alert gr_spell gr_inline_cards gr_disable_anim_appear ContextualSpelling ins-del" id="3178" data-gr-id="3178"&amp;amp;amp;amp;gt;and y&amp;amp;amp;amp;lt;/g&amp;amp;amp;amp;gt;

  def __iter__(self):
        for batch in self.dl:
              x = getattr(batch, self.x_var) # we assume only one input in this wrapper

              if self.y_vars is &amp;amp;amp;amp;lt;g class="gr_ gr_3177 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Grammar replaceWithoutSep" id="3177" data-gr-id="3177"&amp;amp;amp;amp;gt;not&amp;amp;amp;amp;lt;/g&amp;amp;amp;amp;gt; None: # we will concatenate y into a single tensor
                    y = torch.cat([getattr(batch, feat).unsqueeze(1) for feat in self.y_vars], dim=1).float()
              else:
                    y = torch.zeros((1))

              yield (x, y)

  def __len__(self):
        return len(self.dl)

Output:
Yeah, I guess there is some typo from the author.
I think the correct piece of code is this:

if self.y_vars is not None:
    y = torch.cat([getattr(batch, feat).unsqueeze(1) for feat in self.y_vars], dim=1).float()
else:
    y = torch.zeros((1))

You can see this typo in the comment of line 3 also (in the code in blogpost).


Instruction: What dtype should I use for PyTorch parameters in a neural network that inputs and outputs arrays of integers?

Input:
I'm currently building a neural network in PyTorch that accepts tensors of integers and outputs tensors of integers.  There is only a small number of positive integers that are "allowed"  (like 0, 1, 2, 3, and 4) as elements of the input and output tensors.

Neural networks usually work in continuous space.
For example, the nonlinear activation functions between layers are continuous and map integers to real numbers (including non-integers).

Is it best to use unsigned integers like torch.uint8 internally for the weights and biases of the network plus some custom activation function that maps ints to ints?

Or should I use high precision floats like torch.float32 and then round in the end, by binning real numbers to the nearest integer?  I think this second strategy is the way to go, but maybe I'm missing out on something that would work nicely.

Output: Without knowing too much about your application I would go for torch.float32 with rounding. The main reason being that if you use a GPU to compute your neural network, it will require wights and data to be in float32 datatype. If you are not going to train your neural network and you want to run on CPU, then datatypes like torch.uint8 may help you as you can achieve more instructions per time interval (i.e. your application should run faster). If that doesn't leave you with a clue, then please be more specific about your application.


Instruction: Why filters do not learn same features

Input:
The result of the convolution operation is multiple subsets of data are generated per kernel. For example if 5 kernels are applies to an image of dimension WxDx1 (1 channel) then 5 convolutions are applied to the data which generates a 5 dimensional image representation. WxDx1 becomes W'xD'x5 where W' and D' are smaller in dimension that W * D

Is the fact that each kernel is initialised to different values prevent each kernel from learning the same parameters ? If not what prevents each kernel learning the same parameters ?

If the image is RGB instead of grayscale so dimension WxDx3 instead of WxDx1 does this impact how the kernels learns patterns ?

Output:
As you already mentioned, the sole fact of differing what Kernels learn is due to the random initialization of the weights in the beginning.

A great explanation is delivered here and also applies for the convolutional kernels in CNNs.
I regard this as distinct enough to not highlight it as a duplicate, but essentially it works the same.


Instruction: Transforms not applying to the dataset

Input:
I'm new to pytorch and would like to understand something.

I am loading MNIST as follows:

transform_train = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Resize(size, interpolation=2),
     # transforms.Grayscale(num_output_channels=1),
     transforms.RandomHorizontalFlip(p=0.5),
     transforms.Normalize((mean), (std))])

trainset = torchvision.datasets.MNIST(root='./data', train=True,
                                      download=True, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

However, when I explore the dataset, i.e. trainloader.dataset.train_data[0], I am getting a tensor in range [0,255] with shape (28,28).

What am I missing? Is this because the transforms are not applied directly to the dataloader but only in runtime? How can I explore my data otherwise?

Output:
The transforms are applied when the __getitem__ method of the Dataset is called. For example look at the __getitem__ method of the MNIST dataset class: https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py#L62

def __getitem__(self, index):
    """
    Args:
        index (int): Index
    Returns:
        tuple: (image, target) where target is index of the target class.
    """
    img, target = self.data[index], self.targets[index]

    # doing this so that it is consistent with all other datasets
    # to return a PIL Image
    img = Image.fromarray(img.numpy(), mode='L')

    if self.transform is not None:
        img = self.transform(img)

    if self.target_transform is not None:
        target = self.target_transform(target)

    return img, target

The __getitem__ method gets called when you index your MNIST instance for the training set, e.g.:

trainset[0]

For more information on __getitem__: https://docs.python.org/3.6/reference/datamodel.html#object.getitem

The reason why Resize and RandomHorizontalFlip should be before ToTensor is that they act on PIL Images and all the datasets in Pytorch for consistency load the data as PIL Images first. In fact you can see that here they force that behavior through:

img = Image.fromarray(img.numpy(), mode='L')

Once you have the PIL Image of the corresponding index, the transforms are applied with

if self.transform is not None:
    img = self.transform(img)

ToTensor transforms the PIL Image to a torch.Tensor and Normalize subtracts the mean and divides by the standard deviation you provide.

Eventually some transforms are applied to the label with

if self.target_transform is not None:
    target = self.target_transform(target)

Finally the processed image and the processed label are returned. All of this happens in a single trainset[key] call.

import torch
from torchvision.transforms import *
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader

transform_train = Compose([Resize(28, interpolation=2),
                           RandomHorizontalFlip(p=0.5),
                           ToTensor(),
                           Normalize([0.], [1.])])

trainset = MNIST(root='./data', train=True, download=True,
                 transform=transform_train)
trainloader = DataLoader(trainset, batch_size=32, shuffle=True, num_workers=2)
print(trainset[0][0].size(), trainset[0][0].min(), trainset[0][0].max())

shows

(torch.Size([1, 28, 28]), tensor(0.), tensor(1.))


Instruction: How to ensure all PyTorch code fully utilises GPU on Google Colab

Input:
I am new to PyTorch and have been doing some tutorial on CIFAR10, specifically with Google Colab since I personally do not have a GPU to experiment on it yet.

I have successfully trained my neural network but I'm not sure whether my code is using the GPU from Colab, because the training time taken with Colab is not significantly faster than my 2014 MacBook Pro (without GPU).

I checked and my notebook is indeed running Tesla K80 but somehow the training speed is slow. So I think perhaps my code is not equipped with GPU syntax but I couldn't figure out which part is that.

# install PyTorch
from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'
!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.0-{platform}-linux_x86_64.whl torchvision

import torch
import torch.nn as nn
from torch.optim import Adam
from torchvision import transforms
from torch.autograd import Variable
import torchvision.datasets as datasets
from torch.utils.data import DataLoader, TensorDataset
import matplotlib.pyplot as plt

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)

# hyperparameters
n_epochs = 50
n_batch_size = 200
n_display_step = 200
n_learning_rate = 1e-3
n_download_cifar = True

# import cifar
# more about cifar https://www.cs.toronto.edu/~kriz/cifar.html

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_dataset = datasets.CIFAR10(
                    root="../datasets/cifar",
                    train=True,
                    transform=transform,
                    download=n_download_cifar)
test_dataset = datasets.CIFAR10(
                    root="../datasets/cifar",
                    train=False,
                    transform=transform)

# create data loader
train_loader = DataLoader(train_dataset, batch_size=n_batch_size, shuffle=True, num_workers=2)
test_loader = DataLoader(test_dataset, batch_size=n_batch_size, shuffle=False)

# build CNN
class CNN(nn.Module):

    def __init__(self):
        super(CNN, self).__init__()

        # (3, 32, 32)
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))

        # (32, 16, 16)
        self.conv2 = nn.Sequential(
            nn.Conv2d(32, 16, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))

        # (16, 8, 8)
        self.out = nn.Linear(16 * 8 * 8, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        out = self.out(x)
        return out

net = CNN()
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = Adam(net.parameters(), lr=n_learning_rate)

def get_accuracy(model, loader):
    model.eval()
    n_samples = 0
    n_correct = 0

    with torch.no_grad():
        for step, (x, y) in enumerate(loader):
            x, y = Variable(x).to(device), Variable(y).to(device)
            out = model(x)
            _, pred = torch.max(out, 1)
            n_samples += y.size(0)
            n_correct += (pred == y).sum().item()

    return n_correct / n_samples

def train(model, criterion, optimizer, epochs, train_loader, test_loader):
    for epoch in range(epochs):
        for step, (x, y) in enumerate(train_loader):
            model.train()
            x, y = Variable(x).to(device), Variable(y).to(device)
            out = model(x)
            loss = criterion(out, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if step % n_display_step == 0:
                print("Epoch {:2d} Loss {:.4f} Accuracy (Train | Test) {:.4f} {:.4f}".format(epoch, loss.item(), get_accuracy(model, train_loader), get_accuracy(model, test_loader)))

train(net, criterion, optimizer, n_epochs, train_loader, test_loader)

Output:
Your code seems appropriate and I ran it on my MacBook, a GPU-enabled machine, and Google Colab. I compared the training time taken and my experiments show your code is optimized for GPU.

Can you try running this code from this thread and see how much GPU RAM Google has allocated for you? My guess is you've only given 5% GPU usage.

Regards,

Rex.


Instruction: pytorch where is Embedding "max_norm" implemented?

Input:
The "embedding" class documentation https://pytorch.org/docs/stable/nn.html says

max_norm (float, optional) – If given, will renormalize the embedding vectors to have a norm lesser than this before extracting.

1) In my model, I use this embedding class as a parameter, not just as an input (the model learns the embedding.) In this case, I assume every time when updates happen, the embedding gets renormalized, not only when it's initialized. Is my understanding correct?

2) I wanted to confirm 1) by looking at the source, but I couldn't find the implementation in pytorch embedding class. https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html
Can someone point me to the max_norm implementation?

Output:
If you see forward function in Embedding class here, there is a reference to torch.nn.functional.embedding which uses embedding_renorm_ which is in the cpp documentation here which means it is a cpp implementation. Some github search on pytorch repo pointed to this files (1, 2).

Answer to 1 is yes. Answer to 2 is above.


Instruction: Installing packages but still python interpreter doesn't recognize using pycharm

Input:
I'm trying to install pytorch, but when I'm trying to import, my pycharm doesn't recognize this package although I'm sure I've installed this package on the same interpreter.

What I am missing?!

Thanks for your help.

Output:
You are installing the packages in your base interpreter i.e your system interpreter. And I guess you started your Pycharm project using a virtual environment. So to install to your Pycharm venv. You need to start the console below i.e The Terminal in your Pycharm project and then perform pip install package_name again. This should install it to your virtual env. And then python will pick it up.

If you want to have all base packages from your system into your Pycharm venv. You can check the inherit base packages option while you are starting a new project with a new virtual env.


Instruction: How to transform a vector to a matrix with each line equal to the vector in pytorch?

Input:
for exemple I have the vector [1, 2, 3]

I want to get a matrix like

[[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]

How to do that efficiently?

Output:
You should look at .repeat():

In [1]: torch.Tensor([1, 2, 3]).repeat(4, 1)
Out[1]:
tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.]])


Instruction: How do we analyse a loss vs epochs graph?

Input:
I'm training a language model and the loss vs epochs is plotted each time of training. I'm attaching two samples from it.

Obviously, the second one is showing better performance. But, from these graphs, when do we take a decision to stop training (early stopping)?

Can we understand overfitting and underfitting from these graphs or do I need to plot additional learning curves?

What are the additional inferences that can be made from these plots?

Output:
The first conclusion is obviously that the first model performs worse than the second, and that is generally true, as long as you use the same data for validation. In the case where you train a model with different splits, that might not necessarily be the case.

Furthermore, to answer your question regarding overfitting/underfitting:
A typical graph for overfitting looks like this:

So, in your case, you clearly just reach convergence, but don't actually overfit! (This is great news!) On the other hand, you could ask yourself whether you could achieve even better results. I am assuming that you are decaying your learning rate, which lets you pan out at some form of plateau. If that is the case, try reducing the learning rate less at first, and see if you can reduce your loss even further.
Moreover, if you still see a very long plateau, you can also consider stopping early, since you effectively gain no more improvements. Depending on your framework, there are implementations of this (for example, Keras has callbacks for early stopping, which is generally tied to the validation/testing error). If your validation error increases, similar to the image, you should consider using the loweste validation error as a point for early stopping. One way I like to do this is to checkpoint the model every now and then, but only if the validation error improved.
Another inference you can make is the learning rate in general: Is it too large, your graph will likely be very "jumpy/jagged", whereas a very low learning rate will have only a small decline in the error, and not so much exponentially decaying behavior.
You can see a weak form of this by comparing the steepness of the decline in the first few epochs in your two examples, where the first one (with the lower learning rate) takes longer to converge.

Lastly, if your training and test error are very far apart (as in the first case), you might ask yourself whether you are actually accurately describing or modeling the problem; in some instances, you might realize that there is some problem in the (data) distribution that you might have overlooked. Since the second graph is way better, though, I doubt this is the case in your problem.


Instruction: Best loss function for multi-class classification when the dataset is imbalance?

Input:
I'm currently using the Cross Entropy Loss function but with the imbalance data-set the performance is not great.

Is there better lost function?

Output:
It's a very broad subject, but IMHO, you should try focal loss: It was introduced by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He and Piotr Dollar to handle imbalance prediction in object detection. Since introduced it was also used in the context of segmentation.
The idea of the focal loss is to reduce both loss and gradient for correct (or almost correct) prediction while emphasizing the gradient of errors.

As you can see in the graph:

Blue curve is the regular cross entropy loss: it has on the one hand non-negligible loss and gradient even for well classified examples, and on the other hand it has weaker gradient for the erroneously classified examples.
In contrast, focal loss (all other curves) has smaller loss and weaker gradient for the well classified examples and stronger gradients for the  erroneously classified examples.


Instruction: Coreml model float input for a pytorch model

Input:
I have a pytorch model that takes 3 x width x height image as input with the pixel values normalized between 0-1

E.g., input in pytorch

img = io.imread(img_path)
input_img =  torch.from_numpy( np.transpose(img, (2,0,1)) ).contiguous().float()/255.0

I converted this model to coreml and exported an mlmodel which takes the input with correct dimensions

Image (Color width x height)

However, my predictions are incorrect as the model is expecting a float value between 0-1 and cvpixelbuffer is a int bwetween 0-255

I tried to normalize the values inside the model like so,

z = x.mul(1.0/255.0) # div op is not supported for export yet

However, when this op is done inside model at coreml level,  int * float is casted as int and all values are essentially 0

Cast op is not supported for export e.g., x = x.float()

How can I make sure my input is properly shaped for prediction? Essentially, I want to take the pixel rgb and float divide 255.0 and pass it to the model for inference?

Output:
I solved it using the coreml onnx coverter's preprocessing_args like so,

preprocessing_args= {'image_scale' : (1.0/255.0)}

Hope this helps someone


Instruction: Pytorch model accuracy test

Input:
I'm using Pytorch to classify a series of images.
The NN is defined as follows:

model = models.vgg16(pretrained=True)
model.cuda()
for param in model.parameters(): param.requires_grad = False

classifier = nn.Sequential(OrderedDict([
                           ('fc1', nn.Linear(25088, 4096)),
                           ('relu', nn.ReLU()),
                           ('fc2', nn.Linear(4096, 102)),
                           ('output', nn.LogSoftmax(dim=1))
                           ]))

model.classifier = classifier

The criterions and optimizers are as follows:

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

My validation function is as follows:

def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0
    for images, labels in testloader:

        images.resize_(images.shape[0], 784)

        output = model.forward(images)
        test_loss += criterion(output, labels).item()

        ps = torch.exp(output)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

    return test_loss, accuracy

This is the piece of code that is throwing the following error:

  RuntimeError: input has less dimensions than expected

epochs = 3
print_every = 40
steps = 0
running_loss = 0
testloader = dataloaders['test']

# change to cuda
model.to('cuda')

for e in range(epochs):
    running_loss = 0
    for ii, (inputs, labels) in enumerate(dataloaders['train']):
        steps += 1

        inputs, labels = inputs.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # Forward and backward passes
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            model.eval()
            with torch.no_grad():
                test_loss, accuracy = validation(model, testloader, criterion)

            print("Epoch: {}/{}.. ".format(e+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                  "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
                  "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

            running_loss = 0

Any help?

Output:
Just in case it helps someone.

If you don't have a GPU system (say you are developing on a laptop and will eventually test on a server with GPU) you can do the same using:

if torch.cuda.is_available():
        inputs =inputs.to('cuda')
    else:
        inputs = inputs.to('cuda')

Also, if you are wondering why there is a LogSoftmax, instead of Softmax that is because he is using NLLLoss as his loss function. You can read more about softmax here


Instruction: Pytorch Validating Model Error: Expected input batch_size (3) to match target batch_size (4)

Input:
I'm building a NN in Pytorch that is supposed to classify across 102 classes.

I've got the following validation function:

def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0

    for inputs, classes in testloader:
        inputs = inputs.to('cuda')
        output = model.forward(inputs)
        test_loss += criterion(output, labels).item()

        ps = torch.exp(output)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

    return test_loss, accuracy

Code for training (calls validation):

epochs = 3
print_every = 40
steps = 0
running_loss = 0
testloader = dataloaders['test']

# change to cuda
model.to('cuda')

for e in range(epochs):
    running_loss = 0
    for ii, (inputs, labels) in enumerate(dataloaders['train']):
        steps += 1

        inputs, labels = inputs.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # Forward and backward passes
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            model.eval()
            with torch.no_grad():
                test_loss, accuracy = validation(model, testloader, criterion)

            print("Epoch: {}/{}.. ".format(e+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                  "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
                  "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

            running_loss = 0
            model.train()

Im getting this error message:

ValueError: Expected input batch_size (3) to match target batch_size (4).

Full Traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-63-f9f67ed13b94> in <module>()
     28             model.eval()
     29             with torch.no_grad():
---> 30                 test_loss, accuracy = validation(model, testloader, criterion)
     31
     32             print("Epoch: {}/{}.. ".format(e+1, epochs),

<ipython-input-62-dbc77acbda5e> in validation(model, testloader, criterion)
      6         inputs = inputs.to('cuda')
      7         output = model.forward(inputs)
----> 8         test_loss += criterion(output, labels).item()
      9
     10         ps = torch.exp(output)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    191         _assert_no_grad(target)
    192         return F.nll_loss(input, target, self.weight, self.size_average,
--> 193                           self.ignore_index, self.reduce)
    194
    195

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1328     if input.size(0) != target.size(0):
   1329         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1330                          .format(input.size(0), target.size(0)))
   1331     if dim == 2:
   1332         return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)

ValueError: Expected input batch_size (3) to match target batch_size (4).

I don't understand where the error comes from. Indeed, without the validation code the training part works perfectly.

Output:
In your validation function,

def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0

    for inputs, classes in testloader:
        inputs = inputs.to('cuda')
        output = model.forward(inputs)
        test_loss += criterion(output, labels).item()

        ps = torch.exp(output)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

    return test_loss, accuracy

You are iterating on testloader and passing values to the variables inputs, classes but you are passing labels to your criterion.