Instruction: Pytorch: NN function approximator, 2 in 1 out

Input:
[Please be aware of the Edit History below, as the major problem statement has changed.]

We are trying to implement a neural network in pytorch, that approximates a function f(x,y)=z. So there are two real numbers as input and one as ouput, we therefore want 2 nodes in the input layer and one in the output layer. We constructed a test set of 5050 samples and had pretty good results for that task in Keras with Tensorflow backend, with 3 hidden layers with a configuration of the nodes like: 2(in) - 4 - 16 - 4 - 1(out); and ReLU activation functions on all hidden layers, linear on in- and output.

Now in Pytorch we tried to implement a similar network but our loss function still literally explodes: It changes in the first few steps and converges then to some value around 10^7. In Keras we had an error around 10 percent. We already tried different network configurations without any improvement. Maybe someone could have a look on our code and suggest any change?

To explain: tr_data is a list, containing 5050 2*1 numpy arrays which are the inputs for the network. tr_labels is a list, containing 5050 numbers which are the outputs we want to learn. loadData() just load those two lists.

import torch.nn as nn
import torch.nn.functional as F

BATCH_SIZE     = 5050
DIM_IN         = 2
DIM_HIDDEN_1   = 4
DIM_HIDDEN_2   = 16
DIM_HIDDEN_3   = 4
DIM_OUT        = 1
LEARN_RATE     = 1e-4
EPOCH_NUM      = 500

class Net(nn.Module):
    def __init__(self):
        #super(Net, self).__init__()
        super().__init__()
        self.hidden1 = nn.Linear(DIM_IN, DIM_HIDDEN_1)
        self.hidden2 = nn.Linear(DIM_HIDDEN_1, DIM_HIDDEN_2)
        self.hidden3 = nn.Linear(DIM_HIDDEN_2, DIM_HIDDEN_3)
        self.out     = nn.Linear(DIM_HIDDEN_3, DIM_OUT)

    def forward(self, x):
        x = F.relu(self.hidden1(x))
        x = F.tanh(self.hidden2(x))
        x = F.tanh(self.hidden3(x))
        x = self.out(x)
        return x

model = Net()

loss_fn = nn.MSELoss(size_average=False)

optimizer = torch.optim.Adam(model.parameters(), lr=LEARN_RATE)

tr_data,tr_labels = loadData()

tr_data_torch    = torch.zeros(BATCH_SIZE, DIM_IN)
tr_labels_torch  = torch.zeros(BATCH_SIZE, DIM_OUT)

for i in range(BATCH_SIZE):
    tr_data_torch[i]   = torch.from_numpy(tr_data[i])
    tr_labels_torch[i] = tr_labels[i]

for t in range(EPOCH_NUM):

    labels_pred = model(tr_data_torch)
    loss = loss_fn(labels_pred, tr_labels_torch)
    #print(t, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

I have to say, those are our first steps in Pytorch, so please forgive me if there are some obvious, dumb mistakes. I appreciate any help or hint,
Thank you!

EDIT 1 ------------------------------------------------------------------

Following the comments and answers, we improved our code. The Loss function has now for the first time reasonable values, around 250. Our new class definition looks like:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #super().__init__()
        self.hidden1 = nn.Sequential(nn.Linear(DIM_IN, DIM_HIDDEN_1), nn.ReLU())
        self.hidden2 = nn.Sequential(nn.Linear(DIM_HIDDEN_1, DIM_HIDDEN_2), nn.ReLU())
        self.hidden3 = nn.Sequential(nn.Linear(DIM_HIDDEN_2, DIM_HIDDEN_3), nn.ReLU())
        self.out = nn.Linear(DIM_HIDDEN_3, DIM_OUT)

    def forward(self, x):
        x = self.hidden1(x)
        x = self.hidden2(x)
        x = self.hidden3(x)
        x = self.out(x)
        return x

and the loss function:

loss_fn = nn.MSELoss(size_average=True, reduce=True)

As we stated before, we already had far more satisfying results in keras with tensorflow backend. The loss function was around 30, with a similar network configuration. I share the essential parts(!) of our keras code here:

model = Sequential()
model.add(Dense(4, activation="linear", input_shape=(2,)))
model.add(Dense(16, activation="relu"))
model.add(Dense(4, activation="relu"))
model.add(Dense(1, activation="linear" ))
model.summary()

model.compile ( loss="mean_squared_error", optimizer="adam", metrics=["mse"] )

history=model.fit ( np.array(tr_data), np.array(tr_labels), \
                    validation_data = ( np.array(val_data), np.array(val_labels) ),
                    batch_size=50, epochs=200, callbacks = [ cbk ] )

Thank your already for all the help! If anybody still has suggestions to improve the network, we would be happy about it. As somebody already asked for the data, we want to share a pickle file here:

https://mega.nz/#!RDYxSYLY!P4a9mEDtZ7A5Bl7ZRjRk8EzLXQt2gyURa3wN3NCWFPA

together with the code to access it:

import pickle
f=open("data.pcl","rb")
tr_data=pickle.load ( f )
tr_labels=pickle.load ( f )
val_data=pickle.load ( f )
val_labels=pickle.load ( f )
f.close()

Output:
It should be interesting for you to point out the differences between torch.nn and torch.nn.functional (see here). Essentially, it might be that your backpropagation graph might be executed not 100% correct due to a different specification.

As pointed out by previous commenters, I would suggest to define your layers including the activations. My personal favorite way is to use nn.Sequential(), which allows you to specify multiple opeations chained together, like so:

self.hidden1 = nn.Sequential(nn.Linear(DIM_IN, DIM_HIDDEN1), nn.ReLU())

and then simply calling self.hidden1 later (without wrapping it in F.relu()).

May I also ask why you do not call the commented super(Net, self).__init__() (which is the generally recommended way)?
Additionally, if that should not fix the problem, can you maybe just share the code for Keras in comparison?


Instruction: Model taking long time to train

Input:
I have added an LSTM layer after a convolution in the VGG-16 model using PyTorch. Overtime, the model learns just fine. However, after adding just one LSTM layer, which consists of 32 LSTM cells, the process of training and evaluating takes about 10x longer.

I added the LSTM layer to a VGG framework as follows

def make_layers(cfg, batch_norm=False):
   # print("Making layers!")
    layers = []
    in_channels = 3
    count=0
    for v in cfg:
        count+=1
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
                in_channels=v
        if count==5:
            rlstm =RLSTM(v)
            rlstm=rlstm.cuda()
            layers+=[rlstm]

RLSTM is my custom class, which implements RowLSTM, from Google’s Pixel RNN paper.

Is this a common issue?  Do LSTM layers just take long to train in general?

Output: Yes, since LSTM (and many other RNNs) rely on sequential feeding of information you lose a big portion of parallelization speed ups you generally have with CNNs. There are other types of RNNs you can explore that leverage more parallelizable algorithms but the verdict on their predictive performance compared to LSTM/GRU is still not out


Instruction: How do you change require_grad to false for each parameters in your model in pytorch?

Input:
My code is below which I thought would do what I want but the output shows require_grad didn't change to false.

import torch
import torch.nn as nn

encoder = nn.Sequential( nn.Conv2d(1, 4, 1), nn.Sigmoid())

for params in encoder.parameters():

    params.require_grad = False
    print(params.requires_grad) # prints two True statements?

What am I doing wrong?

Output:
You just have a typo :) Simply add an s at the end of grad in     params.require_grad = False

Change this to params.requires_grad = False (note the added s)

Typos can be hard to catch sometimes ;)


Instruction: Cloud Storage Buckets for PyTorch

Input:
For a particular task I'm working on I have a dataset that is about 25 GB. I'm still experimenting with several methods of preprocessing and definitely don't have my data to it's final form yet. I'm not sure what the common workflow is for this sort of problem, so here is what I'm thinking:

Copy dataset from bucket storage to Compute Engine machine SSD (maybe use around 50 GB SSD) using gcsfuse.
Apply various preprocessing operations as an experiment.
Run training with PyTorch on the data stored on the local disk (SSD)
Copy newly processed data back to storage bucket with gcsfuse if it was successful.
Upload results and delete the persistent disk that was used during training.

The alternative approach is this:

Run the processing operations on the data within the Cloud Bucket itself using the mounted directory with gcsfuse
Run training with PyTorch directly on the mounted gcsfuse Bucket directory, using a compute engine instance with very limited storage.
Upload Results and Delete Compute Engine Instance.

Which of these approaches is suggested? Which will incur fewer charges and is used most often when running these kind of operations. Is there a different workflow that I'm not seeing here?

Output: On the billing side, the charges would be the same, as the fuse operations are charged like any other Cloud Storage interface according to the documentation. In your use case I don’t know how you are going to train the data, but if you do more than one operation to files it would be better to have them downloaded, trained locally and then the final result uploaded, which would be 2 object operations. If you do, for example, more than one change or read to a file during the training, every operation would be an object operation. On the workflow side, the proposed one looks good to me.


Instruction: Can a Neural Network learn a simple interpolation?

Input:
I’ve tried to train a 2 layer neural network on a simple linear interpolation for a discrete function, I’ve tried lots of different learning rates as well as different activation functions, and it seems like nothing is being learned!

I’ve literally spent the last 6 hours trying to debug the following code, but it seems like there’s no bug! What's the explanation?

    from torch.utils.data import Dataset
    import os
    import torch
    import numpy as np
    import torch.nn as nn
    import torch.optim as optim
    import random

    LOW_X=255

    MID_X=40000

    HIGH_X=200000

    LOW_Y=torch.Tensor([0,0,1])
    MID_Y=torch.Tensor([0.2,0.5,0.3])
    HIGH_Y=torch.Tensor([1,0,0])

    BATCH_SIZE=4

    def x_to_tensor(x):
        if x<=MID_X:
            return LOW_Y+(x-LOW_X)*(MID_Y-LOW_Y)/(MID_X-LOW_X)
        if x<=HIGH_X:
            return MID_Y+(x-MID_X)*(HIGH_Y-MID_Y)/(HIGH_X-MID_X)
        return HIGH_Y

    class XYDataset(Dataset):
        LENGTH=10000
        def __len__(self):
            return self.LENGTH

        def __getitem__(self, idx):
            x=random.randint(LOW_X,HIGH_X)
            y=x_to_tensor(x)
            return x,y

    class Interpolate(nn.Module):

        def __init__(self, num_outputs,hidden_size=10):
            super(Interpolate, self).__init__()
            self.hidden_size=hidden_size
            self.x_to_hidden = nn.Linear(1, hidden_size)
            self.hidden_to_out = nn.Linear(hidden_size,num_outputs)
            self.activation = nn.Tanh() #I have tried Sigmoid and Relu activations as well
            self.softmax=torch.nn.Softmax(dim=1)

        def forward(self, x):
            out = self.x_to_hidden(x)
            out = self.activation(out)
            out = self.hidden_to_out(out)
            out = self.softmax(out)
            return out

    dataset=XYDataset()

    trainloader = torch.utils.data.DataLoader(dataset, batch_size=BATCH_SIZE,
                                              shuffle=True, num_workers=4)

    criterion= nn.MSELoss()

    def train_net(net,epochs=10,lr=5.137871216190041e-05,l2_regularization=2.181622809797563e-12):
            optimizer= optim.Adam(net.parameters(),lr=lr,weight_decay=l2_regularization)
            net.train(True)
            running_loss=0.0
            for epoch in range(epochs):

                for i,data in enumerate(trainloader):
                    inputs,targets=data
                    inputs,targets=torch.FloatTensor(inputs.float()).view(-1,1),torch.FloatTensor(targets.float())
                    optimizer.zero_grad()
                    outputs=net(inputs)
                    loss=criterion(outputs,targets)
                    loss.backward()
                    optimizer.step()
                    running_loss+=loss.item()
                    if (len(trainloader)*epoch+i)%200==199:
                        running_loss=running_loss/(200*BATCH_SIZE)
                        print('[%d,%5d] loss: %.6f ' % (epoch+1,i+1,running_loss))
                        running_loss=0.0

    for i in range(-11,3):
        net=Interpolate(num_outputs=3)
        train_net(net,lr=10**i,epochs=1)
        print('for learning rate {} net output on low x is {}'.format(i,net(torch.Tensor([255]).view(-1,1))))

Output:
Although your problem is quite simple, it is poorly scaled: x ranges from 255 to 200K. This poor scaling leads to numerical instability and overall makes the training process unnecessarily unstable.
To overcome this technical issue, you simply need to scale your inputs to [-1, 1] (or [0, 1]) range.

Note that this scaling is quite ubiquitous in deep-learning: images are scaled to [-1, 1] range (see, e.g., torchvision.transforms.Normalize).
To understand better the importance of scaled responses, you can look into the mathematical analysis done in this paper.


Instruction: pytorch how to remove cuda() from tensor

Input:
I got TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor).

How do I convert torch.cuda.FloatTensor to torch.LongTensor?

  Traceback (most recent call last):
  File "train_v2.py", line 110, in <module>
    main()
  File "train_v2.py", line 81, in main
    model.update(batch)
  File "/home/Desktop/squad_vteam/src/model.py", line 131, in update
    loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
  File "/home/Desktop/squad_vteam/src/model.py", line 94, in adversarial_loss
    adv_embedding = torch.LongTensor(adv_embedding)
TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor)

Output:
You have a float tensor f and want to convert it to long, you do long_tensor = f.long()

You have cuda tensor i.e data is on gpu and want to move it to cpu you can do cuda_tensor.cpu().

So to convert a torch.cuda.Float tensor A to torch.long do A.long().cpu()


Instruction: Data Augmentation in PyTorch

Input:
I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn't seem like happening in PyTorch. As far as I understood from the references, when we use data.transforms in PyTorch, then it applies them one by one. So for example:

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Here , for the training, we are first randomly cropping the image and resizing it to shape (224,224). Then we are taking these (224,224) images and horizontally flipping them. Therefore, our dataset is now containing ONLY the horizontally flipped images, so our original images are lost in this case.

Am I right? Is this understanding correct? If not, then where do we tell PyTorch in this code above (taken from Official Documentation) to keep the original images and resize them to the expected shape (224,224)?

Thanks

Output:
The transforms operations are applied to your original images at every batch generation. So your dataset is left unchanged, only the batch images are copied and transformed every iteration.

The confusion may come from the fact that often, like in your example, transforms are used both for data preparation (resizing/cropping to expected dimensions, normalizing values, etc.) and for data augmentation (randomizing the resizing/cropping, randomly flipping the images, etc.).

What your data_transforms['train'] does is:

Randomly resize the provided image and randomly crop it to obtain a (224, 224) patch
Apply or not a random horizontal flip to this patch, with a 50/50 chance
Convert it to a Tensor
Normalize the resulting Tensor, given the mean and deviation values you provided

What your data_transforms['val'] does is:

Resize your image to (256, 256)
Center crop the resized image to obtain a (224, 224) patch
Convert it to a Tensor
Normalize the resulting Tensor, given the mean and deviation values you provided

(i.e. the random resizing/cropping for the training data is replaced by a fixed operation for the validation one, to have reliable validation results)

If you don't want your training images to be horizontally flipped with a 50/50 chance, just remove the transforms.RandomHorizontalFlip() line.

Similarly, if you want your images to always be center-cropped, replace transforms.RandomResizedCrop by transforms.Resize and transforms.CenterCrop, as done for data_transforms['val'].


Instruction: AttributeError: 'Image' object has no attribute 'new' occurs when trying to use Pytorchs AlexNet Lighting preprocessing

Input:
I tried to train my model on ImageNet using inception and Alexnet like preprocessing. I used Fast-ai imagenet training script provided script.   Pytorch has support for inception like preprocessing but for AlexNets Lighting, we have to implement it ourselves :

__imagenet_pca = {
    'eigval': torch.Tensor([0.2175, 0.0188, 0.0045]),
    'eigvec': torch.Tensor([
        [-0.5675,  0.7192,  0.4009],
        [-0.5808, -0.0045, -0.8140],
        [-0.5836, -0.6948,  0.4203],
    ])
}

# Lighting data augmentation taken from here - https://github.com/eladhoffer/convNet.pytorch/blob/master/preprocess.py
class Lighting(object):
    """Lighting noise(AlexNet - style PCA - based noise)"""

    def __init__(self, alphastd, eigval, eigvec):
        self.alphastd = alphastd
        self.eigval = eigval
        self.eigvec = eigvec

    def __call__(self, img):
        if self.alphastd == 0:
            return img

        alpha = img.new().resize_(3).normal_(0, self.alphastd)
        rgb = self.eigvec.type_as(img).clone()\
            .mul(alpha.view(1, 3).expand(3, 3))\
            .mul(self.eigval.view(1, 3).expand(3, 3))\
            .sum(1).squeeze()
        return img.add(rgb.view(3, 1, 1).expand_as(img))

which is finally used like this :

train_tfms = transforms.Compose([
        transforms.RandomResizedCrop(size),
        transforms.RandomHorizontalFlip(),
        transforms.ColorJitter(.4,.4,.4),
        transforms.ToTensor(),
        Lighting(0.1, __imagenet_pca['eigval'], __imagenet_pca['eigvec']),
        normalize,
    ])
    train_dataset = datasets.ImageFolder(traindir, train_tfms)
    train_sampler = (torch.utils.data.distributed.DistributedSampler(train_dataset)
                     if args.distributed else None)
    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler)

However, the problem is, whenever I run the script I get the:

  'AttributeError: 'Image' object has no attribute 'new''

Which complains about this line:

alpha = img.new().resize_(3).normal_(0, self.alphastd)

I am clueless as to why this is happening. I'm using Pytorch 0.4 by the way.

Output:
Thanks to @iacolippo's comment, I finally found the cause!

Unlike the example I wrote here, in my actual script, I had used transforms.ToTensor() after the lighting() method. Doing so resulted in a PIL image being sent as the input for lightining()which expects a Tensor and that's why the error occurs.

So basically the snippet I posted in the question is correct and .ToTensor has to be used prior to calling Lighting().


Instruction: CUDA runtime error (59) : device-side assert triggered

Input:
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "main.py", line 109, in <module>
    train(loader_train, model, criterion, optimizer)
  File "main.py", line 54, in train
    optimizer.step()
  File "/usr/local/anaconda35/lib/python3.6/site-packages/torch/optim/sgd.py", line 93, in step
    d_p.add_(weight_decay, p.data)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:265

How do I resolve this error?

Output:
In general, when encountering cuda runtine errors, it is advisable to run your program again using the CUDA_LAUNCH_BLOCKING=1 flag to obtain an accurate stack trace.

In your specific case, the targets of your data were too high (or low) for the specified number of classes.


Instruction: Best way to bound outputs from neural networks on reinforcement learning

Input:
I am training a neural network (feedforward, Tanh hidden layers) that receives states as inputs and gives actions as outputs. I am following the REINFORCE algorithm for policy-gradient reinforcement learning.

However, I need my control actions to be bounded (let us say from 0-5). Currently the way I am doing this is by using a sigmoid output function and multiplying the output by 5. Although my algorithm has a moderate performance, I find the following drawback from using this “bounding scheme” for the output:

I know for regression (hence I guess for reinforcement learning) a linear output is best, and although the sigmoid has a linear part I am afraid the network has not been able to capture this linear output behaviour correctly, or it captures it way too slowly (as its best performance is for classification, therefore polarizing the output).

I am wondering what other alternatives there are, and maybe some heuristics on the matter.

Output:
Have you considered using nn.ReLU6()? This is a bounded version of the rectified linear unit, which output is defined as

out = min( max(x, 0), 6)


Instruction: How to construct a network with two inputs in PyTorch

Input:
Suppose I want to have the general neural network architecture:

Input1 --> CNNLayer
                    \
                     ---> FCLayer ---> Output
                    /
Input2 --> FCLayer

Input1 is image data, input2 is non-image data. I have implemented this architecture in Tensorflow.

All pytorch examples I have found are one input go through each layer. How can I define forward func to process 2 inputs separately then combine them in a middle layer?

Output:
By "combine them" I assume you mean to concatenate the two inputs.
Assuming you concat along the second dimension:

import torch
from torch import nn

class TwoInputsNet(nn.Module):
  def __init__(self):
    super(TwoInputsNet, self).__init__()
    self.conv = nn.Conv2d( ... )  # set up your layer here
    self.fc1 = nn.Linear( ... )  # set up first FC layer
    self.fc2 = nn.Linear( ... )  # set up the other FC layer

  def forward(self, input1, input2):
    c = self.conv(input1)
    f = self.fc1(input2)
    # now we can reshape `c` and `f` to 2D and concat them
    combined = torch.cat((c.view(c.size(0), -1),
                          f.view(f.size(0), -1)), dim=1)
    out = self.fc2(combined)
    return out

Note that when you define the number of inputs to self.fc2 you need to take into account both out_channels of self.conv as well as the output spatial dimensions of c.


Instruction: Issue with running a single prediction with PyTorch

Input:
I have a trained model using PyTorch now I want to simpy run it on one example

>>> model
nn.Sequential {
  [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> output]
  (0): nn.SpatialConvolutionMap
  (1): nn.Tanh
  (2): nn.SpatialMaxPooling(2x2, 2, 2)
  (3): nn.SpatialConvolutionMap
  (4): nn.Tanh
  (5): nn.SpatialMaxPooling(2x2, 2, 2)
  (6): nn.Reshape(6400)
  (7): nn.Linear(6400 -> 128)
  (8): nn.Tanh
  (9): nn.Linear(128 -> 5)
  (10): nn.LogSoftMax
}

Then I load an image from my test set:

image = cv2.imread('image.png',cv2.IMREAD_GRAYSCALE)
transformation = transforms.Compose([transforms.ToTensor()])
image_tensor = transformation(image).float()
inp = Variable(image_tensor)

and finally try to run the network

output = model(inp)

But I get error TypeError: 'Sequential' object is not callable

Output:
It seems like your model is not nn.Sequential (pytorch Sequential), but rather torch.legacy.nn.Sequential (a legacy lua torch model).
Try using this model forward() explicitly:

output = model.forward(inp[None, ...])  # don't forget to add "batch" dimension


Instruction: Make personnal Dataloader with PYTORCH

Input:
I'm searching to create a personnal dataloader with a specific format to use Pytorch library, someone have an idea how can I do it ? I have follow Pytorch Tutorial but I don't find my answer!

I need a DataLoader that yields the tuples of the following format:
        (Bx3xHxW FloatTensor x, BxHxW LongTensor y, BxN LongTensor y_cls) where
        x - batch of input images,
        y - batch of groung truth seg maps,
        y_cls - batch of 1D tensors of dimensionality N: N total number of classes,
        y_cls[i, T] = 1 if class T is present in image i, 0 otherwise

I hope that someone can unlock the problem .. :) Thanks !

Output:
You simply need to have a database derived from torch.utils.data.Dataset, where __getitem__(index) returns a tuple (x, y, y_cls) of the types you want, pytorch will take care of everything else.

from torch.utils import data

class MyTupleDataset(data.Dataset):
  def __init__(self):
    super(MyTupleDataset, self).__init__()
    # init your dataset here...

  def __getitem__(index):
    x = torch.Tensor(3, H, W)  # batch dim is handled by the data loader
    y = torch.Tensor(H, W).to(torch.long)
    y_cls = torch.Tensor(N).to(torch.long)
    return x, y, y_cls

That's it. Provide pytorch's  torch.utils.data.DataLoader with MyTupleDataset  and you are done.


Instruction: Pytorch tensor - How to get the indexes by a specific tensor

Input:
I have a tensor

t = torch.tensor([[1, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], [1, 0, 0, 0]])

and a query tensor

q = torch.tensor([1, 0, 0, 0])

Is there a way to get the indexes of q like

indexes = t.index(q) # get back [0, 3]

in pytorch?

Output:
How about

In [1]: torch.nonzero((t == q).sum(dim=1) == t.size(1))
Out[1]:
tensor([[ 0],
        [ 3]])

Comparing t == q performs element-wise comparison between t and q, since you are looking for entire row match, you need to .sum(dim=1) along the rows and see what row is a perfect match == t.size(1).

As of v0.4.1, torch.all() supports dim argument:

torch.all(t==q, dim=1)


Instruction: Bidirectional RNN cells - shared or not?

Input: Should I use the same weights to compute forward and backward passes in a bidirectional RNN, or should those weights be learned independently?

Output:
They should be learned independently as they learn different patterns, unless you have palindromes. In fact that is the default in the Bidirectional wrapper in Keras:

self.forward_layer = copy.copy(layer)
config = layer.get_config()
config['go_backwards'] = not config['go_backwards']
self.backward_layer = layer.__class__.from_config(config)

In the above source code the opposite direction is a copy with independent weights from the original direction.


Instruction: Why is it in Pytorch when I make a COPY of a network's weight it would be automatically updated after back-propagation?

Input:
I wrote the following code as a test because in my original network I use a ModuleDict and depends on what index I feed it would slice and train only parts of that network.

I wanted to make sure that only the sliced layers would update their weight so I wrote some test code to double check. Well I am getting some weird results. Say if my model has 2 layers, layer1 is an FC and layer 2 is Conv2d, if I slice the network and ONLY use layer2 I would expect layer1's weight to be unchanged because they are unused and layer2's weight will get updated after 1 epoch.

So my plan was to used a for loop to grab all the weights from the network Before training then I would do it AFTER 1 optimizer.step(). Both times I would store those weights completely separate in 2 Python lists so I can compare their results later. Well for some reason the two lists are completely the same if I compare them with torch.equal() I thought its because maybe there is still some sort of hidden link in memory? So I tried to use .detach() on the weights when I grab them from the loop and the result is still the same. Layer2's weight should be different in this case because it should contain weights from the network before training.

Noted in the code below I am actually using layer1 and ignoring layer2.

Full code:

class mymodel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 5)
        self.layer2 = nn.Conv2d(1, 5, 4, 2, 1)
        self.act = nn.Sigmoid()
    def forward(self, x):
        x = self.layer1(x) #only layer1 and act are used layer 2 is ignored so only layer1 and act's weight should be updated
        x = self.act(x)
        return x
model = mymodel()

weights = []

for param in model.parameters(): # loop the weights in the model before updating and store them
    print(param.size())
    weights.append(param)

critertion = nn.BCELoss() #criterion and optimizer setup
optimizer = optim.Adam(model.parameters(), lr = 0.001)

foo = torch.randn(3, 10) #fake input
target = torch.randn(3, 5) #fake target

result = model(foo) #predictions and comparison and backprop
loss = criterion(result, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()

weights_after_backprop = [] # weights after backprop
for param in model.parameters():
    weights_after_backprop.append(param) # only layer1's weight should update, layer2 is not used

for i in zip(weights, weights_after_backprop):
    print(torch.equal(i[0], i[1]))

# **prints all Trues when "layer1" and "act" should be different, I have also tried to call param.detach in the loop but I got the same result.

Output:
You have to clone the parameters, otherwise you just copy the reference.

weights = []

for param in model.parameters():
    weights.append(param.clone())

criterion = nn.BCELoss() # criterion and optimizer setup
optimizer = optim.Adam(model.parameters(), lr=0.001)

foo = torch.randn(3, 10) # fake input
target = torch.randn(3, 5) # fake target

result = model(foo) # predictions and comparison and backprop
loss = criterion(result, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()

weights_after_backprop = [] # weights after backprop
for param in model.parameters():
    weights_after_backprop.append(param.clone()) # only layer1's weight should update, layer2 is not used

for i in zip(weights, weights_after_backprop):
    print(torch.equal(i[0], i[1]))

which gives

False
False
True
True


Instruction: Skorch training object from scratch

Input:
I'm trying to use skorch class to execut GridSearch on a classifier.
I tried running with the vanilla NeuralNetClassifier object, but I haven't found a way to pass the Adam optimizer only the trainable weights (I'm using pre-trained embeddings and I would like to keep them frozen). It's doable if a module is initialized, and then pass those weights with the optimizer__params option, but module needs an uninitialized model. Is there a way around this?

net = NeuralNetClassifier(module=RNN, module__vocab_size=vocab_size, module__hidden_size=hidden_size,
                          module__embedding_dim=embedding_dim, module__pad_id=pad_id,
                          module__dataset=ClaimsDataset, lr=lr, criterion=nn.CrossEntropyLoss,
                          optimizer=torch.optim.Adam, optimizer__weight_decay=35e-3, device='cuda',
                          max_epochs=nb_epochs, warm_start=True)

The code above works. However, with the batch_size set at 64, I've got to run the model for the specified number of epochs on every batch! Which is not the behavior I'm seeking. I'd be grateful if someone could suggest a nicer way to do this.

My other issue is with subclassing skorch.NeuralNet. I run into a similar issue: figuring out a way to pass only the trainable weights to Adam optimizer. The code below is what I've got so far.

class Train(skorch.NeuralNet):
def __init__(self, module, lr, norm, *args, **kwargs):
    self.module = module
    self.lr = lr
    self.norm = norm
    self.params = [p for p in self.module.parameters(self) if p.requires_grad]
    super(Train, self).__init__(*args, **kwargs)

def initialize_optimizer(self):
    self.optimizer = torch.optim.Adam(params=self.params, lr=self.lr, weight_decay=35e-3, amsgrad=True)

def train_step(self, Xi, yi, **fit_params):
    self.module.train()

    self.optimizer.zero_grad()
    yi = variable(yi)

    output = self.module(Xi)

    loss = self.criterion(output, yi)
    loss.backward()

    nn.utils.clip_grad_norm_(self.params, max_norm=self.norm)
    self.optimizer.step()

def score(self, y_t, y_p):
    return accuracy_score(y_t, y_p)

Initializing the class gives the error:

Traceback (most recent call last):
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1658, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1068, in run
pydev_imports.execfile(file, globals, locals)  # execute the script
File "/snap/pycharm-community/74/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/l/Documents/Bsrc/cv.py", line 115, in <module>
    main()
File "/home/l/B/src/cv.py", line 86, in main
    trainer = Train(module=RNN, criterion=nn.CrossEntropyLoss, lr=lr, norm=max_norm)
File "/home/l/B/src/cv.py", line 22, in __init__
   self.params = [p for p in self.module.parameters(self) if p.requires_grad]
File "/home/l/B/src/cv.py", line 22, in <listcomp>
   self.params = [p for p in self.module.parameters(self) if p.requires_grad]
File "/home/l/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 739, in parameters
    for name, param in self.named_parameters():
AttributeError: 'Train' object has no attribute 'named_parameters'

Output:
  but module needs an uninitialized model

That is not correct, you can pass an initialized model as well. The documentation  of the model parameter states:

  It is, however, also possible to pass an instantiated module, e.g. a PyTorch Sequential instance.

The problem is that when passing an initialized model you cannot pass any module__ parameters to the NeuralNet as this would require the module to be re-initialized. But of course that's problematic if you want to do a grid search over module parameters.

A solution for this would be to overwrite initialize_model and after creating a new instance loading and freezing the parameters (by setting the parameter's requires_grad attribute to False):

def _load_embedding_weights(self):
    return torch.randn(1, 100)

def initialize_module(self):
    kwargs = self._get_params_for('module')
    self.module_ = self.module(**kwargs)

    # load weights
    self.module_.embedding0.weight = self._load_embedding_weights()
    # freeze layer
    self.module_.embedding0.weight.requires_grad = False

    return self


Instruction: Where do I get a CPU-only version of PyTorch?

Input:
I'm trying to get a basic app running with Flask + PyTorch, and host it on Heroku. However, I run into the issue that the maximum slug size is 500mb on the free version, and PyTorch itself is ~500mb.

After some google searching, someone wrote about finding a cpu-only version of PyTorch, and using that, which is much smaller here.

However, I'm pretty lost as to how this is done, and the person didn't document this at all. Any advice is appreciated, thanks.

EDIT:

To be more specific about my problem, I tried installing torch by (as far as I understand), including a requirements.txt which listed torch as a dependency. Current I have: torch==0.4.1. However this doesn't work bc of size.

My question is, do you know what I could write in the requirements file to get the cpu-only version of torch that is smaller, or alternatively, if the requirements.txt doesn't work for this, what I would do instead, to get the cpu version.

Output:
Per the Pytorch website, you can install pytorch-cpu with

conda install pytorch-cpu torchvision-cpu -c pytorch

You can see from the files on Anaconda cloud, that the size varies between 26 and 56MB depending on the OS where you want to install it.

You can get the wheel from http://download.pytorch.org/whl/cpu/.
The wheel is 87MB.

You can setup the installation by putting the link to the wheel in the requirements.txt file. If you use Python 3.6 on Heroku:

http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl

otherwise, for Python 2.7:

http://download.pytorch.org/whl/cpu/torch-0.4.1-cp27-cp27mu-linux_x86_64.whl

For example if your requirements are pytorch-cpu, numpy and scipy and you're using Python 3.6, the requirements.txt would look like:

http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl
numpy
scipy


Instruction: How to include batch size in pytorch basic example?

Input:
I am new to pytorch. The following is the basic example of using nn module to train a simple one-layer model with some random data (from here)

import torch
N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
for t in range(500):
    y_pred = model(x)

    loss = loss_fn(y_pred, y)
    print(t, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

As far as I understand, the batch size is equal to 1 in the example, in other words, a single point (out of 64) is used to calculate gradients and update parameters. My question is: how to modify this example to train the model with the batch size greater than one?

Output:
In fact N is the batch size. So you just need to modify N currently its set to 64. So you have in every training batch 64 vectors with size / dim D_in.

I checked the link you posted, you can also take a look at the comments - there is some explanation too :)

# -*- coding: utf-8 -*-
import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2


Instruction: Is .data still useful in pytorch?

Input: I'm new to pytorch. I read much pytorch code which heavily uses tensor's .data member. But I search .data in the official document and Google, finding little. I guess .data contains the data in the tensor, but I don't know when we need it and when not?

Output:
.data was an attribute of Variable (object representing Tensor with history tracking e.g. for automatic update), not Tensor. Actually, .data was giving access to the Variable's underlying Tensor.

However, since PyTorch version 0.4.0, Variable and Tensor have been merged (into an updated Tensor structure), so .data disappeared along the previous Variable object (well Variable is still there for backward-compatibility, but is deprecated).

Paragraph from Release Notes for version 0.4.0 (I recommend reading the whole section about Variable/Tensor updates):

  What about .data?

  .data was the primary way to get the underlying Tensor from a
  Variable. After this merge, calling y = x.data still has similar
  semantics. So y will be a Tensor that shares the same data with
  x, is unrelated with the computation history of x, and has
  requires_grad=False.

  However, .data can be unsafe in some cases. Any changes on x.data
  wouldn't be tracked by autograd, and the computed gradients would be
  incorrect if x is needed in a backward pass. A safer alternative is
  to use x.detach(), which also returns a Tensor that shares data
  with requires_grad=False, but will have its in-place changes
  reported by autograd if x is needed in backward.


Instruction: pytorch how to set .requires_grad False

Input:
I want to set some of my model frozen. Following the official docs:

with torch.no_grad():
    linear = nn.Linear(1, 1)
    linear.eval()
    print(linear.weight.requires_grad)

But it prints True instead of False. If I want to set the model in eval mode, what should I do?

Output:
requires_grad=False

If you want to freeze part of your model and train the rest, you can set requires_grad of the parameters you want to freeze to False.

For example, if you only want to keep the convolutional part of VGG16 fixed:

model = torchvision.models.vgg16(pretrained=True)
for param in model.features.parameters():
    param.requires_grad = False

By switching the requires_grad flags to False, no intermediate buffers will be saved, until the computation gets to some point where one of the inputs of the operation requires the gradient.

torch.no_grad()

Using the context manager torch.no_grad is a different way to achieve that goal: in the no_grad context, all the results of the computations will have requires_grad=False, even if the inputs have requires_grad=True. Notice that you won't be able to backpropagate the gradient to layers before the no_grad. For example:

x = torch.randn(2, 2)
x.requires_grad = True

lin0 = nn.Linear(2, 2)
lin1 = nn.Linear(2, 2)
lin2 = nn.Linear(2, 2)
x1 = lin0(x)
with torch.no_grad():
    x2 = lin1(x1)
x3 = lin2(x2)
x3.sum().backward()
print(lin0.weight.grad, lin1.weight.grad, lin2.weight.grad)

outputs:

(None, None, tensor([[-1.4481, -1.1789],
         [-1.4481, -1.1789]]))

Here lin1.weight.requires_grad was True, but the gradient wasn't computed because the oepration was done in the no_grad context.

model.eval()

If your goal is not to finetune, but to set your model in inference mode, the most convenient way is to use the torch.no_grad context manager. In this case you also have to set your model to evaluation mode, this is achieved by calling eval() on the nn.Module, for example:

model = torchvision.models.vgg16(pretrained=True)
model.eval()

This operation sets the attribute self.training of the layers to False, in practice this will change the behavior of operations like Dropout or BatchNorm that must behave differently at training and test time.


Instruction: PyTorch - Effect of normal() initialization on gradients

Input:
Suppose I have a neural network where I use a normal distribution initialization and I want to use the mean value which is used for initialization as a parameter of the network.

I have a small example:

import torch
parameter_vector = torch.tensor(range(10), dtype=torch.float, requires_grad=True)
sigma = torch.ones(parameter_vector.size(0), dtype=torch.float)*0.1
init_result = torch.normal(parameter_vector, sigma)
print('requires_grad:', init_result.requires_grad)
print('result:       ', init_result)

This results in:

requires_grad: True
result:        tensor([ 0.1026,  0.9183,  1.9586,  3.1778,  4.0538,  4.8056,  5.9561,
         6.9501,  7.7653,  8.9583])

So the requires_grad flag was obviously taken over from the mean value tensor resp. parameter_vector.

But does this automatically mean that the parameter_vector will be updated through backward() in a larger network where init_result does affect the end result?

Especially as normal() does not really seem like normal operation because it involves randomness.

Output:
Thanks to @iacolippo (see comments below the question) the problem is solved now. I just wanted to supplement this by posting what code I am using now, so this may help anyone else.
As presumed in the question and also stated by @iacolippo the code posted in the question is not backpropable:
import torch
parameter_vector = torch.tensor(range(5), dtype=torch.float, requires_grad=True)
print('- initial parameter weights:', parameter_vector)
sigma = torch.ones(parameter_vector.size(0), dtype=torch.float)*0.1
init_result = torch.normal(parameter_vector, sigma)
print('- normal init result requires_grad:', init_result.requires_grad)
print('- normal init vector', init_result)
#print('result:       ', init_result)
sum_result = init_result.sum()
sum_result.backward()
print('- summed dummy-loss:', sum_result)
optimizer = torch.optim.SGD([parameter_vector], lr = 0.01, momentum=0.9)
optimizer.step()
print()
print('- parameter weights after update:', parameter_vector)

Out:
- initial parameter weights: tensor([0., 1., 2., 3., 4.], requires_grad=True)
- normal init result requires_grad: True
- normal init vector tensor([-0.0909,  1.1136,  2.1143,  2.8838,  3.9340], grad_fn=<NormalBackward3>)
- summed dummy-loss: tensor(9.9548, grad_fn=<SumBackward0>)

- parameter weights after update: tensor([0., 1., 2., 3., 4.], requires_grad=True)

As you can see calling backward() does not raise an error (see linked issue in comments above), but the parameters won't get updated either with SGD-Step.

Working Example 1
One solution is to use the formula/trick given here: https://stats.stackexchange.com/a/342815/133099

x=μ+σ sample(N(0,1))

To archive this:
sigma = torch.ones(parameter_vector.size(0), dtype=torch.float)*0.1
init_result = torch.normal(parameter_vector, sigma)

Changes to:
dim = parameter_vector.size(0)
sigma = 0.1
init_result = parameter_vector + sigma*torch.normal(torch.zeros(dim), torch.ones(dim))

After changing these lines the code gets backprobable and the parameter vector gets updated after calling backward() and SGD-Step.
Output with changed lines:
- initial parameter weights: tensor([0., 1., 2., 3., 4.], requires_grad=True)
- normal init result requires_grad: True
- normal init vector tensor([-0.1802,  0.9261,  1.9482,  3.0817,  3.9773], grad_fn=<ThAddBackward>)
- summed dummy-loss: tensor(9.7532, grad_fn=<SumBackward0>)

- parameter weights after update: tensor([-0.0100,  0.9900,  1.9900,  2.9900,  3.9900], requires_grad=True)

Working Example 2
Another way would be using torch.distributions (Documentation Link).
Do do so the respective lines in the code above have to be replaced by:
i = torch.ones(parameter_vector.size(0))
sigma = 0.1
m = torch.distributions.Normal(parameter_vector, sigma*i)
init_result = m.rsample()

Output with changed lines:
- initial parameter weights: tensor([0., 1., 2., 3., 4.], requires_grad=True)
- normal init result requires_grad: True
- normal init vector tensor([-0.0767,  0.9971,  2.0448,  2.9408,  4.1321], grad_fn=<ThAddBackward>)
- summed dummy-loss: tensor(10.0381, grad_fn=<SumBackward0>)

- parameter weights after update: tensor([-0.0100,  0.9900,  1.9900,  2.9900,  3.9900], requires_grad=True)

As it can be seen in the output above - using torch.distributions yields also to backprobable code where the parameter vector gets updated after calling backward() and SGD-Step.
I hope this is helpful for someone.


Instruction: How to re-use old weights in a slightly modified model?

Input:
I have a CNN network built like this for a particular task.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)

        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x))

        x = F.softmax(x, 2)

        return x

The model is stored using the torch built-in method like this.

net = Net()
optimizer = optim.SGD(net.parameters(), lr=1e-3)
state = {
    'state_dict': net.state_dict()
    'opt': optimizer.state_dict()
}
torch.save(state, 'model.pt')

I have increased a single layer in the network while the rest of the model was kept the same.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 256, kernel_size=3, padding=1) # (new added)
        self.conv14 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)

        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x)) (new added)
        x = F.relu(self.conv14(x))

        x = F.softmax(x, 2)

        return x

Since the other conv layers are kept the same, is there any way I can re-use the saved model to load the weights to conv11, conv12 and conv14 ? Instead of starting to train from beginning ?

Output:
Assume you trained the following model and now you make a minor modification to it (like adding a layer) and want to use your trained weights

import torch
import torch.nn as nn
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)
        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x))
        x = F.softmax(x, 2)
            return x

net = Net()
optimizer = optim.SGD(net.parameters(), lr=1e-3)

you save the model (and the optimizer state) with:

state = {'state_dict': net.state_dict(),
         'opt': optimizer.state_dict()
        }
torch.save(state, 'state.pt')

Your new model is (note that corresponding layers keep the same name, so you don't make conv13 -> conv14):

class NewNet(nn.Module):
    def __init__(self):
        super(NewNet, self).__init__()
        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.convnew = nn.Conv2d(256, 256, kernel_size=3, padding=1) # (new added)
        self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)
        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.convnew(x)) # (new added)
        x = F.relu(self.conv13(x))
        x = F.softmax(x, 2)
        return x

Now you can load your model.pt file:

state = torch.load('state.pt')

state is a dict, state['opt'] contains all the parameters that you had for your optimizer, for example state['opt']['param_groups'][0]['lr'] gives

0.001

Assuming corresponding layers kept the same name, you can recover your parameters and initialize the appropriate layers by:

net = NewNet()
for name, param in net.named_parameters():
    if name in state['state_dict'].keys():
        param = param.data
        param.copy_(state['state_dict'][name])


Instruction: How do I turn a Pytorch Dataloader into a numpy array to display image data with matplotlib?

Input:
I am new to Pytorch. I have been trying to learn how to view my input images before I begin training on my CNN. I am having a very hard time changing the images into a form that can be used with matplotlib.

So far I have tried this:

from multiprocessing import freeze_support

import torch
from torch import nn
import torchvision
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np
import PIL

num_classes = 5
batch_size = 100
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToPImage(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    print(npimg)
    plt.imshow(np.transpose(npimg, (1, 2, 0, 1)))

def main():
    # get some random training images
    dataiter = iter(train_loader)
    images, labels = dataiter.next()

    # show images
    imshow(images)
    # print labels
    print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

if __name__ == "__main__":
    main()

However, this throws and error:

  [[0.27058825 0.18431371 0.31764707 ... 0.18823528 0.3882353
    0.27450982]
   [0.23137254 0.11372548 0.24313724 ... 0.16862744 0.14117646
    0.40784314]
   [0.25490198 0.19607842 0.30588236 ... 0.27450982 0.25882354
    0.34509805]
   ...
   [0.2784314  0.21960783 0.2352941  ... 0.5803922  0.46666667
    0.25882354]
   [0.26666668 0.16862744 0.23137254 ... 0.2901961  0.29803923
    0.2509804 ]
   [0.30980393 0.39607844 0.28627452 ... 0.1490196  0.10588235
    0.19607842]]

  [[0.2352941  0.06274509 0.15686274 ... 0.09411764 0.3019608
    0.19215685]
   [0.22745097 0.07843137 0.12549019 ... 0.07843137 0.10588235
    0.3019608 ]
   [0.20392156 0.13333333 0.1607843  ... 0.16862744 0.2117647
    0.22745097]
   ...
   [0.18039215 0.16862744 0.1490196  ... 0.45882353 0.36078432
    0.16470587]
   [0.1607843  0.10588235 0.14117646 ... 0.2117647  0.18039215
    0.10980392]
   [0.18039215 0.3019608  0.2117647  ... 0.11372548 0.06274509
    0.04705882]]]

 ...

 [[[0.8980392  0.8784314  0.8509804  ... 0.627451   0.627451
    0.627451  ]
   [0.8509804  0.8235294  0.7921569  ... 0.54901963 0.5568628
    0.56078434]
   [0.7921569  0.7529412  0.7176471  ... 0.47058824 0.48235294
    0.49411765]
   ...
   [0.3764706  0.38431373 0.3764706  ... 0.4509804  0.43137255
    0.39607844]
   [0.38431373 0.39607844 0.3882353  ... 0.4509804  0.43137255
    0.39607844]
   [0.3882353  0.4        0.39607844 ... 0.44313726 0.42352942
    0.39215687]]

  [[0.9254902  0.90588236 0.88235295 ... 0.60784316 0.6
    0.5921569 ]
   [0.88235295 0.85490197 0.8235294  ... 0.5411765  0.5372549
    0.53333336]
   [0.8235294  0.7882353  0.75686276 ... 0.47058824 0.47058824
    0.47058824]
   ...
   [0.50980395 0.5176471  0.5137255  ... 0.58431375 0.5647059
    0.53333336]
   [0.5137255  0.53333336 0.5254902  ... 0.58431375 0.5686275
    0.53333336]
   [0.5176471  0.53333336 0.5294118  ... 0.5764706  0.56078434
    0.5294118 ]]

  [[0.95686275 0.9372549  0.90588236 ... 0.18823528 0.19999999
    0.20784312]
   [0.9098039  0.8784314  0.8352941  ... 0.1607843  0.17254901
    0.18039215]
   [0.84313726 0.7921569  0.7490196  ... 0.1372549  0.14509803
    0.15294117]
   ...
   [0.03921568 0.05490196 0.05098039 ... 0.11764705 0.09411764
    0.02745098]
   [0.04705882 0.07843137 0.06666666 ... 0.12156862 0.10196078
    0.03529412]
   [0.05098039 0.0745098  0.07843137 ... 0.12549019 0.10196078
    0.04705882]]]

 [[[0.30588236 0.28627452 0.24313724 ... 0.2901961  0.26666668
    0.21568626]
   [0.8156863  0.6666667  0.5921569  ... 0.18039215 0.23921567
    0.21568626]
   [0.9019608  0.83137256 0.85490197 ... 0.21960783 0.36862746
    0.23921567]
   ...
   [0.7058824  0.83137256 0.85490197 ... 0.2627451  0.24313724
    0.20784312]
   [0.7137255  0.84313726 0.84705883 ... 0.26666668 0.29803923
    0.21568626]
   [0.7254902  0.8235294  0.8392157  ... 0.2509804  0.27058825
    0.2352941 ]]

  [[0.24705881 0.22745097 0.19215685 ... 0.2784314  0.25490198
    0.19607842]
   [0.59607846 0.37254903 0.29803923 ... 0.16470587 0.22745097
    0.20392156]
   [0.5921569  0.4509804  0.49803922 ... 0.20784312 0.3764706
    0.2352941 ]
   ...
   [0.42352942 0.4627451  0.42352942 ... 0.23921567 0.23137254
    0.19999999]
   [0.45882353 0.5176471  0.35686275 ... 0.23921567 0.26666668
    0.19607842]
   [0.41568628 0.44313726 0.34901962 ... 0.21960783 0.23921567
    0.21568626]]

  [[0.23137254 0.20784312 0.1490196  ... 0.30588236 0.28627452
    0.19607842]
   [0.61960787 0.3764706  0.26666668 ... 0.16470587 0.24313724
    0.21568626]
   [0.57254905 0.43137255 0.48235294 ... 0.2235294  0.40392157
    0.25882354]
   ...
   [0.4        0.42352942 0.37254903 ... 0.25490198 0.24705881
    0.21568626]
   [0.43137255 0.4509804  0.29411766 ... 0.25882354 0.28235295
    0.20392156]
   [0.38431373 0.3529412  0.25490198 ... 0.2352941  0.25490198
    0.23137254]]]

 [[[0.06274509 0.09019607 0.11372548 ... 0.5803922  0.5176471
    0.59607846]
   [0.09411764 0.14509803 0.1372549  ... 0.5294118  0.49803922
    0.5058824 ]
   [0.04705882 0.09411764 0.10196078 ... 0.45882353 0.42352942
    0.38431373]
   ...
   [0.15294117 0.12941176 0.1607843  ... 0.85882354 0.8509804
    0.80784315]
   [0.14509803 0.10588235 0.1607843  ... 0.8666667  0.85882354
    0.8       ]
   [0.1490196  0.10588235 0.16470587 ... 0.827451   0.8156863
    0.7921569 ]]

  [[0.06666666 0.12156862 0.17647058 ... 0.59607846 0.5529412
    0.6039216 ]
   [0.07058823 0.10588235 0.11764705 ... 0.56078434 0.5254902
    0.5372549 ]
   [0.03921568 0.0745098  0.09803921 ... 0.48235294 0.4392157
    0.4117647 ]
   ...
   [0.2117647  0.14509803 0.2784314  ... 0.43137255 0.3529412
    0.34117648]
   [0.2235294  0.11372548 0.2509804  ... 0.4509804  0.39607844
    0.2509804 ]
   [0.25490198 0.12156862 0.24705881 ... 0.38039216 0.36078432
    0.3254902 ]]

  [[0.05490196 0.09803921 0.12549019 ... 0.46666667 0.38039216
    0.45490196]
   [0.06274509 0.09803921 0.10196078 ... 0.44705883 0.41568628
    0.3882353 ]
   [0.03921568 0.06666666 0.0862745  ... 0.3764706  0.33333334
    0.28235295]
   ...
   [0.12156862 0.14509803 0.16862744 ... 0.15686274 0.0745098
    0.09411764]
   [0.10588235 0.11372548 0.16862744 ... 0.25882354 0.18431371
    0.05490196]
   [0.12156862 0.11372548 0.17254901 ... 0.2352941  0.17254901
    0.14117646]]]]
Traceback (most recent call last):
  File "image_loader.py", line 51, in <module>
    main()
  File "image_loader.py", line 46, in main
    imshow(images)
  File "image_loader.py", line 38, in imshow
    plt.imshow(np.transpose(npimg, (1, 2, 0, 1)))
  File "C:\Users\Aeryes\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\fromnumeric.py", line 598, in transpose
    return _wrapfunc(a, 'transpose', axes)
  File "C:\Users\Aeryes\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\fromnumeric.py", line 51, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: repeated axis in transpose

I tried to print out the arrays to get the dimensions but I do not know what to make of this. It is very confusing.

Here is my direct question: How do I view the input images before training using the tensors in my DataLoader object?

Output:
First of all, dataloader output 4 dimensional tensor - [batch, channel, height, width]. Matplotlib and other image processing libraries often requires [height, width, channel]. You are right about using the transpose, just not in the right way.

There will be a lot of images in your images so first you need to pick one (or write a for loop to save all of them). This will be simply images[i], typically I use i=0.

Then, your transpose should convert a now [channel, height, width] tensor to a [height, width, channel] one. To do this, use np.transpose(image.numpy(), (1, 2, 0)), very much like yours.

Putting them together, you should have

plt.imshow(np.transpose(images[0].numpy(), (1, 2, 0)))

Sometimes you need to call .detach() (detach this part from the computational graph) and .cpu() (transfer data from GPU to CPU) depending on the use case, that will be

plt.imshow(np.transpose(images[0].cpu().detach().numpy(), (1, 2, 0)))


Instruction: In pytorch how do you use add_param_group () with a optimizer?

Input:
The documentation is pretty vague and there aren't example codes to show you how to use it. The documentation for it is

  Add a param group to the Optimizer s param_groups.

  This can be useful when fine tuning a pre-trained network as frozen
  layers can be made trainable and added to the Optimizer as training
  progresses.

  Parameters:    param_group (dict) – Specifies what Tensors should be
  optimized along with group optimization options. (specific) –

I am assuming I can get a param_group parameter by feeding the values I get from a model's state_dict()? E.g. all the actual weight values? I am asking this because I want to make a progressive network, which means I need to constantly feed Adam parameters from newly created convolutions and activations modules.

Output:
Per the docs, the add_param_group method accepts a param_group parameter that is a dict. Example of use:

import torch
import torch.optim as optim

w1 = torch.randn(3, 3)
w1.requires_grad = True
w2 = torch.randn(3, 3)
w2.requires_grad = True
o = optim.Adam([w1])
print(o.param_groups)

gives

[{'amsgrad': False,
  'betas': (0.9, 0.999),
  'eps': 1e-08,
  'lr': 0.001,
  'params': [tensor([[ 2.9064, -0.2141, -0.4037],
           [-0.5718,  1.0375, -0.6862],
           [-0.8372,  0.4380, -0.1572]])],
  'weight_decay': 0}]

now

o.add_param_group({'params': w2})
print(o.param_groups)

gives:

[{'amsgrad': False,
  'betas': (0.9, 0.999),
  'eps': 1e-08,
  'lr': 0.001,
  'params': [tensor([[ 2.9064, -0.2141, -0.4037],
           [-0.5718,  1.0375, -0.6862],
           [-0.8372,  0.4380, -0.1572]])],
  'weight_decay': 0},
 {'amsgrad': False,
  'betas': (0.9, 0.999),
  'eps': 1e-08,
  'lr': 0.001,
  'params': [tensor([[-0.0560,  0.4585, -0.7589],
           [-0.1994,  0.4557,  0.5648],
           [-0.1280, -0.0333, -1.1886]])],
  'weight_decay': 0}]


Instruction: Why does pytorch F.mse_loss behave differently w.r.t. Tensor and Parameter?

Input:
Below is my code:

import torch as pt
from torch.nn import functional as F

a = pt.Tensor([[0, 1], [2, 3]])
b = pt.Tensor([[1, 0], [5, 4]])
print(F.mse_loss(a, b), F.mse_loss(a, b, reduction='elementwise_mean'))

a = pt.nn.Parameter(a)
b = pt.nn.Parameter(b)
print(F.mse_loss(a, b), F.mse_loss(a, b, reduction='elementwise_mean'))

The output was:

tensor(3.) tensor(3.)
tensor(12., grad_fn=<SumBackward0>) tensor(12., grad_fn=<SumBackward0>)

I wonder why they gave two different results?

Environment setting:
python 3.6
pytorch 0.4.1

Output: It is a bug according to pytorch forum.


Instruction: Is it possible to create a FIFO queue with pyTorch?

Input:
I need to create a fixed length Tensor in pyTorch that acts like a FIFO queue.

I have this fuction to do it:

def push_to_tensor(tensor, x):
    tensor[:-1] = tensor[1:]
    tensor[-1] = x
    return tensor

For example, I have:

tensor = Tensor([1,2,3,4])

>> tensor([ 1.,  2.,  3.,  4.])

then using the function will give:

push_to_tensor(tensor, 5)

>> tensor([ 2.,  3.,  4.,  5.])

However, I was wondering:

Does pyTorch have a native method for doing this?
If not, is there a more clever way of doing it?

Output:
I implemented another FIFO queue:

def push_to_tensor_alternative(tensor, x):
    return torch.cat((tensor[1:], Tensor([x])))

The functionality is the same, but then I checked their performance in speed:

# Small Tensor
tensor = Tensor([1,2,3,4])

%timeit push_to_tensor(tensor, 5)
>> 30.9 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit push_to_tensor_alternative(tensor, 5)
>> 22.1 µs ± 2.25 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Larger Tensor
tensor = torch.arange(10000)

%timeit push_to_tensor(tensor, 5)
>> 57.7 µs ± 4.88 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit push_to_tensor_alternative(tensor, 5)
>> 28.9 µs ± 570 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Seems like this push_to_tensor_alternative which uses torch.cat (instead of shifting all items to the left) is faster.


Instruction: pytorch skip connection in a sequential model

Input:
I am trying to wrap my head around skip connections in a sequential model. With the functional API I would be doing something as easy as (quick example, maybe not be 100% syntactically correct but should get the idea):

x1 = self.conv1(inp)
x = self.conv2(x)
x = self.conv3(x)
x = self.conv4(x)

x = self.deconv4(x)
x = self.deconv3(x)
x = self.deconv2(x)
x = torch.cat((x, x1), 1))
x = self.deconv1(x)

I am now using a sequential model and trying to do something similar, create a skip connection that brings the activations of the first conv layer all the way to the last convTranspose. I have taken a look at the U-net architecture implemented here and it's a bit confusing, it does something like this:

upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,
                                    kernel_size=4, stride=2,
                                    padding=1, bias=use_bias)
down = [downrelu, downconv, downnorm]
up = [uprelu, upconv, upnorm]

if use_dropout:
    model = down + [submodule] + up + [nn.Dropout(0.5)]
else:
    model = down + [submodule] + up

Isn't this just adding layers to the sequential model well, sequentially? There is the down conv which is followed by submodule (which recursively adds inner layers) and then concatenated to up which is the upconv layer. I am probably missing something important on how the Sequential API works, but how does the code snipped from U-NET actually implements the skip?

Output:
Your observations are correct, but you may have missed the definition of UnetSkipConnectionBlock.forward() (UnetSkipConnectionBlock being the Module defining the U-Net block you shared), which may clarify this implementation:

(from pytorch-CycleGAN-and-pix2pix/models/networks.py#L259)

# Defines the submodule with skip connection.
# X -------------------identity---------------------- X
#   |-- downsampling -- |submodule| -- upsampling --|
class UnetSkipConnectionBlock(nn.Module):

    # ...

    def forward(self, x):
        if self.outermost:
            return self.model(x)
        else:
            return torch.cat([x, self.model(x)], 1)

The last line is the key (applied for all inner blocks). The skip layer is simply done by concatenating the input x and the (recursive) block output self.model(x), with self.model the list of operations you mentioned -- so not so differently from the Functional code you wrote.


Instruction: Pytorch trying to make a NN received an invalid combination of arguments

Input:
I am trying to build my first NN with pytroch and got an issue.

  TypeError: new() received an invalid combination of arguments - got (float, int, int, int), but expected one of:
   * (torch.device device)
   * (torch.Storage storage)
   * (Tensor other)
   * (tuple of ints size, torch.device device)
   * (object data, torch.device device)

Now I know what this is saying in that I am not passing the right type to the method  or init. But I dont know what I should pass as it looks right to me.

def main():
#Get the time and data
now = datetime.datetime.now()
hourGlassToStack = 2 #Hourglasses to stack
numModules= 2        #Residual Modules for each hourglass
numFeats = 256      #Number of features in each hourglass
numRegModules = 2   #Depth regression modules

print("Creating Model")
model = HourglassNet3D(hourGlassToStack, numModules, numFeats,numRegModules).cuda()
print("Model Created")

this is the main method that created the model.
It then calls this methods.

class HourglassNet3D(nn.Module):

  def __init__(self, nStack, nModules, nFeats, nRegModules):
    super(HourglassNet3D, self).__init__()
    self.nStack = nStack
    self.nModules = nModules
    self.nFeats = nFeats
    self.nRegModules = nRegModules
    self.conv1_ = nn.Conv2d(3, 64, bias = True, kernel_size = 7, stride = 2, padding = 3)
   self.bn1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace = True)
    self.r1 = Residual(64, 128)
    self.maxpool = nn.MaxPool2d(kernel_size = 2, stride = 2)
    self.r4 = Residual(128, 128)
    self.r5 = Residual(128, self.nFeats)

    _hourglass, _Residual, _lin_, _tmpOut, _ll_, _tmpOut_, _reg_ = [], [], [], [], [], [], []
    for i in range(self.nStack):
      _hourglass.append(Hourglass(4, self.nModules, self.nFeats))
      for j in range(self.nModules):
        _Residual.append(Residual(self.nFeats, self.nFeats))
      lin = nn.Sequential(nn.Conv2d(self.nFeats, self.nFeats, bias = True, kernel_size = 1, stride = 1),
                      nn.BatchNorm2d(self.nFeats), self.relu)
      _lin_.append(lin)
      _tmpOut.append(nn.Conv2d(self.nFeats, 16, bias = True, kernel_size = 1, stride = 1))
  _ll_.append(nn.Conv2d(self.nFeats, self.nFeats, bias = True, kernel_size = 1, stride = 1))
  _tmpOut_.append(nn.Conv2d(16, self.nFeats, bias = True, kernel_size = 1, stride = 1))

for i in range(4):
  for j in range(self.nRegModules):
    _reg_.append(Residual(self.nFeats, self.nFeats))

self.hourglass = nn.ModuleList(_hourglass)
self.Residual = nn.ModuleList(_Residual)
self.lin_ = nn.ModuleList(_lin_)
self.tmpOut = nn.ModuleList(_tmpOut)
self.ll_ = nn.ModuleList(_ll_)
self.tmpOut_ = nn.ModuleList(_tmpOut_)
self.reg_ = nn.ModuleList(_reg_)

self.reg = nn.Linear(4 * 4 * self.nFeats,16 )

And this then call this

class Residual(nn.Module):
#set the number ofinput and output for each layer
def __init__(self, numIn, numOut):
   super(Residual, self).__init__()
   self.numIn = numIn
   self.numOut = numOut
   self.bn = nn.BatchNorm2d(self.numIn)
   self.relu = nn.ReLU(inplace = True)
   self.conv1 = nn.Conv2d(self.numIn, self.numOut / 2, bias = True, kernel_size = 1)
   self.bn1 = nn.BatchNorm2d(self.numOut / 2)
   self.conv2 = nn.Conv2d(self.numOut / 2, self.numOut / 2, bias = True, kernel_size = 3, stride = 1, padding = 1)
   self.bn2 = nn.BatchNorm2d(self.numOut / 2)
   self.conv3 = nn.Conv2d(self.numOut / 2, self.numOut, bias = True, kernel_size = 1)

   if self.numIn != self.numOut:
       self.conv4 = nn.Conv2d(self.numIn, self.numOut, bias = True, kernel_size = 1)

all of this looks fine to me, but I dont know how I am suppose to pass this then if I am doing it wrong.
Thank you for any help

Output:
You might have to look out as to what you are passing to your convolutional layers in the Residual class. Per default, Python 3 will convert any division operation into a float variable.

Try casting your variables back to an integer, and see if that helps. Fixed code for Residual:

class Residual(nn.Module):
#set the number ofinput and output for each layer

    def __init__(self, numIn, numOut):
       super(Residual, self).__init__()
       self.numIn = numIn
       self.numOut = numOut
       self.bn = nn.BatchNorm2d(self.numIn)
       self.relu = nn.ReLU(inplace = True)
       self.conv1 = nn.Conv2d(self.numIn, int(self.numOut / 2), bias = True, kernel_size = 1)
       self.bn1 = nn.BatchNorm2d(int(self.numOut / 2))
       self.conv2 = nn.Conv2d(int(self.numOut / 2), int(self.numOut / 2), bias = True, kernel_size = 3, stride = 1, padding = 1)
       self.bn2 = nn.BatchNorm2d(int(self.numOut / 2))
       self.conv3 = nn.Conv2d(int(self.numOut / 2), self.numOut, bias = True, kernel_size = 1)

       if self.numIn != self.numOut:
           self.conv4 = nn.Conv2d(self.numIn, self.numOut, bias = True, kernel_size = 1)


Instruction: Understanding Bilinear Layers

Input:
When having a bilinear layer in PyTorch I can't wrap my head around how the calculation is done.

Here is a small example where I tried to figure out how it works:

In:

import torch.nn as nn
B = nn.Bilinear(2, 2, 1)
print(B.weight)

Out:

Parameter containing:
tensor([[[-0.4394, -0.4920],
         [ 0.6137,  0.4174]]], requires_grad=True)

I am putting through a zero-vector and a one-vector.

In:

print(B(torch.ones(2), torch.zeros(2)))
print(B(torch.zeros(2), torch.ones(2)))

Out:

tensor([0.2175], grad_fn=<ThAddBackward>)
tensor([0.2175], grad_fn=<ThAddBackward>)

I tried adding up the weights in various ways but I'm not getting the same result.

Thanks in advance!

Output:
The operation done by nn.Bilinear is B(x1, x2) = x1*A*x2 + b (c.f. doc) with:

A stored in nn.Bilinear.weight
b stored in nn.Bilinear.bias

If you take into account the (optional) bias, you should obtain the expected results.

import torch
import torch.nn as nn

def manual_bilinear(x1, x2, A, b):
    return torch.mm(x1, torch.mm(A, x2)) + b

x_ones = torch.ones(2)
x_zeros = torch.zeros(2)

# ---------------------------
# With Bias:

B = nn.Bilinear(2, 2, 1)
A = B.weight
print(B.bias)
# > tensor([-0.6748], requires_grad=True)
b = B.bias

print(B(x_ones, x_zeros))
# > tensor([-0.6748], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_zeros.view(2, 1), A.squeeze(), b))
# > tensor([[-0.6748]], grad_fn=<ThAddBackward>)

print(B(x_ones, x_ones))
# > tensor([-1.7684], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_ones.view(2, 1), A.squeeze(), b))
# > tensor([[-1.7684]], grad_fn=<ThAddBackward>)

# ---------------------------
# Without Bias:

B = nn.Bilinear(2, 2, 1, bias=False)
A = B.weight
print(B.bias)
# None
b = torch.zeros(1)

print(B(x_ones, x_zeros))
# > tensor([0.], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_zeros.view(2, 1), A.squeeze(), b))
# > tensor([0.], grad_fn=<ThAddBackward>)

print(B(x_ones, x_ones))
# > tensor([-0.7897], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_ones.view(2, 1), A.squeeze(), b))
# > tensor([[-0.7897]], grad_fn=<ThAddBackward>)


Instruction: Torchtext TabularDataset: data.Field doesn't contain actual imported data?

Input:
I learned from the Torchtext documentation that the way to import csv files is through TabularDataset. I did it like this:

train = data.TabularDataset(path='./data.csv',
                        format='csv',
                        fields=[("label",data.Field(use_vocab=True,include_lengths=False)),
                                ("statement",data.Field(use_vocab=True,include_lengths=True))],
                        skip_header=True)

"label" and "statement" are the header names of the 2 columns in my csv file. I defined them as data.Field, but "label" and "statement" don't seem to actually contain the data from my csv file, despite being recognized as data field objects by the console with no problem. I found out this issue when I tried to build a vocab list with statement.build_vocab(train, max_size=25000). I printed len(statement.vocab), the return is "2", which obviously doesn't reflect the actual data in the csv file. Did I do something wrong when importing the csv data or is my vocab building done wrong? Is there a separate method to put the data in the field objects? Thanks!!

Output:
The fields must be defined separately like this

    TEXT = data.Field(sequential=True,tokenize=tokenize, lower=True, include_lengths=True)
    LABEL = data.Field(sequential=True,tokenize=tokenize, lower=True)
    train = data.TabularDataset(path='./data.csv',
                        format='csv',
                        fields=[("label",LABEL),
                                ("statement",TEXT)],
                        skip_header=True)
    test = data.TabularDataset(path='./test.csv',
                        format='csv',
                        fields=[("label",LABEL),
                                ("statement",TEXT)],
                        skip_header=True)


Instruction: How to apply layer-wise learning rate in Pytorch?

Input:
I know that it is possible to freeze single layers in a network for example to train only the last layers of a pre-trained model. What I’m looking for is a way to apply certain learning rates to different layers.

So for example a very low learning rate of 0.000001 for the first layer and then increasing the learning rate gradually for each of the following layers. So that the last layer then ends up with a learning rate of 0.01 or so.

Is this possible in pytorch? Any idea how I can archive this?

Output:
Here is the solution:

from torch.optim import Adam

model = Net()

optim = Adam(
    [
        {"params": model.fc.parameters(), "lr": 1e-3},
        {"params": model.agroupoflayer.parameters()},
        {"params": model.lastlayer.parameters(), "lr": 4e-2},
    ],
    lr=5e-4,
)

Other parameters that are didn't specify in optimizer will not optimize. So you should state all layers or groups(OR the layers you want to optimize). and if you didn't specify the learning rate it will take the global learning rate(5e-4).
The trick is when you create the model you should give names to the layers or you can group it.


Instruction: What exactly is the definition of a 'Module' in PyTorch?

Input:
Please excuse the novice question, but is Module just the same as saying model?

That's what it sounds like, when the documentation says:

  Whenever you want a model more complex than a simple sequence of existing Modules you will need to define your model (as a custom Module subclass).

Or... when they mention Module, are they referring to something more formal and computer-sciency, like a protocol / interface type thing?

Output:
It's a simple container.

From the docs of nn.Module

  Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes. Submodules assigned in this way will be registered, and will have their parameters converted too when you call .cuda(), etc.

From the tutorial:

  All network components should inherit from nn.Module and override the forward() method. That is about it, as far as the boilerplate is concerned. Inheriting from nn.Module provides functionality to your component. For example, it makes it keep track of its trainable parameters, you can swap it between CPU and GPU with the .to(device) method, where device can be a CPU device torch.device("cpu") or CUDA device torch.device("cuda:0").

A module is a container from which layers, model subparts (e.g. BasicBlock in resnet in torchvision) and models should inherit. Why should they? Because the inheritance from nn.Module allows you to call methods like to("cuda:0"), .eval(), .parameters() or register hooks easily.

why not just call the 'module' a model, and call the layers 'layers'? I suppose maybe it's just semantics and splitting hairs, but still...

That's an API design choice and I find having only a Module class instead of two separate Model and Layers to be cleaner and to allow more freedom (it's easier to send just a part of the model to GPU, to get parameters only for some layers...).


Instruction: TypeError: tensor is not a torch image

Input:
While working through the AI course at Udacity I came across this error during the Transfer Learning section. Here is the code that seems to be causing the trouble:

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models

data_dir = 'filename'

# TODO: Define transforms for the training data and testing data
train_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])
test_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)

Output:
The problem is with the order of the transforms. The ToTensor transform should come before the Normalize transform, since the latter expects a tensor, but the Resize transform returns an image. Correct code with the faulty lines changed:

train_transforms = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
test_transforms = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])


Instruction: Is GEMM or BLAS used in Tensorflow, Theano, Pytorch

Input:
I know that Caffe uses GEneral Matrix to Matrix Multiplication (GEMM) which is part of Basic Linear Algebra Subprograms (BLAS) library for performing convolution operations. Where a convolution is converted to matrix multiplication operation. I have referred below article. https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

I want to understand how other deep learning frameworks like Theano, Tensorflow, Pytorch perform convolution operations. Do they use similar libraries in the backend. There might be some articles present on this topic. If someone can point me to those or can explain with an answer.

PS: I posted the same question on datascience.stackexchange.com. As I didn't get a reply there, I am posting it here as well. If there is a better forum to post this question please let me know.

Output:
tensorflow has multiple alternatives for the operations.

for GPU, cuda support is used. Most of the operations are implemented with cuDNN, some use cuBLAS, and others use cuda.

You can also use openCL instead of cuda, but you should compile tensorflow by yourself.

for CPU, intel mkl is used as the blas library.

I'm not familiar with pytorch and theano, but some commonly used blas libraries are listed below:

cuDNN, cuBLAS, and cuda: nvidia GPU support, most popular library
openCL: common GPU support, I don't know about it at all.
MKL: CPU blas library provided by intel
openBLAS: CPU library


Instruction: Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

Input:
I am using pytorch for training models. But I got an runtime error when it was computing the cross-entropy loss.

Traceback (most recent call last):
  File "deparser.py", line 402, in <module>
    d.train()
  File "deparser.py", line 331, in train
    total, correct, avgloss = self.train_util()
  File "deparser.py", line 362, in train_util
    loss = self.step(X_train, Y_train, correct, total)
  File "deparser.py", line 214, in step
    loss = nn.CrossEntropyLoss()(out.long(), y)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 862, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 975, in log_softmax
    return input.log_softmax(dim)
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

I think this is because the .cuda() function or conversion between torch.Float and torch.Long. But I have tried many ways to change the variable by .cpu()/.cuda() and .long()/.float(), but it still not work. This error message can't be found when searching it on google. Can anyone helps me? Thanks!!!

This is the code cause error:

def step(self, x, y, correct, total):
    self.optimizer.zero_grad()
    out = self.forward(*x)
    loss = nn.CrossEntropyLoss()(out.long(), y)
    loss.backward()
    self.optimizer.step()
    _, predicted = torch.max(out.data, 1)
    total += y.size(0)
    correct += int((predicted == y).sum().data)
    return loss.data

And this function step() is called by:

def train_util(self):
    total = 0
    correct = 0
    avgloss = 0
    for i in range(self.step_num_per_epoch):
        X_train, Y_train = self.trainloader()
        self.optimizer.zero_grad()
        if torch.cuda.is_available():
            self.cuda()
            for i in range(len(X_train)):
                X_train[i] = Variable(torch.from_numpy(X_train[i]))
                X_train[i].requires_grad = False
                X_train[i] = X_train[i].cuda()
            Y_train = torch.from_numpy(Y_train)
            Y_train.requires_grad = False
            Y_train = Y_train.cuda()
        loss = self.step(X_train, Y_train, correct, total)
        avgloss+=float(loss)*Y_train.size(0)
        self.optimizer.step()
        if i%100==99:
            print('STEP %d, Loss: %.4f, Acc: %.4f'%(i+1,loss,correct/total))

    return total, correct, avgloss/self.data_len

The input data X_train, Y_train = self.trainloader() are numpy arrays at begining.

This is a data sample:

>>> X_train, Y_train = d.trainloader()
>>> X_train[0].dtype
dtype('int64')
>>> X_train[1].dtype
dtype('int64')
>>> X_train[2].dtype
dtype('int64')
>>> Y_train.dtype
dtype('float32')
>>> X_train[0]
array([[   0,    6,    0, ...,    0,    0,    0],
       [   0, 1944, 8168, ...,    0,    0,    0],
       [   0,  815,  317, ...,    0,    0,    0],
       ...,
       [   0,    0,    0, ...,    0,    0,    0],
       [   0,   23,    6, ...,    0,    0,    0],
       [   0,    0,  297, ...,    0,    0,    0]])
>>> X_train[1]
array([ 6,  7,  8, 21,  2, 34,  3,  4, 19, 14, 15,  2, 13,  3, 11, 22,  4,
   13, 34, 10, 13,  3, 48, 18, 16, 19, 16, 17, 48,  3,  3, 13])
>>> X_train[2]
array([ 4,  5,  8, 36,  2, 33,  5,  3, 17, 16, 11,  0,  9,  3, 10, 20,  1,
   14, 33, 25, 19,  1, 46, 17, 14, 24, 15, 15, 51,  2,  1, 14])
>>> Y_train
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       ...,
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
      dtype=float32)

Try all possible combinations:

case 1:
loss = nn.CrossEntropyLoss()(out, y)
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 2:
loss = nn.CrossEntropyLoss()(out.long(), y)
as description above

case 3:
loss = nn.CrossEntropyLoss()(out.float(), y)
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 4:
loss = nn.CrossEntropyLoss()(out, y.long())
I get:
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

case 5:
loss = nn.CrossEntropyLoss()(out.long(), y.long())
I get:
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

case 6:
loss = nn.CrossEntropyLoss()(out.float(), y.long())
I get:
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

case 7:
loss = nn.CrossEntropyLoss()(out, y.float())
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 8:
loss = nn.CrossEntropyLoss()(out.long(), y.float())
I get:
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

case 9:
loss = nn.CrossEntropyLoss()(out.float(), y.float())
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

Output:
I know where the problem is.

y should be in torch.int64 dtype without one-hot encoding.
And CrossEntropyLoss() will auto encoding it with one-hot (while out is the probability distribution of prediction like one-hot format).

It can run now!


Instruction: Why the loss function can be apply on different size tensors

Input:
For example, I have a net that take tensor [N, 7](N is the samples num) as input and tensor [N, 4] as output, the “4” represents the different classes’ probabilities.

And the training data’s labels are the form of tensor [N], from range 0 to 3(represent the ground-truth class).

Here’s my question, I’ve seen some demos, they directly apply the loss function on the output tensor and label tensor. I wonder why this can work, since they have different size, and there sizes seems don’t fit the “broadcasting semantics”.

Here’s the minimal demo.

import torch
import torch.nn as nn
import torch.optim as optim

if __name__ == '__main__':
    features = torch.randn(2, 7)
    gt = torch.tensor([1, 1])
    model = nn.Sequential(
        nn.Linear(7, 4),
        nn.ReLU(),
        nn.Linear(4, 4)
    )
    optimizer = optim.SGD(model.parameters(), lr=0.005)
    f = nn.CrossEntropyLoss()

    for epoch in range(1000):
        optimizer.zero_grad()
        output = model(features)
        loss = f(output, gt)
        loss.backward()
        optimizer.step()

Output:
In PyTorch the implementation is:

Link to the Documentation: https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss

So implementing this formula in pytorch you get:
import torch
import torch.nn.functional as F

output = torch.tensor([ 0.1998, -0.2261, -0.0388,  0.1457])
target = torch.LongTensor([1])

# implementing the formula above
print('manual  cross-entropy:', (-output[target] + torch.log(torch.sum(torch.exp(output))))[0])

# calling build in cross entropy function to check the result
print('pytorch cross-entropy:', F.cross_entropy(output.unsqueeze(0), target))

Output:
manual  cross-entropy: tensor(1.6462)
pytorch cross-entropy: tensor(1.6462)

I hope this helps!


Instruction: How to print a tensor without showing gradients

Input:
If I do something like this:

tmp = torch.ones(3, 2, 2, requires_grad=True)
out = tmp ** 2
print("\n{}".format(out))

I get as an output:

tensor([[[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]]], grad_fn=<PowBackward0>)

I would like to print out just the values, not the grad_fn part.

However, doing

print("\n{}".format(out[0]))

results in:

tensor([[1., 1.],
        [1., 1.]], grad_fn=<SelectBackward>)

The only way I know is to out.detach() or there is another/better way? Just to clarify, I am happy the gradient is calculated. I just want to show the vector values without additional data.

Output:
Using the data should do the job for you:

tmp = torch.ones(3, 2, 2, requires_grad=True)
out = tmp ** 2
print("\n{}".format(out.data))

Output:

tensor([[[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]]])


Instruction: PyTorch LSTM States

Input:
Consider the following code snipped:

lstm = nn.LSTM(10, 5, batch_first=True)
states = (torch.rand(1, 1, 5), torch.rand(1, 1, 5))
h, states = lstm(torch.rand(1, 1, 10), states)
print('h:')
print(h)
print('states[0]:')
print(states[0])

Output:

h:
tensor([[[0.2808, 0.3357, 0.1290, 0.1413, 0.2648]]], grad_fn=<TransposeBackward0>)
states[0]:
tensor([[[0.2808, 0.3357, 0.1290, 0.1413, 0.2648]]], grad_fn=<ViewBackward>)

Because I have to hand over states as parameter for the forward() anyway I'd prefer using states[0] over h.

I've just noticed that the grad_fn is different, therefore I'm wondering if it makes any difference for backpropagation if using h or states for further computation of the outputs.

I can hardly imagine that there is a difference, so I'd probably just continue with states[0] but I also would like to understand why it is different.

Thanks in advance!

Output:
It's best practice and more intuitive to use h or (often called output) since states are meant to be passed into the lstm for internal use (think of tensorflow's dynamic_rnn to see why this would be the case.

That said you are correct that it actually doesn't make a difference. I'm not sure why the grad_fns are different, but empirically they function the same:

import torch
from torch import nn

lstm = nn.LSTM(10, 5, batch_first=True)
state = (torch.rand(1, 1, 5), torch.rand(1, 1, 5))
inp = torch.rand(1, 1, 10)
h, states = lstm(inp, state)

param = next(lstm.parameters())

l1 = h.sum()
l1.backward(retain_graph=True)
g1 = param.grad.clone()

param.grad.zero_()

l2 = states[0].sum()
l2.backward(retain_graph=True)
g2 = param.grad.clone()

print((g1 == g2).all())  # 1


Instruction: pytorch .stack final shape after .squeeze

Input:
I had a pandas dataframe 200 columns by 2500 rows which I made it into a tensor

tensor = torch.tensor(df.values)
tensor.size() => ([2500,200])

which i chunked and enumerated

list=[]
for i,chunk in enumerate(tensor.chunk(100,dim=0))
    chunk.size =>([25,200])
    output = hiddenlayer(chunks)
    output.size() => ([25,1])
    list += output

chunks were fed through some layers and outputted as 1 feature tensors. So now I have a list of 100 tensors, each with 25 blocks of 1, 100x25x1

so i

stacked = torch.stack(list, 1).squeeze(2)
stacked.size()=([25,100])

I've played around with the stacking and squeezing but i can't seem to get back to ([2500,1]) which is what I want. Am I missing something? If you could quickly help me understand what stacking and squeezing is doing and why it's not working for me I'd be forever in your debt! Thanks

Output:
Renaming list to tensor_list since it's bad practice to use reserved keywords as variable names.

tensor_list =[]
for i,chunk in enumerate(tensor.chunk(100,dim=0)):
    output = hiddenlayer(chunk).squeeze()
    tensor_list.append(output)

result = torch.reshape(torch.stack(tensor_list,0), (-1, 1))

result.size() should now return torch.Size([2500, 1])


Instruction: Parsing CSV into Pytorch tensors

Input:
I have a CSV files with all numeric values except the header row. When trying to build tensors, I get the following exception:

Traceback (most recent call last):
  File "pytorch.py", line 14, in <module>
    test_tensor = torch.tensor(test)
ValueError: could not determine the shape of object type 'DataFrame'

This is my code:

import torch
import dask.dataframe as dd

device = torch.device("cuda:0")

print("Loading CSV...")
test = dd.read_csv("test.csv", encoding = "UTF-8")
train = dd.read_csv("train.csv", encoding = "UTF-8")

print("Converting to Tensor...")
test_tensor = torch.tensor(test)
train_tensor = torch.tensor(train)

Using pandas instead of Dask for CSV parsing produced the same error. I also tried to specify dtype=torch.float64 inside the call to torch.tensor(data), but got the same error again.

Output:
Try converting it to an array first:

test_tensor = torch.Tensor(test.values)