Instruction: How to deal with mini-batch loss in Pytorch?

Input:
I feed mini-batch data to model, and I just want to know how to deal with the loss. Could I accumulate the loss, then call the backward like:

    ...
    def neg_log_likelihood(self, sentences, tags, length):
        self.batch_size = sentences.size(0)

        logits = self.__get_lstm_features(sentences, length)
        real_path_score = torch.zeros(1)
        total_score = torch.zeros(1)
        if USE_GPU:
            real_path_score = real_path_score.cuda()
            total_score = total_score.cuda()

        for logit, tag, leng in zip(logits, tags, length):
            logit = logit[:leng]
            tag = tag[:leng]
            real_path_score += self.real_path_score(logit, tag)
            total_score += self.total_score(logit, tag)
        return total_score - real_path_score
    ...
loss = model.neg_log_likelihood(sentences, tags, length)
loss.backward()
optimizer.step()

I wonder that if the accumulation could lead to gradient explosion?

So, should I call the backward in loop:

for sentence, tag , leng in zip(sentences, tags, length):
    loss = model.neg_log_likelihood(sentence, tag, leng)
    loss.backward()
    optimizer.step()

Or, use the mean loss just like the reduce_mean in tensorflow

loss = reduce_mean(losses)
loss.backward()

Output: The loss has to be reduced by mean using the mini-batch size. If you look at the native PyTorch loss functions such as CrossEntropyLoss, there is a separate parameter reduction just for this and the default behaviour is to do mean on the mini-batch size.


Instruction: Torch allocates zero GPU memory on PyTorch

Input:
I am trying to use GPU to train my model but it seems that torch fails to allocate GPU memory.

My model is a RNN built on PyTorch

device = torch.device('cuda: 0' if torch.cuda.is_available() else "cpu")

rnn = RNN(n_letters, n_hidden, n_categories_train)
rnn.to(device)
criterion = nn.NLLLoss()
criterion.to(device)
optimizer = torch.optim.SGD(rnn.parameters(), lr=learning_rate, weight_decay=.9)

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.cuda()
        hidden = hidden.cuda()

        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)

        output = output.cuda()
        hidden = hidden.cuda()

        return output, hidden

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size).cuda())

Training function:

def train(category_tensor, line_tensor, rnn, optimizer, criterion):
    rnn.zero_grad()
    hidden = rnn.init_hidden()

    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)

    loss = criterion(output, category_tensor)
    loss.backward()

    optimizer.step()

    return output, loss.item()

The function to get category_tensor and line_tensor:

def random_training_pair(category_lines, n_letters, all_letters):
    category = random.choice(all_categories_train)
    line = random.choice(category_lines[category])
    category_tensor = Variable(torch.LongTensor([all_categories_train.index(category)]).cuda())
    line_tensor = Variable(process_data.line_to_tensor(line, n_letters, all_letters)).cuda()

    return category, line, category_tensor, line_tensor

I ran the following the code:

 print(torch.cuda.get_device_name(0))
 print('Memory Usage:')
 print('Allocated:', round(torch.cuda.memory_allocated(0) / 1024 ** 3, 1), 'GB')
 print('Cached:   ', round(torch.cuda.memory_cached(0) / 1024 ** 3, 1), 'GB')

and I got:

GeForce GTX 1060
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB

I did not get any errors but GPU usage is just 1% while CPU usage is around 31%.

I am using Windows 10 and Anaconda, where my PyTorch is installed. CUDA and cuDNN is installed from .exe file downloaded from Nvidia website.

Output:
Your problem is that to() is not an in-place operation. If you call rnn.to(device) it will return a new object / model located on the desired device. But it will not move the old object anywhere!

So changing:

rnn = RNN(n_letters, n_hidden, n_categories_train)
rnn.to(device)

to:

rnn = RNN(n_letters, n_hidden, n_categories_train).to(device)

For all other instances you used to this way, you have to change it as well.

Should do the trick for you!

Note: All tensors and parameters you perform operations with have to be on the same device. If your model is on GPU but your input tensor is on CPU you will get an error message.


Instruction: In Colaboratory, CUDA cannot be used for the torch

Input:
The error message is as follows:
RuntimeError   Traceback (most recent call last)
<ipython-input-24-06e96beb03a5> in <module>()
     11
     12 x_test = np.array(test_features)
---> 13 x_test_cuda = torch.tensor(x_test, dtype=torch.float).cuda()
     14 test = torch.utils.data.TensorDataset(x_test_cuda)
     15 test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)

/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py in _lazy_init()
    160 class CudaError(RuntimeError):
    161     def __init__(self, code):
--> 162         msg = cudart().cudaGetErrorString(code).decode('utf-8')
    163         super(CudaError, self).__init__('{0} ({1})'.format(msg, code))
    164

RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:51

Output:
Click on Runtime and select Change runtime type.
Now in Hardware Acceleration, select GPU and hit Save.


Instruction: How to train a neural network model with bert embeddings instead of static embeddings like glove/fasttext?

Input:
I am looking for some heads up to train a conventional neural network model with bert embeddings that are generated dynamically (BERT contextualized embeddings which generates different embeddings for the same word which when comes under different context).

In normal neural network model, we would initialize the model with glove or fasttext embeddings like,

import torch.nn as nn

embed = nn.Embedding(vocab_size, vector_size)

embed.weight.data.copy_(some_variable_containing_vectors)

Instead of copying static vectors like this and use it for training, I want to pass every input to a BERT model and generate embedding for the words on the fly, and feed them to the model for training.

So should I work on changing the forward function in the model for incorporating those embeddings?

Any help would be appreciated!

Output:
If you are using Pytorch. You can use https://github.com/huggingface/pytorch-pretrained-BERT which is the most popular BERT implementation for Pytorch (it is also a pip package!). Here I'm just going to outline how to use it properly.

For this particular problem there are 2 approaches - where you obviously cannot use the Embedding layer:

You can incorporate generating BERT embeddings into your data preprocessing pipeline. You will need to use BERT's own tokenizer and word-to-ids dictionary. The repo's README has examples on preprocessing.

You can write a loop for generating BERT tokens for strings like this (assuming - because BERT consumes a lot of GPU memory):

(Note: to be more proper you should also add attention masks - which are LongTensor of 1 & 0 masking the sentence lengths)

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel

batch_size = 32
X_train, y_train = samples_from_file('train.csv') # Put your own data loading function here
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
X_train = [tokenizer.tokenize('[CLS] ' + sent + ' [SEP]') for sent in X_train] # Appending [CLS] and [SEP] tokens - this probably can be done in a cleaner way
bert_model = BertModel.from_pretrained('bert-base-uncased')
bert_model = bert_model.cuda()

X_train_tokens = [tokenizer.convert_tokens_to_ids(sent) for sent in X_train]
results = torch.zeros((len(X_test_tokens), bert_model.config.hidden_size)).long()
with torch.no_grad():
    for stidx in range(0, len(X_test_tokens), batch_size):
        X = X_test_tokens[stidx:stidx + batch_size]
        X = torch.LongTensor(X).cuda()
        _, pooled_output = bert_model(X)
        results[stidx:stidx + batch_size,:] = pooled_output.cpu()

After which you obtain the results tensor which contains the calculated embeddings, where you can use it as an input to your model.

The full (and more proper) code for this is provided here

This method has the advantage of not having to re-calculate these embeddings every epoch.

With this method, e.g for classification your model should only consist of a Linear(bert_model.config.hidden_size, num_labels) layer, inputs to the model should be the results tensor in the above code

Second, and arguably cleaner method: If you check out the repo, you can find there is wrappers for various tasks (e.g BertForSequenceClassification). It should also be easy to implement your custom classes that inherits from BertPretrainedModel and utilizes the various Bert classes from the repo.

For example, you can use:

model = BertForSequenceClassification.from_pretrained('bert-base-uncased', labels=num_labels) # Where num_labels is the number of labels you need to classify.

After which you can continue with the preprocessing, up until generating token ids. Then you can train the entire model (but with a low learning rate e.g Adam 3e-5 for batch_size = 32)

With this you can fine-tune BERT's embeddings itself, or use techniques like freezing BERT for a few epochs to train the classifier only, then unfreeze to fine-tune etc. But it is also more computationally expensive.

An example for this is also provided in the repo


Instruction: Trying to understand Pytorch's implementation of LSTM

Input:
I have a dataset containing 1000 examples where each example has 5 features (a,b,c,d,e). I want to feed 7 examples to an LSTM so it predicts the feature (a) of the 8th day.

Reading Pytorchs documentation of nn.LSTM() I came up with the following:

input_size = 5
hidden_size = 10
num_layers = 1
output_size = 1

lstm = nn.LSTM(input_size, hidden_size, num_layers)
fc = nn.Linear(hidden_size, output_size)

out, hidden = lstm(X)  # Where X's shape is ([7,1,5])
output = fc(out[-1])

output  # output's shape is ([7,1])

According to the docs:

The input of the nn.LSTM is "input of shape (seq_len, batch, input_size)" with "input_size – The number of expected features in the input x",

And the output is: "output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t."

In this case, I thought seq_len would be the sequence of 7 examples, batchis 1 and input_size is 5. So the lstm would consume each example containing 5 features refeeding the hidden layer every iteration.

What am I missing?

Output:
When I extend your code to a full example -- I also added some comments to may help -- I get the following:

import torch
import torch.nn as nn

input_size = 5
hidden_size = 10
num_layers = 1
output_size = 1

lstm = nn.LSTM(input_size, hidden_size, num_layers)
fc = nn.Linear(hidden_size, output_size)

X = [
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
    [[1,2,3,4,5]],
]

X = torch.tensor(X, dtype=torch.float32)

print(X.shape)         # (seq_len, batch_size, input_size) = (7, 1, 5)
out, hidden = lstm(X)  # Where X's shape is ([7,1,5])
print(out.shape)       # (seq_len, batch_size, hidden_size) = (7, 1, 10)
out = out[-1]          # Get output of last step
print(out.shape)       # (batch, hidden_size) = (1, 10)
out = fc(out)          # Push through linear layer
print(out.shape)       # (batch_size, output_size) = (1, 1)

This makes sense to me, given your batch_size = 1 and output_size = 1 (I assume, you're doing regression). I don't know where your output.shape = (7, 1) come from.

Are you sure that your X has the correct dimensions? Did you create nn.LSTM maybe with batch_first=True? There are lot of little things that can sneak in.


Instruction: PyTorch datasets: ImageFolder and subfolder filtering

Input:
I would like to use ImageFolder to create an Image Dataset.

My current image directory structure looks like this:

/root
-- train/
---- 001.jpg
---- 002.jpg
---- ....
-- test/
---- 001.jpg
---- 002.jpg
---- ....

I would like to have a dataset dedicated to training data, and a dataset dedicated to test data.

As I understand, doing so:

dataset = ImageFolder(root='root/train')

does not find any images.

Doing

dataset = ImageFolder(root='root')

find images but train and test images are just scrambled together.

ImageFolder has argument loader but I did not manage to find any use-case for it.

How can I discriminate images in the root folder according to the subfolder they belong to?

Output:
ImageFolder expects the data folder (the one that you pass as root) to contain subfolders representing the classes to which its images belong. Something like this:
data/
├── train/
|   ├── class_0/
|   |   ├── 001.jpg
|   |   ├── 002.jpg
|   |   └── 003.jpg
|   └── class_1/
|       ├── 004.jpg
|       └── 005.jpg
└── test/
    ├── class_0/
    |   ├── 006.jpg
    |   └── 007.jpg
    └── class_1/
        ├── 008.jpg
        └── 009.jpg

Having the above folder structure you can do the following:
train_dataset = ImageFolder(root='data/train')
test_dataset  = ImageFolder(root='data/test')

Since you don't have that structure, one obvious option is to create class-subfolders and put the images into them. Another option is to create a custom Dataset, see here.


Instruction: ModuleNotFoundError: No module named 'torch._C'

Input:
I want to import the torch,but then the interpreter return this result,i have no idea how to deal with it

Traceback (most recent call last):
  File "D:/Programing/tool/Python/learn_ml_the_hard_way/ML/scipy1.py", line 1, in <module>
    import  torch
  File "D:\Programing\python\Anaconda3.5\lib\site-packages\torch\__init__.py", line 84, in <module>
    from torch._C import *
ModuleNotFoundError: No module named 'torch._C'

Output:
I had the same problem and followed the instructions in this link
You can also find the torch path with this command if needed:
sudo find / -iname torch


Instruction: Pytorch Not Updating Variables in .step()

Input:
I'm attempting to convert old code to PyTorch code as an experiment. Ultimately, I will be doing regression on a 10,000+ x 100 Matrix, updating weights and whatnot appropriately.

Trying to learn, I'm slowly scaling up on toy examples. I'm hitting a wall with the following sample code.

import torch
import torch.nn as nn
import torch.nn.functional as funct
from torch.autograd import Variable

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

x_data = Variable( torch.Tensor( [ [1.0, 2.0], [2.0, 3.0], [3.0, 4.0] ] ),
requires_grad=True )
y_data = Variable( torch.Tensor( [ [2.0], [4.0], [6.0] ] ) )

w = Variable( torch.randn( 2, 1, requires_grad=True ) )

b = Variable( torch.randn( 1, 1, requires_grad=True ) )

class Model(torch.nn.Module) :
    def __init__(self) :
        super( Model, self).__init__()
        self.linear = torch.nn.Linear(2,1) ## 2 features per entry. 1 output
    def forward(self, x2, w2, b2) :
        y_pred = x2 @ w2 + b2
        return y_pred

model = Model()

criterion = torch.nn.MSELoss( size_average=False )
optimizer = torch.optim.SGD( model.parameters(), lr=0.01 )

for epoch in range(10) :
    y_pred = model( x_data,w,b ) # Get prediction
    loss = criterion( y_pred, y_data ) # Calc loss
    print( epoch, loss.data.item() ) # Print loss
    optimizer.zero_grad() # Zero gradient
    loss.backward() # Calculate gradients
    optimizer.step() # Update w, b

However, doing so, my loss is always the same, and investigating shows my w and b never actually change.  I'm a bit lost at what's going on here.

Ultimately, I'd like to be able to store the results of the "new" w and b to compare across iterations and datasets.

Output:
It looks like a case of cargo programming to me.

Notice that your Model class doesn't make use of self in forward, so it is effectively a "regular" (non-method) function, and model is entirely stateless. The simplest fix to your code is to make optimizer aware of w and b, by creating it as optimizer = torch.optim.SGD([w, b], lr=0.01). I also rewrite model to be a function

import torch
import torch.nn as nn
# torch.autograd.Variable is roughly equivalent to requires_grad=True
# and is deprecated in PyTorch 1.0

# your code gives not reason to have `requires_grad=True` on `x_data`
x_data = torch.tensor( [ [1.0, 2.0], [2.0, 3.0], [3.0, 4.0] ])
y_data = torch.tensor( [ [2.0], [4.0], [6.0] ] )

w = torch.randn( 2, 1, requires_grad=True )
b = torch.randn( 1, 1, requires_grad=True )

def model(x2, w2, b2):
    return x2 @ w2 + b2

criterion = torch.nn.MSELoss( size_average=False )
optimizer = torch.optim.SGD([w, b], lr=0.01 )

for epoch in range(10) :
    y_pred = model( x_data,w,b )
    loss = criterion( y_pred, y_data )
    print( epoch, loss.data.item() )
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

That being said, nn.Linear is built to simplify this procedure. It automatically creates an equivalent of both w and b, called self.weight and self.bias, respectively. Also, self.__call__(x) is equivalent to the definition of forward of your Model, in that it returns self.weight @ x + self.bias. In other words, you can also use alternative code

import torch
import torch.nn as nn

x_data = torch.tensor( [ [1.0, 2.0], [2.0, 3.0], [3.0, 4.0] ] )
y_data = torch.tensor( [ [2.0], [4.0], [6.0] ] )

model = nn.Linear(2, 1)

criterion = torch.nn.MSELoss( size_average=False )
optimizer = torch.optim.SGD(model.parameters(), lr=0.01 )

for epoch in range(10) :
    y_pred = model(x_data)
    loss = criterion( y_pred, y_data )
    print( epoch, loss.data.item() )
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

where model.parameters() can be used to enumerate model parameters (equivalent to the manually created list [w, b] above). To access your parameters (load, save, print, whatever) use model.weight and model.bias.


Instruction: Querying an image with bilinear interpolation e.e. finding RGB value in the fractional coordinates using Pytorch

Input:
I have a input T1 of size (1,256,256,3) i.e. images/grid of batch size 1. I have another tensor T2 of size (1, N, 2) i.e. tensor consisting of coordinates i.e. [ [10.5 , 200.787], [150.568, 190.456], …]. How do I compute functional values (using bilinear interpolation) of coordinates in T2 from T1 data?

Thanks for any help

I have tested the same functionality using tensorflow with the function "tf.contrib.resampler.resampler"

Output:
try grid_sample:

torch.nn.functional.grid_sample(input, grid, mode='bilinear', padding_mode='zeros')

Given an input and a flow-field grid, computes the output using input values and pixel locations from grid.

For each output location output[n, :, h, w], the size-2 vector grid[n, h, w] specifies input pixel locations x and y, which are used to interpolate the output value output[n, :, h, w]. mode argument specifies nearest or bilinear interpolation method to sample the input pixels.

coordinate should be in range of [-1, 1]. This is because the pixel locations are normalized by the input spatial dimensions.

sampler git example
pytorch documentation


Instruction: Iterate over two Pytorch tensors at once?

Input:
I have two Pytorch tensors (really, just 1-D lists), t1 and t2. Is it possible to iterate over them in parallel, i.e. do something like

for a,b in zip(t1,t2)
?

Thanks.

Output:
For me (Python version 3.7.3 and PyTorch version 1.0.0) the zip function works as expected with PyTorch tensors:

>>> import torch
>>> t1 = torch.ones(3)
>>> t2 = torch.zeros(3)
>>> list(zip(t1, t2))
[(tensor(1.), tensor(0.)), (tensor(1.), tensor(0.)), (tensor(1.), tensor(0.))]

The list call is just needed to display the result. Iterating over zip works normally.


Instruction: Unpickling saved pytorch model throws AttributeError: Can't get attribute 'Net' on <module '__main__' despite adding class definition inline

Input:
I'm trying to serve a pytorch model in a flask app. This code was working when I ran this on a jupyter notebook earlier but now I'm running this within a virtual env and apparently it can't get attribute 'Net' even though the class definition is right there. All the other similar questions tell me to add the class definition of the saved model in the same script. But it still doesn't work. The torch version is 1.0.1 (where the saved model was trained as well as the virtualenv)
What am I doing wrong?
Here's my code.

import os
import numpy as np
from flask import Flask, request, jsonify
import requests

import torch
from torch import nn
from torch.nn import functional as F

MODEL_URL = 'https://storage.googleapis.com/judy-pytorch-model/classifier.pt'

r = requests.get(MODEL_URL)
file = open("model.pth", "wb")
file.write(r.content)
file.close()

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        x = self.fc3(x)

        return F.log_softmax(x, dim=-1)

model = torch.load('model.pth')

app = Flask(__name__)

@app.route("/")
def hello():
    return "Binary classification example\n"

@app.route('/predict', methods=['GET'])
def predict():

    x_data = request.args['x_data']

    x_data =  x_data.split()
    x_data = list(map(float, x_data))

    sample = np.array(x_data)

    sample_tensor = torch.from_numpy(sample).float()

    out = model(sample_tensor)

    _, predicted = torch.max(out.data, -1)

    if predicted.item() == 0:
         pred_class = "Has no liver damage - ", predicted.item()
    elif predicted.item() == 1:
        pred_class = "Has liver damage - ", predicted.item()

    return jsonify(pred_class)

Here's the full traceback:

Traceback (most recent call last):
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/bin/flask", line 10, in <module>
    sys.exit(main())
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 894, in main
    cli.main(args=args, prog_name=name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 557, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 767, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 293, in __init__
    self._load_unlocked()
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 317, in _load_unlocked
    self._app = rv = self.loader()
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 372, in load_app
    app = locate_app(self, import_name, name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 235, in locate_app
    __import__(module_name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/app.py", line 34, in <module>
    model = torch.load('model.pth')
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/torch/serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/torch/serialization.py", line 542, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'Net' on <module '__main__' from '/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/bin/flask'>

This doesn't solve my issue. I do not want to change the way I persist the model. torch.save() worked fine for me outside the virtual env. I don't mind adding the class definition to the script. I'm trying to see what's causing the error despite that.

Output:
(This is a partial answer)
I don't think torch.save(model,'model.pt') works from the command prompt, or when a model is saved from one script running as '__main__' and loaded from another.
The reason is that torch must be automatically loading the module that was used to save the file, and it gets the module name from __name__.
Now for the partial part: It's unclear how to fix this issue, especially when you have virtualenvs in the mix.
Thanks to Jatentaki for starting the conversation in this direction.


Instruction: How to make the convolution in pytorch associative?

Input:
The discrete convolution is by definition associative. But when I try to verify this in pytorch, I can not find get a plausible result.

The associative law is $f*(g*\psi)=(f * g)*\psi$, so I create three discrete functions centered at zero(as tensors) and convolve them with proper zero paddings so that all non-zero element in result map is obtained.

import torch
import torch.nn as nn

def test_conv_compst():
    # $\psi$
    inputs = torch.randn((1,4,7,7))
    # $g$
    a = torch.randn((7, 4, 3, 3))
    # $f$
    b = torch.randn((3, 7, 3, 3))
    int_1 = torch.conv2d(inputs, a, padding=2)
    # results obtained by the first order
    res_1 = torch.conv2d(int_1, b, padding=2)

    comp_k = torch.conv2d(a.transpose(1, 0), b, padding=2).transpose(1, 0)
    print(comp_k.shape)
    # results obtained through the second order
    res_2 = torch.conv2d(inputs, comp_k, padding=4)
    print(res_1.shape)
    print(res_2.shape)
    print(torch.max(torch.abs(res_2-res_1)))

The expected result is that the difference from the two results is negligible. But it returns:

torch.Size([3, 4, 5, 5])
torch.Size([1, 3, 11, 11])
torch.Size([1, 3, 11, 11])
tensor(164.8044)

Output:
Long story short, this is because of batching. The first argument of torch.conv2d is interpreted as [batch, channel, height, width], the second as [out_channel, in_channel, height, width] and the output as [batch, channel, height, width]. So if you call conv2d(a, conv2d(b, c)), you treat b's leading dimension as batch and if you call conv2d(conv2d(a, b), c), you treat it as out_channels.

That being said, I get the impression that you're asking about math here, so let me expand. Your idea is correct in theory: convolutions are linear operators and should be associative. However, since we provide them with kernels rather than the actual matrices representing the linear operators, there is some "conversion" that needs to happen behind the scenes so that the kernels are interpreted as matrices properly. Classically, this can be done by constructing the corresponding circulant matrices (border conditions aside). If we denote the kernels with a, b, c and the circulant matrix creation operator with M, we get that M(a) @ [M(b) @ M(c)] = [M(a) @ M(b)] @ M(c), where @ denotes matrix-matrix multiplication.

Convolution implementations return an image (vector, kernel, however you call it) and not the associated circulant matrix, which is ridiculously redundant and wouldn't fit in the memory in most cases. Therefore we also need some circulant-to-vector operator V(matrix), which returns the first column of matrix and is therefore the inverse of M. In abstract mathematical terms, functions such as scipy.signal.convolve (actually correlate, since convolution requires an extra flip of one of the inputs, which I skip for clarity) are implemented as convolve = lambda a, b: V(M(a) @ M(b)) and thus

convolve(a, convolve(b, c)) =
                            = V(M(a) @ M(V[M(b) @ M(c)])
                            = V(M(a) @ M(b) @ M(c))
                            = V(M(V[M(a) @ M(b)]) @ M(c))
                            = convolve(convolve(a, b), c)

I hope I haven't lost you, this is just converting one into the other by making use of the fact that V is the inverse of M and of the associativeness of matrix multiplication to move the parentheses. Note that the middle line is basically the "raw" ABC. We can verify with the following code:

import numpy as np
import scipy.signal as sig

c2d = sig.convolve2d

a = np.random.randn(7, 7)
b = np.random.randn(3, 3)
c = np.random.randn(3, 3)

ab = c2d(a, b)
ab_c = c2d(ab, c)

bc = c2d(b, c)
a_bc = c2d(a, bc)

print((a_bc - ab_c).max())

The problem with PyTorch is that it interprets the first input as [batch, channel, height, width] and the second as [out_channels, in_channels, height, width]. This means that the "conversion" operator M is different for the first argument and the second argument. Let's call them M and N, respectively. Since there is only one output, there is only one V and it can be the inverse of either M or N, but not both (since they are different). If you rewrite the above equation taking care to distinguish between M and N you will see that, depending on your choice whether V inverts one or the other, you're unable to write the equality either between lines 2 and 3 or 3 and 4.

In practice, there also the additional issue of the channel dimension, which is not there in the classic definition of convolutions, however my first guess is that it could be dealt with with a single lifting operator M for both operands, unlike batching.


Instruction: How to change (assign) new value in FloatTensor, Pytorch?

Input:
I am changing/assigning the value on the array(torch.cuda.floatTensor). I tried some way but it does not work. Please help me!

#1

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0) #3185
for i in range (0,s,1):
    if (dis[i,0] < 0):
        dis[i,0]== 0
#There is no error but It does not work.

#2

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0)
a = torch.zeros(s, 1).cuda()
idx = (dis > a)
dis[idx] = a[idx]

AssertionError: can't compare Variable and tensor

#3

#dis is [torch.cuda.FloatTensor of size 3185x1 (GPU 0)]

s = dis.size(0)
a = torch.zeros(s, 1).cuda()
for i in range (0,s,1):
    if (dis[i,0] < a[i, 0]):
        dis[i,0]==a[i, 0]

#RuntimeError: bool value of Variable objects containing non-empty      torch.cuda.ByteTensor is ambiguous

Output:
IIUC, you need to replace values smaller than 0 with 0, Just use torch.clamp, which is meant for such use cases:

dis = dis.clamp(min=0)

Example:

import torch
dis = torch.tensor([[1], [-3], [0]])
#tensor([[ 1],
#        [-3],
#        [ 0]])

dis.clamp(min=0)
#tensor([[1],
#        [0],
#        [0]])


Instruction: Differences between F.relu(X) and torch.max(X, 0)

Input:
I am trying to implement the following loss function

To me, the most straight forword implementation would be using torch.max

losses = torch.max(ap_distances - an_distances + margin, torch.Tensor([0]))

However, I saw other implementations on github using F.relu

losses = F.relu(ap_distances - an_distances + margin)

They give essential the same output, but I wonder if there's any fundamental difference between the two methods.

Output:
torch.max is not differentiable according to this discussion.
A loss function needs to be continuous and differentiable to do backprop. relu is differentiable as it can be approximated and hence the use of it in a loss function.


Instruction: PyTorch will not fit straight line to two data points

Input:
I'm facing issues in fitting a simple y= 4x1 line with 2 data points using pytorch. While running the inference code, the model seems to output same value to any input which is strange. Pls find the code attached along with the data files used by me. Appreciate any help here.

import torch
import numpy as np
import pandas as pd

df = pd.read_csv('data.csv')
test_data = pd.read_csv('test_data.csv')

inputs = df[['x1']]
target = df['y']
inputs = torch.tensor(inputs.values).float()
target = torch.tensor(target.values).float()

test_data = torch.tensor(test_data.values).float()
#Defining Network Architecture
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

  def __init__(self):
    super(Net,self).__init__()

    hidden1 = 3
#     hidden2 = 5

    self.fc1 = nn.Linear(1,hidden1)
    self.fc3 = nn.Linear(hidden1,1)

  def forward(self,x):
    x = F.relu(self.fc1(x))
    x = self.fc3(x)
    return x

#instantiate the model

model = Net()
print(model)

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

model.train()

#epochs
epochs = 100

for x in range(epochs):
  #initialize the training loss to 0
  train_loss = 0
  #clear out gradients
  optimizer.zero_grad()

  #calculate the output
  output = model(inputs)

  #calculate loss
  loss = criterion(output,target)

  #backpropagate
  loss.backward()

  #update parameters
  optimizer.step()

  if ((x%5)==0):
    print('Training Loss after epoch {:2d} is {:2.6f}'.format(x,loss))

#set the model in evaluation mode
model.eval()

#Test the model on unseen data

test_output = model(test_data)

print(test_output)

Below is the model output

#model output
tensor([[56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579]], grad_fn=<AddmmBackward>)

Output:
Your model is collapsing. You can probably see that based on the prints. You may want to use a lower learning rate (1e-5, 1e-6, etc.). Switching from SGD(...)to Adam(...) may be easier if you do not have experience and want less trouble fine-tuning these hparams. Also, maybe 100 epochs is not enough. As you did not share an MCVE, I cannot tell you for sure what it is. Here is an MCVE of linefitting using the same Net you used:

import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F

epochs = 1000
max_range = 40
interval = 4

# DATA
x_train = torch.arange(0, max_range, interval).view(-1, 1).float()
x_train += torch.rand(x_train.size(0), 1) - 0.5  # small noise
y_train = (4 * x_train)
y_train += torch.rand(x_train.size(0), 1) - 0.5  # small noise

x_test  = torch.arange(interval // 2, max_range, interval).view(-1, 1).float()
y_test  = 4 * x_test

class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    hidden1 = 3
    self.fc1 = nn.Linear(1, hidden1)
    self.fc3 = nn.Linear(hidden1, 1)

  def forward(self, x):
    x = F.relu(self.fc1(x))
    x = self.fc3(x)
    return x

model = Net()
print(model)

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-5)

# TRAIN
model.train()
for epoch in range(epochs):
  optimizer.zero_grad()
  y_pred = model(x_train)
  loss = criterion(y_pred, y_train)
  loss.backward()
  optimizer.step()

  if epoch % 10 == 0:
    print('Training Loss after epoch {:2d} is {:2.6f}'.format(epoch, loss))

# TEST
model.eval()
y_pred = model(x_test)
print(torch.cat((x_test, y_pred, y_test), dim=-1))

This is what the data looks like:

And this is what the training looks like:

Training Loss after epoch  0 is 7416.805664
Training Loss after epoch 10 is 6645.655273
Training Loss after epoch 20 is 5792.936523
Training Loss after epoch 30 is 4700.106445
Training Loss after epoch 40 is 3245.384277
Training Loss after epoch 50 is 1779.370728
Training Loss after epoch 60 is 747.418579
Training Loss after epoch 70 is 246.781311
Training Loss after epoch 80 is 68.635155
Training Loss after epoch 90 is 17.332235
Training Loss after epoch 100 is 4.280161
Training Loss after epoch 110 is 1.170808
Training Loss after epoch 120 is 0.453974
...
Training Loss after epoch 970 is 0.232296
Training Loss after epoch 980 is 0.232090
Training Loss after epoch 990 is 0.231888

And this is what the output looks like:

|  x_test |  y_pred  |  y_test  |
|:-------:|:--------:|:--------:|
|  2.0000 |   8.6135 |   8.0000 |
|  6.0000 |  24.5276 |  24.0000 |
| 10.0000 |  40.4418 |  40.0000 |
| 14.0000 |  56.3303 |  56.0000 |
| 18.0000 |  72.1884 |  72.0000 |
| 22.0000 |  88.0465 |  88.0000 |
| 26.0000 | 103.9047 | 104.0000 |
| 30.0000 | 119.7628 | 120.0000 |
| 34.0000 | 135.6210 | 136.0000 |
| 38.0000 | 151.4791 | 152.0000 |


Instruction: How to do element wise multiplication for two 4D unequal size tensors in pytorch?

Input:
I have got a tensor A and Tensor B.

Size of A = [2,64,56,56]

Size of B = [2,64,29,29]

How can I perform torch.mul(A,B)? The tensors are of unequal size.

  RuntimeError: shape [2, 64, 56, 56] is invalid for input of size 107648

Output:
You can check out the documentation here: https://pytorch.org/docs/stable/torch.html#torch.mul

There, you can read:

  The shapes of input and other must be broadcastable.

You can read about broadcastability here: https://pytorch.org/docs/stable/notes/broadcasting.html#broadcasting-semantics

Lastly, it probably makes sense to think about what you want to achieve. I'm not sure what you wanted such operation to do since to have the element-wise multiplication you need the tensors to have the same shapes.


Instruction: How can i solve backward() got an unexpected keyword argument 'retain_variables'?

Input:
I write the code following below but I got this error:

TypeError: backward() got an unexpected keyword argument 'retain_variables'

My code is:

def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
    outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
    next_outputs = self.model(batch_next_state).detach().max(1)[0]
    target = self.gamma*next_outputs + batch_reward
    td_loss = F.smooth_l1_loss(outputs, target)
    self.optimizer.zero_grad()
    td_loss.backward(retain_variables = True)
    self.optimizer.step()

Output:
I was having the same problem. This solution worked for me.

td_loss.backward(retain_graph = True)

It worked.


Instruction: I don't understand the code for training a classifier in pytorch

Input:
I don't understand the line labels.size(0). I'm new to Pytorch and been quite confused about the data structure.

correct = 0
total = 0
with torch.no_grad():
   for data in testloader:
      images, labels = data
      outputs = net(images)
      _, predicted = torch.max(outputs.data, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))`

Output: labels is a Tensor with dimensions [N, 1], where N is equal to the number of samples in the batch. .size(...) returns a subclass of tuple (torch.Size) with the dimensions of the Tensor, and .size(0) returns an integer with the value of the first (0-based) dimension (i.e., N).


Instruction: Explicit slicing across a particular dimension

Input:
I've got a 3D tensor x (e.g 4x4x100). I want to obtain a subset of this by explicitly choosing elements across the last dimension. This would have been easy if I was choosing the same elements across last dimension (e.g. x[:,:,30:50] but I want to target different elements across that dimension using the 2D tensor indices which specifies the idx across third dimension. Is there an easy way to do this in numpy?

A simpler 2D example:

x = [[1,2,3,4,5,6],[10,20,30,40,50,60]]
indices = [1,3]

Let's say I want to grab two elements across third dimension of x starting from points specified by indices. So my desired output is:

[[2,3],[40,50]]

Update: I think I could use a combination of take() and ravel_multi_index() but some of the platforms that are inspired by numpy (like PyTorch) don't seem to have ravel_multi_index so I'm looking for alternative solutions

Output:
Iterating over the idx, and collecting the slices is not a bad option if the number of 'rows' isn't too large (and the size of the sizes is relatively big).

In [55]: x = np.array([[1,2,3,4,5,6],[10,20,30,40,50,60]])
In [56]: idx = [1,3]
In [57]: np.array([x[j,i:i+2] for j,i in enumerate(idx)])
Out[57]:
array([[ 2,  3],
       [40, 50]])

Joining the slices like this only works if they all are the same size.

An alternative is to collect the indices into an array, and do one indexing.

For example with a similar iteration:

idxs = np.array([np.arange(i,i+2) for i in idx])

But broadcasted addition may be better:

In [58]: idxs = np.array(idx)[:,None]+np.arange(2)
In [59]: idxs
Out[59]:
array([[1, 2],
       [3, 4]])
In [60]: x[np.arange(2)[:,None], idxs]
Out[60]:
array([[ 2,  3],
       [40, 50]])

ravel_multi_index is not hard to replicate (if you don't need clipping etc):

In [65]: np.ravel_multi_index((np.arange(2)[:,None],idxs),x.shape)
Out[65]:
array([[ 1,  2],
       [ 9, 10]])
In [66]: x.flat[_]
Out[66]:
array([[ 2,  3],
       [40, 50]])
In [67]: np.arange(2)[:,None]*x.shape[1]+idxs
Out[67]:
array([[ 1,  2],
       [ 9, 10]])


Instruction: Whats the equivalent of tf.nn.softmax_cross_entropy_with_logits in pytorch?

Input:
I was trying to replicate a code ,which was written in tensorflow ,with pytorch. I came across a loss function in tensorflow, softmax_cross_entropy_with_logits.I was looking for an equivalent of it in pytorch and i found torch.nn.MultiLabelSoftMarginLoss,though im not quite sure it is the right function.Also i dont know how to measure the accuracy of my model when i use this loss function and no relu layer at the end of the network here is my code :

# GRADED FUNCTION: compute_cost

def compute_cost(Z3, Y):

    loss = torch.nn.MultiLabelSoftMarginLoss()
    return loss(Z3,Y)

def model(net,X_train, y_train, X_test, y_test, learning_rate = 0.009,
          num_epochs = 100, minibatch_size = 64, print_cost = True):

    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
    optimizer.zero_grad()

    total_train_acc=0

    for epoch in range(num_epochs):
        for i, data in enumerate(train_loader, 0):
            running_loss = 0.0

            inputs, labels = data

            inputs, labels = Variable(inputs), Variable(labels)

            Z3 = net(inputs)

            # Cost function
            cost = compute_cost(Z3, labels)

            # Backpropagation: Define the optimizer.
            # Use an AdamOptimizer that minimizes the cost.

            cost.backward()
            optimizer.step()

            running_loss += cost.item()

            # Measuring the accuracy of minibatch
            acc = (labels==Z3).sum()
            total_train_acc += acc.item()
            #Print every 10th batch of an epoch
            if epoch%1 == 0:
            print("Cost after epoch {} :
            {:.3f}".format(epoch,running_loss/len(train_loader)))

Output:
Use torch.nn.CrossEntropyLoss(). It combines both softmax and cross-entropy. From documentation:

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

Example:

# define loss function
loss_fn = torch.nn.CrossEntropyLoss(reduction='mean')

# during training
for (x, y) in train_loader:
    model.train()
    y_pred = model(x) # your input `torch.FloatTensor`
    loss_val = loss_fn(y_pred, y)
    print(loss_val.item()) # prints numpy value

    optimizer.zero_grad()
    loss_val.backward()
    optimizer.step()

Make sure that the types of x and y are correct. Usually the conversion is done like this: loss_fn(y_pred.type(torch.FloatTensor), y.type(torch.LongTensor)).

To measure accuracy you can define a custom function:

def compute_accuracy(y_pred, y):
   if list(y_pred.size()) != list(y.size()):
      raise ValueError('Inputs have different shapes.',
                       list(y_pred.size()), 'and', list(y.size()))

  result = [1 if y1==y2 else 0 for y1, y2 in zip(y_pred, y)]

  return sum(result) / len(result)

And use both like this:

model.train()
y_pred = model(x)

loss_val = loss_fn(y_pred.type(torch.FloatTensor), y.type(torch.LongTensor))
_, y_pred = torch.max(y_pred, 1)
accuracy_val = compute_accuracy(y_pred, y)
print(loss_val.item()) # print loss value
print(accuracy_val) # print accuracy value
# update step e.t.c

If your input data is one-hot encoded you can convert it to regular encoding before you use loss_fn:

_, targets = y.max(dim=1)
y_pred = model(x)
loss_val = loss_fn(y_pred, targets)


Instruction: Unable to save Pytorch model to Google Drive in Google Colab?

Input:
I am trying to save my model to my drive on google colab. I have used the following code to mount my Google Drive-

from google.colab import drive
drive.mount('/content/gdrive')

After all the preprocessing, model definition and training, I want to save my model to the drive because training it will take a long time. So, I will save it to drive at regular intervals and reload from that point to continue.
The code to save my model is:

def save_model(model, model_name, iter):
  path = f'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'
  print(f'Saving {model_name} model...')
  torch.save(model.state_dict(), path)
  print(f'{model_name} saved successfully.')

EMBEDDING_DIMS = 128
HIDDEN_SIZE = 256

gru = GRU(vocab.n_words, EMBEDDING_DIMS, HIDDEN_SIZE, 2).to(device)
save_model(gru, 'gru', 0)

I am getting the following error:

Saving gru model...
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-27-d2510611a9d4> in <module>()
      9
     10 gru = GRU(vocab.n_words, EMBEDDING_DIMS, HIDDEN_SIZE, 2).to(device)
---> 11 save_model(gru, 'gru', 0)

<ipython-input-27-d2510611a9d4> in save_model(model, model_name, iter)
      2   path = f'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'
      3   print(f'Saving {model_name} model...')
----> 4   torch.save(model.state_dict(), path)
      5   print(f'{model_name} saved successfully.')
      6

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol)
    217         >>> torch.save(x, buffer)
    218     """
--> 219     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
    220
    221

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _with_file_like(f, mode, body)
    140             (sys.version_info[0] == 3 and isinstance(f, pathlib.Path)):
    141         new_fd = True
--> 142         f = open(f, mode)
    143     try:
    144         return body(f)

FileNotFoundError: [Errno 2] No such file or directory: 'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_gru_iter_0.pth'

I have manually created the folder in my drive and only the file needs to be created. Still, the error persists. Though, I am sure that manually creating the folder was not required. The problem is something else.
Where am I going wrong?

Output:
You likely need a leading / in your path.

Try changing this line:

  path = f'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'

to:

  path = f'/content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'


Instruction: How does the apply(fn) function in pytorch work with a function without return statement as argument?

Input:
I have some questions about the following code fragments:

>>> def init_weights(m):
        print(m)
        if type(m) == nn.Linear:
            m.weight.data.fill_(1.0)
            print(m.weight)

>>> net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
>>> net.apply(init_weights)

apply() is part of the pytorch.nn package. You find the code in the documentation of this package. The final questions:
1. Why does this code sample work, although there is no argument or brackets added to init_weights() when it is given to apply()?
2. Where does the function init_weights(m) gets its argument m from, when it's given as a parameter to the function apply() without brackets and an m?

Output:
We find the answers to your questions in said documentation of torch.nn.Module.apply(fn):

  Applies fn recursively to every submodule (as returned by .children())
  as well as self. Typical use includes initializing the parameters of a model
  (see also torch-nn-init).

Why does this code sample work, althouh there is no argument or brackets added to init_weights() when it is given to apply()?

The given function init_weights isn't called prior to the apply call, precisely because there are no parentheses, rather a reference to init_weights is given to apply, and only from within apply later on init_weights is called.

Where does the function init_weights(m) gets its argument m from, when it's given as a parameter to the function apply() without brackets and an m?

It gets its argument with each call within apply, and, as the documentation tells, it is called for m iterating over every submodule of (in this case) net as well as net itself, due to the method call net.apply(…).


Instruction: Using Multiple GPUs outside of training in PyTorch

Input:
I'm calculating the accumulated distance between each pair of kernel inside a nn.Conv2d layer. However for large layers it runs out of memory using a Titan X with 12gb of memory. I'd like to know if it is possible to divide such calculations across two gpus.
The code follows:

def ac_distance(layer):
    total = 0
    for p in layer.weight:
      for q in layer.weight:
         total += distance(p,q)
    return total

Where layer is instance of nn.Conv2d and distance returns the sum of the differences between p and q. I can't detach the graph, however, for I need it later on. I tried wrapping my model around a nn.DataParallel, but all calculations in ac_distance are done using only 1 gpu, however it trains using both.

Output:
Parallelism while training neural networks can be achieved in two ways.

Data Parallelism - Split a large batch into two and do the same set of operations but individually on two different GPUs respectively
Model Parallelism - Split the computations and run them on different GPUs

As you have asked in the question, you would like to split the calculation which falls into the second category. There are no out-of-the-box ways to achieve model parallelism. PyTorch provides primitives for parallel processing using the torch.distributed package. This tutorial comprehensively goes through the details of the package and you can cook up an approach to achieve model parallelism that you need.

However, model parallelism can be very complex to achieve. The general way is to do data parallelism with either torch.nn.DataParallel or torch.nn.DistributedDataParallel. In both the methods, you would run the same model on two different GPUs, however one huge batch would be split into two smaller chunks. The gradients will be accumulated on a single GPU and optimization happens. Optimization takes place on a single GPU in Dataparallel and parallely across GPUs in DistributedDataParallel by using multiprocessing.

In your case, if you use DataParallel, the computation would still take place on two different GPUs. If you notice imbalance in GPU usage it could be because of the way DataParallel has been designed. You can try using DistributedDataParallel which is the fastest way to train on multiple GPUs according to the docs.

There are other ways to process very large batches too. This article goes through them in detail and I'm sure it would be helpful. Few important points:

Do gradient accumulation for larger batches
Use DataParallel
If that doesn't suffice, go with DistributedDataParallel


Instruction: How does one dynamically add new parameters to optimizers in Pytorch?

Input:
I was going through this post in the pytorch forum, and I also wanted to do this. The original post removes and adds layers but I think my situation is not that different. I also want to add layers or more filters or word embeddings. My main motivation is that the AI agent does not know the whole vocabulary/dictionary in advance because its large. I prefer strongly (for the moment) to not do character by character RNNs.

So what will happen for me is when the agent starts a forward pass it might find new words it has never seen and will need to add them to the embedding table (or perhaps add new filters before it starts the forward pass).

So what I want to make sure is:

embeddings are added correctly (at the right time, when a new computation graph is made) so that they are updatable by the optimizer
no issues with stored info of past parameters e.g. if its using some sort of momentum

How does one do this? Any sample code that works?

Output:
Just to add an answer to the title of your question: "How does one dynamically add new parameters to optimizers in Pytorch?"

You can append params at any time to the optimizer:

import torch
import torch.optim as optim

model = torch.nn.Linear(2, 2)

# Initialize optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001, momentum=0.9)

extra_params = torch.randn(2, 2)
optimizer.param_groups.append({'params': extra_params })

#then you can print your `extra_params`
print("extra params", extra_params)
print("optimizer params", optimizer.param_groups)


Instruction: How to convert BatchNorm weight of caffe to pytorch BathNorm?

Input:
BathNorm and Scale weight of caffe model can be read from pycaffe, which are three weights in BatchNorm and two weights in Scale. I tried to copy those weights to pytorch BatchNorm with codes like this:

if 'conv3_final_bn' == name:
    assert len(blobs) == 3, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.running_mean'] = blobs[0].data
    torch_mod['conv3_final_bn.running_var'] = blobs[1].data
elif 'conv3_final_scale' == name:
    assert len(blobs) == 2, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.weight'] = blobs[0].data
    torch_mod['conv3_final_bn.bias'] = blobs[1].data

The two BatchNorm acts differently. I also tried to set conv3_final_bn.weight=1 and conv3_final_bn.bias=0 to verify the BN layer of caffe, the results didn't match either.

How should I deal with the wrong matching?

Output:
Got it! There is still a third parameter in BatchNorm of caffe. Codes should be:

if 'conv3_final_bn' == name:
    assert len(blobs) == 3, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.running_mean'] = blobs[0].data / blobs[2].data[0]
    torch_mod['conv3_final_bn.running_var'] = blobs[1].data / blobs[2].data[0]
elif 'conv3_final_scale' == name:
    assert len(blobs) == 2, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.weight'] = blobs[0].data
    torch_mod['conv3_final_bn.bias'] = blobs[1].data


Instruction: How to use torch.nn.CrossEntropyLoss as autoencoder's reconstruction loss?

Input:
I want to compute the reconstruction accuracy of my autoencoder using CrossEntropyLoss:

ae_criterion = nn.CrossEntropyLoss()
ae_loss = ae_criterion(X, Y)

where X is the autoencoder's reconstruction and Y is the target (since it is an autoencoder, Y is the same as the original input X).
Both X and Y have shape [42, 32, 130] = [batch_size, timesteps, number_of_classes]. When I run the code above I get the following error:

  ValueError: Expected target size (42, 130), got torch.Size([42, 32,
  130])

After looking the docs, I'm still unsure on how should I call nn.CrossEntropyLoss() in the appropriate way. It seems that I should change Y to be of shape [42, 32, 1], with each element being a scalar in the interval [0, 129] (or [1, 130]), am I right?

Is there a way to avoid this? Since X and Y are between 0 and 1, could I just use binary cross-entropy loss element-wise in an equivalent way?

Output:
For CrossEntropyLoss, shape of the Y must be (42, 32), each element must be a Long scalar in the interval [0, 129].

You may want to use BCELoss or BCEWithLogitsLoss for your problem.


Instruction: How can I use KNN, Random Forest models in Pytorch?

Input: This may seem like a X Y problem, but initially I had huge data and I was not able to train in given resources (RAM problem). So I thought I could use batch feature of Pytorch. But I want to use Methods like KNN, Random Forest, Clustering except Deep Learning. So is it possible or can I use scikit libraries in Pytorch?

Output:
Update
Currently, there are some sklearn alternatives utilizing GPU, most prominent being cuML (link here) provided by rapidsai.
Previous answer
I would advise against using PyTorch solely for the purpose of using batches.
Argumentation goes as follows:

scikit-learn has docs about scaling where one can find MiniBatchKMeans and there are other options like partial_fit method or warm_start arguments (as is the case with RandomForest, check this approach).
KNN cannot be easily used without hand-made implementation with disk caching as it stores whole dataset in memory (and you lack RAM). This approach would be horribly inefficient either way, do not try.
You most probably will not be able to create algorithms on-par with those from scikit (at least not solo and not without considerable amount of work). Your best bet is to go with quite battle-tested solutions (even though it's still 0.2x currently). It should be possible to get some speed improvements through numba but that's beside the scope of this question. Maybe you could utilize CUDA for different algorithms but it's even more non-trivial task.

All in all PyTorch is suited for deep learning computations with heavy CUDA usage. If you need neural networks, this framework is one of the best out there, otherwise go with something like sklearn or other frameworks allowing incremental training. You can always bridge those two easily with numpy() and few other  calls in pytorch.
EDIT: I have found KNN implementation possibly suiting your requirements in this github repository


Instruction: How to fix 'ImportError: /home/... .../lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize' in DGL?

Input:
I get an error in the importation of pytorch inside dgl (Deep Graph Library by DeepMind), concretely:

ImportError: /home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize

I tried to reinstall pytorch (uninstall reinstall with conda un/install). I also search in google and I found this https://github.com/pytorch/pytorch/issues/14973. There, they solve it linking in libnvrtc.so and libcuda.so, but I have no idea what that means. Does anyone know it?

This is the basic code:

import dgl
from parseador import train_df

g = dgl.DGLGraph()
g.add_nodes(5)
g.add_edges([0, 0, 0, 0], [1, 2, 3, 4])
g.ndata['h'] = th.randn(5, 3)
g.edata['h'] = th.randn(4, 4)

And this is the error:

Traceback (most recent call last):
  File "/home/user/Documentos/Repo/grafos.py", line 1, in <module>
    import dgl
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/__init__.py", line 2, in <module>
    from . import function
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/function/__init__.py", line 5, in <module>
    from .message import *
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/function/message.py", line 7, in <module>
    from .. import backend as F
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/__init__.py", line 46, in <module>
    load_backend(os.environ.get('DGLBACKEND', 'pytorch').lower())
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/__init__.py", line 18, in load_backend
    mod = importlib.import_module('.%s' % mod_name, __name__)
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/pytorch/__init__.py", line 1, in <module>
    from .tensor import *
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/pytorch/tensor.py", line 5, in <module>
    import torch as th
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/__init__.py", line 102, in <module>
    from torch._C import *
ImportError: /home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize

How can I fix this error? some time ago I run correctly this code in Windows 10, now I'm running Ubuntu 18.04.

Output:
I also ran into this, but I actually wanted to use GPU, so installing pytorch-cpu was not an option for me.

Instead, installing pytorch package from pytorch channel (instead of defaults) solved the issue for me: conda install pytorch --channel pytorch


Instruction: Manage memory differently on train and test time pytorch

Input:
Currently I'm writing a segmentation model based on U-net with pytorch and I want to use something similar to inverted residual introduced on mobilenet v2 to improve the model's speed on cpu. pytorch code for mobile netv2

Then I realize that the model uses a lot more memory on train phase and test phase. While the model should use more memory on train phase because all the mid-step tensors(feature maps) are saved and with separable convolution there are more tensors created for each "convolution" operation.
But on run time, actually only few last step tensors must be saved to be used for skip connection and all the other tensors can be deleted once it's next step is created. The memory efficiency should be the same for u-net with normal convolution and u-net with separable convolution on test phase.

I'm newbee to pytorch so I don't know how to write code that prevents unnecessary memory cost on test time. Since pytorch is binded with python. I guess I can manually delete all the unnecessary tensors in forward function with del. But I guess that if I just delete variables on forward function, it will influence training stage.
Is here more advanced functionality on pytorch that is able to optimize test phase memory usage with a 'network graph'?
I'm also curious if tensorflow deals with those problems automatically since it has a more abstract and complex graph building logic.

Output:
After reading the official pytorch code for resnet, I realize I shouldn't give all variables a name.aka I shouldn't write:

conv1 = self.conv1(x)
conv2 = self.conv2(conv1)

I should just write:

out = self.conv1(x)
out = self.conv2(out)

On this way nothing refers to obj corresponds to conv1 after it is used and python is able to clean it.

because there are residual connections between blocks, I need to have one more python variable to refer to the variable: aka

out = self.conv1(x)
residual_connect = out
out = self.conv2(out)
out = conv1 + out

But on upsampling stage only out is needed. So I deleted residual_connect at the beginning of the decoding stage.

del residual_connect

It seems like a hack and I'm surprised that it didn't cause problem on training stage. The ram usage for my model is greatly reduced now but I feel here should be a more elegant way to solve the problem.


Instruction: Torch.cuda.is_available() keeps switching to False

Input:
I have tried several solutions which hinted at what to do when the CUDA GPU is available and CUDA is installed but the Torch.cuda.is_available() returns False. They did help but only temporarily, meaning torch.cuda-is_available() reported True but after some time, it switched back to False. I use CUDA 9.0.176 and GTX 1080. What should I do to get the permanent effect?
I tried the following methods:
https://forums.fast.ai/t/torch-cuda-is-available-returns-false/16721/5
https://github.com/pytorch/pytorch/issues/15612
Note: When torch.cuda.is_available() works fine but then at some point switches to False, then I have to restart the computer and then it works again (for some time).

Output:
The reason for torch.cuda.is_available() resulting False is the incompatibility between the versions of pytorch and cudatoolkit.
As on Jun-2022, the current version of pytorch is compatible with cudatoolkit=11.3 whereas the current cuda toolkit version = 11.7. Source
Solution:

Uninstall Pytorch for a fresh installation. You cannot install an old version on top of a new version without force installation (using pip install --upgrade --force-reinstall <package_name>.

Run conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch to install pytorch.

Install CUDA 11.3 version from https://developer.nvidia.com/cuda-11.3.0-download-archive.

You are good to go.


Instruction: PyTorch: Batch Outer-Addition

Input: I have two PyTorch tensors: A and B, both of shape (b, c, 3). I want to make outer product C of A and B so that the resulting shape is (b, c, 3, 3), and replace the multiplication operation with addition. How should I do it?

Output:
You can add a corresponding singleton dimension:

C = A[..., None] + B[..., None, :]

For example, with batch and channel dimensions equal to 1 (b=1, c=1):

import torch
A = torch.tensor([[[1, 2, 3.]]])
B = torch.tensor([[[4., 5., 6.]]])
A[..., None] + B[..., None, :]
Out[ ]:
tensor([[[[5., 6., 7.],
          [6., 7., 8.],
          [7., 8., 9.]]]])


Instruction: Missing Keys in state_dict

Input:
I am having problems loading my model on google colab. here is the code:

I have attached the code below

I have tried changing the name of the statedict and it does not help
basically, I am trying to save my model for later use, but, this is becoming extremely difficult since I am not being able to properly save and load it. Please help me with the problem. After the section of the code, you will also find the error that I have attached below.

here is the code

from zipfile import ZipFile
file_name = 'data.zip'
with ZipFile(file_name, 'r') as zip:
  zip.extractall()

from zipfile import ZipFile
file_name = 'results.zip'
with ZipFile(file_name, 'r') as zip:
  zip.extractall()

!pip install tensorflow-gpu

from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
from torch.autograd import Variable

batchSize = 64
imageSize = 64

transform = transforms.Compose([transforms.Resize(imageSize), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),])

dataset = dset.CIFAR10(root = './data', download = True, transform = transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size = batchSize, shuffle = True, num_workers = 2)

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

class G(nn.Module):

    def __init__(self):
        super(G, self).__init__()
        self.main = nn.Sequential(
            nn.ConvTranspose2d(100, 512, 4, 1, 0, bias = False),
            nn.BatchNorm2d(512),
            nn.ReLU(True),
            nn.ConvTranspose2d(512, 256, 4, 2, 1, bias = False),
            nn.BatchNorm2d(256),
            nn.ReLU(True),
            nn.ConvTranspose2d(256, 128, 4, 2, 1, bias = False),
            nn.BatchNorm2d(128),
            nn.ReLU(True),
            nn.ConvTranspose2d(128, 64, 4, 2, 1, bias = False),
            nn.BatchNorm2d(64),
            nn.ReLU(True),
            nn.ConvTranspose2d(64, 3, 4, 2, 1, bias = False),
            nn.Tanh()
        )

    def forward(self, input):
        output = self.main(input)
        return output

netG = G()
netG.load_state_dict(torch.load('generator.pth'))
netG.eval()
#netG.apply(weights_init)

class D(nn.Module):

    def __init__(self):
        super(D, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(3, 64, 4, 2, 1, bias = False),
            nn.LeakyReLU(0.2, inplace = True),
            nn.Conv2d(64, 128, 4, 2, 1, bias = False),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2, inplace = True),
            nn.Conv2d(128, 256, 4, 2, 1, bias = False),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(0.2, inplace = True),
            nn.Conv2d(256, 512, 4, 2, 1, bias = False),
            nn.BatchNorm2d(512),
            nn.LeakyReLU(0.2, inplace = True),
            nn.Conv2d(512, 1, 4, 1, 0, bias = False),
            nn.Sigmoid()
        )

    def forward(self, input):
        output = self.main(input)
        return output.view(-1)

netD = D()
netD.load_state_dict(torch.load('discriminator.pth'))
netD.eval()
#netD.apply(weights_init)

criterion = nn.BCELoss()
checkpoint = torch.load('discriminator.pth')
optimizerD = optim.Adam(netD.parameters(), lr = 0.0002, betas = (0.5, 0.999))
optimizerD.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
errD = checkpoint['loss']
checkpoint1 = torch.load('genrator.pth')
optimizerG = optim.Adam(netG.parameters(), lr = 0.0002, betas = (0.5, 0.999))
optimizerG.load_state_dict(checkpoint1['optimizer_state_dict'])
errG = checkpoint1['loss']
k = epoch
for j in range(k, 10):

    for i, data in enumerate(dataloader, 0):

        netD.zero_grad()

        real, _ = data
        input = Variable(real)
        target = Variable(torch.ones(input.size()[0]))
        output = netD(input)
        errD_real = criterion(output, target)

        noise = Variable(torch.randn(input.size()[0], 100, 1, 1))
        fake = netG(noise)
        target = Variable(torch.zeros(input.size()[0]))
        output = netD(fake.detach())
        errD_fake = criterion(output, target)

        errD = errD_real + errD_fake
        errD.backward()
        optimizerD.step()

        netG.zero_grad()
        target = Variable(torch.ones(input.size()[0]))
        output = netD(fake)
        errG = criterion(output, target)
        errG.backward()
        optimizerG.step()

        print('[%d/%d][%d/%d] Loss_D: %.4f Loss_G: %.4f' % (epoch+1, 10, i+1, len(dataloader), errD.data, errG.data))
        if i % 100 == 0:
            vutils.save_image(real, '%s/real_samples.png' % "./results", normalize = True)
            fake = netG(noise)
            vutils.save_image(fake.data, '%s/fake_samples_epoch_%03d.png' % ("./results", epoch+1), normalize = True)

torch.save({
            'epoch': epoch,
            'model_state_dict': netD.state_dict(),
            'optimizer_state_dict': optimizerD.state_dict(),
            'loss': errD
            }, 'discriminator.pth')
torch.save({
            'epoch': epoch,
            'model_state_dict': netG.state_dict(),
            'optimizer_state_dict': optimizerG.state_dict(),
            'loss': errG
            }, 'generator.pth')

here is the error

RuntimeError                              Traceback (most recent call last)
<ipython-input-23-3e55546152c7> in <module>()
     26 # Creating the generator
     27 netG = G()
---> 28 netG.load_state_dict(torch.load('generator.pth'))
     29 netG.eval()
     30 #netG.apply(weights_init)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
    767         if len(error_msgs) > 0:
    768             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 769                                self.__class__.__name__, "\n\t".join(error_msgs)))
    770
    771     def _named_members(self, get_members_fn, prefix='', recurse=True):

RuntimeError: Error(s) in loading state_dict for G:
    Missing key(s) in state_dict: "main.0.weight", "main.1.weight", "main.1.bias", "main.1.running_mean", "main.1.running_var", "main.3.weight", "main.4.weight", "main.4.bias", "main.4.running_mean", "main.4.running_var", "main.6.weight", "main.7.weight", "main.7.bias", "main.7.running_mean", "main.7.running_var", "main.9.weight", "main.10.weight", "main.10.bias", "main.10.running_mean", "main.10.running_var", "main.12.weight".
    Unexpected key(s) in state_dict: "epoch", "model_state_dict", "optimizer_state_dict", "loss".

Output:
You need to access the 'model_state_dict' key inside the loaded checkpoint.
Try:

netG.load_state_dict(torch.load('generator.pth')['model_state_dict'])

You'll probably need to apply the same fix to the discriminator as well.


Instruction: Model learns with SGD but not Adam

Input:
I was going through a basic PyTorch MNIST example here and noticed that when I changed the optimizer from SGD to Adam the model did not converge. Specifically, I changed line 106 from

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

to

optimizer = optim.Adam(model.parameters(), lr=args.lr)

I thought this would have no effect on the model. With SGD the loss quickly dropped to low values after about a quarter of an epoch. However with Adam, the loss did not drop at all even after 10 epochs. I'm curious to why this is happening; it seems to me these should have nearly identical performance.

I ran this on Win10/Py3.6/PyTorch1.01/CUDA9

And to save you a tiny bit of code digging, here are the hyperparams:

lr=0.01
momentum=0.5
batch_size=64

Output:
Adam is famous for working out of the box with its default paremeters, which, in almost all frameworks, include a learning rate of 0.001 (see the default values in Keras, PyTorch, and Tensorflow), which is indeed the value suggested in the Adam paper.

So, I would suggest changing to

optimizer = optim.Adam(model.parameters(), lr=0.001)

or simply

optimizer = optim.Adam(model.parameters())

in order to leave lr in its default value (although I would say I am surprised, as MNIST is famous nowadays for working practically with whatever you may throw into it).


Instruction: What is the meaning of "x:" and of the following line?

Input:
What is the meaning of x: and of the following line?

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])

Can somebody explain the syntax of this line?

It is from PyTorch tutorial:
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'data/hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Output:
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}

x comes from the for you have below. for x in ['train', 'val'], so for each value in ['train', 'val'], you are creating a dict in which that x is the key.

This kind of construct, as Kabanus said, its a dictionary comprehension, a dictionary generator.

You can know more about generators in here: https://docs.python.org/3/tutorial/classes.html#generators


Instruction: error: unrecognized arguments: - pytorch code in colab

Input:
I tried to run my code on google colab. but I get this message (error: unrecognized arguments)when I'm trying to call this function :

def parse_opts():
 parser = argparse.ArgumentParser()
 parser.add_argument(
     '--root_path',
     default='/root/data/ActivityNet',
     type=str,
     help='Root directory path of data')
  parser.add_argument(
     '--video_path',
     default='video_kinetics_jpg',
     type=str,
     help='Directory path of Videos')
 args = parser.parse_args()

return args

but this is failed and I get this error

  tester_video.py: error: unrecognized arguments:
  cifar_comp_20_200_0.01_0.1 20 10 0.01 0.1

I tried to use Easydict but it seems its not working
thinks

Output:
The problem only appears in Jupyter notebook/lab/colab.
Change
args = parser.parse_args()
to
args = parser.parse_args(args=[])
and it should fix it.


Instruction: RuntimeError: CUDA out of memory. Problem when re-loading the model in a loop

Input:
I am running into the classic: CUDA out of memory.

What I want to do: I want to load the same model using a different matrix of embeddings every time. I have to do that 300 times, one for each dimension of the word embeddings.

I am not training the model, that is why I am using model.eval(), I thought that would be enough to keep Pytorch from creating a graph.

Please notice that I never pass the model, nor the data, to cuda. In fact, I wanted to debug the code using cpu before sending the code to be performed by a GPU.

The loop below is executed once, a RuntimeError is raised in the second iteration.

My guess is that the code is loading a new model into a GPU memory at each iteration (which I did not know was possible without explicitly pointing it to do so). The emb_matrix is quite heavy and could cause the GPU memory to crash.

emb_dim = 300
acc_dim = torch.zeros((emb_dim, 4))
for d in range(emb_dim):

    #create embeddings with one dimension shuffled
    emb_matrix = text_f.vocab.vectors.clone()

    #get a random permutation across one of the dimensions
    rand_index = torch.randperm(text_f.vocab.vectors.shape[0])
    emb_matrix[:, d] =  text_f.vocab.vectors[rand_index, d]

    #load model with the scrumbled embeddings
    model = load_classifier(emb_matrix,
                            encoder_type = encoder_type)
    model.eval()
    for batch in batch_iters["test"]:
        x_pre = batch.premise
        x_hyp = batch.hypothesis
        y = batch.label

        #perform forward pass
        y_pred = model.forward(x_pre, x_hyp)

        #calculate accuracies
        acc_dim[d] += accuracy(y_pred, y)/test_batches

        #avoid memory issues
        y_pred.detach()

    print(f"Dimension {d} accuracies: {acc_dim[d]}")

I get the following error:
RuntimeError: CUDA out of memory. Tried to allocate 146.88 MiB (GPU 0; 2.00 GiB total capacity; 374.63 MiB already allocated; 0 bytes free; 1015.00 KiB cached)

I tried passing the model and the data to CPU, but I get precisely the same error.

I looked around for how to fix the problem, but I could not find an obvious solution. Any suggestions on how to load the model and data in the correct place, or how to clean the GPU's memory after each iteration are welcome.

Output:
It looks like acc_dim accumulates the grad history - see https://pytorch.org/docs/stable/notes/faq.html

Because you're only do inference, with torch.no_grad(): should be used.
This will completely sidestep the possible issue with accumulating grad history.

model.eval() doesn't prevent grad bookkeeping from happening, it just switches behavior of some layers like dropout. Both model.eval() and with torch.no_grad(): together should be used for inference.


Instruction: Check if the expected object is of backend CUDA or CPU?

Input:
  I am trying to run a code on both CPU and CUDA.
  The problem arise when I create objects, as I need to know what's expected.

I need to determine if the computer is expecting a CUDA or CPU tensor, before it is created.

Code:

def initilize(self, input):
     self.x = torch.nn.Parameter(torch.zeros((1,M))

def run(self,x,state):
  B = torch.cat((self.x,h)

This outputs:

Error: 'Expected object of backend CUDA but got backend CPU for argument #1'

Code idea:

def initilize(self, input):
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  if (expecting_cuda == True):
     self.x = torch.nn.Parameter(torch.zeros((1,M)).to(device))
  else
     self.x = torch.nn.Parameter(torch.zeros((1,M))

def run(self,h):
  B = torch.cat((self.x,h)

Question:
How to figure out what the computer expects?
Limitations:
I am running on a pre-defined "check" procedure, so I can not send an argument into the function 'initilize' with information about CUDA or CPU.

Output: You can just use self.x = torch.nn.Parameter(torch.zeros((1,M)).to(device)), no need for if (expecting_cuda == True): because to(device) will also work for cpu.


Instruction: Error in loading state_dict for customed model

Input:
I had problems when loading the weights of model. Here's some parts of the model

class InceptionV4(nn.Module):

   def __init__(self, num_classes=1001):
       super(InceptionV4, self).__init__()
       # Special attributs
       self.input_space = None
       self.input_size = (299, 299, 3)
       self.mean = None
       self.std = None
       # Modules
       self.features = nn.Sequential(
           BasicConv2d(3, 32, kernel_size=3, stride=2),
           BasicConv2d(32, 32, kernel_size=3, stride=1),
           BasicConv2d(32, 64, kernel_size=3, stride=1, padding=1),
           Mixed_3a(),
           Mixed_4a(),
           Mixed_5a(),
           Inception_A(),
           Inception_A(),
           Inception_A(),
           ...
       )
       self.avg_pool = nn.AvgPool2d(8, count_include_pad=False)
       self.last_linear = nn.Linear(1536, num_classes)

I have tried to save the weights, something like torch.save(model.state_dict(), weight_name) and then reload again model.load_state_dict(torch.load(weight_name))
but got these errors:

Missing key(s) in state_dict: "features.0.conv.weight", "features.0.bn.weight", "features.0.bn.bias", "features.0.bn.running_mean", "features.0.bn.running_var", "features.1.conv.weight", "features.1.bn.weight", "features.1.bn.bias", "features.1.bn.running_mean", "features.1.bn.running_var", "features.2.conv.weight", "features.2.bn.weight

and also:

Unexpected key(s) in state_dict: "conv.0.conv1.0.weight", "conv.0.conv1.0.bias", "conv.0.conv1.2.weight", "conv.0.conv1.2.bias", "conv.0.conv1.2.running_mean", "conv.0.conv1.2.running_var", "conv.0.conv1.2.num_batches_tracked", "conv.0.conv2.0.weight", "conv.0.conv2.0.bias", "conv.0.conv2.2.weight", "conv.0.conv2.2.bias", "conv.0.conv2.2.running_mean", "conv.0.conv2.2.running_var", "conv.0.conv2.2.num_batches_tracked", "conv.1.conv1.0.weight", "conv.1.conv1.0.bias", "conv.1.conv1.2.weight", "conv.1.conv1.2.bias", "conv.1.conv1.2.running_mean", "conv.1.conv1.2.running_var", "conv.1.conv1.2.num_batches_tracked

Any hints on this? Thanks in advance.

Output:
I faced this problem several times. The error indicates that your model state_dict has different names from the pre-trained weights that you load.

I don't see the pretrained model for Inception_v4 in torchvision model zoo, so it would be a little difficult to tell exactly where your InceptionV4 class has a problem with mismatched dict.

Regardless of where you get your the pre-trained file, but the key point is to define your model the same as the pre-trained model code, and you can load the weight file smoothly.

Here are some indicators where your code is different from the model:

# change self.features -> self.conv: This helps in solving mismatched names.

self.conv = nn.Sequential(...)

# Google how to change the BatchNorm in your current pytorch version
# and  the older pytorch version which the pretrained model was defined.

conv.1.conv1.2.num_batches_tracked  # it is deprecated in pytorch version 0.4 or newer

The hint is:

# Define your model (or parts you want to reuse) the same as the original

Hope this helps :)


Instruction: Different `grad_fn` for similar looking operations in Pytorch (1.0)

Input:
I am working on an attention model, and before running the final model, I was going through the tensor shapes which flow through the code. I have an operation where I need to reshape the tensor. The tensor is of the shape torch.Size([[30, 8, 9, 64]])  where 30 is the batch_size, 8 is the number of attention head (this is not relevant to my question) 9 is the number of words in the sentence and 64 is some intermediate embedding representation of the word. I have to reshape the tensor to a size of torch.size([30, 9, 512]) before processing it further. So I was looking into some reference online and they have done the following x.transpose(1, 2).contiguous().view(30, -1, 512)
whereas I was thinking that this should work x.transpose(1, 2).reshape(30, -1, 512).

In the first case the grad_fn is <ViewBackward>, whereas in my case it is <UnsafeViewBackward>. Aren't these two the same operations? Will this result in a training error?

Output:
Aren't these two the same operations?

No. While they produce effectively the same tensor, the operations are not the same, and they are not guaranteed to have the same storage.
TensorShape.cpp:
// _unsafe_view() differs from view() in that the returned tensor isn't treated
// as a view for the purposes of automatic differentiation. (It's not listed in
// VIEW_FUNCTIONS in gen_autograd.py).  It's only safe to use if the `self` tensor
// is temporary. For example, the viewed tensor here (a + b) is discarded immediately
// after viewing:
//
//  res = at::_unsafe_view(a + b, size);
//
// This is a hack because in-place operations on tensors treated like views
// can be much more expensive than the same operations on non-view tensors.

Note this can produce an error if applied to complex inputs, but this is generally not yet fully supported in PyTorch and not unique to this function.


Instruction: Pytorch LSTM text-generator repeats same words

Input:
UPDATE: It was a mistake in the logic generating new characters. See answer below.

ORIGINAL QUESTION: I built an LSTM for character-level text generation with Pytorch. The model trains well (loss decreases reasonably etc.) but the trained model ends up outputting the last handful of words of the input repeated over and over again (e.g. Input: "She told her to come back later, but she never did"; Output: ", but she never did, but she never did, but she never did" and so on).

I have played around with the hyperparameters a bit, and the problem persists. I'm currently using:

Loss function: BCE
Optimizer: Adam
Learning rate: 0.001
Sequence length: 64
Batch size: 32
Embedding dim: 128
Hidden dim: 512
LSTM layers: 2

I also tried not always choosing the top choice, but this only introduces incorrect words and doesn't break the loop. I've been looking at countless tutorials, and I can't quite figure out what I'm doing differently/wrong.

The following is the code for training the model. training_data is one long string and I'm looping over it predicting the next character for each substring of length SEQ_LEN. I'm not sure if my mistake is here or elsewhere but any comment or direction is highly appreciated!

loss_dict = dict()
for e in range(EPOCHS):
    print("------ EPOCH {} OF {} ------".format(e+1, EPOCHS))

    lstm.reset_cell()

    for i in range(0, DATA_LEN, BATCH_SIZE):

        if i % 50000 == 0:
            print(i/float(DATA_LEN))

        optimizer.zero_grad()

        input_vector = torch.tensor([[
            vocab.get(char, len(vocab))
            for char in training_data[i+b:i+b+SEQ_LEN]
        ] for b in range(BATCH_SIZE)])

        if USE_CUDA and torch.cuda.is_available():
            input_vector = input_vector.cuda()

        output_vector = lstm(input_vector)

        target_vector = torch.zeros(output_vector.shape)

        if USE_CUDA and torch.cuda.is_available():
            target_vector = target_vector.cuda()

        for b in range(BATCH_SIZE):
            target_vector[b][vocab.get(training_data[i+b+SEQ_LEN])] = 1

        error = loss(output_vector, target_vector)

        error.backward()
        optimizer.step()

        loss_dict[(e, int(i/BATCH_SIZE))] = error.detach().item()

Output:
ANSWER: I had made a stupid mistake when producing the characters with the trained model: I got confused with the batch size and assumed that at each step the network would predict an entire batch of new characters when in fact it only predicts a single one… That's why it simply repeated the end of the input. Yikes!

Anyways, if you run into this problem DOUBLE CHECK that you have the right logic for producing new output with the trained model (especially if you're using batches). If it's not that and the problem persists, you can try fine-tuning the following:

sequence length
greediness (e.g. probabilistic choice vs. top choice for next character)
batch size
epochs


Instruction: Pytorch is installed, but while importing getting error

Input:
I have installed Anaconda on windows 8, after that done with the PyTorch.
But while importing torch getting error.

  import torch

  AttributeError                            Traceback (most recent
call last)

<ipython-input-4-eb42ca6e4af3> in <module>

----> 1 import torch

I:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py in
<module>
 51         from ctypes.wintypes import DWORD, HMODULE
 52

---> 53         AddDllDirectory = windll.kernel32.AddDllDirectory
 54         AddDllDirectory.restype = DWORD
 55         AddDllDirectory.argtypes = [c_wchar_p]

I:\ProgramData\Anaconda3\lib\ctypes\__init__.py in __getattr__(self,
name)
367         if name.startswith('__') and name.endswith('__'):
368             raise AttributeError(name)

--> 369         func = self.__getitem__(name)
370         setattr(self, name, func)
371         return func

I:\ProgramData\Anaconda3\lib\ctypes\__init__.py in __getitem__(self,
name_or_ordinal)
372
373     def __getitem__(self, name_or_ordinal):

--> 374         func = self._FuncPtr((name_or_ordinal, self))
375         if not isinstance(name_or_ordinal, int):
376             func.__name__ = name_or_ordinal

AttributeError: function 'AddDllDirectory' not found

Also when I used the following command it is showing the pytorch in
Anaconda command prompt.

(base) C:\Users\rk88>conda list pytorch

# packages in environment at I:\ProgramData\Anaconda3:

#

# Name                    Version                   Build  Channel

pytorch                   1.0.1           py3.7_cuda100_cudnn7_1
PyTorch

Anyone can please help me how to resolve this, if you faced this issue earlier.

Thanks,

Output:
Try out the below  in your anaconda prompt

conda create -n <env_name>python=3.6

conda activate <env_name>

conda install pytorch-cpu torchvision-cpu -c pytorch

If you are  using a GPU

 conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

Please refer https://pytorch.org/


Instruction: How get a Python pathlib Path from an Azure blob datastore?

Input:
I am trying to do some custom manipulation of a torch.utils.data.DataLoader in AzureML but cannot get it to instantiate directly from my azureml.core.Datastore :

ws = Workspace( # ... etc ... )
ds = Datastore.get(ws, datastore_name='my_ds')
am = ds.as_mount()

# HOW DO I GET base_path, data_file from am?

dataloader = DataLoader(
        ListDataset(base_path, data_file), #... etc...
    )

The value of am.path() is "$AZUREML_DATAREFERENCE_my_ds" but I cannot figure out how to go from that to a pathlib.Path as is expected by the constructor to ListDataset. Things I've tried include Path(am.path()) and Path(os.environ[am.path()]) but they don't seem to work.

It's clear that there's some answer, since :

script_params = {
    '--base_path': ds.as_mount(),
    '--epochs': 30,
    '--batch_size' : 16,
    '--use_cuda': 'true'
}
torch = PyTorch(source_directory='./',
                        script_params=script_params,
                        compute_target=compute_target,
                        entry_script='train.py',
                        pip_packages=packages,
                        use_gpu=True)

seems to create a legit object.

Output:
You can perhaps try using the DataPath class. It exposes attributes such as path_on_datastore which might be the path you're looking for.

To construct this class from your DataReference object i.e. variable am; you can use  create_from_data_reference() method.

Example:

ds = Datastore.get(ws, datastore_name='my_ds')
am = ds.as_mount()

dp = DataPath().create_from_data_reference(am)

base_path = dp.path_on_datastore