Instruction: How to add hidden neurons to a Pytorch RNN

Input:
How can I add hidden neurons to a Recurrent Neural Network in pytorch? In my understanding, torch.nn.RNN has n neurons with inputs being the input and the hidden state, where n is equal to the size of the hidden state.
How can I add additional layers before the neurons go back to the hidden state? E.g. If i only have 1 input and 1 output, but want to be able to model more complex functions?
I tried using the num_layers parameter but this just adds more layers of single neurons. I also tried using torch.nn.Sequential to stack individual RNNs with different sized inputs/outputs but this didnt work as Sequential objects dont seem to pass through additional parameters (h0, the initial hidden state).
Im trying to model f(x)=sin(x) with the initial hidden state beign the initial value of the sinewave (sin(x_0)), the inputs being x and the outputs being sin(x).

Output:
You can't define rnn without defining hidden neurons.
Lets look at the official example:
class RNNTutorial(Module):
    def __init__(self, input_size, hidden_size,
                 output_size):
        super(RNNTutorial, self).__init__()
        self.hidden_size = hidden_size
        size_sum = input_size + hidden_size
        self.i2h = Linear(size_sum, hidden_size)
        self.i2o = Linear(size_sum, output_size)
        self.softmax = LogSoftmax(dim=1)

    def forward(self, input_, hidden_):
        combined = cat(tensors=(input_, hidden_), dim=1)
        hidden_ = self.i2h(input=combined)
        hidden_ = relu(hidden_)
        output = self.i2o(input=combined)
        output = self.softmax(input=output)
        return output, hidden_

    def init_hidden(self):
        return zeros(1, self.hidden_size)

Above is a two-layer RNN structure. On the 1st layer
self.i2h = Linear(size_sum, hidden_size)

The hidden neuron input size: size_sum and the output hidden_size
how to add neuron? You can change the parameter values.
For instance: size_sum + 1 now you add one more hidden neuron.


Instruction: when importing pytorch microsoft visual C++ Redistributable is not installed

Input:
I work in a windows machine with GPU.
I have installed pytorch in a conda environment with
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

then I run python and inside of python I do  import torch and I get this error
           Python 3.6.10 |Anaconda, Inc.| (default, May  7 2020, 19:46:08) [MSC v.1916 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.
It can be downloaded at https://aka.ms/vs/16/release/vc_redist.x64.exe
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
File "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\__init__.py", line 127, in <module>
raise err
OSError: [WinError 126] 指定されたモジュールが見つかりません。
Error loading "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\lib\asmjit.dll"
or one of its dependencies.
>>>

How can I correct this error?

Output:
Get the Microsoft Visual C++ Redistributable installer from the link in the error, in this case is this.
Run the installer and launch again your shell with conda configured when finished


Instruction: PyTorch tensors have same value after being added to a list

Input:
While learning gradients and optimizing the process by Pytorch, I wanted to figure out the change of loss function values vs weights values with the graph. While I tried to graph, I used both numpy and torch, because I wanted to compare. During to store list of grad. and loss function values.
It works at numpy:
def gradient(x,y, y_predicted):
  return np.dot(2*x, y_predicted-y).mean()

dw = gradient(x,y,y_pred)
dw_list.append(dw)

[-120.0, -112.8, -106.032, -99.67008, -93.68988, -88.06848, -82.78437, -77.81731, -73.14827, -68.75938]
It doesn't works at torch:
  for epoch in range(n_iters):
    # prediction = forward pass
    y_pred = forward(x)

    # loss
    l = loss(y, y_pred)
    loss_list.append(l)
    print('loss')
    print(l_list)
    # gradients = backward pass
    l.backward()  # calculate w.grad = dl/dw   and cumulative

    # update weights
    with torch.no_grad():
        w -= learning_rate * w.grad
    print(f'w.grad before zero setting = {w.grad}')
    dw_list.append(w.grad)
    print(dw_list)
    #print(f'w.grad before zero setting = {w.grad}')
    # zero gradients
    w.grad.zero_()

[tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485)]
Why dw_list.append(dw) works at numpy, but dw_list.append(w.grad) does not work at torch?
Why only the new value of grad had been filled up the whole array at each iter. at torch tensor?

Output:
w.grad is a tensor; it (the same tensor) is appended to the list at each iteration, so the list contains copies of the same tensor, not copies of its value at each point in time as you'd probably intend.
The standard way of handling this is to use:
dw_list.append(w.grad.detach().cpu().numpy())

Please have a look at this discussion for why detach() is necessary:
https://discuss.pytorch.org/t/should-it-really-be-necessary-to-do-var-detach-cpu-numpy/35489/6
By contrast, np.mean() returns a new python float object every time it is called, and so the values are different at the end.  The list append() is not doing anything different in the two cases.
P.S. I think this would also work:
dw_list.append(w.grad.clone())

however it would keep the cloned tensors in the graph (and on the gpu, if the originals were on the gpu).  That may or may not be what you want.


Instruction: Number of layers vs list(net.parameters())

Input:
New to convolutional neural nets so sorry if this doesn't make much sense. I have this code:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

and I think this has 5 layers. However, when I print len((list(net.parameters())) I get 10. Shouldn't this be a list of size 5 with parameters for each layer?

Output:
Quick answer : You get an extra parameter array for each layer containing the bias vector associated to the layer.
Detailled answer:
I will try to guide you in my process of investigating your questions.
It seems like a good idea to see what our 10 parameters are :
for param in net.parameters():
    print(type(param), param.size())

<class 'torch.nn.parameter.Parameter'> torch.Size([6, 3, 5, 5])
<class 'torch.nn.parameter.Parameter'> torch.Size([6])
<class 'torch.nn.parameter.Parameter'> torch.Size([16, 6, 5, 5])
<class 'torch.nn.parameter.Parameter'> torch.Size([16])
<class 'torch.nn.parameter.Parameter'> torch.Size([120, 400])
<class 'torch.nn.parameter.Parameter'> torch.Size([120])
<class 'torch.nn.parameter.Parameter'> torch.Size([84, 120])
<class 'torch.nn.parameter.Parameter'> torch.Size([84])
<class 'torch.nn.parameter.Parameter'> torch.Size([10, 84])
<class 'torch.nn.parameter.Parameter'> torch.Size([10])

We can recognize our 5 layers and an extra line for each layer. For instance, if we look at a specific layer for instance the first conv layer, we get :
for param in net.conv1.parameters():
    print(type(param), param.size())

<class 'torch.nn.parameter.Parameter'> torch.Size([6, 3, 5, 5])
<class 'torch.nn.parameter.Parameter'> torch.Size([6])

So now that we know we have two array of parameters per layer, the question is why. The first 6*3*5*5 array corresponds to your 6 kernels of size 5*5 with 3 channels, the second one corresponds the the bias associated to each of your kernel. Mathematically speaking, to compute the value at the next layer associated to a given kernel, you make the convolution between the area under your desired pixel and the kernel and you add a real number. That number is called the bias and it is empirically proven that using a bias gives better results.
Now you can also create a layer without bias, and then you will only get one parameter array :
layer = nn.Conv2d(3,6,5, bias= False)
for param in layer.parameters():
    print(type(param), param.size())

<class 'torch.nn.parameter.Parameter'> torch.Size([6, 3, 5, 5])


Instruction: Fine-Tuning DistilBertForSequenceClassification: Is not learning, why is loss not changing? Weights not updated?

Input:
I am relatively new to PyTorch and Huggingface-transformers and experimented with DistillBertForSequenceClassification on this Kaggle-Dataset.
from transformers import DistilBertForSequenceClassification
import torch.optim as optim
import torch.nn as nn
from transformers import get_linear_schedule_with_warmup

n_epochs = 5 # or whatever
batch_size = 32 # or whatever

bert_distil = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
#bert_distil.classifier = nn.Sequential(nn.Linear(in_features=768, out_features=1), nn.Sigmoid())
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(bert_distil.parameters(), lr=0.1)

X_train = []
Y_train = []

for row in train_df.iterrows():
    seq = tokenizer.encode(preprocess_text(row[1]['text']),  add_special_tokens=True, pad_to_max_length=True)
    X_train.append(torch.tensor(seq).unsqueeze(0))
    Y_train.append(torch.tensor([row[1]['target']]).unsqueeze(0))
X_train = torch.cat(X_train)
Y_train = torch.cat(Y_train)

running_loss = 0.0
bert_distil.cuda()
bert_distil.train(True)
for epoch in range(n_epochs):
    permutation = torch.randperm(len(X_train))
    j = 0
    for i in range(0,len(X_train), batch_size):
        optimizer.zero_grad()
        indices = permutation[i:i+batch_size]
        batch_x, batch_y = X_train[indices], Y_train[indices]
        batch_x.cuda()
        batch_y.cuda()
        outputs = bert_distil.forward(batch_x.cuda())
        loss = criterion(outputs[0],batch_y.squeeze().cuda())
        loss.requires_grad = True

        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        j+=1
        if j == 20:
            #print(outputs[0])
            print('[%d, %5d] running loss: %.3f loss: %.3f ' %
              (epoch + 1, i*1, running_loss / 20, loss.item()))
            running_loss = 0.0
            j = 0

[1,   608] running loss: 0.689 loss: 0.687
[1,  1248] running loss: 0.693 loss: 0.694
[1,  1888] running loss: 0.693 loss: 0.683
[1,  2528] running loss: 0.689 loss: 0.701
[1,  3168] running loss: 0.690 loss: 0.684
[1,  3808] running loss: 0.689 loss: 0.688
[1,  4448] running loss: 0.689 loss: 0.692  etc...

Regardless on what I tried, loss did never decrease, or even increase, nor did the prediction get better. It seems to me that I forgot something so that weights are actually not updated. Someone has an idea?
O
what I tried

Different loss functions

BCE
CrossEntropy
even MSE-loss

One-Hot Encoding vs A single neuron output
Different learning rates, and optimizers
I even changed all the targets to only one single label, but even then, the network did'nt converge.

Output:
Looking at running loss and minibatch loss is easily misleading. You should look at epoch loss, because the inputs are the same for every loss.
Besides, there are some problems in your code, fixing all of them and the behavior is as expected: the loss slowly decreases after each epoch, and it can also overfit to a small minibatch. Please look at the code, changes include: using model(x) instead of model.forward(x), cuda() only called once, smaller learning rate, etc.
Tuning and fine-tuning ML models are difficult work.
n_epochs = 5
batch_size = 1

bert_distil = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(bert_distil.parameters(), lr=1e-3)

X_train = []
Y_train = []
for row in train_df.iterrows():
    seq = tokenizer.encode(row[1]['text'],  add_special_tokens=True, pad_to_max_length=True)[:100]
    X_train.append(torch.tensor(seq).unsqueeze(0))
    Y_train.append(torch.tensor([row[1]['target']]))
X_train = torch.cat(X_train)
Y_train = torch.cat(Y_train)

running_loss = 0.0
bert_distil.cuda()
bert_distil.train(True)
for epoch in range(n_epochs):
    permutation = torch.randperm(len(X_train))
    for i in range(0,len(X_train), batch_size):
        optimizer.zero_grad()
        indices = permutation[i:i+batch_size]
        batch_x, batch_y = X_train[indices].cuda(), Y_train[indices].cuda()
        outputs = bert_distil(batch_x)
        loss = criterion(outputs[0], batch_y)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    print('[%d] epoch loss: %.3f' %
      (epoch + 1, running_loss / len(X_train) * batch_size))
    running_loss = 0.0

Output:
[1] epoch loss: 0.695
[2] epoch loss: 0.690
[3] epoch loss: 0.687
[4] epoch loss: 0.685
[5] epoch loss: 0.684


Instruction: RuntimeError: DataLoader worker (pid 27351) is killed by signal: Killed

Input:
I'm running the data loader below which applies a filter to a microscopy image prior to training. In order to count the red and green. This code filters the red cells. Since I have applied this to the code I keep on getting the error message above. I have tried increasing the memory allocation to the maximum allowance possible but that didn't help. Is there a way I could modify the filter so it isn't causing this issue, please? Many thanks in advance
import os

import numpy as np
import torch
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms, utils
#from torchvision.transforms import Grayscalei
import pandas as pd
import pdb
import cv2

class CellsDataset(Dataset):
    # a very simple dataset

    def __init__(self, root_dir, transform=None, return_filenames=False):
        self.root = root_dir
        self.transform = transform
        self.return_filenames = return_filenames
        self.files = [os.path.join(self.root,filename) for filename in os.listdir(self.root)]
        self.files = [path for path in self.files
                      if os.path.isfile(path) and os.path.splitext(path)[1]=='.png']

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        path = self.files[idx]
        image = cv2.imread(path)
        sample = image.copy()
        # set blue and green channels to 0
        sample[:, :, 0] = 0
        sample[:, :, 1] = 0

channel.
        if self.transform:
            sample = self.transform(sample)

        if self.return_filenames:
            return sample, path
        else:
            return sample

Output:
Similar problem I met before.
One possible solution is to disable cv2 multi-processing by
def __getitem__(self, idx):
    import cv2
    cv2.setNumThreads(0)
    # ...

in your dataloader. It might be because the cv2 multi-processing is conflict with torch's DataLoader with multi-processing. Although it does not work for me since mine does not involve OpenCV. But indeed you might first try this.

For my occasion, it's worth mentioning that torch's multiprocessing with CUDA access must use spawn or forkserver instead of fork to spawn new process. To do this, you should
if __name__ == '__main__':
    import multiprocessing
    multiprocessing.set_start_method('spawn')
    # ... **The all rest code**

Notice you shall put all remaining code in the if __name__ == '__main__' block, including the imports, because it might be other imports (like cv2) setting it to fork and your loaders does not work.


Instruction: How can I cross-validate by Pytorch and Optuna

Input:
I want to use cross-validation against the official Optuna and pytorch-based sample code (https://github.com/optuna/optuna/blob/master/examples/pytorch_simple.py).
I thought about splitting the data for cross-validation and trying parameter tuning for each fold, but it seems that the average accuracy of each parameter cannot be obtained because the parameters that can be checked in study.trials_dataframe() are different each time.

Output:
I think we need to evaluate all folds and calculate the mean inside an objective function. I create an example notebook, so please take a look.
In the notebook, I slightly modified the objective function to pass the dataset with the arguments and added a wrapper function objective_cv to call the objective function with the split dataset. Then, I optimized the objective_cv instead of the objective function.
def objective(trial, train_loader, valid_loader):

    # Remove the following line.
    # train_loader, valid_loader = get_mnist()

    ...

    return accuracy

def objective_cv(trial):

    # Get the MNIST dataset.
    dataset = datasets.MNIST(DIR, train=True, download=True, transform=transforms.ToTensor())

    fold = KFold(n_splits=3, shuffle=True, random_state=0)
    scores = []
    for fold_idx, (train_idx, valid_idx) in enumerate(fold.split(range(len(dataset)))):
        train_data = torch.utils.data.Subset(dataset, train_idx)
        valid_data = torch.utils.data.Subset(dataset, valid_idx)

        train_loader = torch.utils.data.DataLoader(
            train_data,
            batch_size=BATCHSIZE,
            shuffle=True,
        )
        valid_loader = torch.utils.data.DataLoader(
            valid_data,
            batch_size=BATCHSIZE,
            shuffle=True,
        )

        accuracy = objective(trial, train_loader, valid_loader)
        scores.append(accuracy)
    return np.mean(scores)

study = optuna.create_study(direction="maximize")
study.optimize(objective_cv, n_trials=20, timeout=600)


Instruction: Torch sum subsets of tensor

Input:
if the tensor is of shape [20, 5] then I need to take 10 at a time and sum them, so result is [2,5].
eg:
shape[20,5] -> shape[2, 5] (sum 10 at a time)
shape[100, 20] -> shape[10,20] (sum 10 at a time)
Is there any faster/optimal way to do this?
eg:
[[1, 1], [1, 2], [3, 4], [1,2]] i want [[2, 3], [4, 6]] by taking sum of 2 rows.

Output:
It is not completely clear, but I cannot use a comment for this, so.
For the first case you have:
t1 = torch.tensor([[1., 1.], [1., 2.], [3., 4.], [1.,2.]])
t1.shape #=> torch.Size([4, 2])
t1
tensor([[1., 1.],
        [1., 2.],
        [3., 4.],
        [1., 2.]])

To get the desired output you should reshape:
tr1 = t1.reshape([2, 2, 2])
res1 = torch.sum(tr1, axis = 1)
res1.shape #=> torch.Size([2, 2])
res1
tensor([[2., 3.],
        [4., 6.]])

Let's take a tensor with all one elements (torch.ones) for the second case.
t2 = torch.ones((20, 5))
t2.shape #=> torch.Size([20, 5])
t2
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

So, reshaping to get the required (?) result:
tr2 = tensor.reshape((10, 2, 5))
res2 = torch.sum(tr2, axis = 0)
res2.shape #=> torch.Size([2, 5])
res2
tensor([[10., 10., 10., 10., 10.],
        [10., 10., 10., 10., 10.]])

Is this what you are looking for?


Instruction: NameError: name 'base' is not defined, while running open AI gym in GOOGLE COLAB

Input:
I am going through the tutorial DQN reinforcement learning in Pytorch.org,https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html
But here when I am trying to render a screen and display using python display, I am getting name base not found. Can anyone help me here? If you want to clear any clarity about the question, I am here
Thanks in advance

Output:
Use the gpu on colab. If with that isn't enought execute this cell at the beginning
!apt install xvfb -y
!pip install pyvirtualdisplay
!pip install piglet

from pyvirtualdisplay import Display
display = Display(visible=0, size=(1400, 900))
display.start()


Instruction: Unable to create custom dataset and dataloader using torchtext

Input:
I have questions regarding building custom dataset and iterator using torchtext. I used the following code found in this post and modified based on my case:
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
text_field = Field(sequential=True, eos_token="[CLS]", tokenize=tokenizer)
label_field = Field(sequential=False, use_vocab=False)
data_fields = [("file", None),
               ("text", text_field),
               ("label", label_field)]
train, val = train_test_split(input_dt, test_size=0.1)
train.to_csv("train_output_path", index=False)
val.to_csv("val_output_path", index=False)
train, val = TabularDataset(path="path", train="train.csv", validation="val.csv",
                            format="csv", skip_header=True, fields=data_fields)

When it comes to text_field.build_vocab(train), I got this error: TypeError: '<' not supported between instances of 'list' and 'int'.
The only difference between my code and the post is the pre-trained word embeddings. In the post, the author used glove, which I use XLNetTokenizer from transformers package. I also searched for other posts who used the similar method, but they all used the pre-trained word embeddings, therefore they did have such an issue.
Does anyone know how to fix this issue? Many thanks!

Output:
I think as you are using a predefined tokenizer you dont't need to build vocab, instead you can follow this steps. Showing an example of how to do it using BERT tokenizer.
Sentences: it is a list of of text data
lables: is the label associated
###tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

# Tokenize all of the sentences and map the tokens to thier word IDs.

input_ids = []
attention_masks = []

# For every sentence...

for sent in sentences:
    # `encode_plus` will:
    #   (1) Tokenize the sentence.
    #   (2) Prepend the `[CLS]` token to the start.
    #   (3) Append the `[SEP]` token to the end.
    #   (4) Map tokens to their IDs.
    #   (5) Pad or truncate the sentence to `max_length`
    #   (6) Create attention masks for [PAD] tokens.
    encoded_dict = tokenizer.encode_plus(
                        sent,                      # Sentence to encode.
                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                        max_length = 100,           # Pad & truncate all sentences.
                        pad_to_max_length = True,
                        return_attention_mask = True,   # Construct attn. masks.
                        return_tensors = 'pt',     # Return pytorch tensors.
                   )

    # Add the encoded sentence to the list.
    input_ids.append(encoded_dict['input_ids'])

    # And its attention mask (simply differentiates padding from non-padding).
    attention_masks.append(encoded_dict['attention_mask'])

# Convert the lists into tensors.
input_ids = torch.cat(input_ids, dim=0)
attention_masks = torch.cat(attention_masks, dim=0)
labels = torch.tensor(labels)

# Print sentence 0, now as a list of IDs.
print('Original: ', sentences[0])
print('Token IDs:', input_ids[0])

### Not combine the input id , mask and labels and divide the dataset

#:
from torch.utils.data import TensorDataset, random_split

# Combine the training inputs into a TensorDataset.
dataset = TensorDataset(input_ids, attention_masks, labels)

# Create a 90-10 train-validation split.

# Calculate the number of samples to include in each set.
train_size = int(0.90 * len(dataset))
val_size = len(dataset) - train_size

# Divide the dataset by randomly selecting samples.
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])

print('{:>5,} training samples'.format(train_size))
print('{:>5,} validation samples'.format(val_size))

### Not you call loader of these datasets

from torch.utils.data import DataLoader, RandomSampler, SequentialSampler

# The DataLoader needs to know our batch size for training, so we specify it
# here. For fine-tuning BERT on a specific task, the authors recommend a batch
# size of 16 or 32.
batch_size = 32

# Create the DataLoaders for our training and validation sets.
# We'll take training samples in random order.
train_dataloader = DataLoader(
            train_dataset,  # The training samples.
            sampler = RandomSampler(train_dataset), # Select batches randomly
            batch_size = batch_size # Trains with this batch size.
        )

# For validation the order doesn't matter, so we'll just read them sequentially.
validation_dataloader = DataLoader(
            val_dataset, # The validation samples.
            sampler = SequentialSampler(val_dataset), # Pull out batches sequentially.
            batch_size = batch_size # Evaluate with this batch size.
        )


Instruction: Why cant I use ONNX Runtime training with pytorch?

Input:
When I run
from onnxruntime.capi.ort_trainer import ORTTrainer
as stated at https://github.com/microsoft/onnxruntime/#training-start, I get this error:
ModuleNotFoundError: No module named 'onnxruntime.capi.ort_trainer'
What can I do to fix this?
I have onnxruntime installed via pip but I couldn't even find "ort_trainer.py" in [python path]/site_packages/onnx-runtime/capi

Output:
You should build ort training from source.
https://www.onnxruntime.ai/docs/how-to/build.html#training


Instruction: Pytorch and data augmentation: how to augmentate data with blur, rotations, etc

Input:
I want to do some data augmentation with Pytorch, but i don't know the libraries very well:
I tried this:
def gaussian_blur(img):
    image = np.array(img)
    image_blur = cv2.GaussianBlur(image,(65,65),10)
    new_image = image_blur
    im = Image.fromarray(new_image)
    return im

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomRotation([-8,+8]),
        transforms.Lambda(gaussian_blur),
        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),
        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.RandomRotation([-8,+8]),
        transforms.Lambda(gaussian_blur),
        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),
        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Because the effects i want to do are: gaussian blur/rotation/contrast/gamma+random noise
But i have errors considering several aspects, like the size of the images doesn't match.
Any suggestions?

Output:
If you want to apply Gausian blur there is also already a pytorch class for:
torchvision.transforms.GaussianBlur(kernel_size, sigma=(0.1, 2.0))


Instruction: CUDA out of memory runtime error, anyway to delete pytorch "reserved memory"

Input:
Like many othersm I'm getting a Runtime error of Cuda out of memory, but for some reason pytorch has reserved a large amount of it.
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 6.00 GiB total capacity; 4.31 GiB already allocated; 844.80 KiB free; 4.71 GiB reserved in total by PyTorch)
I've tried the torch.cuda.empy_cache(), but this isn't working either and none of the other CUDA out of memory posts have helped me either.
When I've checked my gpu usage(nvidia-smi) before running my python program, it is plenty free.

Output:
From the given description it seems that the problem is not allocated memory by Pytorch so far before the execution but cuda ran out of memory while allocating the data that means the 4.31GB got already allocated (not cached) but failed to allocate the 2MB last block.
Possible solution already worked for me, is to decrease the batch size, hope that helps!


Instruction: Overlap and add tensor in pytorch using nn.fold

Input:
I have the following problem, I have tensor with shape (1,16198), first i would like to dive it into chunks so I used unfold like:
my_tensor.unfold(-1, 178, 89) -> tensor with shape (1, 182, 178)

which means 182 overlapping chunks of size 178, that's perfect.
But now I would like to undo the operation adding the overlapping chunks and get back a (1,16198) tensor. I believe that the fold method would do that, but I've burned through several hours trying to understand how it works to no progress at all and I cant seem to find a good explanation source for how to use it and I am afraid the official documentation is way to complex for my understanding.

Output:
Got it to work last night, i had to transpose to the (1, 182, 178) tensor to (1, 178, 182) then I used
nn.Fold(my_tensor, (1, 16198), kernel_size=(1,178), stride=(1,89))

and i get
back my tensor with overlapping sections added, thus completing the overlap and add algorithm, still not sure how nn.Fold works.


Instruction: Difference between hidden dimension and n_layers in rnn using pytorch

Input: I am stuck between hidden dimension and n_layers. What I understood so far, is that n_layers in the parameters of RNN using pytorch, is number of hidden layers. If n_layers represents the number if hidden layers than what is hidden dimension?

Output:
Actually documentation is really clear about their differences. Hidden size is number of features of the hidden state for RNN. So if you increase hidden size then you compute bigger feature as hidden state output.
However, num_layers is just multiple RNN units which contain hidden states with given hidden size.

num_layers=2 would mean stacking two RNNs together to form a stacked
RNN, with the second RNN taking in outputs of the first RNN and
computing the final results

I want to explain how RNN works with the image below. Each RNN unit (blue rectangles) takes one h_n (hidden state) and one input. Hidden dimension determines the feature vector size of the h_n (hidden state). At each timestep (t, horizontal propagation in the image) your rnn will take a h_n and input. Then if you have n_layers >1 it will create a intermediate output and give it to the upper layer(vertical). So hidden dimension determine the size of horizontal h_n in the image, whereas num_layers determine the number of blue cells in vertical axis in the image.


Instruction: Loading enormous custom dataset using IterableDataset

Input:
I have a huge dataset with features (input_id,input_mask,segment_id,label_id) saved in batches of 64 in a pickle file. I read this file, create a TensorDataset and pass to dataloader for training. Since the features file is too big to create the full TensorDataset, I want to convert TensorDataset to IterableDataset so that one batch of samples can be retrieved from the features file at a time and passed on to the dataloader. But while training, I get the following error:
TypeError: iter() returned non-iterator of type 'TensorDataset'
Following is the custom dataset class I wrote:
class MyDataset(IterableDataset):

    def __init__(self,args):
        self.args=args

    def get_features(self,filename):
        with open(filename, "rb") as f:
            while True:
                try:
                    yield pickle.load(f)
                except EOFError:
                    break

    def process(self,args):
        if args.cached_features_file:
            cached_features_file = args.cached_features_file

        if os.path.exists(cached_features_file):
            features=self.get_features(cached_features_file)

        feat = next (features)
        li=list(feat)
        all_input_ids=torch.tensor([f.input_ids for f in li ], dtype=torch.long)
        all_input_mask= torch.tensor([f.input_mask for f in li ], dtype=torch.long)
        all_segment_ids= torch.tensor([f.segment_ids for f in li], dtype=torch.long)
        all_label_ids = torch.tensor([f.label_id for f in li ], dtype=torch.long)

        dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
        return dataset

    def __iter__(self):
        dataset=self.process(self.args)
        return dataset

And I use it like this:
train_dataset=MyDataset(args)
train_dataloader = DataLoader(train_dataset, batch_size=args.train_batch_size)

I understand that TensorDataset is map-style requiring index while IterableDataset is iterable-style which is the reason for error. Even if I return a list/tuple of feature tensors instead of TensorDataset, I get similar error. Can someone please tell me how to load a batched dataset in the correct way with IterableDataset?

Output:
I solved the problem by saving the dataset in a different manner. I saved the features as dictionary objects incrementally pickled in a pickle file and just read them off one at a time and pass on to dataloader for processing. Batching is done automatically by the dataloader. This is how the custom class looks now:
class MyDataset(IterableDataset):

    def __init__(self,filename):

        self.filename=filename
        super().__init__()

    def process(self,filename):
        with open(filename, "rb") as f:
            while True:
                try:
                    yield pickle.load(f)
                except EOFError:
                    break

    def __iter__(self):
        dataset=self.process(self.filename)
        return dataset


Instruction: Index multidimensional torch tensor with array of variable length

Input:
I have a list of indices and a tensor with shape:
shape = [batch_size, d_0, d_1, ..., d_k]
idx = [i_0, i_1, ..., i_k]

Is there a way to efficiently index the tensor on each dim d_0, ..., d_k with the indices i_0, ..., i_k? (k is available only at run time)
The result should be:
tensor[:, i_0, i_1, ..., i_k] #tensor.shape = [batch_size]

At the moment I'm creating a tuple of slices, one for each dimension:
idx = (slice(tensor.shape[0]),) + tuple(slice(i, i+1) for i in idx)
tensor[idx]

but I would prefer something like:
tensor[:, *idx]

Example:
a = torch.randint(0,10,[3,3,3,3])
indexes = torch.LongTensor([1,1,1])

I would like to index only the last len(indexes) dimensions like:
a[:, indexes[0], indexes[1], indexes[2]]

but in the general case where I don't know how long indexes is.

Note: this answer does not help since it indexes all the dimensions, and does not work for a proper subset!

Output:
Unfortunately you can't provide1 a mix of slices and iterators to an indexing (e.g. a[:,*idx]). However, you can achieve almost the same thing by wrapping it in brackets to cast to an iterator:
a[(slice(None), *idx)]

In Python, x[(exp1, exp2, ..., expN)] is equivalent to x[exp1, exp2, ..., expN]; the latter is just syntactic sugar for the former.

https://numpy.org/doc/stable/reference/arrays.indexing.html


Instruction: trying to import png images to torchvision

Input:
I am attempting to import images for use with torch and torchvision. But I am receiving this error:
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 81, in default_collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

Based on this post, I am converting them to Tensor:
https://discuss.pytorch.org/t/typeerror-default-collate-batch-must-contain-tensors-numpy-arrays-numbers-dicts-or-lists-found-class-imageio-core-util-array/62667
Here is my code:
import torch
import torchvision
import torchvision.transforms
from torchvision import datasets, transforms

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.ToTensor()
])

dataset = torchvision.datasets.ImageFolder('datasets')

dataloader = torch.utils.data.DataLoader(dataset,
                                          batch_size=16,
                                          shuffle=True,
                                          num_workers=12)

tensor_dataset = []

for i, data in enumerate(dataloader, 0):
    Tensor = torch.tensor(data)
    tensor_dataset.append(Tensor.flatten)

The first last part is from https://github.com/TerragonDE/PyTorch but I have had no success.  The data I am trying to load is from here:
http://www.cvlibs.net/datasets/kitti/
How can I solve this?
UPDATE:
Thanks @trialNerror, but now I am getting this error:
ValueError                                Traceback (most recent call last)
<ipython-input-6-aa72392b67e8> in <module>
      1 for i, data in enumerate(dataloader, 0):
----> 2     Tensor = torch.tensor(data)
      3     tensor_dataset.append(Tensor.flatten)

ValueError: only one element tensors can be converted to Python scalars

This is what I have found so far but am not sure how to apply it:
https://discuss.pytorch.org/t/pytorch-autograd-grad-only-one-element-tensors-can-be-converted-to-python-scalars/56681
UPDATE 2:
The reason why I didn't end up using the dataloader is because I end up getting this error:
num_epochs = 10
loss_values = list()

for epoch in range(1, num_epochs):
    for i, data in enumerate(train_array, 0):
        outputs = model(data.unsqueeze(0))
        loss = criterion(outputs,data.unsqueeze(0))

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print('Epoch - %d, loss - %0.5f '%(epoch, loss.item()))
        loss_values.append(loss.item())

torch.Size([1, 16, 198, 660])
torch.Size([1, 32, 97, 328])
torch.Size([1, 1018112])

RuntimeError                              Traceback (most recent call last)
<ipython-input-106-5e6fa86df079> in <module>
      4 for epoch in range(1, num_epochs):
      5     for i, data in enumerate(train_array, 0):
----> 6         outputs = model(data.unsqueeze(0))
      7         loss = criterion(outputs,data.unsqueeze(0))
      8

c:\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-90-467a3f84a03f> in forward(self, x)
     29         print(out.shape)
     30
---> 31         out = self.fc(out)
     32         print(out.shape)
     33

c:\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\python38\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

c:\python38\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1608     if input.dim() == 2 and bias is not None:
   1609         # fused op is marginally faster
-> 1610         ret = torch.addmm(bias, input, weight.t())
   1611     else:
   1612         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [1 x 1018112], m2: [512 x 10] at C:\w\b\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:41

I realize that if you have m1: [a * b] and m2: [c * d] then b and c have to be the same value, but I am not sure, what is the best way to resize my images?

Output:
Please note that I had wanted to automatically load all PNG images in a directory as pytorch tensors.  I had previously looked at posts like this (and many other web pages):
Loading a huge dataset batch-wise to train pytorch
But instead I ended up using this where I am using Image.open one by one on all the images in the directory instead of the torch DataLoader:
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt

image = Image.open("datasets/image_02/data/my_image.png").convert('RGB')

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms as tf

transforms = tf.Compose([tf.Resize(400),
                        tf.ToTensor()])

img_tensor = transforms(image)


Instruction: How to transform labels in pytorch to onehot

Input:
How to give target_transform a function for changing the labels to onehot encoding?
For example, the MNIST dataset in torchvision:
train_dataset = torchvision.datasets.MNIST(root='./mnist_data/',
                                           train=True,
                                           download=True,
                                           transform=train_transform,
                                           target_transform=<????>)

Tried F.onehot() but it didn't work.

Output:
This is how I implemented it. Not sure if there's a cleaner way.
train_dataset = torchvision.datasets.MNIST(root='./data/', train=True,
                                 transform=torchvision.transforms.ToTensor(),
                                 target_transform=torchvision.transforms.Compose([
                                 lambda x:torch.LongTensor([x]), # or just torch.tensor
                                 lambda x:F.one_hot(x,10)]),
                                 download=True)

It needs to be an index tensor? i.e. int64

Can't use torchvision.ToTensor because it's not an image

Also torch.LongTensor and torch.tensor behave differently with int input

Need to provide number of classes


Instruction: Cant install pytorch - ModuleNotFoundError: No module named 'tools.nnwrap'

Input:
I cannot install PyTorch on python 3.7. This error occurs for others but the suggested fixes did not work. I have tried instaling wheel, and importing tools but neither work.
Got error: ModuleNotFoundError: No module named 'tools.nnwrap'
ran command pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html
full output: Looking in links: https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html Collecting torch Using cached torch-0.1.2.post2.tar.gz (128 kB) Collecting torchvision Using cached torchvision-0.2.2.post3-py2.py3-none-any.whl (64 kB) Requirement already satisfied: pyyaml in c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\lib\site-packages (from torch) (5.3.1) Collecting pillow>=4.1.1 Using cached Pillow-7.2.0-cp37-cp37m-win32.whl (1.8 MB) Collecting numpy Using cached numpy-1.19.1-cp37-cp37m-win32.whl (10.9 MB) Collecting six Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Using legacy 'setup.py install' for torch, since package 'wheel' is not installed. Installing collected packages: torch, pillow, numpy, six, torchvision Running setup.py install for torch ... error ERROR: Command errored out with exit status 1: command: 'c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[ 0] = '"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup .py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --reco rd 'C:\Users\LV7682\AppData\Local\Temp\pip-record-z5_fy7cc\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\lv7682\documents\repos\security _patch\spms\webapp\securitypatchmanagementsystem\venv\venv\include\site\python3.7\torch' cwd: C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\ Complete output (23 lines): running install running build_deps Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 265, in <module> description="Tensors and Dynamic neural networks in Python with strong GPU acceleration", File "c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\lib\site-packages\setuptools\__init__.py", line 163, in setup return distutils.core.setup(**attrs) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\core.py", line 148, in setup dist.run_commands() File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 966, in run_commands self.run_command(cmd) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 99, in run self.run_command('build_deps') File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 51, in run from tools.nnwrap import generate_wrappers as generate_nn_wrappers ModuleNotFoundError: No module named 'tools.nnwrap' ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\scripts\python.exe' -u -c 'import sys , setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\p ip-install-4b80gt33\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\LV7682\AppData\Local\Temp\pip-record-z5_fy7cc\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\user s\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\include\site\python3.7\torch' Check the logs for full command output.

Output: Fixed: needed to use python 64-bit not 32-bit


Instruction: pytorch dataset map-style vs iterable-style

Input:
A map-style dataset in Pytorch has the __getitem__() and __len__() and iterable-style datasets has __iter__() protocol. If we use map-style, we can access the data with dataset[idx] which is great, however with the iterable dataset we can't.
My question is why this distinction was necessary? What makes the data random read so expensive or even improbable?

Output:
I wrote a short post on how to use PyTorch datasets, and the difference between map-style and iterable-style dataset.
In essence, you should use map-style datasets when possible. Map-style datasets give you their size ahead of time, are easier to shuffle, and allow for easy parallel loading.
It’s a common misconception that if your data doesn’t fit in memory, you have to use iterable-style dataset. That is not true. You can implement a map-style dataset such that it retrives data as needed.
Check out the full post here.


Instruction: pytorch code sudden fails on colab with NVIDIA driver on your system is too old

Input:
I had some code which worked on colab (gpu runtime) just a short while ago. Suddenly I am getting
The NVIDIA driver on your system is too old (found version 10010).
nvcc shows
Cuda compilation tools, release 10.1, V10.1.243
I tried torch versions 1.5.1, then 1.13.0. Both keep getting this error.
There is a discussion showing other people having doubts. with no clear resolution.
https://github.com/pytorch/pytorch/issues/27738
Anyone having the same problem?

Output:
The light-the-torch package is designed to solve exactly this type of issue. Try this:
!pip install light-the-torch
!ltt install torch torchvision


Instruction: will loop decrease the utilization of the GPU?

Input:
In PyTorch, I have a loop in my DeepLearning Pipeline's forward part to normalize the intermediate result.
Will it run on CPU and decrease the utilization of the GPU?
some snippet as follow:
def forward(self):
    ...
    for b in range(batch_size):
        self.points[b] = self.unit_cube(self.points[b])
    ....

Output:
In Pytorch, whether an operation is done on the GPU or CPU is decided by where the data is. One of the main selling points of Pytorch is that you don't (usually) have to care where the data is; the interface is the same.
If the tensor data is on the GPU, then the operation is done on the GPU. If it's on the CPU, then the operation is done on the CPU. How you choose to organise those operations (ifs, for loops, etc) have no impact on it.
>>> import torch
>>> a = torch.randn(3,4,5)
>>> b = a.cuda()
>>> a.device
device(type='cpu')
>>> b.device
device(type='cuda', index=0)
>>> c = b
>>> for x in range(10):
...   c = c * 2
...
>>> c.device
device(type='cuda', index=0)

In the above example, I used a for loop to double b 10 times, storing the result in c. This was all done on the GPU, and I equally could have done this on a, making it happen on the CPU.


Instruction: torch.no_grad() affects on model accuracy

Input:
I am getting an error "CUDA out of memory" then i add torch.no_grad() function into my code. Is it affect on my accuracy?
for iters in range(args.iterations):

with torch.no_grad():
    encoded, encoder_h_1, encoder_h_2, encoder_h_3 = encoder(
    res, encoder_h_1, encoder_h_2, encoder_h_3)

with torch.no_grad():
    code = binarizer(encoded)

with torch.no_grad():
    output, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4 = decoder(
    code, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4)

res = res - output.detach()
codes.append(code.data.cpu().numpy())
torch.cuda.empty_cache()
print('Iter: {:02d}; Loss: {:.06f}'.format(iters, res.data.abs().mean()))

Output:
torch.no_grad() just disables the tracking of any calculations required to later calculate a gradient.
It won't have any effect on accuracy in a pure inference mode, since gradients are not needed there. Of course you can't use it during training time since we need the gradients to train and optimize.
In general if you go for inference you always want to set the network to eval mode and disable gradients. This saves run time and memory consumption and won't affect accuracy.
Answer to a similar questions, explaining eval() and no_grad() https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615/2


Instruction: Why is the accuracy difference so much when I use the image data set and pytorch's own data set directly?

Input:
For example, for the cifar10 data set, directly using the data set that comes with pytorch, the accuracy rate can reach 96% under the same network structure, but after I converted cifar10 into a picture, I tested it and the accuracy rate was only 92%. why?
This is the previous code：
train_dataset = dset.CIFAR10(args.data_path, train=True, transform=train_transform, download=True)
test_dataset = dset.CIFAR10(args.data_path, train=False, transform=test_transform, download=True)

This is the modified code：
train_dataset = datasets.ImageFolder(root='/home/ubuntu/bigdisk/DataSets/cifar10/static/orig/train/',
                                         transform=train_transform
                                         )
test_dataset = datasets.ImageFolder(root='/home/ubuntu/bigdisk/DataSets/cifar10/static/orig/test/',
                                        transform=test_transform
                                        )

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True,
                                               num_workers=args.prefetch, pin_memory=True)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=args.test_bs, shuffle=False,
                                              num_workers=args.prefetch, pin_memory=True)

Output:
If the downloaded dataset, hyperparameters(such as batch size or learning rate), dataset transformation, etc were equal, I think it is because the randomness.
Your dataloader shuffles the dataset randomly. The shuffled dataset will always be different every time you shuffle it, which might have led to the accuracy difference.
Also, the model will be initialized with different values each time. (Unless you have used some initialization method that always initializes the model with the same values.)
You could check https://pytorch.org/docs/stable/notes/randomness.html for more information.


Instruction: Google Cloud Function Build timeout - all requirements have been loaded

Input:
I have the following code on my cloud function -
import os
import numpy as np
import requests

import torch
from torch import nn
from torch.nn import functional as F
import math
from torch.nn import BCEWithLogitsLoss
from torch.utils.data import TensorDataset
from transformers import AdamW, XLNetTokenizer, XLNetModel, XLNetLMHeadModel, XLNetConfig
from keras.preprocessing.sequence import pad_sequences
import numpy as np
import pandas as pd

def polarization(request):

   MODEL_URL = 'https://polarization.s3-us-west-1.amazonaws.com/classifier_state_dict.pt'
   print(MODEL_URL)
   r = requests.get(MODEL_URL)
   print(r)

   #Cloud function vm is a read only s/m. The only writable place is the tmp folder
   file = open("/tmp/model.pth", "wb")
   file.write(r.content)
   file.close()

   print("Wrote to the tmp file")
   # State dict requires model object
   model = XLNetForPolarizationClassification(num_labels=1)
   model.load_state_dict(torch.load('/tmp/model.pth'))

   # Tokenize the embedded article
   embeddedArticle = request["embeddedArticle"]
   tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)
   textIds = tokenize_inputs(embeddedArticle, tokenizer, num_embeddings=250)

   # Generate the attention masks and padding
   masks = create_attn_masks(textIds)
   article = pd.DataFrame()
   article["features"] = textIds.tolist()
   article["masks"] = masks

   # Call generate_predictions
   pred = generate_predictions(model, article, 1)

   return pred

## Extracting parameter and returning prediction
def generate_predictions(model, df, num_labels, device="cpu"):

   model.eval()

   X = df_subset["features"].values.tolist()
   masks = df_subset["masks"].values.tolist()

   X = torch.tensor(X)
   masks = torch.tensor(masks, dtype=torch.long)
   with torch.no_grad():
       # Run the model with the input_ids and attention_masks separately
       logits = model(input_ids=X, attention_mask=masks)
       # Get the logits for each class
       logits = logits.sigmoid().detach().cpu().numpy()

   return round(logits)

class XLNetForPolarizationClassification(torch.nn.Module):

   def __init__(self, num_labels=2):
       super(XLNetForPolarizationClassification, self).__init__()
       self.num_labels = num_labels
       self.xlnet = XLNetModel.from_pretrained('xlnet-base-cased')
       self.classifier = torch.nn.Linear(768, 1)
       torch.nn.init.xavier_normal_(self.classifier.weight)

   def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):

       last_hidden_state = self.xlnet(input_ids=input_ids,
                                       attention_mask=attention_mask,
                                       token_type_ids=token_type_ids)

       mean_last_hidden_state = self.pool_hidden_state(last_hidden_state)
       logits = self.classifier(mean_last_hidden_state)

       # If you know the labels, compute the loss otherwise
       if labels is not None:
           loss_fct = BCEWithLogitsLoss()
           loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
           return loss
       else:
           return logits

   def pool_hidden_state(self, last_hidden_state):
       last_hidden_state = last_hidden_state[0]
       mean_last_hidden_state = torch.mean(last_hidden_state, 1)
       return mean_last_hidden_state

def create_attn_masks(input_ids):
 """
 This will set a 1 or 0 based on if it is a mask or an actual input it for the word
 """
 attention_masks = []

 for seq in input_ids:
   seq_mask = [float(i>0) for i in seq]
   attention_masks.append(seq_mask)
 return attention_masks

def tokenize_inputs(text, tokenizer, num_embeddings=250):

   # tokenize the text, then truncate sequence to the desired length minus 2 for
   # the 2 special characters
   tokenized_texts = list(map(lambda t: tokenizer.tokenize(t)[:num_embeddings-2], text))

   # convert tokenized text into numeric ids for the appropriate LM
   input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]

   # append special token "<s>" and </s> to end of sentence
   input_ids = [tokenizer.build_inputs_with_special_tokens(x) for x in input_ids]

   # pad sequences
   input_ids = pad_sequences(input_ids, maxlen=num_embeddings, dtype="long", truncating="post", padding="post")
   return input_ids

and requirements.txt
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
cycler==0.10.0
filelock==3.0.12
future==0.18.2
h5py==2.10.0
idna==2.10
joblib==0.16.0
Keras==2.4.3
kiwisolver==1.2.0
matplotlib==3.3.0
numpy==1.19.1
packaging==20.4
Pillow==7.2.0
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.3.1
regex==2020.7.14
requests==2.24.0
sacremoses==0.0.43
scipy==1.5.2
sentencepiece==0.1.91
six==1.15.0
tokenizers==0.8.1rc1
torch==1.6.0
tqdm==4.48.2
transformers==3.0.2
urllib3==1.25.10

However when I deploy, it gives me build timed out. The logs don't show any errors and show that each of the dependencies in the requirements.txt file have built.  The first print statement doesn't get logged either. I don't see what part of the model/ requirements is causing the time out issue.
Here's a screenshot of the logs - there are no errors all the logs are info until theres a Conext deadline exceeded statement. I can't share the actual logs without giving access to the function I believe.  I've set the timeout to 9 minutes (540 seconds)1

Output:
It might be due to the torch import, as this causes you to import the PyTorch lib that contains CUDA and hence requires a GPU, which isn't available on Cloud Functions.
Instead, you can use a direct link to the cpu only version in your requirements.txt, like this:
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
cycler==0.10.0
filelock==3.0.12
future==0.18.2
h5py==2.10.0
idna==2.10
joblib==0.16.0
Keras==2.4.3
kiwisolver==1.2.0
matplotlib==3.3.0
numpy==1.19.1
packaging==20.4
Pillow==7.2.0
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.3.1
regex==2020.7.14
requests==2.24.0
sacremoses==0.0.43
scipy==1.5.2
sentencepiece==0.1.91
six==1.15.0
tokenizers==0.8.1rc1
https://download.pytorch.org/whl/cpu/torch-1.6.0%2Bcpu-cp37-cp37m-linux_x86_64.whl
tqdm==4.48.2
transformers==3.0.2
urllib3==1.25.10

See this answer about how to select a different PyTorch version.


Instruction: Max function in PyTorch versus in NumPy

Input:
I am developing an algorithm that involves a CPU case, allowing for NumPy, and a GPU case, allowing for PyTorch. The object will almost always be 4D. Two versions of the object are as follows.
B = [
    [[[0.5000, 0.5625],
      [0.5000, 0.5625]],

     [[1.2500, 0.5000],
      [0.5625, 0.6875]],

     [[0.5625, 0.6250],
      [0.5000, 0.5625]]]
]

B_array = np.array(B)

B_tensor = torch.Tensor(B)

I want to take the max of each 2D matrix, such that I get a result of:
max_array_fn(B_array)   # returns array([0.5625, 1.250, 0.6250])
max_tensor_fn(B_tensor) # returns tensor([0.5625, 1.250, 0.6250])

Part of the solution was discussed here, but this is only for NumPy on CPU:
Max of each 2D matrix in 4D NumPy array
However, on the GPU, it seems that PyTorch does not use the same convention as NumPy.
If this was defined as a NumPy array, we could solve the problem using np.array(B, axis=(0,2,3)). The usage of axis in PyTorch similar to the NumPy example is not supported, as suggested here:
PyTorch torch.max over multiple dimensions
Is there an alternative vectorization of the method? Why is it only able to be vectorized on the CPU with NumPy and not on GPU with PyTorch?
The correct solution should not use any loops and ideally one function call.

Output:
Not the most elegant way:
B.max(dim=3)[0].max(dim=2)[0].max(dim=0)[0]


Instruction: Pytorch rename labels of a trained model

Input:
I have a trained model. During training there was some problem with string characters. So i converted my labels into numbers like:
red : 0
blue: 1
green: 2
Now is it possible to rename my label back to actual label names.
Hours of training. Would be  helpful if anyone has an idea.
Train and validate the model
for epoch in range(1, epoch_num + 1):
    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)
    loss_val, acc_val = validate(val_loader, model, criterion, epoch)
    total_loss_val.append(loss_val)
    total_acc_val.append(acc_val)

Test Single Image:
def eval_image(file_path):
    model = torch.load(file_path)
    model.eval()
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    X = Image.open('red.jpeg')
    test_transforms = transforms.Compose([transforms.ToTensor()])
    image_tensor = test_transforms(X).float()
    image_tensor = image_tensor.unsqueeze_(0)
    input = Variable(image_tensor)
    input = input.to(device)
    output = model(input)
    index = output.data.cpu().numpy().argmax()
    print(index)

Once the training is done, The evaluation script cannot be made generic. I will have to pass following code
idx_to_class = {
        0: "red",
        1: "blue",
        2: "green",
    }
    class_name = idx_to_class[index]

But I do not want to pass the above code. As my evaluation script needs to generic.

Output:
Use a dictionary for it!
labels = [0, 1, 2, 1, 0, 2, 1]

dct = {0: "red", 1: "blue", 2: "green"}

renamed_labels = [dct[x] for x in labels]

renamed_labels ## ["red", "blue", "green", "blue", "red", "green", "blue"]


Instruction: Summing over specific indices PyTorch (similar to scatter_add)

Input:
I have some matrix where rows belong to some label, unordered. I want to sum all rows for each label.
Here is how it can be done with a loop:
labels = torch.tensor([0, 1, 0])
x = torch.tensor([[1, 2, 3],[4, 5, 6],[7, 8, 9]])
torch.stack([torch.sum(x[labels == i], dim=0) for i in torch.unique(labels)])

desired output:
tensor([[ 8, 10, 12],
        [ 4,  5,  6]])

EDIT: Just to make it clear, I have the labels tensor, I know which labels repeat, I am interested in computing the final line without the use of a loop. I was thinking scatter_add_ or gather might help.

Output:
Just use torch.index_add function.
labels = torch.tensor([0, 1, 0])
x = torch.tensor([[1, 2, 3],[4, 5, 6],[7, 8, 9]])

nrow = torch.unique(labels).size(0)
ncol = x.size(1)
out = torch.zeros((nrow, ncol), dtype=x.dtype)
out.index_add_(0, labels, x)
print(out)

# the output will be
# tensor([[ 8, 10, 12],
#         [ 4,  5,  6]])


Instruction: Creating a criterion that measures the F1 Loss

Input:
I am currently creating criterion to measure the MSE loss function using:
loss_fcn = torch.nn.MSELoss()

loss = loss_fcn(logits[getMaskForBatch(subgraph)], labels.float())

Now I need to change it to F1 score but I cannot seem to find one library that could be used for it

Output:
In particular, depending on the task you need to have a specific loss function.
Loss function also known as objective, cost or error function is somehow opposite to the optimization function. Loss function creates the loss, optimization function reduces the loss. :). These two functions should live in equilibrium so we don't overfit.
PyTorch Regression losses:

nn.L1Loss L1 Loss (MAE)
nn.MSELoss L2 Loss (MSE)
nn.SmoothL1Loss Huber

PyTorch Classification losses:

nn.CrossEntropyLoss
nn.KLDivLoss
nn.NLLLoss

PyTorch GAN training

nn.MarginRankingLoss

So if you used nn.MSELoss you probably need to stay with regression, because F1  is a classification metric.
If you really need F1 score for some other reason, you may use scikit learn.


Instruction: torch.jit.script(module) vs @torch.jit.script decorator

Input:
Why is adding the decorator "@torch.jit.script" results in an error, while I can call torch.jit.script on that module, e.g. this fails:
import torch

@torch.jit.script
class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

"C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\torch\jit\__init__.py", line 1262, in script
    raise RuntimeError("Type '{}' cannot be compiled since it inherits"
RuntimeError: Type '<class '__main__.MyCell'>' cannot be compiled since it inherits from nn.Module, pass an instance instead

While the following code works well:
class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

This question is also featured on PyTorch forums.

Output:
Reason for your error is here, this bulletpoint precisely:

No support for inheritance or any other polymorphism strategy, except
for inheriting from object to specify a new-style class.

Also, as stated at the top:

TorchScript class support is experimental. Currently it is best suited
for simple record-like types (think a NamedTuple with methods
attached).

Currently, it's purpose is for simple Python classes (see other points in the link I've provided) and functions, see link I've provided for more information.
You can also check torch.jit.script source code to get a better grasp of how it works.
From what it seems, when you pass an instance, all attributes which should be preserved are recursively parsed (source). You can follow this function along (quite commented, but too long for an answer, see here), though exact reason why this is the case (and why it was designed this way) is beyond my knowledge (so hopefully someone with expertise in torch.jit's inner workings will speak more about it).


Instruction: Pytorch: How to train a network with two loss functions?

Input:
I want to pretrain a network with reconstruction loss first, then finetune it by crossentropy loss. But it seems that I have to define two network in this two stage. How to achieve it?
class Net():
    def __init__(self,pretrain):
        self.pretrain = pretrain
    def encoder(self,x):
        # do something here
        return x
    def decoder(self,x):
        # do something here
        return x

    def forward(self):
        e_x = self.encoder(x)
        if self.pretrain:
            return decoder(e_x)
        else:
            return e_x

def train(x,y):
    pretrain = True
    if pretrain:
        network = Net(pretrain=True)
        output = network(x)
        loss = MSE(x,output)
     else:
        network = Net(pretrain=False)
        output = network(x)
        loss = crossentropy(output,y)
    loss.backward()

Output:
You can achieve this by simply defining the two-loss functions and loss.backward will be good to go. See the relevant discussion here
MSE = torch.nn.MSELoss()
crossentropy = torch.nn.CrossEntropyLoss()

def train(x,y):
        pretrain = True
        if pretrain:
            network = Net(pretrain=True)
            output = network(x)
            loss = MSE(x,output)
        else:
            network = Net(pretrain=False)
            output = network(x)
            loss = crossentropy(output,y)
        loss.backward()


Instruction: The use of meshgrid in pytorch/numpy

Input:
Below code is a snippet taken from https://blog.paperspace.com/how-to-implement-a-yolo-v3-object-detector-from-scratch-in-pytorch-part-3/, and I am confused as to what it is trying to achieve.
    grid = np.arange(grid_size)
    a,b = np.meshgrid(grid, grid)

    x_offset = torch.FloatTensor(a).view(-1,1)
    y_offset = torch.FloatTensor(b).view(-1,1)

    if CUDA:
        x_offset = x_offset.cuda()
        y_offset = y_offset.cuda()

    x_y_offset = torch.cat((x_offset, y_offset), 1).repeat(1,num_anchors).view(-1,2).unsqueeze(0)

I tried the case when grid_size = 3, and it outputed:
tensor([[0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.],
        [0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.],
        [0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.]])

I cannot quite see what is the pattern here. According to the description in the given link I think I should really expect something like:
tensor([[0,0],
        [0,0],
        [0,0],
        [0,1],
        [0,1],
        [0,1],
        [0,2],
        [0,2],
        ...]])

Output:
If you are expecting the second output you show, simply change
x_y_offset = (
    torch.cat((x_offset, y_offset), 1).repeat(1, num_anchors).view(-1, 2).unsqueeze(0)
)

to
x_y_offset = (
    torch.cat((y_offset, x_offset), 1).repeat(1, num_anchors).view(-1, 2).unsqueeze(0)
)

It just has to do with the ordering of the output of meshgrid.


Instruction: Three dimensional autoencoder has low MSE loss but gives noisy reconstruction image

Input:
I am re-constructing spatio-temporal cuboids of 3-dimensional size with width and height equal to 32 and depth equal to 20. I am using Conv3d layers in my autoencoder architecture.
So my input shape is 32x32x20, which I am reducing to size 2048, and then reconstructing it back to 32x32x20.
The MSE loss of the model has nice convergence even though the reconstruction is just noise.
My encoder architecture:
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv3d-1       [-1, 64, 20, 32, 32]           5,248
         AvgPool3d-2       [-1, 64, 10, 16, 16]               0
            Conv3d-3      [-1, 128, 10, 16, 16]         221,312
         AvgPool3d-4         [-1, 128, 5, 8, 8]               0
            Conv3d-5         [-1, 256, 5, 8, 8]         884,992
         AvgPool3d-6         [-1, 256, 2, 4, 4]               0
            Conv3d-7         [-1, 512, 2, 4, 4]       3,539,456
         AvgPool3d-8         [-1, 512, 1, 2, 2]               0

My decoder architecture:
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
       Interpolate-1         [-1, 512, 2, 4, 4]               0
            Conv3d-2         [-1, 256, 4, 4, 4]       3,539,200
       Interpolate-3         [-1, 256, 8, 8, 8]               0
            Conv3d-4        [-1, 128, 10, 8, 8]         884,864
       Interpolate-5      [-1, 128, 20, 16, 16]               0
            Conv3d-6       [-1, 64, 20, 16, 16]         221,248
       Interpolate-7       [-1, 64, 40, 32, 32]               0
            Conv3d-8        [-1, 3, 40, 32, 32]           5,187
            Conv3d-9        [-1, 3, 20, 32, 32]             246

My reconstruction code:
from mpl_toolkits.axes_grid1 import ImageGrid
# recon_batch is the last batch of the autoencoder output.
# recon_batch has shape (batch_size, 3, 20, 32, 32)
recon_batch = recon_batch.permute(0, 2, 3, 4, 1) # new shape = (batch_size, 20, 32, 32, 3)
recon_batch = recon_batch.detach().cpu()
recon_batch_numpy = recon_batch.detach().cpu().numpy()
for k in range(3):  # K=3 because I want to display 3 frames
    fig = plt.figure(figsize=(4., 4.))
    grid = ImageGrid(fig, 111,
                      nrows_ncols=(4, 4),
                      axes_pad=0.1,
                      )
    images = [recon_batch_numpy[i][k] for i in range(16)]  # 16 because Original Image is of size (128x128), when I make 32x32 patches, 16 sub frames are formed
    for ax, im in zip(grid, images):
        ax.imshow((rgb2gray(im) * 255).astype(np.uint8), cmap='gray', vmin=0, vmax=255)
    name = str(num_epochs) + 'th_figure_' + str(k)
    plt.savefig(name)

Original input image(actual cuboids of size 32x32x20, the image is reconstructed by taking the first frame for depth 0 of each such cuboid):

The reconstructed output image:

If needed, I can also add a loss plot, the loss starts around 10,000 and converges around 200.

Output:
I'm not sure what your loss term or training process looks like, but you may want to consider a couple of things:

training for more epochs since 200 may be a relatively large MSE (and you may be stuck in a local minima for which case a higher gradient or different optimizer/optimizer hyper-parameters may help)

changing you loss term by using a different loss or by adding additional terms to the loss; such as, KL-divergence


Instruction: pytorch model saved from TPU run on CPU

Input:
I found interesting model - question generator, but can't run it. I got an error:
Traceback (most recent call last):
  File "qg.py", line 5, in <module>
    model = AutoModelWithLMHead.from_pretrained("/home/user/ml-experiments/gamesgen/t5-base-finetuned-question-generation-ap/")
  File "/home/user/.virtualenvs/hugging/lib/python3.7/site-packages/transformers/modeling_auto.py", line 806, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/user/.virtualenvs/hugging/lib/python3.7/site-packages/transformers/modeling_utils.py", line 798, in from_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'

I briefly googled and found that "torch_xla" is a something that is used to train pytorch model on TPU. But I would like to run it localy on cpu (for inference, of course) and got this error when pytorch tried to load tpu-bound tensors.
How can I fix it?
this is model, which I tried: https://huggingface.co/mrm8488/t5-base-finetuned-question-generation-ap

Output: As @cronoik suggested, I have installed transformers library form github. I clonned latest version, and executed python3 setup.py install in it's directory. This bug was fixed, but fix still not released in python's packets repository.


Instruction: Using pytorch Cuda on MacBook Pro

Input:
I am using MacBook Pro (16-inch, 2019, macOS 10.15.5 (19F96))
GPU

AMD Radeon Pro 5300M
Intel UHD Graphics 630

I am trying to use Pytorch with Cuda on my mac.
All of the guides I saw assume that i have Nvidia graphic card.
I found this: https://github.com/pytorch/pytorch/issues/10657 issue, but it looks like I need to install ROCm, and according to their Supported Operating Systems, it only supports Linux.
Is it possible to run Pytorch on GPU using mac and AMD Graphic card?

Output:
No.
CUDA works only with supported NVidia GPUs, not with AMD GPUs.
There is an ongoing effort to support acceleration for AMD GPUs with PyTorch (via ROCm, which does not work on MacOS).


Instruction: AssersionError: Torch not compiled with CUDA enabled

Input:
I want to run this repo. I installed everything that is needed for this project.
I have Windows 8.1 operating system, seems that I don't have NVIDIA GPU (from Device Manager: Display adapters - AMD Radeon HD 7660G + 7670M Dual Graphics and AMD Radeon HD 7670M).
I installed torch with command that is presented on Pytorch web-site
pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

But when I run the project I receive error - AssersionError: Tourch not compiled with CUDA enabled.
Then I tried to install torch with CUDA enabled.
pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

But when I run the project I receive error - AssersionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from https://www.nvidia.com/Download/index.aspx.
Please, help me to solve my issue and run the project without errors.

Output:
I have already fixed this issue. There was a problem in source code where they use
opt.device

in main.py and violin_dataset.py. But this was declared as
opt.device = torch.device('cuda:0')

in config.py even if you didn't have cuda support.
So I changed it to
opt.device = torch.device('cpu')

And everything works fine now.


Instruction: Use SageMaker Pytorch image for training

Input:
I am trying to containerize the training process for a fine tuned BERT model and run it on SageMaker. I was planning to use the pre-built SageMaker Pytorch GPU containers (https://aws.amazon.com/releasenotes/available-deep-learning-containers-images/) as my starting point but I am having issues pulling the images during my build process.
My Dockerfile looks like this:
# SageMaker PyTorch image
FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.0-gpu-py36-cu101-ubuntu16.04

ENV PATH="/opt/ml/code:${PATH}"

# /opt/ml and all subdirectories are utilized by SageMaker, we use the /code subdirectory to store our user code.
COPY /bert /opt/ml/code

# this environment variable is used by the SageMaker PyTorch container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code

# this environment variable is used by the SageMaker PyTorch container to determine our program entry point
# for training and serving.
# For more information: https://github.com/aws/sagemaker-pytorch-container
ENV SAGEMAKER_PROGRAM bert/train

My build_and_push script:
#!/usr/bin/env bash

# This script shows how to build the Docker image and push it to ECR to be ready for use
# by SageMaker.

# The argument to this script is the image name. This will be used as the image on the local
# machine and combined with the account and region to form the repository name for ECR.
IMAGE="my-bert"

# parameters
PY_VERSION="py36"

# Get the account number associated with the current IAM credentials
account=$(aws sts get-caller-identity --query Account --output text)

if [ $? -ne 0 ]
then
    exit 255
fi

chmod +x bert/train

# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=$(aws configure get region)
region=${region:-us-east-2}

# If the repository doesn't exist in ECR, create it.
aws ecr describe-repositories --repository-names ${IMAGE} || aws ecr create-repository --repository-name ${IMAGE}

echo "---> repository done.."
# Get the login command from ECR and execute it directly
aws ecr get-login-password --region $region | docker login --username AWS --password-stdin $account.dkr.ecr.$region.amazonaws.com
echo "---> logged in to account ecr.."

# Get the login command from ECR in order to pull down the SageMaker PyTorch image
# aws ecr get-login-password --region $region | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com
# echo "---> logged in to pytorch ecr.."

echo "Building image with arch=gpu, region=${region}"
TAG="gpu-${PY_VERSION}"
FULLNAME="${account}.dkr.ecr.${region}.amazonaws.com/${IMAGE}:${TAG}"
docker build -t ${IMAGE}:${TAG} --build-arg ARCH="$arch" -f "Dockerfile" .
docker tag ${IMAGE}:${TAG} ${FULLNAME}
docker push ${FULLNAME}

I get the following message during the push and the sagemaker pytorch image is not pulled:
Get https://763104351884.dkr.ecr.us-east-1.amazonaws.com/v2/pytorch-training/manifests/1.5.0-gpu-py36-cu101-ubuntu16.04: no basic auth credentials

Please let me know if this is the correct way to use a pre-built SageMaker image and what I could do to fix this error.

Output:
You should run something like this, before running the docker build:
aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin 763104351884.dkr.ecr.${region}.amazonaws.com


Instruction: How to know whether the data passed to GPU will cause CUDA out of memory or not

Input:
I am using GPU to run some very large deep learning models, when I choose a batch size of 8, it can fit into the memory, but if I use a batch size of 16, it will cause CUDA out-of-memory error, and I have to kill the process.
My question is, before actually passing the data into GPU, is there a way that I could know how large the data will occupy in the GPU?
For example, the following code is about how I create a pytorch dataloader and pass each batch of the dataloader to the GPU, could I know how large it is before I call batch.to(device)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)
for step, batch in enumerate(train_dataloader):
    b_input_ids = batch[0].to(device)
    b_input_mask = batch[1].to(device)
    b_labels = batch[2].to(device)

Output:
I would recommend using the torchsummary package here.
pip install torchsummary

and in use
from torchsummary import summary
myModel.cuda()
summary(myModel, (shapeOfInput)) # where shapeOfInput is a tuple of the sample's dimensions

This will give you the size of the model, the size of the forward pass, and the size of the backpass in MB for a batch size of 1, and you can then multiple out by your batch size.


Instruction: Runtime error while executing code from google colab document for creating Deepfakes image animation

Input:
enter image description hereI'm getting runtime error while executing code from google colab document for creating Deepfakes image animation.
RuntimeError                              Traceback (most recent call last)

<ipython-input-5-dbd18151b569> in <module>()
  1 from demo import load_checkpoints
  2 generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml',

----> 3                             checkpoint_path='/content/gdrive/My Drive/first-order-motion-model/vox-cpk.pth.tar')
10 frames
/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py in _lazy_init()
188             raise AssertionError(

189                 "libcudart functions unavailable. It looks like you have a broken build?")

--> 190         torch._C._cuda_init()
191         # Some of the queued calls may reentrantly call _lazy_init();

192         # we need to just return without initializing in that case.

RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47

Output:
You may be running on a colab environment that only has TPU's available and not GPU's in which case you need to utilize XLA with PyTorch. You might find this notebook and repository very helpful if this is the case:
https://colab.research.google.com/github/pytorch/xla/blob/master/contrib/colab/resnet18-training.ipynb
https://github.com/pytorch/xla


Instruction: What is the difference between `.zero_grad()` and `.zero_grad`?

Input: I am working on neural network and I find that, with *.grad_zero() I get loss function values properly and also converge to zero. Where, with *.grad_zero (with out bracket) gives loss function values in 5 digits. (13,564.23). So, what is the difference between them? Why "()" important in FPP. Thank you.

Output: optimizer.zero_grad is a function, so you need to call it with parentheses. If you don't use the parentheses, you are just referencing the function object but never calling it.


Instruction: Proper Usage of PyTorch's non_blocking=True for Data Prefetching

Input:
I am looking into prefetching data into the GPU from the CPU when the model is being trained on the GPU. Overlapping CPU-to-GPU data transfer with GPU model training appears to require both

Transferring data to GPU using data = data.cuda(non_blocking=True)
Pin data to CPU memory using train_loader = DataLoader(..., pin_memory=True)

However, I cannot understand how non-blocking transfer is being performed in this official PyTorch example, specifically this code block:
for i, (images, target) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)

        if args.gpu is not None:
            images = images.cuda(args.gpu, non_blocking=True)
        if torch.cuda.is_available():
            target = target.cuda(args.gpu, non_blocking=True)

        # compute output
        output = model(images)
        loss = criterion(output, target)

Won't images.cuda(non_blocking=True) and target.cuda(non_blocking=True) have to be completed before output = model(images) is executed. Since this is a synchronization point, images must be first fully transferred to the CUDA device, so the data transfer steps are effectively no longer non-blocking.
Since output = model(images) is blocking, images.cuda() and target.cuda() in the next i iteration of the for loop will not occur until the model output is computed, meaning no prefetching in the next loop iteration.
If this is correct, what is the correct way to perform data prefetching to the GPU?

Output:
I think where you are off is that output = model(images) is a synchronization point. It seems the computation is handled by a different part of a GPU. Quote from official PyTorch docs:

Also, once you pin a tensor or storage, you can use asynchronous GPU
copies. Just pass an additional non_blocking=True argument to a
to() or a cuda() call. This can be used to overlap data
transfers with computation.


Instruction: What is the right calculation of epoch loss in training?

Input:
I am reading Pytorch official tutorial for fine tuning and I am faced with one problem and that is calculation of loss in each epoch.
Before this , I calculate loss for batch of data, accumulate these batch losses and find mean of these values as loss of epoch. But in that example, the calculation is as follow:
for inputs, labels in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)

my question is in this line running_loss += loss.item() * inputs.size(0). It is multiply loss value of batch in bach size. What is the true way to calculate loss of epoch?
and what is the unit of loss? What is the range of loss value?

Output:
Yes the code snippet adds multiplication of batch size with batch mean error. If you want to calculate true summation. You can use
torch.nn.CrossEntropyLoss(reduction = "sum")

which will give you the sum of errors for the batch. Then you can directly sum for each batch as follows:
running_loss += loss.item()

The range of the loss value depends on your number of classes and feature vector. The code in your question will have same running_loss if you use reduction="sum" because your code basically makes
(loss/batch_size) * batch_size

which is the same thing with loss value. However, backpropagation changes because on the one hand you backprop according to the sum of losses, on the other hand you calculate backprop according to the mean loss.


Instruction: AttributeError: module 'torch.utils' has no attribute 'tensorboard'

Input:
I tried to use tensorboard in torch.utils, but it says "module 'torch.utils' has no attribute 'tensorboard'".
My torch version is "1.6.0+cu101"
PS C:\Users\kelekelekle> python
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 01:54:44) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
1.6.0+cu101
>>> writer = torch.utils.tensorboard.SummaryWriter()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch.utils' has no attribute 'tensorboard'
>>>

Output:
You have to install tensorboard via:
pip install tensorboard

(or a-like). Given that is done, you should import tensorboard module from torch.utils package:
from torch.utils import tensorboard

tensorboard.SummaryWriter("foo")

Or you can import SummaryWriter directly:
from torch.utils.tensorboard import SummaryWriter

SummaryWriter("bar")


Instruction: Vectorized way to apply a 3-dimension mask to RGB in pytorch

Input:
I have a HxWx3 tensor representing an RGB image and a HxWx3 mask (boolean) tensor as input.
It is assumed that for each (i,j) in the mask tensor there's exactly one true value (that is exactly one of R\G\B is on).
I want to apply the mask to the image to result in a HxW (or HxWx1) tensor V where V[i,j]='the matching R\G\B value according to the mask'.
Using Problem applying binary mask to an RGB image with numpy I was able to achieve the following:
>>> X*mask
tensor([[[ 9., 10.],
         [ 0.,  0.]],

        [[ 0.,  0.],
         [ 0., 20.]],

        [[ 0.,  0.],
         [30.,  0.]]])

But as stated, I want a single dim HxW and not HxWx3 as result.
Illustration:

Output:
Assuming that for each i,j only a single R/G/B value is retained, you can simply do:
(X*mask).sum(axis=2)

This should give you your desired (HxW) output.


Instruction: TypeError: reshape(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

Input:
I am a high school student who doesn't having much experience in using PyTorch and LIME. I'm having a lot of trouble with my image shape. Initially my image shape was (3,224,224), however the LIME algorithm only works with images that are in this shape(...,...,3). As a result, I tried transposing the image earlier. It seemed that I made some more progress by doing that, however, now I am getting a different error. Here is some of my code to understand what I have been doing before the error came up.
def get_preprocess_transform():
transf = transforms.Compose([
  #  transforms.ToPILImage(), #had to convert image to PIL as error was showing up two cells below about needing it in pil
    transforms.Resize(input_size),
    transforms.CenterCrop(input_size),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),

])

return transf

preprocess_transform = get_preprocess_transform() ## use your data_transform but in a method version

def batch_predict(image):
    model_ft.eval()
    batch = torch.reshape(image,(1,3,224,224))
    print(type(batch))

    logits = model_ft(batch)
    probs = F.softmax(logits, dim=1)
    return probs.detach().cpu().numpy()

print(img_t.shape)
img_t = torch.reshape(img_t,(1,3,224,224))
test_pred = batch_predict(img_t)
test_pred.squeeze().argmax()

img_t = np.ones((3, 224, 224))
np.transpose(img_t, (2,1,0)).shape

img_x = np.transpose(img_t, (2, 1, 0))
print(img_x.shape)

from lime import lime_image

explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(img_x, ## pass your image, do not transform
                                        batch_predict, # classification function
                                        top_labels=5,
                                        hide_color=0,
                                        num_samples=1000)

Here is the error message that comes from the "explainer cell"

Output:
use this command to convert numpy.ndarray to tensor
img = torch.from_numpy(img).float() #use appropriate name of variable


Instruction: How can torchaudio.transform.Resample be called without __call__ function inside?

Input:
if sample_rate != sr:
        waveform = torchaudio.transforms.Resample(sample_rate, sr)(waveform)
        sample_rate = sr

I was wondering how this Resamle works in there. So took a look at the docs of torchaudio. I thought there would be __call__ function. Because Resample is used as a function. I mean that Resample()(waveform). But inside, there are only __init__ and forward function. I think the forward function is the working function but I don't know why it is named 'forward' not __call__.  What am I missing?
class Resample(torch.nn.Module):
    r"""Resample a signal from one frequency to another. A resampling method can be given.

    Args:
        orig_freq (float, optional): The original frequency of the signal. (Default: ``16000``)
        new_freq (float, optional): The desired frequency. (Default: ``16000``)
        resampling_method (str, optional): The resampling method. (Default: ``'sinc_interpolation'``)
    """

    def __init__(self,
                 orig_freq: int = 16000,
                 new_freq: int = 16000,
                 resampling_method: str = 'sinc_interpolation') -> None:
        super(Resample, self).__init__()
        self.orig_freq = orig_freq
        self.new_freq = new_freq
        self.resampling_method = resampling_method

    def forward(self, waveform: Tensor) -> Tensor:
        r"""
        Args:
            waveform (Tensor): Tensor of audio of dimension (..., time).

        Returns:
            Tensor: Output signal of dimension (..., time).
        """
        if self.resampling_method == 'sinc_interpolation':

            # pack batch
            shape = waveform.size()
            waveform = waveform.view(-1, shape[-1])

            waveform = kaldi.resample_waveform(waveform, self.orig_freq, self.new_freq)

            # unpack batch
            waveform = waveform.view(shape[:-1] + waveform.shape[-1:])

            return waveform

        raise ValueError('Invalid resampling method: %s' % (self.resampling_method))

--edit--
I looked around torch.nn.module. There is no def __call__. But only
__call__ : Callable[..., Any] = _call_impl Would it be the way?

Output:
Here's simple similar demonstrates of how forward function works in PyTorch.
Check this:
from typing import Callable, Any

class parent:
    def _unimplemented_forward(self, *input):
        raise NotImplementedError

    def _call_impl(self, *args):
        # original nn.Module _call_impl function contains lot more code
        # to handle exceptions, to handle hooks and for other purposes
        self.forward(*args)

    forward : Callable[..., Any]  = _unimplemented_forward
    __call__ : Callable[..., Any] = _call_impl

class child(parent):
    def forward(self, *args):
        print('forward function')

class child_2(parent):
    pass

Runtime:
>>> c1 = child_1()
>>> c1()
forward function
>>> c2 = child_2()
>>> c2()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".\callable.py", line 8, in _call_impl
    self.forward(*args)
  File ".\callable.py", line 5, in _unimplemented_forward
    raise NotImplementedError
NotImplementedError


Instruction: Export pytorch model parameters into separate files according to layer hierarchy

Input:
Is it possible to export the trained parameters of a Pytorch model into separate binary files (float32/64, not text) under a folder hierarchy reflecting the layers defined by the model's architecture?
I wish to examine a sizeable pretrained model without the framework overhead and also split the checkpoint into manageable chunks.

Output:
There is no direct way to do this, but it should take only a few lines of code. For example, consider I have a model of the following structure:
class ConvBlock(nn.Module):
    def __init__(self, C_in, C_out, kernel, pool):
        super().__init__()
        self.conv = nn.Conv2d(C_in, C_out, kernel)
        self.relu = nn.ReLU(inplace = True)
        self.pool = nn.MaxPool2d(2,2) if pool else nn.Identity()

    def forward(self, input):
        out = self.conv(input)
        out = self.relu(out)
        out = self.pool(out)
        return out

class LeNet5(nn.Module):
    def __init__(self):
        super().__init__()
        self.block1 = ConvBlock(1, 6, 5, pool = True)
        self.block2 = ConvBlock(6, 16, 5, pool = True)
        self.block3 = ConvBlock(16, 120, 5, pool = False)
        self.fc = nn.Sequential(
            nn.Linear(120, 84),
            nn.ReLU(inplace = True),
            nn.Linear(84, 10)
        )

    def forward(self, input):
        out = self.block1(input)
        out = self.block2(out)
        out = self.block3(out)
        out = out.view(-1,120)
        out = self.fc(out)
        return out

To binarize individual parameters, all you have to do is iterate through them.
net = LeNet5()

basedir = 'lenet_params'

for name, param in net.named_parameters():
    name = name.split('.')
    out_dir, filename = os.path.join(basedir, *name[:-1]), name[-1]+'.pth'
    out_path = os.path.join(out_dir, filename)

    if not os.path.exists(out_dir):
        os.makedirs(out_dir, exist_ok=True)
    torch.save(param, out_path)

This will produce the directory structure below:
lenet_params
|---block1
|   |---conv
|   |   |---weight.pth
|   |   |---bias.pth
|---block2
|   |---conv
|   |   |---weight.pth
|   |   |---bias.pth
|---block3
|   |---conv
|   |   |---weight.pth
|   |   |---bias.pth
|---fc
|   |---0
|   |   |---weight.pth
|   |   |---bias.pth
|   |---2
|   |   |---weight.pth
|   |   |---bias.pth