Instruction: Explanation behind the following Pytorch results

Input:
I am trying to get a deeper understanding of how Pytorch's autograd works. I am unable to explain the following results:
import torch
def fn(a):
 b = torch.tensor(5,dtype=torch.float32,requires_grad=True)
 return a*b

a  = torch.tensor(10,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)

The output is tensor(5.). But my question is that the variable b is created within the function and  so should be removed from memory after the function returns a*b right? So when I call backward how is the value of b still present for allowing this computation?
As far as I understand each operation in Pytorch has a context variable which tracks "which" tensor to use for backward computation and there are also versions present in each tensor, and if the version changes then backward should raise an error right?
Now when I try to run the following code,
import torch
def fn(a):
 b = a**2
 for i in range(5):
   b *= b
 return b

a  = torch.tensor(10,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)

I get the following error: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of MulBackward0, is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
But if I run the following code, there is no error:
import torch
def fn(a):
  b = a**2
  for i in range(2):
    b = b*b
  return b

def fn2(a):
  b = a**2
  c = a**2
  for i in range(2):
    c *= b
  return c

a  = torch.tensor(5,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)
output2 = fn2(a)
output2.backward()
print(a.grad)

The output for this is :
tensor(625000.)
tensor(643750.)
So for a standard computation graphs with quite a few variables, in the same function, I am able to understand how the computation graph works. But when there is a variable changing before the call of backward function, I am having a lot of trouble understanding the results. Can someone explain?

Output:
Please note that b *=b is not same as b = b*b.
It is perhaps confusing, but the underlying operations vary.
In case of b *=b, an in-place operation takes place which messes up with the gradients and hence the RuntimeError.
In case of b = b*b, two tensor objects gets multiplied and the resulting object is assigned the name b. Thus no RuntimeError when you run this way.
Here is a SO question on the underlying python operation: The difference between x += y and x = x + y
Now what is the difference between fn in first case and fn2 in the second case? The operation c*=b does not destroy the graph links to b from c. The operation c*=c would make it impossible to have a graph connecting two tensors via an operation.
Well, I cannot work with tensors to showcase that because they raise RuntimeError. So I'll try with python list.
>>> x = [1,2]
>>> y = [3]
>>> id(x), id(y)
(140192646516680, 140192646927112)
>>>
>>> x += y
>>> x, y
([1, 2, 3], [3])
>>> id(x), id(y)
(140192646516680, 140192646927112)

Notice that there is no new object created. So it is not possible to trace from the output to initial variables. We cannot distinguish the object_140192646516680 to be an output or an input. So how does one create a graph with that..
Consider the following alternate case:
>>> a = [1,2]
>>> b = [3]
>>>
>>> id(a), id(b)
(140192666168008, 140192666168264)
>>>
>>> a = a + b
>>> a, b
([1, 2, 3], [3])
>>> id(a), id(b)
(140192666168328, 140192666168264)
>>>

Notice that the new list a is in fact a new object with id 140192666168328. Here we can trace that the object_140192666168328 came from the addition operation between two other objects object_140192666168008 and object_140192666168264. Thus a graph can be dynamically created and gradients can be propagated back from output to previous layers.


Instruction: Can I extract all indices that correspond to a certain key in a pytorch tensor?

Input: Say I have a pytorch tensor tensor([3,5,7,3,9,3,0]). I'd like to extract the indices where 3 appears, i.e. tensor([0,3,5]). Is there a built-in function for this?

Output:
There is a dedicated function for this:
   torch.where(my_tensor == the_number)


Instruction: RuntimeError: _th_exp_out not supported on CUDAType for Long

Input:
Questions and Help
I'm trying to finetune a transformer model for question answering, and am using the BCEWithLogitsLoss function.
However when I try calculating loss i get this error:

RuntimeError: _th_exp_out not supported on CUDAType for Long

I'm inputting matrices with the shape and dtype of [16, 2] and Long
Bud sadly I've no idea to solve this. I tried using another dtype (int32, float32, double), which didn't work.
Here is the code:
def loss_fn(self, preds, labels):
    return torch.nn.BCEWithLogitsLoss()(preds, labels)

def train_fn(self, dataloader, model, optimizer, device):
    # Some other stuff here
    pred = model(
            token_ids = token_ids,
            attention_mask = attention_mask,
            token_type_ids = token_type_ids)
    start_scores = torch.argmax(pred[0], dim=1)
    end_scores = torch.argmax(pred[1], dim=1)

    pred = torch.tensor(list(zip(start_scores, end_scores)))
    pred = pred.to(device, dtype=torch.long)

    batch_loss = self.loss_fn(pred, label)

Output: Passing both, pred and label as float worked in my case.


Instruction: pretrained fasterrcnn_resnet50_fpn: "forward() takes 2 positional arguments but 5 were given" on single image test

Input:
I am receving an error when executing the code below:
import torch
import torch.nn as nn
import torchvision.transforms as transforms
from torchvision.models.detection import fasterrcnn_resnet50_fpn
import PIL.Image as Image

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model = fasterrcnn_resnet50_fpn(pretrained=True)
model.roi_heads = nn.Sequential()
model.to(device)

img = Image.open('frame_00001.jpg')
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                    std=[0.229, 0.224, 0.225])])
img = transform(img).unsqueeze(0).to(device)

model.eval()
output = model(img)

The last line causes a "TypeError: forward() takes 2 positional arguments but 5 were given."
The size of img is [1, 3, 960, 1280].
If I add square brackets around img before passing into the model (output = model([img])), I get a "ValueError: images is expected to be a list of 3d tensors of shape [C, H, W], got torch.Size([1, 3, 960, 1280])." Then, if I use "img.view(3, 960, 1280)," I get the original error again.
What is the solution to this problem? Thank you.

Output:
I'm not sure what you're trying to do with the line:

model.roi_heads = nn.Sequential()

This line is causing the problem, I ran you're code excerpt without this line and it worked as expected (minus a few warnings about deprecation coming from the model being used, but the output looks reasonable)


Instruction: PyTorch save prediction as CSV for each epoch

Input:
I'm trying save the predictions I am getting from a model in PyTorch as csv. The following code. However, the code seem to overwrite the prediction at each epoch and the final .csv file only contains 3 values. Any idea what I am doing wrong please:
output = model(images)
preds = output.sum(dim=[1,2,3])
np.savetxt('preds.csv', preds.detach())

Output:
Assuming your output will be float, you can map that to str and apply join as a comma separated string, finally append row to csv.
import pandas as pd
import numpy as np
import torch

output = torch.rand(5,5,5,5)
preds = output.sum(dim=[1,2,3])

print(preds)

with open('preds.csv','a') as fd:
    fd.write( ','.join(map(str, preds.detach().tolist())) + '\n')


Instruction: Unable to load quantized pytorch mobile model on android

Input:
I am struggling to get my quantized pytorch mobile model (custom MobilenetV3) running on android. I have followed this tutorial https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html and I managed to quantize my model without any problems. However, when I try to load the model via. module = Module.load(assetFilePath(this, MODEL_NAME));
I get the following Exception:

Unknown builtin op: quantized::linear_unpack_fp16.
Could not find any similar ops to quantized::linear_unpack_fp16. This
op may not exist or may not be currently supported in TorchScript.

Why are there even float16 values in the quantized model, I thought quantization would replace all float32 vlaues with qint8/quint8? Any ideas on how this can be fixed?
This is how the pytorch model was quantized and saved:
torch.quantization.get_default_qconfig(backend='qnnpack')
model.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(model, inplace=True)
torch.quantization.convert(model, inplace=True)
traced_script_module = torch.jit.trace(model,dummyInput)
traced_script_module.save("model/modelQuantized.pt")

Output:
I found that this error was caused by a version mismatch between the pytorch version that was used for model quantization and pytorch_android.
The model was quantized with pytorch 1.5.1 torchvision 0.6.1 cudatoolkit 10.2.89 but I used org.pytorch:pytorch_android:1.4.0 for building.
Switching to org.pytorch:pytorch_android:1.5.0 solved it.


Instruction: cannot install pip install torchvision

Input:
I am using python 3.8.3 version, I tried to install torchvision and torch module and faced this error, none of them are installed. The error comes as ERROR:could not find a version that satisfies the requirement torch==1.4.0(from versions:0.1.2,0.1.2,post1,0.1.2.post2)
Error:No matching distribution found for torch==1.4.0

Output:
According to PyTorch's website, you must specify if you are using cpu or the version of CUDA when installing from pip.
For instance, if I wanted to install on a Linux system without CUDA, version 1.5.1 of PyTorch, I would run:
pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

You can use the link I provided above to get the syntax for your specific environment.


Instruction: Finetuning embeddings with torchtext - nn.Embedding vs. nn.Embedding.from_pretrained

Input:
I have been working with pretrained embeddings (Glove) and would like to allow these to be finetuned. I currently use embeddings like this:
word_embeddingsA = nn.Embedding(vocab_size, embedding_length)
word_embeddingsA.weight = nn.Parameter(TEXT.vocab.vectors, requires_grad=False)

Should I simply set requires_grad=True to allow the embeddings to be trained? Or should I do something like this
word_embeddingsA = nn.Embedding.from_pretrained(TEXT.vocab.vectors, freeze=False)

Are these equivalent, and do I have a way to check that the embeddings are getting trained?

Output:
Yes they are equivalent as states in embedding:

freeze (boolean, optional) – If True, the tensor does not get updated in the learning process. Equivalent to embedding.weight.requires_grad = False. Default: True

If word_embeddingsA.requires_grad == True, then embedding is getting trained, else it's not.


Instruction: Using PyTorch for computing derivative of a function

Input:
When I execute the code below:
import torch

def g(x):
    return 4*x + 3
x=3.0
g_hat=torch.tensor(g(x), requires_grad= True)
g_hat.backward()

I get the following output:
>>>g_hat.grad
tensor(1.)

But this is not the result that I was expecting to get from my code above... what I want to do is, I want to find the value of dg/dx, at x = 3.0 (so in the example above, the correct output should be tensor(4.)).
How can I achieve this with PyTorch? or if I can't carry out this task with PyTorch, how can I do this type of task easily on Python?
Thank you,

Output:
x = torch.autograd.Variable(torch.Tensor([1.0]),requires_grad=True)
y = 4 * x + 3
y.backward()
x.grad

Works fine.


Instruction: Hugginface transformers module not recognized by anaconda

Input:
I am using Anaconda, python 3.7, windows 10.
I tried to install transformers by https://huggingface.co/transformers/ on my env.
I am aware that I must have either pytorch or TF installed, I have pytorch installed - as seen in anaconda navigator environments.
I would get many kinds of errors, depending on where (anaconda / prompt) I uninstalled and reinstalled pytorch and transformers. Last attempt using
conda install pytorch torchvision cpuonly -c pytorch and
conda install -c conda-forge transformers
I get an error:
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

def tok(dataset):
    input_ids = []
    attention_masks = []
    sentences = dataset.Answer2EN.values
    labels = dataset.Class.values
    for sent in sentences:
        encoded_sent = bert_tokenizer.encode(sent,
                                             add_special_tokens=True,
                                             max_length = 64,
                                             pad_to_max_length =True)

TypeError: _tokenize() got an unexpected keyword argument
'pad_to_max_length'

Does anyone know a secure installation of transformers using Anaconda?
Thank you

Output:
The problem is that conda only offers the transformers library in version 2.1.1 (repository information) and this version didn't have a pad_to_max_length argument. I'm don't want to look it up if there was a different parameter, but you can simply pad the result (which is just a list of integers):
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

sentences = ['this is just a test', 'this is another test']

max_length = 64

for sent in sentences:
    encoded_sent = bert_tokenizer.encode(sent,
                                         add_special_tokens=True,
                                         max_length = max_length)
    encoded_sent.extend([0]* (max_length - len(encoded_sent)))

    ###your other stuff

The better option in my opinion is to create a new conda environment and install everything via pip and not via conda. This will allow you to work with the most recent transformers version (2.11).


Instruction: Display Pytorch tensor as image using Matplotlib

Input:
I am trying to display an image stored as a pytorch tensor.
trainset = datasets.ImageFolder('data/Cat_Dog_data/train/', transform=transforms)
trainload = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True)

images, labels = iter(trainload).next()
image = images[0]
image.shape

>>> torch.Size([3, 224, 224]) # pyplot doesn't like this, so reshape

image = image.reshape(224,224,3)
plt.imshow(image.numpy())

This method is displaying a 3 by 3 grid of the same image, always in greyscale. For example:

How do I fix this so that the single color image is displayed correctly?

Output:
That's very odd. Try putting the channels last by permuting rather than reshaping:
image.permute(1, 2, 0)


Instruction: How to use pytorch's cat function for K-Fold Validation (i.e. concatenate a list of pytorch chunks together)

Input:
I'm working on splitting up a data set for k-fold cross validation but having trouble with concatenating a list of tensors using Pytorch's stack/cat functions.
First, I split the training and test set into chunks using the .chunk method as follows
  x_train_folds = torch.chunk(x_train, num_folds)
  y_train_folds = torch.chunk(y_train, num_folds)

Where x_train is a tensor of torch.Size([5000, 3, 32, 32]) and y_train is a tensor of torch.Size([5000])
x_train_folds and y_train_folds are now tuples of num_folds tensors
Then, I need to setup a series of nested loops to iterate through the different values for K, and the various folds, while always excluding one fold from the training set to be used at test/validation time:
  for k in k_choices:
    k_to_accuracies[k] = [] # create empty space to append for a given k-value
    for fold in range(num_folds):
      # create training sets by excluding the current loop index fold and using that as the test set
      x_train_cross_val = torch.cat((x_train_folds[:fold], x_train_folds[fold+1:]), 0)
      y_train_cross_val = torch.cat((y_train_folds[:fold], y_train_folds[fold+1:]), 0)
      classifier = KnnClassifier(x_train_cross_val, y_train_cross_val)
      k_to_accuracies[k].append(classifier.check_accuracy(x_train_folds[fold], y_train_folds[fold], k=k))

As you can see, I'm always skipping one fold from the original training set to be used for validation. This is standard K-fold cross validation.
Unfortunately, I am getting the following error which I cannot seem to figure out:
TypeError: expected Tensor as element 0 in argument 0, but got tuple
As you can see in the API listing, .cat seems to need a tuple of tensors which is what I have.
https://pytorch.org/docs/stable/torch.html#torch.cat
Does anyone have any suggestions?
Much appreciated
-Drew

Output:
try:
x_train_cross_val = torch.cat((*x_train_folds[:fold], *x_train_folds[fold+1:]), 0)
y_train_cross_val = torch.cat((*y_train_folds[:fold], *y_train_folds[fold+1:]), 0)

torch.cat receives a tuple whose elements are torch.Tensor type. However, the elements in your tuple x_train_folds[:fold] are still tuple. So, you need to remove the tuple 'decorator' of your tensors.


Instruction: What does next() and iter() do in PyTorch's DataLoader()

Input:
I have the following code:
import torch
import numpy as np
import pandas as pd
from torch.utils.data import TensorDataset, DataLoader

# Load dataset
df = pd.read_csv(r'../iris.csv')

# Extract features and target
data = df.drop('target',axis=1).values
labels = df['target'].values

# Create tensor dataset
iris = TensorDataset(torch.FloatTensor(data),torch.LongTensor(labels))

# Create random batches
iris_loader = DataLoader(iris, batch_size=105, shuffle=True)

next(iter(iris_loader))

What does next() and iter() do in the above code? I have went through PyTorch's documentation and still can quite understand what is next() and iter() doing here. Can anyone help in explaining this? Many thanks in advance.

Output:
These are built-in functions of python, they are used for working with iterables.
Basically iter() calls the __iter__() method on the iris_loader which returns an iterator. next() then calls the __next__() method on that iterator to get the first iteration. Running next() again will get the second item of the iterator, etc.
This logic often happens 'behind the scenes', for example when running a for loop. It calls the __iter__() method on the iterable, and then calls __next__() on the returned iterator until it reaches the end of the iterator. It then raises a stopIteration and the loop stops.
Please see the documentation for further details and some nuances: https://docs.python.org/3/library/functions.html#iter


Instruction: How the gradient is calculated in pytorch

Input:
I have an example code. When I calculate dloss/dw manually I get the result 8, but the following code gives me a 16. Please tell me how the gradient is 16.
import torch
x = torch.tensor(2.0)
y = torch.tensor(2.0)
w = torch.tensor(3.0, requires_grad=True)
# forward
y_hat = w * x
s = y_hat - y
loss = s**2
#backward
loss.backward()
print(w.grad)

Output:
I think you simply miscalculated.
The derivation of loss = (w * x - y) ^ 2 is:
dloss/dw = 2 * (w * x - y) * x = 2 * (3 * 2 - 2) * 2 = 16
Keep in mind that back-propagation in neural networks is done by applying the chain rule: I think you forgot the *x at the end of the derivation
To be specific:
chain rule for derivation says that df(g(x))/dx = f'(g(x)) * g'(x) (derivated with respect to x)
the whole loss function in your case is built like this:
loss(y_hat) = (y_hat - y)^2
y_hat(x) = w * x
thus: loss(y_hat(x)) = (y_hat(x) - y)^2
the derivation of this is according to chain rule:
dloss(y_hat(x))/dw = loss'(y_hat(x)) * dy_hat(x)/dw
for any z:
loss'(z) = 2 * (z - y) * 1 and dy_hat(z)/dw = z
thus: dloss((y_hat(x))/dw = dloss(y_hat(x))/dw = loss'(y_hat(x)) * y_hat'(x) = 2 * (y_hat(x) - z) * dy_hat(x)/dw = 2 * (y_hat(x) - z) * x = 2 * (w * x - z) * x = 16
pytorch knows that in your forward pass each layer applies some kind of function to its input and that your forward pass is 1 * loss(y_hat(x)) and than keeps applying the chain rule for the backward pass (each layer requires one application of the chain rule).


Instruction: pytorch: how to multiply 3d tensor with 2d tensor

Input:
I got a 3D tensor three and a 2D tensor two, which need to be multiplied. For example, the dimensions are:
three.shape = 4x100x700
two.shape = 4x100

Output shape should be:
output.shape = 4x100x700

So basically, in output[a,b] there should be 700 scalars which were computed by multiplying all 700 scalars from three[a,b] with the single scalar from two[a,b].

Output:
You can simply add an extra dimension to two:
output = three * two.unsqueeze(-1)

There are alternative syntax, e.g.:
output = three * two[..., None]


Instruction: Why am I getting a divide by zero error here?

Input:
So I'm following along this tutorial in the docs on custom datasets. I'm using the MNIST dataset instead of the fancy one in the tutorial. This is the extension of the Dataset class I wrote:
class KaggleMNIST(Dataset):

    def __init__(self, csv_file, transform=None):
        self.pixel_frame = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.pixel_frame)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.tolist()

        image = self.pixel_frame.iloc[index, 1:]
        image = np.array([image])

        if self.transform:
            image = self.transform(image)

        return image

It works, until I try to use a transform on it:
tsf = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,), (0.5,))
                         ])

trainset = KaggleMNIST('train/train.csv', transform=tsf)

image0 = trainset[0]

I've looked at the stack trace, and it seems like the normalization is happening in this line of code:
c:\program files\python38\lib\site-packages\torchvision\transforms\functional.py in normalize(tensor, mean, std, inplace)
--> 218     tensor.sub_(mean[:, None, None]).div_(std[:, None, None])

So I don't get why there is divide by zero since std should be 0.5, nowhere remotely close to a small value.
Thanks for your help!
EDIT:
This does not answer my question, but I discovered that if I change these lines of code:
image = self.pixel_frame.iloc[index, 1:]
image = np.array([image])

to
image = self.pixel_frame.iloc[index, 1:].to_numpy(dtype='float64').reshape(1, -1)

Essentially, making sure the datatype was float64 fixed the problem. I'm still not sure why the problem existed in the first place, so I'd still be happy for a well-explained answer!

Output:
The dtype of the data read is int64
img = np.array([pixel_frame.iloc[0, 1:]])
img.dtype
# output
dtype('int64')

This forces the mean and std to be converted to int64 and as std is 0.5, it becomes 0, and raises the following error:
>>> tsf(img)
ValueError: std evaluated to zero after conversion to torch.int64, leading to division by zero.

It's because the mean and std are converted to dtype of the dataset during normalization.
def normalize(tensor, mean, std, inplace=False):
    ...
    dtype = tensor.dtype
    mean = torch.as_tensor(mean, dtype=dtype, device=tensor.device)
    std = torch.as_tensor(std, dtype=dtype, device=tensor.device)
    if (std == 0).any():
        raise ValueError('std evaluated to zero after conversion to {}, leading to division by zero.'.format(dtype))

That's why converting the dtype to float fixes the error.


Instruction: PyTorch code stops with message "Killed". What killed it?

Input:
I train a network on GPU with Pytorch. However, after at most 3 epochs, code stops with a message :
Killed
No other error message is given.
I monitored the memory and gpu usage, there were still space during the run. I reviewed the /var/sys/dmesg to find a detailed message regarding to this, however no message with "kill" was entered. What might be the problem?
Cuda version: 9.0
Pytorch version: 1.1.0

Output:
In order to give an idea to people who will enconter this:
Apparently, Slurm was installed on the machine so that I needed to give the tasks on Slurm.


Instruction: Does cv2.rectangle() have an argument named “rec”?

Input:
I was trying to draw some rectangles within an image but the system gave me this error:
Here is my code:
cv2.rectangle(img=sample,
                      pt1=(box[0], box[1]),
                      pt2=(box[2], box[3]),
                      color=(220, 0, 0), thickness=2)

Any advice will be appreciated!
PS: I tried this piece of codes on kaggle notebok and it ran successfully but it crashed when I want to apply streamlit to deploy a web app (using my local machine). Not sure if it makes a difference, just FYI.

Output: I figured out this issue, it is about the version. When I downgraded to 4.1.0.25, problem is solved.


Instruction: Invalid syntax error when setting root = to in PyTorch

Input:
import torchvision
from torchvision import transforms
train_data_path="./train/"
transforms = transforms.Compose([
transforms.Resize(64),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225] )
])
train_data = torchvision.datasets.ImageFolder
(root=train_data_path,transform=transforms)

This is the error message:
File "<ipython-input-4-e470172b3902>", line 8
    (root=train_data_path,transform=transforms)
         ^
SyntaxError: invalid syntax

How would I be able to fix this?

Output:
You need the opening parentheses to be directly connected to the function, without any whitespace separating them. Try replacing the final two lines with:
train_data = torchvision.datasets.ImageFolder(
    root=train_data_path, transform=transforms
)


Instruction: importing error default_collate is not defined

Input:
I tried to import default_collate from torch.utils.data.dataloader but it gives me default_collate is not exsit and another function is exist (_collate_fn_t)
from torch.utils.data.dataloader import default_collate
from torch.utils.data.dataloader import _collate_fn_t
Are these commands are similar

Output:
The _collate_fn_t is the type definition of a collate function in general, it is defined here as:
_collate_fn_t = Callable[[List[T]], Any]

default_collate is the default collate function used by the DataLoader class.
Importing these is not equivalent, you should check your spelling and try again with:
from torch.utils.data.dataloader import default_collate


Instruction: How many weight convolution layer has?

Input:
I have a simple convolution network:
import torch.nn as nn

class model(nn.Module):
def __init__(self, ks=1):
    super(model, self).__init__()
    self.conv1 = nn.Conv2d(in_channels=4, out_channels=32, kernel_size=ks, stride=1)

    self.fc1 = nn.Linear(8*8*32*ks, 64)
    self.fc2 = nn.Linear(64, 64)

def forward(self, x):
    x = F.relu(self.conv1(x))
    x = x.view(x.size(0), -1)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

cnn = model(1)

Since the kernel size is 1 and the output channel is 32, I assume that there should be 32*1*1 weights in this layer. But, when I ask pytorch about the shape of the weight matrix cnn.conv1.weight.shape, it returns torch.Size([32, 4, 1, 1]). Why the number of input channel should matter on the weight of a conv2d layer?
Am I missing something?

Output: It matters because you are doing 2D convolution over the images which means the depth of the filter(kernel) must be equal to the number of in_channels(pytorch sets it for you) so the true kernel size is [in_channels,1,1]. On the other hands we can say that out_channels number is the number of kernels so the number of weights = number of kernels * size of kernel = out_channels * (in_channels * kernel_size). Here is 2D conv with 3D input


Instruction: Pytorch differences between two tensors

Input:
I have two tensors like this:
1st tensor
[[0,0],[0,1],[0,2],[1,3],[1,4],[2,1],[2,4]]

2nd tensor
[[0,1],[0,2],[1,4],[2,4]]

I want the result tensor to be like this:
[[0,0],[1,3],[2,1]] # differences between 1st tensor and 2nd tensor

I have tried to use set, list, torch.where,.. and couldn't find any good way to achieve this. Is there any way to get the different rows between two different sizes of tensors? (need to be efficient)

Output:
You can perform a pairwairse comparation to see which elements of the first tensor are present in the second vector.
a = torch.as_tensor([[0,0],[0,1],[0,2],[1,3],[1,4],[2,1],[2,4]])
b = torch.as_tensor([[0,1],[0,2],[1,4],[2,4]])

# Expand a to (7, 1, 2) to broadcast to all b
a_exp = a.unsqueeze(1)

# c: (7, 4, 2)
c = a_exp == b
# Since we want to know that all components of the vector are equal, we reduce over the last fim
# c: (7, 4)
c = c.all(-1)
print(c)
# Out: Each row i compares the ith element of a against all elements in b
# Therefore, if all row is false means that the a element is not present in b
tensor([[False, False, False, False],
        [ True, False, False, False],
        [False,  True, False, False],
        [False, False, False, False],
        [False, False,  True, False],
        [False, False, False, False],
        [False, False, False,  True]])
non_repeat_mask = ~c.any(-1)

# Apply the mask to a
print(a[non_repeat_mask])
tensor([[0, 0],
        [1, 3],
        [2, 1]])

If you feel cool you can do it one liner :)
a[~a.unsqueeze(1).eq(b).all(-1).any(-1)]


Instruction: Why use multiple ReLU objects in Neural Net class definition?

Input:
Recently I observed that a lot of times while defining the neural nets we define separate ReLU objects for each layer. Why can't we use the same ReLU object wherever it is needed.
For example instead of writing like this-
def __init__(self):
    self.fc1     = nn.Linear(784, 500)
    self.ReLU_1  = nn.ReLU()
    self.fc2     = nn.Linear(500, 300)
    self.ReLU_2  = nn.ReLU()

def forward(x):
    x = self.fc1(x)
    x = self.ReLU_1(x)
    x = self.fc2(x)
    x = self.ReLU_2(x)

why can't we use
def __init__(self):
    self.fc1    = nn.Linear(784, 500)
    self.ReLU   = nn.ReLU()
    self.fc2    = nn.Linear(500, 300)

def forward(x):
    x = self.fc1(x)
    x = self.ReLU(x)
    x = self.fc2(x)
    x = self.ReLU(x)

Is this something specific to PyTorch?

Output: We can do so. First variant is just for clarity.


Instruction: What does 'requires grad' do in PyTorch and should I use it?

Input:
I have a network where I need to add my own parameter that I want to be trainable. I am using nn.Parameter() to add this but there's a 'Requires Grad' argument and I can't really make sense as to whether I want this to be true or false from reading the documentation. It makes sense that I would set this to be true, because I want this parameter to be optimised as part of the learning process -- but the need for this argument confuses me: if False means it is not optimised as part of the training process, then why would you use nn.Parameter() rather than just using a normal Tensor?
From the documentation I see that it adds the parameter to the list of utterable parameters that you obtain from the model, but I don't see why you would want this if you're not optimising it.

Output: As far as I know, sometimes you might need to freeze/unfreeze some part of your neural network and avoid/let some of the parameters to be optimized during the training. "requires_grad" argument provides an easy way to include or exclude your network's parameters in the backpropagation phase. You just set it to True or False and it's done.


Instruction: Slicing pytorch tensors and use of data_ptr()

Input:
a = tensor([[ 1,  2,  3,  4,  5],
        [ 6,  7,  8,  8, 10],
        [11, 12, 13, 14, 15]])

I have a torch tensor and I need to index a tensor c such that c = [[3], [8], [13]]
So I did c = a[:,[2]] which gave me the expected answer, but it still failed on the autograder.
The autograder uses a check function as follows -
def check(orig, actual, expected):
  expected = torch.tensor(expected)
  same_elements = (actual == expected).all().item() == 1
  same_storage = (orig.storage().data_ptr() == actual.storage().data_ptr())
  return same_elements and same_storage

print('c correct:', check(a, c, [[3], [8], [13]]))

I tried debugging it and it turns out that same_storage is false, I don't understand why orig.storage().data_ptr() == actual.storage().data_ptr() should be True, and how it makes a difference.
Update
I was able to get the correct answer by doing c = a[:, 2:3] instead of c = a[:, [2]] what is the difference?

Output:
PyTorch allows tensor to be a "view" of an existing tensor, such that it shares the same underlying data with its base tensor, thus avoiding explicit data copy to be able to perform fast and memory efficient operations.
As mentioned in the Tensor View docs,

When accessing the contents of a tensor via indexing, PyTorch follows Numpy behaviors that basic indexing returns views, while advanced indexing returns a copy.

In your example, c = a[:, 2:3] is basic indexing, while, c = a[:, [2]] is advanced indexing. That's why a view is created in the first case only. Thus, .storage().data_ptr() gives same result.
You can read about basic and advanced indexing in Numpy indexing docs.

Advanced indexing is triggered when the selection object, obj, is a non-tuple sequence object, an ndarray (of data type integer or bool), or a tuple with at least one sequence object or ndarray (of data type integer or bool).


Instruction: How to convert a string of pixels into an image with python?

Input:
I am having a string of pixels of a B/W image in this manner:-
48 34 21 18 16 21 26 36 40...44 53 57 64 82 95 9

of string length 547, and I want these pixels to be converted into an image of 48*48 pixel B/W image.
Is there any python library or PyTorch function to make it possible?
Link to the Notebook

Output:
You can use PIL.Image.frombuffer() function for example.
from PIL import Image

pixels = '126 126 129 120 110 168 174 172 173 174 170 157 134 97 78 75 118 119 97 117 142 86 58 71 124 161 162 134 100 119 124 118 153 135 143 180 180 172 153 128 98 67 60 75 85 89 85 98 123 126 131 119 119 172 180 171 155 127 100 82 77 72 54 97 184 151 157 176 125 74 71 114 173 171 136 109 116 118 118 150 152 150 178 191 188 187 180 169 152 127 86 60 74 91 92 95 119 131 127 115 113 153 130 94 71 62 65 69 83 95 79 137 181 174 196 148 88 75 103 149 135 107 100 105 104 112 140 162 178 191 187 181 171 166 159 156 159 155 144 97 54 69 83 91 112 116 100 85 74 73 59 58 67 75 74 80 104 110 96 153 180 182 151 100 103 139 167 172 152 141 135 135 141 163 182 191 191 183 179 169 144 127 132 147 156 154 152 143 92 52 56 78 97 86 69 60 51 48 55 65 74 74 71 92 121 122 115 155 177 166 151 158 181 200 201 208 207 198 187 181 182 189 187 187 179 170 154 119 91 86 101 124 139 154 153 153 132 73 51 60 63 52 48 45 46 54 65 78 79 86 96 131 163 146 133 162 179 193 202 211 205 208 216 218 209 204 191 183 184 183 176 170 153 127 96 98 127 148 148 142 134 140 150 154 149 102 49 53 57 58 65 68 70 75 75 74 90 121 151 176 176 148 156 170 193 209 215 211 203 213 219 212 206 201 183 174 173 166 162 135 101 86 99 147 176 175 166 164 157 146 148 155 154 128 55 53 54 54 77 92 89 68 58 61 69 94 143 171 155 150 161 163 185 197 203 202 207 213 212 209 202 195 176 163 157 153 125 83 86 118 160 179 172 165 161 161 166 164 155 155 153 138 74 54 40 70 110 122 106 92 112 123 116 108 96 94 118 146 155 159 166 174 180 187 198 213 216 205 196 192 178 155 139 117 88 96 131 163 159 142 130 126 123 123 132 139 147 156 158 145 75 59 60 117 163 157 138 157 176 179 186 189 167 121 98 105 123 149 161 162 165 178 196 213 214 199 188 181 171 147 117 97 104 124 134 127 123 131 135 139 138 137 144 146 139 147 135 106 75 69 98 155 184 180 178 189 187 182 181 190 203 198 165 126 102 109 138 162 171 193 216 225 222 204 191 182 154 130 116 118 137 143 125 108 100 96 96 102 122 129 145 158 142 96 90 127 88 50 125 175 195 191 191 189 178 166 165 172 175 178 177 162 139 116 112 130 158 194 211 216 218 202 189 165 132 129 134 143 129 104 86 98 80 90 128 97 70 90 117 149 161 109 127 151 112 58 145 187 200 198 192 185 174 171 180 162 135 117 120 126 128 128 124 116 124 157 171 166 156 126 107 92 101 127 129 104 93 105 119 137 113 136 145 136 118 107 119 149 162 147 152 152 125 77 142 183 189 176 177 178 174 163 118 109 118 109 95 83 94 101 114 134 125 120 152 153 154 152 156 124 99 103 122 111 112 136 151 156 163 160 160 163 155 143 145 159 163 137 141 147 126 86 119 158 161 184 183 185 163 100 98 167 135 126 89 99 116 104 101 146 144 115 162 197 201 191 189 156 112 94 103 108 112 127 141 150 153 155 156 148 141 149 156 157 165 155 144 146 129 86 112 140 156 191 190 179 144 126 159 179 160 150 148 143 135 127 135 138 137 117 151 199 195 178 177 169 123 105 100 106 115 132 138 143 145 145 143 139 149 156 156 151 141 141 143 143 130 94 131 136 142 181 198 188 177 189 193 185 183 166 152 136 123 121 130 125 132 143 186 207 192 177 177 169 140 119 128 141 142 147 154 159 158 155 158 156 153 146 132 114 122 136 136 134 127 105 136 143 151 167 184 190 198 189 175 168 160 148 138 131 131 144 160 181 176 168 212 206 202 198 185 173 160 150 133 139 148 155 158 164 165 158 146 133 119 113 118 127 130 127 126 125 124 113 176 181 163 183 177 177 184 186 183 172 164 160 160 164 165 174 189 184 171 206 218 212 206 198 187 180 177 171 157 125 113 116 121 127 126 126 126 129 133 136 134 128 121 118 117 115 113 111 175 199 166 177 196 190 188 189 190 190 188 186 184 182 183 189 179 150 171 208 219 209 201 191 181 173 167 152 150 159 157 148 149 154 155 157 153 146 142 140 130 124 117 112 113 108 100 104 175 199 190 163 181 191 187 187 191 191 189 185 179 170 161 155 150 168 168 185 221 204 193 181 168 165 157 137 120 126 158 167 164 168 168 166 160 152 147 144 138 127 118 113 106 103 95 95 173 196 200 206 189 175 168 169 171 167 166 159 154 155 162 176 190 172 147 191 213 202 193 179 162 158 154 137 124 108 123 155 163 167 169 167 164 157 151 144 135 128 120 113 106 99 92 89 164 192 203 215 214 208 206 193 183 175 173 176 179 185 194 196 182 142 148 202 213 204 202 187 173 165 154 151 135 112 102 131 154 157 162 161 158 149 142 137 129 123 117 108 101 95 88 85 156 189 205 211 206 206 206 202 197 198 194 197 201 201 195 178 156 129 171 211 216 214 208 190 179 171 168 149 132 117 102 112 134 152 154 150 150 143 137 131 123 119 112 103 96 91 83 84 148 189 202 204 201 198 199 201 202 201 197 194 192 191 177 159 133 136 189 206 208 210 199 186 180 176 161 129 129 117 106 114 126 144 149 149 145 141 135 130 122 116 108 98 93 90 81 83 149 190 200 199 194 196 199 194 193 192 185 182 180 172 161 141 129 159 188 186 193 194 185 176 173 164 123 93 105 104 117 127 125 132 145 149 142 137 134 129 119 111 103 96 92 87 79 83 159 190 197 198 190 188 188 186 182 180 174 169 165 160 145 126 134 180 187 154 158 186 175 154 151 137 99 84 87 101 124 130 131 131 140 146 143 136 133 126 117 109 99 92 88 85 76 80 171 189 192 187 184 182 182 179 173 169 168 160 156 148 135 124 158 191 196 172 158 169 146 126 123 117 103 102 104 114 128 132 135 133 133 143 144 138 130 122 112 104 96 90 87 84 76 73 175 190 184 180 178 173 171 169 167 162 159 156 152 139 123 131 177 189 192 188 177 167 147 122 111 101 100 110 116 117 131 133 133 135 134 140 144 138 129 117 108 103 97 95 92 85 76 69 179 191 186 178 171 168 164 162 156 151 153 150 144 131 118 151 181 191 198 193 183 175 163 136 112 109 115 118 121 126 136 132 130 132 129 137 144 140 127 116 108 105 101 99 92 86 76 67 160 190 184 173 169 161 156 152 148 144 146 147 141 120 126 172 184 195 203 200 191 184 174 159 142 139 133 129 133 138 135 131 124 124 127 130 141 145 132 119 112 109 106 104 96 88 73 64 110 184 188 176 168 158 145 143 144 147 146 146 136 115 145 176 184 198 199 196 197 190 178 163 159 157 149 145 144 141 134 129 121 119 126 128 138 151 141 124 117 113 110 106 98 88 69 60 89 172 189 178 170 163 149 138 144 152 155 154 128 128 166 178 184 193 197 200 198 192 182 170 163 168 162 156 147 140 136 132 121 115 118 124 130 155 152 132 126 118 110 106 98 86 66 56 60 157 186 180 173 167 153 142 155 167 166 156 125 151 174 178 178 185 193 189 191 188 175 168 167 162 144 130 126 122 122 124 118 112 117 121 124 155 163 137 129 118 110 107 104 86 63 57 41 140 184 178 178 170 159 154 172 183 181 158 134 161 170 176 176 184 184 169 156 138 129 134 136 114 101 99 100 99 99 98 103 108 119 128 131 160 171 140 126 113 106 112 108 82 64 60 39 116 184 173 179 171 165 161 182 199 198 170 149 170 170 174 185 173 148 132 123 122 126 120 106 109 124 129 124 97 64 61 58 78 112 130 136 165 177 143 121 106 104 124 107 75 63 60 40 86 181 175 176 175 169 162 175 203 203 179 164 179 175 172 143 109 97 84 106 158 175 184 171 164 168 134 76 43 33 65 86 90 105 122 142 174 175 134 116 98 119 135 99 70 60 59 48 69 169 183 173 174 170 164 169 197 209 190 164 179 170 142 130 126 79 44 55 82 127 142 124 112 105 72 54 71 89 111 102 88 96 120 149 180 163 127 110 100 136 126 84 60 57 60 50 59 146 190 174 173 174 168 167 189 204 199 175 164 150 145 168 171 143 111 125 135 144 135 133 139 136 129 128 128 126 112 85 78 102 128 155 174 141 127 100 119 144 114 73 58 60 62 50 50 118 189 181 173 175 171 167 180 196 194 181 166 162 161 163 151 155 164 168 166 164 155 155 155 148 141 135 124 105 85 78 90 122 138 156 149 140 124 98 140 138 94 64 60 63 64 47 48 80 170 191 177 174 180 172 169 187 189 177 170 172 172 169 159 148 142 141 140 139 137 139 133 125 116 106 96 86 81 88 113 122 127 135 142 155 102 119 154 118 76 62 68 65 65 39 46 52 130 185 185 174 182 178 170 174 186 178 171 176 187 178 164 151 135 121 115 118 121 116 110 104 99 93 92 92 93 111 127 125 137 138 161 126 94 144 139 96 70 69 71 61 74 35 40 44 84 166 188 183 176 180 178 174 179 185 180 193 193 180 171 160 150 151 154 152 151 146 141 135 119 108 101 96 102 128 137 145 144 133 126 91 118 147 117 82 74 78 72 61 101 38 36 44 53 127 181 187 178 174 181 180 173 183 190 190 196 195 178 164 161 167 172 168 165 171 163 154 141 128 115 104 117 149 155 139 124 111 96 99 139 135 102 88 86 82 72 64 140 43 37 41 43 75 161 181 183 173 174 179 180 175 190 199 197 198 189 181 173 166 171 173 180 176 158 148 141 126 115 113 129 147 136 132 115 97 91 125 148 124 105 94 87 77 66 82 181 44 41 35 41 45 114 177 177 181 177 175 183 184 176 192 198 184 186 195 198 188 173 174 172 167 157 145 138 121 121 115 113 125 146 133 98 83 111 148 145 122 107 90 85 75 58 118 203 48 44 35 36 42 58 144 181 189 188 182 182 187 181 172 180 189 192 193 198 192 186 178 165 160 152 138 128 128 129 110 111 139 132 105 83 97 143 153 137 118 97 87 79 68 68 168 211 45 45 41 32 38 40 90 163 179 190 184 186 185 189 182 176 185 194 199 187 175 173 162 156 156 145 128 125 127 121 104 111 121 110 89 87 131 153 148 130 106 85 77 71 60 107 199 208'

b = bytes(int(p) for p in pixels.split())
i = Image.frombuffer('L', (48, 48), b)
print(i)
i.save('example.png')

Outputs example.png:


Instruction: Can't import sklearn ([WinError 126] The specified module could not be found)

Input:
from sklearn import datasets
I can't import sklearn
here is the error:
OSError                                   Traceback (most recent call last)

<ipython-input-13-f9e6334b9a20> in <module>
  1 import torch
  2 import numpy as np
   ----> 3 from sklearn import datasets
  4
  5 X_numpy, y_numpy = datasets.make_regression(n_samples=100, n_features=1, noise=20,
  random_state=1)

 c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-
packages\sklearn\__init__.py in <module>
 78     from . import _distributor_init  # noqa: F401
 79     from . import __check_build  # noqa: F401
---> 80     from .base import clone
 81     from .utils._show_versions import show_versions
 82

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\sklearn\base.py in <module>
 19 from . import __version__
 20 from ._config import get_config
---> 21 from .utils import _IS_32BIT
 22 from .utils.validation import check_X_y
 23 from .utils.validation import check_array

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\sklearn\utils\__init__.py in <module>
 18 import warnings
 19 import numpy as np
---> 20 from scipy.sparse import issparse
 21
 22 from .murmurhash import murmurhash3_32

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\scipy\__init__.py in <module>
102
103 # Allow distributors to run custom init code
--> 104 from . import _distributor_init
105
106 __all__ += _num.__all__

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\scipy\_distributor_init.py in <module>
 59             os.chdir(libs_path)
 60             for filename in glob.glob(os.path.join(libs_path, '*dll')):
 ---> 61                 WinDLL(os.path.abspath(filename))
 62         finally:
 63             os.chdir(owd)

 c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
362
363         if handle is None:
--> 364             self._handle = _dlopen(self._name, mode)
365         else:
366             self._handle = handle

OSError: [WinError 126] The specified module could not be found

when I type pip install sklearn I get Requirement already satisfied:
Requirement already satisfied: sklearn in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (0.0) Requirement already satisfied: scikit-learn in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from sklearn) (0.23.1) Requirement already satisfied: joblib>=0.11 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (0.15.1) Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (2.1.0) Requirement already satisfied: numpy>=1.13.3 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (1.18.5) Requirement already satisfied: scipy>=0.19.1 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (1.5.0)
so why can't I use sklearn, any help is appreciated thanks.
EDIT: when I import sklearn in cmd it works with no issues, however, when I run using Jupyter Notebook I get the error above.

Output:
Just make sure  OpenSlide DLL in your library search path
Otherwise:
download the OpenSlide Windows binaries, then you need to add the 'bin' folder to your environmental path


Instruction: Where do the input flatten value come from in fully connected 1 (fc1) layer (MNIST Example)

Input:
here's some of the convolutional neural network sample code from Pytorch's examples directory on their github:
https://github.com/pytorch/examples/blob/master/mnist/main.py
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

If I understand this, we need to flatten the output from the last convolutional layer before we can pass it through a linear layer (fc1). So, looking at this code, we see the input to the first fully connected layer is: 9216.
Where has this number (9216) come from?

Output:
You also need to look at the forward method and the network input shape in order to compute the input shape of the linear/fully-connected layer. In the case of MNIST we have a single channel 28x28 input image. Using the following formulas from the docs you can compute the output shape of each convolution operation. The max-pooling operation follows the same input-output relationship as convolution layers.
Since the shape of the input before flattening is a 64 channel 12x12 feature map, then the total size of the feature is 64*12*12 = 9216.
Input/Output relation for conv2d and max_pool2d operations

def forward(self, x):
    """ For each line which changes the feature shape additional comment
        indicates <input_shape> -> <output_shape> """
    x = self.conv1(x)                # [1, 28, 28] -> [32, 26, 26]
    x = F.relu(x)
    x = self.conv2(x)                # [32, 26, 26] -> [64, 24, 24]
    x = F.relu(x)
    x = F.max_pool2d(x, 2)           # [64, 24, 24] -> [64, 12, 12]
    x = self.dropout1(x)
    x = torch.flatten(x, 1)          # [64, 12, 12] -> [9216]
    x = self.fc1(x)                  # [9216] -> [128]
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x)                  # [128] -> [10]
    output = F.log_softmax(x, dim=1)
    return output


Instruction: what the difference between att_mask and key_padding_mask in MultiHeadAttnetion

Input:
What the difference between att_mask and key_padding_mask in MultiHeadAttnetion of pytorch:

key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored

attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.

Thanks in advance.

Output:
The key_padding_mask is used to mask out positions that are padding, i.e., after the end of the input sequence. This is always specific to the input batch and depends on how long are the sequence in the batch compared to the longest one. It is a 2D tensor of shape batch size × input length.
On the other hand, attn_mask says what key-value pairs are valid. In a Transformer decoder, a triangle mask is used to simulate the inference time and prevent the attending to the "future" positions. This is what att_mask is usually used for. If it is a 2D tensor, the shape is input length × input length. You can also have a mask that is specific to every item in a batch. In that case, you can use a 3D tensor of shape (batch size × num heads) × input length × input length. (So, in theory, you can simulate key_padding_mask with a 3D att_mask.)


Instruction: Argmax indexing in pytorch with 2 tensors of equal shape

Input:
Summarize the problem
I am working with high dimensional tensors in pytorch and I need to index one tensor with the argmax values from another tensor. So I need to index tensor y of dim [3,4] with the results from the argmax of tensor xwith dim [3,4]. If tensors are:
import torch as T
# Tensor to get argmax from
# expected argmax: [2, 0, 1]
x = T.tensor([[1, 2, 8, 3],
              [6, 3, 3, 5],
              [2, 8, 1, 7]])

# Tensor to index with argmax from preivous
# expected tensor to retrieve [2, 4, 9]
y = T.tensor([[0,  1,  2,  3],
              [4,  5,  6,  7],
              [8,  9, 10, 11]])
# argmax
x_max, x_argmax = T.max(x, dim=1)

I would like an operation that given the argmax indexes of x, or x_argmax, retrieves the values in tensor y in the same indexes x_argmax indexes.
Describe what you’ve tried
This is what I have tried:
# What I have tried
print(y[x_argmax])
print(y[:, x_argmax])
print(y[..., x_argmax])
print(y[x_argmax.unsqueeze(1)])

I have been reading a lot about numpy indexing, basic indexing, advanced indexing and combined indexing. I have been trying to use combined indexing (since I want a slice in first dimension of the tensor and the indexes values on the second one). But I have not been able to come up with a solution for this use case.

Output:
You are looking for torch.gather:
idx = torch.argmax(x, dim=1, keepdim=true)  # get argmax directly, w/o max
out = torch.gather(y, 1, idx)

Resulting with
tensor([[2],
        [4],
        [9]])


Instruction: PyTorch: batching from multiple datasets

Input:
I have multiple datasets that I want to use in the training. I want each batch to be from one dataset but have batches from (possibly) all of the datasets in each epoch.
Merging the datasets into one simple Dataset object and using the default Dataloader leads to having samples from different datasets in one batch.
My own guess is to have a separate Dataset object for each dataset and override the Dataloader or the sampler, but I don't know how to do it.

Output: I think the best way to solve your problem is to have a single merged dataset with a single data loader, but have a custom BatchSampler that yields indices based on the different datasets inside the merged dataset.


Instruction: Mask for neural network training

Input:
I am training a neural network and I found a dataset of pictures taken by drones. It says that it has 20 classes that should be detected. However the masks all looks like this (I'm sorry the image is really dark!):
This is my mask
When I try to train the network it always says that the image only has 3 channels. (I figured it was probably RGB). The problem is that my network expects a 20 channels input for the mask (one for each category to detect like tree, car, human, etc.) Is there a way for me to transform the image in a 20 channels image? I looked if the mask only had 20 different values for the pixel colors but it has 254 so I do not think there is something to do with that...
Thank you! (It is my first question on StackOverflow so if there is a problem in the question just tell me! :-) )

Output: I am not sure I understand your question correctly but yes, there is a way to upsample the image to make it a 20 channel input. All you have to do is take the original image (lets assume its size is [batch, height, width, #channels]) and convolve it with a kernel ([#channels, height, width, 20]) while keeping the padding mode 'same'. This would convert your 3 channel image into a 20 channel array (wouldn't call it an image anymore).


Instruction: PyTorch arguments not valid on android

Input:
I want to use this model in my android app. But when I start the app it falls with an error. The model works fine on my PC.
To Reproduce
Steps to reproduce the behavior:

Clone repository and use instructions in readme to run the model.
Add code below to save the model

            traced_script_module = torch.jit.trace(i2d, data)
            traced_script_module.save("i2d.pt")

I used PyTorch Android DemoApp link to run the model on android.

Error:
 E/AndroidRuntime: FATAL EXCEPTION: ModuleActivity
    Process: com.hypersphere.depthvisor, PID: 4765
    com.facebook.jni.CppException:
    Arguments for call are not valid.
    The following variants are available:

      aten::upsample_bilinear2d(Tensor self, int[2] output_size, bool align_corners) -> (Tensor):
      Expected at most 3 arguments but found 5 positional arguments.

      aten::upsample_bilinear2d.out(Tensor self, int[2] output_size, bool align_corners, *, Tensor(a!) out) -> (Tensor(a!)):
      Argument out not provided.

    The original call is:
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\functional.py(3013): interpolate
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\functional.py(2797): upsample
    <ipython-input-1-e1d92bec6901>(75): _upsample_add
    <ipython-input-1-e1d92bec6901>(89): forward
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\modules\module.py(534): _slow_forward
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\modules\module.py(548): __call__
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\jit\__init__.py(1027): trace_module
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\jit\__init__.py(875): trace
    <ipython-input-12-19d2ccccece4>(16): <module>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3343): run_code
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3263): run_ast_nodes
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3072): run_cell_async
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\async_helpers.py(68): _pseudo_sync_runner
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(2895): _run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(2867): run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\zmqshell.py(536): run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\ipkernel.py(300): do_execute
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(545): execute_request
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(268): dispatch_shell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(365): process_one
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(748): run
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(787): inner
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\ioloop.py(743): _run_callback
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\ioloop.py(690): <lambda>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\events.py(88): _run
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\base_events.py(1786): _run_once
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\base_events.py(541): run_forever
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\platform\asyncio.py(149): start
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelapp.py(597): start
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\traitlets\config\application.py(664): launch_instance
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel_launcher.py(16): <module>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\runpy.py(85): _run_code
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\runpy.py(193): _run_module_as_main
    Serialized   File "code/__torch__/___torch_mangle_907.py", line 39
        _17 = ops.prim.NumToTensor(torch.size(_16, 2))
        _18 = ops.prim.NumToTensor(torch.size(_16, 3))
2020-06-29 23:50:09.536 4765-4872/com.hypersphere.depthvisor E/AndroidRuntime:     _19 = torch.upsample_bilinear2d(_15, [int(_17), int(_18)], False, None, None)
              ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        input = torch.add(_19, _16, alpha=1)
        _20 = (_6).forward(input, )

        at org.pytorch.NativePeer.initHybrid(Native Method)
        at org.pytorch.NativePeer.<init>(NativePeer.java:18)
        at org.pytorch.Module.load(Module.java:23)
        at com.hypersphere.depthvisor.MainActivity.analyzeImage(MainActivity.java:56)
        at com.hypersphere.depthvisor.MainActivity.analyzeImage(MainActivity.java:21)
        at com.hypersphere.depthvisor.AbstractCameraXActivity.lambda$setupCameraX$2$AbstractCameraXActivity(AbstractCameraXActivity.java:86)
        at com.hypersphere.depthvisor.-$$Lambda$AbstractCameraXActivity$KgCZmrRflavSsq5aSHYb53Fi-P4.analyze(Unknown Source:2)
        at androidx.camera.core.ImageAnalysisAbstractAnalyzer.analyzeImage(ImageAnalysisAbstractAnalyzer.java:57)
        at androidx.camera.core.ImageAnalysisNonBlockingAnalyzer$1.run(ImageAnalysisNonBlockingAnalyzer.java:135)
        at android.os.Handler.handleCallback(Handler.java:873)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:65)

Environment
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS:  Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.7
Is CUDA available: No
CUDA runtime version: 10.2.89
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.5.0
[pip3] torchvision==0.6.0
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              h74a9793_1
[conda] libmklml                  2019.0.5                      0
[conda] mkl                       2019.4                      245
[conda] mkl-service               2.3.0            py37hb782905_0
[conda] mkl_fft                   1.1.0            py37h45dec08_0
[conda] mkl_random                1.1.0            py37h675688f_0
[conda] numpy                     1.18.5           py37h6530119_0
[conda] numpy-base                1.18.5           py37hc3f5095_0
[conda] pytorch                   1.5.0           cpu_py37h9f948e0_0
[conda] torchvision               0.6.0                py37_cu102    pytorch

Android Studio 4.0
Device: Samsung s8 plus
Android version: 9

Output:
My pc PyTorch version was 1.5 and in dependences were 1.4. So solution is:
implementation 'org.pytorch:pytorch_android:1.5.0'
implementation 'org.pytorch:pytorch_android_torchvision:1.5.0'


Instruction: What's the best way to load two pretrained model partially from state-dict in Pytorch?

Input:
I am trying to load two separately trained models except for the last layer and want to train the last layer separately combining these two models. I defined e new nn.Module class and load these pretrained model inside that class and in the forward path tried to return the value before the last layer.
class New_net(nn.Module):
    def __init__(self):
        super(New_net, self).__init__()
        self.net1 = net1()
        self.net2 = net2()
        self.fc= nn.Linear(512, 2)
        self._initialize_weights()

    def _initialize_weights(self):
        checkpoint = torch.load('save_model/checkpoint_net1.t7')
        self.net1.load_state_dict(checkpoint['state_dict'])

        checkpoint = torch.load('save_model/checkpoint_net2.t7')
        self.net2.load_state_dict(checkpoint['state_dict'])

    def forward(self, x):
        x1 = self.net1(x)
        x2 = self.net2(x)
        x=torch.cat((x1,x2),dim=1)
        x=self.fc(x)
        return x

but it seems it is not loading the model accurately. What's the correct way to do that

Output:
I figured that. Instead of weight initialization, I did the following
#load net1 model partially
checkpoint = torch.load('save_model/checkpoint_net1.t7')
pretrained_dict=checkpoint['state_dict']

net1_dict=net.net1.state_dict()
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in net1_dict}
net1_dict.update(pretrained_dict)
net.net1.load_state_dict(net1_dict)

#load net2 model partially
checkpoint = torch.load('save_model/checkpoint_net2.t7')
pretrained_dict=checkpoint['state_dict']
net2_dict=net.net2.state_dict()
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in net2_dict}
net2_dict.update(pretrained_dict)
net.net2.load_state_dict(net2_dict)


Instruction: How can I load a partially trained model in PyTorch?

Input: Let's suppose that the model I'm loading has 4 layers (layer0, layer1, layer2, layer3) for simplicity. If I only wanted that model to be pretrained for say layer0 and layer1, but have randomly initialized parameters for layer2 and layer3, how would I be able to do that?

Output:
You can do it by freezing the layers you want it not to change(pretrained) and leave the other unfreezed(they will continue to train)
model_ft = models.resnet50(pretrained=True)
ct = 0
for child in model_ft.children():
    ct += 1
    if ct < 7:
        for param in child.parameters():
            param.requires_grad = False

This freezes layers 1-6 in the total 10 layers of Resnet50.


Instruction: PyTorch Multi Class Classification using CrossEntropyLoss - not converging

Input:
I am trying to get a simple network to output the probability that a number is in one of three classes. These are, smaller than 1.1, between 1.1 and 1.5 and bigger than 1.5. I am using cross entropy loss with class labels of 0, 1 and 2, but cannot solve the problem.
Every time I train, the network outputs the maximum probability for class 2, regardless of input. The lowest loss I seem to be able to achieve is 0.9ish. Any advice on where I am going wrong would be greatly appreciated!! All code is below.
class gating_net(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(gating_net, self).__init__()
        self.linear1 = nn.Linear(input_dim, 32)
        self.linear2 = nn.Linear(32, output_dim)

    def forward(self, x):
        # The original input (action) is used as the residual.
        x = F.relu(self.linear1(x))
        x = F.sigmoid(self.linear2(x))
        return x

learning_rate = 0.01
batch_size = 64
epochs = 500
test = 1

gating_network = gating_net(1,3)

optimizer = torch.optim.SGD(gating_network.parameters(), lr=learning_rate, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=20, verbose=True)

for epoch in range (epochs):
    input_ = []
    label_ = []

    for i in range (batch_size):
        scale = random.randint(10,20)/10

        input = scale
        if scale < 1.1:
            label = np.array([0])
        elif 1.1 < scale < 1.5:
            label = np.array([1])
        else:
            label = np.array([2])

        input_.append(np.array([input]))
        label_.append(label)

    optimizer.zero_grad()

    # get output from the model, given the inputs
    output = gating_network.forward(torch.FloatTensor(input_))
    old_label  = torch.FloatTensor(label_)

    # get loss for the predicted output
    loss = nn.CrossEntropyLoss()(output, old_label.squeeze().long())

    # get gradients w.r.t to parameters
    loss.backward()
    # update parameters
    optimizer.step()
    scheduler.step(loss)

    print('epoch {}, loss {}'.format(epoch, loss.item()))

    if loss.item() < 0.01:
        print("########## Solved! ##########")
        torch.save(mod_network.state_dict(), './supervised_learning/run_{}.pth'.format(test))
        break

    # save every 500 episodes
    if epoch % 100 == 0:
        torch.save(gating_network.state_dict(), './run_{}.pth'.format(test))

Output:
Your code generates training data every epochs (which is also every batch in this case). This is very redundant, but it doesn't mean the code won't work. However one thing that does influence the training is the imbalance of training data between classes. With your code majority of the training data is always labeled 2. So intuitively, your network will always learn more about class 2. That is why with the very small epoch of 500, the network classified all the classes as 2, because it is a fast and easy way to lower the loss. However when the network can't get the loss much lower by applying the knowledge about label 2, it will learn about 1 and 0 too. So it is possible to train the network, though not so efficient.
Continuing from the previous issue, using ReduceLROnPlateau also isn't efficient because at the time the network starts to learn about label 0 and 1, the learning rate is already to small(intuitively speaking).  Doesn't mean it is untrainable, but it will probably take a lot of time.
CrossEntropyLoss calculates LogSoftmax internally, so having Sigmoid at the end of the network means you have a Softmax layer right after Sigmoid layer, which is probably not what you want. I think the network isn't necessarily 'wrong', but it will be much harder to train.
actually scale 1.1 is being labeled 2 because you have <1.1 and >1.1.

TL;DR
Get rid of sigmoid and scheduler.
I was able to get Solved! somewhere around 15000 epoch (with learning rate and batch size as same as your code).


Instruction: How can I visualize network architectures effectively?

Input:
Is there some sort of software that can do so? Specifically, I would like to visualize Resnet18. Is there no other way other than to just draw it myself? Here is an example of what I want to see:
Sample Architecture Visualization

Output: You can use this one : http://alexlenail.me/NN-SVG/LeNet.html . It lets you visualize neural networks by letting you modify several parameters and finally lets you export the architectures as SVG files. You can also choose between 3 visualization styles, namely FCNN, LeNet & AlexNet.


Instruction: Process got killed when building docker image on GCP

Input:
I am a newbie of cloud computing and struggling to deploy my web app on google cloud platform for the first time. When I was building docker image, which means I ran the following code
docker build -t gcr.io/${PROJECT_ID}/insurance-streamlit:v1 .

the process got killed, showing the following error:
Killed
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 137

I guessed my app might be too large because the weight file has been more than 100MB. So is there a way to fix it? Please tell me the details, thanks in advance!
PS: my Dockerfile is as below:
FROM python:3.7
RUN pip install virtualenv
ENV VIRTUAL_ENV=/venv
RUN virtualenv venv -p python3
ENV PATH="VIRTUAL_ENV/bin:$PATH"

WORKDIR /app
ADD . /app

# Install dependencies
RUN pip install -r requirements.txt

# copying all files over
COPY . /app

# Expose port
ENV PORT 8501

# cmd to launch app when container is run
CMD streamlit run app.py

# streamlit-specific commands for config
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
RUN mkdir -p /root/.streamlit
RUN bash -c 'echo -e "\
[general]\n\
email = \"\"\n\
" > /root/.streamlit/credentials.toml'

RUN bash -c 'echo -e "\
[server]\n\
enableCORS = false\n\
" > /root/.streamlit/config.toml

And my requirements.txt is like:
albumentations==0.4.5
numpy==1.19.0
opencv-python==4.1.0.25
opencv-python-headless==4.2.0.34
pandas==1.0.5
Pillow==7.1.2
streamlit==0.62.0
torch==1.4.0
torchvision==0.5.0
matplotlib

Output:
I found that to build your Docker image you should have enough disk space and Python 3.7 installed, also there's a typo at your Docker file - no single quotes ' at the end of the last string. Beside of that, everything looks good and runs.
Please find my steps below:

Enable Google Container Registry API

Create VM instance:

gcloud compute instances create instance-4 --zone=europe-west3-a --machine-type=e2-medium --image=ubuntu-1804-bionic-v20200701 --image-project=ubuntu-os-cloud --boot-disk-size=50GB

Follow documentation Pushing and pulling images.
Install Python 3.7:

sudo apt install python3.7

Build Docker image:

docker build -t gcr.io/test-prj/testimage:v1 .
...
Step 16/16 : RUN bash -c 'echo -e "[server]\nenableCORS = false\n" > /root/.streamlit/config.toml
 ---> Running in 57502f97cfbe
/bin/sh: 1: Syntax error: Unterminated quoted string
The command '/bin/sh -c bash -c 'echo -e "[server]\nenableCORS = false\n" > /root/.streamlit/config.toml' returned a non-zero code: 2

Change last string of the Docker file:

" > /root/.streamlit/config.toml'

Build Docker image again:

docker build -t gcr.io/test-prj/testimage:v1 .
...
Step 16/16 : RUN bash -c 'echo -e "[server]\nenableCORS = false\n" > /root/.streamlit/config.toml'
 ---> Running in c1c1f81a2d09
Removing intermediate container c1c1f81a2d09
 ---> 24b6609de554
Successfully built 24b6609de554
Successfully tagged gcr.io/test-prj/testimage:v1
$ docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
gcr.io/test-prj/testimage             v1                  24b6609de554        14 minutes ago      3.87GB

Push Docker image to the registry:

gcloud docker -- push gcr.io/test-prj/testimage:v1

Create new VM instance and deploy image:

gcloud compute instances create-with-container instance-5 --zone=europe-west3-a --machine-type=e2-medium --image=cos-stable-81-12871-148-0 --image-project=cos-cloud --boot-disk-size=50GB --container-image=gcr.io/test-prj/testimage:v1

Check status of Docker container:

instance-5 ~ $ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                                  PORTS               NAMES
e21b80dc0de7        gcr.io/test-prj/testimage:v1   "/bin/sh -c 'streaml…"   28 seconds ago      Restarting (2) Less than a second ago                       klt-instance-5-caqx

and it doesn't look very good.

Stop container:

instance-5 ~ $docker stop e21b80dc0de7

Follow the documentation and run container interactively:

instance-5 ~ $docker run --name test -it gcr.io/test-prj/testimage:v1
Usage: streamlit run [OPTIONS] TARGET [ARGS]...

Error: Invalid value: File does not exist: app.py

no surprise because I don't have app.py.
After that, I've added some dummy app.py, rebuild and finally it works:
instance-6 ~ $ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS               NAMES
1de2e8ded5d8        gcr.io/test-prj/testimage:v2   "/bin/sh -c 'streaml…"   7 minutes ago       Up 7 minutes                           klt-instance-6-yezv


Instruction: What is the role of pytorch dataset.imagefolder's loader option?

Input:
Loader option
What is the role of PyTorch dataset.imagefolder's loader option?
Actually, now I have a 10,000 images in a folder, but I want to use only 100 images. Now I load all the image and subset dataset by index.
from torch.utils.data import Subset
a = dset.ImageFolder(root=path_F,transform=transform)
index = list(range(0,100))
b = Subset(a,index)

Can loader option make my code simple?

Output:
loader is responsible for loading the image given path. By default PyTorch uses pillow and it's Image.open(path) functionality see docs.
You can specify custom loading like this (load and rotate by 45 degrees):
import torchvision
from PIL import Image

def loader(path):
    return Image.open(path).rotate(45)

dataset = torchvision.datasets.ImageFolder("images", loader=loader)

So no, you shouldn't use it for choosing images.
You could, in principle, use is_valid_file argument to do it like this though:
class Chooser:
    def __init__(self):
        self._counter = -1

    def __call__(self, path):
        self._counter += 1
        return self._counter < 100

dataset = torchvision.datasets.ImageFolder("images", is_valid_file=Chooser())

I would highly discourage this approach though as the intent isn't immediately clear and it will go through whole dataset checking each image. Your current way is the way to go.


Instruction: Multi class classification - RuntimeError: 1D target tensor expected, multi-target not supported

Input:
My goal is to build a multi-class image classifier using Pytorch and based on the EMNIST dataset (black and white pictures of letters).
The shape of my training data X_train is (124800, 28, 28).
The shape of the original target variables y_train is (124800, 1), however I created a one-hot encoding so that now the shape is (124800, 26).
The model that I am building should have 26 output variables, each representing the probability of one letter.
I read in my data as follows:
import scipy .io
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

Then, I created a one-hot-encoding as follows:
y_train_one_hot = np.zeros([len(y_train), 27])

for i in range (0, len(y_train)):
    y_train_one_hot[i, y_train[i][0]] = 1

y_train_one_hot = np.delete(y_train_one_hot, 0, 1)

I create the dataset with:
train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train_one_hot))

batch_size = 128
n_iters = 3000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

And then I build my model as follows:
class CNNModel(nn.Module):

    def __init__(self):
        super(CNNModel, self).__init__()

        # Convolution 1
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)
        self.relu1 = nn.ReLU()

        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(2,2)

        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)
        self.relu2 = nn.ReLU()

        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        # Fully connected 1 (readout)
        self.fc1 = nn.Linear(32 * 4 * 4, 26)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x.float())
        out = self.relu1(out)

        # Max pool 1
        out = self.maxpool1(out)

        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)

        # Max pool 2
        out = self.maxpool2(out)

        # Resize
        # Original size: (100, 32, 7, 7)
        # out.size(0): 100
        # New out size: (100, 32*7*7)
        out = out.view(out.size(0), -1)

        # Linear function (readout)
        out = self.fc1(out)

        return out

model = CNNModel()

criterion = nn.CrossEntropyLoss()

learning_rate = 0.01

optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)

And then I train the model as follows:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, when I run this, I get the following error:
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-c26c43bbc32e> in <module>()
     21
     22         # Calculate Loss: softmax --> cross entropy loss
---> 23         loss = criterion(outputs, labels)
     24
     25         # Getting gradients w.r.t. parameters

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    930     def forward(self, input, target):
    931         return F.cross_entropy(input, target, weight=self.weight,
--> 932                                ignore_index=self.ignore_index, reduction=self.reduction)
    933
    934

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2315     if size_average is not None or reduce is not None:
   2316         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2317     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2318
   2319

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

I expect that I do something wrong when I initialize/use my loss function. What can I do so that I can start training my model?

Output:
If you are using crossentropy loss you shouldn't one-hot encode your target variable y.
Pytorch crossentropy expects just the class indices as target not their one-hot encoded version.
To cite the doc https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html  :
This criterion expects a class index in the range [0, C-1] as the target for each value of a 1D tensor of size minibatch;


Instruction: PyTorch differentiable mask

Input:
How would I go about blacking out a portion of an image or feature map such that AutoGrad can backprop through the operation?
Specifically I want to black out everything except for n layers of border pixels. So if we consider a single channel of the feature map which looks like:
[
  [1, 1, 1, 1],
  [1, 1, 1, 1],
  [1, 1, 1, 1],
  [1, 1, 1, 1],
]

I set a constant n=1 so my operation does the following to the input:
[
  [1, 1, 1, 1],
  [1, 0, 0, 1],
  [1, 0, 0, 1],
  [1, 1, 1, 1],
]

In my case I'd be doing it to a multi channel feature map and all channels would be treated the same way.
If possible, I want to do it in a functional manner.

Output:
Considering the comments you added, i.e. that you don't need the output to be differentiable wrt. to the mask (said differently, the mask is constant), you could just store the indices of the 1s in the mask and act only on the corresponding elements of whatever Tensor you're considering. Or if you don't want to deal with fancy indexing, you could just keep the mask as a Tensor of 0s and 1s and do an element-wise multiplication of it with whatever Tensor you're considering. Or, if you truly just need to compute a loss along just the border pixels, just extract the first and last row, and first and last column, and avoid double-counting the corners. This latter solution is essentially just the first solution recast in a special case.
To address the question in your comment to my answer:
x = torch.tensor([[1.0,2,3],[4,5,6]], requires_grad = True)
print(x[:,0])

gives
tensor([1., 4.], grad_fn=<SelectBackward>)

, so we see that slicing does not mess with the autograd engine (it's still tracking the contribution to the gradient). It is not too surprising that this works automatically; slicing can be viewed as the (mathematical) function that of projecting onto a subspace of R^n, for which it's easy to compute the gradient.


Instruction: Change expectation to multi dimensional rather than 1D in nn.crossentropyloss

Input:
My goal is to do multi class image classification in Pytorch using the EMNIST dataset.
As a loss function, I would like to use Multi-Class Cross-Entropy Loss.
Currently, I define my loss function as follows:
criterion = nn.CrossEntropyLoss()

I train my model as follows:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, the error that I get is:
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-c26c43bbc32e> in <module>()
     21
     22         # Calculate Loss: softmax --> cross entropy loss
---> 23         loss = criterion(outputs, labels)
     24
     25         # Getting gradients w.r.t. parameters

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

My CNN outputs 26 variables and my target variables are also 26D.
How can I change my code so that nn.crossentropyloss() expects a 26D input rather than 1D?

Output:
nn.CrossEntropy()(input, target) expects input to be an one-hot vector with size batchsize X num_classes, and target to be the id of true class with size batchsize.
So in short, you can change your target with target = torch.argmax(target, dim=1) to have it fit nn.CrossEntropy().


Instruction: Get variable from a Numpy array

Input:
For a image classification problem with Pytorch, I read in my data as follows:
import scipy .io
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

X_test = data ['test'][0, 0]['images'][0, 0]
X_test = X_test.reshape((-1,28,28), order = 'F')

y_test = data ['test'][0, 0]['labels'][0, 0]

I aim to create a dataset, using:
train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))

Currently, when I run an instance of y_train, the output is an array:
y_train[0]
>>> array([23], dtype=uint8)

However, I want train_dataset to contain only the number that's inside the array at the 0th index (in this case 23) instead of the entire array.
How can I change my code so that the TensorDataset that is created contains only the first element of the array of y_train, instead of the entire array?

Output:
You can use np.squeeze() to get rid of dimensions of the data with size 1. If you mean to remove a specific dimension, pass it to squeeze.
import numpy as np

arr = np.random.randn(1, 2, 1, 3, 1)
arr.squeeze().shape # (2, 3)

arr.squeeze(2).shape # (1, 2, 3, 1)


Instruction: How to disable TOKENIZERS_PARALLELISM=(true | false) warning?

Input:
I use pytorch to train huggingface-transformers model, but every epoch, always output the warning:
The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)

How to disable this warning?

Output:
Set the environment variable to the string "false"
either by
TOKENIZERS_PARALLELISM=false

in your shell
or by:
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"

in the Python script


Instruction: Unexpected model output running Onnx model in Unity using Barracuda

Input:
Context
I am trying to use a pre-trained model in ONNX format to do inference on image data in Unity. The model is linked to the executing component in Unity as an asset called modelAsset. I am using Barracuda version 1.0.0 for this and executing the model as follows:
// Initialisation
this.model = ModelLoader.Load(this.modelAsset);
this.worker = WorkerFactory.CreateWorker(WorkerFactory.Type.CSharpBurst, model);

// Loop
Tensor tensor = new Tensor(1, IMAGE_H, IMAGE_W, 3, data);
worker.Execute(tensor);
Tensor modelOutput = worker.PeekOutput(OUTPUT_NAME);

The data going into the input tensor (of which the model has only 1) is image data of h * w with 3 channels for RGB values between -0.5 and 0.5. The model has multiple outputs which I retrieve in the last line shown above.
Expected behavior
Using the same input data, the PyTorch model and converted ONNX model produce the same output data in Python (ONNXRuntime and PyTorch) as in Barracuda in Unity.
Problem
In python both the ONNX and PyTorch model produce the same output. However, the same ONNX model running in Barracuda produces a different output. The difference is mainly that we expect a heatmap but Barracuda consistently produces values somewhere between 0.001 and -0.0004 in these patterns:

This makes it almost seem like the model weights are not properly loaded.
What we found
When converting to ONNX as per the Barracuda manual we found that if we did not set the model to inference mode in the PyTorch net before conversion (link), these same, incorrect, results were generated by ONNXRuntime in Python. In other words, it looks like this inference mode is saved in the ONNX model and is recognized by ONNXRuntime in Python but not in Barracuda.
Our question
In general:

How do we get this model in Barracuda in Unity to produce the same results as ONNXRuntime/PyTorch in Python?

And potentially:

How does the inference mode get embedded into the ONNX file and how is it used in ONNXRuntime vs Barracuda?

Output:
So it turned out that there were 2 problems.
First, the input data had been orchestrated according to the ONNX model dimensions, however, Barracuda expects differently oriented data. "The native ONNX data layout is NCHW, or channels-first. Barracuda automatically converts ONNX models to NHWC layout." So our data was flattened into an array similar to the Python implementation which created the first mismatch.
Secondly, the Y-axis of the input image was inverted, making the model unable to recognize any people.
After correcting for these issues, the implementation works fine!


Instruction: Using torch.cat on list of tensors

Input:
I have a list of torch tensors
list_tensor = [tensor([[1, 2, 3],
                       [3, 4, 5]]),
               tensor([[4, 5, 6],
                       [6, 4, 3]]),
               tensor([[4, 2, 1],
                       [3, 3, 1]]),
               tensor([[1, 4, 5],
                       [3, 1, 0]]),
               tensor([[1, 3, 3],
                       [2, 2, 2]])]

I want to do a cross validation on this set, so I want to consider four tensors as training, and keep 1 for testing - and I want to do this for len(list_tensor) times.
So I thought of doing,
for num in range(1, len(list_tensor) + 1):
       train_x = torch.cat((list_tensor[:num], list_tensor[num:]))

The problem is I cannot use lists for a torch.cat operation because both  list_tensor[:num] and  list_tensor[num:] return lists. For example, for num = 1,
list_tensor[:num] = [tensor([[1, 2, 3], [3, 4, 5]])]
list_tensor[num:] = [tensor([[4, 5, 6], [6, 4, 3]]), tensor([[4, 2, 1],[3, 3, 1]]), tensor([[1, 4, 5],
                      [3, 1, 0]]), tensor([[1, 3, 3], [2, 2, 2]])]

How do I perform torch.cat on this?

Output:
I found a work around without using reduce.
train_x = torch.cat((torch.cat(list_tensor[:num+1]),torch.cat(list_tensor[num+1:])))
Basically concatenate all tensors in the individual list, this returns a torch.tensor object, then use torch.cat on both.


Instruction: Does tf.math.reduce_max allows gradient flow like torch.max?

Input:
I am trying to build a multi-label binary classification model in Tensorflow. The model has a tf.math.reduce_max operator between two layers (It is not Max Pooling, it's for a different purpose).
And the number of classes is 3.
I am using Binary Cross Entropy loss and using Adam optimizer.
Even after hours of training, when I check the predictions, all the predictions are in the range 0.49 to 0.51.
It seems that the model is not learning anything and is making random predictions, which is making me think that using a tf.math.reduce_max function may be causing the problems.
However, I read on the web that the torch.max function allows back propagation of gradients through it.
When I checked the Graph in Tensorboard, I saw that the graph is showing unconnected at the tf.math.reduce_max operator.
SO, does this operator allows gradients ot back propagate through it?
EDIT :
Addin the code
input_tensor = Input(shape=(256, 256, 3))
base_model_toc = VGG16(input_tensor=input_tensor,weights='imagenet',pooling=None, include_top=False)

x = base_model.output

x = GlobalAveragePooling2D()(x)

x = tf.math.reduce_max(x,axis=0,keepdims=True)

x = Dense(1024,activation='relu')(x)

output_1 = Dense(3, activation='sigmoid')(x)

model_a = Model(inputs=base_model_toc.input, outputs=output_1)

for layer in base_model.layers:
    layer.trainable = True

THe tf.math.reduce_max is done along axis = 0 becasue that is what needs to be done in this model
Optimizer that I am using is Adam with initial learning rate 0.00001

Output:
Yes, tf.math.reduce_max does allow gradients to flow. It is easy to check (this is TensorFlow 2.x but it is the same result in 1.x):
import tensorflow as tf

with tf.GradientTape() as tape:
    x = tf.linspace(0., 2. * 3.1416, 10)
    tape.watch(x)
    # A sequence of operations involving reduce_max
    y = tf.math.square(tf.math.reduce_max(tf.math.sin(x)))
# Check gradients
g = tape.gradient(y, x)
print(g.numpy())
# [ 0.         0.         0.3420142 -0.        -0.        -0.
#  -0.         0.         0.         0.       ]

As you can see, there is a valid gradient for y with respect to x. Only one of the values is not zero, because it is the value that then resulted in the maximum value, so it is the only value in x that affects the value of y. This is the correct gradient for the operation.


Instruction: Why PyTorch model takes multiple image size inside the model?

Input:
I am using a simple object detection model in PyTorch and using a Pytoch Model for Inferencing.
When I am using a simple iterator over the code
for k, image_path in enumerate(image_list):
    image = imgproc.loadImage(image_path)
    print(image.shape)
    with torch.no_grad():
        y, feature = net(x)
    result = image.cuda()

It prints our variable sized images such as
torch.Size([1, 3, 384, 320])

torch.Size([1, 3, 704, 1024])

torch.Size([1, 3, 1280, 1280])

So When I am using Batch Inferencing using a DataLoader applying the same transformation the code is not running.
However, when I am resizing all the images as 600.600 the batch processing runs successfully.
I am having Two Doubts,
First why Pytorch is capable of inputting dynamically sized inputs in Deep Learning Model and Why dynamic sized input is failing in Batch Processing.

Output:
PyTorch has what is called a Dynamic Computational Graph (other explanation).
It allows the graph of the neural network to dynamically adapt to its input size, from one input to the next, during training or inference.
This is what you observe in your first example: providing an image as a Tensor of size [1, 3, 384, 320] to your model, then another one as a Tensor of size [1, 3, 384, 1024], and so forth, is completely fine, as, for each input, your model will dynamically adapt.
However, if your input is a actually a collection of inputs (a batch), it is another story. A batch, for PyTorch, will be transformed to a single Tensor input with one extra dimension. For example, if you provide a list of n images, each of the size [1, 3, 384, 320], PyTorch will stack them, so that your model has a single Tensor input, of the shape [n, 1, 3, 384, 320].
This "stacking" can only happen between images of the same shape. To provide a more "intuitive" explanation than previous answers, this stacking operation cannot be done between images of different shapes, because the network cannot "guess" how the different images should "align" with one another in a batch, if they are not all the same size.
No matter if it happens during training or testing, if you create a batch out of images of varying size, PyTorch will refuse your input.
Several solutions are usually in use: reshaping as you did, adding padding (often small or null values on the border of your images) to extend your smaller images to the size of the biggest one, and so forth.