Instruction: Interpreting the output tokenization of BERT for a given word

Input:
from bert_embedding import BertEmbedding
bert_embedding = BertEmbedding(model='bert_12_768_12', dataset_name='wiki_multilingual_cased')
output = bert_embedding("any")

I need clarification on the output of mBERT embeddings. I'm aware that WordPiece tokenization is used to break up the input text. Also I observed that on providing a single word (say "any") as input, the output has length equal to the number of characters in the input (in our case, 3). output[i] is a tuple of lists where the first list contains the character at ith  position with the 'unknown' token preceding and following it as different elements in the array. Following this are three (= length of the input word) arrays (embeddings) of size 768 each. Why does the output seem to be tokenized character-wise (rather than wordpiece tokenized)?
Also found out the output form changes when the input is given in a list as:bert_embedding(["any"]). The output now is a single tuple with ['[UNK]', 'state', '[UNK]'] as the first element followed by three different embeddings conceivably corresponding to the three tokens listed above.
If I need the embedding of the last subword (not simply of the last character or the whole word) for a given input word, how do I access it?

Output: Checked their github page. About the input format: YES it is expected as a list (of strings). Also this particular implementation provides token ( = word ) level embeddings; so subword level embedings can't be retrieved directly although it provides a choice on how the word embeddings should be derived from their subword components ( by taking avg which is default or taking sum or just the last subword embedding). Refer to the Hugggingface interface for BERT for a finer control over how the embeddings are taken e.g. from the different layers and using which operations.


Instruction: Understanding input and output size for Conv2d

Input:
I'm learning image classification using PyTorch (using CIFAR-10 dataset) following this link.
I'm trying to understand the input & output parameters for the given Conv2d code:
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

My conv2d() understanding (Please correct if I am wrong/missing anything):

since image has 3 channels that's why first parameter is 3.
6 is no of filters (randomly chosen)
5 is kernel size (5, 5) (randomly chosen)
likewise we create next layer (previous layer output is input of this layer)
Now creating a fully connected layer using linear function:
self.fc1 = nn.Linear(16 * 5 * 5, 120)

16 * 5 * 5: here 16 is the output of last conv2d layer, But what is 5 * 5 in this?.
Is this kernel size ? or something else? How to know we need to multiply by 5*5 or 4*4 or 3*3.....
I researched & got to know that since image size is 32*32, applying max pool(2) 2 times, so image size would be 32 -> 16 -> 8, so we should multiply it by last_ouput_size * 8 * 8 But in this link its 5*5.
Could anyone please explain?

Output:
These are the dimensions of the image size itself (i.e. Height x Width).
Unpadded convolutions
Unless you pad your image with zeros, a convolutional filter will shrink the size of your output image by filter_size - 1 across the height and width:

3-filter takes a 5x5 image to a (5-(3-1) x 5-(3-1)) image
Zero padding preserves image dimensions

You can add padding in Pytorch by setting Conv2d(padding=...).
Chain of transformations
Since it has gone through:

Layer
Shape Transformation

one conv layer (without padding)
(h, w) -> (h-4, w-4)

a MaxPool
-> ((h-4)//2, (w-4)//2)

another conv layer (without padding)
-> ((h-8)//2, (w-8)//2)

another MaxPool
-> ((h-8)//4, (w-8)//4)

a Flatten
-> ((h-8)//4 * (w-8)//4)

We go from the original image size of (32,32) to (28,28) to (14,14) to (10,10) to (5,5) to (5x5).

To visualise this you can use the torchsummary package:
from torchsummary import summary

input_shape = (3,32,32)
summary(Net(), input_shape)

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1            [-1, 6, 28, 28]             456
         MaxPool2d-2            [-1, 6, 14, 14]               0
            Conv2d-3           [-1, 16, 10, 10]           2,416
         MaxPool2d-4             [-1, 16, 5, 5]               0
            Linear-5                  [-1, 120]          48,120
            Linear-6                   [-1, 84]          10,164
            Linear-7                   [-1, 10]             850
================================================================


Instruction: PyTorch: concat and flatten inputs of different shape

Input:
I have multiple inputs of different shape: (7,), (), (6,), how can I concatenate and flatten them to a single flatten input. My desired output shape is (14,).
For example: arr1= [1, 2, 3], arr2=6, arr3=[6,7], output=[1,2,3,6,6,7]. Although I can use multiple numpy.append, but that would be ugly.

Output:
You can use torch.cat:
import torch

arr1 = torch.tensor([1, 2, 3])
arr2 = torch.tensor([6])
arr3 = torch.tensor([6,7])

torch.cat((arr1,arr2,arr3))
>>> tensor([1, 2, 3, 6, 6, 7])


Instruction: Confusion when displaying an image from matplotlib.pyplot to tensorflow

Input:
I have this error: TypeError: Invalid shape (28, 28, 1) for image data
Here is my code:
import torch
import torchvision
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import torch.nn as nn
import torch.nn.functional as F
from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
from torchvision.utils import make_grid
from torch.utils.data.dataloader import DataLoader
from torch.utils.data import random_split
%matplotlib inline

# Load dataset

!wget www.di.ens.fr/~lelarge/MNIST.tar.gz
!tar -zxvf MNIST.tar.gz

from torchvision.datasets import MNIST

dataset = MNIST(root = './', train=True, download=True, transform=ToTensor())
#val_data = MNIST(root = './', train=False, download=True, transform=transform)

image, label = dataset[0]
print('image.shape:', image.shape)
plt.imshow(image.permute(1, 2, 0), cmap='gray') # HELP WITH THIS LINE
print('Label:', label)

I know that the pytorch does processing via this way: C x H x W,
and that matplotlib does it this way: H x W x C, yet when I change it to  matplotlib's way, it gives me an error here. Am I missing something? Why does this happen?

Output:
plt.imshow() expects 2D or 3D arrays. If the array has 3 dimensions then the last dimension should be 3 or 4. In your case the array has shape (28,28,1) and this is considered as a 3D array.
So the last dimension should be squeezed out in order to match imshow()'s requirements.
plt.imshow(np.squeeze(image.permute(1, 2, 0), axis = 2), cmap='gray')


Instruction: Gradients vanishing despite using Kaiming initialization

Input:
I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by stacking 100 such conv and activation blocks on top of each other), I noticed that the output I am getting values of the order of  10^(-10). Is this normal, considering I am stacking upto 100 layers. Adding a small bias to each layer fixes the problem. But in Kaiming initialization the biases are supposed to be zero.
Here is the conv block code
from collections import Iterable

def convBlock(
    input_channels, output_channels, kernel_size=3, padding=None, activation="prelu"
):
    """
    Initializes a conv block using Kaiming Initialization
    """
    padding_par = 0
    if padding == "same":
        padding_par = same_padding(kernel_size)
    conv = nn.Conv2d(input_channels, output_channels, kernel_size, padding=padding_par)
    relu_negative_slope = 0.25
    act = None
    if activation == "prelu" or activation == "leaky_relu":
        nn.init.kaiming_normal_(conv.weight, a=relu_negative_slope, mode="fan_in")
        if activation == "prelu":
            act = nn.PReLU(init=relu_negative_slope)
        else:
            act = nn.LeakyReLU(negative_slope=relu_negative_slope)
    if activation == "relu":
        nn.init.kaiming_normal_(conv.weight, nonlinearity="relu")
        act = nn.ReLU()
    nn.init.constant_(conv.bias.data, 0)
    block = nn.Sequential(conv, act)
    return block

def flatten(lis):
    for item in lis:
        if isinstance(item, Iterable) and not isinstance(item, str):
            for x in flatten(item):
                yield x
        else:
            yield item

def Sequential(args):
    flattened_args = list(flatten(args))
    return nn.Sequential(*flattened_args)

This is the test Code
ls=[]
for i in range(100):
    ls.append(convBlock(3,3,3,"same"))
model=Sequential(ls)

test=np.ones((1,3,5,5))
model(torch.Tensor(test))

And the output I am getting is
tensor([[[[-1.7771e-10, -3.5088e-10,  5.9369e-09,  4.2668e-09,  9.8803e-10],
          [ 1.8657e-09, -4.0271e-10,  3.1189e-09,  1.5117e-09,  6.6546e-09],
          [ 2.4237e-09, -6.2249e-10, -5.7327e-10,  4.2867e-09,  6.0034e-09],
          [-1.8757e-10,  5.5446e-09,  1.7641e-09,  5.7018e-09,  6.4347e-09],
          [ 1.2352e-09, -3.4732e-10,  4.1553e-10, -1.2996e-09,  3.8971e-09]],

         [[ 2.6607e-09,  1.7756e-09, -1.0923e-09, -1.4272e-09, -1.1840e-09],
          [ 2.0668e-10, -1.8130e-09, -2.3864e-09, -1.7061e-09, -1.7147e-10],
          [-6.7161e-10, -1.3440e-09, -6.3196e-10, -8.7677e-10, -1.4851e-09],
          [ 3.1475e-09, -1.6574e-09, -3.4180e-09, -3.5224e-09, -2.6642e-09],
          [-1.9703e-09, -3.2277e-09, -2.4733e-09, -2.3707e-09, -8.7598e-10]],

         [[ 3.5573e-09,  7.8113e-09,  6.8232e-09,  1.2285e-09, -9.3973e-10],
          [ 6.6368e-09,  8.2877e-09,  9.2108e-10,  9.7531e-10,  7.0011e-10],
          [ 6.6954e-09,  9.1019e-09,  1.5128e-08,  3.3151e-09,  2.1899e-10],
          [ 1.2152e-08,  7.7002e-09,  1.6406e-08,  1.4948e-08, -6.0882e-10],
          [ 6.9930e-09,  7.3222e-09, -7.4308e-10,  5.2505e-09,  3.4365e-09]]]],
       grad_fn=<PreluBackward>)

Output:
Amazing question (and welcome to StackOverflow)! Research paper for quick reference.
TLDR

Try wider networks (64 channels)
Add Batch Normalization after activation (or even before, shouldn't make much difference)
Add residual connections (shouldn't improve much over batch norm, last resort)

Please check this out in this order and give a comment what (and if) any of that worked in your case (as I'm also curious).
Things you do differently

Your neural network is very deep, yet very narrow (81 parameters per layer only!)

Due to above, one cannot reliably create those weights from normal distribution as the sample is just too small.
Try wider networks, 64 channels or more

You are trying much deeper network than they did

Section: Comparison Experiments

We conducted comparisons on a deep but efficient model with  14  weight
layers (actually 22 was also tested in comparison with Xavier)

That was due to date of release of this paper (2015) and hardware limitations "back in the days" (let's say)
Is this normal?
Approach itself is quite strange with layers of this depth, at least currently;

each conv block is usually followed by activation like ReLU and Batch Normalization (which normalizes signal and helps with exploding/vanishing signals)
usually networks of this depth (even of depth half of what you've got) use also residual connections (though this is not directly linked to vanishing/small signal, more connected to degradation problem of even deep networks, like 1000 layers)


Instruction: Legacy torchtext 0.9.0

Input: In the latest release of torchtext they moved a lot of features to torchtext.legacy, I want to do the same things like torchtext.legacy.data.Field and other features without using legacy, is that can be done? and how?

Output:
EDIT:

here is a release note about 0.9.0 version
here is the migration guide

Also, in the first link, there are counterparts for legacy Datasets.
Old answer (might be useful)
You could go for an alias, namely:
import torchtext.legacy as torchtext

But this is a bad idea for multiple reasons:

It became legacy for a reason (you can always change your existing code to torchtext.legacy.data.Field)
Very confusing - torchtext should torchtext, not torchtext.legacy
Unable to import torchtext as... torchtext - because this alias is already taken.

You could also do some workarounds like assigning torch.legacy.data.Field to torch.data.Field and what not, but it is a horrible idea.
If you want to work with this legacy stuff, you can always stay with smaller version of torchtext like 0.8.0 and this would make sense


Instruction: Serialiazation of pytorch state_dict changes after loading into new model instance

Input:
Why do the bytes obtained from serializing pytorch state_dicts change after loading the a state_dict into a new instance of the same model architecture?
Have a look:
import binascii
import torch.nn as nn
import pickle

lin1 = nn.Linear(1, 1, bias=False)
lin1s = pickle.dumps(lin1.state_dict())
print("--- original model ---")
print(f"hash of state dict: {hex(binascii.crc32(lin1s))}")
print(f"weight: {lin1.state_dict()['weight'].item()}")

lin2 = nn.Linear(1, 1, bias=False)
lin2.load_state_dict(pickle.loads(lin1s))
lin2s = pickle.dumps(lin2.state_dict())
print("\n--- model from deserialized state dict ---")
print(f"hash of state dict: {hex(binascii.crc32(lin2s))}")
print(f"weight: {lin2.state_dict()['weight'].item()}")

prints
--- original model ---
hash of state dict: 0x4806e6b6
weight: -0.30337071418762207

--- model from deserialized state dict ---
hash of state dict: 0xe2881422
weight: -0.30337071418762207

As you can see, the hashes of the (pickles of the) state_dicts are different whereas the weight is copied over correctly. I would assume that a state_dict from the new model equals the old one in every aspect. Seemingly, it does not, hence the different hashes.

Output:
This might be because pickle is not expected to produce a repr suitable for hashing (See Using pickle.dumps to hash mutable objects). It might be a better idea to compare keys, and then compare tensors stored in the dict-keys for equality/closeness.
Below is a rough implementation of that idea.
def compare_state_dict(dict1, dict2):
    # compare keys
    for key in dict1:
        if key not in dict2:
            return False

    for key in dict2:
        if key not in dict1:
            return False

    for (k,v) in dict1.items():
        if not torch.all(torch.isclose(v, dict2[k]))
            return False

    return True

However, if you would still like to hash a state-dict and avoid using comparisons like isclose above, you can use a function like below.
def dict_hash(dictionary):
    for (k,v) in dictionary.items():
        # it did not work without hashing the tensor
        dictionary[k] = hash(v)

    # dictionaries are not hashable and need to be converted to frozenset.
    return hash(frozenset(sorted(dictionary.items(), key=lambda x: x[0])))


Instruction: Flask app serving GPT2 on Google Cloud Run not persisting downloaded files?

Input:
I have a Flask app running on Google Cloud Run, which needs to download a large model (GPT-2 from huggingface). This takes a while to download, so I am trying to set up so that it only downloads on deployment and then just serves this up for subsequent visits. That is I have the following code in a script that is imported by my main flask app app.py:
import torch
# from transformers import GPT2Tokenizer, GPT2LMHeadModel
from transformers import AutoTokenizer, AutoModelWithLMHead
# Disable gradient calculation - Useful for inference
torch.set_grad_enabled(False)

# Check if gpu or cpu
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Load tokenizer and model
try:
    tokenizer = AutoTokenizer.from_pretrained("./gpt2-xl")
    model = AutoModelWithLMHead.from_pretrained("./gpt2-xl")
except Exception as e:

    print('no model found! Downloading....')

    AutoTokenizer.from_pretrained('gpt2').save_pretrained('./gpt2-xl')
    AutoModelWithLMHead.from_pretrained('gpt2').save_pretrained('./gpt2-xl')
    tokenizer = AutoTokenizer.from_pretrained("./gpt2-xl")
    model = AutoModelWithLMHead.from_pretrained("./gpt2-xl")

model = model.to(device)

This basically tries to load the the downloaded model, and if that fails it downloads a new copy of the model. I have autoscaling set to a minimum of 1 which I thought would mean something would always be running and therefore the downloaded file would persist even after activity. But it keeps having to redownload the model which freezes up the app when some people try to use it. I am trying to recreate something like this app https://text-generator-gpt2-app-6q7gvhilqq-lz.a.run.app/ which does not appear to have the same load time issue . In the flask app itself I have the following:
@app.route('/')
@cross_origin()
def index():
    prompt = wp[random.randint(0, len(wp)-1)]
    res = generate(prompt, size=75)
    generated = res.split(prompt)[-1] + '\n \n...TO BE CONTINUED'
    #generated = prompt
    return flask.render_template('main.html', prompt = prompt, output = generated)

if __name__ == "__main__":
    app.run(host='0.0.0.0',
            debug=True,
            port=PORT)

But it seems to redownload the models every few hours...how can I avoid having the app re-downloading the models and the app freezing for those who want to try it?

Output:
Data written to the filesystem does not persist when the container instance is stopped.
Cloud Run lifetime is the time between an HTTP Request and the HTTP response. Overlapped requests extend this lifetime. Once the final HTTP response is sent your container can be stopped.
Cloud Run instances can run on different hardware (clusters). One instance will not have the same temporary data as another instance. Instances can be moved. Your strategy of downloading a large file and saving it to the in-memory file system will not work consistently.
Filesystem access
Also note that the file system is in-memory, which means you need to have additional memory to store files.


Instruction: How to extract individual JPEG images from a HDF5 file

Input:
I have a big HDF5 file with the images and its corresponding ground truth density map.
I want to put them into the network CRSNet and it requires the images in separate files.
How can I achieve that? Thank you very much.
-- Basic info I have a HDF5 file with two keys "images" and "density_maps". Their shapes are (300, 380, 676, 1).
300 stands for the number of images, 380 and 676 refer to the height and width respectively.
-- What I need to put into the CRSNet network  are the images (jpg) with their corresponding HDF5 files.  The shape of them would be (572, 945).
Thanks a lot for any comment and discussion!

Output:
For starters, a quick clarification on h5py and HDF5. h5py is a Python package to read HDF5 files. You can also read HDF5 files with the PyTables package (and with other languages: C, C++, FORTRAN).
I'm not entirely sure what you mean by "the images (jpg) with their corresponding h5py (HDF5) files" As I understand all of your data is in 1 HDF5 file. Also, I don't understand what you mean by: "The shape of them would be (572, 945)." This is different from the image data, right?  Please update your post to clarify these items.
It's relatively easy to extract data from a dataset. This is how you can get the "images" as NumPy arrays and and use cv2 to write as individual jpg files. See code below:
with h5py.File('yourfile.h5','r') as h5f:
    for i in range(h5f['images'].shape[0]):
        img_arr = h5f['images'][i,:]   # slice notation gets [i,:,:,:]
        cv2.imwrite(f'test_img_{i:03}.jpg',img_arr)

Before you start coding, are you sure you need the images as individual image files, or individual image data (usually NumPy arrays)? I ask because the first step in most CNN processes is reading the images and converting them to arrays for downstream processing. You already have the arrays in the HDF5 file. All you may need to do is read each array and save to the appropriate data structure for CRSNet to process them. For example, here is the code to create a list of arrays (used by TensorFlow and Keras):
image_list = []
with h5py.File('yourfile.h5','r') as h5f:
    for i in range(h5f['images'].shape[0]):
        image_list.append( h5f['images'][i,:] )  # gets slice [i,:,:,:]


Instruction: AllenNLP Multi-Task Model: Keep encoder weights for new heads

Input:
I have trained a (AllenNLP) multi-task model. I would like to keep the encoder/backbone weights and continue training with new heads on new datasets. How can I do that with AllenNLP?
I have two basic ideas for how to do that:

I followed this AllenNLP tutorial to load the trained model and then instead of just making predictions I wanted to change the configuration and the model-heads to continue training on the new datasets...but I am kinda lost in how to do that.

I guess it should be possible to (a) save the state-dict of the previously trained encoder in a file and then (b) point to those weights in the configuration file for the new model (instead of pointing to "bert-base-cased"-weights for example). But looking at the PretrainedTransformerEmbedder-class I don't see how I could pass my own model-weights to that class.

As an additional question: Is it also possible to save the weights of the heads separately and initialize new heads with those weights?
Any help is appreciated :)

Output: Your second idea is the preferred way, which you can accomplish by using a PretrainedModelInitializer. See the CopyNet model for an example of how to add this to your model.


Instruction: Receiving coordinates from inference Pytorch

Input:
I'm trying to get the coordinates of the pixels inside of a mask that is generated by Pytorches DefaultPredictor, to later on get the polygon corners and use this in my application.
However, DefaultPredictor produced a tensor of pred_masks, in the following format: [False, False ... False], ... [False, False, .. False]
Where the length of each individual list is length of the image, and the number of total lists is the height of the image.
Now, as I need to get the pixel coordinates that are inside of the mask, the simple solution seemed to be looping through the pred_masks, checking the value and if == "True" creating tuples of these and adding them to a list. However, as we are talking about images with width x height of about 3200 x 1600, this is a relatively slow process (~4 seconds to loop through a single 3200x1600, yet as there are quite some objects for which I need to get the inference in the end - this will end up being incredibly slow).
What would be the smarter way to get the the coordinates (mask) of the detected object using the pytorch (detectron2) model?
Please find my code below for reference:
from __future__ import print_function

from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.data import MetadataCatalog
from detectron2.data.datasets import register_coco_instances

import cv2
import time

# get image
start = time.time()

im = cv2.imread("inputImage.jpg")

# Create config
cfg = get_cfg()
cfg.merge_from_file("detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # Set threshold for this model
cfg.MODEL.WEIGHTS = "model_final.pth" # Set path model .pth
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
cfg.MODEL.DEVICE='cpu'

register_coco_instances("dataset_test",{},"testval.json","Images_path")
test_metadata = MetadataCatalog.get("dataset_test")

# Create predictor
predictor = DefaultPredictor(cfg)

# Make prediction
outputs = predictor(im)

#Loop through the pred_masks and check which ones are equal to TRUE, if equal, add the pixel values to the true_cords_list
outputnump = outputs["instances"].pred_masks.numpy()

true_cords_list = []
x_length = range(len(outputnump[0][0]))
#y kordinaat on range number
for y_cord in range(len(outputnump[0])):
    #x cord
    for x_cord in x_length:
        if str(outputnump[0][y_cord][x_cord]) == "True":
            inputcoords = (x_cord,y_cord)
            true_cords_list.append(inputcoords)

print(str(true_cords_list))

end = time.time()

print(f"Runtime of the program is {end - start}") # 14.29468035697937

//

EDIT:
After changing the for loop partially to compress - I've managed to reduce the runtime of the for loop by ~3x - however, ideally I would like to receive this from the predictor itself if possible.
y_length = len(outputnump[0])
x_length = len(outputnump[0][0])
true_cords_list = []
for y_cord in range(y_length):
    x_cords = list(compress(range(x_length), outputnump[0][y_cord]))
    if x_cords:
        for x_cord in x_cords:
            inputcoords = (x_cord,y_cord)
            true_cords_list.append(inputcoords)

Output:
The problem is easily solvable with sufficient knowledge about NumPy or PyTorch native array handling, which allows 100x speedups compared to Python loops. You can study the NumPy library, and PyTorch tensors are similar to NumPy in behaviour.
How to get indices of values in NumPy:
import numpy as np
arr = np.random.rand(3,4) > 0.5
ind = np.argwhere(arr)[:, ::-1]
print(arr)
print(ind)

In your particular case this will be
ind = np.argwhere(outputnump[0])[:, ::-1]

How to get indices of values in PyTorch:
import torch
arr = torch.rand(3, 4) > 0.5
ind = arr.nonzero()
ind = torch.flip(ind, [1])
print(arr)
print(ind)

[::-1] and .flip are used to inverse the order of coordinates from (y, x) to (x, y).
NumPy and PyTorch even allow checking simple conditions and getting the indices of values that meet these conditions, for further understanding see the according NumPy docs article

When asking, you should provide links for your problem context. This question is actually about Facebook object detector, where they provide a nice demo Colab notebook.


Instruction: How to get the partial derivative of probability to input in pyTorch?

Input:
I want to generate attack samples via the following steps:

Find a pre-trained CNN classification model, whose input is X and output is P(y|X), and the most possible result of X is y.

I want to input X' and get y_fool, where X' is not far away from X and y_fool is not equal to y

The steps for getting X' is:enter image description here

How can I get the partial derivative described in the image?

Here is my code but I got None: (The model is Vgg16)
x = torch.autograd.Variable(image, requires_grad=True)
output = model(image)
prob = nn.functional.softmax(output[0], dim=0)

prob.backward(torch.ones(prob.size()))
print(x.grad)

How should I modify my codes? Could someone help me? I would be absolutely grateful.

Output:
Here, the point is to backpropagate a "false" example through the network, in other words you need to maximize one particular coordinate of your output which does not correspond to the actual label of x.
Let's say for example that your model outputs N-dimensional vectors, that x label should be [1, 0, 0, ...] and that we will try to make the model actually predict [0, 1, 0, 0, ...] (so y_fool actually has its second coordinate set to 1, instead of the first one).
Quick note on the side : Variable is deprecated, just set the requires_grad flag to True. So you get :
x = torch.tensor(image, requires_grad=True)
output = model(x)
# If the model is well trained, prob_vector[1] should be almost 0 at the beginning
prob_vector = nn.functional.softmax(output, dim=0)
# We want to fool the model and maximize this coordinate instead of prob_vector[0]
fool_prob = prob_vector[1]
# fool_prob is a scalar tensor, so we can backward it easy
fool_prob.backward()
# and you should have your gradients :
print(x.grad)

After that, if you want to use an optimizer in your loop to modify x, remember that pytorch optimizer.step method tries to minimize the loss, whereas you want to maximize it. So either you use a negative learning rate or you change the backprop sign :
# Maximizing a scalar is minimizing its opposite
(-fool_prob).backward()


Instruction: torchvision.io cannot find reference read_image() in __init.py__

Input:
I am trying to import read_image from torchvision.io
and when i hover the error it says
torchvision.io cannot find reference 'read_image' in '__init.py__'
from torchvision.io import read_image

I am following this example
https://pytorch.org/tutorials/beginner/basics/data_tutorial.html
from torchvision.io import read_image ImportError: cannot import name 'read_image' from 'torchvision.io' (C:\Users\X\.conda\envs\Pytorch37\lib\site-packages\torchvision\io\__init__.py)
This is the error I'm getting.

Output:
You need to upgrade the version of your pytorch
# Just for my version of system and cuda, you need to replace the cmd below based on your condition
pip install torch torchvision torchaudio


Instruction: PyTorch tensors: new tensor based on old tensor and indices

Input:
I'm new to tensors and having a headache over this problem:
I have an index tensor of size k with values between 0 and k-1:
tensor([0,1,2,0])

and the following matrix:
tensor([[[0, 9],
     [1, 8],
     [2, 3],
     [4, 9]]])

I want to create a new tensor which contains the rows specified in index, in that order. So I want:
tensor([[[0, 9],
     [1, 8],
     [2, 3],
     [0, 9]]])

Outside tensors I'd do this operation more or less like this:
new_matrix = [matrix[i] for i in index]

How do I do something similar in PyTorch on tensors?

Output:
You use fancy indexing:
from torch import tensor

index = tensor([0,1,2,0])
t = tensor([[[0, 9],
     [1, 8],
     [2, 3],
     [0, 9]]])

result = t[:, index, :]

to get
tensor([[[0, 9],
         [1, 8],
         [2, 3],
         [0, 9]]])

Note that t.shape == (1, 4, 2) and you want to index on the second axis; so we apply it in the second argument and keep the rest the same via :s i.e. [:, index, :].


Instruction: create representation of questions using LSTM via a pre-trained word embedding such as GloVe

Input:
I am new in LSTM and python. My goal is to represent the sentence using LSTM.
Could u tell me I am doing the right? how to fix the error when running the following code ?
"TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not str"
import torch
import torch.nn as nn
import numpy as np
from torch import optim
from torch.nn.utils.rnn import pad_packed_sequence, pack_padded_sequence
import torchvision.datasets as datasets  # Standard datasets
import torchvision.transforms as transforms
import json

class RNN_LSTM(nn.Module)

    def __init__(self, input_size, hidden_size, num_layers, num_classes, vocab_size,
                 lstm_dropout, device, word_emb_file):
        super(RNN_LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm_dropout = lstm_dropout
        self.lstm_drop = nn.Dropout(p=self.lstm_dropout)
        self.word_emb_file = word_emb_file
        self.device = device

        # initialize text embeddings
        self.word_embeddings = nn.Embedding(vocab_size, input_size)
        self.word_embeddings.weight = nn.Parameter(
                torch.from_numpy(
                    np.pad(np.load(self.word_emb_file), ((0, 1), (0, 0)), 'constant')).type(
                    'torch.FloatTensor'))

        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)

    def forward(self, sentence, question_len):
        embeds = self.word_embeddings(sentence)
        packed_output = pack_padded_sequence(embeds, question_len, batch_first=True)
        outputs, (hidden, cell_state) = self.lstm(packed_output)
        outputs, outputs_length = pad_packed_sequence(outputs, batch_first=True)
        outputs = torch.cat([hidden[0,:,:], hidden[1,:,:]], dim=-1)
        return outputs

lstm_dropout = 0.3
input_size = 300
hidden_size = 256
num_classes = 10
num_layers = 1
device = 'cpu'
vocab_size = 2000
word_emb_file = "/home/project/word_emb_300d.npy"

model = RNN_LSTM(input_size, hidden_size, num_layers, num_classes, vocab_size, lstm_dropout, device, word_emb_file)

model.word_embeddings('Blue Skye')

Output:
Please see torch embedding tutorial and use embedding with keras for knowledge about word embeddings.
Basically nn.Embedding(vocab_size, embed_size) is a matrix vocab_size*embed_size, where each row corresponds to a word's representation.
In order to know which word correspond to which row you should define a vocabulary that can transform a word into an index (for example a python dictionary {'hello': 0, 'word': 1}).
And before that to transform a sentence into word (in order to compute the vocabulary) you need to tokenize the sentence (using nltk.word_tokenize or str.split() for example).
Though torch.nn.Embedding expects a Tensor, so if you want to process multiple sentence into batches, and the sentence have different length you will need to pad the sentence with empty tokens in order to fit them into a Tensor.
# This is pseudo code

# Preprocess data
documents = ['first sentence', 'the second sentence']
tokenized_documents = [d.split(' ') for d in documents]
# Create vocabulary and add a special token for padding
words = [w for d in documents for word in tokenized_documents]
vocabulary = {w: i+1 for i, w in enumerate(set(words))}
vocabulary['PAD'] = 0
indexed_documents = [[vocabulary[w] for w in d] for d in tokenized_documents]
# indexed_documents will look like : [[0, 1], [2, 3, 1]]
padded_documents = torch.nn.utils.rnn.pad_sequence(
    indexed_documents,
    padding_value=vocabulary['PAD'])

# Data can be fed to the neural network
model.word_embeddings(torch.tensor(padded_documents))


Instruction: New tensor based on old tensor and 2d indices

Input:
I previously asked:
PyTorch tensors: new tensor based on old tensor and indices
I have the same problem now but need to use a 2d index tensor.
I have a tensor col of size [batch_size, k] with values between 0 and k-1:
idx = tensor([[0,1,2,0],
        [0,3,2,2],
        ...])

and the following matrix:
x = tensor([[[0, 9],
 [1, 8],
 [2, 3],
 [4, 9]],
 [[0, 0],
 [1, 2],
 [3, 4],
 [5, 6]]])

I want to create a new tensor which contains the rows specified in index, in that order. So I want:
tensor([[[0, 9],
 [1, 8],
 [2, 3],
 [0, 9]],
 [[0, 0],
 [5, 6],
 [3, 4],
 [3, 4]]])

Currently I'm doing it like this:
for i, batch in enumerate(t):
    t[i] = batch[col[i]]

How can I do it more efficiently?

Output:
you should use torch gather to achieve this. It would actually also work for the otehr question you linked, but this is left as an exercise to the reader :p
Let us call idx your first tensor and source the second one. Their respective dimensions are (B,N) and (B, K, p) (with p=2 in your example), and all values of idx are between 0 and K-1.
So to use torch gather, we first need to express your operation as a nested for loop. In your case, what you actually want to achieve is :
for b in range(B):
    for i in range(N):
        for j in range(p):
            # This kind of nested for loops is what torch.gether actually does
            target[b,i,j] = source[b, idx[b,i,j], j]

But that does not work because idx is a 2D tensor, not a 3D one. Well, no big deal, let's make it a 3D tensor. We want it to have shape (B, N, p) and be actually constant along the last dimension. Then we can replace the for loop with a call to gather:
reshaped_idx = idx.unsqueeze(-1).repeat(1,1,2)
target = source.gather(1, reshaped_idx)
# or : target = torch.gather(source, 1, reshaped_idx)


Instruction: Size Mismatch for Functional Linear Layer

Input:
I apologize that this is probably a simple question that has been answered before, but I could not find the answer. I’m attempting to use a CNN to extract features and then input that into a FC network that outputs 2 variables. I’m attempting to use the functional linear layer as a way to dynamically handle the flattened features. The self.cnn is a Sequential container which last layer is the nn.Flatten(). When I print the size of x after the CNN I see it is 15x152064, so I’m unclear why the F.linear layer is failing to run with the error below. Any help would be appreciated.
RuntimeError: size mismatch, get 15, 15x152064,2
x = self.cnn(x)
batch_size, channels = x.size()
x = F.linear(x, torch.Tensor([256,channels]))
y_hat = self.FC(x)

Output:
torch.Tensor([256, channels]) does not create a tensor of size (256, channels) but the 1D tensor containing the values 256 and channels instead. I don't know how you want to initialize your weights, but there are a couple options :
# Identity transform:
x = F.linear(x, torch.ones(256,channels))
# Random transform :
x = F.linear(x, torch.randn(256,channels))


Instruction: PyTorch shows the error " 'NoneType' object has no attribute 'zero_' " when calling the zero_ method

Input:
Can someone please answer why my code is showing an error.
Thanks in advance.
Code:
import torch
torch.manual_seed(0)

a = torch.rand((1, 3), requires_grad = True)
w1 = torch.rand((3, 3), requires_grad = True)
w2 = torch.rand((3, 1), requires_grad = True)

d = torch.matmul(torch.matmul(a, w1), w2)

L = (10 - d)

L.backward()

w1 = w1 - w1.grad*0.001
w1.grad.zero_()

Error:
AttributeError
'NoneType' object has no attribute 'zero_'

Output:
The line
w1 = w1 - w1.grad*0.001

is reassigning w1, so afterwards w1 no longer refers to the same tensor it did before. To maintain all the internal state of w1 (e.g. the .grad member) you must update w1 in place. Since this is a leaf tensor we also need to disable construction of the computation graph.
with torch.no_grad():
    w1.sub_(w1.grad * 0.001)


Instruction: Torch is not saving my freezed and optimized model

Input:
when I start my script it runs fine until it hits the traced_model.save(args.save_path) statement after that the script just stop running.
Could someone please help me out with this?
import argparse
import torch
from model import SpeechRecognition
from collections import OrderedDict

def trace(model):
    model.eval()
    x = torch.rand(1, 81, 300)
    hidden = model._init_hidden(1)
    traced = torch.jit.trace(model, (x, hidden))
    return traced

def main(args):
    print("loading model from", args.model_checkpoint)
    checkpoint = torch.load(args.model_checkpoint, map_location=torch.device('cpu'))
    h_params = SpeechRecognition.hyper_parameters
    model = SpeechRecognition(**h_params)

    model_state_dict = checkpoint['state_dict']
    new_state_dict = OrderedDict()
    for k, v in model_state_dict.items():
        name = k.replace("model.", "") # remove `model.`
        new_state_dict[name] = v

    model.load_state_dict(new_state_dict)

    print("tracing model...")
    traced_model = trace(model)
    print("saving to", args.save_path)
    traced_model.save(args.save_path)
    print("Done!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="testing the wakeword engine")
    parser.add_argument('--model_checkpoint', type=str, default='your/checkpoint_file', required=False,
                        help='Checkpoint of model to optimize')
    parser.add_argument('--save_path', type=str, default='path/where/you/want/to/save/the/model', required=False,
                        help='path to save optmized model')

    args = parser.parse_args()
    main(args)

If you start the script you can even see where it stops working because print("Done!") is not executed.
Here is what it looks in the terminal when I run the script:
loading model from C:/Users/supre/Documents/Python Programs/epoch=0-step=11999.ckpt
tracing model...
saving to C:/Users/supre/Documents/Python Programs

Output:
According to the PyTorch documentation, a common PyTorch convention is to save models using either a .pt or .pth file extension.
To save model checkpoints or multiple components, organize them in a dictionary and use torch.save() to serialize the dictionary.
For example,
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)

A common PyTorch convention is to save these checkpoints using the .tar file extension.
Hope this answers your question.


Instruction: Neural Network on GPU using system RAM instead of GPU memory

Input:
I built a basic chatbot using PyTorch, and in the training code, I moved both the neural network as well as the training data to the gpu. However, when I run the program, it uses up to 2GB of my ram. There is a little gpu memory that is used, but not that much. When i run the same program, but this time on the cpu, it takes only about 900mb of my ram.
Can anyone tell me why is is happening? i have attached my code, as well as a couple of screenshots.
Sorry if the answer is pretty obvious, I'm new to deep learning.
My code:
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from nltk_utils import tokenizer, stem, bag_of_words
import pandas as pd

device = torch.device("cuda")

#Neural Network
class NeuralNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.l1 = nn.Linear(input_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, hidden_size)
        self.l4 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = self.relu(self.l1(x))
        out = self.relu(self.l2(out))
        out = self.relu(self.l3(out))
        out = self.l4(out)
        return out

#data initialization
data = pd.read_csv("path_to_train_data")
data = data.dropna(axis=0)

allwords = []
taglist = []
xy = []
ignorewords = ["?", "!", "'", ","]

taglist.extend(x for x in data["tag"] if x not in taglist)

#developing vocabulary
for x in data["pattern"]:
    w = tokenizer(x)
    allwords.extend(stem(y) for y in w if y not in ignorewords)

#making training data
for indx, x in enumerate(data["pattern"]):
    w = tokenizer(x)
    bag = bag_of_words(w, allwords)
    tag = taglist.index(data["tag"].iloc[indx])
    xy.append((bag, tag))

xtrain = np.array([x[0] for x in xy])
ytrain = np.array([x[1] for x in xy])

class TestDataset(Dataset):
    def __init__(self):
        self.num_classes = len(xtrain)
        self.xdata = torch.from_numpy(xtrain.astype(np.float32))
        self.ydata = torch.from_numpy(ytrain.astype(np.float32))

    def __getitem__(self, index):
        return self.xdata[index], self.ydata[index]

    def __len__(self):
        return self.num_classes

dataset = TestDataset()
train_data = DataLoader(dataset=dataset,
                        batch_size=8,
                        shuffle=True,
                        num_workers=0,
                        pin_memory=True)

inputSize = len(xtrain[0])
hiddenSize = 32
outputSize = len(taglist)

model = NeuralNetwork(inputSize, hiddenSize, outputSize).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())

epochs = 5000

for epoch in range(epochs):
    for (words, labels) in train_data:
        words = words.to(device)
        labels = labels.type(torch.LongTensor)
        labels = labels.to(device)

        y_pred = model(words)

        loss = criterion(y_pred, labels)

        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()

    with torch.no_grad():
        if (epoch + 1) % 10 == 0:
            print(f"epoch: {epoch + 1}, loss: {loss:.30f}")

While running on GPU:
GPU utilization,
RAM utilization
While running on CPU:
RAM utilization

Output:
RAM is the place where your data gets stacked to be proceed, the stacks are then transferred to SRAM or otherwise know as cache which is the nearest memory to CPU (referred as host). The general rule to select the memory of RAM for a system is considered as equivalent or greater than GPU memory.
For Example: If the GPU memory is 8GB you need to have a RAM of 8GB or greater to ensure optimal performance.
So, when running on GPU's (referred as Device) your data is directly transferred to GPU to perform tensor operations. But on CPU it can not perform parallel computations thus using lesser memory of RAM.
You can try it out with different batch_sizes and observe the RAM usage. Note that if your batch of data is not fitting in GPU memory, you will be seeing an CUDA out of memory error.


Instruction: Select specific indexes of 3D Pytorch Tensor using a 1D long tensor that represents indexes

Input:
So I have a tensor that is M x B x C, where M is the number of models, B is the batch and C is the classes and each cell is the probability of a class for a given model and batch. Then I have a tensor of the correct answers which is just a 1D of size B we'll call "t". How do I use the 1D of size B to just return a M x B x 1, where the returned tensor is just the value at the correct class? Say the M x B x C tensor is called "blah" I've tried
blah[:, :, C]

for i in range(M):
    blah[i, :, C]

blah[:, C, :]

The top 2 just return the values of indexes t in the 3rd dimension of every slice. The last one returns the values at t indexes in the 2nd dimension. How do I do this?

Output:
We can get the desired result by combining advanced and basic indexing
import torch

# shape [2, 3, 4]
blah = torch.tensor([
    [[ 0,  1,  2,  3],
     [ 4,  5,  6,  7],
     [ 8,  9, 10, 11]],
    [[12, 13, 14, 15],
     [16, 17, 18, 19],
     [20, 21, 22, 23]]])

# shape [3]
t = torch.tensor([2, 1, 0])
b = torch.arange(blah.shape[1]).type_as(t)

# shape [2, 3, 1]
result = blah[:, b, t].unsqueeze(-1)

which results in
>>> result
tensor([[[ 2],
         [ 5],
         [ 8]],
        [[14],
         [17],
         [20]]])


Instruction: How to run LSTM on very long sequence using Truncated Backpropagation in Pytorch (lightning)?

Input:
I have a very long time series I want to feed into an LSTM for classification per-frame.
My data is labeled per frame, and I know some rare events happen that influence the classification heavily ever since they occur.
Thus, I have to feed the entire sequence to get meaningful predictions.
It is known that just feeding very long sequences into LSTM is sub-optimal, since the gradients vanish or explode just like normal RNNs.

I wanted to use a simple technique of cutting the sequence to shorter (say, 100-long) sequences, and run the LSTM on each, then pass the final LSTM hidden and cell states as the start hidden and cell state of the next forward pass.
Here is an example I found of someone who did just that. There it is called "Truncated Back propagation through time". I was not able to make the same work for me.

My attempt in Pytorch lightning (stripped of irrelevant parts):
def __init__(self, config, n_classes, datamodule):
    ...
    self._criterion = nn.CrossEntropyLoss(
        reduction='mean',
    )

    num_layers = 1
    hidden_size = 50
    batch_size=1

    self._lstm1 = nn.LSTM(input_size=len(self._in_features), hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
    self._log_probs = nn.Linear(hidden_size, self._n_predicted_classes)
    self._last_h_n = torch.zeros((num_layers, batch_size, hidden_size), device='cuda', dtype=torch.double, requires_grad=False)
    self._last_c_n = torch.zeros((num_layers, batch_size, hidden_size), device='cuda', dtype=torch.double, requires_grad=False)

def training_step(self, batch, batch_index):
    orig_batch, label_batch = batch
    n_labels_in_batch = np.prod(label_batch.shape)
    lstm_out, (self._last_h_n, self._last_c_n) = self._lstm1(orig_batch, (self._last_h_n, self._last_c_n))
    log_probs = self._log_probs(lstm_out)
    loss = self._criterion(log_probs.view(n_labels_in_batch, -1), label_batch.view(n_labels_in_batch))

    return loss

Running this code gives the following error:

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

The same happens if I add
def on_after_backward(self) -> None:
    self._last_h_n.detach()
    self._last_c_n.detach()

The error does not happen if I use
lstm_out, (self._last_h_n, self._last_c_n) = self._lstm1(orig_batch,)

But obviously this is useless, as the output from the current frame-batch is not forwarded to the next one.

What is causing this error? I thought detaching the output h_n and c_n should be enough.
How do I pass the output of a previous frame-batch to the next one and have torch back propagate each frame batch separately?

Output:
Apparently, I missed the trailing _ for detach():
Using
def on_after_backward(self) -> None:
    self._last_h_n.detach_()
    self._last_c_n.detach_()

works.

The problem was self._last_h_n.detach() does not update the reference to the new memory allocated by detach(), thus the graph is still de-referencing the old variable which backprop went through.
The reference answer solved that by H = H.detach().
Cleaner (and probably faster) is self._last_h_n.detach_() which does the operation in place.


Instruction: NumPy: pass subset of multi-dimensional indices to array

Input:
Suppose I have an array, and a (sub)set of indices:
arr = np.array([[[0,1,2],
                 [3,4,5],
                 [6,7,8]],

                [[9, 10,11],
                 [12,13,14],
                 [15,16,17]]])
idx = [1,2]

And I wish to get, for each element of dim=0 of the array, the respective idx slice, i.e.:
>>> arr[:, idx[0], idx[1]]
array([ 5, 14])

Is there a way to do such slicing without hard-coding each index? Something like:
ar[:, *idx]

Note, the following is my current workaround:
idx = [slice(a.shape[0]), *idx]
a[idx]

but I was wondering if NumPy (or PyTorch, or a related library) supported a more 'elegant'/natural multi-dimensional indexing syntax for such cases.

Output:
I think your workaround is pretty much as close as you can get.
You can try this to make it slightly more concise:
import numpy as np

arr = np.array([[[0,1,2],
                 [3,4,5],
                 [6,7,8]],

                [[9, 10,11],
                 [12,13,14],
                 [15,16,17]]])
idx = [1,2]

print(
    arr[(slice(None), *idx)]
)

This works because : is the same as slice(None)


Instruction: PyTorch ROCm is out - How to select Radeon GPU as device

Input:
since Pytorch released the ROCm version, which enables me to use other gpus than nvidias, how can I select my radeon gpu as device in python?
Obviously, code like device = torch.cuda.is_available or device = torch.device("cuda") is not working. Thanks for any help. :)

Output:
This took me forever to figure out. I'm still having some configuration issues with my AMD GPU, so I haven't been able to test that this works, but, according to this github pytorch thread, the Rocm integration is written so you can just call torch.device('cuda') and no actual porting is required!
See the thread here: https://github.com/pytorch/pytorch/issues/10670


Instruction: How to get layer execution time on an AI model saved as .pth file?

Input:
I'm trying to run a Resnet-like image classification model on a CPU, and want to know the breakdown of time it takes to run each layer of the model.
The issue I'm facing is the github link https://github.com/facebookresearch/semi-supervised-ImageNet1K-models has the model saved as a .pth file. It is very large (100s of MB), and I don't know exactly how it differs from pytorch except it's binary.
I load the model from this file using the following script. But I don't see a way to modify the model or insert the t = time.time() variables/statements in between model layers to break down the time in each layer.
Questions:

Would running the model in the following script give a correct estimate of end-to-end time (t2-t1) it takes to run the model on the CPU, or would it also include pytorch compilation time?

How to insert time statements between consecutive layers to get a breakdown?

There is no inference/training script at the github link and only has the .pth file. So how exactly is one supposed to run inference or training? How to insert additional layers between consecutive layers of the .pth model and save them?

#!/usr/bin/env python
import torch torchvision time

model=torch.hub.load('facebookresearch/semi-supervised-ImageNet1K-models', 'resnext50_32x4d_swsl', force_reload=False)
in = torch.randn(1, 3, 224, 224)
t1 = time.time()
out = model.forward(in)
t2 = time.time()
```**strong text**

Output:
A simple way to implement such a requirement is by registering forward hooks on each module of the model which updates a global variable for storing the time and computes the time difference between the last and current computations.
For example:
import torch
import torchvision
import time

global_time = None
exec_times = []

def store_time(self, input, output):
    global global_time, exec_times
    exec_times.append(time.time() - global_time)
    global_time = time.time()

model = torch.hub.load('facebookresearch/semi-supervised-ImageNet1K-models', 'resnext50_32x4d_swsl', force_reload=False)
x = torch.randn(1, 3, 224, 224)

# Register a hook for each module for computing the time difference
for module in model.modules():
    module.register_forward_hook(store_time)

global_time = time.time()
out = model(x)
t2 = time.time()

for module, t in zip(model.modules(), exec_times):
    print(f"{module.__class__}: {t}")

The output I get is:
<class 'torchvision.models.resnet.ResNet'>: 0.004999876022338867
<class 'torch.nn.modules.conv.Conv2d'>: 0.002006053924560547
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009946823120117188
<class 'torch.nn.modules.activation.ReLU'>: 0.007998466491699219
<class 'torch.nn.modules.pooling.MaxPool2d'>: 0.0010004043579101562
<class 'torch.nn.modules.container.Sequential'>: 0.0020003318786621094
<class 'torchvision.models.resnet.Bottleneck'>: 0.0010023117065429688
<class 'torch.nn.modules.conv.Conv2d'>: 0.017997026443481445
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010018348693847656
<class 'torch.nn.modules.conv.Conv2d'>: 0.0009999275207519531
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.003000497817993164
<class 'torch.nn.modules.conv.Conv2d'>: 0.003999948501586914
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.001997232437133789
<class 'torch.nn.modules.activation.ReLU'>: 0.004001140594482422
<class 'torch.nn.modules.container.Sequential'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.001999378204345703
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torchvision.models.resnet.Bottleneck'>: 0.003001689910888672
<class 'torch.nn.modules.conv.Conv2d'>: 0.0020008087158203125
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009992122650146484
<class 'torch.nn.modules.conv.Conv2d'>: 0.0019991397857666016
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010001659393310547
<class 'torch.nn.modules.conv.Conv2d'>: 0.0009999275207519531
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002998828887939453
<class 'torch.nn.modules.activation.ReLU'>: 0.0010013580322265625
<class 'torchvision.models.resnet.Bottleneck'>: 0.0029997825622558594
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002999544143676758
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010006427764892578
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.001001119613647461
<class 'torch.nn.modules.conv.Conv2d'>: 0.0019979476928710938
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010018348693847656
<class 'torch.nn.modules.activation.ReLU'>: 0.0010001659393310547
<class 'torch.nn.modules.container.Sequential'>: 0.00299835205078125
<class 'torchvision.models.resnet.Bottleneck'>: 0.002004384994506836
<class 'torch.nn.modules.conv.Conv2d'>: 0.0009975433349609375
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.005999088287353516
<class 'torch.nn.modules.conv.Conv2d'>: 0.0020003318786621094
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010001659393310547
<class 'torch.nn.modules.activation.ReLU'>: 0.0020017623901367188
<class 'torch.nn.modules.container.Sequential'>: 0.0009970664978027344
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0029997825622558594
<class 'torchvision.models.resnet.Bottleneck'>: 0.0010008811950683594
<class 'torch.nn.modules.conv.Conv2d'>: 0.00500035285949707
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009984970092773438
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0020020008087158203
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019979476928710938
<class 'torch.nn.modules.activation.ReLU'>: 0.0010018348693847656
<class 'torchvision.models.resnet.Bottleneck'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.00099945068359375
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.001001119613647461
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002997875213623047
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010013580322265625
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002000570297241211
<class 'torch.nn.modules.activation.ReLU'>: 0.0
<class 'torchvision.models.resnet.Bottleneck'>: 0.001997232437133789
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010008811950683594
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.001001596450805664
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.00099945068359375
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002998828887939453
<class 'torch.nn.modules.activation.ReLU'>: 0.0010020732879638672
<class 'torch.nn.modules.container.Sequential'>: 0.0010020732879638672
<class 'torchvision.models.resnet.Bottleneck'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.001995563507080078
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.002001523971557617
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010001659393310547
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010008811950683594
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.activation.ReLU'>: 0.0029985904693603516
<class 'torch.nn.modules.container.Sequential'>: 0.0009989738464355469
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010068416595458984
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torchvision.models.resnet.Bottleneck'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.004993438720703125
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010013580322265625
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010001659393310547
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010018348693847656
<class 'torch.nn.modules.conv.Conv2d'>: 0.001997709274291992
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.activation.ReLU'>: 0.0019991397857666016
<class 'torchvision.models.resnet.Bottleneck'>: 0.0029990673065185547
<class 'torch.nn.modules.conv.Conv2d'>: 0.0030128955841064453
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019872188568115234
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0029993057250976562
<class 'torch.nn.modules.activation.ReLU'>: 0.0010008811950683594
<class 'torchvision.models.resnet.Bottleneck'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010006427764892578
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009992122650146484
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.003001689910888672
<class 'torch.nn.modules.conv.Conv2d'>: 0.0019986629486083984
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0010008811950683594
<class 'torch.nn.modules.activation.ReLU'>: 0.0
<class 'torchvision.models.resnet.Bottleneck'>: 0.002000093460083008
<class 'torch.nn.modules.conv.Conv2d'>: 0.0019986629486083984
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0020012855529785156
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019981861114501953
<class 'torch.nn.modules.activation.ReLU'>: 0.0030014514923095703
<class 'torchvision.models.resnet.Bottleneck'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0029985904693603516
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010008811950683594
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0010013580322265625
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009989738464355469
<class 'torch.nn.modules.activation.ReLU'>: 0.0
<class 'torch.nn.modules.container.Sequential'>: 0.002998828887939453
<class 'torchvision.models.resnet.Bottleneck'>: 0.002000570297241211
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.003000497817993164
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0020020008087158203
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0009982585906982422
<class 'torch.nn.modules.activation.ReLU'>: 0.0009996891021728516
<class 'torch.nn.modules.container.Sequential'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0029990673065185547
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0020003318786621094
<class 'torchvision.models.resnet.Bottleneck'>: 0.0010025501251220703
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019981861114501953
<class 'torch.nn.modules.conv.Conv2d'>: 0.0019996166229248047
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019996166229248047
<class 'torch.nn.modules.activation.ReLU'>: 0.0
<class 'torchvision.models.resnet.Bottleneck'>: 0.0030002593994140625
<class 'torch.nn.modules.conv.Conv2d'>: 0.0020012855529785156
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.0
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0
<class 'torch.nn.modules.conv.Conv2d'>: 0.006000518798828125
<class 'torch.nn.modules.batchnorm.BatchNorm2d'>: 0.0019979476928710938
<class 'torch.nn.modules.activation.ReLU'>: 0.0
<class 'torch.nn.modules.pooling.AdaptiveAvgPool2d'>: 0.002003192901611328
<class 'torch.nn.modules.linear.Linear'>: 0.0019965171813964844

Process finished with exit code 0


Instruction: Positive and negative edges in train_test_split_edges from Pytorch geometric package

Input: I am trying to find the explanation for negative and positive edges in a graph as explained in the opening of the function train_test_split_edges Pytorch Geometric doc. According to the doc file it says the function is supposed to split a graph into "positive and negative train/val/test edges". What is the meaning of a positive edge or a negative edge for that matter. According to the code the positive edges "seems" to be the connections of edges in the upper triangle of the adjacency matrix of the graph and the negative edges are the edges in the lower triangle of the adjacency matrix. So if (1,0) is considered to be a positive edge then in an undirected graph (0,1) is a negative edge. Am I correct? I am not finding anywhere about the meaning of positive edge / negative edge where in comes to graphs.

Output:
In link prediction tasks, it is usual to treat existing edges in the graph as positive examples, and non-existent edges as negative examples.
i.e. in training/prediction you feed the network a subset of all edges in the complete graph, and the associated targets are "this is a real edge" (positive) and "this is not a real edge" (negative).


Instruction: Using pytorch cuda in AWS sagemaker notebook instance

Input:
In colab, whenever we need GPU, we simply click change runtime type and change hardware accelarator to GPU
and cuda becomes available, torch.cuda.is_available() is True
How to do this is AWS sagemaker, i.e. turning on cuda.
I am new to AWS and trying to train model using pytorch in aws sagemaker, where Pytorch code is first tested in colab environment.
my sagemaker notebook insatnce is ml.t2.medium

Output:
Using AWS Sagemaker you don't need to worry about the GPU, you simply select an instance type with GPU ans Sagemaker will use it. Specifically ml.t2.medium doesn't have a GPU but it's anyway not the right way to train a model.
Basically you have 2 canonical ways to use Sagemaker (look at the documentation and examples please), the first is to use a notebook with a limited computing resource to spin up a training job using a prebuilt image, in that case when you call the estimator you simply specify what instance type you want (you'll choose one with GPU, looking at the costs). The second way is to use your own container, push it to ECR and launch a training job from the console, where you specify the instance type.


Instruction: PyTorch: Dynamic Programming as Tensor Operation

Input:
Is it possible to get the following loop with a Tensor operation?
a = torch.Tensor([1, 0, 0, 0])
b = torch.Tensor([1, 2, 3, 4])

for i in range(1, a.shape[0]):
    a[i] = b[i] + a[i-1]

print(a) # [1, 3, 6, 10]

The operation depends on the previous values in a and the values that are computed on the way (in a dynamic programming fashion).
Is it possible to get this type of sequential computation with a tensor operation?

Output:
You can achieve this with a cumulative sum:
b.cumsum(0)


Instruction: I wonder why the corresponding storage of pytorch.tensor changes every time it is resized?

Input:
import torch

a = torch.tensor([3,2,3,4])
b = a.view(2,2)
c = a.resize(2,2)
d = a.resize_(2,2)
print(id(a.storage()))
print(id(b.storage()))
print(id(c.storage()))
print(id(d.storage()))

run at first time
2356950450056
2356950450056
2356950450056
2356950450056

run at second time
2206021857352
2206301638600
2206021857352
2206301638600

Why does id change sometimes but not change sometimes, I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this. (apologies for my poor English)
Thanks in advance.

Output:
You don't need to view or resize the object to observe this behaviour, calling storage on the same object can return different id's:
a = torch.tensor([3,2,3,4])
print(id(a.storage()))
print(id(a.storage()))
>>> 2308579152
>>> 2308422224

This is because the python storage object is constructed when calling .storage().


Instruction: PyTorch Tensor Operation for adding the maximum of the previous row to the next

Input:
Follow-Up question to PyTorch: Dynamic Programming as Tensor Operation.
Could the following be written as a tensor operation instead of a loop?
a = torch.Tensor([[1, 2, 3, 4],
                  [5, 6, 7, 8],
                  [9, 10, 11, 12]])

print(a.shape)
# (3, 4)

for i in range(1, a.shape[0]):
    a[i] = a[i-1].max(dim=0)[0] + a[i]

print(a)
# tensor([[ 1,  2,  3,  4],
#         [ 9, 10, 11, 12],
#         [21, 22, 23, 24]])

Basically adding the maximum of the previous row to all elements of the next.
The interesting part is that you can't compute the maximum for each row beforehand and then add that to the respective row, because adding the first maximum influences what the maximum of the next row is.

Output:
Not entirely sure why you're trying to do this, but yes, this is possible. It's basically the same as your last question:
max_vals, _ = a.max(axis=1, keepdim=True)
additions = max_vals.cumsum(0)[:-1]
a[1:, :] += additions

This is because the marginal addition from one row to the next is equivalent to the maximum, so you can take the maximums first, then cumulatively sum them and add them to the original tensor.


Instruction: Can I apply softmax only on specific output neurons?

Input:
I am building an Actor-Critic neural network model in pytorch in order to train an agent to play the game of Quoridor (hopefully). For this reason, I have a neural network with two heads, one for the actor output which does a softmax on all the possible moves and one for the critic output which is just one neuron (for regressing the value of the input state).
Now, in quoridor, most of the times not all moves will be legal and as such I am wondering if I can exclude output neurons on the actor's head that correspond to illegal moves for the input state e.g. by passing a list of indices of all the neurons that correspond to legal moves. Thus, I want to not sum these outputs on the denominator of softmax.
Is there a functionality like this on pytorch (because I cannot find one)? Should I attempt to implement such a Softmax myself (kinda scared to, pytorch probably knows best, I ve been adviced to use LogSoftmax as well)?
Furthermore, do you think this approach of dealing with illegal moves is good? Or should I just let him guess illegal moves and penalize him (negative reward) for it in the hopes that eventually it will not pick illegal moves?
Or should I let the softmax be over all the outputs and then just set illegal ones to zero? The rest won't sum to 1 but maybe I can solve that by plain normalization (i.e. dividing by the L2 norm)?

Output:
An easy solution would be to mask out illegal moves with a large negative value, this will practically force very low (log)softmax values (example below).
# 3 dummy actions for a batch size of 2
>>> actions = torch.rand(2, 3)
>>> actions
tensor([[0.9357, 0.2386, 0.3264],
        [0.0179, 0.8989, 0.9156]])
# dummy mask assigning 0 to valid actions and 1 to invalid ones
>>> mask = torch.randint(low=0, high=2, size=(2, 3))
>>> mask
tensor([[1, 0, 0],
        [0, 0, 0]])
# set actions marked as invalid to very large negative value
>>> actions = actions.masked_fill_(mask.eq(1), value=-1e10)
>>> actions
tensor([[-1.0000e+10,  2.3862e-01,  3.2636e-01],
        [ 1.7921e-02,  8.9890e-01,  9.1564e-01]])
# softmax assigns no probability mass to illegal actions
>>> actions.softmax(dim=-1)
tensor([[0.0000, 0.4781, 0.5219],
        [0.1704, 0.4113, 0.4183]])


Instruction: Understanding the output of LSTM predictions

Input:
It's a 15-class classification model, OUTPUT_DIM = 15. I'm trying to input a frequency vector like this one 'hi my name is' => [1,43,2,56].

When I call predictions = model(x_train[0]) I get an array of size torch.Size([100, 15]), instead of just a 1D array with 15 classes like this: torch.Size([15]). What's happening? Why is this the output? How can I fix it? Thank you in advance, more info below.

The model (from main docs) is the following:
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):

        super().__init__()

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2tag = nn.Linear(hidden_dim, output_dim)

    def forward(self, text):

        embeds = self.word_embeddings(text)
        lstm_out, _ = self.lstm(embeds.view(len(text), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(text), -1))
        tag_scores = F.log_softmax(tag_space, dim=1)

        return tag_scores

Parameters:
INPUT_DIM = 62288
EMBEDDING_DIM = 64
HIDDEN_DIM = 128
OUTPUT_DIM = 15

Output:
The LSTM function in Pytorch returns not just the output of the last timestep but all outputs instead (this is useful in some cases). So in your example you seem to have exactly 100 timesteps (the amount of timesteps is just your sequence length).
But since you are doing classification you just care about the last output. You can normally get it like this:
outputs, _ = self.lstm(embeddings)
# shape: batch_size x 100 x 15
output = outputs[:, -1]
# shape: batch_size x 1 x 15


Instruction: Subclass of PyTorch dataset class cannot find dataset files

Input:
I'm trying to create a subclass of the PyTorch MNIST dataset class, which I call  CustomMNISTDataset, as follows:
import torchvision.datasets as datasets

class CustomMNISTDataset(datasets.MNIST):

    def __init__(self, root='/home/psando'):
        super().__init__(root=root,
                         download=False)

but when I execute:
dataset = CustomMNISTDataset()

it fails with error: "RuntimeError: Dataset not found. You can use download=True to download it".
However, when I run the following in the same file:
dataset = datasets.MNIST(root='/home/psando', download=False)
print(len(dataset))

it succeeds and prints "60000", as expected.
Since CustomMNISTDataset subclasses datasets.MNIST why is the behavior different? I've verified that the path '/home/psando' contains the MNIST directory with raw and processed subdirectories (otherwise, explicitly calling the constructor for datasets.MNIST() would have failed). The current behavior implies that the call to super().__init__() within CustomMNISTDataset is not calling the constructor for datasets.MNIST which is very strange!
Other details: I'm using Python 3.6.8 with torch==1.6.0 and
torchvision==0.7.0. Any help would be appreciated!

Output:
This requires some source-diving, but your problem is this function. The path to the dataset is dependant on the name of the class, so when you subclass MNIST the root folder changes to /home/psando/CustomMNISTDataset
So if you rename /home/psando/MNIST to /home/psando/CustomMNISTDataset it works.


Instruction: NumPy/PyTorch slicing using an array of indices from an argmax call

Input:
I have the following tensors / ndarrays I am operating with.
a_intents of shape (b, n_i) - Scalar at position ij is the activation of intent j for the i-th element in the batch.
u_intents of shape (b, n_i, d_m) - Vector of dimension d_m at index ij is the pose vector of intent j for the i-th element in the batch.
I want to get the index of the intent with the biggest activation scalar, and I do:
 max_activations = argmax(a_intents, dim=-1, keepdim=False)

Now, using those indices, I want to get the corresponding vectors in u_intents.
max_activation_poses = u_intents[?, ?,:]

How do I use max_activations to indicate only those indices on dim 1? My intuition tells me I will end with an incorrect shape if I do
[:, max_activations, :]

The shape I am trying to get is (b, d_m) - The vector with the same index as the highest activation, for each element in the batch.
Thanks

Output:
It's easy if you treat the u_intents vector as 2-dimensional, and offset each argmax index by its batch index times the number of elements.
# dummy values for demonstration
>>> b, n_i, d_m = 2, 3, 5
>>> a_intents = torch.rand(b, n_i)
>>> a_intents
tensor([[0.1733, 0.9965, 0.4790],
        [0.6056, 0.4040, 0.0236]])
>>> u_intents = torch.rand(b, n_i, d_m)
>>> u_intents
tensor([[[0.3301, 0.8153, 0.1356, 0.6623, 0.4385],
         [0.1902, 0.1748, 0.4131, 0.3887, 0.5363],
         [0.1211, 0.5773, 0.2405, 0.6313, 0.2064]],
        [[0.2592, 0.5127, 0.7301, 0.8883, 0.5665],
         [0.2767, 0.6545, 0.7595, 0.2677, 0.5163],
         [0.8158, 0.4940, 0.0492, 0.0911, 0.0465]]])
# add to each index the batch start
>>> max_activations = a_intents.argmax(dim=-1) + torch.arange(0, b*n_i, step=n_i)
# elements 1 and 3 of 0..5
>>> max_activations
tensor([1, 3])
>>> poses = u_intents.view(b*n_i, d_m).index_select(0, max_activations)
# tensor of shape (b, d_m) correctly indexing the maxima.
>>> poses
tensor([[0.1902, 0.1748, 0.4131, 0.3887, 0.5363],
        [0.2592, 0.5127, 0.7301, 0.8883, 0.5665]])


Instruction: Pytorch model training without using forward

Input:
I'm working on training CLIP model.
Here's the source code of the model https://github.com/openai/CLIP/blob/main/clip/model.py
Basically the CLIP object is constructed like this :
class CLIP(nn.module):
   ...
   def encode_image(self, image):
     return self.visual(image.type(self.dtype))

   def encode_text(self, text):
    x = ...
    ...
    return x

   def forward(self, image, text):
     image_features = self.encode_image(image)
     text_features = self.encode_text(text)
     ...
     return logits_per_image, logits_per_text

The forward method except pair of image and text, since I want to repurpose CLIP for other task(text-text pairs),  I'm not using forward from CLIP, but I'm using others method defined inside CLIP. My training code look like this :
for k in range(epoch):
  for batch in dataloader :
    x,y = batch
    y1 = model.encode_text(x[first_text_part])
    y2 = model.encode_text(x[second_text_part])
    <calculate loss, backward, step, etc>

The problem is, after 1 epoch, all the gradients turn out to be nan even though the loss is not nan.
My suspicion is PyTorch only able to propagate the gradient through the forward method.
Some source says that forward is not that special (https://discuss.pytorch.org/t/must-use-forward-function-in-nn-module/50943/3), but other source say coding with torch must use the forward (https://stackoverflow.com/a/58660175/12082666).
The question is, can we train Pytorch network without using forward method?

Output:
The forward() in pytorch in nothing new. It just attaches the graph of your network when called. Backpropagation doesnt rely much on forward() because, the gradients are propagated through the graph.
The only difference is that in pytorch source, forward is similar to call() method with all the hooks registered in nn.Module.


Instruction: Getting predict.proba from BERT classififer

Input:
I have a classifier on top of BERT, and I would like to see the predict probability for creating the ROC curve. How do I get the predict proba?. The predicted probas will be used to calculate the TPR FPR and threshold for ROC curve.
here is the code
class BertBinaryClassifier(nn.Module):
    def __init__(self, dropout=0.1):
        super(BertBinaryClassifier, self).__init__()
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        self.dropout = nn.Dropout(dropout)
        self.linear = nn.Linear(768, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, tokens, masks=None):
        _, pooled_output = self.bert(tokens, attention_mask=masks, output_all_encoded_layers=False)
        dropout_output = self.dropout(pooled_output)
        linear_output = self.linear(dropout_output)
        prediction = self.sigmoid(linear_output)
        return prediction
# Config setting
BATCH_SIZE = 4
EPOCHS = 5
# Making dataloaders
train_dataset =  torch.utils.data.TensorDataset(train_tokens_tensor, train_masks_tensor, train_y_tensor)
train_sampler =  torch.utils.data.RandomSampler(train_dataset)
train_dataloader =  torch.utils.data.DataLoader(train_dataset, sampler=train_sampler, batch_size=BATCH_SIZE)
test_dataset =  torch.utils.data.TensorDataset(test_tokens_tensor, test_masks_tensor, test_y_tensor)
test_sampler =  torch.utils.data.SequentialSampler(test_dataset)
test_dataloader =  torch.utils.data.DataLoader(test_dataset, sampler=test_sampler, batch_size=BATCH_SIZE)

bert_clf = BertBinaryClassifier()
bert_clf = bert_clf.cuda()
#wandb.watch(bert_clf)
optimizer = torch.optim.Adam(bert_clf.parameters(), lr=3e-6)

# training
for epoch_num in range(EPOCHS):
    bert_clf.train()
    train_loss = 0
    for step_num, batch_data in enumerate(train_dataloader):
        token_ids, masks, labels = tuple(t for t in batch_data)
        token_ids, masks, labels = token_ids.to(device), masks.to(device), labels.to(device)
        preds = bert_clf(token_ids, masks)
        loss_func = nn.BCELoss()
        batch_loss = loss_func(preds, labels)
        train_loss += batch_loss.item()
        bert_clf.zero_grad()
        batch_loss.backward()
        optimizer.step()
        #wandb.log({"Training loss": train_loss})
        print('Epoch: ', epoch_num + 1)
        print("\r" + "{0}/{1} loss: {2} ".format(step_num, len(train_data) / BATCH_SIZE, train_loss / (step_num + 1)))

# evaluating on test
bert_clf.eval()
bert_predicted = []
all_logits = []
probs=[]
with torch.no_grad():
    test_loss = 0
    for step_num, batch_data in enumerate(test_dataloader):
        token_ids, masks, labels = tuple(t for t in batch_data)
        token_ids, masks, labels = token_ids.to(device), masks.to(device), labels.to(device)
        logits = bert_clf(token_ids, masks)
        pr=logits.ravel()
        probs+=pr
        loss_func = nn.BCELoss()
        loss = loss_func(logits, labels)
        test_loss += loss.item()
        numpy_logits = logits.cpu().detach().numpy()
        #print(numpy_logits)
        #wandb.log({"Testing loss": test_loss})
        bert_predicted += list(numpy_logits[:, 0] > 0.5)
        all_logits += list(numpy_logits[:, 0])

I am able to get the prediction score to calculate the accuracy or f1 score. But not the probability for creating ROC curve.
Thanks

Output:
In your forward, you:
def forward(self, tokens, masks=None):
    _, pooled_output = self.bert(...)             # Get output of BERT
    dropout_output = self.dropout(pooled_output)
    linear_output = self.linear(dropout_output)   # Take linear combination of outputs
                                                  # (unconstrained score - "logits")

    prediction = self.sigmoid(linear_output)      # Normalise scores
                                                  # (constrained between [0,1] - "probabilities")
    return prediction

Hence the result of calling your model can be directly supplied to calculate the False Positive and True Positive rates e.g:
from sklearn import metrics

...

test_probs = bert_clf(token_ids, masks)
fpr, tpr, thresholds = metrics.roc_curve(labels, test_probs)
roc_auc = metrics.auc(fpr, tpr)


Instruction: I try to implement vggnet, but it does not be trained well

Input:
I am new at CNN.
I try to train vggnet.
class Net(nn.Module) :
def __init__(self) :
    super(Net, self).__init__()

    self.conv = nn.Sequential (
        #1
        #####
        nn.Conv2d(3,64,3, padding=1),  nn.ReLU(inplace=True),
        nn.Conv2d(64,64,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #2
        #####
        nn.Conv2d(64,128,3, padding=1), nn.ReLU(inplace=True),
        nn.Conv2d(128,128,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #####
        #3
        #####
        nn.Conv2d(128,256,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(256,256,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(256,256,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #4
        #####
        nn.Conv2d(256,512,3, padding=1), nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1), nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #5
        #####
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #####
    )
    self.fc = nn.Sequential(
       nn.Linear(512 * 7 * 7, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),
       nn.Linear(4096, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),
       nn.Linear(4096, 1000)
    )

def forward(self, x):

    # diffrent depending on the model
    x = self.conv(x)
    # change dimension
    x = torch.flatten(x, 1)
    # same
    x = self.fc(x)

    return x

But the loss is near 6.9077.
After epoch 0, it rarely change.
Even if I change weight decay to 0 (not use L2 normalization), the loss is slightly change.
My optimizer and scheduler is
optimizer = torch.optim.SGD(net.parameters(), lr=0.1, weight_decay=5e-4)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, factor = 0.1 , patience= 2 ,mode='min')
What is the problem.
Sometimes, it print bytes but only got 0.
warnings.warn(str(msg))
That is related to my problem ?

Output:
Your loss value 6.9077 is equal to -log(1/1000), which basically means your network produces random outputs out of all possible 1000 classes.
It is a bit tricky to train VGG nets from scratch, especially if you do not include batch-norm layers.
Try to reduce the learning rate to 0.01, and add momentum to your SGD.
Add more input augmentations (e.g., flips color jittering, etc.).


Instruction: Is there any alternate for tfagents in pytorch

Input:
We are using tfagents in tensorflow for reinforcement learning, because of limitations with static computation graphs we are planning to migrate our code to pytorch.
tfagents is great and have very good documentation and reduce a lot of time doing the same task again
We are wondering if the pytorch community have a similar kind of stuff?

Output: rllib is an alternative which supports PyTorch.


Instruction: How do I predict using a PyTorch model?

Input:
I created a pyTorch Model to classify images.
I saved it once via state_dict and the entire model like that:
torch.save(model.state_dict(), "model1_statedict")
torch.save(model, "model1_complete")

How can i use these models?
I'd like to check them with some images to see if they're good.
I am loading the model with:
model = torch.load(path_model)
model.eval()

This works alright, but i have no idea how to use it to predict on a new picture.

Output:
def predict(self, test_images):
    self.eval()
    # model is self(VGG class's object)

    count = test_images.shape[0]
    result_np = []

    for idx in range(0, count):
        # print(idx)
        img = test_images[idx, :, :, :]
        img = np.expand_dims(img, axis=0)
        img = torch.Tensor(img).permute(0, 3, 1, 2).to(device)
        # print(img.shape)
        pred = self(img)
        pred_np = pred.cpu().detach().numpy()
        for elem in pred_np:
            result_np.append(elem)
    return result_np

network is VGG-19 and ref my source code.
like this architecture:
class VGG(object):
    def __init__(self):
    ...

    def train(self, train_images, valid_images):
        train_dataset = torch.utils.data.Dataset(train_images)
        valid_dataset = torch.utils.data.Dataset(valid_images)

        trainloader = torch.utils.data.DataLoader(train_dataset)
        validloader = torch.utils.data.DataLoader(valid_dataset)

        self.optimizer = Adam(...)
        self.criterion = CrossEntropyLoss(...)

        for epoch in range(0, epochs):
            ...
            self.evaluate(validloader, model=self, criterion=self.criterion)
    ...

    def evaluate(self, dataloader, model, criterion):
        model.eval()
        for i, sample in enumerate(dataloader):
    ...

    def predict(self, test_images):

    ...

if __name__ == "__main__":
    network = VGG()
    trainset, validset = get_dataset()    # abstract function for showing
    testset = get_test_dataset()

    network.train(trainset, validset)

    result = network.predict(testset)


Instruction: How to convert a CNN LSTM form keras to pytorch

Input:
I am trying to convert a CNN LSTM for keras to pytorch but I have trouble.
ConvNN_model = models.Sequential()
ConvNN_model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 1)))
ConvNN_model.add(layers.MaxPooling2D((2, 2)))
ConvNN_model.add(layers.Conv2D(64, (3, 3), activation='relu'))
ConvNN_model.add(TimeDistributed(LSTM(128, activation='relu')))
ConvNN_model.add(Dropout(0.2))
ConvNN_model.add(LSTM(128, activation='relu'))
ConvNN_model.add(layers.Dense(64, activation='relu'))
ConvNN_model.add(layers.Dropout(0.25))
ConvNN_model.add(layers.Dense(15, activation='softmax'))

How to convert the above code from Keras to Pytorch?

Output:
This is your CNN in Keras:
ConvNN_model = models.Sequential()
ConvNN_model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(64, 64, 1)))
ConvNN_model.add(layers.MaxPooling2D((2, 2)))
ConvNN_model.add(layers.Conv2D(64, (3, 3), activation='relu'))
ConvNN_model.add(TimeDistributed(LSTM(128, activation='relu')))
ConvNN_model.add(Dropout(0.2))
ConvNN_model.add(LSTM(128, activation='relu'))
ConvNN_model.add(layers.Dense(64, activation='relu'))
ConvNN_model.add(layers.Dropout(0.25))
ConvNN_model.add(layers.Dense(15, activation='softmax'))

This is the equivalent code in PyTorch:
class ConvNN_model(nn.Module):
    def __init__(self):
        super(ConvNN_model, self).__init__()
        self.layers = nn.Sequential(
                         nn.Conv2d(1, 32, kernel_size=3),
                         nn.ReLU(),
                         nn.MaxPool2d((2, 2)),
                         nn.Conv2d(32, 64, kernel_size=3),
                         nn.ReLU(),
                         TimeDistributed(nn.LSTM(128, 128)),
                         nn.Dropout(0.2),
                         nn.LSTM(128, 128),
                         nn.ReLU(),
                         nn.Linear(128, 64),
                         nn.ReLU(),
                         nn.Dropout(0.25),
                         nn.Linear(64, 15),
                         nn.Softmax()
                         )
    def forward(self, x):
        return self.layers(x)

Keep in mind that there is no equivalent module for the TimeDistributed class in PyTorch, so you have to build it yourself. Here is one that you can use (from here):
class TimeDistributed(nn.Module):
    def __init__(self, module, batch_first=False):
        super(TimeDistributed, self).__init__()
        self.module = module
        self.batch_first = batch_first

    def forward(self, x):

        if len(x.size()) <= 2:
            return self.module(x)

        # Squash samples and timesteps into a single axis
        x_reshape = x.contiguous().view(-1, x.size(-1))  # (samples * timesteps, input_size)

        y = self.module(x_reshape)

        # We have to reshape Y
        if self.batch_first:
            y = y.contiguous().view(x.size(0), -1, y.size(-1))  # (samples, timesteps, output_size)
        else:
            y = y.view(-1, x.size(1), y.size(-1))  # (timesteps, samples, output_size)

        return y

There are a million-and-one ways to skin a cat; you do not necessarily have to create the entire network in the nn.Sequential block as I did. Or if you wanted to stick to the sequential method to stay consistent with Keras, you don't need to subclass nn.Module and use the sequential layers altogether.


Instruction: Shuffle patches in image batch

Input:
I am trying to create a transform that shuffles the patches of each image in a batch.
I aim to use it in the same manner as the rest of the transformations in torchvision:
trans = transforms.Compose([
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
            ShufflePatches(patch_size=(16,16)) # our new transform
        ])

More specifically, the input is a BxCxHxW tensor. I want to split each image in the batch into non-overlapping patches of size patch_size, shuffle them, and regroup into a single image.
Given the image (of size 224x224):

Using ShufflePatches(patch_size=(112,112)) I would like to produce the output image:

I think the solution has to do with torch.unfold and torch.fold, but didn't manage to get any further.
Any help would be appreciated!

Output:
Indeed unfold and fold seem appropriate in this case.
import torch
import torch.nn.functional as nnf

class ShufflePatches(object):
  def __init__(self, patch_size):
    self.ps = patch_size

  def __call__(self, x):
    # divide the batch of images into non-overlapping patches
    u = nnf.unfold(x, kernel_size=self.ps, stride=self.ps, padding=0)
    # permute the patches of each image in the batch
    pu = torch.cat([b_[:, torch.randperm(b_.shape[-1])][None,...] for b_ in u], dim=0)
    # fold the permuted patches back together
    f = nnf.fold(pu, x.shape[-2:], kernel_size=self.ps, stride=self.ps, padding=0)
    return f

Here's an example with patch size=16:


Instruction: Understanding Loss Functions

Input: I was following this pytorch tutorial on how to setup a neural network, but I don't understand the loss function, loss = (y_pred - y).pow(2).sum().item(). Why is this being used rather than the derivative of the function used to calculate the predicted y value? I also don't understand what that function returns.

Output:
That function is the euclidean L2-norm. It returns the sum of the squared errors between network output and expected output.
As for the derivative of the function, or better its gradient, it it computed internally by the deep learning framework you are using (here pytorch I assume) and is needed to update the network parameter. For most use cases, you do not need to think at it. Its computation is totally automatic.
One note: if you call .item() on a tensor, you are extracting its raw value, i.e. what you get is no more a tensor but just a number. This means that you cannot compute the gradient from it (call .backward()).


Instruction: CNN in pytorch "Expected 4-dimensional input for 4-dimensional weight [32, 1, 5, 5], but got 3-dimensional input of size [16, 64, 64] instead"

Input:
I am new to pytorch. I am trying to use chinese mnist dataset to train the neural network that shows in below code. Is that a problem of the neural network input or something else goes wrong in my code. I have tried many ways to fix it but instead it shows me other errors
train_df = chin_mnist_df.groupby('value').apply(lambda x: x.sample(700, random_state=SEED)).reset_index(drop=True)
x_train, y_train  = train_df.iloc[:, :-2], train_df.iloc[:, -2]

valid_df = chin_mnist_df.groupby('value').apply(lambda x: x.sample(200, random_state=SEED)).reset_index(drop=True)
x_valid, y_valid  = valid_df.iloc[:, :-2], valid_df.iloc[:, -2]

test_df  = chin_mnist_df.groupby('value').apply(lambda x: x.sample(100, random_state=SEED)).reset_index(drop=True)
x_test, y_test    = test_df.iloc[:, :-2], test_df.iloc[:, -2]

train_ds = Dataset(x_train, y_train)
train_dataloader = torch.utils.data.DataLoader(train_ds, batch_size=16, shuffle=True)

valid_ds = Dataset(x_valid, y_valid)
valid_dataloader = torch.utils.data.DataLoader(valid_ds, batch_size=16, shuffle=True)

test_ds = Dataset(x_test, y_test)
test_dataloader = torch.utils.data.DataLoader(test_ds, batch_size=16, shuffle=True)

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(7 * 7 * 64, 1000)
        self.fc2 = nn.Linear(1000, 15)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

model = ConvNet()
klisi=[]
apoklisi=[]
apoklisi2=[]
klisi2=[]
olatalr=[]
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
total_step = len(train_dataloader)
loss_list = []
acc_list = []
for epoch in range(num_epochs):
    for i,data in enumerate(train_dataloader):#(images, labels)
        batch_inputs, batch_labels = data[0][:].to(device).type(torch.float), data[1][:].to(device)
        # Run the forward pass
        outputs = model(batch_inputs)
        loss = criterion(outputs, batch_labels)

Output:
Your training images are greyscale images. That is, they only have one channel (as opposed to the three RGB color channels in color images).
It seems like your Dataset (implicitly) "squeezes" this singleton dimension, and instead of having a batch of shape BxCxHxW = 16x1x64x64, you end up with a batch of shape 16x64x64.
Try:
# ...
batch_inputs, batch_labels = data[0][:].to(device).type(torch.float), data[1][:].to(device)
batch_inputs = batch_inputs[:, None, ...]  # explicitly add the singleton channel dimension
# Run the forward pass
# ...