Instruction: What is the purpose of [np.arange(0, self.batch_size), action] after the neural network?

Input:
I followed a PyTorch tutorial to learn reinforcement learning(TRAIN A MARIO-PLAYING RL AGENT) but I am confused about the following code:
current_Q = self.net(state, model="online")[np.arange(0, self.batch_size), action] # Q_online(s,a)

What's the purpose of [np.arange(0, self.batch_size), action] after the neural network?(I know that TD_estimate takes in state and action, just confused about this on the programming side) What is this usage(put a list after self.net)?
More related code referenced from the tutorial:
class MarioNet(nn.Module):

def __init__(self, input_dim, output_dim):
    super().__init__()
    c, h, w = input_dim

    if h != 84:
        raise ValueError(f"Expecting input height: 84, got: {h}")
    if w != 84:
        raise ValueError(f"Expecting input width: 84, got: {w}")

    self.online = nn.Sequential(
        nn.Conv2d(in_channels=c, out_channels=32, kernel_size=8, stride=4),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
        nn.ReLU(),
        nn.Flatten(),
        nn.Linear(3136, 512),
        nn.ReLU(),
        nn.Linear(512, output_dim),
    )

    self.target = copy.deepcopy(self.online)

    # Q_target parameters are frozen.
    for p in self.target.parameters():
        p.requires_grad = False

def forward(self, input, model):
    if model == "online":
        return self.online(input)
    elif model == "target":
        return self.target(input)

self.net:
self.net = MarioNet(self.state_dim, self.action_dim).float()

Thanks for any help!

Output:
Essentially, what happens here is that the output of the net is being sliced to get the desired part of the Q table.
The (somewhat confusing) index of [np.arange(0, self.batch_size), action] indexes each axis. So, for axis with index 1, we pick the item indicated by action. For index 0, we pick all items between 0 and self.batch_size.
If self.batch_size is the same as the length of dimension 0 of this array, then this slice can be simplified to [:, action] which is probably more familiar to most users.


Instruction: Clarification about Gradient Accumulation

Input:
I'm trying to get a better understanding of how Gradient Accumulation works and why it is useful. To this end, I wanted to ask what is the difference (if any) between these two possible PyTorch-like implementations of a custom training loop with gradient accumulation:
gradient_accumulation_steps = 5
for batch_idx, batch in enumerate(dataset):
  x_batch, y_true_batch = batch
  y_pred_batch = model(x_batch)

  loss = loss_fn(y_true_batch, y_pred_batch)
  loss.backward()

  if (batch_idx + 1) % gradient_accumulation_steps == 0: # (assumption: the number of batches is a multiple of gradient_accumulation_steps)
    optimizer.step()
    optimizer.zero_grad()

y_true_batches, y_pred_batches = [], []
gradient_accumulation_steps = 5
for batch_idx, batch in enumerate(dataset):
  x_batch, y_true_batch = batch
  y_pred_batch = model(x_batch)

  y_true_batches.append(y_true_batch)
  y_pred_batches.append(y_pred_batch)

  if (batch_idx + 1) % gradient_accumulation_steps == 0: # (assumption: the number of batches is a multiple of gradient_accumulation_steps)
    y_true = stack_vertically(y_true_batches)
    y_pred = stack_vertically(y_pred_batches)

    loss = loss_fn(y_true, y_pred)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    y_true_batches.clear()
    y_pred_batches.clear()

Also, kind of as an unrelated question: Since the purpose of gradient accumulation is to mimic a larger batch size in cases where you have memory constraints, does it mean that I should also increase the learning rate proportionally?

Output:
1. The difference between the two programs:
Conceptually, your two implementations are the same: you forward gradient_accumulation_steps batches for each weight update.
As you already observed, the second method requires more memory resources than the first one.
There is, however, a slight difference: usually, loss functions implementation use mean to reduce the loss over the batch. When you use gradient accumulation (first implementation) you reduce using mean over each mini-batch, but using sum over the accumulated gradient_accumulation_steps mini-batches. To make sure the accumulated gradient implementation is identical to large batches implementation you need to be very careful in the way the loss function is reduced. In many cases you will need to divide the accumulated loss by gradient_accumulation_steps. See this answer for a detailed imlpementation.

2. Batch size and learning rate:
Learning rate and batch size are indeed related. When increasing the batch size one usually reduces the learning rate.
See, e.g.:
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, Don't Decay the Learning Rate, Increase the Batch Size (ICLR 2018).


Instruction: How to calculate perplexity of a sentence using huggingface masked language models?

Input:
I have several masked language models (mainly Bert, Roberta, Albert, Electra). I also have a dataset of sentences. How can I get the perplexity of each sentence?
From the huggingface documentation here they mentioned that perplexity "is not well defined for masked language models like BERT", though I still see people somehow calculate it.
For example in this SO question they calculated it using the function
def score(model, tokenizer, sentence,  mask_token_id=103):
  tensor_input = tokenizer.encode(sentence, return_tensors='pt')
  repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
  mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
  masked_input = repeat_input.masked_fill(mask == 1, 103)
  labels = repeat_input.masked_fill( masked_input != 103, -100)
  loss,_ = model(masked_input, masked_lm_labels=labels)
  result = np.exp(loss.item())
  return result

score(model, tokenizer, '我爱你') # returns 45.63794545581973

However, when I try to use the code I get TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'.
I tried it with a couple of my models:
from transformers import pipeline, BertForMaskedLM, BertForMaskedLM, AutoTokenizer, RobertaForMaskedLM, AlbertForMaskedLM, ElectraForMaskedLM
import torch

1)
tokenizer = AutoTokenizer.from_pretrained("bioformers/bioformer-cased-v1.0")
model = BertForMaskedLM.from_pretrained("bioformers/bioformer-cased-v1.0")
2)
tokenizer = AutoTokenizer.from_pretrained("sultan/BioM-ELECTRA-Large-Generator")
model = ElectraForMaskedLM.from_pretrained("sultan/BioM-ELECTRA-Large-Generator")

This SO question also used the masked_lm_labels as an input and it seemed to work somehow.

Output:
There is a paper Masked Language Model Scoring that explores pseudo-perplexity from masked language models and shows that pseudo-perplexity, while not being theoretically well justified, still performs well for comparing "naturalness" of texts.
As for the code, your snippet is perfectly correct but for one detail: in recent implementations of Huggingface BERT, masked_lm_labels are renamed to simply labels, to make interfaces of various models more compatible. I have also replaced the hard-coded 103 with the generic tokenizer.mask_token_id. So the snippet below should work:
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
import numpy as np

model_name = 'cointegrated/rubert-tiny'
model = AutoModelForMaskedLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

def score(model, tokenizer, sentence):
    tensor_input = tokenizer.encode(sentence, return_tensors='pt')
    repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
    mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
    masked_input = repeat_input.masked_fill(mask == 1, tokenizer.mask_token_id)
    labels = repeat_input.masked_fill( masked_input != tokenizer.mask_token_id, -100)
    with torch.inference_mode():
        loss = model(masked_input, labels=labels).loss
    return np.exp(loss.item())

print(score(sentence='London is the capital of Great Britain.', model=model, tokenizer=tokenizer))
# 4.541251105675365
print(score(sentence='London is the capital of South America.', model=model, tokenizer=tokenizer))
# 6.162017238332462

You can try this code in Google Colab by running this gist.


Instruction: Evaluating my object detection model using COCO metric shows 0 and -1 values

Input:
I'm currently trying to solve an object detection problem and decided to use faster RCNN on for this. I followed this Youtube video and their Code. The loss decreases but the big problem is it won't evaluate correctly no matter how I try to. I've tried looking into the inputs, if there is any sort of size mismatch or missing information but it still doesn't work. It's always showing -1 and 0 values for all of its metrics like this.
creating index...
index created!
Test:  [0/1]  eta: 0:00:08  model_time: 0.4803 (0.4803)  evaluator_time: 0.0304 (0.0304)  time: 8.4784  data: 7.9563  max mem: 7653
Test: Total time: 0:00:08 (8.6452 s / it)
Averaged stats: model_time: 0.4803 (0.4803)  evaluator_time: 0.0304 (0.0304)
Accumulating evaluation results...
DONE (t=0.01s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
<coco_eval.CocoEvaluator at 0x7ff9989fea10>

Here is my current code: Colab notebook

Output:
My labels were given wrong. I figured this out by trying to plot my dataset image with its labels and I figured out that it either wasn't showing the labels or not showing it accurately.
This evaluation function is based on COCO metric. It evaluates labels of all sizes so it is showing -1.000 for area=large. My current guess that it is because my dataset doesn't have varying sizes of labels. They are all of equal sizes and they are medium/small in size. I might be wrong.


Instruction: Pytorch Tensors using all RAM

Input:
I have a list of tensors, which is too heavy for my RAM. I would like to save them in filesystem and load them when needed
torch.save(single_tensor, 'tensor_<idx>.pt')

If I want to use batches while training, is there an automatic way to load tensors when needed?
I was thinking about using TensorDataset and DataLoader, but since now I don't have tensors in a list but in filesystem, how should I build them?

Output:
Firstly save the tensors one by one to file with torch.save()
torch.save(tensor, 'path/to/file.pt')

Then this Dataset class allows to load the tensors only when they are really needed:
class EmbedDataset(torch.utils.data.Dataset):
    def __init__(self, first_embed_path, second_embed_path, labels):
        self.first_embed_path = first_embed_path
        self.second_embed_path = second_embed_path
        self.labels = labels

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, i):

        label = self.labels[i]

        embed = torch.load(os.path.join(self.first_embed_path, str(i) + '.pt'))

        pos = torch.load(os.path.join(self.second_embed_path, str(i) + '.pt'))

        tensor = torch.cat((embed, pos))

        return tensor, label

Here the tensors are named with numbers, eg 1.pt or 1816.pt


Instruction: Does pytorch use the cudatoolkit in the conda environment or the system?

Input:
I installed pytorch and torchvision in my conda environment with pip install torch==1.5.1+cu101 torchvision==0.6.1+cu101, to my understanding this means that the pytorch library is compiled with cuda10.1. And upon running nvcc --version , I get
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

And I assume this means that the cudatoolkit in my system is cuda9.1, but if I then go on to install a different version of cudatoolkit in my conda environment with conda install -c anaconda cudatoolkit=10.1. Which cudatoolkit will pytorch use?

I used pip install for pytorch because this was the instruction given in the original repo I am planning to use.

Output:
Yes, the pip wheels and conda binaries ship with their own CUDA runtime (as well as cuDNN. NCCL etc.) so you would only need to install the NVIDIA driver. If you want to build PyTorch from source or a custom CUDA extension, the local CUDA toolkit will be used.

As answered in the link here.


Instruction: Concatenate layers with different sizes in PyTorch

Input:
In Keras, it is possible to concatenate two layers of different sizes:
# Keras — this works, conceptually
layer_1 = Embedding(50, 5)(inputs)
layer_2 = Embedding(300, 20)(inputs)
concat = Concatenate()([layer_1, layer_2])
# -> `concat` now has shape `(*, 25)`, as desired

But PyTorch keeps complaining that the two layers have different sizes:
# PyTorch — this does not work
class MyModel(torch.nn.Module):
    def __init__(self):
        self.layer1 = Embedding(50, 5)
        self.layer2 = Embedding(300, 20)

    def forward(self, inputs):
        layer_1 = self.layer1(inputs)
        layer_2 = self.layer2(inputs)
        concat = torch.cat([layer_1, layer_2])

The code just above results in this error:

RuntimeError: Sizes of tensors must match except in dimension 0.
Expected size 5 but got size 20 for tensor number 1 in the list.

The final concat layer I want is a layer of size 25 made of the concatenation of the two source layers.
As the two source layers are Embedding layers, I do not see as optimal that they would share the same dimension. In this example, using an embedding dimension of 5 for a vocabulary of 50 items, and an embedding dimension of size 20 for a vocabulary of 200 items.
How should this problem be solved in PyTorch?

Output:
Indeed torch.cat will apply the concatenation on the first axis. Since you are looking to concatenate on the second axis, you should provide the dim argument as:
>>> concat = torch.cat([layer_1, layer_2], dim=1)


Instruction: DeepSORT's Feature extractor cannot be used for Person ReIdentification

Input:
I am using this repo for DeepSORT - https://github.com/nwojke/deep_sort
I am trying to build a Multi Camera Person Tracking system. I want to save and utilize the features extracted by one camera in footage from other cameras.
The Feature extractor which is trained on Mars dataset, doesn't seem to help in differentiating between two different people.
I wrote the below snippet to check the Cosine Distance between images from same person and different person.
extr = Extractor("./deep_sort_pytorch/deep_sort/deep/checkpoint/ckpt.t7")
list = glob.glob("mars/*.jpg")
features_list = []

for i in list:
    im = cv2.imread(i)
    im_crops = [im]
    features = extr(im_crops)
    features_list.append(features)

for f in features_list:
    print(_cosine_distance(f, features_list[0]),"<<")

def _cosine_distance(a, b, data_is_normalized=False):
    cos = nn.CosineSimilarity(dim=1, eps=1e-6)
    return (1. - cos(torch.from_numpy(a), torch.from_numpy(b)))

As expected, the cosine distance between images of same person is very low.
But unexpectedly the cosine distance between crops of two different people is also similarly low.
I thought the Feature extractor will help me in differentiating.
Shall I increase the latent space dimensions from 512 to a bigger size?
Or maybe I am mistaking the role of Feature extractor.

Output:
A slightly larger feature space may help. But your main issue is the architecture of the feature extractor. In order to match people and distinguish them from impostors, features corresponding small local regions (e.g. shoes, glasses) and global whole body regions are equally important. This is not captured by the simple feature extractor provided by https://github.com/nwojke/deep_sort. For more information on this check: https://arxiv.org/pdf/1905.00953.pdf. I recommend you to try any of the OSNet models provided here: https://kaiyangzhou.github.io/deep-person-reid/MODEL_ZOO
I can also recommend you to check out my repository: https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch. It seems to provide all you need: multi-camera multi-object tracking and OSNet models


Instruction: How to generate indices like [0,2,4,1,3,5] without using explicit loop for reorganizing rows of a tensors in Pytorch?

Input:
Suppose I have a Tensor like
a = torch.tensor([[3, 1, 5, 0, 4, 2],
                  [2, 1, 3, 4, 5, 0],
                  [0, 4, 5, 1, 2, 3],
                  [3, 1, 4, 5, 0, 2],
                  [3, 5, 4, 2, 0, 1],
                  [5, 3, 0, 4, 1, 2]])

and I want to reorganize the rows of the tensor by applying the transformation a[c] where
c = torch.tensor([0,2,4,1,3,5])

to get
b = torch.tensor([[3, 1, 5, 0, 4, 2],
                  [0, 4, 5, 1, 2, 3],
                  [3, 5, 4, 2, 0, 1],
                  [2, 1, 3, 4, 5, 0],
                  [3, 1, 4, 5, 0, 2],
                  [5, 3, 0, 4, 1, 2]])

For doing it, I want to generate the tensor c so that I can do this transformation irrespective of the size of tensor a and the stepping size (which I have taken to be equal to 2 in this example for simplicity). Can anyone let me know how do I generate such a tensor for the general case without using an explicit for loop in PyTorch?

Output:
You can use torch.index_select, so:
b = torch.index_select(a, 0, c)

The explanation in the official docs is pretty clear.


Instruction: name '_get_ade20k_pairs' is not defined

Input:
So I'm trying to make function for preprocessing dataaset in semantic segmentation. but it tells me that my function is not define. Whereas is actually define on there. my code is like this
import os
import random
import numpy as np
import torch
import torch.utils.data as data

from PIL import Image, ImageOps, ImageFilter

__all__ = ['ADE20KSegmentation']

class ADE20KSegmentation(data.Dataset):

    BASE_DIR = 'ADEChallengeData2016'
    NUM_CLASS = 150
    CLASSES = ("wall", "building, edifice", "sky", "floor, flooring", "tree",
               "ceiling", "road, route", "bed", "windowpane, window", "grass",
               "cabinet", "sidewalk, pavement",
               "person, individual, someone, somebody, mortal, soul",
               "earth, ground", "door, double door", "table", "mountain, mount",
               "plant, flora, plant life", "curtain, drape, drapery, mantle, pall",
               "chair", "car, auto, automobile, machine, motorcar",
               "water", "painting, picture", "sofa, couch, lounge", "shelf",
               "house", "sea", "mirror", "rug, carpet, carpeting", "field", "armchair",
               "seat", "fence, fencing", "desk", "rock, stone", "wardrobe, closet, press",
               "lamp", "bathtub, bathing tub, bath, tub", "railing, rail", "cushion",
               "base, pedestal, stand", "box", "column, pillar", "signboard, sign",
               "chest of drawers, chest, bureau, dresser", "counter", "sand", "sink",
               "skyscraper", "fireplace, hearth, open fireplace", "refrigerator, icebox",
               "grandstand, covered stand", "path", "stairs, steps", "runway",
               "case, display case, showcase, vitrine",
               "pool table, billiard table, snooker table", "pillow",
               "screen door, screen", "stairway, staircase", "river", "bridge, span",
               "bookcase", "blind, screen", "coffee table, cocktail table",
               "toilet, can, commode, crapper, pot, potty, stool, throne",
               "flower", "book", "hill", "bench", "countertop",
               "stove, kitchen stove, range, kitchen range, cooking stove",
               "palm, palm tree", "kitchen island",
               "computer, computing machine, computing device, data processor, "
               "electronic computer, information processing system",
               "swivel chair", "boat", "bar", "arcade machine",
               "hovel, hut, hutch, shack, shanty",
               "bus, autobus, coach, charabanc, double-decker, jitney, motorbus, "
               "motorcoach, omnibus, passenger vehicle",
               "towel", "light, light source", "truck, motortruck", "tower",
               "chandelier, pendant, pendent", "awning, sunshade, sunblind",
               "streetlight, street lamp", "booth, cubicle, stall, kiosk",
               "television receiver, television, television set, tv, tv set, idiot "
               "box, boob tube, telly, goggle box",
               "airplane, aeroplane, plane", "dirt track",
               "apparel, wearing apparel, dress, clothes",
               "pole", "land, ground, soil",
               "bannister, banister, balustrade, balusters, handrail",
               "escalator, moving staircase, moving stairway",
               "ottoman, pouf, pouffe, puff, hassock",
               "bottle", "buffet, counter, sideboard",
               "poster, posting, placard, notice, bill, card",
               "stage", "van", "ship", "fountain",
               "conveyer belt, conveyor belt, conveyer, conveyor, transporter",
               "canopy", "washer, automatic washer, washing machine",
               "plaything, toy", "swimming pool, swimming bath, natatorium",
               "stool", "barrel, cask", "basket, handbasket", "waterfall, falls",
               "tent, collapsible shelter", "bag", "minibike, motorbike", "cradle",
               "oven", "ball", "food, solid food", "step, stair", "tank, storage tank",
               "trade name, brand name, brand, marque", "microwave, microwave oven",
               "pot, flowerpot", "animal, animate being, beast, brute, creature, fauna",
               "bicycle, bike, wheel, cycle", "lake",
               "dishwasher, dish washer, dishwashing machine",
               "screen, silver screen, projection screen",
               "blanket, cover", "sculpture", "hood, exhaust hood", "sconce", "vase",
               "traffic light, traffic signal, stoplight", "tray",
               "ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, "
               "dustbin, trash barrel, trash bin",
               "fan", "pier, wharf, wharfage, dock", "crt screen",
               "plate", "monitor, monitoring device", "bulletin board, notice board",
               "shower", "radiator", "glass, drinking glass", "clock", "flag")
    def __init__(self, root='/content/dataset', split='training', mode=None, transform=None,
                 base_size=520, crop_size=480, **kwargs):
        super(ADE20KSegmentation, self).__init__()
        self.root = root
        self.split = split
        self.mode = mode if mode is not None else split
        self.transform = transform
        self.base_size = base_size
        self.crop_size = crop_size
        self.images, self.mask_paths = _get_ade20k_pairs(self.root, self.split)
        assert (len(self.images) == len(self.mask_paths))
        if len(self.images) == 0:
            raise RuntimeError("Found 0 images in subfolders of: " + self.root + "\n")
        self.valid_classes = [7, 8, 11, 12, 13, 17, 19, 20, 21, 22,
                              23, 24, 25, 26, 27, 28, 31, 32, 33]
        self._key = np.array([-1, -1, -1, -1, -1, -1,
                              -1, -1, 0, 1, -1, -1,
                              2, 3, 4, -1, -1, -1,
                              5, -1, 6, 7, 8, 9,
                              10, 11, 12, 13, 14, 15,
                              -1, -1, 16, 17, 18])
        self._mapping = np.array(range(-1, len(self._key) - 1)).astype('int32')

    def _class_to_index(self, mask):
        values = np.unique(mask)
        for value in values:
            assert (value in self._mapping)
        index = np.digitize(mask.ravel(), self._mapping, right=True)
        return self._key[index].reshape(mask.shape)

    def __getitem__(self, index):
        img = PIL.Image.open(self.images[index]).convert('RGB')
        if self.mode == 'test':
            if self.transform is not None:
                img = self.transform(img)
            return img, os.path.basename(self.images[index])
        mask = PIL.Image.open(self.mask_paths[index])
        # synchrosized transform
        if self.mode == 'training':
            img, mask = self._sync_transform(img, mask)
        elif self.mode == 'valdation':
            img, mask = self._val_sync_transform(img, mask)
        else:
            assert self.mode == 'testval'
            img, mask = self._img_transform(img), self._mask_transform(mask)
        # general resize, normalize and toTensor
        if self.transform is not None:
            img = self.transform(img)
        return img, mask

    def _val_sync_transform(self, img, mask):
        outsize = self.crop_size
        short_size = outsize
        w, h = img.size
        if w > h:
            oh = short_size
            ow = int(1.0 * w * oh / h)
        else:
            ow = short_size
            oh = int(1.0 * h * ow / w)
        img = img.resize((ow, oh), Image.BILINEAR)
        mask = mask.resize((ow, oh), Image.NEAREST)
        # center crop
        w, h = img.size
        x1 = int(round((w - outsize) / 2.))
        y1 = int(round((h - outsize) / 2.))
        img = img.crop((x1, y1, x1 + outsize, y1 + outsize))
        mask = mask.crop((x1, y1, x1 + outsize, y1 + outsize))
        # final transform
        img, mask = self._img_transform(img), self._mask_transform(mask)
        return img, mask

    def _sync_transform(self, img, mask):
        # random mirror
        if random.random() < 0.5:
            img = img.transpose(Image.FLIP_LEFT_RIGHT)
            mask = mask.transpose(Image.FLIP_LEFT_RIGHT)
        crop_size = self.crop_size
        # random scale (short edge)
        short_size = random.randint(int(self.base_size * 0.5), int(self.base_size * 2.0))
        w, h = img.size
        if h > w:
            ow = short_size
            oh = int(1.0 * h * ow / w)
        else:
            oh = short_size
            ow = int(1.0 * w * oh / h)
        img = img.resize((ow, oh), Image.BILINEAR)
        mask = mask.resize((ow, oh), Image.NEAREST)
        # pad crop
        if short_size < crop_size:
            padh = crop_size - oh if oh < crop_size else 0
            padw = crop_size - ow if ow < crop_size else 0
            img = ImageOps.expand(img, border=(0, 0, padw, padh), fill=0)
            mask = ImageOps.expand(mask, border=(0, 0, padw, padh), fill=0)
        # random crop crop_size
        w, h = img.size
        x1 = random.randint(0, w - crop_size)
        y1 = random.randint(0, h - crop_size)
        img = img.crop((x1, y1, x1 + crop_size, y1 + crop_size))
        mask = mask.crop((x1, y1, x1 + crop_size, y1 + crop_size))
        # gaussian blur as in PSP
        if random.random() < 0.5:
            img = img.filter(ImageFilter.GaussianBlur(
                radius=random.random()))
        # final transform
        img, mask = self._img_transform(img), self._mask_transform(mask)
        return img, mask

    def _img_transform(self, img):
        return np.array(img)

    def _mask_transform(self, mask):
        target = self._class_to_index(np.array(mask).astype('int32'))
        return torch.LongTensor(np.array(target).astype('int32'))

    def __len__(self):
        return len(self.images)

    @property
    def classes(self):
        """Category names."""
        return type(self).CLASSES

    @property
    def pred_offset(self):
        return 1

    def _get_ade20k_pairs(folder, mode='training'):
        def get_path_pairs(img_folder, mask_folder):
          img_paths = []
          mask_paths = []
          for root, _, files in os.walk(img_folder):
              for filename in files:
                  if filename.endswith(".jpg"):
                      imgpath = os.path.join(root, filename)
                      foldername = os.path.basename(os.path.dirname(imgpath))
                      maskname = filename.replace('images', 'annotations')
                      maskpath = os.path.join(mask_folder, foldername, maskname)
                      if os.path.isfile(imgpath) and os.path.isfile(maskpath):
                          img_paths.append(imgpath)
                          mask_paths.append(maskpath)
                      else:
                          print('cannot find the mask or image:', imgpath, maskpath)
          print('Found {} images in the folder {}'.format(len(img_paths), img_folder))
          return img_paths, mask_paths

        if split in ('training', 'validation'):
            img_folder = os.path.join(folder, 'images/' + split)
            mask_folder = os.path.join(folder, 'annotations/' + split)
            img_paths, mask_paths = get_path_pairs(img_folder, mask_folder)
            return img_paths, mask_paths
        else:
            assert split == 'trainval'
            print('trainval set')
            train_img_folder = os.path.join(folder, 'images/training')
            train_mask_folder = os.path.join(folder, 'annotations/training')
            val_img_folder = os.path.join(folder, 'images/validation')
            val_mask_folder = os.path.join(folder, 'annotations/validation')
            train_img_paths, train_mask_paths = get_path_pairs(train_img_folder, train_mask_folder)
            val_img_paths, val_mask_paths = get_path_pairs(val_img_folder, val_mask_folder)
            img_paths = train_img_paths + val_img_paths
            mask_paths = train_mask_paths + val_mask_paths
        return img_paths, mask_paths

if __name__ == '__main__':
    dataset = ADE20KSegmentation()
    img, label = dataset[0]

when I run that it always showing NameError: name '_get_ade20k_pairs' is not defined. whats wrong with my definition? because sometime it can get run and sometime it can't

Output:
I suppose you were copying the code from here and you failed to copy _get_ade20k_pairs correctly.
You need it indented with 0 tabs.


Instruction: Is it possible to execute from the point where the neural network model is interrupted?

Input:
Assume that I am training a neural network model. I am storing the tensor file of the neural network model for every 15 epochs in .pth format.
I need to run 1000 epochs in total. Suppose I stopped my program during the 501st epoch, then I have the following files
15.pth, 30.pth, 45.pth, 60.pth, 75.pth,.... 420.pth, 435.pth, 450.pth, 465.pth, 480.pth, 495.pth

Then my doubt is
Is it possible to use the last stored model 495.pth and continue execution as it generally happens if done without any interruption? In short, I am asking for something similar to the "resumption" of the training phase with a few modifications to the existing code. I am just asking for such a possibility.
I am asking for general practice and not particular to any code. If such a method exists, I will be free to stop any program under execution and can resume later. Currently, I cannot use resources for shorter programs if longer programs are in execution and hence I am asking this question.

Output:
I order to resume training from a checkpoint, you need to save the entire state of your training process. This includes:

Current weights of the model.
State of the optimizer: most optimizers keep track of different statistics of the updates, e.g., momentum, variance etc.
State of the learning rate scheduler.
Additional "state" variables unique to your code.

If you saved all this information, you should be able to fully restore the "state" of your training process and resume from that point.


Instruction: Why torch does train using mini-batches?

Input:
I'm currently tring to understand how to train models via pytorch. And while this I saw a pretty interesting feature: passing to train data --- is a mini-batch. For ex.
There is a code fragment from official pytorch web-site
...

data_dir = 'data/hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}

...

...

for inputs, labels in dataloaders[phase]:
    inputs = inputs.to(device)
    labels = labels.to(device)

    optimizer.zero_grad()

    with torch.set_grad_enabled(phase == 'train'):
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)

...

According to this code, inputs that passing to model is a mini-batch.
I've tied to fiend some info about this, but unsuccessfully.
But I'm really curious, is it some kind of boost (parallel run, etc.) or neccussary thing. So, would you mind to help me figure it out and tell me, why there is a mini-batch passing to train function?
N.B. Will not refuse a link to the paper [smily :)].

Output:
Imagine it this way; say you want to learn the difference between a dog and a cat, and you have never seen them before.
The batches would be that we show you, say, 10 images of a dog and a cat at a time. You can, rather fast, learn the some of differences of cats and dogs after say 4x10 images (4 batches) but you'll of course be biased if e.g all the dogs you've been showed so far have been large dogs, thus you might classify all small dogs as cats. After enough batches you'll learn, and then unlearn, different features since you don't see them all at once, but the important thing is, you start learning something fast.
On the other hand, say we show you 100 images instead of 10. It will take much longer for you to look all the images through and compare them to each other, but you'll learn the differences in "one go" so to speak.
Either way; when you have processed those images (either as batches or the entire dataset), I can then show you one image, and you can tell me if it is a dog or a cat, even though you have learned from multiple images.


Instruction: Get the mean of each column for a 2D PyTorch tensor object

Input:
I want to know what is the most simple way to get the mean of the matrix along each column, namely my tensor only has two dimensions with shape (m X n).
For example, if I have a tensor object
T = torch.FloatTensor([[2.6, 5], [1.7, 6], [3.2, 7], [2.1, 8]])

I want some function that could return a tensor object as following
([2.4, 6.5])

I know I could do with for loop, but I wonder if there is a PyTorch built-in method that could do this for me, or is there some more elegant way for this purpose.
for i in range(T.size(1)):
  mean[i] = torch.mean(T[:, i])

Thanks for any help.

Output:
Please read the doc on torch.mean.
It has an optional parameter dim:

dim (int or tuple of ints) – the dimension or dimensions to reduce.

In your case:
T.mean(dim=0)

Will give you the mean along the first dim.
Note that many other "reduction" operations in pytorch (and in numpy) also have this option to reduce along a specific dimension only (e.g., std, sum etc.).


Instruction: Pytorch Model Optimization: Automatic Mixed Precision vs Quantization?

Input:
I'm trying to optimize my pytorch model. I understand the basics of quantization (changing 32 bit floats to other data types in 16 bit or 8bit), but I'm lost on how the two methods differ or what to choose.
I see AMP (Automatic Mixed Precision) https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html
and regular Quantization https://pytorch.org/tutorials/recipes/quantization.html.
Could someone please explain the difference and applications?
Thank you.

Output:
Automatic Mixed Precision (AMP)'s main goal is to reduce training time. On the other hand, quantization's goal is to increase inference speed.
AMP: Not all layers and operations require the precision of fp32, hence it's better to use lower precision. AMP takes care of what precision to use for what operation. It eventually helps speed up the training.

Mixed precision tries to match each op to its appropriate datatype, which can reduce your network’s runtime and memory footprint.

Also, note that the max performance gain is observed on Tensor Core-enabled GPU architectures.
Quantization converts 32-bit floating numbers in your model parameters to 8-bit integers. This will significantly decrease the model size and increase the inference speed. However, it could severly impact the model's accuracy. That's why you can utilize techniques like Quantization Aware Training (QAT). Rest you can read on the tutorials you shared.


Instruction: How to invert the permutations in each row of a 2D Pytorch tensor without using for loop?

Input:
Suppose I have a 2D tensor that contains a permutation in each row, for example
a = torch.tensor([[4, 3, 5, 0, 2, 1],
                  [5, 0, 3, 4, 2, 1],
                  [3, 1, 0, 2, 4, 5],
                  [5, 0, 4, 3, 2, 1],
                  [2, 4, 0, 1, 5, 3]])

I want to invert all the permutations in tensor a to get a tensor b. For example, after doing this operation on the above tensor a my desired output should be
>>> b
tensor([[3, 5, 4, 1, 0, 2],
        [1, 5, 4, 2, 3, 0],
        [2, 1, 3, 0, 4, 5],
        [1, 5, 4, 3, 2, 0],
        [2, 3, 0, 5, 1, 4]])

I tried to search online and found out this answer. How should I generalize the approach mentioned in that answer to invert all the permutations without using a for loop?

Output:
It can be done using torch.Tensor.scatter_
b = torch.zeros_like(a)
b.scatter_(1, a, torch.arange(a.size(1)).expand(a.size(0),-1))


Instruction: How to index tensor by using all but given indices in pytorch?

Input:
Suppose I have a Tensor a
>>> a
tensor([[2, 8],
        [3, 0],
        [4, 2],
        [2, 2],
        [6, 8]])

I also have a tensor idx consisting of indices that I don't want to select
>>> idx
tensor([0, 3, 4])

How should I go ahead and select all rows except the ones whose indices are present in the tensor idx?
For the above tensors a and idx, after selection, I should get the output Tensor b as
>>> b
tensor([[3, 0],
        [4, 2]])

Output: a[[i for i in range(len(a)) if i not in idx]]


Instruction: How to dynamically slice array without eval?

Input:
I am in a situation where I need to dynamically slice an array of variable shape/ dimension. So I assemble a string to access it.
This works fine when accessing the data.
some_data = eval("arr[:,[ci]]")

#or

some_data = eval("arr[:,[ci],:]")

#or

some_data = eval("arr[:,:,:,[ci]]")

#etc.

But the limitations of eval() prevent me from assigning using this approach.
eval("arr[:,[ci]]") = some_data

Is there a way I can slice without eval()?

Output:
In Python, the indexing operation for a type is implemented by the __getitem__/__setitem__/__delitem__ magic methods. To understand how indexing syntax is interpreted, and thus how these methods are called, we can simply define our own class:
class test:
    def __getitem__(self, idx):
        return idx

test()[:, [1,2,3], ...]
# result: (slice(None, None, None), [1, 2, 3], Ellipsis)

This slice type is a built-in type mentioned earlier in the same documentation, with a link to more specific information.
Ellipsis is simply the name of the object created by the literal syntax .... It's what Numpy uses to represent the complete extent of any number of axes - so this lets us avoid worrying about the dimensionality of the array we're working with.
The important thing to realize is that we can use that same index tuple directly, to get the same result.
Thus, given a dimension along which to slice with [ci], we can construct the necessary tuple, and index with it:
all = slice(None)
def make_index(ci, dimension):
    return (all,) * dimension + ([ci], Ellipsis)

# Now we can do things like:
arr[make_index(ci, dimension)] = some_data


Instruction: Does using an image transform significantly slow down training?

Input:
I see image transforms used quite often by many deep learning researchers. They seem to be treated as if they are free GPU or CPU cycles.
Example:
transformations = transforms.Compose([
        transforms.Resize(255),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    ])

train_set = datasets.ImageFolder(data_dir + "/train", transform = transformations)

In this specific case would it not be infinitely better to process the images upfront and save them out for future use in some other format? I see this sometimes but extremely rarely.
Or am I wrong, and transformers on a GPU are just so fast it's not worth the extra code or hassle?

Output: It really depends on how you set up the dataloader. Generally, the transforms are performed on the CPU, and then the transformed data is moved to the GPU. Pytorch dataloaders have a 'prefetch_factor' argument that allows them to pre-compute your data (with transforms) in parallel with the GPU computing the model. That being said, with fixed transforms like you have here, pre-computing the entire dataset and saving it prior to computing could also be a valid strategy.


Instruction: Cython_bbox and lap installation error, #include "Python.h" not found

Input:
I have encountered these strange errors upon trying to install these 2 libraries (Cython_bbox and lap), which are part of other libraries that I need when running pip install -r requirements.txt,
which contains the following
yacs
opencv-python
PyYAML
cython-bbox
scipy
progress
motmetrics
matplotlib
lap
openpyxl
Pillow
tensorboardX
fvcore

This is found from the following link at :https://github.com/ifzhang/FairMOT
This is the big bunch of errors that I got.
rc/cython_bbox.c:31:10: fatal error: Python.h: No such file or directory
     31 | #include "Python.h"
        |          ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for cython-bbox
  Running setup.py clean for cython-bbox
  Building wheel for lap (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/aevas/Desktop/pyenvs/fairmotpy39/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dweujmo4/lap_14d06a4a011f45a69d78f1a214bd5715/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dweujmo4/lap_14d06a4a011f45a69d78f1a214bd5715/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-pzh48d16
       cwd: /tmp/pip-install-dweujmo4/lap_14d06a4a011f45a69d78f1a214bd5715/
  Complete output (34 lines):
  Partial import of lap during the build process.
  Generating cython files
  running bdist_wheel
  running build
  running config_cc
  running config_fc
  running build_src

SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/lap
  copying lap/lapmod.py -> build/lib.linux-x86_64-3.9/lap
  copying lap/__init__.py -> build/lib.linux-x86_64-3.9/lap
  running build_ext
  creating /tmp/tmpijpimqcg/home
  creating /tmp/tmpijpimqcg/home/aevas
  creating /tmp/tmpijpimqcg/home/aevas/Desktop
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/distutils
  creating /tmp/tmpijpimqcg/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/distutils/checks
  CCompilerOpt.generate_dispatch_header[2281] : dispatch header dir build/src.linux-x86_64-3.9/numpy/distutils/include does not exist, creating it
  creating build/temp.linux-x86_64-3.9/lap
  lap/_lapjv.cpp:4:10: fatal error: Python.h: No such file or directory
      4 | #include "Python.h"
        |          ^~~~~~~~~~
  compilation terminated.
  error: Command "x86_64-linux-gnu-g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-FZ7wim/python3.9-3.9.5=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/core/include -Ilap -I/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/home/aevas/Desktop/pyenvs/fairmotpy39/include -I/usr/include/python3.9 -c lap/_lapjv.cpp -o build/temp.linux-x86_64-3.9/lap/_lapjv.o -MMD -MF build/temp.linux-x86_64-3.9/lap/_lapjv.o.d -msse -msse2 -msse3" failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for lap

  Running setup.py install for lap ... error
    ERROR: Command errored out with exit status 1:

 warnings.warn(
    running build
    running config_cc
    running config_fc
    running build_src
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/lap
    copying lap/lapmod.py -> build/lib.linux-x86_64-3.9/lap
    copying lap/__init__.py -> build/lib.linux-x86_64-3.9/lap
    running build_ext
    creating /tmp/tmplmbzif7h/home
    creating /tmp/tmplmbzif7h/home/aevas
    creating /tmp/tmplmbzif7h/home/aevas/Desktop
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/distutils
    creating /tmp/tmplmbzif7h/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/distutils/checks
    CCompilerOpt.generate_dispatch_header[2281] : dispatch header dir build/src.linux-x86_64-3.9/numpy/distutils/include does not exist, creating it
    creating build/temp.linux-x86_64-3.9/lap
    lap/_lapjv.cpp:4:10: fatal error: Python.h: No such file or directory
        4 | #include "Python.h"
          |          ^~~~~~~~~~
    compilation terminated.
    error: Command "x86_64-linux-gnu-g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-FZ7wim/python3.9-3.9.5=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/core/include -Ilap -I/home/aevas/Desktop/pyenvs/fairmotpy39/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/home/aevas/Desktop/pyenvs/fairmotpy39/include -I/usr/include/python3.9 -c lap/_lapjv.cpp -o build/temp.linux-x86_64-3.9/lap/_lapjv.o -MMD -MF build/temp.linux-x86_64-3.9/lap/_lapjv.o.d -msse -msse2 -msse3" failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1:

I was installing it in a brand new virtual environment running python3.9. I am not using any conda environment.
Here are my system specs, if required
11th Gen Intel® Core™ i7-11850H @ 2.50GHz × 16
llvmpipe (LLVM 12.0.0, 256 bits) / Mesa Intel® UHD Graphics (TGL GT1)
Ubuntu 21.04 64-bit

Things I've tried

looking through stackoverflow.com/search?q=%23include+%22Python.h%22+not+found

Output:
Try this :
sudo apt install libpython3.9-dev

which will install header files like Python.h


Instruction: PyTorch training loop within a sklearn pipeline

Input:
What I am playing around with right now is to work with PyTorch within a pipeline, where all of the preprocessing will be handled.
I am able to make it work. However, the results I am getting are a bit off. The loss function seems to be not decreasing and gets stuck (presumably in local optima?) as the training loop progresses.
I follow the standard PyTorch training loop and wrap it inside the fit method as this is what sklearn wants:
import torch
from sklearn.base import BaseEstimator, TransformerMixin

import torch.nn.functional as F

from IPython.core.debugger import set_trace

# +
import pandas as pd
import seaborn as sns
import numpy as np

from tqdm import tqdm
import random
# -

df = sns.load_dataset("tips")
df.head()

# +
class LinearRegressionModel(torch.nn.Module, BaseEstimator, TransformerMixin):

    def __init__(self, loss_func = torch.nn.MSELoss()):
        super(LinearRegressionModel, self).__init__()
        self.linear = torch.nn.Linear(3, 1)  # One in and one out
        self.loss_func = loss_func
        self.optimizer = torch.optim.SGD(self.parameters(), lr = 0.01)

    def forward(self, x):
        y_pred = F.relu(self.linear(x))
        return y_pred

    def fit(self, X, y):

#         set_trace()

        X = torch.from_numpy(X.astype(np.float32))
        y = torch.from_numpy(y.values.astype(np.float32))

        for epoch in tqdm(range(0, 12)):

            pred_y = self.forward(X)

            # Compute and print loss
            loss = self.loss_func(pred_y, X)

            # Zero gradients, perform a backward pass,
            # and update the weights.
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
            print('epoch {}, loss {}'.format(epoch, loss.item()))

# +
from sklearn.pipeline import Pipeline

from sklego.preprocessing import PatsyTransformer
# -

my_model = LinearRegressionModel()

pipe = Pipeline([
    ("patsy", PatsyTransformer("tip + size")),
    ("model", my_model)
])

pipe.fit(df, df['total_bill'])

It is not only due to the model being to simple. If I use sklearn linear regression estimated via stochastic gradient descent (SGDRegressor) the results seem nice. Therefore, I am concluding that problem is within my PyTorch class
# +
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

pipe2 = Pipeline([
    ("patsy", PatsyTransformer("tip + C(size) + C(time)")),
    ("model", LinearRegression())
])

pipe2.fit(df, df['total_bill'])
# -

mean_squared_error(df['total_bill'], pipe2.predict(df))

Output:
The problem in this implementation is in the fit method.
We are comparing prediction and design matrix
# Compute and print loss
loss = self.loss_func(pred_y, X)

Should be prediction and real value y:
loss = self.loss_func(pred_y, y)


Instruction: PyTorch - save only positive pictures where pedestrian are detected

Input:
I'm a bit stuck on some code.
I want to save ONLY POSITIVE IMAGE when one or more pedestrian are detected. When nothing is detected, do nothing.
I started by reading : https://github.com/ultralytics/yolov5/issues/36
I wrote this :
import torch
import os

f = []
for dirpath, subdirs, files in os.walk('MyFolderWithPictures'):
    for x in files:
        if x.endswith(".jpg"):
            f.append(os.path.join(dirpath, x))

model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

model.conf = 0.25  # NMS confidence threshold
model.iou = 0.45  # NMS IoU threshold
model.classes = 0   # Only pedestrian
model.multi_label = False  # NMS multiple labels per box
model.max_det = 1000  # maximum number of detections per image

img = f  # list of pictures

results = model(img)

results.print()
results.save()

But this print and save ALL images (positive and negative).
I want to save only images with pedestrian.
Can you help me ? Thanks in advance.
ps : the output give :
image 1/13: 1080x1920 1 person
image 2/13: 1080x1920 (no detections)
image 3/13: 1080x1920 (no detections)
image 4/13: 1080x1920 (no detections)
image 5/13: 1080x1920 (no detections)
image 6/13: 1080x1920 (no detections)
image 7/13: 1080x1920 (no detections)
image 8/13: 1080x1920 (no detections)
image 9/13: 1080x1920 (no detections)
image 10/13: 1080x1920 (no detections)
image 11/13: 1080x1920 (no detections)
image 12/13: 1080x1920 1 person
image 13/13: 1080x1920 (no detections)
Speed: 18.6ms pre-process, 119.8ms inference, 1.9ms NMS per image at shape (13, 3, 384, 640)
Saved 13 images to runs\detect\exp

ADD SOLUTION :
for item in f:
    # Images
    img = item  # or file, Path, PIL, OpenCV, numpy, list
    # Inference
    results = model(img)
    # Results
    results.print()  # or .show(), .save(), .crop(), .pandas(), etc.
    if 0 in results.pandas().xyxy[0]['class']:
        results.save()

Output:
model(img) will always return some kinds of results, even if there are no objects detected. What you need to do is inspect the results and see if it includes the class that you are interested in. The results can easily be converted to a Pandas Dataframe so that you can query them.
Here is an example to check if the results contain an instance of class 0 and then save the results if it does.
if 0 in results.pandas().xyxy[0]['class']:
  results.save()


Instruction: ERROR WHEN IMPORTING PYTORCH (The filename or extension is too long)

Input:
I'm using Anconda to run my Transformers project locally in google colab.
I've created a new environment (tf_gpu) and installed (supposedly) everything I need.
And everything works fine, but when I try to simply import pytorch, this error appears:
[WinError 206] The filename or extension is too long: 'C:\\Users\\34662\\anaconda3\\envs\\tf_gpu\\lib\\site-packages\\torch\\lib'

When clearly the path is not long enough to trigger this error.
My python version is 3.8, and my GPU is a Nvidia GeForce GTX 1650, so it shouldn't be a GPU problem
Does anybody knows why this happens?
Any help is good at this point, I don't know how to solve this.
Here I leave a screenshot of the complete error message
Thank you in advance.

Output: Your problem is that the error ist not a too long path error it is a file not found error which mean that pytorch is not correctly installed


Instruction: Cross entropy yields different results for vectors with identical distributions

Input:
I am training a neural network to distinguish between three classes. Naturally, I went for PyTorch's CrossEntropyLoss. During experimentation, I realized that the loss was significantly higher when a Softmax layer was put at the end of the model. So I decided to experiment further:
import torch
from torch import nn

pred_1 = torch.Tensor([[0.1, 0.2, 0.7]])
pred_2 = torch.Tensor([[1, 2, 7]])
pred_3 = torch.Tensor([[2, 4, 14]])
true = torch.Tensor([2]).long()

loss = nn.CrossEntropyLoss()

print(loss(pred_1, true))
print(loss(pred_2, true))
print(loss(pred_3, true))

The result of this code is as follows:
0.7679
0.0092
5.1497e-05

I also tried what happens when multiplying the input with some constant.

Several sources (1, 2) stated that the loss has a softmax built in, but if that were the case, I would have expected all of the examples above to return the same loss, which clearly isn't the case.
This poses the following question: if bigger outputs lead to a lower loss, wouldn't the network optimize towards outputting bigger values, thereby skewing the loss curves? If so, it seems like a Softmax layer would fix that. But since this results in a higher loss overall, how useful would the resulting loss actually be?

Output:
From the docs, the input to CrossEntropyLoss "is expected to contain raw, unnormalized scores for each class". Those are typically called logits.
There are two questions:

Scaling the logits should not yield the same cross-entropy. You might be thinking of a linear normalization, but the (implicit) softmax in the cross-entropy normalizes the exponential of the logits.
This causes the learning to optimize toward larger values of the logits. This is exactly what you want because it means that the network is more "confident" of the classification prediction. (The posterior p(c|x) is closer to the ground truth.)


Instruction: How do you know if a Pytorch Save contains a model and/or just the weights?

Input:
I'm fairly new to pytorch and this might be a version issue, but I see torch.load and torch.load_state_dict used, but in both cases the file extension is commonly ".pth"
Models that I have created, I can Save and Load them via torch.Save and torch.Load and call model.eval()
I have another model file that I'm fairly sure is just the state dictionary, as model.eval() fails after a load.
How would I inspect the file and know that one has a full model in it?
Thanks much.

Output:
As far as I know there isn't a foolproof way to figure this out. torch.save uses Python's pickle under the hood (ref: Pytorch docs), so users can save arbitrary Python objects. For example, the following code wraps the state dicts in a dictionary:
# example from https://github.com/lucidrains/lightweight-gan/blob/fce20938562a0cc289c915f7317722a8241abd37/lightweight_gan/lightweight_gan.py#L1437
save_data = {
    'GAN': self.GAN.state_dict(),
    'version': __version__,
    'G_scaler': self.G_scaler.state_dict(),
    'D_scaler': self.D_scaler.state_dict()
}
torch.save(save_data, self.model_name(num))

If it helps, state dicts themselves are OrderedDict objects. If isinstance(model, collections.OrderedDict) returns True, you can be fairly confident that model is a state dict. (Remember to import collections)
Models themselves are subclasses of torch.nn.Module, so you can check if something is a model by verifying that isinstance(model, torch.nn.Module) returns True.


Instruction: Unable to use Pytorch with CUDA in Celery task

Input:
I have a celery task that uses torch library, which internally uses CUDA. When I run the task, it fails saying
"Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method"
When I browsed on this a little, I got this - https://github.com/celery/celery/issues/6036
This issue says Celery supports only fork and not spawn.
Is there any workaround/alternative to this?

Output:
you should load pytorch model and use it as global variable. The instance of model and task will run in the same process
from celery.signals import worker_process_init
# for more information about worker_process_init, read here:
# https://docs.celeryproject.org/en/stable/userguide/signals.html#worker-process-init

pytorch_model = None

@worker_process_init.connect()
def init_worker_process(**kwargs):
    """
    load model before running tasks
    :param kwargs:
    :return:
    """
    global pytorch_model
    pytorch_model = load_model()

@app.task
def predict_task(image: np.ndarray):
    return pytorch_model.predict(image)


Instruction: how to use pretrained model .pth in pytorch?

Input: I have one question, how to use the saved pre-trained model. For example, I have a model and I train it with the dataset I have and then I save it with the .pth extension. how to use file .pth to test new image. thanks

Output:
You can load the parameters inside from a.pt/h into a model like this:
# initialize a model with the same architecture as the model which parameters you saved into the .pt/h file
model = Model()

# load the parameters into the model
model.load_state_dict(torch.load("parameters.pth"))


Instruction: Pytorch LSTM - generating sentence- word by word?

Input:
I'm trying to implement a neural network to generate sentences (image captions), and I'm using Pytorch's LSTM (nn.LSTM) for that.
The input I want to feed in the training is from size batch_size * seq_size * embedding_size, such that seq_size is the maximal size of a sentence. For example - 64*30*512.
After the LSTM there is one FC layer (nn.Linear).
As far as I understand, this type of networks work with hidden state (h,c in this case), and predict the next word each time.
My question is- in the training - do we have to manually feed the sentence word by word to the LSTM in the forward function, or the LSTM knows how to do it itself?
My forward function looks like this:
    def forward(self, features, caption, h = None, c = None):
        batch_size = caption.size(0)
        caption_size = caption.size(1)

        no_hc = False
        if h == None and c == None:
            no_hc = True
            h,c = self.init_hidden(batch_size)

        embeddings = self.embedding(caption)
        output = torch.empty((batch_size, caption_size, self.vocab_size)).to(device)

        for i in range(caption_size): #go over the words in the sentence
            if i==0:
                lstm_input = features.unsqueeze(1)
            else:
                lstm_input = embeddings[:,i-1,:].unsqueeze(1)

            out, (h,c) = self.lstm(lstm_input, (h,c))
            out = self.fc(out)

            output[:,i,:] = out.squeeze()

        if no_hc:
            return output

        return output, h,c

(took inspiration from here)
The output of the forward here is from size batch_size * seq_size * vocab_size, which is good because it can be compared with the original batch_size * seq_size sized caption in the loss function.
The question is whether this for loop inside the forward that feeds the words one after the other is really necessary, or I can somehow feed the entire sentence at once and get the same results?
(I saw some example that do that, for example this one, but I'm not sure if it's really equivalent)

Output:
The answer is, LSTM knows how to do it on its own. You do not have to manually feed each word one by one.
An intuitive way to understand is that the shape of the batch that you send, contains seq_length (batch.shape[1]), using which it decides the number of words in the sentence. The words are passed through LSTM Cell generating the hidden states and C.


Instruction: big data in pytorch, help for tuning steps

Input:
I've previously splitted my bigdata:
# X_train.shape : 4M samples x 2K features
# X_test.shape : 2M samples x 2K features

I've prepared the dataloaders
target = torch.tensor(y_train.to_numpy())
features = torch.tensor(X_train.values)
train = data_utils.TensorDataset(features, target)
train_loader = data_utils.DataLoader(train, batch_size=10000, shuffle=True)

testtarget = torch.tensor(y_test.to_numpy())
testfeatures = torch.tensor(X_test.values)
test = data_utils.TensorDataset(testfeatures, testtarget)
validation_generator = data_utils.DataLoader(test, batch_size=20000, shuffle=True)

I copied from an online course this example for a network (no idea if other model are better)
base_elastic_model = ElasticNet()
param_grid = {'alpha':[0.1,1,5,10,50,100],
              'l1_ratio':[.1, .5, .7, .9, .95, .99, 1]}
grid_model = GridSearchCV(estimator=base_elastic_model,
                          param_grid=param_grid,
                          scoring='neg_mean_squared_error',
                          cv=5,
                          verbose=0)

I've built this fitting
for epoch in range(1):
    # Training
    cont=0
    total = 0
    correct = 0
    for local_batch, local_labels in train_loader:
        cont+=1
        with torch.set_grad_enabled(True):
            grid_model.fit(local_batch,local_labels)
        with torch.set_grad_enabled(False):
            predicted = grid_model.predict(local_batch)
            total += len(local_labels)
            correct += ((1*(predicted>.5)) == np.array(local_labels)).sum()
        #print stats

    # Validation
    total = 0
    correct = 0

    with torch.set_grad_enabled(False):
        for local_batch, local_labels in validation_generator:
            predicted = grid_model.predict(local_batch)
            total += len(local_labels)
            correct += ((1*(predicted>.5)) == np.array(local_labels)).sum()
            #print stats

Maybe my grandchildren will have the results for 1 epoch!
I need some advises:

how/where (in the code) can I use quickly less data for a first tuning?
some advise for the steps to have a result in the 2022?
because I've added "with torch.set_grad_enabled(False):" for stats printing, have I to add (as  done) "with torch.set_grad_enabled(True):" ?
I have got a GPU (useful without images??). I've the function "get_device()". Where have I to put ".to(get_device())" to use CUDA?
I'm learning putting together pieces of information, do you have general advising for my exercise?

Output:
To shorten the training process by simply stopping the training for loop after a certain number like so.
for local_batch, local_labels in train_loader:

   cont+=1
   if cont== number_u_want_to_stop:
      break #Breaks out of the for Loop and continues with the rest.

Always use your GPU for training and "inferencing" aka (using a model to make predictions) bs it is more than 20 faster than even the best CPU.

No you don't have to make it true again. That's the main point of using the  "with" syntax so after the code that is in the with the block is finished the properties will just dissolve into air :). So u can delete this line with a torch.set_grad_enabled(False):

Like I said in the 2nd point use your GPU for all your projects but keep in mind u will have to use a graphics card with at least 4GB to train even little models.
here the install cmd for using the GPU on windows:
pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113  torchaudio===0.10.1+cu113 -f  https://download.pytorch.org/whl/cu113/torch_stable.html
and here is the one for Linux
pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

and here is a link  to the PyTorch doc that explains to you how to use the GPU in PyTorch

A very nice starter project that probably everyone has done when he started with machine learning especially those who want to use computer vision. The Implementation of the image classification using the MNIST Dataset. There are many great tutorials out there. So at first, it will be very overwhelming with all those new words but I will promise it will get better when you start to speak the same language as the guys writing those tutorials. So first follow the tutorial and if u don't understand any word just google it by itself and work through it in little pieces bc otherwise, it will be very hard to comprehend. After u gained some basic knowledge u can start to build your own little projects. Start with something little. So keep grinding :)


Instruction: Attention does not seem to be applied at TransformerEncoderLayer and MultiheadAttention PyTorch

Input:
Changing something at one position in my input does not affect the outputs at other positions of my transformer encoder. I made a test in isolation in PyTorch:
# My encoder layer
encoder_layer = nn.TransformerEncoderLayer(d_model=8, nhead=2)
# Turn off dropout
encoder_layer.eval()
# Random input
src = torch.rand(2, 10, 8)
# Predict the output
out_0 = encoder_layer(src)
# Change the values at one of the positions (position 3 in this case)
src[:,3,:] += 1
# Predict once again the output
out_1 = encoder_layer(src)
# Check at which positions the outcomes are different between the two cases
# I summed in the embedding space direction
print(np.sum(np.abs(out_0.detach().numpy()),axis=-1) - np.sum(np.abs(out_1.detach().numpy()),axis=-1))

Output:
[[ 0. 0.  0.  -0.15470695  0.   0. 0.  0.     0.  0.   ]
 [ 0.   0.  0.  -0.27988768  0.  0.0.   0.   0.   0.   ]]

However, this does work in TensorFlow:
# My encoder layer
encoder_layer = TransformerBlock(8, 2, 8)
# Random input
src = np.random.randn(2, 10, 8)
# Predict the output
out_0 = encoder_layer(src, training=False)
# Change the values at one of the positions (position 3 in this case)
src[:,3,:] += 1
# Predict once again the output
out_1 = encoder_layer(src, training=False)
# Check at which positions the outcomes are different between the two cases
# I summed in the embedding space direction
print(np.sum(np.abs(out_0),axis=-1) )

Output:
[[6.4196725 6.775745  6.946576  7.26213   6.473065  5.520765  6.201167
  7.1266503 6.3147016 6.614853 ]
 [5.565378  7.030789  6.768366  6.6065626 6.7277775 7.480627  6.6785836
  6.4560523 6.4248576 6.6436586]]

My question is: Why aren't the values at all the position affected by changing the input at one input in PyTorch?

Output:
From the documentation:

batch_first – If True, then the input and output tensors are provided as (batch, seq, feature). Default: False.

In other words, your input are 10 8-dimensional batches of sequence length 2 each. What you are doing is add 1 to all dimensions of all inputs of sample #4 in the batch, which --unsurprisingly-- alters only the output values of that specific sample.


Instruction: PyTorch how to set the minimum value along each row of a tensor to zero?

Input:
We don't know the shape of the input tensor and we shouldn't use any loops. Only reduction and indexing operations. How do we set the minimum value of each row to zero?
For example:
input:
x = torch.tensor([[
      [10, 20, 30]
      [2, 5, 1]
    ]])

output:
torch.tensor([
  [0, 20, 30],
  [2, 5, 0]
])

I couldn't figure it out and couldn't find any related questions. I'm stuck.

Output:
One way to do so is to compute row-wise minimum x.min(dim=-1), get minimal values x.min(dim=-1).values (indices won't work in case of multiple minimal elements), get mask indicating locations of non-minimal elements using comparison and multiply by it:
axis = -1  # Minimum iterating over the last dimension
min_values = x.min(dim=axis).values  # Get minimal values
min_values_shape_corrected = min_values.unsqueeze(axis)  # Reshape the minimal values so we can compare it with `x`
mask = (x != min_values_shape_corrected)  # Get the mask of non-minimal elements
result = x * mask  # Multiplying by a boolean mask leaves only True elements and sets False ones to 0

Or in one line
x * (x != x.min(dim=axis).values.unsqueeze(axis))


Instruction: SimpleTransformers "max_seq_length" argument results in CUDA out of memory error in Kaggle and Google Colab

Input:
When fine-tuning the sloBERTa Transformer model, based on CamemBERT, for a multiclass classification task with SimpleTransformers, I want to use the model argument "max_seq_length": 512, as previous work states that it gives better results than 128, but the inclusion of this argument triggers the error below. The error is the same in Kaggle and Google Colab environment, and terminating the execution and reruning it does not help. The error is triggered not matter how small the number of training epochs is, and the dataset contains only 600 instances (with text as strings, and labels as integers). I've tried lowering the max_seq_length to 509, 500 and 128, but the error persists.
The setup without this argument works normally and allows training with 90 epochs, so I otherwise have enough memory.
from simpletransformers.classification import ClassificationModel

# define hyperparameter
model_args ={"overwrite_output_dir": True,
             "num_train_epochs": 90,
             "labels_list": LABELS_NUM,
             "learning_rate": 1e-5,
             "train_batch_size": 32,
             "no_cache": True,
             "no_save": True,
             #"max_seq_length": 512,
             "save_steps": -1,
             }

model = ClassificationModel(
    "camembert", "EMBEDDIA/sloberta",
    use_cuda = device,
    num_labels = NUM_LABELS,
    args = model_args)

model.train_model(train_df)

This is the error:
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_34/2529369927.py in <module>
    19     args = model_args)
    20
---> 21 model.train_model(train_df)

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs)
   610             eval_df=eval_df,
   611             verbose=verbose,
--> 612             **kwargs,
   613         )
   614

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py in train(self, train_dataloader, output_dir, multi_label, show_running_loss, eval_df, test_df, verbose, **kwargs)
   883                             loss_fct=self.loss_fct,
   884                             num_labels=self.num_labels,
--> 885                             args=self.args,
   886                         )
   887                 else:

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py in _calculate_loss(self, model, inputs, loss_fct, num_labels, args)
  2256
  2257     def _calculate_loss(self, model, inputs, loss_fct, num_labels, args):
-> 2258         outputs = model(**inputs)
  2259         # model outputs are always tuple in pytorch-transformers (see doc)
  2260         loss = outputs[0]

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
  1210             output_attentions=output_attentions,
  1211             output_hidden_states=output_hidden_states,
-> 1212             return_dict=return_dict,
  1213         )
  1214         sequence_output = outputs[0]

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   859             output_attentions=output_attentions,
   860             output_hidden_states=output_hidden_states,
--> 861             return_dict=return_dict,
   862         )
   863         sequence_output = encoder_outputs[0]

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   531                     encoder_attention_mask,
   532                     past_key_value,
--> 533                     output_attentions,
   534                 )
   535

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)
   415             head_mask,
   416             output_attentions=output_attentions,
--> 417             past_key_value=self_attn_past_key_value,
   418         )
   419         attention_output = self_attention_outputs[0]

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)
   344             encoder_attention_mask,
   345             past_key_value,
--> 346             output_attentions,
   347         )
   348         attention_output = self.output(self_outputs[0], hidden_states)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   720             result = self._slow_forward(*input, **kwargs)
   721         else:
--> 722             result = self.forward(*input, **kwargs)
   723         for hook in itertools.chain(
   724                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/transformers/models/roberta/modeling_roberta.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)
   273             attention_probs = attention_probs * head_mask
   274
--> 275         context_layer = torch.matmul(attention_probs, value_layer)
   276
   277         context_layer = context_layer.permute(0, 2, 1, 3).contiguous()

RuntimeError: CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 15.90 GiB total capacity; 15.04 GiB already allocated; 15.75 MiB free; 15.12 GiB reserved in total by PyTorch)

Additional code (if it helps - I've tried everything regarding the pytorch I found on the web - the full code can be accessed at https://www.kaggle.com/tajakuz/0-sloberta-example-max-seq-length-error):
!conda install --yes pytorch>=1.6 cudatoolkit=11.0 -c pytorch

# install simpletransformers
!pip install -q transformers
!pip install --upgrade transformers
!pip install -q simpletransformers

# check installed version
!pip freeze | grep simpletransformers

!pip uninstall -q torch -y
!pip install -q torch==1.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html

# pytorch libraries
import torch # the main pytorch library
import torch.nn as nn # the sub-library containing Softmax, Module and other useful functions
import torch.optim as optim # the sub-library containing the common optimizers (SGD, Adam, etc.)
from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler

from torch import cuda
device = 'cuda' if cuda.is_available() else 'cpu'

#importing other necessary packages and ClassificationModel for bert
from tqdm import tqdm
import warnings
warnings.simplefilter('ignore')

from scipy.special import softmax

Thank you so much for your help, it is really appreciated!

Output:
This happened because max_seq_length defines the number of input neurons for the model thus increasing the number of trainable parameters which will require it to allocate more memory which might exceed your memory limits on those platforms.
Most of the time, max_seq_length is up the dataset, and sometimes adding too much could be wasteful in terms of training time and model size.
What you can do is to find the max number of words per sample in your training dataset and use that as your max_seq_length.


Instruction: Neural net loss exponentially rises after first propogation

Input:
I am training a neural network on video frames (converted to greyscale) to output a tensor with two values. The first iteration always evaluates an acceptable loss (mean squared error generally between 15-40), followed by an exponential rise in the second pass, and then infinite.
The net is quite vanilla:
class NeuralNetwork(nn.Module):

    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(100 * 291, 29100),
            nn.ReLU(),
            nn.Linear(29100, 29100),
            nn.ReLU(),
            nn.Linear(29100, 2),
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

As is the training loop:
def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to("cpu"), y.to("cpu")

        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropogation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

Example of loss function growth:
ITERATION 1
prediction: tensor([[-1.2239, -8.2337]], grad_fn=<AddmmBackward>)
actual:     tensor([[0.0321, 0.0325]])
loss:       tensor(34.9545, grad_fn=<MseLossBackward>)

ITERATION 2
prediction: tensor([[ 314636.5625, 2063098.2500]], grad_fn=<AddmmBackward>)
actual:     tensor([[0.0330, 0.0323]])
loss:       tensor(2.1777e+12, grad_fn=<MseLossBackward>)

ITERATION 3
prediction: tensor([[-8.0924e+22, -5.3062e+23]], grad_fn=<AddmmBackward>)
actual:     tensor([[0.0334, 0.0317]])
loss:       tensor(inf, grad_fn=<MseLossBackward>)

Here is an example of the video data: it's a 291x100 greyscale image and there are 1100 of them in the training dataset:
dataset.video_frames.size()
> torch.Size([1100, 100, 291])

dataset.video_frames[0]
> tensor([[21., 29., 28.,  ..., 33., 27., 26.],
        [22., 27., 25.,  ..., 25., 25., 30.],
        [23., 26., 26.,  ..., 24., 24., 28.],
        ...,
        [24., 33., 31.,  ..., 41., 40., 42.],
        [26., 34., 31.,  ..., 26., 20., 22.],
        [25., 32., 32.,  ..., 21., 20., 18.]])

And the labeled training data:
dataset.y.size()
> torch.Size([1100, 2])

dataset.y[0]
> tensor([0.0335, 0.0315], dtype=torch.float)

I've fiddled the learning rate, number of hidden layers, and nothing seems to keep the loss from going to infinite.

Output:
Properly scaling the inputs is crucial for proper training.
Weights are initialized based on some assumptions on the way inputs are scaled.
See this part of a lecture on weight initialization and see how critical it is for proper convergence.
More details on the mathematical analysis of the influence of weight initialization can be found in Sec. 2 of this paper:
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (ICCV 2015).


Instruction: Should we actively use the weight argument in loss functions

Input:
Most of the current machine learning libraries have loss functions that comes with a weight argument, which allows us to tackle unbalanced datasets. However should this feature be actively made use of? If not, are there certain guidelines as to when we should use it(e.g. if the dataset is skewed to a certain extent). Will the model eventually learn to predict the rare cases anyway if it is complex(for the lack of a better word, I understand complexity doesn't equate to performance) enough?
I had this question because I was training a model with an unbalanced dataset(but not to the extreme), however I am adjusting the weights in the loss function somewhat arbitrarily according to the proportion of each class present in the dataset.

Output:
You can use the weighted version of loss functions if you are certain the real world data your model will need to generalize for is similarly imbalanced. If not, you are introducing man-made bias into the system.
The choice to use weight cannot be based solely on model performance during training, validation or testing, but has to be made based on close scrutiny of the data set and how it was built.
A clear example where it may help is tumor detection in CT scans, where background and foreground often have a ratio of 20:1.


Instruction: Some weights of Actor Critic model not updating

Input:
I am working on an Actor-Critic model in Pytorch. The model first receives the input in an RNN and then the policy net comes into play. The code for Policy net is:
class Policy(nn.Module):
    """
    implements both actor and critic in one model
    """
    def __init__(self):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(state_size+1, 128)

        self.fc2 = nn.Linear(128, 64)

        # actor's layer
        self.action_head = nn.Linear(64, action_size)
        self.mu = nn.Sigmoid()
        self.var = nn.Softplus()

        # critic's layer
        self.value_head = nn.Linear(64, 1)

    def forward(self, x):
        """
        forward of both actor and critic
        """
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))

        # actor: choses action to take from state s_t
        # by returning probability of each action
        action_prob = self.action_head(x)
        mu = self.mu(action_prob)
        var = self.var(action_prob)

        # critic: evaluates being in the state s_t
        state_values = self.value_head(x)

        return mu, var, state_values
policy = Policy()

In model class, we are calling this policy after the rnn. And in agent class’s act method, we are calling the model to get the action like this:
 def act(self, some_input, state):
      mu, var, state_value = self.model(some_input, state)
      mu = mu.data.cpu().numpy()
      sigma = torch.sqrt(var).data.cpu().numpy()
      action = np.random.normal(mu, sigma)
      action = np.clip(action, 0, 1)
      action = torch.from_numpy(action/1000)
      return action, state_value

I must mention that in optimizer, we are calling the model.parameters. When we print all the trainable parameters in each epoch, we see that everything else is changing except for the policy.action_head. Any idea why this is happening? I must also mention how the losses are calculated now:
       advantage = reward - Value
       Lp = -math.log(pdf_prob_now)*advantage
       policy_losses.append(Lp)
       #similar for value_losses
#after all the runs in the epoch is done
loss = torch.stack(policy_losses).sum() + alpha*torch.stack(value_losses).sum()
loss.backward()

Here Value is the state_value (the 2nd output from agent.act) and the pdf_prob_now is the probability of the action from all possible actions which is calculated like this:
def find_pdf(policy, action, rnn_output):
    mu, var, _ = policy(rnn_output)
    mu = mu.data.cpu().numpy()
    sigma = torch.sqrt(var).data.cpu().numpy()
    pdf_probability = stats.norm.pdf(action.cpu(), loc=mu, scale=sigma)
    return pdf_probability

Is there some logical error here?

Output:
the bug is in act function
def act(self, some_input, state):
    # mu contains info required for gradient
    mu, var, state_value = self.model(some_input, state)
    # mu is detached and now has forgot all the operations performed
    # in self.action_head
    mu = mu.data.cpu().numpy()
    sigma = torch.sqrt(var).data.cpu().numpy()
    action = np.random.normal(mu, sigma)
    action = np.clip(action, 0, 1)
    action = torch.from_numpy(action/1000)
    return action, state_value

for the further process, if loss is calculated using tensor operations performed on action, it can not be traced back to update self.action_head weights, as you detached the tensor mu which removes it from the computation graph and so you do not see any updates in self.action_head.


Instruction: How to use nn.TransTransformerEncoder from pytorch

Input:
I am trying to use PyTorch's '''nn.TransformerEncoder''' module for a classification task.
I have data points of varying lengths i.e I have sequences of different lengths. All sequences have one corresponding output(target which is either 0 or 1).[![enter code here][1]][1]
This image outlines my dataset
This image shows how the sequences vary in length

However the entries of each sequence are of the same length.
I want to use this dataset to train an encoder part of the Transformer to be able to predict the corresponding outputs.
How can I go about doing this? And are there any examples I can check online?

Output: It depends on how your data actually looks like and what kind of output you expect. In general I would suggest to you to use the Transformers Library from HuggingFace, they have a lot of documentation and detailed code examples that you can work on -- plus an active forum. Here is a link to their description of Encoder-Decoder Models. I hope that helps you a bit.


Instruction: How can I do calculations on tensors that have "requires_grad = true"?

Input:
I have this program you see below.
import torch

def dht_calculate_transformation_of_single_joint(para_dht_parameters):
    var_a = para_dht_parameters[0]
    var_d = para_dht_parameters[1]
    var_alpha = para_dht_parameters[2]
    var_theta = para_dht_parameters[3]

    var_transformation = torch.tensor(data=[
        [torch.cos(var_theta), -1 * torch.sin(var_theta) * torch.cos(var_alpha), torch.sin(var_theta) * torch.sin(var_alpha), var_a * torch.cos(var_theta)],
        [torch.sin(var_theta), torch.cos(var_theta) * torch.cos(var_alpha), -1 * torch.cos(var_theta) * torch.sin(var_alpha), var_a * torch.sin(var_theta)],
        [0, torch.sin(var_alpha), torch.cos(var_alpha), var_d],
        [0, 0, 0, 1]
    ], dtype=torch.float32, requires_grad=True)

    return var_transformation

def dht_calculate_positions_of_all_joints(para_all_transformations_of_joints):
    var_all_positions_of_joints = torch.zeros(size=[27], dtype=torch.float32, requires_grad=True)
    var_index_all_positions_of_joints = 0
    var_transformation_to_joint = torch.zeros(size=[4, 4], dtype=torch.float32, requires_grad=True)

    for var_index_of_transformation_of_joint, var_transformation_of_joint in enumerate(para_all_transformations_of_joints):
        if var_index_of_transformation_of_joint == 0:
            var_transformation_to_joint = var_transformation_of_joint
        else:
            var_transformation_to_joint = torch.matmul(var_transformation_to_joint, var_transformation_of_joint)

        var_all_positions_of_joints[var_index_all_positions_of_joints + 0] = var_transformation_to_joint[0][3]
        var_all_positions_of_joints[var_index_all_positions_of_joints + 1] = var_transformation_to_joint[1][3]
        var_all_positions_of_joints[var_index_all_positions_of_joints + 2] = var_transformation_to_joint[2][3]
        var_index_all_positions_of_joints += 3

    return var_all_positions_of_joints

def dht_complete_calculation(para_input):
    var_input_reshaped = para_input.view(-1, 9, 4)
    var_output = torch.zeros(size=[para_input.shape[0], 27], dtype=torch.float32, requires_grad=True)  # Tensor ist x Zeilen (Datenreihen) * 27 Spalten (Positionen von Joints) groß.

    for var_index_of_current_row, var_current_row in enumerate(var_input_reshaped):
        var_all_transformations_of_joints = torch.zeros(size=[9, 4, 4], dtype=torch.float32, requires_grad=True)
        for var_index_of_current_column, var_current_column in enumerate(var_current_row):
            var_all_transformations_of_joints[var_index_of_current_column] = dht_calculate_transformation_of_single_joint(var_current_column)

        var_output[var_index_of_current_row] = dht_calculate_positions_of_all_joints(var_all_transformations_of_joints)

    return var_output

if __name__ == "__main__":
    inp = torch.tensor(data=
        [
            [5.1016, 5.2750, 5.0043, 5.2184,
             4.8471, 5.3377, 5.0113, 5.0789,
             4.8800, 5.0455, 5.0394, 4.9092,
             4.6609, 5.5003, 5.1327, 4.7121,
             4.9442, 5.0918, 4.8083, 4.3548,
             5.0163, 4.8840, 4.7491, 4.8089,
             4.8919, 5.0975, 4.9931, 5.0999,
             4.6400, 5.0069, 4.7420, 5.3347,
             4.6725, 5.0338, 5.0310, 5.0470],
            [4.9628, 5.0113, 5.0834, 4.7143,
             5.0336, 5.1864, 5.4348, 5.0918,
             5.1570, 4.8881, 4.5411, 4.6745,
             4.6072, 4.9938, 4.9655, 5.2279,
             5.5559, 5.1952, 5.2229, 5.0727,
             5.1382, 4.7613, 4.6449, 4.3832,
             5.1866, 5.6650, 4.9886, 4.8088,
             4.9390, 5.3506, 5.1028, 4.4640,
             5.1076, 5.0772, 4.8219, 5.1303]
        ]
    , requires_grad=True)

    t1 = dht_complete_calculation(inp)
    print("Endergebins \n", t1, t1.shape)

I get the following message when I execute the main:
Traceback (most recent call last):
  File "dht.py", line 77, in <module>
    t1 = dht_complete_calculation(inp)
  File "dht.py", line 46, in dht_complete_calculation
    var_all_transformations_of_joints[var_index_of_current_column] = dht_calculate_transformation_of_single_joint(var_current_column)
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

The thing is that the "dht_complete_calculation" function will be used with a neural Network (which isn't in the code fragment and isn't relevant to the question). The Output of the neural Network will be inputted in the "dht_complete_calculation" function. That is why the outputtensor and every tensor used in the calculation need to have "requires_grad = true".
The "dht_complete_calculation" function gets a tensor with x rows and 36 columns as input and should output a tensor with x rows and 27 columns. The calculations you see there are correct, because if I remove "requires_grad = true" from every tensor it works.
This is the desired output:
tensor([[ 2.4727e+00, -4.4623e+00,  5.2750e+00,  6.6468e+00, -4.1351e+00,
          1.1145e+01,  1.3516e+01, -4.3618e+00,  1.2571e+01,  1.7557e+01,
         -1.0147e+01,  1.4048e+01,  1.8344e+01, -1.2500e+01,  2.0697e+01,
          2.4276e+01, -1.4575e+01,  2.3784e+01,  2.6110e+01, -2.0825e+01,
          2.6521e+01,  2.6707e+01, -2.4291e+01,  3.2371e+01,  3.1856e+01,
         -2.4376e+01,  3.6915e+01],
        [ 9.4848e-03, -4.9628e+00,  5.0113e+00,  3.1514e+00, -6.8211e+00,
          1.1249e+01,  9.8675e+00, -6.9772e+00,  1.3564e+01,  1.1752e+01,
         -9.6508e+00,  1.9519e+01,  1.1553e+01, -8.3219e+00,  2.7006e+01,
          1.4205e+01, -2.2681e+00,  2.9327e+01,  1.6872e+01, -2.0226e+00,
          3.6526e+01,  1.2353e+01, -5.7472e-01,  4.2049e+01,  1.0814e+01,
          3.8157e+00,  4.7547e+01]]) torch.Size([2, 27])

Process finished with exit code 0

However with "requires_grad = true" removed the network wouldn't learn anything, which is not what I want.
Can you help me to understand which part of the code triggers this error and how to fix it?

Output:
The problem here is not that you are doing computations on a requires_grad=True tensor. After all this is how one gets gradients! By doing computations on such tensors :)
The issue is that you are doing what are called in-place operations.
By in-place we mean that a previously existing variable's memory location is now replaced with a new variable. As a result the computational graph is broken, and thus, no gradient backpropagation can be achieved.
How does this look like? I found a few quick examples in this Pytorch-forum question
In particular:
>>> x = torch.rand(1)
>>> y = torch.rand(1)
>>> x
tensor([0.2738])
>>> id(x)
140736259305336
>>> x = x + y   # Normal operation
>>> id(x)
140726604827672 # New location
>>> x += y
>>> id(x)
140726604827672 # Existing location used (in-place)

So, you might then ask, where do you do that?
One such place is
var_all_positions_of_joints[var_index_all_positions_of_joints + 0] = var_transformation_to_joint[0][3]
    var_all_positions_of_joints[var_index_all_positions_of_joints + 1] = var_transformation_to_joint[1][3]
    var_all_positions_of_joints[var_index_all_positions_of_joints + 2] = var_transformation_to_joint[2][3]

Instead of doing that, you should instead collect all var_transofrmation_to_joint variables in a list and then do torch.stack or torch.cat depending on your situation. Alternatively, if in the future you seek to re-arrange the location of elements in a tensor, I recommend using something like einops for a highly-efficient and framework-independent solution.