Instruction: How to train Pytorch model on custom data

Input:
I am very rookie in transferring my code from Keras/Tensorflow to Pytorch and I am trying to retrain my TF model in Pytorch, however, my dataset has some particularities which make it difficult to me to make it run in Pytorch.
To understand my issues, recall that I have a custom dataset initialized this way:
class MyDataSet(torch.utils.data.Dataset):
    def __init__(self, x, y, transform=None):
        super(MyDataSet, self).__init__()
        # store the raw tensors
        self._x = np.load(x)
        self._y = np.load(y)

        self._x=np.swapaxes(self._x,3,2)
        self._x=np.swapaxes(self._x,2,1)
        self.transform = transform

    def __len__(self):
        # a DataSet must know it size
        return self._x.shape[0]

    def __getitem__(self, index):
        x = self._x[index, :]
        y = self._y[index]

        return x, y

The shape of _self._x is (12000, 3, 224, 224) and the shape of self._y is (12000,)
I am fine-tuning a pre-trained RESNET-50 in this data, and the training happens the following way:
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import numpy as np
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
import time
import copy

def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

#Transform dataset
print("Loading Data")
transform = transforms.Compose([transforms.ToTensor()])
dataset = MyDataSet("me/train1-features.npy","/me/train1-classes.npy",transform=transform)
dataloader = DataLoader(dataset, batch_size=4)

print("Configuring network")
feature_extract = True
num_epochs = 15
num_classes=12

model_ft = resnet50(pretrained=True)
set_parameter_requires_grad(model_ft, feature_extract)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, num_classes)

if torch.cuda.is_available():
    model_ft.cuda()

params_to_update = model_ft.parameters()

print("Params to learn:")
if feature_extract:
    params_to_update = []
    for name,param in model_ft.named_parameters():
        if param.requires_grad == True:
            params_to_update.append(param)
            print("\t",name)

else:
    for name,param in model_ft.named_parameters():
        if param.requires_grad == True:
            print("\t",name)

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9)

# Setup the loss fxn
criterion = nn.CrossEntropyLoss()

#Train (how to validate?)

for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(dataloader, 0):

        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        #transfer labels and inputs to cuda()
        inputs,labels=inputs.cuda(), labels.cuda()

        # zero the parameter gradients
        optimizer_ft.zero_grad()

        # forward + backward + optimize
        outputs = model_ft(inputs)
        loss = loss_func(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

However, whenever I run this code, I receive the following error
    Traceback (most recent call last):
  File "train_my_data_example.py", line 114, in <module>
    outputs = model_ft(inputs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/models/resnet.py", line 249, in forward
    return self._forward_impl(x)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/models/resnet.py", line 232, in _forward_impl
    x = self.conv1(x)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

I also can do the train and validation procedures normally on TF/Keras, but I don't know how to do that in my custom Dataset with Pytorch.
How can I solve my problem and also run train/val loop with Pytorch in my custom data?

Output:
It seems that np.load is loading binary data to X so ToTensor() is trying to preserve the dtype by coercing it to a ByteTensor. You can fix this by making a small change in __getitem__:
def __getitem__(self, index):
    x = self._x[index, :]
    y = self._y[index]

    return x.astype(np.float32), y


Instruction: Can I debug a python package after installing it?

Input:
I'm interested in a package and wanted to play around with the code: https://github.com/aiqm/torchani
The package itself is not complex and the key modules are included in the torchani folder. I wanted to use the VSCode debugger to do some experiments with the components and track the code. Do I need to run python setup.py --install, or I should simply go to the folder and run the modules without installing?
The problem is: there will be a lot of relative import issues if I directly run the code in the parent folder. If I install the package, then the code will probably be compiled and my changes will not be executed.

Output:
You can install the package with python setup.py install (or pip install [-e] .).
For the debugging part you can use the debugger of VSCode, just set justMyCode: False in the launch.json.


Instruction: torch.nn.functional.interpolate: difference between "linear" and "bilinear"?

Input:
In torch.nn.functional.interpolate what's the difference between the modes linear and bilinear?
To me, these are usually synonyms with regards to image resizing...

Output:
Pytorch is explicitly differentiating between 1d interpolation (linear) and 2d interpolation (bilinear).
They differ in the the dimensionality of the input argument they are allowed to work on ( see here ). Specifically, linear works on 3D inputs and bilinear works on 4D inputs because the first two dimensions (mini-batch x channels) are understood not to be interpolated.


Instruction: Is there any way to change parameters in pretrained DetrFeatureExtractor?

Input:
For my model, I load pretrained version of DetrFeatureExtractor:
feature_extractor = DetrFeatureExtractor(return_tensors="pt"
                                        ,do_normalize = True
                                        ,size = 400).from_pretrained("facebook/detr-resnet-50")

But when I output parameters of this variable, I get:
DetrFeatureExtractor {
  "do_normalize": true,
  "do_resize": true,
  "feature_extractor_type": "DetrFeatureExtractor",
  "format": "coco_detection",
  "image_mean": [
    0.485,
    0.456,
    0.406
  ],
  "image_std": [
    0.229,
    0.224,
    0.225
  ],
  "max_size": 1333,
  "size": 800
}

which still has size = 800. Is that possible to change parameters of pretrained feature extractor and, if yes, how can I change them?

Output:
Your values to the constructor are not taken, because you are calling .from_pretrained(), which loads all values from the respective config file of the pretrained model (in your case, the corresponding config file can be viewed here).
Even if some values might not be specified in the config, they will be primarily taken from the default values, instead of whatever you're passing before.
If you want to change attributes, you can do so after loading:
from transformers import DetrFeatureExtractor

model = DetrFeatureExtractor.from_pretrained("facebook/detr-resnet-50")
model.size = 400
print(model)  # will show the correct size

I do want to point out that changing parameters of pretrained networks might lead to unexpected behavior - after all, the model wasn't originally trained with this particular setting. So just be aware that this might lead to detrimental performance, and experiment at your own risk ;-)


Instruction: What is the official implementation of first order MAML using the higher PyTorch library?

Input:
After noticing that my custom implementation of first order MAML might be wrong I decided to google how the official way to do first order MAML is. I found a useful gitissue that suggests to stop tracking the higher order gradients. Which makes complete sense to me. No more derivatives over the derivatives. But when I tried setting it to false (so that no higher derivatives are tracked) I got that there was no more training of my models and the .grad fiedl was None. Which is obviously wrong.
Is this a bug in higher or what is going on?

To reproduce run the official MAML example higher has but slightly modified here. The main code is this though:
#!/usr/bin/env python3
#
# Copyright (c) Facebook, Inc. and its affiliates.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
This example shows how to use higher to do Model Agnostic Meta Learning (MAML)
for few-shot Omniglot classification.
For more details see the original MAML paper:
https://arxiv.org/abs/1703.03400

This code has been modified from Jackie Loong's PyTorch MAML implementation:
https://github.com/dragen1860/MAML-Pytorch/blob/master/omniglot_train.py

Our MAML++ fork and experiments are available at:
https://github.com/bamos/HowToTrainYourMAMLPytorch
"""

import argparse
import time
import typing

import pandas as pd
import numpy as np
import matplotlib as mpl

mpl.use('Agg')
import matplotlib.pyplot as plt

plt.style.use('bmh')

import torch
from torch import nn
import torch.nn.functional as F
import torch.optim as optim

import higher

from support.omniglot_loaders import OmniglotNShot

def main():
    argparser = argparse.ArgumentParser()
    argparser.add_argument('--n_way', type=int, help='n way', default=5)
    argparser.add_argument(
        '--k_spt', type=int, help='k shot for support set', default=5)
    argparser.add_argument(
        '--k_qry', type=int, help='k shot for query set', default=15)
    argparser.add_argument(
        '--task_num',
        type=int,
        help='meta batch size, namely task num',
        default=32)
    argparser.add_argument('--seed', type=int, help='random seed', default=1)
    args = argparser.parse_args()

    torch.manual_seed(args.seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(args.seed)
    np.random.seed(args.seed)

    # Set up the Omniglot loader.
    # device = torch.device('cuda')
    # from uutils.torch_uu import get_device
    # device = get_device()
    device =  torch.device(f"cuda:{gpu_idx}" if torch.cuda.is_available() else "cpu")
    db = OmniglotNShot(
        '/tmp/omniglot-data',
        batchsz=args.task_num,
        n_way=args.n_way,
        k_shot=args.k_spt,
        k_query=args.k_qry,
        imgsz=28,
        device=device,
    )

    # Create a vanilla PyTorch neural network that will be
    # automatically monkey-patched by higher later.
    # Before higher, models could *not* be created like this
    # and the parameters needed to be manually updated and copied
    # for the updates.
    net = nn.Sequential(
        nn.Conv2d(1, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(64, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        nn.Conv2d(64, 64, 3),
        nn.BatchNorm2d(64, momentum=1, affine=True),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(2, 2),
        Flatten(),
        nn.Linear(64, args.n_way)).to(device)

    # We will use Adam to (meta-)optimize the initial parameters
    # to be adapted.
    meta_opt = optim.Adam(net.parameters(), lr=1e-3)

    log = []
    for epoch in range(100):
        train(db, net, device, meta_opt, epoch, log)
        test(db, net, device, epoch, log)
        # plot(log)

def train(db, net, device, meta_opt, epoch, log):
    net.train()
    n_train_iter = db.x_train.shape[0] // db.batchsz

    for batch_idx in range(n_train_iter):
        start_time = time.time()
        # Sample a batch of support and query images and labels.
        x_spt, y_spt, x_qry, y_qry = db.next()

        task_num, setsz, c_, h, w = x_spt.size()
        querysz = x_qry.size(1)

        # TODO: Maybe pull this out into a separate module so it
        # doesn't have to be duplicated between `train` and `test`?

        # Initialize the inner optimizer to adapt the parameters to
        # the support set.
        n_inner_iter = 5
        inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

        qry_losses = []
        qry_accs = []
        meta_opt.zero_grad()
        for i in range(task_num):
            with higher.innerloop_ctx(
                    net, inner_opt, copy_initial_weights=False,
                    # track_higher_grads=True,
                    track_higher_grads=False,
            ) as (fnet, diffopt):
                # Optimize the likelihood of the support set by taking
                # gradient steps w.r.t. the model's parameters.
                # This adapts the model's meta-parameters to the task.
                # higher is able to automatically keep copies of
                # your network's parameters as they are being updated.
                for _ in range(n_inner_iter):
                    spt_logits = fnet(x_spt[i])
                    spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                    diffopt.step(spt_loss)

                # The final set of adapted parameters will induce some
                # final loss and accuracy on the query dataset.
                # These will be used to update the model's meta-parameters.
                qry_logits = fnet(x_qry[i])
                qry_loss = F.cross_entropy(qry_logits, y_qry[i])
                qry_losses.append(qry_loss.detach())
                qry_acc = (qry_logits.argmax(
                    dim=1) == y_qry[i]).sum().item() / querysz
                qry_accs.append(qry_acc)

                # Update the model's meta-parameters to optimize the query
                # losses across all of the tasks sampled in this batch.
                # This unrolls through the gradient steps.
                qry_loss.backward()

        assert meta_opt.param_groups[0]['params'][0].grad is not None
        meta_opt.step()
        qry_losses = sum(qry_losses) / task_num
        qry_accs = 100. * sum(qry_accs) / task_num
        i = epoch + float(batch_idx) / n_train_iter
        iter_time = time.time() - start_time
        if batch_idx % 4 == 0:
            print(
                f'[Epoch {i:.2f}] Train Loss: {qry_losses:.2f} | Acc: {qry_accs:.2f} | Time: {iter_time:.2f}'
            )

        log.append({
            'epoch': i,
            'loss': qry_losses,
            'acc': qry_accs,
            'mode': 'train',
            'time': time.time(),
        })

def test(db, net, device, epoch, log):
    # Crucially in our testing procedure here, we do *not* fine-tune
    # the model during testing for simplicity.
    # Most research papers using MAML for this task do an extra
    # stage of fine-tuning here that should be added if you are
    # adapting this code for research.
    net.train()
    n_test_iter = db.x_test.shape[0] // db.batchsz

    qry_losses = []
    qry_accs = []

    for batch_idx in range(n_test_iter):
        x_spt, y_spt, x_qry, y_qry = db.next('test')

        task_num, setsz, c_, h, w = x_spt.size()
        querysz = x_qry.size(1)

        # doesn't have to be duplicated between `train` and `test`?
        n_inner_iter = 5
        inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

        for i in range(task_num):
            with higher.innerloop_ctx(net, inner_opt, track_higher_grads=False) as (fnet, diffopt):
                # Optimize the likelihood of the support set by taking
                # gradient steps w.r.t. the model's parameters.
                # This adapts the model's meta-parameters to the task.
                for _ in range(n_inner_iter):
                    spt_logits = fnet(x_spt[i])
                    spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                    diffopt.step(spt_loss)

                # The query loss and acc induced by these parameters.
                qry_logits = fnet(x_qry[i]).detach()
                qry_loss = F.cross_entropy(
                    qry_logits, y_qry[i], reduction='none')
                qry_losses.append(qry_loss.detach())
                qry_accs.append(
                    (qry_logits.argmax(dim=1) == y_qry[i]).detach())

    qry_losses = torch.cat(qry_losses).mean().item()
    qry_accs = 100. * torch.cat(qry_accs).float().mean().item()
    print(
        f'[Epoch {epoch + 1:.2f}] Test Loss: {qry_losses:.2f} | Acc: {qry_accs:.2f}'
    )
    log.append({
        'epoch': epoch + 1,
        'loss': qry_losses,
        'acc': qry_accs,
        'mode': 'test',
        'time': time.time(),
    })

def plot(log):
    # Generally you should pull your plotting code out of your training
    # script but we are doing it here for brevity.
    df = pd.DataFrame(log)

    fig, ax = plt.subplots(figsize=(6, 4))
    train_df = df[df['mode'] == 'train']
    test_df = df[df['mode'] == 'test']
    ax.plot(train_df['epoch'], train_df['acc'], label='Train')
    ax.plot(test_df['epoch'], test_df['acc'], label='Test')
    ax.set_xlabel('Epoch')
    ax.set_ylabel('Accuracy')
    ax.set_ylim(70, 100)
    fig.legend(ncol=2, loc='lower right')
    fig.tight_layout()
    fname = 'maml-accs.png'
    print(f'--- Plotting accuracy to {fname}')
    fig.savefig(fname)
    plt.close(fig)

# Won't need this after this PR is merged in:
# https://github.com/pytorch/pytorch/pull/22245
class Flatten(nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

if __name__ == '__main__':
    main()

Note:
I asked a similar question here Would making the gradient "data" by detaching them implement first order MAML using PyTorch's higher library? but that one is slightly different. It is asking about a custom implementation that detaches the gradients directly to make them "data". This one is asking why the setting track_higher_grads=False screws up the population of gradients -- which as I understand should not.

related:

bug report since from the discussion I expect the flag to solve the issues: https://github.com/facebookresearch/higher/issues/129
https://github.com/facebookresearch/higher/issues?q=is%3Aissue+first+order+maml+is%3Aclosed
https://github.com/facebookresearch/higher/issues/63
https://github.com/facebookresearch/higher/issues/128
https://www.reddit.com/r/pytorch/comments/sixdqd/what_is_the_official_implementation_of_first/
https://www.reddit.com/r/pytorch/comments/si5xv1/would_making_the_gradient_data_by_detaching_them/

Bounty
Explain the reasoning of why the solution here works i.e. why
track_higher_grads = True
...
diffopt.step(inner_loss, grad_callback=lambda grads: [g.detach() for g in grads])

computed FO maml but:
 new_params = params[:]
 for group, mapping in zip(self.param_groups, self._group_to_param_list):
     for p, index in zip(group['params'], mapping):
         if self._track_higher_grads:
             new_params[index] = p
         else:
             new_params[index] = p.detach().requires_grad_() # LIKELY THIS LINE!!!

does not allow FO to work properly and sets .grads to None (not populate the grad field). The assignment with p.detach().requires_grad_() honestly looks the same to me. This .requires_grad_() evens seems extra "safe".

Output: The reason why track_higher_grads=False doesn't actually work is that it detaches the gradients of the post-adaptation parameters rather than just the gradients (see here). So you get no gradient at all from your outer loop loss. What you really want is just to detach the gradients on just the inner loop-computed gradients, but leave the (otherwise trivial) computation graph between model initialization and adapted parameters intact.


Instruction: IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 3)

Input:
I created a random function with the line below
x = torch.randn(4, 3)

and used the transpose function as shown here
torch.transpose(x, 0, 1)

I got the error line below. Who can assist with a solution

IndexError                                Traceback (most recent call last)
<ipython-input-19-28494ba2cedc> in <module>()
----> 1 torch.transpose(x, 0, 3)

IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 3)

Output: You are trying to transpose x = torch.randn(4, 3), which is 2D. torch.transpose(x, 0, 1) works fine because you swap dimensions 0 and 1. But then, you try to swap dimensions 0 and 3 by doing torch.transpose(x, 0, 3) but your x does not have the 3rd dimension


Instruction: Take multiply slices in numpy/pytorch

Input:
I have a big one dimensional array X.shape = (10000,), and a vector of indices y = [0, 7, 9995].
I would like to get a matrix with rows
[
 X[0 : 100],
 X[7 : 107],
 concat(X[9995:], X[:95]),
]

That is, slices of length 100, starting at each index, with wrap-around.
I can do that with a python loop, but I'm wondering if there's a smarter batched way of doing it in pytorch or numpy, since my arrays can be quite large.

Output:
Quite simple, actually.

For each element E in y, create a range from E to E + 100
Concatenate all the ranges horizontally
Modulo the resulting array by the length of X

indexes = np.hstack([np.arange(v, v + 100) for v in y]) % X.shape[0]

Output:
>>> indexes
array([   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
         11,   12,   13,   14,   15,   16,   17,   18,   19,   20,   21,
         22,   23,   24,   25,   26,   27,   28,   29,   30,   31,   32,
         33,   34,   35,   36,   37,   38,   39,   40,   41,   42,   43,
         44,   45,   46,   47,   48,   49,   50,   51,   52,   53,   54,
         55,   56,   57,   58,   59,   60,   61,   62,   63,   64,   65,
         66,   67,   68,   69,   70,   71,   72,   73,   74,   75,   76,
         77,   78,   79,   80,   81,   82,   83,   84,   85,   86,   87,
         88,   89,   90,   91,   92,   93,   94,   95,   96,   97,   98,
         99,    7,    8,    9,   10,   11,   12,   13,   14,   15,   16,
         17,   18,   19,   20,   21,   22,   23,   24,   25,   26,   27,
         28,   29,   30,   31,   32,   33,   34,   35,   36,   37,   38,
         39,   40,   41,   42,   43,   44,   45,   46,   47,   48,   49,
         50,   51,   52,   53,   54,   55,   56,   57,   58,   59,   60,
         61,   62,   63,   64,   65,   66,   67,   68,   69,   70,   71,
         72,   73,   74,   75,   76,   77,   78,   79,   80,   81,   82,
         83,   84,   85,   86,   87,   88,   89,   90,   91,   92,   93,
         94,   95,   96,   97,   98,   99,  100,  101,  102,  103,  104,
        105,  106, 9995, 9996, 9997, 9998, 9999,    0,    1,    2,    3,
          4,    5,    6,    7,    8,    9,   10,   11,   12,   13,   14,
         15,   16,   17,   18,   19,   20,   21,   22,   23,   24,   25,
         26,   27,   28,   29,   30,   31,   32,   33,   34,   35,   36,
         37,   38,   39,   40,   41,   42,   43,   44,   45,   46,   47,
         48,   49,   50,   51,   52,   53,   54,   55,   56,   57,   58,
         59,   60,   61,   62,   63,   64,   65,   66,   67,   68,   69,
         70,   71,   72,   73,   74,   75,   76,   77,   78,   79,   80,
         81,   82,   83,   84,   85,   86,   87,   88,   89,   90,   91,
         92,   93,   94])

Now just use index X with that:
X[indexes]


Instruction: Getting a 'KeyError" while trying to access a Custom Dataset

Input:
I am trying to learn PyTorch and make use of a custom dataset. Code Credit - https://github.com/vineeth2309/Custom-Dataset-and-Dataloader-in-Torch.
However, when I run the code I get a 'KeyError'.
import glob
import cv2
import numpy as np
import torch
from torch.utils.data import Dataset, DataLoader

class CustomDataset(Dataset):
    def __init__(self):
        self.imgs_path = "Dog_Cat_Dataset/"
        file_list = glob.glob(self.imgs_path + "*")
        print(file_list)
        self.data = []
        for class_path in file_list:
            class_name = class_path.split("/")[-1]
            for img_path in glob.glob(class_path + "/*.jpeg"):
                self.data.append([img_path, class_name])
        print(self.data)
        self.class_map = {"dogs" : 0, "cats": 1}
        self.img_dim = (416, 416)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        img_path, class_name = self.data[idx]
        img = cv2.imread(img_path)
        img = cv2.resize(img, self.img_dim)
        class_id = self.class_map[class_name]
        img_tensor = torch.from_numpy(img)
        img_tensor = img_tensor.permute(2, 0, 1)
        class_id = torch.tensor([class_id])
        return img_tensor, class_id

if __name__ == "__main__":
    dataset = CustomDataset()
    print (dataset)
    data_loader = DataLoader(dataset, batch_size=4, shuffle=True)
    print (data_loader)
    for imgs, labels in data_loader:
        print("Batch of images has shape: ",imgs.shape)
        print("Batch of labels has shape: ", labels.shape)

Stack-Trace :
C:\Users\parag\anaconda3\envs\tf-gpu\python.exe C:/Users/parag/PycharmProjects/Custom-Dataset-and-Dataloader-in-Torch/Main.py
['Dog_Cat_Dataset\\cats', 'Dog_Cat_Dataset\\dogs']
[['Dog_Cat_Dataset\\cats\\1.jpeg', 'Dog_Cat_Dataset\\cats'], ['Dog_Cat_Dataset\\cats\\2.jpeg', 'Dog_Cat_Dataset\\cats'], ['Dog_Cat_Dataset\\cats\\3.jpeg', 'Dog_Cat_Dataset\\cats'], ['Dog_Cat_Dataset\\cats\\4.jpeg', 'Dog_Cat_Dataset\\cats'], ['Dog_Cat_Dataset\\cats\\5.jpeg', 'Dog_Cat_Dataset\\cats'], ['Dog_Cat_Dataset\\dogs\\1.jpeg', 'Dog_Cat_Dataset\\dogs'], ['Dog_Cat_Dataset\\dogs\\2.jpeg', 'Dog_Cat_Dataset\\dogs'], ['Dog_Cat_Dataset\\dogs\\3.jpeg', 'Dog_Cat_Dataset\\dogs'], ['Dog_Cat_Dataset\\dogs\\4.jpeg', 'Dog_Cat_Dataset\\dogs'], ['Dog_Cat_Dataset\\dogs\\5.jpeg', 'Dog_Cat_Dataset\\dogs']]
<__main__.CustomDataset object at 0x00000254C0568FD0>
<torch.utils.data.dataloader.DataLoader object at 0x00000254C250A7F0>
Traceback (most recent call last):
  File "C:\Users\parag\PycharmProjects\Custom-Dataset-and-Dataloader-in-Torch\Main.py", line 40, in <module>
    for imgs, labels in data_loader.dataset:
  File "C:\Users\parag\PycharmProjects\Custom-Dataset-and-Dataloader-in-Torch\Main.py", line 29, in __getitem__
    class_id = self.class_map[class_name]
KeyError: 'Dog_Cat_Dataset\\cats'

Process finished with exit code 1

[My Folder Structure][1]
I have tried, but not able to resolve the error. Could someone please help me with this?
[1]: https://i.stack.imgur.com/1giJw.png

Output:
You've used forward slash when it should be backslash:
    for class_path in file_list:
        class_name = class_path.split("\\")[-1]
        for img_path in glob.glob(class_path + "\*.jpeg"):
            self.data.append([img_path, class_name])

I'm guessing you're running on windows when the example is from linux.


Instruction: Way to define a generic tensor-like type (like tf.Tensor | torch.Tensor | np.ndarray | float) in python 3.8?

Input: I'm annotating my code for a high level neural network library. I want to make it as flexible as possible while still clarifying type expectations for developers using the library. In many cases, this requires writing 'generic' functions that operate on tensors defined in arbitrary backends (like tf.Tensor, torch.Tensor, np.ndarray, as well as python types like float, int, and bool; it'd be nice if I could statically enforce dtype's as well, but that not my concern here) Is there a way to define a generic Tensor-like type in python 3.8?

Output:
After coding for a few hours, I began to see that you don't want to do this in most cases. Here's some reasons why:

it's not pythonic. "Avoid the magical wand" when it adds needless complexity
you'll have duck-typing issues like calling .size in torch but .shape in tf which results in a lot of if/elif/else structures.
you're going to be repeating yourself for a lot of generic-backend logic

I would recommend for other developers facing this prediciment to seriously ask themselves if they are willing to maintain a high level framework on two backends. You see, any time you want to perform an operation on the tensor (say transpose()), you have to ask if that operation has an identical signature in all supported backends. If not, you start ending up with huge swaths of repeated if/elif/else code and development productivity really starts to suffer. If willing to proceed, try using an existing library like keras.backend which has limited tensorflow and thaneo support, or write your own cross-framework tensor-framework abstract for us all to use. (And please post a comment if you made or found one)
In most cases however, it suffices to imitate the einops approach and simply define an arbitrary TypeVar like so:
Tensor = TypeVar('Tensor')

Also, CoPilot recommended this to me:
TensorType = type(None)

Then you can use this type var in your annotations as expected:
def iterate(self, a: Tensor, b: Tensor) -> Tensor:
  c = self.neural_net(a)
  d = c + a - b
  return d / a


Instruction: How to deal with CUDA version?

Input:
How to set up different versions of CUDA in one OS?
Here is my problem: Lastest Tensorflow with GPU support requires CUDA 11.2, whereas Pytorch works with 11.3. So what is the solution to install both libraries in Windows and Ubuntu?

Output:
One solution is to use Docker Container Environment, which would only need the Nvidia Driver to be of version XYZ.AB; in this way, you can use both PyTorch and TensorFlow versions.
A very good starting point for your problem would be this one(ML-WORKSPACE) : https://github.com/ml-tooling/ml-workspace


Instruction: Is it possible to add tensors of different sizes together in pytorch?

Input:
I have an image gradient of size (3, 224, 224) and a patch of (1, 768). is it possible to add this gradient to the patch to get a size of the patch (1, 768)?
Forgive my inquisitiveness. I know pytorch too utilizes broadcasting and I am not sure if I will able to do so with two different tensors in way similar to the line below:
torch.add(a, b)

For example:

The end product would be the same patch on the left with the gradient of an entire image on the right added to it. My understanding is that it’s not possible, but knowledge isn’t bounded.

Output: I figured out to do it myself. I divided the image gradient (right) into 16 x 16 patches, created a loop that adds each patch to the original image patch (left). This way, I was able to add a 224 x 224 image gradient into a 16 x 16 patch. I just wanted to see what would happen if I do such


Instruction: What does it mean torch.rand(1, 3, 64, 64)?

Input:
I am beginner in PyTorch. In one tutorial, I saw: torch.rand(1, 3, 64, 64), I understand that it creates a Tensor with random numbers following standard normal distribution.
The outputs looks like:
tensor([[[[0.1352, 0.5110, 0.7585,  ..., 0.9067, 0.4730, 0.8077],
          [0.2471, 0.8726, 0.3580,  ..., 0.4983, 0.9747, 0.5219],
          [0.8554, 0.4266, 0.0718,  ..., 0.6734, 0.8739, 0.6137],
          ...,
          [0.2132, 0.9319, 0.5361,  ..., 0.3981, 0.2057, 0.7032],
          [0.3347, 0.5330, 0.7019,  ..., 0.6713, 0.0936, 0.4706],
          [0.6257, 0.6656, 0.3322,  ..., 0.6664, 0.8149, 0.1887]],

         [[0.3210, 0.6469, 0.7772,  ..., 0.3175, 0.5102, 0.9079],
          [0.3054, 0.2940, 0.6611,  ..., 0.0941, 0.3826, 0.3103],
          [0.7484, 0.3442, 0.1034,  ..., 0.8028, 0.4643, 0.2800],
          ...,
          [0.9946, 0.5868, 0.8709,  ..., 0.4837, 0.6691, 0.5303],
          [0.1770, 0.5355, 0.8048,  ..., 0.1843, 0.0658, 0.3817],
          [0.9612, 0.0122, 0.5012,  ..., 0.4198, 0.3294, 0.2106]],

         [[0.5800, 0.5174, 0.5454,  ..., 0.3881, 0.3277, 0.5470],
          [0.8871, 0.7536, 0.9928,  ..., 0.8455, 0.8071, 0.0062],
          [0.2199, 0.0449, 0.2999,  ..., 0.3570, 0.7996, 0.3253],
          ...,
          [0.8238, 0.1100, 0.1489,  ..., 0.0265, 0.2165, 0.2919],
          [0.4074, 0.5817, 0.8021,  ..., 0.3417, 0.1280, 0.9279],
          [0.0047, 0.1796, 0.4522,  ..., 0.3257, 0.2657, 0.4405]]]])

But what do the 4 parameters (1, 3, 64, 64) mean exactly? Thanks!

Output:
This is the shape of the output tensor. Specifically, it means that your output tensor has

(1, 3, 64, 64): 1 element of shape (3, 64, 64) in dimension 0

(1, 3, 64, 64): 3 elements of shape (64, 64) in dimension 1 for a given dimension 0

(1, 3, 64, 64): 64 elements of shape (64,) in dimension 2 for a given dimension 1 and dimension 0

(1, 3, 64, 64): 64 scalars in dimension 3 for a given dimension 2, dimension 1, and dimension 0

You can confirm this by comparing the number of elements with the tensor's "capacity":
>>> torch.rand(1,3,64,64).numel()
12288
>>> 1 * 3 * 64 * 64
12288


Instruction: how to get mentions in pytorch NER instead of toknes?

Input:
I am using PyTorch and a pre-trained model.
Here is my code:
class NER(object):
    def __init__(self, model_name_or_path, tokenizer_name_or_path):
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
        self.model = AutoModelForTokenClassification.from_pretrained(
            model_name_or_path)
        self.nlp = pipeline("ner", model=self.model, tokenizer=self.tokenizer)

    def get_mention_entities(self, query):
        return self.nlp(query)

when I call get_mention_entities and print its output for "اینجا دانشگاه صنعتی امیرکبیر است."
it gives:
[{'entity': 'B-FAC', 'score': 0.9454591, 'index': 2, 'word': 'دانشگاه', 'start': 6, 'end': 13}, {'entity': 'I-FAC', 'score': 0.9713519, 'index': 3, 'word': 'صنعتی', 'start': 14, 'end': 19}, {'entity': 'I-FAC', 'score': 0.9860724, 'index': 4, 'word': 'امیرکبیر', 'start': 20, 'end': 28}]

As you can see, it can recognize the university name, but there are three tokens in the list.
Is there any standard way to combine these tokens based on the "entity" attribute?
desired output is something like:
[{'entity': 'FAC', 'word': 'دانشگاه صنعتی امیرکبیر', 'start': 6, 'end': 28}]

Finally, I can write a function to iterate, compare, and merge the tokens based on the "entity" attribute, but I want a standard way like an internal PyTorch function or something like this.
my question is similar to this question.
PS: "دانشگاه صنعتی امیرکبیر" is a university name.

Output:
Huggingface's NER pipeline has an argument grouped_entities=True which will do exactly what you seek: group BI into unified entities.
Adding
self.nlp = pipeline("ner", model=self.model, tokenizer=self.tokenizer, grouped_entities=True)

should do the trick


Instruction: Function of Activation functions

Input:
is it possible to define a function of activation function? I tried to do :
def activation():
    # return nn.Sin()
    # return nn.Tanh()
    # return nn.Sigmoid()
    # return nn.Tanhshrink()
    return nn.HardTanh(-1,1)
    # return nn.Hardswish()
    # return nn.functionnal.silu()

But i get an error when trying to call it. Here is an example:
def f():
  return nn.Tanh()
input = torch.randn(2)
output = f(input)
print(output)

it outputs "TypeError: f() takes 0 positional arguments but 1 was given". It doesn't work even i gave it an argument x.

Output:
You can either use the object-oriented approach:
>>> f = nn.Tanh()
>>> output = f(x)

Or the functional approach where you will find the equivalent for nn.Tanh inside nn.functional as tanh.
>>> f = nn.functional.tanh
>>> output = f(x)


Instruction: Which model/technique to use for specific sentence extraction?

Input:
I have a dataset of tens of thousands of dialogues / conversations between a customer and customer support. These dialogues, which could be forum posts, or long-winded email conversations, have been hand-annotated to highlight the sentence containing the customers problem. For example:

Dear agent, I am writing to you because I have a very annoying problem with my washing machine. I bought it three weeks ago and was very happy with it. However, this morning the door does not lock properly. Please help

Dear customer.... etc

The highlighted sentence would be:

However, this morning the door does not lock properly.

What approaches can I take to model this, so that in future I can automatically extract the customers problem? The domain of the datasets are broad, but within the hardware space, so it could be appliances, gadgets, machinery etc.
What is this type of problem called?
I thought this might be called "intent recognition", but most guides seem to refer to multiclass classification. The sentence either is or isn't the customers problem. I considered analysing each sentence and performing binary classification, but I'd like to explore options that take into account the context of the rest of the conversation if possible.
What resources are available to research how to implement this in Python (using tensorflow or pytorch)

I found a model on HuggingFace which has been pre-trained with customer dialogues, and have read the research paper, so I was considering fine-tuning this as a starting point, but I only have experience with text (multiclass/multilabel) classification when it comes to transformers.

Output:
If you want to get a specific sentence (without any modification) from the original input text, that is often referred to as  'span classification'  where the output is the index of the first and last word of the specific sentence.
The state-of-the-art now is the attention models like BERT
.You can check the Bert models that are designed for the 'span classification' problem in huggingface as RobertaForQuestionAnswering https://huggingface.co/docs/transformers/model_doc/roberta#transformers.TFRobertaForQuestionAnswering that uses TensorFlow or PyTorch library.


Instruction: How to reload hydra config with enumerations

Input:
Is there a better way to reload a hydra config from an experiment with enumerations? Right now I reload it like so:
initialize_config_dir(config_dir=exp_dir, ".hydra"), job_name=config_name)
cfg = compose(config_name, overrides=overrides)
print(cfg.enum)
>>> ENUM1

But ENUM1 is actually an enumeration that normally loads as
>>> <SomeEnumClass.ENUM1: 'enum1'>

I am able to fix this by adding a configstore default to the experiment hydra file:
defaults:
  - base_config_cs

Which now results in
initialize_config_dir(config_dir=exp_dir, ".hydra"), job_name=config_name)
cfg = compose(config_name, overrides=overrides)
print(cfg.enum)
>>> <SomeEnumClass.ENUM1: 'enum1'>

Is there a better way to do this without adding this? Or can I add the default in the python code?

Output:
This is a good question -- reliably reloading configs from previous Hydra runs is an area that could be improved.
As you've discovered, loading the saved file config.yaml directly results in an untyped DictConfig object.
The solution below involves a script called reload.py that creates a config node with a defaults list that loads both the schema base_config_cs and the saved file config.yaml.
At the end of this post I also give a simple solution that involves loading .hydra/overrides.yaml to re-run the config composition process.

Suppose you've run a Hydra job with the following setup:
# app.py
from dataclasses import dataclass
from enum import Enum
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import DictConfig

class SomeEnumClass(Enum):
    ENUM1 = 1
    ENUM2 = 2

@dataclass
class Schema:
    enum: SomeEnumClass
    x: int = 123
    y: str = "abc"

def store_schema() -> None:
    cs = ConfigStore.instance()
    cs.store(name="base_config_cs", node=Schema)

@hydra.main(config_path=".", config_name="foo")
def app(cfg: DictConfig) -> None:
    print(cfg)

if __name__ == "__main__":
    store_schema()
    app()

# foo.yaml
defaults:
  - base_config_cs
  - _self_
enum: ENUM1
x: 456

$ python app.py y=xyz
{'enum': <SomeEnumClass.ENUM1: 1>, 'x': 456, 'y': 'xyz'}

After running app.py, there exists a directory outputs/2022-02-05/06-42-42/.hydra containing the saved file config.yaml.
As you correctly pointed out in your question, to reload the saved config you must merge the schema base_config_cs with the contents of config.yaml. Here is a pattern for accomplishing that:
# reload.py
import os
from hydra import compose, initialize_config_dir
from hydra.core.config_store import ConfigStore
from app import store_schema

config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-19-56")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
assert os.path.exists(f"{saved_cfg_dir}/{config_name}.yaml")

store_schema()  # stores `base_config_cs`
cs = ConfigStore.instance()
cs.store(
    name="reload_conf",
    node={
        "defaults": [
            "base_config_cs",
            config_name,
        ]
    },
)

with initialize_config_dir(config_dir=saved_cfg_dir):
    cfg = compose("reload_conf")
print(cfg)

$ python reload.py
{'enum': <SomeEnumClass.ENUM1: 1>, 'x': 456, 'y': 'xyz'}

In the above, python file reload.py, we store a node called reload_conf in the ConfigStore. Storing reload_conf this way is equivalent to creating a file called reload_conf.yaml that is discoverable by Hydra on the config search path. This reload_conf node has a defaults list that loads both the schema base_config_cs and config. For this to work, the following two conditions must be met:

the schema base_config_cs must be stored in the ConfigStore. This is accomplished by calling the store_schema function that we have imported from app.py.
a config node with name specified by the variable config_name, i.e. config.yaml in this example, must be discoverable by Hydra (which is taken care of here by calling initialize_config_dir).

Note that in foo.yaml we have a defaults list ["base_config_cs", "_self_"] that loads the schema base_config_cs before loading the contents _self_ of foo. In order for reload_conf to reconstruct the app's config with the same merge order, base_config_cs should come before config_name in the defaults list belonging to reload_conf.

The above approach could be taken one step further by removing the defaults list from foo.yaml and using cs.store to ensure the same defaults list is used in both the app and the reloading script
# app2.py
from dataclasses import dataclass
from enum import Enum
from typing import Any, List
import hydra
from hydra.core.config_store import ConfigStore
from omegaconf import MISSING, DictConfig

class SomeEnumClass(Enum):
    ENUM1 = 1
    ENUM2 = 2

@dataclass
class RootConfig:
    defaults: List[Any] = MISSING
    enum: SomeEnumClass = MISSING
    x: int = 123
    y: str = "abc"

def store_root_config(primary_config_name: str) -> None:
    cs = ConfigStore.instance()
    # defaults list defined here:
    cs.store(
        name="root_config", node=RootConfig(defaults=["_self_", primary_config_name])
    )

@hydra.main(config_path=".", config_name="root_config")
def app(cfg: DictConfig) -> None:
    print(cfg)

if __name__ == "__main__":
    store_root_config("foo2")
    app()

# foo2.yaml (note NO DEFAULTS LIST)
enum: ENUM1
x: 456

$ python app2.py hydra.job.chdir=false y=xyz
{'enum': <SomeEnumClass.ENUM1: 1>, 'x': 456, 'y': 'xyz'}

# reload2.py
import os
from hydra import compose, initialize_config_dir
from hydra.core.config_store import ConfigStore
from app2 import store_root_config

config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-45-43")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
assert os.path.exists(f"{saved_cfg_dir}/{config_name}.yaml")

store_root_config("config")
with initialize_config_dir(config_dir=saved_cfg_dir):
    cfg = compose("root_config")
print(cfg)

$ python reload2.py
{'enum': <SomeEnumClass.ENUM1: 1>, 'x': 456, 'y': 'xyz'}

A simpler alternative approach is to use .hydra/overrides.yaml to recompose the app's configuration based on the overrides that were originally passed to Hydra:
# reload3.py
import os
import yaml
from hydra import compose, initialize
from app import store_schema

config_name = "config"
exp_dir = os.path.abspath("outputs/2022-02-05/07-19-56")
saved_cfg_dir = os.path.join(exp_dir, ".hydra")
overrides_path = f"{saved_cfg_dir}/overrides.yaml"
assert os.path.exists(overrides_path)

overrides = yaml.unsafe_load(open(overrides_path, "r"))
print(f"{overrides=}")
store_schema()
with initialize(config_path="."):
    cfg = compose("foo", overrides=overrides)
print(cfg)

$ python reload3.py
overrides=['y=xyz']
{'enum': <SomeEnumClass.ENUM1: 1>, 'x': 456, 'y': 'xyz'}

This approach has its drawbacks: if your app's configuration involves some non-hermetic operation like querying a timestamp (e.g. via Hydra's now resolver) or looking up an environment variable (e.g. via the oc.env resolver), the configuration composed by reload.py might be different from the original version loaded in app.py.


Instruction: I think I have reconstructed the computational graph before, but it hints me "Trying to backward through the graph a second time ", why?

Input:
A image to discribe my question
From my point of view, every iterations, the computational graph will be constructed at the first arrow, and it will be used and delete at the second arrow in backward pass. So, why it tells me that:

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

Here is my code:
def train(num_epoch = 10,len_vocab = 1, num_hidden=256,embedding_dim = 8):
    data = get_data()
    model = MyRNN(len_vocab,num_hidden,embedding_dim)
    if os.path.exists('QingBinLi'):
        model.load_state_dict(torch.load('QingBinLi'))
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=1e-5)
    loss_for_draw = []
    for epoch in range(num_epoch+1):

        h = torch.randn(1,1,num_hidden)
        loss_average = []
        for i in range(data.shape[-2]):
            optimizer.zero_grad()
            #I think my computational graph will be constructed there
            pre,h = model(data[:,:,i,:] ,h)
            pre = pre.unsqueeze(0).unsqueeze(0)
            loss = criterion(pre, data[:,:,i+1,:])
            loss_average.append(loss)
            #I think everytime the backward pass will delete the computational graph.
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), max_norm=10)
            optimizer.step()
            print(f"finish {i+1} times")

        loss_for_draw.append(sum(loss_average)/len(loss_average))
        torch.save(model.state_dict(), 'QingBinLi')
        print(f'now epoch:{epoch}, loss = {loss_for_draw[-1]}')

    return loss_for_draw

class MyRNN(nn.Module):
    def __init__(self,len_vocab, num_hidden=256,embedding_dim = 8):
        super(MyRNN,self).__init__()
        self.rnn = nn.RNN(embedding_dim, num_hidden)
        self.num_directions=1
        self.output_model = nn.Linear(num_hidden, embedding_dim)

    def forward(self, x, h):
        y, h = self.rnn(x, h)

        output = self.output_model(y.reshape((-1)))

        return output, h

So, if I'm right, it shouldn't tell me "Trying to backward through the graph a second time"..
So, where did i go wrong

Output:
Variable h and data requires gradient, so we must add 2 lines:
h = h.detach()
data = data.detach()


Instruction: Pytorch-index on multiple dimension tensor in a batch

Input:
Given a 1-d tensor:
A = torch.tensor([1, 2, 3, 4])

suppose we have some "indexer tensor"
ind1 = torch.tensor([3, 0, 1])
ind2 = torch.tensor([[3, 0], [1, 2]])

as we run A[ind1] & A[ind2]
we get results tensor([4, 1, 2]) & tensor([[4, 1],[2, 3]])
which is the same shape of the indexed tensor (ind1 and ind2) and its value are mapped from tensor A.
I want to ask how can I index on higher dimension tensors?

Currently I have one solution:
For a N-d tensor A, suppose we have the indexer tensor IND,
IND is like [[i11, i12, ... i1N], [i21, i22, ... i2N], ...[iM1, i22, ... iMN], where M is the number of indexed elements.
We can divide IND into N tensors, where
IND_1 = torch.tensor([i11, i21, ... iM1])
...
IND_N = torch.tensor([i1N, i2N, ... iMN])

as we run A[IND_1, ... IND_N], we got tensor(v1, v2, ... vM)
Example:
A = tensor([[1, 2], [3, 4]], [[5, 6], [7, 8]]])  # [2 * 2 * 2]
ind1 = tensor([1, 0, 1])
ind2 = tensor([1, 1, 0])
ind3 = tensor([0, 1, 0])
A[ind1, ind2, ind3]
=> tensor([7, 4, 5])
# and the good thing is you can control the shape of result tensor by modifying the inds' shape.
ind1 = tensor([[0, 0], [1, 0]])
ind2 = tensor([[1, 1], [0, 1]])
ind3 = tensor([[0, 1], [0, 0]])
A[ind1, ind2, ind3]
=> tensor([[3, 4],[5, 3]])  # same as inds' shape

Anyone has more elegant solutions?

Output:
1- Manual approach using unraveled indices on flattened input.
If you want to index on an arbitrary number of axes (all axes of A) then one straightforward approach is to flatten all dimensions and unravel the indices. Let's assume that A is 3D and we want to index it using a stack of ind1, ind2, and ind3:
>>> ind = torch.stack((ind1, ind2, ind3))

You can first unravel the indices using A's strides:
>>> unraveled = torch.tensor(A.stride()) @ ind.flatten(1)

Then flatten A, index it with unraveled and reshape to the final form:
>>> A.flatten()[unraveled].reshape_as(ind[0])

2- Using a simple split of ind.
You can actually perform the same operation using torch.chunk:
>>> A[ind.chunk(len(ind))][0]

Or alternatively torch.split which is identical:
>>> A[ind.split(1)][0]

3- Initial answer for single-axis indexing.
Let's take a minimal multi-dimensional example with A being a 2-D tensor defined as:
>>> A = torch.tensor([[1, 2, 3, 4],
                      [5, 6, 7, 8]])

From your description of the problem:

the same shape of index tensor and its value are mapped from tensor A.

Then the indexer tensor would require to have the same shape as the indexed tensor A, since this one is no longer flat. Otherwise, what would the result of A (shaped (2, 4)) indexed by ind1 (shape (3,)) be?
If you are indexing on a single dimension then you can utilize torch.gather:
>>> A.gather(1, ind2)
tensor([[4, 1],
        [6, 7]])


Instruction: PyTorch: Multi-class segmentation loss value != 0 when using target image as the prediction

Input:
I was performing semantic segmentation using PyTorch. There are a total of 103 different classes in the dataset and the targets are RGB images with only the Red channel containing the labels. I was using nn.CrossEntropyLoss as my loss function. For sanity, I wanted to check if using nn.CrossEntropyLoss is correct for this problem and whether it has the expected behaviour
I pick a random mask from my dataset and create a categorical version of it using this custom transform
class ToCategorical:
    def __init__(self, n_classes: int) -> None:
        self.n_classes = n_classes

    def __call__(self, sample: torch.Tensor):
        mask = sample.permute(1, 2, 0)
        categories = torch.unique(mask).tolist()[1:]  # get all categories other than 0
        # build a tensor with `n_classes` channels
        one_hot_image = torch.zeros(self.n_classes, *mask.shape[:-1])
        for category in categories:
            # get spacial locs where the categ is present
            rows, cols, _ = torch.where(mask == category)
            # in same spacial loc but in `categ` channel fill 1
            one_hot_image[category, rows, cols] = 1
        return one_hot_image

And then I send this image as the output (prediction) and use the ground truth mask as the target to the loss function.
import torch.nn as nn

mask = T.PILToTensor()(Image.open("path_to_image").convert("RGB"))
categorical_mask = ToCategorical(103)(mask).unsqueeze(0)
mask = mask[0].unsqueeze(0)  # get only the red channel, add fake batch_dim

loss_fn = nn.CrossEntropyLoss()

target = mask
output = categorical_mask

print(output.shape, target.shape)
print(loss_fn(output, target.to(torch.long)))

I expected the loss to be zero but to my surprise, the output is as follows
torch.Size([1, 103, 600, 800]) torch.Size([1, 600, 800])
tensor(4.2836)

I verified with other samples in the dataset and I obtained similar values for other masks as well. Am I doing something wrong? I expect the loss to be = 0 when the output is the same as the target.
PS. I also know that nn.CrossEntropyLoss is the same as using log_softmax followed by nn.NLLLoss() but even I obtained the same value by using nllloss as well
For Reference
Dataset used: UECFoodPixComplete

Output:
I would like to adress this:

I expect the loss to be = 0 when the output is the same as the target.

If the prediction matches the target, i.e. the prediction corresponds to a one-hot-encoding of the labels contained in the dense target tensor, but the loss itself is not supposed to equal to zero. Actually, it can never be equal to zero because the nn.CrossEntropyLoss function is always positive by definition.
Let us take a minimal example with number of #C classes and a target y_pred and a prediction y_pred consisting of prefect predictions:
As a quick reminder:

The softmax is applied on the logits (q_i) as p_i = log(exp(q_i)/sum_j(exp(q_j)):
>>> p = F.softmax(y_pred, 1)

Similarly if you are using the log-softmax, defined as logp_i = log(p_i):
>>> logp = F.log_softmax(y_pred, 1)

Then comes the negative likelihood function computed between x the input and y the target: -y*x. In association with the softmax, it comes down to -y*p, or -y*logp respectively. In any case, whether you apply the log or not, only the predictions corresponding to the true classes will remain since the others ones are zeroed-out.

That being said, applying the NLLLoss on y_pred would indeed result with a 0 as you expected in your question. However, here we apply it on the probability distribution or log-probability: p, or logp respectively!
In our specific case, p_i = 1 for the true class and p_i = 0 for all other classes (there are #C - 1 of those). This means the softmax of the logit associated with the true class will equal to exp(1)/sum_i(p_i). And since sum_i(p_i) = (#C-1)*exp(0) + exp(1). We therefore have:
softmax(p) = e / (#C - 1 + e)

Similarly for log-softmax:
log-softmax(p) = log(e / (#C-1 + e)) = 1 - log(#C - 1 + e)

If we proceed by applying the negative likelihood function we simply get cross-entropy(y_pred, y_true) = (nllloss o log-softmax)(y_pred, y_true). This results in:
loss = - (1 - log(#C - 1 + e)) = log(#C - 1 + e) - 1

This effectively corresponds to the minimum of the nn.CrossEntropyLoss function.

Regarding your specific case where #C = 103, you may have an issue in your code... since the average loss should equal to log(102 + e) - 1 i.e. around 3.65.
>>> y_true = torch.randint(0,103,(1,1,2,5))
>>> y_pred = torch.zeros(1,103,2,5).scatter(1, y_true, value=1)

You can see for yourself with one of the provided methods:

the builtin function nn.functional.cross_entropy:
>>> F.cross_entropy(y_pred, y_true[:,0])
tensor(3.6513)

manually computing the quantity:
>>> logp = F.log_softmax(y_pred, 1)
>>> -logp.gather(1, y_true).mean()
tensor(3.6513)

analytical result:
>>> log(102 + e) - 1
3.6513


Instruction: Why GPU is much slower than cpu in google colab?

Input:
I'm training a RNN on google colab and this is my first time using gpu to train a neural network. From my point of view, GPU should be much faster than cpu, and changing device from cpu to gpu only need to add .to('cuda') in the definition of model/loss/variable and set google colab 'running on gpu'.
When I train it on cpu, the average speed is 650 iteration/s
Training on cpu in google colab
But when I train it on gpu, the average speed is only 340 iterations/s, only half of the cpu
Training on gpu in google colab
and this happened on every epoch
Here is my code.
def train(num_epoch = 30,len_vocab = 1, num_hidden=256,embedding_dim = 8,batch_size = 100):
    data = get_data()

    model = MyRNN(len_vocab,num_hidden,embedding_dim).to('cuda') #here
    if os.path.exists('QingBinLi'):
        model.load_state_dict(torch.load('QingBinLi'))

    criterion = nn.MSELoss().to('cuda')   #here
    optimizer = torch.optim.Adam(model.parameters(), lr=0.1, weight_decay=1e-5)
    loss_for_draw = []
    model.train()
    data = data.detach().to('cuda') #here

    for epoch in range(num_epoch+1):

        h = torch.randn(1,batch_size,num_hidden).to('cuda')  #here
        loss_average = 0
        for i in tqdm(range(data.shape[-2] -batch_size)):
            optimizer.zero_grad()
            pre,h = model(data[:,:,i:i+batch_size,:].squeeze(0) ,h)
            h = h.detach()
            pre = pre.unsqueeze(0).unsqueeze(0)
            loss = criterion(pre, data[:,:,i+1:i+1+batch_size,:].squeeze(0))
            loss_average += loss.item()
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), max_norm=10)
            optimizer.step()

        loss_for_draw.append(loss_average/(data.shape[-2] -batch_size))
        torch.save(model.state_dict(), 'QingBinLi')
        print(f'now epoch:{epoch}, loss = {loss_for_draw[-1]}')

    return loss_for_draw

I just add '.to('cuda')' there when I try to run it on gpu.
So, why it's much slower when I running my code on gpu? Maybe I should modify more code?

Output:
My brother says that when the tensor is very big, such as 1 million dimension, gpu can be faster than cpu, otherwise we don't even need parallel computing because computing are not mainly on tensor multiply, but on copy tensors and other things like that.
My RNN has about 256x256+256x8 parameters and batch_size is 100, and the dimention of that is much lower than 1 million. So gpu is much slower.
And, when I change my batch_size to 10000, gpu is 145 iteration/s while cpu is only 15iterations/s. This time gpu is much faster.
A CNN, with stride one, in gpu we can calculate filter_size *image_size * batch_size, about 2,415,919,104 times multiply simultaneously. So in this kind of computing, gpu is much faster.


Instruction: Is there anyway to add same index multiple times in one operation?

Input:
Let's say that I have two tensors value and index, which contain the data and all indexes we need. I want to add one to data in value with the corresponding index. If an index shows k times in tensor index, then this data should be added by k, instead of one.
Here's an example:
value = torch.zeros(3) # [0, 0, 0]
index = torch.zeros(10).long() #[0,0,0,0,0,0,0,0,0,0]
ret = some_func(value, index) # [10, 0, 0]

I know that using a for loop to go through all indexes in index could solve the problem, but I just wanna ask if there is more elegant way?

Output:
One way you can do this is with scatter_add:
In [54]: value = torch.zeros(3)

In [55]: index = torch.tensor([0, 0, 1, 0, 2, 2, 1, 1, 1])

In [56]: value.scatter_add(0, index, torch.ones_like(index, dtype=value.dtype))
Out[56]: tensor([3., 4., 2.])

You can use scatter_add_ to operate on value in place.

You might find it more efficient to use bincount():
In [63]: index = torch.tensor([0, 0, 1, 0, 2, 2, 1, 1, 1])

In [64]: counts = index.bincount(minlength=value.shape[0])

In [65]: counts
Out[65]: tensor([3, 4, 2])

If in your actual problem, value is initialized with 0s, then you're done--just use counts as the result.  Otherwise, add counts to value.


Instruction: serving pytorch model using flask

Input:
If multiple clients make a request to one server(pytorch model), how does the problem arise in the multithread of the flask or elsewhere?
As far as I know, if running the pytorch model with multithreading using flask, the data will get tangled. If I’m wrong, please correct it.

Output:
Use a multithreaded approach for Flask as well, the request will get throttled if you just initiate a single threaded usual Flask server.
Use Gunicorn: https://gunicorn.org/ to start your Flask server, with a good number of threads, passed as a parameter value to Gunicorn.


Instruction: using random_split() in python to split the Trainset to train and validation

Input:
train_dataset = torchvision.datasets.FashionMNIST(data_dir, train=True, download=True)
test_dataset  = torchvision.datasets.FashionMNIST(data_dir, train=False, download=True)

using two above lines I load the Mnist dataset and then transfered them to Tensor and Dataloader using below lines of code
tr =transforms.Compose([transforms.ToTensor(),])
train_dataset.transform = tr
test_dataset.transform = tr
train_dataloader = DataLoader(train_dataset, batch_size=256, shuffle=True)
test_dataloader = DataLoader(test_dataset, batch_size=256, shuffle=False)

and then by using the for loop such as bellow I iterate over the data and do training model in pytorch.
for i in train_dataloader:

But when I split the training data into two parts using random_split I get error using the for loop
train_dataset, val_dataset = random_split(train_dataset, (50000, 10000))

train_dataset.transform = tr
test_dataset.transform = tr
val_dataset.transform = tr

train_dataloader = DataLoader(train_dataset, batch_size=256, shuffle=True)
validation_dataloader = DataLoader(val_dataset, batch_size=256, shuffle=True)
test_dataloader = DataLoader(test_dataset, batch_size=256, shuffle=False)

The error is:
default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

How can solve the issue?

Output:
You should pass transform to your FashionMNIST dataset's constructor directly.
train_dataset = torchvision.datasets.FashionMNIST(data_dir, train=True, download=True, transform=tr)
test_dataset  = torchvision.datasets.FashionMNIST(data_dir, train=False, download=True, transform=tr)


Instruction: Is it possible to run scatter matmul in pytorch?

Input:
Edit: apparently DGL is working on it already: https://github.com/dmlc/dgl/pull/3641
I have several types of embeddings and each one needs its own linear projection. I can solve the problem with a for loop of type:
emb_out = dict()
for ntype in ntypes:
    emb_out[ntype] = self.lin_layer[ntype](emb[ntype])

But ideally, I wanted to do some sort of scatter operation to run it in parallel. Something like:
pytorch_scatter(lin_layers, embeddings, layer_map, reduce='matmul'), where the layer map tells which embedding should go through which layer. If I have 2 types of linear layers and batch_size = 5, then layer_map would be something like [1,0,1,1,0].
Would it be possible to vectorize the for loop in a efficient way, like in pytorch_scatter? Please check below minimal examples.
import torch
import random
import numpy as np

seed = 42
torch.manual_seed(seed)
random.seed(seed)

def matmul_single_embtype(lin_layers, embeddings, layer_map):
   #run single linear layer over all embeddings, irrespective of type
   output_embeddings = torch.matmul(lin_layers[0], embeddings.T).T
   return output_embeddings

def matmul_for_loop(lin_layers, embeddings, layer_map):
   #let each embedding type have its own projection, looping over emb types
   output_embeddings = dict()
   for emb_type in np.unique(layer_map):
       output_embeddings[emb_type] = torch.matmul(lin_layers[emb_type], embeddings[layer_map == emb_type].T).T
   return output_embeddings

def matmul_scatter(lin_layers, embeddings, layer_map):
   #parallelize the for loop by creating a diagonal matrix of lin layers
   #this is very innefficient, because creates a copy of the layer for each embedding, instead of broadcasting
   mapped_lin_layers = [lin_layers[i] for i in layer_map]
   mapped_lin_layers = torch.block_diag(*mapped_lin_layers) #batch_size*inp_size x batch_size*output_size
   embeddings_stacked = embeddings.view(-1,1) #stack all embeddings to multiply the linear block
   output_embeddings = torch.matmul(mapped_lin_layers, embeddings_stacked).view(embeddings.shape)
   return output_embeddings

"""
GENERATE DATA
lin_layers:
   List of matrices of size n_layer x inp_size x output_size
embeddings:
   Matrix of size batch_size x inp_size
layer_map:
   Vector os size batch_size stating which embedding should go thorugh each layer
"""

emb_size = 32
batch_size = 500
emb_types = 20
layer_map = [random.choice(list(range(emb_types))) for i in range(batch_size)]

lin_layers = [torch.arange(emb_size*emb_size, dtype=torch.float32).view(emb_size,emb_size) for i in range(emb_types)]
embeddings = torch.arange(batch_size*emb_size, dtype=torch.float32).view(batch_size,emb_size)
grouped_emb = {i: embeddings[layer_map==i] for i in np.unique(layer_map)} #separate embeddings by embedding type

#Run experiments
%timeit matmul_scatter(lin_layers, embeddings, layer_map)
%timeit matmul_for_loop(lin_layers, embeddings, layer_map)
%timeit matmul_single_embtype(lin_layers, embeddings, layer_map)

>>>>>133 ms ± 2.47 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>>>>1.64 ms ± 14 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
>>>>>31.4 µs ± 805 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Related stackoverflow question: how to vectorize the scatter-matmul operation
Related issue in pytorch: https://github.com/pytorch/pytorch/issues/31942

Output: Just found out that DGL is working on this feature already: https://github.com/dmlc/dgl/pull/3641


Instruction: Pytorch expects each tensor to be equal size

Input:
When running this code: embedding_matrix = torch.stack(embeddings)
I got this error:
RuntimeError: stack expects each tensor to be equal size, but got [7, 768] at entry 0 and [8, 768] at entry 1

I'm trying to get embedding using BERT via:
    split_sent = sent.split()
    tokens_embedding = []
    j = 0
    for full_token in split_sent:
        curr_token = ''
        x = 0
        for i,_ in enumerate(tokenized_sent[1:]):
            token = tokenized_sent[i+j]
            piece_embedding = bert_embedding[i+j]
            if token == full_token and curr_token == '' :
               tokens_embedding.append(piece_embedding)
               j += 1
               break
    sent_embedding = torch.stack(tokens_embedding)
    embeddings.append(sent_embedding)
embedding_matrix = torch.stack(embeddings)

Does anyone know how I can fix this?

Output: As per PyTorch Docs about torch.stack() function, it needs the input tensors in the same shape to stack. I don't know how will you be using the embedding_matrix but either you can add padding to your tensors (which will be a list of zeros at the end till a certain user-defined length and is recommended if you will train with this stacked tensor, refer this tutorial) to make them equidimensional or you can simply use something like torch.cat(data,dim=0).


Instruction: Converting a PyG graph to a NetworkX graph

Input:
I am trying to convert my PyG graph to a NetworkX graph using to_networkx
According to the docs I can optionally pass node and edge attributes as str iterables, in addition to the Data object.
Below are by node and edge attribute lists, with values converted to strings:
Nodes:  ['3.3375725746154785', '2.0086510181427',..., '1.5960148572921753', '3.621992349624634']

Edges:  ['0.9940207804344958', '0.48573804411542043', ..., '0.7245483440145621', '0.24117984598949904']

to_networkx runs fine when I only pass it the Data object. However, when I also pass these attribute lists, I get the below error:
G[u][v][key] = values[key][i]
KeyError: '0.30194718370332896'

I've looked at the source code, but can't make out what it is doing. Could someone please help explain what is wrong with my attribute lists and what I need to change for them to be accepted.
What I can make out is that this error is specifically referring to my edge attributes. If I remove them, I get the following similar error related to the node attributes:
feat_dict.update({key: values[key][i]})
KeyError: '0.0'

How I construct my graph and pass it to to_networkx:
n1 = np.repeat(np.array([0,1,2,3,4,5,6]),5)
n2 = np.array([0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4])
cat = np.stack((n1,n2), axis=0)
e = torch.tensor(cat, dtype=torch.long)
edge_index = e.t().clone().detach()
edge_attr = torch.tensor(np.random.rand(35,1))

x = torch.tensor([[0], [0], [0], [0], [0], [1], [1]], dtype=torch.float)

data = Data(x=x, edge_index=edge_index.t().contiguous(), edge_attr = edge_attr)

Before I pass the node and edge attributes, I do the string conversion to conform to the str iterable requirment:
networkx_node_values = list(map(str, data.x.t()[0].tolist()))
networkx_edge_values = list(map(str, edge_attr.t()[0].tolist()))

networkX_graph = to_networkx(data, node_attrs = networkx_node_values, edge_attrs = networkx_edge_values)

Output:
You need to pass the names of the attributes as a list:
to_networkx(<PyTorchGeometricDataObject>, node_attrs=[<Name of Node Attribute 1>, <Name of Node Attributes 2>, ... ], edge_attr=[<Edge Attribute 1>, ...])

Or in context, based on your given minimal example:
import numpy as np
import torch
from torch_geometric.data import Data
from torch_geometric.utils import to_networkx

n1 = np.repeat(np.array([0,1,2,3,4,5,6]),5)
n2 = np.array([0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4,0,1,2,3,4])
cat = np.stack((n1,n2), axis=0)
e = torch.tensor(cat, dtype=torch.long)
edge_index = e.t().clone().detach()
edge_attr = torch.tensor(np.random.rand(35,1))

x = torch.tensor([[0], [0], [0], [0], [0], [1], [1]], dtype=torch.float)

data = Data(x=x, edge_index=edge_index.t().contiguous(), edge_attr = edge_attr)
print(data)
# Data(edge_attr=[35, 1], edge_index=[2, 35], x=[7, 1])

networkX_graph = to_networkx(data, node_attrs=["x"], edge_attrs=["edge_attr"])

print(networkX_graph.nodes(data=True))
# [(0, {'x': 0.0}), (1, {'x': 0.0}),...
print(networkX_graph.edges(data=True))
# [(0, 0, {'edge_attr': 0.3412137594357493}), ...


Instruction: Clipping a tensor if below certain treshold

Input:
I have a tensor with dim n x 3, containing n 3d vectors. I want to compute a new tensor with dim n x 3. If the norm of a vector drops below a certain treshold I want to set it to the zero vector and get a new tensor which contains the index positions of the changed vectors.
Example: tensor([1, 2, 3], [0, 1, 1], [4, 5, 6], ...) would lead to tensor([1, 2, 3], [0, 0, 0], [4, 5, 6], ...) with tensor([1]) if the treshold was set to 1.5.
How can I achieve that without using a loop? Thanks for your help.

Output:
Simply do a[vector_norm(a,dim=1) < thr] = 0, where thr is your threshold. Here's a demonstration.
import torch
from torch.linalg import vector_norm

n = 10
a = torch.rand(10,3)

print('a before:',a)
thr = 1
ind = vector_norm(a,dim=1) < thr
a[ind] = 0
print('a after:',a)
print('list of indices',ind.nonzero())

Result of an example run:
a before: tensor([[0.0708, 0.7559, 0.3974],
        [0.2969, 0.0974, 0.8652],
        [0.8074, 0.8180, 0.2432],
        [0.9006, 0.2447, 0.1602],
        [0.6289, 0.1976, 0.8543],
        [0.2109, 0.7539, 0.6334],
        [0.9100, 0.2514, 0.2314],
        [0.6657, 0.1940, 0.6565],
        [0.4577, 0.8439, 0.5681],
        [0.5566, 0.9979, 0.1468]])
a after: tensor([[0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.8074, 0.8180, 0.2432],
        [0.0000, 0.0000, 0.0000],
        [0.6289, 0.1976, 0.8543],
        [0.2109, 0.7539, 0.6334],
        [0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000],
        [0.4577, 0.8439, 0.5681],
        [0.5566, 0.9979, 0.1468]])
list of indices tensor([[0],
        [1],
        [3],
        [6],
        [7]])


Instruction: How to use 'same' padding for maxpool1d

Input: Tensorflow tf.keras.layers.MaxPool1D has the option to set padding='same' to make the input shape the same as the output shape. Is there something equivalent for torch.nn.MaxPool1d ? I see that torch.nn.Conv1d has the option to set padding='same', but this option seems to be missing from maxpool. What is the current workaround for this?

Output:
Note that the output of the keras version is only really the same shape as the input whenever you use it with stride and dilation set to 1, so I'll assume the same parameters in this answer.
For any uneven kernel size, this is quite easily achievable in PyTorch by setting the padding to (kernel_size - 1)/2.
For an even kernel size, both sides of the input need to be padded by a different amount, and this seems not possible in the current implementation of MaxPool1d.


Instruction: Loading a pretrained model in PyTorch, error:object not callable

Input:
I am trying to load the Efficientnet-b6 weights using PyTorch and Fastai:
PATH = '../input/EffnetB6/efficientnet_b6.pth'
model = torch.load(PATH)

The above model is part of another model:
class EARUnet(nn.Module):
    def __init__(self, pretrained_net, out_ch=1):
        super(EARUnet, self).__init__()
        # print("EfficientUnet_git_b6_res")
        self.pretrained_net = pretrained_net
        .
        .

When I run:
net = EARUnet(model,1)
learn.fit_flat_cos(10)

I get this error:
TypeError: 'collections.OrderedDict' object is not callable

What is the model's format requested?

Output:
Given the restrained context, I suspect that the problem resides in model, probably containing an OrderedDict of the EfficientNet model state dict, while the EARUnet expects the EfficientNet nn.Module.
You should instead, try something like:
eff_net = EfficientNetB6()
eff_net_state_dict = torch.load(PATH)
eff_net.load_state_dict(eff_net_state_dict)

net = EARUnet(model, 1)

Have a look at this page for more details.


Instruction: LSTM, multi-variate, multi-feature in pytorch

Input:
I'm having trouble understanding the format of data for an LSTM in pytorch.    Lets say i have a CSV file with 4 features, laid out in timestamps one after the other ( a classic time series)
time1 feature1 feature2 feature3 feature4
time2 feature1 feature2 feature3 feature4
time3 feature1 feature2 feature3 feature4
time4 feature1 feature2 feature3 feature4,  label

However, this entire set of 4 sequences only has a single label.  The thing we're trying to classify started at time1, but we don't know how to label it until time 4.
My question is, can a typical pytorch LSTM support this? All of the tutorials i've read, watched, walked through, involve looking at a time sequence of a single feature, or a word model, which is still a dataset with a single dimension.
If it can support it, does the data need to be flattened in some way?
Pytorch's LSTM reference states:
input: tensor of shape (L,N,Hin)(L, N, H_{in})(L,N,Hin) when batch_first=False or (N,L,Hin)(N, L, H_{in})(N,L,Hin) when batch_first=True containing the features of the input sequence. The input can also be a packed variable length sequence.
Does this mean that it cannot support any input that contains multiple sequences?  Or is there another name for this?
I'm really lost here, and could use any advice, pointers, help, so on.   Maybe some disambiguation too.
I've posted a couple times here but gotten no responses at all. If this post is misplaced, could someone kindly direct me towards the correct place to post it?
Edit: Following Daniel's advice, do i understand correctly that the four features should be put together like this:
[(feature1, feature2, feature3, feature4, feature1, feature2, feature3, feature4, feature1, feature2, feature3, feature4, feature1, feature2, feature3, feature4), label]  when given to the LSTM?

If that's correct, is the input size (16) in this case?
Finally, I was under the impression that the output of the LSTM Would be the predicted label.  Do I have that wrong?

Output:
As you show, the LSTM layer's input size is (batch_size, Sequence_length, feature_size). This means that the feature is assumed to be a 1D vector.
So to use it in your case you need to stack your four features into one vector (if they are more then 1D themselves then flatten them first) and use that vector as the layer's input.
Regarding the label. It is defiantly supported to have a label only after a few iterations. The LSTM will output a sequence with the same length as the input sequence, but when training the LSTM you can choose to use any part of that sequence in the loss function. In your case you will want to use the last element only.


Instruction: Apply gradient to a tensor in a sparse way in PyTorch

Input:
I have a very large tensor L (millions of elements), from which I gather a relatively small subtensor S (maybe a thousand of elements).
I then apply my model to S, compute loss, and backpropagate to S and to L with the intent to only update selected elements in L. Problem is PyTorch makes L's gradient to be a continuous tensor, so it basically doubles L's memory usage.
Is there an easy way to compute and apply gradient to L without doubling memory usage?
Sample code to illustrate the problem:
import torch
import torch.nn as nn
from torch.nn.parameter import Parameter

net = nn.Sequential(
  nn.Linear(1, 64),
  nn.ReLU(),
  nn.Linear(64,64),
  nn.ReLU(),
  nn.Linear(64, 1))

L = Parameter(torch.zeros([1024*1024*256], dtype=torch.float32))
L.data.uniform_(-1, 1)

indices = torch.randint(high=256*1024*1024, size=[1024])
S = torch.unsqueeze(L[indices], dim=1)

out = net(S)

loss = out.sum()

loss.backward()

print(loss)
g = L.grad
print(g.shape)  # this is huge!

Output:
You don't actually need requires_grad on L as gradients will be computed and applied manually. Instead, set it on S. That will stop backpropagation at S.
Then, you can update the values of L using S.grad and your preferred optimization. Something along these lines
L = torch.zeros([1024*1024*256], dtype=torch.float32)

...

S = torch.unsqueeze(L[indices], dim=1)
S.requires_grad_()

out = net(S)

loss = torch.abs(out).sum()

loss.backward()

with torch.no_grad():
  L[indices] -= learning_rate * torch.squeeze(S.grad)
  S.grad.zero_()


Instruction: Pytorch to ONNX: Could not find an implementation for RandomNormalLike

Input:
I am trying to convert a fairly complex model from pytorch into ONNX. The conversion succeeds without error, but I am encountering this error when loading the model:
Traceback (most recent call last):
  File "/home/***/***/***.py", line 50, in <module>
    main()
  File "/home/***/***/***.py", line 38, in main
    ort_session = ort.InferenceSession(onnx_path, providers=[
  File "/home/***/miniconda3/envs/***/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 324, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/***/miniconda3/envs/***/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 369, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for RandomNormalLike(1) node with name 'RandomNormalLike_598'

I think that the RandomNormalLike node the error is complaining about might correspond to this module I have:
class NoiseInjection(nn.Module):
    def __init__(self):
        super().__init__()

        self.weight = nn.Parameter(torch.zeros(1), requires_grad=True)

    def forward(
        self,
        feat: torch.Tensor,
        noise: Optional[torch.Tensor] = None,
    ) -> torch.Tensor:
        if noise is None:
            batch, _, height, width = feat.shape
            noise = torch.randn(
                batch, 1, height, width,
                dtype=feat.dtype,
                device=feat.device,
            )

        return feat + self.weight * noise

I also created a different implementation, but it leads to the same error: (edit: This version actually works. I made an unrelated mistake elsewhere that mislead me into thinking it did not work)
    def forward(
        self,
        feat: torch.Tensor,
        noise: Optional[torch.Tensor] = None,
    ) -> torch.Tensor:
        if noise is None:
            noise = torch.randn_like(feat[:, 0:1])

        return feat + self.weight * noise

My pytorch and onnx version are as follows:
$ conda list torch
# Name                    Version                   Build  Channel
torch                     1.10.0+cu113             pypi_0    pypi
torchaudio                0.10.0+cu113             pypi_0    pypi
torchvision               0.11.1+cu113             pypi_0    pypi

$ conda list onnx
# Name                    Version                   Build  Channel
onnx                      1.10.2                   pypi_0    pypi
onnxruntime-gpu           1.9.0                    pypi_0    pypi

What can be done to be able to export such a module to onnx and run it successfully?

Output:
From checking online I found a similar issue on GitHub about conv (https://github.com/microsoft/onnxruntime/issues/3130), could be that the types of the parameters used in torch are not compatible with the implementation of RandomNormalLike available in ONNX.
Could you check in netron what's inside the RandomNormalLike node/nodes to see if they comply with the spec: https://github.com/onnx/onnx/blob/main/docs/Operators.md#RandomNormal or https://github.com/onnx/onnx/blob/main/docs/Operators.md#RandomNormalLike
Cheers
EDIT: turns out the RandomNormal node has a type of 10 which corresponds to fp16
While the onnxruntime implementation only supports float and doubles see source code here: https://github.com/microsoft/onnxruntime/blob/24e35fba3217bf33b0e4064bc71d271a61938ba0/onnxruntime/core/providers/cpu/generator/random.cc#L354
Solution here is either to run the whole model in fp32 or ask explicitely RandomNormalLike to use floats or doubles hoping that torch allows mixed computation on fp16 and fp32/fp64 I guess


Instruction: pycave kmeans-gpu disable model trainer outputs from being printed

Input:
This is a code snippet to run KMeans using GPU.
Documentation-link:https://pycave.borchero.com/sites/generated/clustering/kmeans/pycave.clustering.KMeans.html
import torch
from pycave.clustering import KMeans

X = torch.cat([
    torch.randn(1000, 6) - 5,
    torch.randn(1000, 6),
    torch.randn(1000, 6) + 5,
])

estimator = KMeans(num_clusters = 3, trainer_params=dict(gpus=1,
                                                         enable_progress_bar=0,
                                                         max_epochs=100,))
labels = estimator.fit_predict(X).numpy()

pd.value_counts(labels)

The issue is with how to disable the console output from the estimator.
Current Output:
Running initialization...

{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x000002BE21221700>}

Fitting K-Means...

{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x000002BE21221700>}
{'batch_size': 1, 'sampler': None, 'batch_sampler': <pytorch_lightning.overrides.distributed.IndexBatchSamplerWrapper object at 0x000002BE593A55B0>, 'collate_fn': <function collate_tensor at 0x000002BE21221700>, 'shuffle': False, 'drop_last': False}

0    1000
2    1000
1    1000
dtype: int64

Expected Output:
0    1000
2    1000
1    1000
dtype: int64

Info regarding trainer_params parameter
(Optional[Dict[str, Any]]) --
Initialization parameters to use when initializing a PyTorch Lightning trainer. By default, it disables various stdout logs unless PyCave is configured to do verbose logging. Checkpointing and logging are disabled regardless of the log level.

Output:
The dictionaries that are printed should never be there, that's a bug in a dependency. Resolved in the latest build.
As far as the PyCave logs are concerned (Running initialization... and Fitting K-Means...), you can turn them off easily by adding the following:
import logging
from pycave import set_logging_level

set_logging_level(logging.WARNING)

Note that set_logging_level(logging.WARNING) also turns off the progress bar and the model summary automatically so you don't have to set these flags explicitly.


Instruction: Tensorflow: Create the torch.gather() equivalent in tensorflow

Input:
I want to replicate the torch.gather() function in TensorFlow 2.X.
I have a Tensor A (shape: [2, 4, 3]) and a corresponding Index-Tensor I (shape: [2,2,3]).
Using torch.gather() yields the following:
A = torch.tensor([[[10,20,30], [100,200,300], [1000,2000,3000]],
                  [[50,60,70], [500,600,700], [5000,6000,7000]]])
I = torch.tensor([[[0,1,0], [1,2,1]],
                  [[2,1,2], [1,0,1]]])
torch.gather(A, 1, I)

>
tensor([[[10,   200,   30], [100, 2000, 300]],
         [5000, 600, 7000], [500,   60, 700]]])

I have tried using tf.gather(), but this did not yield pytorch-like results. I also tried to play around with tf.gather_nd(), but I could not find a suitable solution.
I found this StackOverflow post, but this seems not to work for me.
Edit:
When using tf.gather_nd(A, I), I get the following result:
tf.gather_nd(A, I)

>
[[100, 6000],
 [  0,   60]]

The result for tf.gather(A, I) is rather lengthy. It has the shape of [2, 2, 3, 4, 3]

Output:
torch.gather and tf.gather_nd work differently and will therefore yield different results when using the same indices tensor (in some cases an error will also be returned). This is what the indices tensor would have to look like to get the same results:
import tensorflow as tf

A = tf.constant([[
                   [10,20,30], [100,200,300], [1000,2000,3000]],
                  [[50,60,70], [500,600,700], [5000,6000,7000]]])
I = tf.constant([[[
                  [0,0,0],
                  [0,1,1],
                  [0,0,2],
                ],[
                  [0,1,0],
                  [0,2,1],
                  [0,1,2],
                ]],
                 [[
                  [1,2,0],
                  [1,1,1],
                  [1,2,2],
                ],
                  [
                  [1,1,0],
                  [1,0,1],
                  [1,1,2],
                ]]])

print(tf.gather_nd(A, I))

tf.Tensor(
[[[  10  200   30]
  [ 100 2000  300]]

 [[5000  600 7000]
  [ 500   60  700]]], shape=(2, 2, 3), dtype=int32)

So, the question is actually how are you calculating your indices or are they always hard-coded? Also, check out this post on the differences of the two operations.
As for the post you linked that didn't work for you, you just need to cast the indices and everything should be fine:
def torch_gather(x, indices, gather_axis):

    all_indices = tf.where(tf.fill(indices.shape, True))
    gather_locations = tf.reshape(indices, [indices.shape.num_elements()])

    gather_indices = []
    for axis in range(len(indices.shape)):
        if axis == gather_axis:
            gather_indices.append(tf.cast(gather_locations, dtype=tf.int64))
        else:
            gather_indices.append(tf.cast(all_indices[:, axis], dtype=tf.int64))

    gather_indices = tf.stack(gather_indices, axis=-1)
    gathered = tf.gather_nd(x, gather_indices)
    reshaped = tf.reshape(gathered, indices.shape)
    return reshaped

I = tf.constant([[[0,1,0], [1,2,1]],
                  [[2,1,2], [1,0,1]]])
A = tf.constant([[
                   [10,20,30], [100,200,300], [1000,2000,3000]],
                  [[50,60,70], [500,600,700], [5000,6000,7000]]])
print(torch_gather(A, I, 1))

tf.Tensor(
[[[  10  200   30]
  [ 100 2000  300]]

 [[5000  600 7000]
  [ 500   60  700]]], shape=(2, 2, 3), dtype=int32)


Instruction: In the PyTorch Distributed Data Parallel (DDP) tutorial, how does `setup` know it's rank?

Input:
For the tutorial Getting Started with Distributed Data Parallel
How does setup() function knows the rank when mp.spawn() doesn't pass the rank?
def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)
    .......

def run_demo(demo_fn, world_size):
    mp.spawn(demo_fn,
             args=(world_size,),
             nprocs=world_size,
             join=True)

if __name__ == "__main__":
    n_gpus = torch.cuda.device_count()
    assert n_gpus >= 2, f"Requires at least 2 GPUs to run, but got {n_gpus}"
    world_size = n_gpus
    run_demo(demo_basic, world_size)

Output:
mp.spawn does pass the rank to the function it calls.
From the torch.multiprocessing.spawn docs

torch.multiprocessing.spawn(fn, args=(), nprocs=1, join=True, daemon=False, start_method='spawn')
...

fn (function) -
Function is called as the entrypoint of the spawned process. This
function must be defined at the top level of a module so it can be
pickled and spawned. This is a requirement imposed by multiprocessing. The function is called as fn(i, *args), where i is the process index
and args is the passed through tuple of arguments.

So when spawn invokes fn it passes it the process index as the first argument.


Instruction: How to Suppress "Using bos_token, but it is not set yet..." in HuggingFace T5 Tokenizer

Input:
I'd like to turn off the warning that huggingface is generating when I use unique_no_split_tokens
In[2]   tokenizer = T5Tokenizer.from_pretrained("t5-base")
In[3]   tokenizer(" ".join([f"<extra_id_{n}>" for n in range(1,101)]), return_tensors="pt").input_ids.size()
Out[3]: torch.Size([1, 100])
    Using bos_token, but it is not set yet.
    Using cls_token, but it is not set yet.
    Using mask_token, but it is not set yet.
    Using sep_token, but it is not set yet.

Anyone know how to do this?

Output:
This solution worked for me:
tokenizer.add_tokens([f"_{n}" for n in range(1,100)], special_tokens=True)
model.resize_token_embeddings(len(tokenizer))
tokenizer.save_pretrained('pathToExtendedTokenizer/')
tokenizer = T5Tokenizer.from_pretrained("pathToExtendedTokenizer/")


Instruction: Retrieve the PyTorch model from a PyTorch lightning model

Input:
I have trained a PyTorch lightning model that looks like this:
In [16]: MLP
Out[16]:
DecoderMLP(
  (loss): RMSE()
  (logging_metrics): ModuleList(
    (0): SMAPE()
    (1): MAE()
    (2): RMSE()
    (3): MAPE()
    (4): MASE()
  )
  (input_embeddings): MultiEmbedding(
    (embeddings): ModuleDict(
      (LCLid): Embedding(5, 4)
      (sun): Embedding(5, 4)
      (day_of_week): Embedding(7, 5)
      (month): Embedding(12, 6)
      (year): Embedding(3, 3)
      (holidays): Embedding(2, 1)
      (BusinessDay): Embedding(2, 1)
      (day): Embedding(31, 11)
      (hour): Embedding(24, 9)
    )
  )
  (mlp): FullyConnectedModule(
    (sequential): Sequential(
      (0): Linear(in_features=60, out_features=435, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.13371112461182535, inplace=False)
      (3): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (4): Linear(in_features=435, out_features=435, bias=True)
      (5): ReLU()
      (6): Dropout(p=0.13371112461182535, inplace=False)
      (7): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (8): Linear(in_features=435, out_features=435, bias=True)
      (9): ReLU()
      (10): Dropout(p=0.13371112461182535, inplace=False)
      (11): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (12): Linear(in_features=435, out_features=435, bias=True)
      (13): ReLU()
      (14): Dropout(p=0.13371112461182535, inplace=False)
      (15): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (16): Linear(in_features=435, out_features=435, bias=True)
      (17): ReLU()
      (18): Dropout(p=0.13371112461182535, inplace=False)
      (19): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (20): Linear(in_features=435, out_features=435, bias=True)
      (21): ReLU()
      (22): Dropout(p=0.13371112461182535, inplace=False)
      (23): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (24): Linear(in_features=435, out_features=435, bias=True)
      (25): ReLU()
      (26): Dropout(p=0.13371112461182535, inplace=False)
      (27): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (28): Linear(in_features=435, out_features=435, bias=True)
      (29): ReLU()
      (30): Dropout(p=0.13371112461182535, inplace=False)
      (31): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (32): Linear(in_features=435, out_features=435, bias=True)
      (33): ReLU()
      (34): Dropout(p=0.13371112461182535, inplace=False)
      (35): LayerNorm((435,), eps=1e-05, elementwise_affine=True)
      (36): Linear(in_features=435, out_features=1, bias=True)
    )
  )
)

I need the corresponding PyTorch model to use in one of my other applications.
Is there a simple way to do that?
I thought of saving the checkpoint but then I don't know how to do it.
Can you please help?
Thanks

Output:
You can manually save the weights of the torch.nn.Modules in the LightningModule. Something like:
trainer.fit(model, trainloader, valloader)

torch.save(
    model.input_embeddings.state_dict(),
    "input_embeddings.pt"
)
torch.save(model.mlp.state_dict(), "mlp.pt")

Then to load without needing Lightning:
# create the "blank" networks like they
# were created in the Lightning Module
input_embeddings = MultiEmbedding(...)
mlp = FullyConnectedModule(...)

# Load the models for inference
input_embeddings.load_state_dict(
    torch.load("input_embeddings.pt")
)
input_embeddings.eval()

mlp.load_state_dict(
    torch.load("mlp.pt")
)
mlp.eval()

For more information about saving and loading PyTorch Modules see Saving and Loading Models: Saving & Loading Model for Inference in the PyTorch documentation.
Since Lightning automatically saves checkpoints to disk (check the lightning_logs folder if using the default Tensorboard logger), you can also load a pretrained LightningModule and then save the state dicts without needing to repeat all the training. Instead of calling trainer.fit in the previous code, try
model = DecoderMLP.load_from_checkpoint("path/to/checkpoint.ckpt")


Instruction: PyTorch - Neural Network - Output single scalar value

Input:
Let's say we have the following neural network in PyTorch
seq_model = nn.Sequential(
nn.Linear(1, 13),
nn.Tanh(),
nn.Linear(13, 1))

With the following input tensor
input = torch.tensor([1.0, 1.0, 5.0], dtype=torch.float32).unsqueeze(1)

I can run forward through the net and get
seq_model(input)

tensor([[-0.0165],
        [-0.0165],
        [-0.2289]], grad_fn=<TanhBackward0>)

Probably I also can get a single scalar value as an output, but I'm not sure how.
Thank you. I'm trying to use such an network for reinforcment learning, and use it
as an value function approximator for game board state evaluation.

Output:
The first dimension of input represents the number of observations in your minibatch (3), the second dimension represents instead the number of features (1).
If you want to forward a single 3d input, the network must be modified (nn.Linear(1, 13) becomes nn.Linear(3, 13)), and you must remove unsqueeze(1) on input. Otherwise, you can merge the three outputs by using a loss to compute a single scalar from them.


Instruction: Difference between the calculation of the training loss and validation loss using pytorch

Input:
I wanna use the following code of this traditional image classification problem for my regression problem. The code can be found here:
GeeksforGeeks-Training Neural Networks with Validation using Pytorch
class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        self.fc1 = nn.Linear(28*28, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(1,-1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Network()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

epochs = 5

for e in range(epochs):
    train_loss = 0.0
    model.train()     # Optional when not using Model Specific layer
    for data, labels in trainloader:
        if torch.cuda.is_available():
            data, labels = data.cuda(), labels.cuda()

        optimizer.zero_grad()
        target = model(data)
        loss = criterion(target,labels)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()

    valid_loss = 0.0
    model.eval()     # Optional when not using Model Specific layer
    for data, labels in validloader:
        if torch.cuda.is_available():
            data, labels = data.cuda(), labels.cuda()

        target = model(data)
        loss = criterion(target,labels)
        valid_loss = loss.item() * data.size(0)

print(f'Epoch {e+1} \t\t Training Loss: {train_loss / len(trainloader)} \t\t Validation Loss: {valid_loss / len(validloader)}')

I can understand why the training loss is summed up and then divided by the length of the training data in this example, but I can't get why the validation loss is also not summed up and divided by the length. If I understand correctly, the validation loss will be calculated here by using the validation loss of the last batch and then it is multiplied by the length of the batch size.
Is the calulation of the validation loss the correct way to do it? Can I use the code for my regression problem assuming I use regression-specific metrics (e.g. MSE instead of CrossEntropyLoss etc.)?

Output:
Yes, you can use the code for your regression task. The targets of the code example are one-hot vectors or in the MNIST example the numbers 0 to 9, which symbolize the classes. You would make a scalar out of that in the regression case. The loss function, which is the cross-entropy in the example, can be replaced by the MSE in your case.
I assume that the validation loss in this example is only estimated by extrapolating from a single data point to all other data points.
Since data.size represents the batch size, even averaging would only come out with the loss of that single data point.
However, on the web page, the validation loss is calculated over all data points in the validation set, as it should be done.