Instruction: Why conda installs old pytorch with by default with cudatoolkit=11.2

Input:
I am trying to install conda for cudatoolkit=11.2 on google colab using:
conda install pytorch cudatoolkit=11.2 -c pytorch -c nvidia

But why does it install old pytorch=1.0.0 version not something >1.6?
If I try to force install pytorch=1.6, it gives the following error:
UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (cudatoolkit):

  - cudatoolkit=11.2
  - pytorch=1.6 -> cudatoolkit[version='>=10.1,<10.2|>=10.2,<10.3|>=9.2,<9.3']

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package setuptools conflicts for:
setuptools
python=3.7 -> pip -> setuptools
conda[version='>=4.10.3'] -> setuptools[version='>=31.0.1']
wheel -> setuptools
pip -> setuptools
...

EDIT based on the answer:
When I try to use conda install -c conda-forge pytorch cudatoolkit=11.2, it gives the following error.
PackagesNotFoundError: The following packages are not available from current channels:

  - cudatoolkit=11.2
  - __glibc[version='>=2.17,<3.0.a0']

Current channels:

  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/linux-64
  - https://repo.anaconda.com/pkgs/pro/noarch

Output:
The pytorch channel doesn't yet have any pytorch builds compatible with cudatoolkit=11.2. What the solver finds is an old version of PyTorch where they did not have proper upper bounds on the dependency version.
If you insist on cudatoolkit=11.2, then you'll need to stick to the Conda Forge stack:
conda install -c conda-forge pytorch cudatoolkit=11.2

Otherwise, if you want the official PyTorch builds, then they build up to v11.1 compatible versions:
conda install -c pytorch -c conda-forge pytorch cudatoolkit=11.1


Instruction: Pytorch geometric: how to explain the input in the below code-snippet?

Input:
I am reading PyTorch geometric documentation at https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html
On this page, there is a code snippet:
import torch
from torch_geometric.data import Data

edge_index = torch.tensor([[0, 1, 1, 2],
                           [1, 0, 2, 1]], dtype=torch.long)
x = torch.tensor([[-1], [0], [1]], dtype=torch.float)

data = Data(x=x, edge_index=edge_index)

The output of the last line from the above code snippet is:
Data(edge_index=[2, 4], x=[3, 1])

How are the edge_index 2 and 4? If I understand correctly there are four edges being defined with an index starting from 0. Is this assumption wrong? Also, what does x =[3, 1] mean?
Data is a class, so I won't expect it to return anything. Class definition is here: https://pytorch-geometric.readthedocs.io/en/latest/modules/data.html . I read the documentation. x should be Node feature matrix and edge_index should be graph connectivity. But I don't understand the console output that I cross-checked in the jupyter notebook.

Output: Okay, I think I have got the understanding of the output Data(edge_index=[2, 4], x=[3, 1]). Here [2,4] are dimensions of edge_index and [3,1] are dimensions of x. But please, anyone correct me if I am wrong.


Instruction: This torch project keep telling me "Expected 2 or more dimensions (got 1)"

Input:
I was trying to make my own neural network using PyTorch. I do not understand why my code is not working properly.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optimizers
import numpy as np
from tqdm import tqdm
import os
import hashlib

# Only for the first time
MAKE_DATA = False

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2, 100)
        self.fc2 = nn.Linear(100, 200)
        self.fc3 = nn.Linear(200, 200)
        self.fc4 = nn.Linear(200, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.relu(x)

def make_numirical(data):
    data = str(data)
    data = data.encode()
    data = hashlib.md5(data).hexdigest()
    str1 = ''
    for c in data:
        if not (c >= '0' and c <= '9'):
            c = ord(c)
            if c > 10:
                c /= 10
        str1 += str(int(c))
    return int(str1[:20])

def make_train_data():
    HITS = 'Songs_DB/Hits'
    NOHITS = 'Songs_DB/NoHits'
    hits_count = 0
    no_hits_count = 0
    LABELS = {HITS: 0, NOHITS: 0}
    training_data = []
    i = 0
    for label in LABELS:
        for f in tqdm(os.listdir(label)):
            try:
                path = os.path.join(label, f)
                with open(path, 'rb') as file:
                    data = file.read()
                    file.close()
                data = make_numirical(data)
                data = int(data)
                training_data.append([np.array([data]), np.eye(2)[i]])

                if label == HITS:
                    hits_count += 1
                else:
                    no_hits_count += 1
            except:
                pass
        i += 1

        np.random.shuffle(training_data)
        np.save('training_data.npy', training_data)
        print(hits_count)
        print(no_hits_count)

if MAKE_DATA:
    make_train_data()

model = Model()

# 1 = brown, 0 = not brown, 1 = cat, 0 = dog.

Xs = torch.Tensor([[0, 1], [1, 1], [1, 0], [1, 1], [1, 1], [0, 1], [1, 1], [0, 0], [1, 0]])
ys = torch.Tensor([[1], [0], [0], [1], [0], [1], [0], [1], [1]])

i = 0
for x in Xs:
    output = model(x)
    print(output)
    loss = F.nll_loss(output, ys[i])

print(loss)

The program keeps giving me this error:

Expected 2 or more dimensions (got 1)

Can anyone explain what is wrong with my code?

Output:
The tensor you use as the dataset, Xs is shaped (n, 2). So when looping over it each element x ends up as a 1D tensor shaped (2,). However, your module expects a batched tensor as input, i.e. here a 2D tensor shaped (n, 2), just like Xs. You have two possible options, either use a data loader and divide your dataset into batches, or unsqueeze your input x to make it two dimensional shaped (1, 2).

Using a TensorDataset and wrapping it with a DataLoader:
>>> dataset = TensorDataset(Xs, ys)
>>> dataloader = Dataloader(dataset, batch_size=4)

Then iterating over dataloader will return batches of fours (inputs and corresponding labels):
>>> for x, y in dataloader:
...     output = model(x)
...     loss = F.nll_loss(output, y)

TensorDataset and Dataloader are both imported from torch.utils.data.

Or use torch.Tensor.unsqueeze on x to add one extra dimension:
>>> for x, y in zip(Xs, ys):
...     output = model(x.unsqueeze())
...     loss = F.nll_loss(output, y)

Alternatively, you can do x[None] which has the same effect.


Instruction: Expected object of scalar type Long but got scalar type Int for argument #2 in loss function

Input:
I have encountered the following error:
 RuntimeError                              Traceback (most recent call last)
    <ipython-input-42-276f5444b449> in <module>
    ----> 1 train_epocs(model, optimizer, train_dl, valid_dl, epochs=15)

    <ipython-input-39-6f4616cc5f25> in train_epocs(model, optimizer, train_dl, val_dl, epochs, C)
         11             y_bb = y_bb.cuda().float()
         12             out_class, out_bb = model(x)
    ---> 13             loss_class = F.cross_entropy(out_class, y_class, reduction="sum")
         14             loss_bb = F.l1_loss(out_bb, y_bb, reduction="none").sum(1)
         15             loss_bb = loss_bb.sum()

    ~\anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
       2822     if size_average is not None or reduce is not None:
       2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
    -> 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
       2825
       2826

    RuntimeError: Expected object of scalar type Long but got scalar type Int for argument #2 'target' in call to _thnn_nll_loss_forward

I have been training the network with the following set-up. We have 26 classes. The code is adopted from https://jovian.ai/ranerajesh/road-signs-bounding-box-prediction/v/10 I have my own custom dataset that has been structured in a way required for this code to run. However, I have encountered a RunTime error.
    def normalize(im):
        """Normalizes images with Imagenet stats."""
        imagenet_stats = np.array([[0.485, 0.456, 0.406], [0.229, 0.224, 0.225]])
        return (im - imagenet_stats[0])/imagenet_stats[1]

    class RoadDataset(Dataset):
        def __init__(self, paths, bb, y, transforms=False):
            self.transforms = transforms
            self.paths = paths.values
            self.bb = bb.values
            self.y = y.values
        def __len__(self):
            return len(self.paths)

        def __getitem__(self, idx):
            path = self.paths[idx]
            print(path)
            y_class = self.y[idx]
            x, y_bb = transformsXY(path, self.bb[idx], self.transforms)
            x = normalize(x)
            x = np.rollaxis(x, 2)
            return x, y_class, y_bb
train_ds = RoadDataset(X_train['new_path'],X_train['new_bb'] ,y_train, transforms=True)
valid_ds = RoadDataset(X_val['new_path'],X_val['new_bb'],y_val)
batch_size = 2
train_dl = DataLoader(train_ds, batch_size=batch_size, shuffle=True)
valid_dl = DataLoader(valid_ds, batch_size=batch_size)

class BB_model(nn.Module):
    def __init__(self):
        super(BB_model, self).__init__()
        resnet = models.resnet34(pretrained=True)
        layers = list(resnet.children())[:8]
        self.features1 = nn.Sequential(*layers[:6])
        self.features2 = nn.Sequential(*layers[6:])
        self.classifier = nn.Sequential(nn.BatchNorm1d(512), nn.Linear(512, 26))
        self.bb = nn.Sequential(nn.BatchNorm1d(512), nn.Linear(512, 26))

    def forward(self, x):
        x = self.features1(x)
        x = self.features2(x)
        x = F.relu(x)
        x = nn.AdaptiveAvgPool2d((1,1))(x)
        x = x.view(x.shape[0], -1)
        return self.classifier(x), self.bb(x)

def update_optimizer(optimizer, lr):
    for i, param_group in enumerate(optimizer.param_groups):
        param_group["lr"] = lr

def train_epocs(model, optimizer, train_dl, val_dl, epochs=10,C=1000):
    idx = 0
    for i in range(epochs):
        model.train()
        total = 0
        sum_loss = 0
        for x, y_class, y_bb in train_dl:
            batch = y_class.shape[0]
            x = x.cuda().float()
            y_class = y_class.cuda()
            y_bb = y_bb.cuda().float()
            out_class, out_bb = model(x)
            loss_class = F.cross_entropy(out_class, y_class, reduction="sum")
            loss_bb = F.l1_loss(out_bb, y_bb, reduction="none").sum(1)
            loss_bb = loss_bb.sum()
            loss = loss_class + loss_bb/C
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            idx += 1
            total += batch
            sum_loss += loss.item()
        train_loss = sum_loss/total
        val_loss, val_acc = val_metrics(model, valid_dl, C)
        print("train_loss %.3f val_loss %.3f val_acc %.3f" % (train_loss, val_loss, val_acc))
    return sum_loss/total

def val_metrics(model, valid_dl, C=1000):
    model.eval()
    total = 0
    sum_loss = 0
    correct = 0
    for x, y_class, y_bb in valid_dl:
        batch = y_class.shape[0]
        x = x.cuda().float()
        y_class = y_class.cuda()
        y_bb = y_bb.cuda().float()
        out_class, out_bb = model(x)
        loss_class = F.cross_entropy(out_class, y_class, reduction="sum")
        loss_bb = F.l1_loss(out_bb, y_bb, reduction="none").sum(1)
        loss_bb = loss_bb.sum()
        loss = loss_class + loss_bb/C
        _, pred = torch.max(out_class, 1)
        correct += pred.eq(y_class).sum().item()
        sum_loss += loss.item()
        total += batch
    return sum_loss/total, correct/total

model = BB_model().cuda()
parameters = filter(lambda p: p.requires_grad, model.parameters())
optimizer = torch.optim.Adam(parameters, lr=0.006)

train_epocs(model, optimizer, train_dl, valid_dl, epochs=15)

Output:
loss_class = F.cross_entropy(out_class, y_class, reduction="sum")

In the line above, y_class is target of your out_class (for model predictions). The output from model is Long and your y_class has a Float type. So you need to change y_class's type to Long by:
y_class = y_class.long()
loss_class = F.cross_entropy(out_class, y_class, reduction="sum")


Instruction: function for Ordinal Pooling Neural network

Input:
please I want to create a function that computes the Ordinal Pooling neural network like the following figure:

this is my function :
def Ordinal_Pooling_NN(x):
  wights = torch.tensor([0.6, 0.25, 0.10, 0.05])
  top = torch.topk(x, 4, dim = 1)
  wights = wights.repeat(x.shape[0], 1)
  result = torch.sum(wights * (top.values), dim = 1 )
  return result

but as a result, I get the following error:
<ipython-input-112-ddf99c812d56> in Ordinal_Pooling_NN(x)
      9     top = torch.topk(x, 4, dim = 1)
     10     wights = wights.repeat(x.shape[0], 1)
---> 11     result = torch.sum(wights * (top.values), dim = 1 )
     12     return result

RuntimeError: The size of tensor a (4) must match the size of tensor b (16) at non-singleton dimension 2

Output:
Your implementation is actually correct, I believe you did not feed the function with a 2D tensor, the input must have a batch axis. For instance, the code below will run:
>>> Ordinal_Pooling_NN(torch.tensor([[1.9, 0.4, 1.3, 0.8]]))
tensor([1.5650])

Do note you are not required to repeat the weights tensor, it will be broadcasted automatically when computing the point-wise multiplication. You only need the following:
def Ordinal_Pooling_NN(x):
  w = torch.tensor([0.6, 0.25, 0.10, 0.05])
  top = torch.topk(x, k=4, dim=1)
  result = torch.sum(w*top.values, dim=1)
  return result


Instruction: How to get indices of tensors of same value in a 2-d tensor?

Input:
As described in title, given a 2-d tensor, let's say:
tensor([
    [0, 1, 0, 1], # A
    [1, 1, 0, 1], # B
    [1, 0, 0, 1], # C
    [0, 1, 0, 1], # D
    [1, 1, 0, 1], # E
    [1, 1, 0, 1]  # F
])

That's easy enough to tell that "A and D", "B, E and F" are two groups of tensors,
that are of same value(that means A == D and B == E == F).
So my question is:

How to get indices of those groups?

Details:

Input: tensor above
Output: (0, 3), (1, 4, 5)

Output:
A solution using PyTorch functions:
import torch

x = torch.tensor([
    [0, 1, 0, 1], # A
    [1, 1, 0, 1], # B
    [1, 0, 0, 1], # C
    [0, 1, 0, 1], # D
    [1, 1, 0, 1], # E
    [1, 1, 0, 1]  # F
])

_, inv, counts = torch.unique(x, dim=0, return_inverse=True, return_counts=True)
print([tuple(torch.where(inv == i)[0].tolist()) for i, c, in enumerate(counts) if counts[i] > 1])
# > [(0, 3), (1, 4, 5)]


Instruction: pytorch hook function is not executed

Input: May I know why the pytorch hook function does not work ?

Output:
You might want to use torch.nn.Module.named_modules instead of torch.nn.Module.named_children. The latter will only return immediate child modules. In your case graph's immediate child is cells, so you won't be looping over modules inside of cells, i.e. the layers defined inside the ModuleList.
Either use named_modules:
for name, module in graph.named_modules()
    pass

Or use named_children on graph.cells directly:
for name, module in graph.cells.named_children():
    pass

However, the latter alternative won't scale if you ever decide to add additional child modules to Graph.


Instruction: Create a simple PyTorch neural network with a normalized weights

Input: I want to create a simple PyTorch neural network with the sum of its weights equal to 1. To understand my question here is a to give an example:

Output:
You can simply normalize by the sum of all initialized weights:
>>> layer = nn.Linear(4, 1, bias=False)
>>> layer.weight
Parameter containing:
tensor([[-0.2565,  0.4753, -0.1129,  0.2327]], requires_grad=True)

Normalize layer.weight:
>>> layer.weight.data /= layer.weight.data.sum()

Then:
>>> layer.weight
Parameter containing:
tensor([[-0.7573,  1.4034, -0.3333,  0.6872]], requires_grad=True)


Instruction: How to write a forward hook function for nn.Transformer in pytorch?

Input:
I have learnt that forward hook function has the form as hook_fn(m,x,y). m refers to model, x refers to input and y refers to output. I want to write a forward hook function for nn.Transformer.
However there are to input for transformer layer which is src and tgt. For example, >>> out = transformer_model(src, tgt). So how can I differ these inputs?

Output:
Your hook will call your callback function with tuples for x and y. As described in the documentation page of torch.nn.Module.register_forward_hook (it does quite explain the type of x and y though).

The input contains only the positional arguments given to the module.
Keyword arguments won’t be passed to the hooks and only to the
forward. [...].

model = nn.Transformer(nhead=16, num_encoder_layers=12)
src = torch.rand(10, 32, 512)
tgt = torch.rand(20, 32, 512)

Define your callback:
def hook(module, x, y):
    print(f'is tuple={isinstance(x, tuple)} - length={len(x)}')
    src, tgt = x

    print(f'src: {src.shape}')
    print(f'tgt: {tgt.shape}')

Hook to your nn.Module:
>>> model.register_forward_hook(hook)

Do an inference:
>>> out = model(src, tgt)
is tuple=True - length=2
src: torch.Size([10, 32, 512])
tgt: torch.Size([20, 32, 512])


Instruction: Classification with PyTorch is much slower than Tensorflow: 42min vs. 11min

Input:
I have been a Tensorflow user and start to use Pytorch. As a trial, I implemented simple classification tasks with both libraries.
However, PyTorch is much slower than Tensorflow: Pytorch takes 42min while TensorFlow 11min. I referred to PyTorch official Tutorial, and made only little change from it.
Could anyone share some advice for this problem?
Here is the summary what I tried.

environment: Colab Pro+
dataset: Cifar10
classifier: VGG16
optimizer: Adam
loss: crossentropy
batch size: 32

PyTorch
Code:
import torch, torchvision
from torch import nn
from torchvision import transforms, models
from tqdm import tqdm
import time, copy

trans = transforms.Compose([transforms.Resize((224, 224)),
                            transforms.ToTensor(),])

data = {phase: torchvision.datasets.CIFAR10('./', train = (phase=='train'),  transform=trans, download=True) for phase in ['train', 'test']}
dataloaders = {phase: torch.utils.data.DataLoader(data[phase], batch_size=32, shuffle=True) for phase in ['train', 'test']}

def train_model(model, criterion, optimizer, dataloaders, device, num_epochs=5):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'test']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in tqdm(iter(dataloaders[phase])):
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / len(dataloaders[phase])
            epoch_acc = running_corrects.double() / len(dataloaders[phase])

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'test' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = models.vgg16(pretrained=False)
model = model.to(device)

model = train_model(model=model,
                    criterion=nn.CrossEntropyLoss(),
                    optimizer=torch.optim.Adam(model.parameters(), lr=0.001),
                    dataloaders=dataloaders,
                    device=device,
                    )

Result:
Epoch 0/4
----------
  0%|          | 0/1563 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
100%|██████████| 1563/1563 [07:50<00:00,  3.32it/s]
train Loss: 75.5199 Acc: 3.2809
100%|██████████| 313/313 [00:38<00:00,  8.11it/s]
test Loss: 73.7274 Acc: 3.1949

Epoch 1/4
----------
100%|██████████| 1563/1563 [07:50<00:00,  3.33it/s]
train Loss: 73.8162 Acc: 3.2514
100%|██████████| 313/313 [00:38<00:00,  8.13it/s]
test Loss: 73.6114 Acc: 3.1949

Epoch 2/4
----------
100%|██████████| 1563/1563 [07:49<00:00,  3.33it/s]
train Loss: 73.7741 Acc: 3.1369
100%|██████████| 313/313 [00:38<00:00,  8.11it/s]
test Loss: 73.5873 Acc: 3.1949

Epoch 3/4
----------
100%|██████████| 1563/1563 [07:49<00:00,  3.33it/s]
train Loss: 73.7493 Acc: 3.1331
100%|██████████| 313/313 [00:38<00:00,  8.12it/s]
test Loss: 73.6191 Acc: 3.1949

Epoch 4/4
----------
100%|██████████| 1563/1563 [07:49<00:00,  3.33it/s]
train Loss: 73.7289 Acc: 3.1939
100%|██████████| 313/313 [00:38<00:00,  8.13it/s]test Loss: 73.5955 Acc: 3.1949

Training complete in 42m 22s
Best val Acc: 3.194888

Tensorflow
Code:
import tensorflow_datasets as tfds
from tensorflow.keras import applications, models
import tensorflow as tf
import time

ds_test, ds_train = tfds.load('cifar10', split=['test', 'train'])

def resize(ip):
    image = ip['image']
    label = ip['label']
    image = tf.image.resize(image, (224, 224))
    image = tf.expand_dims(image,0)
    label = tf.one_hot(label,10)
    label = tf.expand_dims(label,0)
    return (image, label)

ds_train_ = ds_train.map(resize)
ds_test_ = ds_test.map(resize)

model = applications.vgg16.VGG16(input_shape = (224, 224, 3), weights=None, classes=10)
model.compile(optimizer='adam', loss = 'categorical_crossentropy', metrics= ['accuracy'])

batch_size = 32
since = time.time()
history = model.fit(ds_train_,
                    batch_size = batch_size,
                    steps_per_epoch = len(ds_train)//batch_size,
                    epochs = 5,
                    validation_steps = len(ds_test),
                    validation_data = ds_test_,
                    shuffle = True,)
time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format( time_elapsed // 60, time_elapsed % 60 ))

Result:
Epoch 1/5
1562/1562 [==============================] - 125s 69ms/step - loss: 36.9022 - accuracy: 0.1069 - val_loss: 2.3031 - val_accuracy: 0.1000
Epoch 2/5
1562/1562 [==============================] - 129s 83ms/step - loss: 2.3031 - accuracy: 0.1005 - val_loss: 2.3033 - val_accuracy: 0.1000
Epoch 3/5
1562/1562 [==============================] - 129s 83ms/step - loss: 2.3035 - accuracy: 0.1069 - val_loss: 2.3031 - val_accuracy: 0.1000
Epoch 4/5
1562/1562 [==============================] - 129s 83ms/step - loss: 2.3038 - accuracy: 0.1024 - val_loss: 2.3030 - val_accuracy: 0.1000
Epoch 5/5
1562/1562 [==============================] - 129s 83ms/step - loss: 2.3028 - accuracy: 0.1024 - val_loss: 2.3033 - val_accuracy: 0.1000
Training complete in 11m 23s

Output:
It is because in your tensorflow codes, the data pipeline is feeding a batch of 1 image into the model per step instead of a batch of 32 images.
Passing batch_size into model.fit does not really control the batch size when the data is in the form of datasets. The reason why it showed a seemingly correct steps per epoch from the log is that you passed steps_per_epoch into model.fit.
To correctly set the batch size:
ds_test, ds_train = tfds.load('cifar10', split=['test', 'train'])

def resize(ip):
    image = ip['image']
    label = ip['label']
    image = tf.image.resize(image, (224, 224))
    label = tf.one_hot(label,10)
    return (image, label)

train_size=len(ds_train)
test_size=len(ds_test)
ds_train_ = ds_train.shuffle(train_size).batch(32).map(resize)
ds_test_ = ds_test.shuffle(test_size).batch(32).map(resize)

model.fit call:
history = model.fit(ds_train_,
                    epochs = 1,
                    validation_data = ds_test_)

After fixed the problem, tensorflow got similar speed performance with pytorch. In my machine, pytorch took ~27 minutes per epoch while tensorflow took ~24 minutes per epoch.
According to the benchmarks from NVIDIA, pytorch and tensorflow had similar speed performance in most popular deep learning applications with real-world datasets and problem size. (Reference: https://developer.nvidia.com/deep-learning-performance-training-inference)


Instruction: PyTorch: "one of the variables needed for gradient computation has been modified by an inplace operation"

Input:
I'm training a PyTorch RNN on a text file of song lyrics to predict the next character given a character.
Here's how my RNN is defined:

import torch.nn as nn
import torch.optim

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        # from input, previous hidden state to new hidden state
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)

        # from input, previous hidden state to output
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

        # softmax on output
        self.softmax = nn.LogSoftmax(dim = 1)

    def forward(self, input, hidden):

        combined = torch.cat((input, hidden), 1)

        #get new hidden state
        hidden = self.i2h(combined)

        #get output
        output = self.i2o(combined)

        #apply softmax
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

rnn = RNN(input_size = num_chars, hidden_size = 200, output_size = num_chars)
criterion = nn.NLLLoss()

lr = 0.01
optimizer = torch.optim.AdamW(rnn.parameters(), lr = lr)

Here's my training function:
def train(train, target):

    hidden = rnn.initHidden()

    loss = 0

    for i in range(len(train)):

        optimizer.zero_grad()

        # get output, hidden state from rnn given input char, hidden state
        output, hidden = rnn(train[i].unsqueeze(0), hidden)

        #returns the index with '1' - indentifying the index of the right character
        target_class = (target[i] == 1).nonzero(as_tuple=True)[0]

        loss += criterion(output, target_class)

        loss.backward(retain_graph = True)
        optimizer.step()

        print("done " + str(i) + " loop")

    return output, loss.item() / train.size(0)

When I run my training function, I get this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [274, 74]], which is output 0 of TBackward, is at version 5; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Interestingly, it makes it through two complete loops of the training function before giving me that error.
Now, when I remove the retain_graph = True from loss.backward(), I get this error:
RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.

It shouldn't be trying to go backward through the graph multiple times here. Perhaps the graph is not getting cleared between training loops?

Output:
The issue is you are accumulating your loss values (and at the same time, the computation graphs associated attached to them) on variable loss, here:
    loss += criterion(output, target_class)

In turn, this means at every iteration you are trying to backpropagate through the current and previous loss values that were computed in previous inferences. In this particular instance where you are looping through your dataset, it isn't the right thing to do.
A simple fix is to accumulate loss's underlying value, i.e. the scalar value, not the tensor itself, using item. And, backpropagate on the current loss tensor:
total_loss = 0

for i in range(len(train)):
    optimizer.zero_grad()
    output, hidden = rnn(train[i].unsqueeze(0), hidden)
    target_class = (target[i] == 1).nonzero(as_tuple=True)[0]

    loss = criterion(output, target_class)
    loss.backward()

    total_loss += loss.item()

Since you are updating the model's parameter straight after having done the backpropagation, you don't need to retain the graph in memory.


Instruction: Pytorch: How to index a tensor?

Input:
I am new to PyTorch and am still wrapping my head around how to form a proper gather statement. I have a 4D input tensor of size (1,200,61,1632), where 1632 is the time dimension. I want to index it with a tensor idx which is size (4,1632) where each row of idx is a value I want to extract from the input tensor. So the rows of idx look like:
[0,20,30,0]
[0,150,9,1]
[0,180,100,2]
...

So that the output has size 1632. In other words I want to do this:
output = []
for i in range(1632):
  output.append(input[idx[0,i], idx[1,i], idx[2,i], idx[3,i]])

Is this an appropriate use case for torch.gather? Looking at the documentation for gather, it says the input and index tensors must have the same shape.

Output:
Since PyTorch doesn't offer an implementation of ravel_multi_index, the ugly way of doing it is this one:
output = input[idx[0, :], idx[1, :], idx[2, :], idx[3, :]]

In NumPy, you could do this way:
output = np.take(input, np.ravel_multi_index(idx, input.shape))


Instruction: pic should be Tensor or ndarray. Got

Input:
I am a beginner in PyTorch. I want to train a network using NYU dataset, but I am getting an error.

The error happens while I use the Dataloader to load my local dataset, and I want to print the data to demonstrate the code is right:

test=Mydataset(data_root,transforms,'image_train')
test2=DataLoader(test,batch_size=4,num_workers=0,shuffle=False)
for idx,data in enumerate(test2):
  print(idx)

Here's the rest of the code with the Mydataset definition:
from __future__ import division,absolute_import,print_function
from PIL import Image
from torch.utils.data import DataLoader,Dataset
from torchvision.transforms import transforms
data_root='D:/AuxiliaryDocuments/NYU/'
transforms=transforms.Compose([transforms.ToPILImage(),
                           transforms.Resize(224,101),
                           transforms.ToTensor()])

filename_txt={'image_train':'image_train.txt','image_test':'image_test.txt',
          'depth_train':'depth_train.txt','depth_test':'depth_test.txt'}

class Mydataset(Dataset):
  def __init__(self,data_root,transformation,data_type):
    self.transform=transformation
    self.image_path_txt=filename_txt[data_type]
    self.sample_list=list()
    f=open(data_root+'/'+data_type+'/'+self.image_path_txt)
    lines=f.readlines()
    for line in lines:
        line=line.strip()
        line=line.replace(';','')
        self.sample_list.append(line)
    f.close()

def __getitem__(self, index):
    item=self.sample_list[index]
    img=Image.open(item)
    if self.transform is not None:
        img=self.transform(img)
    idx=index
    return idx,img

def __len__(self):
    return len(self.sample_list)

Output:
The error in the title is different from the one in the image (which you should have posted as text, by the way). Assuming the one from the image is correct, your problem is the following:
Your transforms begins with a transforms.ToPILImage(), but the image is already read as a PIL image by the dataloader. If you remove that transformation, the code should run just fine.
# [...]
transforms = transforms.Compose([
    transforms.ToPILImage(),  # <<< remove this
    transforms.Resize(224, 101),
    transforms.ToTensor()
])

# [...]

class Mydataset(Dataset):
    # [...]
    def __getitem__(self, index):
        item = self.sample_list[index]
        img = Image.open(item)  # <<< this image is already a PIL image
        if self.transform is not None:
            img = self.transform(img)
        idx = index
        return idx, img
    # [...]


Instruction: Pytorch RNN error: RuntimeError: input must have 3 dimensions got 1

Input:
I am trying to train an RNN based off the code here
I also found two similar posts, but was not able to extrapolate from them what I should do to fix my problem here and here
The error is pretty easy to interpret, the model is expecting 3 dimensions, but I am only giving it 1. However, I do not know where to fix the issue.  I know that a good stack post is to include data, but I am not sure how to include example tensors in the post.  Apologies.
My input are 300d word embeddings and my output are one hot encoded vectors of length 11, where the model makes a classification choice in each of the 11 output dimensions.
I will start with the dataloader then go from there with the code.
from torch.utils.data import Dataset, DataLoader

class CustomDataset(Dataset):
    def __init__(self, dat, labels):
        self.labels = labels
        self.dat = dat

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        label = self.labels[idx]
        dat = self.dat[idx]
        sample = {"Sample": dat, "Class": label}
        return sample

I define my vanilla RNN as follows.
class VanillaRNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(VanillaRNN, self).__init__()

        # Defining some parameters
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        #Defining the layers
        # RNN Layer
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        # Fully connected layer
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, inputs):

        batch_size = inputs.size(0)

        # Initializing hidden state for first input using method defined below
        hidden = self.init_hidden(batch_size)

        # Passing in the input and hidden state into the model and obtaining outputs
        out, hidden = self.rnn(inputs, hidden)

        # Reshaping the outputs such that it can be fit into the fully connected layer
        out = out.contiguous().view(-1, self.hidden_dim)
        out = self.fc(out)

        return out, hidden

    def init_hidden(self, batch_size):
        # This method generates the first hidden state of zeros which we'll use in the forward pass
        # We'll send the tensor holding the hidden state to the device we specified earlier as well
        hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
        return hidden

and my training loop as follows
def plot_train_val(x, train, val, train_label,
                   val_label, title, y_label,
                   color):

  plt.plot(x, train, label=train_label, color=color)
  plt.plot(x, val, label=val_label, color=color, linestyle='--')
  plt.legend(loc='lower right')
  plt.xlabel('epoch')
  plt.ylabel(y_label)
  plt.title(title)

def count_parameters(model):
  parameters = sum(p.numel() for p in model.parameters() if p.requires_grad)
  return parameters

def init_weights(m):
  if type(m) in (nn.Linear, nn.Conv1d):
    nn.init.xavier_uniform_(m.weight)

# Training functioN
def train(model, device, train_loader, valid_loader, epochs, learning_rate):

  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

  train_loss, validation_loss = [], []
  train_acc, validation_acc = [], []

  for epoch in range(epochs):
    #train
    model.train()
    running_loss = 0.
    correct, total = 0, 0
    steps = 0
    for idx, batch in enumerate(train_loader):
      text = batch["Sample"].to(device)
      target = batch['Class'].to(device)
      target = torch.autograd.Variable(target).long()
      text, target = text.to(device), target.to(device)
      # add micro for coding training loop
      optimizer.zero_grad()
      output, hideden = model(text)
      print(output.shape, target.shape, target.view(-1).shape)
      loss = criterion(output, target.view(-1))
      loss.backward()
      optimizer.step()
      steps += 1
      running_loss += loss.item()

      # get accuracy
      _, predicted = torch.max(output, 1)
      print(predicted)
      #predicted = torch.round(output.squeeze())
      total += target.size(0)
      correct += (predicted == target).sum().item()

    train_loss.append(running_loss/len(train_loader))
    train_acc.append(correct/total)

    print(f'Epoch: {epoch + 1}, '
          f'Training Loss: {running_loss/len(train_loader):.4f}, '
          f'Training Accuracy: {100*correct/total: .2f}%')

    # evaluate on validation data
    model.eval()
    running_loss = 0.
    correct, total = 0, 0

    with torch.no_grad():
      for idx, batch in enumerate(valid_loader):
        text = batch["Sample"].to(device)
        print(type(text), text.shape)
        target = batch['Class'].to(device)
        target = torch.autograd.Variable(target).long()
        text, target = text.to(device), target.to(device)

        optimizer.zero_grad()
        output = model(text)

        loss = criterion(output, target)
        running_loss += loss.item()

        # get accuracy
        _, predicted = torch.max(output, 1)
        #predicted = torch.round(output.squeeze())
        total += target.size(0)
        correct += (predicted == target).sum().item()

    validation_loss.append(running_loss/len(valid_loader))
    validation_acc.append(correct/total)

    print (f'Validation Loss: {running_loss/len(valid_loader):.4f}, '
           f'Validation Accuracy: {100*correct/total: .2f}%')

  return train_loss, train_acc, validation_loss, validation_acc

When I run the model with the following, I get the error provided below.  Thanks in advance for any help.
# Model hyperparamters
#vocab_size = len(word_array)
learning_rate = 1e-3
output_size = 11
input_size = 300
epochs = 10
hidden_dim = 100
n_layers = 2

# Initialize model, training and testing
set_seed(SEED)
vanilla_rnn_model = VanillaRNN(input_size, output_size, hidden_dim, n_layers)

#vanilla_rnn_model = VanillaRNN(output_size, input_size, RNN_size, fc_size, DEVICE)
vanilla_rnn_model.to(DEVICE)

vanilla_rnn_start_time = time.time()
vanilla_train_loss, vanilla_train_acc, vanilla_validation_loss, vanilla_validation_acc = train(vanilla_rnn_model,
                                                                                DEVICE,
                                                                                train_loader,
                                                                                valid_loader,
                                                                                epochs = epochs,
                                                                                learning_rate = learning_rate)

The error :(
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-bfd2f8f3456f> in <module>()
     19                                                                                valid_loader,
     20                                                                                epochs = epochs,
---> 21                                                                                learning_rate = learning_rate)
     22 print("--- Time taken to train = %s seconds ---" % (time.time() - vanilla_rnn_start_time))
     23 #test_accuracy = test(vanilla_rnn_model, DEVICE, test_iter)

6 frames
<ipython-input-30-db1fa6c8b625> in train(model, device, train_loader, valid_loader, epochs, learning_rate)
     45       # add micro for coding training loop
     46       optimizer.zero_grad()
---> 47       output, hideden = model(text)
     48       print(output.shape, target.shape, target.view(-1).shape)
     49       loss = criterion(output, target.view(-1))

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-26-c34b90b3cbc3> in forward(self, x)
     21
     22         # Passing in the input and hidden state into the model and obtaining outputs
---> 23         out, hidden = self.rnn(x, hidden)
     24
     25         # Reshaping the outputs such that it can be fit into the fully connected layer

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    263         assert hx is not None
    264         input = cast(Tensor, input)
--> 265         self.check_forward_args(input, hx, batch_sizes)
    266         _impl = _rnn_impls[self.mode]
    267         if batch_sizes is None:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    227
    228     def check_forward_args(self, input: Tensor, hidden: Tensor, batch_sizes: Optional[Tensor]):
--> 229         self.check_input(input, batch_sizes)
    230         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    231

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in check_input(self, input, batch_sizes)
    201             raise RuntimeError(
    202                 'input must have {} dimensions, got {}'.format(
--> 203                     expected_input_dim, input.dim()))
    204         if self.input_size != input.size(-1):
    205             raise RuntimeError(

RuntimeError: input must have 3 dimensions, got 1

Output:
First, you need to wrap your dataset in a proper dataloader, and you can do something like this:
from torch.utils.data import DataLoader

# [...]

# define a batch_size, I'll use 4 as an example
batch_size = 4

train_dset = CustomDataset(X2, y)  # your current code (change train_loader to train_dset)
train_loader = DataLoader(train_dset, batch_size=batch_size, shuffle=True))

At this point, text now should be [4, 300].
Then, you said the your sequence length is equal to 1. To fix the error, you can add the length dimension using unsqueeze:
# [...]
output, hideden = model(text.unsqueeze(1))
# [...]

Now, text should be [4, 1, 300], and here you have the 3 dimensions the RNN forward call is expecting (your RNN has batch_first=True):

input: tensor of shape (L, N, H_in) when batch_first=False or (N, L, H_in) when batch_first=True containing the features of the input sequence. (...)


Instruction: Import torch ModuleNotFoundError

Input:
enter image description here
the import torch in my Jupiter notebook is not work, but in my terminal is works fine.

Output:
This looks like an interpreter configuration problem, your jupyter notebook is maybe using another interpreter while your terminal is using another. For eg, if you are using anaconda the default env is (base), so to add a certain module to your environment, you go to anaconda prompt and do this:
$ conda activate base

(base) $ conda install pytorch torchvision torchaudio cpuonly -c pytorch

Use the above command to install non-CUDA version of pytorch. If you're not using anaconda, you can just go to your terminal and do this:
$ workon myenv

(myenv) $ pip3 install torch torchvision torchaudio

And then try again. You need to do all this in the same environment as your jupyter notebook.


Instruction: Memory Leak in Pytorch Autograd of WGAN-GP

Input:
I want to use WGAN-GP, and when I run the code, it gives me an error:
def calculate_gradient_penalty(real_images, fake_images):

    t = torch.rand(real_images.size(0), 1, 1, 1).to(real_images.device)
    t = t.expand(real_images.size())

    interpolates = t * real_images + (1 - t) * fake_images
    interpolates.requires_grad_(True)

    disc_interpolates = D(interpolates)

    grad = torch.autograd.grad(
        outputs=disc_interpolates, inputs=interpolates,
        grad_outputs=torch.ones_like(disc_interpolates),
        create_graph=True, retain_graph=True, allow_unused=True)[0]

    grad_norm = torch.norm(torch.flatten(grad, start_dim=1), dim=1)
    loss_gp = torch.mean((grad_norm - 1) ** 2) * lambda_term

    return loss_gp

RuntimeError                              Traceback (most recent call
last)  in
/opt/conda/lib/python3.8/site-packages/torch/tensor.py in
backward(self, gradient, retain_graph, create_graph, inputs)
243                 create_graph=create_graph,
244                 inputs=inputs)
--> 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
246
247     def register_hook(self, hook):
/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py in
backward(tensors, grad_tensors, retain_graph, create_graph,
grad_variables, inputs)
143         retain_graph = create_graph
144
--> 145     Variable.execution_engine.run_backward(
146         tensors, grad_tensors, retain_graph, create_graph, inputs,
147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 2;
15.75 GiB total capacity; 13.76 GiB already allocated; 2.75 MiB free; 14.50 GiB reserved in total by PyTorch)

The train code:
%%time

d_progress = []
d_fake_progress = []
d_real_progress = []
penalty = []
g_progress = []

data = get_infinite_batches(benign_data_loader)
one = torch.FloatTensor([1]).to(device)
mone = (one * -1).to(device)

for g_iter in range(generator_iters):

    print('----------G Iter-{}----------'.format(g_iter+1))

    for p in D.parameters():
        p.requires_grad = True # This is by Default

    d_loss_real = 0
    d_loss_fake = 0
    Wasserstein_D = 0

    for d_iter in range(critic_iter):
        D.zero_grad()

        images = data.__next__()
        if images.size()[0] != batch_size:
            continue

        # Train Discriminator
        # Real Images
        images = images.to(device)
        z = torch.randn(batch_size, 100, 1, 1).to(device)
        d_loss_real = D(images)
        d_loss_real = d_loss_real.mean(0).view(1)
        d_loss_real.backward(mone)

        # Fake Images
        fake_images = G(z)
        d_loss_fake = D(fake_images)
        d_loss_fake = d_loss_fake.mean(0).view(1)
        d_loss_fake.backward(one)

        # Calculate Penalty
        gradient_penalty = calculate_gradient_penalty(images.data, fake_images.data)
        gradient_penalty.backward()

        # Total Loss
        d_loss = d_loss_fake - d_loss_real + gradient_penalty
        Wasserstein_D = d_loss_real - d_loss_fake
        d_optimizer.step()
        print(f'D Iter:{d_iter+1}/{critic_iter} Loss:{d_loss.detach().cpu().numpy()}')

        time.sleep(0.1)
        d_progress.append(d_loss) # Store Loss
        d_fake_progress.append(d_loss_fake)
        d_real_progress.append(d_loss_real)
        penalty.append(gradient_penalty)

    # Generator Updata
    for p in D.parameters():
        p.requires_grad = False  # Avoid Computation

    # Train Generator
    # Compute with Fake
    G.zero_grad()
    z = torch.randn(batch_size, 100, 1, 1).to(device)
    fake_images = G(z)
    g_loss = D(fake_images)
    g_loss = g_loss.mean().mean(0).view(1)
    g_loss.backward(one)
    # g_cost = -g_loss
    g_optimizer.step()
    print(f'G Iter:{g_iter+1}/{generator_iters} Loss:{g_loss.detach().cpu().numpy()}')

    g_progress.append(g_loss) # Store Loss

Does anyone know how to solve this problem?

Output:
All loss tensors which are saved outside of the optimization cycle (i.e. outside the for g_iter in range(generator_iters) loop) need to be detached from the graph. Otherwise, you are keeping all previous computation graphs in memory.
As such, you should detach anything that gets appended to d_progress, d_fake_progress, d_real_progress, penalty, and g_progress.
You can do so by converting the tensor to a scalar value with torch.Tensor.item, the graph will free itself on the following iteration. Change the following lines:
    d_progress.append(d_loss) # Store Loss
    d_fake_progress.append(d_loss_fake)
    d_real_progress.append(d_loss_real)
    penalty.append(gradient_penalty)

#######

g_progress.append(g_loss) # Store Loss

to:
    d_progress.append(d_loss.item()) # Store Loss
    d_fake_progress.append(d_loss_fake.item())
    d_real_progress.append(d_loss_real.item())
    penalty.append(gradient_penalty.item())

#######

g_progress.append(g_loss.item()) # Store Loss


Instruction: Setting results of torch.gather(...) calls

Input:
I have a 2D pytorch tensor of shape n by m. I want to index the second dimension using a list of indices (which could be done with torch.gather) then then also set new values to the result of the indexing.
Example:
data = torch.tensor([[0,1,2], [3,4,5], [6,7,8]]) # shape (3,3)
indices = torch.tensor([1,2,1], dtype=torch.long).unsqueeze(-1) # shape (3,1)
# data tensor:
# tensor([[0, 1, 2],
#         [3, 4, 5],
#         [6, 7, 8]])

I want to select the specified indices per row (which would be [1,5,7] but then also set these values to another number - e.g. 42
I can select the desired columns row wise by doing:
data.gather(1, indices)
tensor([[1],
        [5],
        [7]])
data.gather(1, indices)[:] = 42 # **This does NOT work**, since the result of gather
                                # does not use the same storage as the original tensor

which is fine, but I would like to change these values now, and have the change also affect the data tensor.
I can do what I want to achieve using this, but it seems to be very un-pythonic:
max_index = torch.max(indices)
for i in range(0, max_index + 1):
  mask = (indices == i).nonzero(as_tuple=True)[0]
  data[mask, i] = 42
print(data)
# tensor([[ 0, 42,  2],
#         [ 3,  4, 42],
#         [ 6, 42,  8]])

Any hints on how to do that more elegantly?

Output:
What you are looking for is torch.scatter_ with the value option.

Tensor.scatter_(dim, index, src, reduce=None) → Tensor
Writes all values from the tensor src into self at the indices specified in the index tensor. For each value in src, its output index is specified by
its index in src for dimension != dim and by the corresponding value
in index for dimension = dim.
With 2D tensors as input and dim=1, the operation is:
self[i][index[i][j]] = src[i][j]

No mention of the value parameter though...

With value=42, and dim=1, this will have the following effect on data:
data[i][index[i][j]] = 42

Here applied in-place:
>>> data.scatter_(index=indices, dim=1, value=42)
>>> data
tensor([[ 0, 42,  2],
        [ 3,  4, 42],
        [ 6, 42,  8]])


Instruction: Hello World aka. MNIST with feed forward gets less accuracy in comparison of plain with DistributedDataParallel (DDP) model with only one node

Input:
This is a cross-post to my question in the Pytorch forum.
When using DistributedDataParallel (DDP) from PyTorch on only one node I expect it to be the same as a script without DistributedDataParallel.
I created a simple MNIST training setup with a three-layer feed forward neural network. It gives significantly lower accuracy (around 10%) if trained with the same hyperparameters, same epochs, and generally same code but the usage of the DDP library.
I created a GitHub repository demonstrating my problem.
I hope it is a usage error of the library, but I do not see how there is a problem, also colleges of mine did already audit the code. Also, I tried it on macOS with a CPU and on three different GPU/ubuntu combinations (one with a 1080-Ti, one with a 2080-Ti and a cluster with P100s inside) all giving the same results. Seeds are fixed for reproducibility.

Output: You are using different batch sizes in your two experiments: batch_size=128, and batch_size=32 for mnist-distributed.py and mnist-plain.py respectively. This would indicate that you won't have the same performance result with those two trainings.


Instruction: Difficulty setting batch size correctltly in 2 layer RNN

Input:
I am building an RNN that makes a multi-class classification output for 11 dimensions in the output. The input are word embeddings that I took from a pretrained glove model.
The error I get is (full traceback at the end of the question):

ValueError: Expected input batch_size (1) to match target batch_size (11).

Note that here I use batch_size=1, and the error says "expected batch size 1 to match target batch_size (11)". However, if I change batch size to 11, the error changes to:

ValueError: Expected input batch_size (11) to match target batch_size (121).

I think that the error is coming from the shape of text which is torch.Size([11, 300]), which lacks a sequence length, but I thought that if I do not assign a seq length it defaults to 1. However, I do not know how to add this in.
Training loop:
def train(model, device, train_loader, valid_loader, epochs, learning_rate):

  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

  train_loss, validation_loss = [], []
  train_acc, validation_acc = [], []

  for epoch in range(epochs):
    #train
    model.train()
    running_loss = 0.
    correct, total = 0, 0
    steps = 0
    for idx, batch in enumerate(train_loader):
      text = batch["Sample"].to(device)
      target = batch['Class'].to(device)
      print(text.shape, target.shape)
      text, target = text.to(device), target.to(device)
      # add micro for coding training loop
      optimizer.zero_grad()
      print(text.shape)
      output, hidden = model(text.unsqueeze(1))
      #print(output.shape, target.shape, target.view(-1).shape)
      loss = criterion(output, target.view(-1))
      loss.backward()
      optimizer.step()
      steps += 1
      running_loss += loss.item()

      # get accuracy
      _, predicted = torch.max(output, 1)
      print(predicted)
      #predicted = torch.round(output.squeeze())
      total += target.size(0)
      correct += (predicted == target).sum().item()

    train_loss.append(running_loss/len(train_loader))
    train_acc.append(correct/total)

    print(f'Epoch: {epoch + 1}, '
          f'Training Loss: {running_loss/len(train_loader):.4f}, '
          f'Training Accuracy: {100*correct/total: .2f}%')

    # evaluate on validation data
    model.eval()
    running_loss = 0.
    correct, total = 0, 0

    with torch.no_grad():
      for idx, batch in enumerate(valid_loader):
        text = batch["Sample"].to(device)
        print(type(text), text.shape)
        target = batch['Class'].to(device)
        target = torch.autograd.Variable(target).long()
        text, target = text.to(device), target.to(device)

        optimizer.zero_grad()
        output = model(text)

        loss = criterion(output, target)
        running_loss += loss.item()

        # get accuracy
        _, predicted = torch.max(output, 1)
        #predicted = torch.round(output.squeeze())
        total += target.size(0)
        correct += (predicted == target).sum().item()

    validation_loss.append(running_loss/len(valid_loader))
    validation_acc.append(correct/total)

    print (f'Validation Loss: {running_loss/len(valid_loader):.4f}, '
           f'Validation Accuracy: {100*correct/total: .2f}%')

  return train_loss, train_acc, validation_loss, validation_acc

This is how I call the training loop:
# Model hyperparamters
#vocab_size = len(word_array)
learning_rate = 1e-3
hidden_dim = 100
output_size = 11
input_size = 300
epochs = 10
n_layers = 2

# Initialize model, training and testing
set_seed(SEED)
vanilla_rnn_model = VanillaRNN(input_size, output_size, hidden_dim, n_layers)
vanilla_rnn_model.to(DEVICE)
vanilla_rnn_start_time = time.time()
vanilla_train_loss, vanilla_train_acc, vanilla_validation_loss, vanilla_validation_acc = train(vanilla_rnn_model,
                                                                                DEVICE,
                                                                                train_loader,
                                                                                valid_loader,
                                                                                epochs = epochs,
                                                                                learning_rate = learning_rate)

This is how I create the dataloaders:
# Splitting dataset
# define a batch_size, I'll use 4 as an example
batch_size = 1

train_dset = CustomDataset(X2, y)  # create data set
train_loader = DataLoader(train_dset, batch_size=batch_size, shuffle=True) #load data with batch size
valid_dset = CustomDataset(X2, y)
valid_loader = DataLoader(valid_dset, batch_size=batch_size, shuffle=True)

g_seed = torch.Generator()
g_seed.manual_seed(SEED)

Full traceback:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-23-bfd2f8f3456f> in <module>()
     19                                                                                valid_loader,
     20                                                                                epochs = epochs,
---> 21                                                                                learning_rate = learning_rate)
     22 print("--- Time taken to train = %s seconds ---" % (time.time() - vanilla_rnn_start_time))
     23 #test_accuracy = test(vanilla_rnn_model, DEVICE, test_iter)

3 frames
<ipython-input-22-16748701034f> in train(model, device, train_loader, valid_loader, epochs, learning_rate)
     47       output, hidden = model(text.unsqueeze(1))
     48       #print(output.shape, target.shape, target.view(-1).shape)
---> 49       loss = criterion(output, target.view(-1))
     50       loss.backward()
     51       optimizer.step()

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1119     def forward(self, input: Tensor, target: Tensor) -> Tensor:
   1120         return F.cross_entropy(input, target, weight=self.weight,
-> 1121                                ignore_index=self.ignore_index, reduction=self.reduction)
   1122
   1123

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2822     if size_average is not None or reduce is not None:
   2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2825
   2826

ValueError: Expected input batch_size (1) to match target batch_size (11).

Output:
You should not be using .view(-1). This line:
loss = criterion(output, target.view(-1))

should be:
loss = criterion(output, target)

It is effectively removing your batch dimension. For batch_size=1, it changes (1, 11) to (11,). When you changed batch_size to 11, then the view changed the shape from (11, 11) to (121,), hence the error.


Instruction: probleme of neural network :mat1 and mat2 shapes cannot be multiplied

Input:
I implemented a simple neuron network like this:
import torch
from torch import nn

class Simple_NN(nn.Module):
  '''
    Multilayer Perceptron.
  '''
  def __init__(self, input_dim):
    super().__init__()
    self.input = input_dim
    #self.out = out_dim
    self.layer = nn.Linear(self.input, 1, bias=False)

  def getweights(self):
    return self.layer.weight
  def normalize(self):
    self.layer.weight.data /= self.layer.weight.data.sum()
    return self.layer.weight

  def forward(self, x, dim = 0):
    sort = torch.sort(x, dim, descending = True)[0]
    #top = torch.topk(x, 4, dim)
    sort = self.layer(sort)
    return sort

when I run this piece of code:
outputs = torch.tensor([[1.9, 0.4, 1.3, 0.8, 0.2, 0.0],[1.7, 1.4, 0.3, 1.8, 1.2, 1.1]])
model = Simple_NN(input_dim = outputs.shape[0])
model.getweights()
model.normalize()

I get the following result:
Parameter containing:
tensor([[0.9772, 0.0228]], requires_grad=True)

but, when I run this line:
model(outputs, dim=0)

I get this error:

<ipython-input-1-dd06de9bb6ad> in forward(self, x, dim)
     20     sort = torch.sort(x, dim, descending = True)[0]
     21     #top = torch.topk(x, 4, dim)
---> 22     sort = self.layer(sort)
     23     return sort

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x6 and 2x1)

How can I solve this problem?

Output:
As you didn't provide more details, here's 2 possible ways to solve this:

If the batch_size=2, the input_dim should be 6, not 2:
model = Simple_NN(input_dim = outputs.shape[1])  # change [0] to [1]

If the batch_size=6, then outputs needs to be transposed:
model(outputs.t(), dim=0)  # add .t()

I think the correct solution to your case is the first one, but both of them work. It depends on what you actually want.


Instruction: To get gpu support to create neural networks is conda environment necessary?

Input:
So I recently started learning about deep learning in the TensorFlow lib in python. And came across a problem, so turns out, (and I genuinely didn't know this) to use the GPU for programming specially TensorFlow and PyTorch you need these CUDA toolkit, cuDNN libs, and Visual Studio community. Which far enough I'll download and already finished downloading vs community.
But what I want to ask is, is the setting up of Conda environment necessary? Can it not be done with pip?
Sorry if this is not the right place to ask this kind of question, but I couldn't find anything in detail about this.

Output:
No, you can just use pip install tensorflow-gpu to install TensorFlow with GPU support. It's your choice if you want to create Conda environment or not. But before using that pip command, make sure you have CUDA 11.2 and cuDNN 8.1.
And in case of PyTorch just go to this site and copy the command and install it.


Instruction: Generating probabilites from patches of image

Input:
I am working with an image of size 512x512. The image is divided into patches using einops with patch size of 32. The number of patches overall is 256, in other words, we get a new "image" of size 256x1024.
Since this image is actually a mask for a segmentation problem, the image is actually comprised of only 4 values (4 classes): 0 for the background, 1 for the first class, 2 for the second class, 3 for the third class.
My goal is to take every patch, and compute for every class C the following:
Number of pixels in this patch / Number of pixels labeled C.
This should give me an array of size 4 where the first entry is the total number of pixels in the patch (1024) over the number of background pixels (labeled as 0), the second, third and fourth entries are the same but for the corresponding class.
In theory, I know that I need to iterate over every single patch and then count how many pixels of each class exists in the current patch, then divide by 1024. Doing this 256 yields exactly what I want. The problem is that I have a (very) large amount of images that I need to do this for, and the size of 512 is just an example to make the question simpler, therefore a for loop is out of question.
I know that I can get the result that I want using numpy. I tried both: numpy.apply_over_axes and numpy.apply_along_axis but I don't know which one is better suited for this task, also there is numpy.where which I don't know how it applies here.
Here is what I did:
from einops import rearrange
import numpy as np

labn = np.random.randint(4,size= (512,512))      # Every pixels in this image is of value: 0,1,2,3
to_patch = rearrange(labn, "(h p1) (w p2) -> (h w) (p1 p2)", p1=32, p2=32)
print(to_patch.shape) # (256,1024)

c0 = np.full(1024, 0)
c1 = np.full(1024, 1)
c2 = np.full(1024, 2)
c3 = np.full(1024, 3)

def f(a):
    _c0 = a == c0
    _c1 = a == c2
    _c2 = a == c2
    _c3 = a == c3
    pr = np.array([np.sum(_c0), np.sum(_c1), np.sum(_c2), np.sum(_c3)]) / 1024
    return pr

resf = np.apply_along_axis(f, 1, to_patch)
print(resf.shape) # (256, 4, 1024)

Two things:

I want the output to be 256x4 where every array along the second axes sums to one.
Is there a faster/better/pythonic way to do this, preferably vectorized?

EDIT: I forgot to add the sum, so now I do get 256x4.

Output:
There is a built-in function to count occurrences called torch.histc, it is similar to Python's collections.Counter.

torch.histc(input, bins=100, min=0, max=0, *, out=None) → Tensor
Computes the histogram of a tensor.
The elements are sorted into equal width bins between min and max. If
min and max are both zero, the minimum and maximum values of the data
are used.
Elements lower than min and higher than max are ignored.

You need to specify the number of bins, here the number of classes C. As well as the min and max values for ordering. Also, it won't work with multi-dimensional tensors as such the resulting tensor will contain global statistics of the input tensor regardless of dimensions. As a possible workaround, you can iterate through your patches, calling torch.histc each time, then stacking the results and normalizing:
resf = torch.stack([torch.histc(patch, C, min=0, max=C-1) for patch in x]) / x.size(1)


Instruction: torch.return_types.max as Tensor

Input:
I try to pass torch.max() return type (torch.return_types.max) as argument to function torch.tile():
torch.tile(torch.max(x), (1, 1, 1, 5))
The error is: TypeError: tile(): argument 'input' (position 1) must be Tensor, not torch.return_types.max.
How can I convert torch.return_types.max to Tensor? Maybe I should use another function to find maximum in Tensor?

Output:
For this error to be true, you have to be using some dim=?, because only then torch.max will return a tuple of (values, indices).
You can fix that error by using only the first output:
torch.tile(torch.max(x, dim=0)[0], (1, 1, 1, 5))


Instruction: With torch or torchvision, how can I resize and crop an image batch, and get both the resizing scales and the new images?

Input:
I want to transform a batch of images such that they are randomly cropped (with fixed ratio) and resized (scaled). However, I want not only the new images but also a tensor of the scale factors applied to each image. For example, this torchvision transform will do the cropping and resizing I want:
scale_transform = torchvision.transforms.RandomResizedCrop(224, scale=(0.08, 1.0), ratio=(1.0, 1.0))
images_scaled = scale_transform(images_original)

But I also want to know the scale factors. How might I get those scale factors, or tackle this in a different way?

Output:
If I understand correctly, you want to get ratio of how much cropped part was resized. You can get it by computing xy size of the cropped part and divide it by size you want to get.
class MyRandomResizedCrop(object):
    def __init__(self, size, scale, ratio):
        self.t = torchvision.transforms.RandomResizedCrop(size, scale=scale, ratio=ratio)
        self.size = size
        self.scale = scale
        self.ratio = ratio

    def __call__(self, sample):
        sample = F.to_pil_image(sample)
        crop_size = self.t.get_params(sample, self.scale, self.ratio)

        x_size = crop_size[2] - crop_size[0]
        y_size = crop_size[3] - crop_size[1]

        x_ratio = sample.size[0] / x_size
        y_ratio = sample.size[1] / y_size
        ratio = (x_ratio, y_ratio)

        output = F.crop(sample, *crop_size)
        output = F.resize(output, self.size)

        return ratio, output

import torchvision
from PIL import Image
import torchvision.transforms.functional as F

size = 244
scale = (0.08, 1.0)
ratio = (1.0, 1.0)

t = MyRandomResizedCrop(size, scale, ratio)

img = torch.rand((3,1024,1024), dtype=torch.float32)

r, img = t(img)


Instruction: Plot the transformed (augmented) images in pytorch

Input:
I want to use one of the image augmentation techniques (for example rotation or horizontal flip) and apply it to some images of the CIFAR-10 dataset and plot them in PyTorch.
I know that we can use the following code to augmented images:
from torchvision import models, datasets, transforms
from torchvision.datasets import CIFAR10

data_transforms = transforms.Compose([
        # add augmentations
        transforms.RandomHorizontalFlip(p=0.5),
        # The output of torchvision datasets are PILImage images of range [0, 1].
        # We transform them to Tensors of normalized range [-1, 1]
        transforms.ToTensor(),
        transforms.Normalize(mean, std)
    ])

and then I used the transforms above when I want to load the Cifar10 dataset:
train_set = CIFAR10(
    root='./data/',
    train=True,
    download=True,
    transform=data_transforms['train'])

As far as I know, when this code is used, all CIFAR10 datasets are transformed.
Question
My question is how can I use data transform or augmentation techniques for some images in data sets and plot them? for example 10 images and their augmented images.

Output:
when this code is used, all CIFAR10 datasets are transformed

Actually, the transform pipeline will only be called when images in the dataset are fetched via the __getitem__ function by the user or through a data loader. So at this point in time, train_set doesn't contain augmented images, they are transformed on the fly.

You will need to construct another dataset without augmentations.
>>> non_augmented = CIFAR10(
...     root='./data/',
...     train=True,
...     download=True)

>>> train_set = CIFAR10(
...     root='./data/',
...     train=True,
...     download=True,
...     transform=data_transforms)

Stack some images together:
>>> imgs = torch.stack((*[non_augmented[i][0] for i in range(10)],
                        *[train_set[i][0] for i in range(10)]))

>>> imgs.shape
torch.Size([20, 3, 32, 32])

Then torchvision.utils.make_grid can be useful to create the desired layout:
>>> grid = torchvision.utils.make_grid(imgs, nrow=10)

There you have it!
>>> transforms.ToPILImage()(grid)


Instruction: torch.utils.data.DataLoader - why it adds a dimension

Input:
from torchvision import datasets, transforms
transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,), (0.5,)),])
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)            # A
trainloader = torch.utils.data.DataLoader(trainset.train_data, batch_size=64, shuffle=True) # B

dataiter = iter(trainloader)
images, labels = dataiter.next() # A
images         = dataiter.next() # B
images.shape

Why the code above, approach #A gives torch.Size([64, 1, 28, 28]) , while #B gives torch.Size([64, 28, 28])? Where does the second dimension with value 1 in #A come from?
Thank you in advance.

Output:
The second dimension describes the color channels which for grayscale is 1. RGB images would have 3 channels (red, green and blue) and would look something like 64, 3, W, H.
So when working with CNNs your data normally has to be in shape batchsize, channels, width, height therefore 64, 1, 28, 28 is correct.


Instruction: PyTorch tutorial using testing dataset in training epoch

Input:
In PyTorch official tutorial: OPTIMIZING MODEL PARAMETERS. The dataset is said to be split into training data and testing data. However, the testing dataset was used in each epoch.
Shouldn't the testing dataset only be used once for the evaluating the final model? Or in this tutorial the 'testing dataset' is actually 'validation dataset' and there is no testing dataset in the code?

Output: Indeed in many basic ML/DL tutorials, the fine distinction between validation and test is often overlooked. In the tutorial you mentioned, since no "decision" is made based on the validation performance (e.g., early stopping), this set can be considered a "test" set and it is okay to monitor test performance during training.


Instruction: AttributeError: module 'torch.fft' has no attribute 'fftfreq'

Input:
I followed the example from
https://pytorch.org/docs/stable/generated/torch.fft.fftshift.html#torch.fft.fftshift
import torch.fft
f = torch.fft.fftfreq(4)
a = torch.fft.fftshift(f)
print(a)

and got the error

AttributeError: module 'torch.fft' has no attribute 'fftfreq'

I tried pip torch==1.7.0+cu110 and pip torch==1.7.1+cu110 and also conda pytorch==1.7.1 with cudatoolkit=11.0.
Others have the same problem https://discuss.pytorch.org/t/unable-to-use-correctly-the-new-torch-fft-module/104560/6
But changing to torch1.7.0 didn't solve the problem.
How to use torch.fft correctly?

Output: Function torch.fft.fftfreq was introduced in PyTorch version 1.8.0. You need to upgrade to this version or higher in order to use it.


Instruction: How to measure the semantic similarities among image features extracted by pre-trained models(e.g. vgg, resnet...)?

Input:
As far as I know, pre-trained models play well in many tasks as a feature-extractor, thanks to their abundant training dataset.
However, I'm wondering that whether the model, let's say vgg-16,

have certain ability to extract some "semantic" information from input image?

If the answer is positive, given an unlabeled dataset,

is it possible to "cluster" images by measuring the semantic similarities of the extracted features?

Actually, I've spent some efforts:

Load pre-trained vgg-16 through Pytorch.
Load Cifar-10 dataset and transform to batched-tensor X, of size(5000, 3, 224, 224).
Fine-tune vgg.classifier, define its output dimension as 4096.
Extract features:

 features = vgg.features(X).view(X.shape[0], -1) # X: (5000, 3, 224, 224)

 features = vgg.classifier(features) # features: (5000, 25088)

 return features # features: (5000, 4096)

Try out cosine similarity, inner product, torch.cdist, however, only to find several bad clusters.

Any suggestion? Thanks in advance.

Output:
You might not want to go all the way to the last layer, as these contain features specific to the classification task at hand. Using features from layers higher up in the classifier might help. Additionally, you want to switch to eval mode since VGG-16 has a dropout layer in its classifier.
>>> vgg16 = torchvision.models.vgg(pretrained=True).eval()

Truncate the classifier:
>>> vgg16.classifier = vgg16.classifier[:4]

Now vgg16's classifier will look like:
(classifier): Sequential(
  (0): Linear(in_features=25088, out_features=4096, bias=True)
  (1): ReLU(inplace=True)
  (2): Dropout(p=0.5, inplace=False)
  (3): Linear(in_features=4096, out_features=4096, bias=True)
)

Then extract the features:
>>> vgg16(torch.rand(1, 3, 124, 124)).shape
torch.Size([1, 4096])


Instruction: How to concatenate a list of tensors on a specific axis?

Input:
I have a list (my_list) of tensors all with the same shape. I want to concatenate them on the channel axis.
Helping code
for i in my_list:
    print(i.shape) #[1, 3, 128, 128] => [batch, channel, width, height]

I would like to get a new tensor i.e. new_tensor = [1, 3*len(my_list), width, height]
I don't want to use torch.stack() to add a new dimension. And i am unable to figure out how can I use torch.cat() to do this?

Output:
Given a example list containing 10 tensors shaped (1, 3, 128, 128):
>>> my_list = [torch.rand(1, 3, 128, 128) for _ in range(10)]

You are looking to concatenate your tensors on axis=1 because the 2nd dimension is where the tensor to concatenate together. You can do so using torch.cat:
>>> res = torch.cat(my_list, axis=1)
>>> res.shape
torch.Size([1, 30, 128, 128])

This is actually equivalent to stacking your tensor in my_list vertically, i.e. by using torch.vstack:
>>> res = torch.vstack(my_list)


Instruction: How can I get a batch of samples from a dataset given a list of idxs in pytorch?

Input:
I have a torch.utils.data.Dataset object, I would like to have a DataLoader or a similar object that accepts a list of idxs and returns a batch of samples with the corresponding idxs.
Example, I have
list_idxs = [10, 109, 7, 12]
I would like to do like:
batch = loader.getbatch(list_idxs)
where batch contains:
[sample10, sample109, sample7, sample12]
Is there a simple and elegant way to do that in an optimized way?

Output:
If I understand your question correctly, you could have a DataLoader return a sequence of hand-selected batches using a custom batch_sampler (you don't even need to pass it a sampler in this case).
Given an arbitrary Dataset:
>>> from torch.utils.data import DataLoader, Dataset
>>> from torch.utils.data.sampler import Sampler
>>> class MyDataset(Dataset):
...     def __getitem__(self, idx):
...         return idx

you can then define something like:
>>> class MyBatchSampler(Sampler):
...     def __init__(self, batches):
...         self.batches = batches
...
...     def __iter__(self):
...         for batch in self.batches:
...             yield batch
...
...     def __len__(self):
...         return len(self.batches)

which just takes a list of lists containing dataset indices to include in each batch.
Then:
>>> dataset = MyDataset()
>>> batch_sampler = MyBatchSampler([[1, 2, 3], [5, 6, 7], [4, 2, 1]])
>>> dataloader = DataLoader(dataset=dataset, batch_sampler=batch_sampler)
>>> for batch in dataloader:
...     print(batch)
...
tensor([1, 2, 3])
tensor([5, 6, 7])
tensor([4, 2, 1])

Should be easy to extend to your actual Dataset, etc.


Instruction: Convert at::Tensor to double in C++ when using LibTorch (PyTorch)

Input:
In the following code, I want to compare the loss (data type at::Tensor) with a lossThreshold (data type double). I want to convert loss to double before making that comparison. How do I do it?
int main() {
    auto const input1(torch::randn({28*28});
    auto const input2(torch::randn({28*28});
    double const lossThreshold{0.05};
    auto const loss{torch::nn::functional::mse_loss(input1, input2)}; // this returns an at::Tensor datatype
    return loss > lossThreshold ? EXIT_FAILURE : EXIT_SUCCESS;
}

Output:
Thanks to GitHub CoPilot which recommended this solution. I guess I should leave my job now. :(
The solution is using the item<T>() template function as follows:
int main() {
    auto const input1(torch::randn({28*28}); // at::Tensor
    auto const input2(torch::randn({28*28}); // at::Tensor
    double const lossThreshold{0.05}; // double
    auto const loss{torch::nn::functional::mse_loss(input1, input2).item<double>()}; // the item<double>() converts at::Tensor to double
    return loss > lossThreshold ? EXIT_FAILURE : EXIT_SUCCESS;
}


Instruction: CrossEntropyLoss equivalence to LogSoftmax + NLLLoss

Input:
According to the docs, CrossEntropyLoss criterion combines LogSoftmax function and NLLLoss criterion.
That is all fine and well, but testing it doesn't seem to substantiate this claim (ie assertion fails):
model_nll = nn.Sequential(nn.Linear(3072, 1024),
                          nn.Tanh(),
                          nn.Linear(1024, 512),
                          nn.Tanh(),
                          nn.Linear(512, 128),
                          nn.Tanh(),
                          nn.Linear(128, 2),
                          nn.LogSoftmax(dim=1))

model_ce = nn.Sequential(nn.Linear(3072, 1024),
                          nn.Tanh(),
                          nn.Linear(1024, 512),
                          nn.Tanh(),
                          nn.Linear(512, 128),
                          nn.Tanh(),
                          nn.Linear(128, 2),
                          nn.LogSoftmax(dim=1))

loss_fn_ce = nn.CrossEntropyLoss()
loss_fn_nll = nn.NLLLoss()

t = torch.rand(1,3072)
target = torch.tensor([1])

with torch.no_grad():
    loss_nll = loss_fn_nll(model_nll(t), target)
    loss_ce = loss_fn_ce(model_ce(t), target)
    assert torch.eq(loss_nll, loss_ce)

I'm obviously missing something basic here.

Output:
As you noticed, the weights are initialized randomly.
One way to get two modules sharing the same weights is to simply export with state_dict the state of one and set it on the other with load_state_dict.
This is a one-liner:
>>> model_ce.load_state_dict(model_nll.state_dict())


Instruction: Trying to backward through the graph a second time with GANs model

Input:
I'm trying to setup a simple GANs training loop but am getting the following error:

RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.

for epoch in range(N_EPOCHS):
    # gets data for the generator
    for i, batch in enumerate(dataloader, 0):

        # passing target images to the Discriminator
        global_disc.zero_grad()
        output_disc = global_disc(batch.to(device))
        error_target = loss(output_disc, torch.ones(output_disc.shape).cuda())
        error_target.backward()

        # apply mask to the images
        batch = apply_mask(batch)

        # passes fake images to the Discriminator
        global_output, local_output = gen(batch.to(device))
        output_disc = global_disc(global_output.detach())
        error_fake = loss(output_disc, torch.zeros(output_disc.shape).to(device))
        error_fake.backward()

        # combines the errors
        error_total = error_target + error_fake
        optimizer_disc.step()

        # updates the generator
        gen.zero_grad()
        error_gen = loss(output_disc, torch.ones(output_disc.shape).to(device))
        error_gen.backward()
        optimizer_gen.step()

        break
    break

As far as I can tell, I have the operations in the right order, I'm zeroing out the gradients, and I'm detaching the output of the generator before it goes into discriminator.
This article was helpful but I'm still running into something I don't understand.

Output:
Two important points come to mind:

You should feed your generator with noise, and not the real input:
global_output, local_output = gen(noise.to(device))

Above noise should have the appropriate shape (it is the input of your generator).

In order to optimize the generator, you are required to recompute the discriminator output, because it has already been backpropagated on. Simply add this line to recompute output_disc:
# updates the generator
gen.zero_grad()
output_disc = global_disc(global_output)
# ...

Please refer to this tutorial provided by PyTorch for a full walkthrough.


Instruction: Understanding why memory allocation occurs during inference, backpropagation, and model update

Input:
In the process of tracking down a GPU OOM error, I made the following checkpoints in my Pytorch code (running on Google Colab P100):
learning_rate = 0.001
num_epochs = 50

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

print('check 1')
!nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

model = MyModel()

print('check 2')
!nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

model = model.to(device)

print('check 3')
!nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

print('check 4')
!nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

for epoch in range(num_epochs):
    train_running_loss = 0.0
    train_accuracy = 0.0

    model = model.train()

    print('check 5')
    !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

    ## training step
    for i, (name, output_array, input) in enumerate(trainloader):

        output_array = output_array.to(device)
        input = input.to(device)
        comb = torch.zeros(1,1,100,1632).to(device)

        print('check 6')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        ## forward + backprop + loss
        output = model(input, comb)

        print('check 7')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        loss = my_loss(output, output_array)

        print('check 8')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        optimizer.zero_grad()

        print('check 9')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        loss.backward()

        print('check 10')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        ## update model params
        optimizer.step()

        print('check 11')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        train_running_loss += loss.detach().item()

        print('check 12')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        temp = get_accuracy(output, output_array)

        print('check 13')
        !nvidia-smi | grep MiB | awk '{print $9 $10 $11}'

        train_accuracy += temp

with the following output:
check 1
2MiB/16160MiB
check 2
2MiB/16160MiB
check 3
3769MiB/16160MiB
check 4
3769MiB/16160MiB
check 5
3769MiB/16160MiB
check 6
3847MiB/16160MiB
check 7
6725MiB/16160MiB
check 8
6725MiB/16160MiB
check 9
6725MiB/16160MiB
check 10
9761MiB/16160MiB
check 11
16053MiB/16160MiB
check 12
16053MiB/16160MiB
check 13
16053MiB/16160MiB
check 6
16053MiB/16160MiB
check 7
16071MiB/16160MiB
check 8
16071MiB/16160MiB
check 9
16071MiB/16160MiB
check 10
16071MiB/16160MiB
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-f566d09448f9> in <module>()
     65
     66         ## update model params
---> 67         optimizer.step()
     68
     69         print('check 11')

3 frames
/usr/local/lib/python3.7/dist-packages/torch/optim/optimizer.py in wrapper(*args, **kwargs)
     86                 profile_name = "Optimizer.step#{}.step".format(obj.__class__.__name__)
     87                 with torch.autograd.profiler.record_function(profile_name):
---> 88                     return func(*args, **kwargs)
     89             return wrapper
     90

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     26         def decorate_context(*args, **kwargs):
     27             with self.__class__():
---> 28                 return func(*args, **kwargs)
     29         return cast(F, decorate_context)
     30

/usr/local/lib/python3.7/dist-packages/torch/optim/adam.py in step(self, closure)
    116                    lr=group['lr'],
    117                    weight_decay=group['weight_decay'],
--> 118                    eps=group['eps'])
    119         return loss

/usr/local/lib/python3.7/dist-packages/torch/optim/_functional.py in adam(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, amsgrad, beta1, beta2, lr, weight_decay, eps)
     92             denom = (max_exp_avg_sqs[i].sqrt() / math.sqrt(bias_correction2)).add_(eps)
     93         else:
---> 94             denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(eps)
     95
     96         step_size = lr / bias_correction1

RuntimeError: CUDA out of memory. Tried to allocate 2.32 GiB (GPU 0; 15.78 GiB total capacity; 11.91 GiB already allocated; 182.75 MiB free; 14.26 GiB reserved in total by PyTorch)

It makes sense to me that model = model.to(device) creates 3.7G of memory.
But why does running the model output = model(input, comb) create another 3G of memory?
And then loss.backward() creates another 3G of memory?
And then optimizer.step() creates another 6.3G of memory?
I would appreciate it if someone could explain how the PyTorch GPU memory allocation model is working in this example.

Output:
Inference
By default, an inference on your model will allocate memory to store the activations of each layer (activation as in intermediate layer inputs). This is needed for backpropagation where those tensors are used to compute the gradients. A simple but effective example is a function defined by f: x -> x². Here, df/dx = 2x, i.e. in order to compute df/dx you are required to keep x in memory.
If you use the torch.no_grad() context manager, you will allow PyTorch to not save those values thus saving memory. This is particularly useful when evaluating or testing your model, i.e. when backpropagation is performed. Of course, you won't be able to use this during training!

Backward propagation
The backward pass call will allocate additional memory on the device to store each parameter's gradient value. Only leaf tensor nodes (model parameters and inputs) get their gradient stored in the grad attribute. This is why the memory usage is only increasing between the inference and backward calls.

Model parameter update
Since you are using a stateful optimizer (Adam), some additional memory is required to save some parameters. Read related PyTorch forum post on that. If you try with a stateless optimizer (for instance SGD) you should not have any memory overhead on the step call.

All three steps can have memory needs. In summary, the memory allocated on your device will effectively depend on three elements:

The size of your neural network: the bigger the model, the more layer activations and gradients will be saved in memory.

Whether you are under the torch.no_grad context: in this case, only the state of your model needs to be in memory (no activations or gradients necessary).

The type of optimizer used: whether it is stateful (saves some running estimates during parameter update, or stateless (doesn't require to).

whether you require to do back