Instruction: AttributeError: 'NoneType' object has no attribute 'shape' even though the tensor is not None

Input:
I'm working with a resnet-based model to generate some feature embeddings
feature = self.m_resnet(input)
when I print
print('feature:', feature)

I get output like,
tensor([[[-5.2228e-01, -2.6507e-01, -1.4583e+00,  ..., -1.1618e+00,
          -3.9355e-01, -6.7108e-01],
         [-5.0633e-01,  9.0730e-01, -7.6286e-01,  ..., -6.7644e-01,
          -6.4372e-01,  4.2130e-02],
         [ 1.3522e+00,  1.1739e+00,  1.1027e+00,  ...,  1.0143e+00,
           1.0382e+00,  5.5187e-01],
         ...,
         [ 4.6489e-01, -1.2791e-01,  1.1394e+00,  ..., -2.3228e-01,
          -4.3149e-01,  3.1564e-01],
         [ 1.0425e+00,  9.7971e-01, -3.5113e-01,  ...,  4.3813e-01,
           3.7757e-01,  3.0367e-01],
         [-9.2531e-01, -3.5561e-01, -1.9557e-01,  ...,  1.2157e-01,
          -4.4008e-01, -9.3977e-02]],

        [[-5.7037e-01, -2.3364e-01, -1.3321e+00,  ..., -1.2070e+00,
          -4.7131e-01, -5.4751e-01],
         [-2.8480e-01,  8.5945e-01, -5.6804e-01,  ..., -8.7505e-01,
          -5.9196e-01, -4.7775e-02],
         [ 1.4179e+00,  1.3121e+00,  1.1915e+00,  ...,  9.6185e-01,
           9.4094e-01,  6.2634e-01],
         ...,
         [ 4.7378e-01, -2.0151e-01,  1.0540e+00,  ..., -2.1641e-01,
          -4.2161e-01,  2.7364e-01],
         [ 1.0599e+00,  8.7958e-01, -1.3885e-01,  ...,  3.7642e-01,
           3.1348e-01,  2.2855e-01],
         [-8.3528e-01, -3.6043e-01, -4.1944e-02,  ...,  7.9550e-02,
          -3.3973e-01, -9.5777e-02]],

        [[-4.6509e-01, -3.1390e-01, -1.3608e+00,  ..., -1.1940e+00,
          -4.0954e-01, -6.8436e-01],
         [-3.8707e-01,  9.0035e-01, -8.3244e-01,  ..., -7.7861e-01,
          -6.3493e-01,  1.8479e-02],
         [ 1.3674e+00,  1.2385e+00,  1.1890e+00,  ...,  9.8861e-01,
           1.1610e+00,  6.2035e-01],
         ...,
         [ 5.5125e-01, -1.4365e-01,  1.2544e+00,  ..., -1.9578e-01,
          -5.9983e-01,  3.9633e-01],
         [ 1.0217e+00,  1.0091e+00, -3.5424e-01,  ...,  3.7400e-01,
           4.3592e-01,  2.3972e-01],
         [-8.4321e-01, -3.9549e-01, -1.7280e-01,  ...,  1.3756e-01,
          -5.2992e-01,  1.3601e-03]],

        [[-4.4482e-01, -3.2837e-01, -1.4795e+00,  ..., -1.2002e+00,
          -4.2828e-01, -7.6532e-01],
         [-4.5860e-01,  9.2647e-01, -8.4050e-01,  ..., -7.8178e-01,
          -6.4275e-01,  6.4469e-02],
         [ 1.3156e+00,  1.2003e+00,  1.2451e+00,  ...,  1.1086e+00,
           1.2272e+00,  7.0043e-01],
         ...,
         [ 5.1705e-01, -1.1838e-01,  1.1950e+00,  ..., -2.6755e-01,
          -6.0813e-01,  4.4875e-01],
         [ 1.0428e+00,  1.0369e+00, -4.6282e-01,  ...,  4.1102e-01,
           4.8060e-01,  2.7509e-01],
         [-8.9088e-01, -3.8145e-01, -3.3285e-01,  ..., -1.1576e-03,
          -5.0087e-01,  3.1878e-02]]], device='cuda:0',

but when I print the shape,
print('shape:', feature.shape)

I get error,AttributeError: 'NoneType' object has no attribute 'shape' on this print line.
According to above lines, feature is not None, then why I'm getting this error?

Output:
I don't see any other explanation than there is some code which runs between your first print command and your second and sets feature=None. Case in point:
>>> feature = "hello"
>>> print(feature)
hello
>>> feature = None
>>> print(feature.shape)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'shape'

If feature was still a tensor (or str or whatever), the error message should reflect that, like so:
>>> feature = "hello"
>>> print(feature)
hello
>>> print(feature.shape)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'shape'

If somehow feature was not in the scope and hadn't been defined at all, this should be the error message:
>>> print(feature_undefined.shape)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'feature_undefined' is not defined


Instruction: Pytorch - Selecting n indices without replacement from dimension x

Input:
Suppose I have the following embeddings emb_user = torch.randn(64, 128, 256). From the second dimension (of length 128), I wish to pick out 16 at random at each instance. I was wondering if there was a more efficient way of doing the following:
idx = torch.multinomial(torch.ones(64, 128), 16)
sampled_emb_user = emb_user[torch.arange(len(emb_user)).unsqueeze(-1), idx]

What I also find curios is that the above multinomial would not work if the weight matrix (torch.ones(64, 128)) exceeded more than 2 dimensions.

Output:
Since in your case you want an uniform distribution you could speed it up with
idx = torch.sort(torch.randint(
    0, 128 - 15, (64, 16), device=device
), axis=1).values + torch.arange(0, 16, device=device).reshape(1, -1)
sampled_emb_user = emb_user[torch.arange(len(emb_user)).unsqueeze(-1), idx]

Instead of
idx = torch.multinomial(torch.ones(64, 128, device=device), 16)
sampled_emb_user = emb_user[torch.arange(len(emb_user)).unsqueeze(-1), idx]

The runtimes on my machine are 427 µs and 784 µs with device='cpu'; 135 µs and 260 µs and 469 µs with device='cuda'.
How it works?
The sorted randint gives the indices for a multinomial distribution with replacement. That is increasing, adding the arange term makes it strictly increasing, thus eliminates the replacements.
Illustrating with a small case
idx = torch.sort(torch.randint(0, 7, (4,))).values
print('Indices with replacement in the range from 0 to 6: ', idx)
print('Indices without replacement in the slice: ', idx + torch.arange(4))

Indices with replacement in the range from 0 to 6:  tensor([0, 5, 5, 6])
Indices without replacement in the slice:  tensor([0, 6, 7, 9])

A possibly faster solution, but not from exactly the same distribution is the following:
idx = torch.cumsum(torch.diff(
torch.sort(torch.randint(
    0, 128 - 16, (64, 17), device=device
), axis=1).values
, axis=1) + 1, axis=1) - 1
sampled_emb_user = emb_user[torch.arange(len(emb_user)).unsqueeze(-1), idx]

One more way, I expect to be closer to the exact method, not very rigorously analyzed.
# 1-rand() to include 1 and exclude zero.
d = torch.cumsum(1 - torch.rand(64, 17, device=device
), axis=1)
# this produces a sorted tensor with values in the range [0:128-16]
d = (((128 - 15) * d[:, :-1]) / d[:, -1:]).to(torch.long)
idx = d + torch.arange(0, 16, device=device).reshape(1, -1)

But in the end it tends to be slower than the method using sort.


Instruction: Bad results using OpenCV with Yolov5 comparing with pure Yolov5

Input:
I'm tryting to recognize lego bricks from video cam using opencv. It performs extremely bad comparing with just running detect.py in Yolov5. Thus I made some experiments about just recognizing images, and I found using openCV still performs dramatically bad as well, is there any clue? Here are the experiments I did.
This is the result from detect.py by just running
python detect.py --weights runs/train/yolo/weights/best.pt --source legos.jpg

This is the result from openCV by implementing this
import torch
import cv2
import numpy as np

model = torch.hub.load('.', 'custom', path='runs/train/yolo/weights/last.pt', source='local')

cap = cv2.VideoCapture('legos.jpg')

while cap.isOpened():
    ret, frame = cap.read()

    # Make detections
    results = model(frame)

    cv2.imshow('YOLO', np.squeeze(results.render()))

    if cv2.waitKey(0) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

If I simply do this, it gives a pretty good result
import torch
results = model('legos.jpg')
results.show()

Any genious ideas?

Output:
Probably your model is trained with RGB images while opencv is using BGR format. Please try to convert the colour space accordingly. Example:
import torch
import cv2

model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# read image and convert to RGB
img = cv2.imread('zidane.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# make detections
results = model(img)

# render results and convert back to BGR
results.render()
out = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)

cv2.imshow('YOLO', out)

cv2.waitKey(-1)
cv2.destroyAllWindows()


Instruction: Dimension error by using Patch Embedding for video processing

Input:
I am working on one of the transformer models that has been proposed for video classification. My input tensor has the shape of [batch=16 ,channels=3 ,frames=16, H=224, W=224] and for applying the patch embedding on the input tensor it uses the following scenario:
patch_dim = in_channels * patch_size ** 2
self.to_patch_embedding = nn.Sequential(
        Rearrange('b t c (h p1) (w p2) -> b t (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),
        nn.Linear(patch_dim, dim),     ***** (Root of the error)******
    )

The parameters that I am using are as follows:
 patch_size =16
 dim = 192
 in_channels = 3

Unfortunately I receive the following error that corresponds to the line that has been shown in the code:
Exception has occured: RuntimeError
mat1 and mat2 shapes cannot be multiplied (9408x4096 and 768x192)

I thought a lot on the reason of the error but I couldn't find out what is the reason. How can I solve the problem?

Output:
The input tensor has shape [batch=16, channels=3, frames=16, H=224, W=224], while Rearrange expects dimensions in order [ b t c h w ]. You expect channels but pass frames. This leads to a last dimension of (p1 * p2 * c) = 16 * 16 * 16 = 4096.
Please try to align positions of channels and frames:
from torch import torch, nn
from einops.layers.torch import Rearrange

patch_size = 16
dim = 192

b, f, c, h, w = 16, 16, 3, 224, 224
input_tensor = torch.randn(b, f, c, h, w)

patch_dim = c * patch_size ** 2

m = nn.Sequential(
    Rearrange('b t c (h p1) (w p2) -> b t (h w) (p1 p2 c)', p1=patch_size, p2=patch_size),
    nn.Linear(patch_dim, dim)
)

print(m(input_tensor).size())

Output:
torch.Size([16, 16, 196, 192])


Instruction: In Pytorch, how do you multiply a (b, c, h, w) size tensor with a tensor of size (c)

Input:
I have to normalize a tensor of size size (b, c, h, w) with two tensors of size (c) which represent the respective mean and standard deviation.
I cannot manage to figure out how to multiply a tensor of shape, let say torch.Size([1, 3, 128, 128]) with a tensor of shape torch.Size([3]).
What I want to accomplish is: take the first element of the smaller tensor and multiply the first [128x128] part of the larger tensor with it. And do this for the second element and second [128x128] tensor etc.
def normalize(img, mean, std):
    """ Normalizes an image tensor.

    # Parameters:
        @img, torch.tensor of size (b, c, h, w)
        @mean, torch.tensor of size (c)
        @std, torch.tensor of size (c)

    # Returns the normalized image
    """
    # TODO: 1. Implement normalization doing channel-wise z-score normalization.

    img * mean                              #try1: this doesn't work
    torch.mul(img.view(3,128,128), mean)    #try2: this doesn't work

    return img

Both of my attempts throw the same error: RuntimeError: The size of tensor a (128) must match the size of tensor b (3) at non-singleton dimension 3.
I imagine you could create a tensor of the needed size, fill it with the values necessary and multiply that, but I would image there is a better solution than that.

Output:
img * mean.reshape(1,3,1,1)

Will reshape the mean tensor so that torch can understand which dimensions you are trying to multiply together.
Edit for details:
Torch reads tensor sizes from lowest to highest dimension, so it can infer some of the higher dimensions (e.g. img * mean.reshape(3,1,1) also works in your case), however you must specify the lower dimensions to either be one, or match the tensor you are trying to multiply with.


Instruction: Tensor repeat for image patches

Input:
I have a batch of 20 flattened tensors representing 256X256 images.
>>> imgs.shape
(20, 65536)

Each image was split into 32x32 patches (a total of 64 patches per image). I have calculated a score for each patch and got a vector with the shape of (20,64)
I would like to multiply each pixel with the corresponding patch score.
imgs * score yields an error and score.repeat(1,1,64) didn't repeat the scores in a way that preserves the score of each pixel.
How can this be achieved?
EDIT:
A simple example can be using
import torch
img_size = 4
patch_size = 2
img = torch.rand((2,img_size,img_size)) # (2,4,4)
score = torch.tensor([[1,2,3,4],[5,6,7,8]]) # (2,4)

And trying to achieve
score = [[1,1,3,3],[2,2,4,4],[5,5,6,6][7,7,8,8]]

Output:
I would suggest reshaping your scores array to preserve information about how it relates to the original image, then using repeat_interleave() twice.
Example:
import torch
img_size = 4
patch_size = 2
patches_per_axis = int(img_size / patch_size)
num_images = 2
img = torch.rand((2,img_size,img_size)) # (2,4,4)
score = torch.tensor([[1,2,3,4],[5,6,7,8]]) # (2,4)

def expand_scores(scores):
    # Unflatten scores
    scores = scores.reshape((num_images, patches_per_axis, patches_per_axis))
    # Repeat scores to match dimensions of image, in vertical direction
    scores = scores.repeat_interleave(repeats=patch_size, axis=1)
    # Repeat scores to match dimensions of image, in horizontal direction
    scores = scores.repeat_interleave(repeats=patch_size, axis=2)
    # Optional: use reshape() to re-flatten scores. If you do that here, you'll need to do it to the image tensor too.
    return scores

(I added two constants at the top to your example, num_images, and patches_per_axis. In your original example, these would be set to 20 and 8, respectively.)
When you call expand_scores(), you'll get the following output:
tensor([[[1, 1, 2, 2],
         [1, 1, 2, 2],
         [3, 3, 4, 4],
         [3, 3, 4, 4]],

        [[5, 5, 6, 6],
         [5, 5, 6, 6],
         [7, 7, 8, 8],
         [7, 7, 8, 8]]])

You can multiply that by the pixel values:
expand_scores(score) * img


Instruction: Efficiently sample batches from only one class at each iteration with PyTorch

Input:
I want to train a classifier on ImageNet dataset (1000 classes) and I need each batch to contain 64 images from the same class and consecutive batches from different classes. So far based on @shai's suggestion and this post I have
import torchvision.transforms as transforms
import torchvision.datasets as datasets
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
import numpy as np
import random
import argparse
import torch
import os

class DS(Dataset):
    def __init__(self, data, num_classes):
        super(DS, self).__init__()
        self.data = data

        self.indices = [[] for _ in range(num_classes)]
        for i, (data, class_label) in enumerate(data):
            # create a list of lists, where every sublist containts the indices of
            # the samples that belong to the class_label
            self.indices[class_label].append(i)

    def classes(self):
        return self.indices

    def __getitem__(self, index):
        return self.data[index]

class BatchSampler:
    def __init__(self, classes, batch_size):
        # classes is a list of lists where each sublist refers to a class and contains
        # the sample ids that belond to this class
        self.classes = classes
        self.n_batches = sum([len(x) for x in classes]) // batch_size
        self.min_class_size = min([len(x) for x in classes])
        self.batch_size = batch_size
        self.class_range = list(range(len(self.classes)))
        random.shuffle(self.class_range)

        assert batch_size < self.min_class_size, 'batch_size should be at least {}'.format(self.min_class_size)

    def __iter__(self):
        batches = []
        for j in range(self.n_batches):
            if j < len(self.class_range):
                batch_class = self.class_range[j]
            else:
                batch_class = random.choice(self.class_range)
            batches.append(np.random.choice(self.classes[batch_class], self.batch_size))
        return iter(batches)

def main():
    # Code about
    _train_dataset = DS(train_dataset, train_dataset.num_classes)
    _batch_sampler = BatchSampler(_train_dataset.classes(), batch_size=args.batch_size)
    _train_loader = DataLoader(dataset=_train_dataset, batch_sampler=_batch_sampler)
    labels = []
    for i, (inputs, _labels) in enumerate(_train_loader):
        labels.append(torch.unique(_labels).item())
        print("Unique labels: {}".format(torch.unique(_labels).item()))

    labels = set(labels)
    print('Length of traversed unique labels: {}'.format(len(labels)))

if __name__ == "__main__":

    parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')

    parser.add_argument('--data', metavar='DIR', nargs='?', default='imagenet',
                        help='path to dataset (default: imagenet)')
    parser.add_argument('--dummy', action='store_true', help="use fake data to benchmark")

    parser.add_argument('-b', '--batch-size', default=64, type=int,
                        metavar='N',
                        help='mini-batch size (default: 256), this is the total '
                             'batch size of all GPUs on the current node when '
                             'using Data Parallel or Distributed Data Parallel')

    parser.add_argument('-j', '--workers', default=4, type=int, metavar='N',
                        help='number of data loading workers (default: 4)')

    args = parser.parse_args()

    if args.dummy:
        print("=> Dummy data is used!")
        num_classes = 100
        train_dataset = datasets.FakeData(size=12811, image_size=(3, 224, 224),
                                          num_classes=num_classes, transform=transforms.ToTensor())
        val_dataset = datasets.FakeData(5000, (3, 224, 224), num_classes, transforms.ToTensor())
    else:
        traindir = os.path.join(args.data, 'train')
        valdir = os.path.join(args.data, 'val')

        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])

        train_dataset = datasets.ImageFolder(
            traindir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))

        val_dataset = datasets.ImageFolder(
            valdir,
            transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,
            ]))

    # Samplers are initialized to None and train_sampler will be replaced
    train_sampler, val_sampler = None, None
    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler)

    val_loader = torch.utils.data.DataLoader(
        val_dataset, batch_size=args.batch_size, shuffle=False,
        num_workers=args.workers, pin_memory=True, sampler=val_sampler)

    main()

which prints: Length of traversed unique labels: 100.
However, creating self.indices in the for loop takes a lot of time. Is there a more efficient way to construct this sampler?
EDIT: yield implementation
import torchvision.transforms as transforms
import torchvision.datasets as datasets
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
import numpy as np
import random
import argparse
import torch
import os
from tqdm import tqdm
import os.path

class DS(Dataset):
    def __init__(self, data, num_classes):
        super(DS, self).__init__()
        self.data = data
        self.data_len = len(data)

        indices = [[] for _ in range(num_classes)]

        for i, (_, class_label) in tqdm(enumerate(data), total=len(data), miniters=1,
                                        desc='Building class indices dataset..'):
            indices[class_label].append(i)

        self.indices = indices

    def per_class_sample_indices(self):
        return self.indices

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.data_len

class BatchSampler:
    def __init__(self, per_class_sample_indices, batch_size):
        # classes is a list of lists where each sublist refers to a class and contains
        # the sample ids that belond to this class
        self.per_class_sample_indices = per_class_sample_indices
        self.n_batches = sum([len(x) for x in per_class_sample_indices]) // batch_size
        self.min_class_size = min([len(x) for x in per_class_sample_indices])
        self.batch_size = batch_size
        self.class_range = list(range(len(self.per_class_sample_indices)))
        random.shuffle(self.class_range)

    def __iter__(self):
        for j in range(self.n_batches):
            if j < len(self.class_range):
                batch_class = self.class_range[j]
            else:
                batch_class = random.choice(self.class_range)
            if self.batch_size <= len(self.per_class_sample_indices[batch_class]):
                batch = np.random.choice(self.per_class_sample_indices[batch_class], self.batch_size)
                # batches.append(np.random.choice(self.per_class_sample_indices[batch_class], self.batch_size))
            else:
                batch = self.per_class_sample_indices[batch_class]
            yield batch

    def n_batches(self):
        return self.n_batches

def main():
    file_path = 'a_file_path'
    file_name = 'per_class_sample_indices.pt'
    if not os.path.exists(os.path.join(file_path, file_name)):
        print('File: {} does not exists. Create it.'.format(file_name))
        per_class_sample_indices = DS(train_dataset, num_classes).per_class_sample_indices()
        torch.save(per_class_sample_indices, os.path.join(file_path, file_name))
    else:
        per_class_sample_indices = torch.load(os.path.join(file_path, file_name))
        print('File: {} exists. Do not create it.'.format(file_name))

    batch_sampler = BatchSampler(per_class_sample_indices,
                                 batch_size=args.batch_size)

    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        # batch_size=args.batch_size,
        # shuffle=(train_sampler is None),
        num_workers=args.workers,
        pin_memory=True,
        # sampler=train_sampler,
        batch_sampler=batch_sampler
    )

    # We do not use sampler for the validation
    # val_loader = torch.utils.data.DataLoader(
    #     val_dataset, batch_size=args.batch_size, shuffle=False,
    #     num_workers=args.workers, pin_memory=True, sampler=None)

    labels = []
    for i, (inputs, _labels) in enumerate(train_loader):
        labels.append(torch.unique(_labels).item())
        print("Unique labels: {}".format(torch.unique(_labels).item()))

    labels = set(labels)
    print('Length of traversed unique labels: {}'.format(len(labels)))

if __name__ == "__main__":

    parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')

    parser.add_argument('--data', metavar='DIR', nargs='?', default='imagenet',
                        help='path to dataset (default: imagenet)')
    parser.add_argument('--dummy', action='store_true', help="use fake data to benchmark")

    parser.add_argument('-b', '--batch-size', default=64, type=int,
                        metavar='N',
                        help='mini-batch size (default: 256), this is the total '
                             'batch size of all GPUs on the current node when '
                             'using Data Parallel or Distributed Data Parallel')

    parser.add_argument('-j', '--workers', default=4, type=int, metavar='N',
                        help='number of data loading workers (default: 4)')

    args = parser.parse_args()

    if args.dummy:
        print("=> Dummy data is used!")
        num_classes = 100
        train_dataset = datasets.FakeData(size=12811, image_size=(3, 224, 224),
                                          num_classes=num_classes, transform=transforms.ToTensor())
        val_dataset = datasets.FakeData(5000, (3, 224, 224), num_classes, transforms.ToTensor())
    else:
        traindir = os.path.join(args.data, 'train')
        valdir = os.path.join(args.data, 'val')

        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])

        train_dataset = datasets.ImageFolder(
            traindir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))

        val_dataset = datasets.ImageFolder(
            valdir,
            transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,
            ]))
        num_classes = len(train_dataset.classes)

    main()

A similar post but in TensorFlow can be found here

Output: You should write your own batch_sampler class for the DataLoader.


Instruction: Why are linear layers used in Binary Classification with Deep Learning?

Input:
In many examples of Binary Classification with Deep learning
Why are linear layers used? I've been trying to look around the internet for information on the reason for the use of linear layers
e.g.
https://github.com/StatsGary/PyTorch_Tutorials/blob/main/01_MLP_Thyroid_Classifier/PyTorch_Binary_From_Scratch.py
https://hutsons-hacks.info/building-a-pytorch-binary-classification-multi-layer-perceptron-from-the-ground-up

Output: Linear layer is just another (a bit mathematically incorrect) name of a fully connected layer, the most standard, classic, and in some sense - powerful building block of neural networks. Networks built purely from fully connected layers are universal approximators, and thus a good starting point for any sort of investigation.


Instruction: How to use Huggingface Trainer streaming Datasets without wrapping it with torchdata's IterableWrapper?

Input:
Given a datasets.iterable_dataset.IterableDataset with stream=True, e.g.
train_data = load_dataset("csv", data_files="../input/tatoeba/tatoeba-sentpairs.tsv",
                  streaming=True, delimiter="\t", split="train")

and trying to use it in a Trainer object, e.g.
# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=train_data,
)

trainer.train()

It throws an error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_27/3002801805.py in <module>
     28 )
     29
---> 30 trainer.train()

/opt/conda/lib/python3.7/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1411             resume_from_checkpoint=resume_from_checkpoint,
   1412             trial=trial,
-> 1413             ignore_keys_for_eval=ignore_keys_for_eval,
   1414         )
   1415

/opt/conda/lib/python3.7/site-packages/transformers/trainer.py in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1623
   1624             step = -1
-> 1625             for step, inputs in enumerate(epoch_iterator):
   1626
   1627                 # Skip past any already trained steps if resuming training

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    528             if self._sampler_iter is None:
    529                 self._reset()
--> 530             data = self._next_data()
    531             self._num_yielded += 1
    532             if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    567
    568     def _next_data(self):
--> 569         index = self._next_index()  # may raise StopIteration
    570         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    571         if self._pin_memory:

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_index(self)
    519
    520     def _next_index(self):
--> 521         return next(self._sampler_iter)  # may raise StopIteration
    522
    523     def _next_data(self):

/opt/conda/lib/python3.7/site-packages/torch/utils/data/sampler.py in __iter__(self)
    224     def __iter__(self) -> Iterator[List[int]]:
    225         batch = []
--> 226         for idx in self.sampler:
    227             batch.append(idx)
    228             if len(batch) == self.batch_size:

/opt/conda/lib/python3.7/site-packages/torch/utils/data/sampler.py in __iter__(self)
     64
     65     def __iter__(self) -> Iterator[int]:
---> 66         return iter(range(len(self.data_source)))
     67
     68     def __len__(self) -> int:

TypeError: object of type 'IterableDataset' has no len()

This can be resolved by wrapping the IterableDataset object with the IterableWrapper from torchdata library.
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper

...

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=IterableWrapper(train_data),
    eval_dataset=IterableWrapper(train_data),
)

trainer.train()

Is it possible to use the IterableDataset with Seq2SeqTrainer without casting it with IterableWrapper?

For reference, a full working code would look something as below, replacing the line where train_dataset=IterableWrapper(train_data) to train_dataset=train_data will replicate the TypeError: object of type 'IterableDataset' has no len() error.
import torch

from datasets import load_dataset
from transformers import EncoderDecoderModel
from transformers import AutoTokenizer
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

from torchdata.datapipes.iter import IterDataPipe, IterableWrapper

multibert = EncoderDecoderModel.from_encoder_decoder_pretrained(
    "bert-base-multilingual-uncased", "bert-base-multilingual-uncased"
)
tokenizer= AutoTokenizer.from_pretrained("bert-base-multilingual-uncased")
tokenizer.bos_token = tokenizer.cls_token
tokenizer.eos_token = tokenizer.sep_token
tokenizer.add_special_tokens({'pad_token': '[PAD]'})

# set special tokens
multibert.config.decoder_start_token_id = tokenizer.bos_token_id
multibert.config.eos_token_id = tokenizer.eos_token_id
multibert.config.pad_token_id = tokenizer.pad_token_id

# sensible parameters for beam search
multibert.config.vocab_size = multibert.config.decoder.vocab_size

def process_data_to_model_inputs(batch, max_len=10):
    inputs = tokenizer(batch["SRC"], padding="max_length",
                       truncation=True, max_length=max_len)
    outputs = tokenizer(batch["TRG"], padding="max_length",
                        truncation=True, max_length=max_len)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["decoder_attention_mask"] = outputs.attention_mask
    batch["labels"] = outputs.input_ids.copy()

    # because BERT automatically shifts the labels, the labels correspond exactly to `decoder_input_ids`.
    # We have to make sure that the PAD token is ignored
    batch["labels"] = [[-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]]

    return batch

# tatoeba-sentpairs.tsv is a pretty large file.
train_data = load_dataset("csv", data_files="../input/tatoeba/tatoeba-sentpairs.tsv",
                  streaming=True, delimiter="\t", split="train")

train_data = ds.map(process_data_to_model_inputs, batched=True)

batch_size = 1

# set training arguments - these params are not really tuned, feel free to change
training_args = Seq2SeqTrainingArguments(
    output_dir="./",
    evaluation_strategy="steps",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_with_generate=True,
    logging_steps=2,  # set to 1000 for full training
    save_steps=16,    # set to 500 for full training
    eval_steps=4,     # set to 8000 for full training
    warmup_steps=1,   # set to 2000 for full training
    max_steps=16,     # delete for full training
    # overwrite_output_dir=True,
    save_total_limit=1,
    #fp16=True,
)

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=IterableWrapper(train_data),
    eval_dataset=IterableWrapper(train_data),
)

trainer.train()

Output:
Found the answer from https://discuss.huggingface.co/t/using-iterabledataset-with-trainer-iterabledataset-has-no-len/15790
By adding the with format to the iterable dataset, like this:
train_data.with_format("torch")

The trainer should work without throwing the len() error.
# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_data.with_format("torch"),
    eval_dataset=train_data.with_format("torch"),
)

trainer.train()


Instruction: How to turn a cifar10 dataset into a tensor

Input:
I am trying to turn the cifar10 dataset into a tensor with
trainset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transforms.ToTensor())
but it continually returns false when I run it torch.is_tensor(trainset) function which means it's not a tensor and it also doesn't work for functions that require tensors in the code I'm running on.
I tried to print out the trainset with
print(trainset)
and I keep getting

Dataset CIFAR10
Number of datapoints: 50000
Root location: ./data
Split: Train
StandardTransform
Transform: ToTensor()

which means it's not yet a tensor.
How exactly can I convert the entire cifar10 dataset to a tensor?

Output:
trainset is a Dataset instance and you do not convert it to a tensor. You should load the data and then transform it.
for i, data in enumerate(trainset, 0):
    do whatever


Instruction: How to apply torch.topk only on non-zero elements of a tensor?

Input:
I want to apply the function torch.topk, but only on the non-zero elements of the tensor (i.e, not to count zero elements in the counting process).
Currently I do this:
torch.topk(tensor.view(-1), k)

But this also considers the zero elements in variable tensor and returns the top largest among them. What should I do to get the topk among non-zero elements?

Output:
# get the top k values in a tensor excluding zeros
top_vals = torch.topk(tensor.view(-1), k)[0]
mask = top_vals != 0
values = top_vals[mask]
print(values)

# Get Indices of Top K Values
indices = torch.nonzero(mask)
print(indices)

credits: this question and its answers


Instruction: Pytorch: RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 1

Input:
The pytorch program needs to return the result of the "Rock, Paper, Scissors" game. Input is given as one-hot tensors: [ [1,0,0], [0,1,0] ] ([1,0,0]-rock, [0,1,0] - scissors) output must be: [1, 0] (first player win, second player lose). What's wrong with this code?
import torch
from torch import nn
import torch.utils.data as data
torch.manual_seed(42)

input = [[1, 0, 0], [0, 1, 0]], [[1, 0, 0], [0, 0, 1]], [[0, 1, 0], [0, 1, 0]], [[0, 0, 1], [1, 0, 0]]
input = torch.tensor(input, dtype=torch.float32)
result = [[1, 0], [0, 1], [1, 1], [1, 0]]
result = torch.tensor(result, dtype=torch.float32)

class LinearRegression(nn.Module):
    def __init__(self, num_inputs, num_outputs):
        super().__init__()
        self.linear = nn.Linear(num_inputs, num_outputs)
        self.act_fn = nn.Sigmoid()
    def forward(self, x):
        x = self.linear(x)
        x = self.act_fn(x)
        return x

model = LinearRegression(num_inputs=3, num_outputs=2)
print(model)
# Training loop
for name, param in model.named_parameters():
    print(f"Parameter {name}, shape {param.shape}")

for x in input:
  print(model(x))

print(model(input))
print('####################')
model.train()
optimizer = torch.optim.SGD(model.parameters(), lr=0.3)
lossfunc = nn.MSELoss()
# Training loop
for _ in range(1000):
    res = model(input)
    res = res.squeeze(dim=1)

    loss = lossfunc(res, result)
    # print(loss)

    ## Backpropagation

    optimizer.zero_grad()
    loss.backward()

    optimizer.step()

print(model(input))

Output:
you have the following problem: You want your model to get as input the rock/paper/scissors of player1 and the rock/paper/scissors of player2. Since you do one-hot encoding you want to input 6 values into your neural network. 3 for player1 and 3 for player2. But you are trying to
Hey, a dense/feed-forward neural networl can only take ONE vector as input. But you are trying to pass two vectors of size three [[1, 0, 0], [0, 1, 0]]. What you can/should do instead is to concat both vectors. That means the first 3 values is the one-hot encoded result of player1 and the last 3 values are the one-hot encoded result of player2. Like this: [1, 0, 0, 0, 1, 0].
Therefore simply change the input data to this:
input = [[1, 0, 0, 0, 1, 0]], [[1, 0, 0, 0, 0, 1]], [[0, 1, 0, 0, 1, 0]], [[0, 0, 1, 1, 0, 0]]

and the model to this:
model = LinearRegression(num_inputs=6, num_outputs=2)

And it works!


Instruction: ImportError: libtinfo.so.5: cannot open shared object file: No such file or directory

Input:
On an Ubuntu-based system, I got this error that I didn't have before in an existing FastAI Python project.
Traceback:
Traceback (most recent call last):
  File "/home/me/PycharmProjects/project/model/predict.py", line 6, in <module>
    from fastai.vision.all import *
  File "/home/me/miniconda3/envs/venv/lib/python3.9/site-packages/fastai/vision/all.py", line 1, in <module>
    from . import models
  File "/home/me/miniconda3/envs/venv/lib/python3.9/site-packages/fastai/vision/models/__init__.py", line 1, in <module>
    from . import xresnet
  File "/home/me/miniconda3/envs/venv/lib/python3.9/site-packages/fastai/vision/models/xresnet.py", line 17, in <module>
    from ...torch_basics import *
  File "/home/me/miniconda3/envs/venv/lib/python3.9/site-packages/fastai/torch_basics.py", line 1, in <module>
    from torch import multiprocessing
  File "/home/me/miniconda3/envs/venv/lib/python3.9/site-packages/torch/__init__.py", line 190, in <module>
    from torch._C import *
ImportError: libtinfo.so.5: cannot open shared object file: No such file or directory

Output:
Terminal:
sudo apt-get install libtinfo5


Instruction: Loss function gradient not computing - FastAI Convolutional VAE

Input:
I modified the example from the Pytorch VAE example to be a convolutional network. I then wanted to implement this in FastAI.
class convVAE(nn.Module):
def __init__(self, dim_z=20):

    super(convVAE, self).__init__()

    self.cv1 = nn.Conv2d(1, 32, 3, stride=2)
    self.cv2 = nn.Conv2d(32, 64, 3, stride=2)
    self.fc31 = nn.Linear(2304, dim_z)
    self.fc32 = nn.Linear(2304, dim_z)
    self.fc4 = nn.Linear(dim_z, 2304)
    self.cv5 = nn.ConvTranspose2d(64, 32, 3, stride=2)
    self.cv6 = nn.ConvTranspose2d(32, 1, 3, stride=2, output_padding=1)

def encode(self, x):
    h1 = F.leaky_relu(self.cv1(x))
    h2 = F.leaky_relu(self.cv2(h1)).view(-1, 2304)

    return self.fc31(h2), self.fc32(h2)

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5*logvar)
    eps = torch.randn_like(std)
    return mu + eps*std

def decode(self, z):
    h5 = F.leaky_relu(self.fc4(z)).view(-1, 64, 6, 6)
    h6 = F.leaky_relu(self.cv5(h5))
    return torch.sigmoid(self.cv6(h6))

def forward(self, x):
    mu, logvar = self.encode(x)
    z = self.reparameterize(mu, logvar)
    return self.decode(z).view(-1, 784), mu, logvar

def get_loss(res,y):
    y_hat, mu, logvar = res

    BCE = F.binary_cross_entropy(
        y.view(-1, 784),
        y_hat,
        reduction='sum')

    KLD = -0.5 * torch.sum(1 + logvar -
                           mu.pow(2) - logvar.exp())

    return BCE + KLD
block = DataBlock(
blocks=(ImageBlock(cls=PILImageBW),ImageBlock(cls=PILImageBW)),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=(lambda x: x),
batch_tfms=aug_transforms(mult=2., do_flip=False))

path = untar_data(URLs.MNIST)
loaders = block.dataloaders(path/“training”,num_workers=0,bs=32)
loaders.train.show_batch(max_n=4, nrows=1)

mdl = convVAE(5)
learn = Learner(loaders, mdl, loss_func = convVAE.get_loss)
learn.fit(1, cbs=ShortEpochCallback())

The gradient is not computing from the loss, as the parameters all become NaN after one step. The loss function does compute but was relatively large O(1e6). The model and loss function work in the native Pytorch implementation.
EDIT: SOLUTION APPEARS TO HAVE BEEN DUE TO def init(.) instead of def __init__(.) facepalm

Output:
There is a mistake in your BCE calculation:
BCE = F.binary_cross_entropy(
    y.view(-1, 784),  # this should be your model prediction
    y_hat,  # this should be the ground truth
    reduction='sum')

A simple fix is to swap the two arguments.


Instruction: Append or combine column wise torch tensors of different shapes

Input:
Tensors A and B below share the row size. The 6 and 3 refers to the columns of the 2 data frame, but the B tensor has at each cell a vector of size 256.
A= torch.Size([17809, 6])
B= torch.Size([17809, 3, 256])

How do I append combine these tensors?
More detail:
A column of 'A' is a numeric vector like 'Age', In B one of the 3 columns has a set of node embeddings (vector) of size 256.

Output:
You can apply torch.nn.Embedding on A to embedding numeric vector then use torch.cat to concat embeding of A and B on the axis=1.
(In the below code I use random tensors).
import torch
from torch import nn

num_embeddings = 10 # fill base Age
embedding_dim = 256 # fill base of B tensor
embedding = nn.Embedding(num_embeddings, embedding_dim)

A = torch.randint(10, (17809, 6))
print(f"A   : {A.shape}")

E_A = embedding(A)
print(f"E_A : {E_A.shape}")

B = torch.rand(17809, 3, 256)
print(f"B   : {B.shape}")

C = torch.cat((E_A, B), 1)
print(f"C   : {C.shape}")

Output:
A   : torch.Size([17809, 6])
E_A : torch.Size([17809, 6, 256])
B   : torch.Size([17809, 3, 256])
C   : torch.Size([17809, 9, 256])


Instruction: AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'

Input:
I am trying to load the dataset using Torch Dataset and DataLoader, but I got the following error:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'

the code I use is:
class WineDataset(Dataset):

    def __init__(self):
        # Initialize data, download, etc.
        # read with numpy or pandas
        xy = np.loadtxt('./data/wine.csv', delimiter=',', dtype=np.float32, skiprows=1)
        self.n_samples = xy.shape[0]

        # here the first column is the class label, the rest are the features
        self.x_data = torch.from_numpy(xy[:, 1:]) # size [n_samples, n_features]
        self.y_data = torch.from_numpy(xy[:, [0]]) # size [n_samples, 1]

    # support indexing such that dataset[i] can be used to get i-th sample
    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    # we can call len(dataset) to return the size
    def __len__(self):
        return self.n_samples

    dataset = WineDataset()

    train_loader = DataLoader(dataset=dataset,
                              batch_size=4,
                              shuffle=True,
                              num_workers=2)

I tried to make the num_workers=0, still have the same error.
Python version 3.8.9
PyTorch version 1.13.0

Output:
I too faced the same issue, when i tried to call the next() method as follows
dataiter = iter(dataloader)
data = dataiter.next()

You need to use the following instead and it works perfectly:
dataiter = iter(dataloader)
data = next(dataiter)

Finally your code should look like follows:
class WineDataset(Dataset):

    def __init__(self):
        # Initialize data, download, etc.
        # read with numpy or pandas
        xy = np.loadtxt('./data/wine.csv', delimiter=',', dtype=np.float32, skiprows=1)
        self.n_samples = xy.shape[0]

        # here the first column is the class label, the rest are the features
        self.x_data = torch.from_numpy(xy[:, 1:]) # size [n_samples, n_features]
        self.y_data = torch.from_numpy(xy[:, [0]]) # size [n_samples, 1]

    # support indexing such that dataset[i] can be used to get i-th sample
    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    # we can call len(dataset) to return the size
    def __len__(self):
        return self.n_samples

    dataset = WineDataset()

    train_loader = DataLoader(dataset=dataset,
                              batch_size=4,
                              shuffle=True,
                              num_workers=2)

dataiter = iter(dataloader)
data = next(dataiter)


Instruction: TypeError: unsupported operand type(s) for /: 'SequenceClassifierOutput' and 'int'

Input:
I am using hugginface library to train a bert model on classification problem.
    model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=10)

    def training_step(self, batch, batch_nb):
       sequence, label = batch
       input_ids, attention_mask, labels = self.prepare_batch(sequence=sequence, label=label)
       loss = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels)
       tensorboard_logs = {'train_loss': loss}

I am getting the following error just before the training starts:
in training_step
closure_loss = closure_loss / self.trainer.accumulate_grad_batches
TypeError: unsupported operand type(s) for /: 'SequenceClassifierOutput' and 'int'

I am using pytorch-lightning

Output:
Calling self.model() returns an object of type SequenceClassifierOutput.
To access the loss you need to call it's loss attribute:
Replace
loss = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels)

by
output = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels)
loss = output.loss


Instruction: Does optuna.integration.TorchDistributedTrial support multinode optimization?

Input:
Does integration.TorchDistributedTrial support multinode optimization?
I'm using Optuna on a SLURM cluster. Suppose I would like to do a distributed hyperparameter optimization using two nodes with two gpus each. Would submitting a script like pytorch_distributed_simple.py to multiple nodes yield expected results?
I assume every node would be responsible for executing their own trials (i.e. no nodes share trials) and every gpu on a node is responsible for its own portion of the data, determined by torch.utils.data.Dataloader's sampler. Is this assumption correct or are edits needed apart from TorchDistributedTrial's requirement to pass None to objective calls on ranks other than 0.
I already tried the above, but I'm not sure how to check every node is responsible for distinct trials.

GitHub issues crosspost

Output:
Apparently, Optuna does allow multiple Optuna processes to do distributed runs. Why wouldn't it :)
Basically, run pytorch_distributed_simple.py on multiple nodes (I use SLURM for this) and make sure every subprocess calls the trial.report() method. Every node is now responsible for its own trial. Trials can use DDP.
My method differs from the provided code in that I use SLURM (different environment variables) and I use sqlite to store study information. Moreover, I use the NCCL backend to initialize process groups, and therefore need to pass a device to TorchDistributedTrial.

Unrelated, but I also wanted to call MaxTrialsCallback() in every subprocess. To achieve this, I passed the callback to the rank 0 study.optimizer method and call it explicitly in local non-rank 0 processes after the objective call.


Instruction: PyTorch runs as expected under Windows, but fails on larger images under Ubuntu

Input:
I've trained a segmentation_models_pytorch.PSPNet model for image segmentation. For prediction I load whole image in PyTorch tensor and scan it with 384x384 pixels window.
result = model.predict(image_tensor[:, :, y:y+384, x:x+384])

My Windows machine has 6Gb GPU, while Ubuntu has 8 Gb GPU. When all models are loaded they consume some 1.4 Gb GPU. When processing a large image on Windows the memory consumption increases to 1.7 Gb GPU.
Under Windows the model can handle 25 M pixel images. Under Ubuntu the same code can only process up to 5 M pixel image. Debugging is difficult because I only have ssh access to the Ubuntu machine.
What could cause this discrepancy and how to debug this issue?

Output:
You can check cuda version, torch version.
To debug, you can just print image_tensor.shape before model.predict, maybe you are running a larger batch size on linux machine.


Instruction: How to remove an element from the torch tensor list?

Input:
I have torch tensor list that looks like below
tensor([[[-1.8510e-01,  1.3181e-01,  3.2903e-01,  ...,  1.9867e-01,
           5.1037e-03,  6.4071e-03],
         [-4.6331e-01,  2.0216e-01,  2.7916e-01,  ...,  2.6695e-01,
          -1.3543e-02,  5.3604e-02],
         [-3.8719e-01,  2.9603e-01,  2.5516e-01,  ...,  1.7509e-01,
           8.9148e-02,  3.7516e-02],

and the shape of this torch tensor is [500, 197, 768]
There are 500 images with 197*768 dimensions. I need to remove the instance of some images. Lets say if I remove 5 images then the shape will be [495, 197, 768]
Can anyone tell me how to remove this using index?

Output:
It depends on where along that dimension you want to remove the items.
To remove the first/last n elements (using normal Python indexing):
new_data = data[n:] # Remove first n elements
new_data = data[:-n] # Remove last n elements

To remove n items inside the tensor, you will need to specify a start-index s (s+n should not be larger than the length along that dimension):
new_data = torch.cat((data[:s], data[s+n:]), dim=0) # Remove n elements starting at s

To remove using a list of indices you could do the following:
indices = [6, 7, 9, 100, 204] # Arbitrary list of indices that should be removed
indices_to_keep = [i for i in range(data.shape[0]) if i not in indices] # List of indices that should be kept
new_data = data[torch.LongTensor(indices_to_keep)]


Instruction: Invalid stoi argument with torch

Input:
I've been tackling python and torch specifically lately as a hobby, and, while some API works, I keep getting invalid stoi argument exception with other very basic API torch provides.
Reproduced with the code below:
import torch
torch.cuda.is_available()
torch.cuda.current_device()

First call (is_available()) works as expected and returns True, but the second throws an exception:
Exception has occurred: RuntimeError
invalid stoi argument
  File "C:\DEV\pthon_test\torch_test.py", line 5, in <module>
    torch.cuda.current_device()

Needless to say, more complicated things (for example, running stable_diffusion_webui) fail if used with GPU (said webui works with CPU), and trying to dig deeper into the code brings me to the same exception.
OS is Windows 11, python Python 3.10.8, torch version checked with torch.__version__ returns 1.12.1+cu113. And, well, GPU is present
List of packages installed:
❯ pip list
Package                 Version
----------------------- ---------------
absl-py                 1.3.0
addict                  2.4.0
antlr4-python3-runtime  4.9.3
basicsr                 1.4.2
beautifulsoup4          4.11.1
cachetools              5.2.0
certifi                 2022.9.24
charset-normalizer      2.1.1
clip                    1.0
colorama                0.4.6
contourpy               1.0.6
cycler                  0.11.0
einops                  0.4.1
facexlib                0.2.5
ffmpy                   0.3.0
filelock                3.8.0
filterpy                1.4.5
font-roboto             0.0.1
fonts                   0.0.3
fonttools               4.38.0
ftfy                    6.1.1
future                  0.18.2
gdown                   4.5.3
gfpgan                  1.3.5
google-auth             2.14.0
google-auth-oauthlib    0.4.6
grpcio                  1.50.0
idna                    3.4
imageio                 2.22.3
kiwisolver              1.4.4
lark                    1.1.2
llvmlite                0.39.1
lmdb                    1.3.0
lpips                   0.1.4
Markdown                3.4.1
MarkupSafe              2.1.1
matplotlib              3.6.2
networkx                2.8.8
numba                   0.56.3
numpy                   1.23.3
oauthlib                3.2.2
omegaconf               2.2.3
opencv-python           4.6.0.66
orjson                  3.8.1
packaging               21.3
piexif                  1.1.3
Pillow                  9.2.0
pip                     22.2.2
protobuf                3.19.6
pyasn1                  0.4.8
pyasn1-modules          0.2.8
pycparser               2.21
pycryptodome            3.15.0
pydantic                1.10.2
pyDeprecate             0.3.2
pydub                   0.25.1
pyparsing               3.0.9
pyrsistent              0.19.2
PySocks                 1.7.1
python-dateutil         2.8.2
python-multipart        0.0.4
pytz                    2022.6
PyWavelets              1.4.1
PyYAML                  6.0
regex                   2022.10.31
requests                2.28.1
requests-oauthlib       1.3.1
resize-right            0.0.2
rfc3986                 1.5.0
rsa                     4.9
scikit-image            0.19.3
scipy                   1.9.3
setuptools              63.2.0
six                     1.16.0
sniffio                 1.3.0
soupsieve               2.3.2.post1
tb-nightly              2.11.0a20221103
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
tifffile                2022.10.10
tokenizers              0.12.1
torch                   1.12.1+cu113
torchvision             0.13.1+cu113
tqdm                    4.64.1
typing_extensions       4.4.0
uc-micro-py             1.0.1
urllib3                 1.26.12
wcwidth                 0.2.5
websockets              10.4
Werkzeug                2.2.2
wheel                   0.37.1
yapf                    0.32.0
zipp                    3.10.0

This brings me to two questions:

What's causing the issue? I do have a rough understanding of the c++ meaning of the error, but not sure why I'm getting this on python
Is there anything I can do to fix the problem?

Thanks

Output:
As there is no other answers, and initial problem seem to have been solved, I thought I'd share some information.
I've noticed the problem was fixed when... I updated my Nvidia driver. No idea what was the problem, and if there's anything else I unknowingly did, but updating to 526.98 did the trick for me.
If there's anyone else to share more details on the original issue, or possible solutions - feel free to edit or give a better answer, I'd mark that as the best option instead.
Have a nice day o/


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x32 and 400x120)

Input:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        #(input channel, output channel, kenel size)
        #channel is a dimension of a tensor which is a container that can house data in N dimensions (matrices)
        self.conv1 = nn.Conv2d(3, 6, 5)
        #shrink the image stack by pooling(kernel size, stride(shift)) and take max value per window
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        #TODO: add conv3
        self.conv3 = nn.Conv2d(16, 32, 5)
        #drop layer deletes 20% of the feautures to help prevent overfitting
        self.drop = nn.Dropout2d(p=0.2)
        #linear predicts the output as a linear function of inputs
        #(output channels, height, width, batch size
        #TODO:
        self.fc1 = nn.Linear(16 * 16 * 5, 120)
        #TODO:
        self.fc1_5 = nn.Linear()
        #layer(size of input, size of output)
        #Linear layer=Fully connected layer
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        #F.ReLUs change negative values to 0. Apply to all stack of images.
        #they are activation functions. We apply it after each liner layer.
        #only used in hidden layers.
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        #Select some feautures to drop after 3rd conv to prevent overfitting
        x = self.drop(F.relu(self.conv3(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch into 1-D
        x = F.relu(self.fc1(x))
        #TODO: add fc1_5
        x = F.relu(self.fc1_5(x))
        x = F.relu(self.fc2(x))
        #Feed to Fully connected layer to predict class
        x = self.fc3(x) # no relu b/c it's a last layer.
        return x

I am using images from CIFAR10 which are of size 3x32x32.
When I ran the code before, it stopped because self.fc1 linear layer size did not work with self.conv3 I've added.
I'm also not sure what to write for self.fc1_5.
Can someone explain me how this is actually working and the solution as well?
Thank you!
I have added an extra convolutional layer and you can see it is
self.conv3 = nn.Conv2d(16, 32, 5).
Lines under the TODO are where I'm stuck at.
I updated the line to:
self.fc1 = nn.Linear(16 * 16 * 5, 120)
before, it was:
self.fc1 = nn.Linear(16 * 5 * 5, 120).

Output:
When you create a CNN for classification with a fixed input size, it's easy to figure out the size of your image by the time it has progressed through your CNN layers. Since we start with images of size [32,32] (channels are unimportant for now):
def __init__(self):
    super().__init__()
    #(input channel, output channel, kenel size)
    #channel is a dimension of a tensor which is a container that can house data in N dimensions (matrices)
    self.conv1 = nn.Conv2d(3, 6, 5) # size 28x28 - lose 2 px from each side with a kernel of size 5
    #shrink the image stack by pooling(kernel size, stride(shift)) and take max value per window
    self.pool = nn.MaxPool2d(2, 2) # size 14x14 - max pooling with K=2 halves the image size
    self.conv2 = nn.Conv2d(6, 16, 5) # size 10x10 -> 5x5 after pooling
    #TODO: add conv3
    self.conv3 = nn.Conv2d(16, 32, 5) # size 1x1
    #drop layer deletes 20% of the feautures to help prevent overfitting
    self.drop = nn.Dropout2d(p=0.2)
    #linear predicts the output as a linear function of inputs
    #(output channels, height, width, batch size
    self.fc1 = nn.Linear(1 * 1 * 32, 120)
    self.fc1_5 = nn.Linear(120,120) # matches the output size of fc1 and input size of fc2

The CNN size losses can be negated by using padding of (K-1)//2, where K=kernel_size.


Instruction: How to add shared or static library using PyTorch C++ extension?

Input:
How do I use torch.utils.cpp_extension.load to link shared or static library from external source?
I wrote some function in C++, and am using it in PyTorch. So I am using load function from torch.utils.cpp_extension to load a PyTorch C++ extension just-in-time (JIT).
This is the wrapper.py file's content:
import os
from torch.utils.cpp_extension import load

dir_path = os.path.dirname(os.path.realpath(__file__))

my_func = load(name='my_func', sources=[os.path.join(dir_path, 'my_func.cpp')], extra_cflags=['-fopenmp', '-O2'], extra_ldflags=['-lgomp','-lrt'])

my_func.cpp uses OpenMP, so I use the above flags.
Now, I am trying to additionally use several functions in zstd library in my_func.cpp. After cloning and makeing zstd repository, shared libraries like libzstd.so, libzstd.so.1, libzstd.so.1.5.3, and static library like libzstd.a have been created.
I've included #include <zstd.h> inside my_func.cpp and used zstd's functions.
I now have to modify wrapper.py to tell the compiler that I am using functions from zstd library.
How can I successfully compile my_func.cpp using PyTorch C++ extension's torch.utils.cpp_extension.load -- which arguments should I modify? Or, is it even possible to add external shared or static library using this method?
Frankly, I'm not familiar with the difference between static and shared library. But it seems that I can compile my_func.cpp with either one of them, i.e., g++ -fopenmp -O2 -lgomp -lrt -o my_func my_func.cpp lib/libzstd.so.1.5.3 and g++ -fopenmp -O2 -lgomp -lrt -o my_func my_func.cpp lib/libzstd.a both works.
I just can't figure out how I can do the exact same compiling using torch.utils.cpp_extension.load.
Sorry for delivering kind of a lengthy question. I just wanted to make things clear.

Output: I've figured this out. extra_ldflags argument in torch.utils.cpp_extension.load can handle this. In my case, I've added libzstd.so file in my repository and added -lzstd in above argument.


Instruction: FastAI Multilayer LSTM not learning, accuracy decreases while training

Input:
I'm following Chapter 12 on RNNs/LSTMs from scratch in the fastai book, but getting stuck trying to train a custom built LSTM from scratch. Here is my code
This is the boilerplate bit (following the examples in the book)
from fastai.text.all import *
path = untar_data(URLs.HUMAN_NUMBERS)

lines = L()
with open(path/'train.txt') as f: lines += L(*f.readlines())
with open(path/'valid.txt') as f: lines += L(*f.readlines())

text = ' . '.join([l.strip() for l in lines])
tokens = text.split(' ')
vocab = L(*tokens).unique()
word2idx = {w:i for i,w in enumerate(vocab)}
nums = L(word2idx[i] for i in tokens)

def group_chunks(ds, bs):
    m = len(ds) // bs
    new_ds = L()
    for i in range(m): new_ds += L(ds[i + m*j] for j in range(bs))
    return new_ds

sl = 3
bs = 64
seqs = L((tensor(nums[i:i+sl]), nums[i+sl])
         for i in range(0,len(nums)-sl-1,sl))
cut = int(len(seqs) * 0.8)
dls = DataLoaders.from_dsets(group_chunks(seqs[:cut], bs),
                             group_chunks(seqs[cut:], bs),
                             bs=bs, drop_last=True, shuffle=False)

And this is the meat of the thing
class LSTMCell(Module):
    def __init__(self, ni, nh):
        self.forget_gate = nn.Linear(ni + nh, nh)
        self.input_gate  = nn.Linear(ni + nh, nh)
        self.cell_gate   = nn.Linear(ni + nh, nh)
        self.output_gate = nn.Linear(ni + nh, nh)

    def forward(self, input, state):
        h, c = state
        h = torch.cat([h, input], dim=1)
        c = c * torch.sigmoid(self.forget_gate(h))
        c = c + torch.sigmoid(self.input_gate(h)) * torch.tanh(self.cell_gate(h))
        h = torch.sigmoid(self.output_gate(h)) * torch.tanh(c)
        return h, (h, c)

class MyModel(Module):
    def __init__(self, vocab_sz, n_hidden):
        self.i_h = nn.Embedding(vocab_sz, n_hidden)
        self.cells = [LSTMCell(bs, n_hidden) for _ in range(sl)]
        self.h_o = nn.Linear(n_hidden, vocab_sz)

        self.h = torch.zeros(bs, n_hidden)
        self.c = torch.zeros(bs, n_hidden)

    def forward(self, x):
        x = self.i_h(x)
        h, c = self.h, self.c
        for i, cell in enumerate(self.cells):
            res, (h, c) = cell(x[:, i, :], (h, c))
        self.h = h.detach()
        self.c = c.detach()
        return self.h_o(res)

    def reset(self):
        self.h.zero_()
        self.c.zero_()

learn = Learner(dls, MyModel(len(vocab), 64), loss_func=CrossEntropyLossFlat(), metrics=accuracy, cbs=ModelResetter)
learn.fit_one_cycle(5, 1e-2)

The training output looks like this

Any help appreciated

Output:
After some playing around I was able to figure it out. The issue was the way I was initialising the list of cells. In MyModule.__init__ I only needed to change the line to
self.cells = nn.ModuleList([LSTMCell(bs, n_hidden) for _ in range(sl)])

The reason it was broken was that by initialising the Modules in a regular list, the parameters were hidden from pytorch/fastai. By using a nn.ModuleList the parameters are registered and can be trained


Instruction: Resizing PIL Image gives a completely black image

Input:
I have a dataset of grayscale images, like this one below:

Now, I open my dataset with the following class:
    """Tabular and Image dataset."""

    def __init__(self, excel_file, image_dir):
        self.image_dir = image_dir
        self.excel_file = excel_file
        self.tabular = pd.read_excel(excel_file)

    def __len__(self):
        return len(self.tabular)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        tabular = self.tabular.iloc[idx, 0:]

        y = tabular["Prognosis"]

        image = PIL.Image.open(f"{self.image_dir}/{tabular['ImageFile']}")
        image = np.array(image)
        #image = image[..., :3]

        image = transforms.functional.to_tensor(image)

        return image, y

If I check the tensors of the image, then I have this:
tensor([[[160, 160, 192,  ...,  52,  40,  40],
         [176, 208, 320,  ...,  96,  80,  16],
         [176, 240, 368,  ..., 160, 160,  52],
         ...,
         [576, 608, 560,  ...,  16,  16,  16],
         [624, 592, 544,  ...,  16,  16,  16],
         [624, 624, 576,  ...,  16,  16,  16]]], dtype=torch.int32)

Now, they should be between 0 and 1, right? Because it is grayscale, or 0-255 in RGB, but there are those big values that I have no idea from where they are coming (indeed imshow shows images with strange distorted colors like yellow and blue rather then grayscale).
However, this is the size of the images torch.Size([1, 2350, 2866]); I want to resize to (1,224,224) for example
This is my function:
def resize_images(images: List[str]):
    for i in images:
        image = PIL.Image.open(f"{data_path}TrainSet/{i}")
        new_image = image.resize((224, 224))
        new_image.save(f"{data_path}TrainImgs/{i}")

resize_images(list(df["ImageFile"]))

However, this code returns all images that are 224x244 but they are all black. All images are completely black!

Output:
You have either shared a different file from the one your code opens, or imgur has changed your file. In any case, the most expedient way to examine the content of your file on Linux/Unix/macOS without installing any software is to use the file command. So, checking your file we can see that it is an 8-bit PNG with alpha channel:
file pZS4l.png
pZS4l.png: PNG image data, 896 x 732, 8-bit/color RGBA, non-interlaced

That immediately tells me it is not the exact image you opened in your code, because there are values exceeding 255 in your pixel dump, and that is not possible in an 8-bit file. So, the next best way to check the contents of an image is with exiftool and that works for Windows users too. That looks like this:
exiftool pZS4l.png

ExifTool Version Number         : 12.30
File Name                       : pZS4l.png
Directory                       : .
File Size                       : 327 KiB
File Modification Date/Time     : 2022:11:05 17:15:22+00:00
File Access Date/Time           : 2022:11:05 17:15:23+00:00
File Inode Change Date/Time     : 2022:11:05 17:15:22+00:00
File Permissions                : -rw-r--r--
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 896
Image Height                    : 732
Bit Depth                       : 8
Color Type                      : RGB with Alpha
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Profile Name                    : ICC Profile
Profile CMM Type                : Apple Computer Inc.
Profile Version                 : 2.1.0
Profile Class                   : Display Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 2022:07:06 14:13:59
Profile File Signature          : acsp
Primary Platform                : Apple Computer Inc.
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : Apple Computer Inc.
Device Model                    :
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 : Apple Computer Inc.
Profile ID                      : 0
Profile Description             : Display
...
...

So, now I see it is a screen-grab made on a Mac. A screen-grab is not the same as the image you are displaying! If you display a 16-bit image on an 8-bit display, the screen-grab will be 8-bit. You should share your original image, not a screen-grab. If imgur is changing your images, you should share them with Dropbox or Google Drive or similar.

Right, on to your question. Assuming you actually open a PNG in your code (which we can't tell because it is incomplete) the data should not be float because PNG cannot store floats, it can only store integers. PNG can store integer samples with bit depths of 1, 2, 4, 8 or 16-bits. If you read about a 24-bit PNG, that is RGB888. If you read about a 32-bit PNG, that is RGBA8888. If you read about a 48-bit PNG, that is RGB with 16 bits/sample. If you read about 64-bit PNG, that is RGBA with 16-bits/sample.
So, the short answer is to run:
file YOURACTUALIMAGE.PNG

and/or:
exiftool YOURACTUALIMAGE.PNG

So, my suspicion is that you have a 16-bit greyscale PNG, which is perfectly able to store samples in the range 0..65535.

Note: If you actually do want to store floats, you probably need to use TIFF, or PFM (Portable Float Map) or EXR format.


Instruction: How to use random_split with percentage split (sum of input lengths does not equal the length of the input dataset)

Input:
I tried to use torch.utils.data.random_split as follows:
import torch
from torch.utils.data import DataLoader, random_split

list_dataset = [1,2,3,4,5,6,7,8,9,10]
dataset = DataLoader(list_dataset, batch_size=1, shuffle=False)

random_split(dataset, [0.8, 0.1, 0.1], generator=torch.Generator().manual_seed(123))

However, when I tried this, I got the error raise ValueError("Sum of input lengths does not equal the length of the input dataset!")
I looked at the docs and it seems like I should be able to pass in decimals that sum to 1, but clearly it's not working.
I also Googled this error and the closest thing that comes up is this issue.
What am I doing wrong?

Output:
You're likely using an older version of PyTorch, such as Pytorch 1.10, which does not have this functionality.
To replicate this functionality in the older version, you can just copy the source code of the newer version:
import math
from torch import default_generator, randperm
from torch._utils import _accumulate
from torch.utils.data.dataset import Subset

def random_split(dataset, lengths,
                 generator=default_generator):
    r"""
    Randomly split a dataset into non-overlapping new datasets of given lengths.

    If a list of fractions that sum up to 1 is given,
    the lengths will be computed automatically as
    floor(frac * len(dataset)) for each fraction provided.

    After computing the lengths, if there are any remainders, 1 count will be
    distributed in round-robin fashion to the lengths
    until there are no remainders left.

    Optionally fix the generator for reproducible results, e.g.:

    >>> random_split(range(10), [3, 7], generator=torch.Generator().manual_seed(42))
    >>> random_split(range(30), [0.3, 0.3, 0.4], generator=torch.Generator(
    ...   ).manual_seed(42))

    Args:
        dataset (Dataset): Dataset to be split
        lengths (sequence): lengths or fractions of splits to be produced
        generator (Generator): Generator used for the random permutation.
    """
    if math.isclose(sum(lengths), 1) and sum(lengths) <= 1:
        subset_lengths: List[int] = []
        for i, frac in enumerate(lengths):
            if frac < 0 or frac > 1:
                raise ValueError(f"Fraction at index {i} is not between 0 and 1")
            n_items_in_split = int(
                math.floor(len(dataset) * frac)  # type: ignore[arg-type]
            )
            subset_lengths.append(n_items_in_split)
        remainder = len(dataset) - sum(subset_lengths)  # type: ignore[arg-type]
        # add 1 to all the lengths in round-robin fashion until the remainder is 0
        for i in range(remainder):
            idx_to_add_at = i % len(subset_lengths)
            subset_lengths[idx_to_add_at] += 1
        lengths = subset_lengths
        for i, length in enumerate(lengths):
            if length == 0:
                warnings.warn(f"Length of split at index {i} is 0. "
                              f"This might result in an empty dataset.")

    # Cannot verify that dataset is Sized
    if sum(lengths) != len(dataset):    # type: ignore[arg-type]
        raise ValueError("Sum of input lengths does not equal the length of the input dataset!")

    indices = randperm(sum(lengths), generator=generator).tolist()  # type: ignore[call-overload]
    return [Subset(dataset, indices[offset - length : offset]) for offset, length in zip(_accumulate(lengths), lengths)]


Instruction: How to get activation values of a layer in pytorch

Input:
I have a pytorch-lightning model that has a dense layer like so:
def __init__(...)
...
self.dense = nn.Linear(channels[-1], 64, bias=True)
...

for my project, I need to get the activation values of this layer as a list
I have tried this code which I found on the pytorch discussion forum:
activation = {}

def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

test_img = cv.imread(f'digimage/100.jpg')
test_img = cv.resize(test_img, (128, 128))
test_img = np.moveaxis(test_img, 2, 0)

modelftr = load_feature_model(**model_dict)
num_ftrs = modelftr.fc.in_features
modelftr.fc = torch.nn.Linear(num_ftrs, 228)
modelftr.load_state_dict(torch.load('...'))
modelftr.dense.register_forward_hook(get_activation('dense'))

with torch.no_grad():
    modelatt.to('cpu')
    modelatt.eval()
    test_img = torch.tensor(test_img).view(-1, 3, 128, 128).float()
    output = modelcat(test_img)
    print(activation['dense'])

But this gives a keyerror:
      8 test_img = torch.tensor(test_img).view(-1, 3, 128, 128).float()
      9 output = modelcat(test_img)
---> 10 print(activation['dense'])

KeyError: 'dense'

Update:
This is my full model code.
As you can see there is a linear layer named dense
class FAtNet(pl.LightningModule):
    def __init__(self, image_size, in_channels, num_blocks, channels,
                 num_classes=20, block_types=['C', 'C', 'T', 'T'], lr=0.0001, loss_function=nn.CrossEntropyLoss()):
        super().__init__()
        self.lr = lr
        self.loss_function = loss_function
        ih, iw = image_size
        block = {'C': MBConv, 'T': Transformer}

        self.s0 = self._make_layer(
            conv_3x3_bn, in_channels, channels[0], num_blocks[0], (ih // 2, iw // 2))
        self.s1 = self._make_layer(
            block[block_types[0]], channels[0], channels[1], num_blocks[1], (ih // 4, iw // 4))
        self.s2 = self._make_layer(
            block[block_types[1]], channels[1], channels[2], num_blocks[2], (ih // 8, iw // 8))
        self.s3 = self._make_layer(
            block[block_types[2]], channels[2], channels[3], num_blocks[3], (ih // 16, iw // 16))
        self.s4 = self._make_layer(
            block[block_types[3]], channels[3], channels[4], num_blocks[4], (ih // 32, iw // 32))

        self.pool = nn.AvgPool2d(ih // 32, 1)
        self.dense = nn.Linear(channels[-1], 64, bias=True)
        self.fc = nn.Linear(64, num_classes, bias=False)

    def forward(self, x):
        x = self.s0(x)
        x = self.s1(x)
        x = self.s2(x)
        x = self.s3(x)
        x = self.s4(x)

        x = self.pool(x).view(-1, x.shape[1])
        x = self.dense(x)
        x = self.fc(x)
        return x

    def _make_layer(self, block, inp, oup, depth, image_size):
        layers = nn.ModuleList([])
        for i in range(depth):
            if i == 0:
                layers.append(block(inp, oup, image_size, downsample=True))
            else:
                layers.append(block(oup, oup, image_size))
        return nn.Sequential(*layers)

    def configure_optimizers(self):
        return optim.Adam(self.parameters(), lr=self.lr)

    def training_step(self, batch, batch_idx):
        X, y = batch
        y_hat = self(X)
        loss = self.loss_function(y_hat, y)
        self.log('train_loss', loss)
        return loss

    def test_step(self, batch, batch_idx):
        X, y = batch
        y_hat = self(X)
        loss = self.loss_function(y_hat, y)
        self.log('test_loss', loss)
        return loss

    ### custom prediction function ###
    def predict(self, dm):
        X_test = dm.X_test

        self.eval()

        X_test = torch.tensor(X_test).float()
        self.to(device='cuda')

        pred = []

        with torch.no_grad():
            for data in X_test:
                output = self(data)
                pred.append(output)

        pred = pred[0].detach()

        pred = pred.cpu()

        self.to(device='cpu')
        self.train()

        return pred

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

Output:
It seems like you model does not have 'dense' layer, only 'fc'.
Try:
modelftr.fc.register_forward_hook(get_activation('fc'))


Instruction: Where to raise a typo in the PyTorch Documentation?

Input: I have found a typo in the official PyTorch Documentation. Where can I raise the flag so that it is rectified?

Output:
From the PyTorch Contribution Guide, in the section on Documentation:

Improving Documentation & Tutorials
We aim to produce high quality documentation and tutorials. On rare
occasions that content includes typos or bugs. If you find something
you can fix, send us a pull request for consideration.


Instruction: Access weights, biases in model - PyTorch pruning

Input:
To implement L1-norm, unstructured, layer-wise pruning with torch.nn.utils.prune l1_unstructured, radom_unstructured methods, I have a toy LeNet-300-100 dense neural network as-
class LeNet300(nn.Module):
    def __init__(self):
        super(LeNet300, self).__init__()

        # Define layers-
        self.fc1 = nn.Linear(in_features = 28 * 28 * 1, out_features = 300)
        self.fc2 = nn.Linear(in_features = 300, out_features = 100)
        self.output_layer = nn.Linear(in_features = 100, out_features = 10)

        self.weights_initialization()

    def forward(self, x):
        x = F.leaky_relu(self.fc1(x))
        x = F.leaky_relu(self.fc2(x))
        x = self.output_layer(x)
        return x

    def weights_initialization(self):
        '''
        When we define all the modules such as the layers in '__init__()'
        method above, these are all stored in 'self.modules()'.
        We go through each module one by one. This is the entire network,
        basically.
        '''
        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight)
                nn.init.constant_(m.bias, 1)

    def shape_computation(self, x):
        print(f"Input shape: {x.shape}")
        x = self.fc1(x)
        print(f"dense1 output shape: {x.shape}")
        x = self.fc2(x)
        print(f"dense2 output shape: {x.shape}")
        x = self.output_layer(x)
        print(f"output shape: {x.shape}")
        del x
        return None

# Initialize architecture-
model = LeNet300().to(device)

This has 266610 trainable parameters. To prune this with 20% for the first two dense layers and 10% for the output layer until 99.5% sparsity, you need 25 pruning rounds. The pruning is done with-
l1_unstructured(module = fc, name = 'weight', amount = 0.2)
Iterating through the layers is done with-
for name, module in model.named_modules():
    if name == '':
        continue
    else:
        print(f"layer: {name}, module: {module}")

However, for a particular module, how to access its weights and biases besides using-
module.weight, module.bias
?
The idea is to use a layer-wise (for now) pruning function as-
# Prune multiple parameters/layers in a given model-
for name, module in model.named_modules():

    # prune 20% of weights/connections in for all hidden layaers-
    if isinstance(module, torch.nn.Linear) and name != 'output_layer':
        prune.l1_unstructured(module = module, name = 'weight', amount = 0.2)

    # prune 10% of weights/connections for output layer-
    elif isinstance(module, torch.nn.Linear) and name == 'output_layer':
        prune.l1_unstructured(module = module, name = 'weight', amount = 0.1)

Bias pruning will also be included.

Output:
The name parameter is the attribute name of the parameter, within the module, on which the pruning will be applied (see documentation page). As such, you can provide either 'weight' or 'bias' in your case since you are focusing on nn.Linear exclusively.
Additionally, you will read that prune.l1_unstructured will:

Modifies module in place (and also return the modified module) by:

adding a named buffer called name+'_mask' corresponding to the binary mask applied to the parameter name by the pruning method.

replacing the parameter name by its pruned version, while the original (unpruned) parameter is stored in a new parameter named
name+'_orig'.

You can access the pruned weights via the weight and bias attributes and the original parameters with weight_orig and bias_orig:
m = nn.Linear(2, 3)
m = prune.l1_unstructured(m, 'weight', amount=.2)
m = prune.l1_unstructured(m, 'bias', amount=.2)

>>> m.state_dict().keys()
odict_keys(['weight_orig', 'bias_orig', 'weight_mask', 'bias_mask'])


Instruction: Upsampling the Spatial Dimensions Of a 4D Tensor using transposed convolution

Input:
I have a tensor with the size of (16, 64, 4,4) and I want to upsample the spatial size of this tensor using transposed convolution. How can I select the kernel size, stride, padding to have a tensor with size of (16, 64, 4,6)?
For example this is my code for upsampling from (16, 64, 1,1) to (16, 64, 4,4):
nn.convTranspose2d(64,64,kernel_size=6,stride=4,padding=1)

Output:
The equation for computing the output size is in the pytorch documentation of ConvTranspose2d. Put in your input size, set your desired output size and solve the equation for kernel size, stride, padding, etc. There may be multiple valid solutions.
For your specific problem, the following works:
from torch import nn

t = torch.ones((16, 64, 4, 4))  # test data
layer = nn.ConvTranspose2d(64, 64, kernel_size=(1, 3))
print(layer(t).shape)  # torch.Size([16, 64, 4, 6])

To demonstrate that there can be multiple valid solutions, here is another solution for up-sampling a tensor with shape (16, 64, 1, 1) to (16, 64, 4, 4).
from torch import nn

t = torch.ones((16, 64, 1, 1))  # test data
layer = nn.ConvTranspose2d(64, 64, kernel_size=(4, 4))
print(layer(t).shape)  # torch.Size([16, 64, 4, 4])


Instruction: Efficient way to use regex compile (Python) with a list of 10.000 strings

Input:
I got a list which contains approx. 10.000 strings and I want to use a regex pattern to detect this in this list. When I use re.compile it takes a lot of time to only apply one regex pattern. Is there any way with Python to make it faster?
Here my code:

import re

list_of_strings = ["I like to eat meat", "I don't like to eat meat", "I like to eat fish", "I don't like to eat fish"]

outcome = [x for x in list_of_strings if len(re.compile(r"I like to eat (.*?)").findall(x)) != 0]

Out[6]: ['I like to eat meat', 'I like to eat fish']

Here I have just 4 strings to demonstrate the case. In reality the code should handle 10.000 strings.
I could also use multiple processing to solve this issue but maybe there is also another solution with pytorch, pyspark or other Frameworks existing.
[Edit]
Thanks for all answers. I should have mentioned that every string is an article. So, it is not just one sentence to be handled from regex.
I also want to say that the regex here ist not that problem. So this is not a topic to be discussed.

Output:
You may also consider looping the list.
new_list = []
for item in list_of_strings:
    if 'I like to eat' in item:
        new_list.append(item)


Instruction: PyTorch's `torch.cuda.max_memory_allocated()` showing different results from `nvidia-smi`?

Input:
I'm currently making my own custom GPU report and am using torch.cuda.max_memory_allocated(device_id) to get the maximum memory that each GPU uses. However, I've noticed that this number is different from when I run nvidia-smi during the process.
According to the documentation for torch.cuda.max_memory_allocated, the output integer is in the form of bytes. And from what I've searched online to convert the number of bytes to the number of gigabytes, you should divide it by 1024 ** 3. I'm currently doing round(max_mem / (1024 ** 3), 2)
Am I doing the calculation wrong, or am I misunderstanding how torch.cuda.max_memory_allocated works entirely? The memory allocated I've observed from one GPU during the entire process was aroudn 32GB, but torch.cuda.max_memory_allocated(0) / (1024 ** 3) returns around 13.5GB.

Output:
Posting a link to the same question asked on the discussion forum for PyTorch. TL;DR torch.cuda.max_memory_allocated is not supposed to accurately resemble the output of nvidia-smi because nvidia-smi actually reserves a lot of memory than what is actually being used. Therefore, torch.cuda.max_memory_reserved will resemble the actual output much closely (although still not entirely accurately).
https://discuss.pytorch.org/t/pytorchs-torch-cuda-max-memory-allocated-showing-different-results-from-nvidia-smi/165706/2


Instruction: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden while using local mode in AWS SageMaker

Input:
trainer = PyTorch(
    entry_point="train.py",
    source_dir= "source_dir",  # directory of your training script
    role=role,
    framework_version="1.5.0",
    py_version="py3",
    instance_type= "local",
    instance_count=1,
    output_path=output_path,
    hyperparameters = hyperparameters
)

This code is running SageMaker NoteNook instance.
Error
Creating dsd3faq5lq-algo-1-ouews ...
Creating dsd3faq5lq-algo-1-ouews ... done
Attaching to dsd3faq5lq-algo-1-ouews
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,444 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,475 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,494 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,507 sagemaker_pytorch_container.training INFO     Invoking user training script.
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,673 sagemaker-training-toolkit ERROR    Reporting training FAILURE
dsd3faq5lq-algo-1-ouews | 2022-11-10 05:39:26,674 sagemaker-training-toolkit ERROR    framework error:
dsd3faq5lq-algo-1-ouews | Traceback (most recent call last):
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_training/trainer.py", line 85, in train
dsd3faq5lq-algo-1-ouews |     entrypoint()
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_pytorch_container/training.py", line 121, in main
dsd3faq5lq-algo-1-ouews |     train(environment.Environment())
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_pytorch_container/training.py", line 73, in train
dsd3faq5lq-algo-1-ouews |     runner_type=runner_type)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_training/entry_point.py", line 92, in run
dsd3faq5lq-algo-1-ouews |     files.download_and_extract(uri=uri, path=environment.code_dir)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_training/files.py", line 131, in download_and_extract
dsd3faq5lq-algo-1-ouews |     s3_download(uri, dst)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/sagemaker_training/files.py", line 167, in s3_download
dsd3faq5lq-algo-1-ouews |     s3.Bucket(bucket).download_file(key, dst)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/boto3/s3/inject.py", line 246, in bucket_download_file
dsd3faq5lq-algo-1-ouews |     ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/boto3/s3/inject.py", line 172, in download_file
dsd3faq5lq-algo-1-ouews |     extra_args=ExtraArgs, callback=Callback)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/boto3/s3/transfer.py", line 307, in download_file
dsd3faq5lq-algo-1-ouews |     future.result()
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
dsd3faq5lq-algo-1-ouews |     return self._coordinator.result()
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
dsd3faq5lq-algo-1-ouews |     raise self._exception
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/s3transfer/tasks.py", line 255, in _main
dsd3faq5lq-algo-1-ouews |     self._submit(transfer_future=transfer_future, **kwargs)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/s3transfer/download.py", line 343, in _submit
dsd3faq5lq-algo-1-ouews |     **transfer_future.meta.call_args.extra_args
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
dsd3faq5lq-algo-1-ouews |     return self._make_api_call(operation_name, kwargs)
dsd3faq5lq-algo-1-ouews |   File "/opt/conda/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call
dsd3faq5lq-algo-1-ouews |     raise error_class(parsed_response, operation_name)
dsd3faq5lq-algo-1-ouews | botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
dsd3faq5lq-algo-1-ouews |
dsd3faq5lq-algo-1-ouews | An error occurred (403) when calling the HeadObject operation: Forbidden
dsd3faq5lq-algo-1-ouews exited with code 1
1
Aborting on container exit...

The same code worked a couple of days back but it is not working from yesterday onwards.
Sagemaker Version: 2.115.0

I tried changing the SageMaker version to 2.0 but to no avail.
I also uploaded training code on s3 and use s3_uri in "source_dir" but still got the same error.
The problem only occurs while using instance_type = "local"
It works perfectly fine if I use a remote instance such as instance_type = "ml.c4.xlarge"
I also tried uploading and downloading objects directly from S3 using boto3 and it worked perfectly fine.
eg;
session = boto3.Session()
s3 = session.resource('s3')
s3_obj = s3.Object(bucket, data_key)
s3_obj.put(Body=data)

s3client = boto3.client('s3')
response = s3client.get_object(Bucket= bucket, Key= data_key)
body = response['Body'].read()

The above code works perfectly fine in same instance.
I am not sure but if it was just an issue with permission, wouldn't the above code also not work.

Output:
I changed the code to below, replacing Sagwmaker.pytorch.Pytorch with sagemaker.estimator.Estimator, providing reference to the relevant Docker Image URI.
from sagemaker.estimator import Estimator

TRAIN_IMAGE_URI= "763104351884.dkr.ecr.ap-south-1.amazonaws.com/pytorch-training:1.12.1-cpu-py38"

trainer = Estimator(
    image_uri = TRAIN_IMAGE_URI,
    entry_point="train.py",
    source_dir="source_dir",  # directory of your training script
    role=role,
    base_job_name = base_job_name,
    instance_type= "local",
    instance_count=1,
    output_path=output_path,
    hyperparameters = hyperparameters
)
trainer.fit()

This seems to work perfectly in my case.
Although I still do not understand why.


Instruction: how sum two tensors with difference shapes_pytorch

Input:
I have two tensors with these shapes:
a_tensor : torch.Size([64, 37])
b_tensor : torch.Size([64, 300])

how can I sum them (a_tensor+b_tensor ) and pad a_tensor with zeros to be size of [64,300]?
For example, a_tensor is [[1 , 2 , 3],[3 , 2 , 1]] with shape of ([2 , 3])
and
b_ensor is [[10 , 20 , 30 , 40] , [40 , 30 , 20 , 10]] with the shape of ([2,4]).
a_tensor + b_tensor would be like this:
[[1,2,3,0],[3,2,1,0]] + [[10,20,30,40],[40,30,20,10]] = [[11,22,33,40],[43,32,21,10]]

Output:
You can do so using nn.functional.pad:
>>> a = torch.rand(64, 37)
>>> b = torch.rand(64, 300)

Measure the padding amount:
>>> r = b.size(-1) - a.size(-1)

Pad and sum:
>>> tf.pad(a, (0,0,r,0)) + b


Instruction: torch Parameter grad return none

Input:
I want to implement learned size quantization algorithm. And I create a quante Linear layer
class QLinear(nn.Module):
    def __init__(self, input_dim, out_dim, bits=8):
        super(QLinear, self).__init__()
        # create a tensor requires_grad=True
        self.up = 2 ** bits - 1
        self.down = 0
        self.fc = nn.Linear(input_dim, out_dim)
        weight = self.fc.weight.data
        self.scale = nn.Parameter(torch.Tensor((torch.max(weight) - torch.min(weight)) / (self.up - self.down)), requires_grad=True)
        self.zero_point = nn.Parameter(torch.Tensor(self.down - (torch.min(weight) / self.scale).round()), requires_grad=True)

    def forward(self, x):
        weight = self.fc.weight
        quant_weight = (round_ste(weight / self.scale) + self.zero_point)
        quant_weight = torch.clamp(quant_weight, self.down, self.up)
        dequant_weight = ((quant_weight - self.zero_point) * self.scale)
        self.fc.weight.data = dequant_weight
        return self.fc(x)

class QNet(nn.Module):
    def __init__(self):
        super(QNet, self).__init__()
        self.fc1 = QLinear(28 * 28, 100)
        self.fc2 = QLinear(100, 10)

    def forward(self, x):
        x = x.view(-1, 28 * 28)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        x = F.softmax(x)
        return x

when I train this network,scale's grad always return None. Why this happen and how can i solve it?

Output:
The issue is that you are passing dequant_weight through data attribute of your parameter which ends up not being registered by autograd. A simple alternative would be to handle weight as a nn.Parameter and apply a linear operator manually in the forward definition directly with the computed weight dequant_weight.
Here is a minimal example that should work:
class QLinear(nn.Module):
    def __init__(self, input_dim, out_dim, bits=8):
        super().__init__()
        self.up = 2 ** bits - 1
        self.down = 0

        self.weight = nn.Parameter(torch.rand(out_dim, input_dim))
        self.scale = nn.Parameter(
            torch.Tensor((self.weight.max() - self.weight.min()) / (self.up - self.down)))
        self.zero_point = nn.Parameter(
            torch.Tensor(self.down - (self.weight.min() / self.scale).round()))

    def forward(self, x):
        quant_weight = (torch.round(self.weight / self.scale) + self.zero_point)
        quant_weight = torch.clamp(quant_weight, self.down, self.up)
        dequant_weight = ((quant_weight - self.zero_point) * self.scale)
        return F.linear(x, dequant_weight)

Side notes:

nn.Parameter requires gradient computation by default (no need to provide requires_grad=True.

Additionally you can reformat QNet by inheriting from nn.Sequential to avoid boilerplate code:
class QNet(nn.Sequential):
   def __init__(self):
       super().__init__(nn.Flatten(),
                        QLinear(28 * 28, 100),
                        nn.ReLU(),
                        QLinear(100, 10),
                        nn.Softmax())


Instruction: A Classifier Network Seems to be "Forgetting" older samples

Input:
This is a strange problem: Imagine a neural network classifier. It is a simple linear layer followed by a sigmoid activation that has an input size of 64, and an output size of 112. There also are 112 training samples, where I expect the output to be a one-hot vector. So the basic structure of a training loop is as follows, where samples is a list of integer indices:
model = nn.Sequential(nn.Linear(64,112),nn.Sequential())
loss_fn = nn.BCELoss()
optimizer = optim.AdamW(model.parameters(),lr=3e-4)
for epoch in range(500):
     for input_state, index in samples:
          one_hot = torch.zeros(112).float()
          one_hot[index] = 1.0
          optimizer.zero_grad()
          prediction = model(input_state)
          loss = loss_fn(prediction,one_hot)
          loss.backward()
          optimizer.step()

This model does not perform well, but I don't think it's a problem with the model itself, but rather how it's trained. I think that this is happening because for the most part, all of the one_hot tensor is zeros, that the model just tends to gravitate toward all of the outputs being zeros, which is what's happening. The question becomes: "How does this get solved?" I tried using the average loss with all the samples, to no avail. So what do I do?

Output: So this is very embarrassing, but the answer actually lies in how I process my data. This is a text-input project, so I used basic python lists to create blocks of messages, but when I did this, I ended up making it so that all of the inputs the net got were the same, but the output was different every time. I solved tho s problem with the copy method.


Instruction: Mismatch in expected results of convolution using Conv2d from Pytorch?

Input:
I am experimenting with the conv2d function implemented in PyTorch. I wrote a code sample below where we have a 3x3 matrix of 1 batch, 2 input channels. I implement my convolution layer to have a kernel the exact size of the matrix (so the stride doesn't matter) and an output channel of 1. I fix the weights at 1.
Basically this should just sum up input tensor. Why are the last two printed values slightly differing? Is this the result of some sort of floating point calculation error?
import torch.nn as nn
import torch
m = nn.Conv2d(2, 1, 3, stride=2)
input = torch.randn(1, 2, 3, 3)
m.weight = torch.nn.Parameter(torch.ones_like(m.weight))

output = m(input)
print(input)
print(torch.sum(input))
print(output)

Output:
The bias of Conv2d is not initialized as zeros.
Try this
import torch.nn as nn
import torch
m = nn.Conv2d(2, 1, 3, stride=2)
input = torch.randn(1, 2, 3, 3)
m.weight = torch.nn.Parameter(torch.ones_like(m.weight))
nn.init.zeros_(m.bias)

output = m(input)
print(input)
print(torch.sum(input))
print(output)