Instruction: pytroch data loader RuntimeError: stack expects each tensor to be equal size, but got [224, 224] at entry 0 and [224, 224, 3] at entry 1

Input:
my problem is i got two tensors in one dataset with headers image and label
when i execute simple loop all look fine
unfortunetly when i make dataloader as below
training_loader = torch.utils.data.DataLoader(training_dataset, batch_size=100, shuffle=True)

and run
for i in training_loader:
print(i)

Im getting error:
RuntimeError: stack expects each tensor to be equal size, but got [224, 224] at entry 0 and [224, 224, 3] at entry 4
what can cause it and how to fix it ? thank you inadvance

Output:
It seems like one (or more) of your images is not a color image, but a gray-scale image.
Modify your loading code to force all images to be treated as color images:
img = Image.load(filename).convert('RGB')

See this answer for more details.


Instruction: Receiving Infinity Infinity in LineString

Input:
I am try get linestring so I can measure the distance and time. Here in this linestring I am getting nan distance and time.
Also, pleased to hear any of your suggestion on my code or logic. Thanks
data:
[[29.87819, 121.54944999999998], [24.23111845, 119.02311485000001], [5.402576549999999, 106.87891215000002], [1.367889, 104.27658300000002], [4.65750565, 98.40456015000001], [5.93498595, 82.50298040000001], [6.895460999999999, 75.83849285000002], [11.087761, 55.21659015], [11.986111, 50.79761100000002], [12.57124165, 44.563427950000005], [15.262399899999998, 41.828814550000004], [27.339266099999996, 34.20131845], [29.927166, 32.566855000000004], [32.36497615, 28.787162800000004], [36.25582884999999, 14.171143199999989], [37.089583, 11.039139000000006], [36.98901405, 4.773231850000002], [36.139162799999994, -4.182775300000003], [36.86918755, -8.487389949999994], [42.41353785, -9.331828900000005], [47.68888635, -5.458406800000006], [50.7011586, 1.0547821000000113], [52.84235105, -6.168939849999987], [53.33306, -6.248889999999989]]

Code:
from shapely.geometry import LineString
from shapely.ops import transform
from functools import partial
import pyproj
path  = LineString(path)
print(path)
# Geometry transform function based on pyproj.transform
project = partial(
    pyproj.transform,
    pyproj.Proj('EPSG:4326'),
    pyproj.Proj('EPSG:32633'))

path = transform(project, path)
print(path)
distance = path.length/1852

print(str(path.length) + " METERS")
print(str(distance) + " NM")
print(str(distance/13) + " Hr will take")

Output:

Output:
Probably, previous pyproj EPSG was switching the opposite interpretation. So I change the EPSG code with below:
project = partial(
        pyproj.transform,
        pyproj.Proj('EPSG:4326'),
        pyproj.Proj('EPSG:3857'))


Instruction: Using CUDA in Colab

Input:
Is it necessary to convert tensors and models to CUDA with tensor.to in colab when I've chosen runtime type as GPU?
I want use CUDA for training my model

Output:
tensor.to(device) transfer data to the given device.
Yes, you need to transfer model, input, labels etc to whatever device you are intending to use


Instruction: CUDA out of memory error when reloading Pytorch model

Input:
Common pytorch error here, but I'm seeing it under a unique circumstance: when reloading a model, I get a CUDA: Out of Memory error, even though I haven't yet placed the model on the GPU.
model = model.load_state_dict(torch.load(model_file_path))
optimizer = optimizer.load_state_dict(torch.load(optimizer_file_path))
# Error happens here ^, before I send the model to the device.
model = model.to(device_id)

Output:
The issue is that I was trying to load to a new GPU (cuda:2) but originally saved the model and optimizer from a different GPU (cuda:0). So even though I didn't explicitly tell it to reload to the previous GPU, the default behavior is to reload to the original GPU (which happened to be occupied).
Adding map_location=device_id to each torch.load call fixed the problem:
model.to(device_id)
model = model.load_state_dict(torch.load(model_file_path, map_location=device_id))
optimizer = optimizer.load_state_dict(torch.load(optimizer_file_path, map_location=device_id))


Instruction: How Conv2D works in Tensorflow/PyTorch when two layers are connected with different filter numbers?

Input:
After reviewing LeNet5 architecture description, when Max Pooling layer (with 6 filters) is connected with Conv2D layer (16 filters), there is a requirement of special filter mapping, in the following way:

Taking inputs from every contiguous subset of 3 feature maps
Taking inputs from every contiguous subset of 4 feature maps
Taking inputs from every contiguous subset of 4 feature maps
Taking inputs from the discontinuous subset of 4 feature maps
Taking all the feature maps

A slightly annotated image from LeNet paper, courtesy of TowardsAI

Here recall this mapping is pretty specialized to 10 <-> 16, whereas Tensorflow/PyTorch is pretty flexible, just wonder how Tensorflow/PyTorch handles it exactly?

Output:
I don't know about the actual implementation of this network in those frameworks. However, here's one way you can implement such an operation in PyTorch.

You can look at this operation as a change of basis: going from feature maps in the 'S2' space to feature maps in the 'C3' space using a transform matrix M. The whole objective is to construct that matrix, it is composed of ones and zeros, where the ones are positioned such that you construct vectors in C3 space using components of vectors in S2 space.
For instance, let's look at the discontinuous subsets of 4 of the table: column #12 requires maps n°0, 1, 3, and 4. The corresponding row in M for vector #12 will therefore be [1,1,0,1,1,0]. Essentially, the 1s here correspond to the crosses shown in the figure. For this particular portion of the transition M will look like:
tensor([[1., 0., 1.],
        [1., 1., 0.],
        [0., 1., 1.],
        [1., 0., 1.],
        [1., 1., 0.],
        [0., 1., 1.]])

To actually perform the matrix multiplication, you can use torch.einsum:
torch.einsum('bchw,cd->bdhw', x, M)

Here's an example: starting from a 6-channel 2x2 map and transitioning to a 3-channel 2x2 map (defined by columns #12, #13, and #14 of the Table I):
>>> x = torch.rand(1,6,2,2)
tensor([[[[0.3134, 0.2468],
          [0.2759, 0.4971]],

         [[0.4150, 0.8735],
          [0.6726, 0.0463]],

         [[0.9547, 0.5338],
          [0.0654, 0.7458]],

         [[0.4099, 0.1984],
          [0.0930, 0.8054]],

         [[0.1695, 0.1586],
          [0.7961, 0.3894]],

         [[0.5535, 0.0678],
          [0.1484, 0.7735]]]])

>>> torch.einsum('bchw,cd->bdhw', x, M)
tensor([[[[1.3077, 1.4773],
          [1.8377, 1.7382]],

         [[2.0926, 1.6338],
          [1.6825, 1.9550]],

         [[2.2315, 1.0467],
          [0.5828, 2.8219]]]])

You can of course expand this operation to the whole of Table I, this would result in a matrix M of size 6x16.


Instruction: neural network trained with PyTorch outputs the mean value for every input

Input:
I am using PyTorch in order to get my neural network to recognize digits from the MNIST database.
import torch
import torchvision

I'd like to implement a very simple design similar to what is shown in 3Blue1Brown's video series about neural networks. The following design in particular achieved an error rate of 1.6%.
class Net(torch.nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = torch.nn.Linear(784, 800)
        self.layer2 = torch.nn.Linear(800, 10)

    def forward(self, x):
        x = torch.sigmoid(self.layer1(x))
        x = torch.sigmoid(self.layer2(x))
        return x

The data is gathered using torchvision and organised in mini batches containing 32 images each.
batch_size = 32
training_set = torchvision.datasets.MNIST("./", download=True, transform=torchvision.transforms.ToTensor())
training_loader = torch.utils.data.DataLoader(training_set, batch_size=32)

I am using the mean squared error as a loss funtion and stochastic gradient descent with a learning rate of 0.001 as my optimization algorithm.
net = Net()
loss_function = torch.nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.001)

Finally the network gets trained and saved using the following code:
for images, labels in training_loader:
    optimizer.zero_grad()
    for i in range(batch_size):
        output = net(torch.flatten(images[i]))
        desired_output = torch.tensor([float(j == labels[i]) for j in range(10)])
        loss = loss_function(output, desired_output)
        loss.backward()
    optimizer.step()
torch.save(net.state_dict(), "./trained_net.pth")

However, here are the outputs of some test images:
tensor([0.0978, 0.1225, 0.1018, 0.0961, 0.1022, 0.0885, 0.1007, 0.1077, 0.0994,
        0.1081], grad_fn=<SigmoidBackward>)
tensor([0.0978, 0.1180, 0.1001, 0.0929, 0.1006, 0.0893, 0.1010, 0.1051, 0.0978,
        0.1067], grad_fn=<SigmoidBackward>)
tensor([0.0981, 0.1227, 0.1018, 0.0970, 0.0979, 0.0908, 0.1001, 0.1092, 0.1011,
        0.1088], grad_fn=<SigmoidBackward>)
tensor([0.1061, 0.1149, 0.1037, 0.1001, 0.0957, 0.0919, 0.1044, 0.1022, 0.0997,
        0.1052], grad_fn=<SigmoidBackward>)
tensor([0.0996, 0.1137, 0.1005, 0.0947, 0.0977, 0.0916, 0.1048, 0.1109, 0.1013,
        0.1085], grad_fn=<SigmoidBackward>)
tensor([0.1008, 0.1154, 0.0986, 0.0996, 0.1031, 0.0952, 0.0995, 0.1063, 0.0982,
        0.1094], grad_fn=<SigmoidBackward>)
tensor([0.0972, 0.1235, 0.1013, 0.0984, 0.0974, 0.0907, 0.1032, 0.1075, 0.1001,
        0.1080], grad_fn=<SigmoidBackward>)
tensor([0.0929, 0.1258, 0.1016, 0.0978, 0.1006, 0.0889, 0.1001, 0.1068, 0.0986,
        0.1024], grad_fn=<SigmoidBackward>)
tensor([0.0982, 0.1207, 0.1040, 0.0990, 0.0999, 0.0910, 0.0980, 0.1051, 0.1039,
        0.1078], grad_fn=<SigmoidBackward>)

As you can see the network seems to approach a state where the answer for every input is:
[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]

This neural network is not better than just guessing. Where did I go wrong in my design or code?

Output:
Here are a few points that would be useful for you:

At first glance your model is not learning since your prediction are as good as a random guess. The first initiative would be to monitor your loss, here you only have a single epoch. At least you could evaluate your model on unseen data:
validation_set = torchvision.datasets.MNIST('./',
    download=True, train=False, transform=T.ToTensor())

validation_loader = DataLoader(validation_set, batch_size=32)

You are using a MSE loss (the L2-norm) to train a classification task which is not the right tool for this kind of task. You could instead be using the negative log-likelihood. PyTorch offers nn.CrossEntropyLoss which includes a log-softmax and the negative log-likelihood loss in one module. This change can be implemented by adding in:
loss_function = nn.CrossEntropyLoss()

and using the right target shapes when applying loss_function (see below). Since the loss function will apply a log-softmax, you shouldn't have an activation function on your model's output.

You are using sigmoid as an activation function, intermediate non-linearities will work better as ReLU (see related post). A sigmoid is more suited for a binary classification task. Again, since we are using nn.CrossEntropyLoss, we have to remove the activation after layer2.
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.flatten = nn.Flatten()
        self.layer1 = torch.nn.Linear(784, 800)
        self.layer2 = torch.nn.Linear(800, 10)

    def forward(self, x):
        x = self.flatten(x)
        x = torch.relu(self.layer1(x))
        x = self.layer2(x)
        return x

A less crucial point is the fact that you could infer estimations on a whole batch instead of looping through each batch one element at a time. A typical training loop for one epoch would look like:
for images, labels in training_loader:
    optimizer.zero_grad()
    output = net(images)
    loss = loss_function(output, labels)
    loss.backward()
    optimizer.step()

With these kinds of modifications, you can expect to have a validation of around 80% after a single epoch.


Instruction: PyTorch: vectorising looping addition of one value from a vector to a vector

Input:
I want to create a matrix that contains every combination of the sums of all elements in two large vectors using Torch, ultimately using CUDA within Torch.
The best way to describe it is with this (inefficient) code:
import numpy as np
import torch

x = torch.Tensor([1.1,2.2,3.3,4.4,5.5])
x_cent = torch.Tensor([10.2,20.2,100.1])

res_matrix = torch.zeros([int((x.shape)[0]), int((x_cent.shape)[0])])
res_col = torch.zeros([int((x.shape)[0])])

for i in x_cent:
    res_col = x.add(i)

for i in range(0,int((x_cent.shape)[0])):
    res_col = x.add(x_cent[i])
    res_matrix[:,i] = res_col

print(res_matrix)

The output of this is:
> tensor([[ 11.3000,  21.3000, 101.2000],
        [ 12.4000,  22.4000, 102.3000],
        [ 13.5000,  23.5000, 103.4000],
        [ 14.6000,  24.6000, 104.5000],
        [ 15.7000,  25.7000, 105.6000]])

There may be a term for this operation, and if someone can point it out, I will edit
this question and include the term.
Can you suggest a more efficient (vectorised?) approach to this that I could implement using the CUDA device on very large vectors? I'm guessing that this is a very simple question, but I am a beginner with torch.
Thanks!

Output:
Prompted by a comment above, and looking around, I see that a similar blog post exists:
every combination of addition of two vectors -- but not a matrix
This can be modified to produce a matrix:
(x.unsqueeze(1) + x_cent.unsqueeze(0))

Any better approaches?


Instruction: Can someone explain the layers code in the following pytorch neural network

Input:
The neural network is a pytorch implementation of the NVIDIA model for self driving cars.
Here I did not understand the first layer of the linear layers, the following is the line.
'nn.Linear(in_features=64 * 2 * 33, out_features=100)'
I can understand that 64 is the output of previous layer and 2 is number of flattened layers (if im not wrong).
Now my question is what's the purpose of '33'?

class NetworkDense(nn.Module):
def __init__(self):
    super(NetworkDense, self).__init__()
    self.conv_layers = nn.Sequential(
        nn.Conv2d(3, 24, 5, stride=2),
        nn.ELU(),
        nn.Conv2d(24, 36, 5, stride=2),
        nn.ELU(),
        nn.Conv2d(36, 48, 5, stride=2),
        nn.ELU(),
        nn.Conv2d(48, 64, 3),
        nn.ELU(),
        nn.Conv2d(64, 64, 3),
        nn.Dropout(0.25)
    )
    self.linear_layers = nn.Sequential(
        nn.Linear(in_features=64 * 2 * 33, out_features=100),
        nn.ELU(),
        nn.Linear(in_features=100, out_features=50),
        nn.ELU(),
        nn.Linear(in_features=50, out_features=10),
        nn.Linear(in_features=10, out_features=1)
    )

def forward(self, input):
    input = input.view(input.size(0), 3, 70, 320)
    output = self.conv_layers(input)
    output = output.view(output.size(0), -1)
    output = self.linear_layers(output)
    return output

Output:
It comes from your first input shape.
input = input.view(input.size(0), 3, 70, 320)
Think about it as an image with a size of (70, 320). If we pass that image to the first conv2d, then it will become (33, 158)..., and at last, if it all passes the conv layer, its shape will become (2, 33).
So, how to calculate these numbers?
It goes like:
(length - (kernel_size - stride)) // stride
Now let's just think about how the 70 will be changed. The first conv2d has a kernel size of 5 with stride 2. So 70 will be like:
(70 - (5 - 2)) // 2 => 33, then for the rest:
(33 - (5 - 2)) // 2 => 15
(15 - (5 - 2)) // 2 => 6
(6 - (3 - 1)) // 1` => 4
(4 - (3 - 1)) // 1` => 2

It's the same as the other length, 320 will become 33.


Instruction: can i use PyTorch Data Loader to load raw data images which are saved in CSV files?

Input: I have raw data images saved in separate CSV files(each image in a file). I want to train a CNN on them using PyTorch. how should I load data to be appropriate for using as CNN's input? (also, it is 1 channel and the image net's input is RGB as the default)

Output:
PyTorch's DataLoader, as the name suggests, is simply a utility class that helps you load your data in parallel, build your batch, shuffle and so on, what you need is instead a custom Dataset implementation.
Ignoring the fact that images stored in CSV files is kind of weird, you simply need something of the sort:
from torch.utils.data import Dataset, DataLoader

class CustomDataset(Dataset):

    def __init__(self, path: Path, ...):
        # do some preliminary checks, e.g. your path exists, files are there...
        assert path.exists()
        ...
        # retrieve your files in some way, e.g. glob
        self.csv_files = list(glob.glob(str(path / "*.csv")))

    def __len__(self) -> int:
        # this lets you know len(dataset) once you instantiate it
        return len(self.csv_files)

    def __getitem__(self, index: int) -> Any:
        # this method is called by the dataloader, each index refers to
        # a CSV file in the list you built in the constructor
        csv = self.csv_files[index]
        # now do whatever you need to do and return some tensors
        image, label = self.load_image(csv)
        return image, label

And that's it, more or less. You can then create your dataset, pass it to a dataloader and iterate the dataloader, something like:
dataset = CustomDataset(Path("path/to/csv/files"))
train_loader = DataLoader(dataset, shuffle=True, num_workers=8,...)

for batch in train_loader:
    ...


Instruction: PyTorch Lightning functools.partial error

Input:
I'm using a combination of PyTorch Forecasting and PyTorch Lightning, and running into an odd error. Some code below.
batch_size = 128
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=8)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=8)

.
.
.

tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.05,
    hidden_size=16,  # biggest influence network size
    attention_head_size=1,
    dropout=0.1,
    hidden_continuous_size=8,
    output_size=7,  # QuantileLoss has 7 quantiles by default
    loss=QuantileLoss(),
    log_interval=10,  # log example every 10 batches
    reduce_on_plateau_patience=4,  # reduce learning automatically
)

trainer.fit(
    tft,
    train_dataloaders=train_dataloader,
    val_dataloaders=val_dataloader
)

However, I then run into this error and I can't figure out why. Can anyone help me figure out what to do with the below error? I tried playing around with changing the syntax for the val_dataloader, but couldn't get anything to work.
Traceback (most recent call last):
  File "/model.py", line 136, in <module>
    val_dataloaders=val_dataloader,
  File "C:\...\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 553, in fit
    self._run(model)
  File "C:\...\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 912, in _run
    self._pre_dispatch()
  File "C:\...\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 941, in _pre_dispatch
    self._log_hyperparams()
  File "C:\...\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 970, in _log_hyperparams
    self.logger.save()
  File "C:\...\venv\lib\site-packages\pytorch_lightning\utilities\distributed.py", line 48, in wrapped_fn
    return fn(*args, **kwargs)
  File "C:\...\venv\lib\site-packages\pytorch_lightning\loggers\tensorboard.py", line 249, in save
    save_hparams_to_yaml(hparams_file, self.hparams)
  File "C:\...\venv\lib\site-packages\pytorch_lightning\core\saving.py", line 405, in save_hparams_to_yaml
    yaml.dump(v)
  File "C:\...\venv\lib\site-packages\yaml\__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "C:\...\venv\lib\site-packages\yaml\__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "C:\...\lib\site-packages\yaml\representer.py", line 27, in represent
    node = self.represent_data(data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 343, in represent_object
    'tag:yaml.org,2002:python/object:'+function_name, state)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 343, in represent_object
    'tag:yaml.org,2002:python/object:'+function_name, state)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 346, in represent_object
    return self.represent_sequence(tag+function_name, args)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 286, in represent_tuple
    return self.represent_sequence('tag:yaml.org,2002:python/tuple', data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "C:\...\venv\lib\site-packages\yaml\representer.py", line 331, in represent_object
    if function.__name__ == '__newobj__':
AttributeError: 'functools.partial' object has no attribute '__name__'

Process finished with exit code 1

Output:
This ended up being caused by an issue with a recent pandas upgrade. Rolling back to 1.2.5 resolved the issue.
pip install --upgrade pandas==1.2.5

Details on the problem in the link below.
https://github.com/pandas-dev/pandas/issues/42748


Instruction: How to train only RPN for torch vision Faster RCNN with pretrained backbone

Input:
As per the title mentioned, if I have already pretrained backbone, and I want to train only the RPN instead of the classifier using the Faster R-CNN from torchvision.
Is there any parameters I can pass in to the create_model function or would I stop the classifier training in my train() function?
I’m on mobile so olease excuse my editting
This is my create model function
Create your backbone from timm
backbone = timm.create_model(
“resnet50”,
pretrained=True,
num_classes=0, # this is important to remove fc layers
global_pool="" # this is important to remove fc layers
)

backbone.out_channels = backbone.feature_info[-1][“num_chs”]

anchor_generator = AnchorGenerator(
sizes=((16, 32, 64, 128, 256),), aspect_ratios=((0.25, 0.5, 1.0, 2.0),)
)
roi_pooler = torchvision.ops.MultiScaleRoIAlign(
featmap_names=[“0”], output_size=7, sampling_ratio=2
)
fastercnn_model = FasterRCNN(
backbone=backbone,
num_classes=1000,
rpn_anchor_generator=anchor_generator,
box_roi_pool=roi_pooler,
)

Output:
You can do the following
# First you can use model.children() method to see the idx of the backbone
for idx, child in enumerate(fastercnn_model.children()):
    if idx == 1:
        # Now set requires_grad for that idx to False
        for param in child.parameters():
            param.requires_grad = False
        break

# ===============  UPDATED  ========================
# This will train only the box_predictor not even the RPN. You can try out
# Different strategies and find the best for you.
# setting everything to false

for child in fastercnn_model.children():
    for param in child.parameters():
        param.requires_grad = False

for idx, child in enumerate(fastercnn_model.children()):
    if idx == 3:
        for i, param in enumerate(child.parameters()):
            if i==1:
                param.requires_grad = True
        break


Instruction: How can I fix cuda runtime error on google colab?

Input:
I'm trying to execute the named entity recognition example using BERT and pytorch following the Hugging Face page: Token Classification with W-NUT Emerging Entities.
There was a related question on stackoverflow, but the error message is different from my case.
cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorMath.cu:29
I have trouble with fixing the above cuda runtime error.
How can I execute the sample code on google colab with the run time type, GPU?
Error
trainer.train()

# Error Message
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    147     Variable._execution_engine.run_backward(
    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    150
    151

RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorMath.cu:29

Code
I didn't change the original data and code introduced on the tutorial, Token Classification with W-NUT Emerging Entities.
Access from the browser to Token Classification with W-NUT Emerging Entities code:
custom_datasets.ipynb - Colaboratory
from pathlib import Path
import re

def read_wnut(file_path):
    file_path = Path(file_path)

    raw_text = file_path.read_text().strip()
    raw_docs = re.split(r'\n\t?\n', raw_text)
    token_docs = []
    tag_docs = []
    for doc in raw_docs:
        tokens = []
        tags = []
        for line in doc.split('\n'):
            token, tag = line.split('\t')
            tokens.append(token)
            tags.append(tag)
        token_docs.append(tokens)
        tag_docs.append(tags)

    return token_docs, tag_docs

texts, tags = read_wnut('wnut17train.conll')

from sklearn.model_selection import train_test_split
train_texts, val_texts, train_tags, val_tags = train_test_split(texts, tags, test_size=.2)
unique_tags = set(tag for doc in tags for tag in doc)
tag2id = {tag: id for id, tag in enumerate(unique_tags)}
id2tag = {id: tag for tag, id in tag2id.items()}

from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-cased')
train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)

import numpy as np

def encode_tags(tags, encodings):
    labels = [[tag2id[tag] for tag in doc] for doc in tags]
    encoded_labels = []
    for doc_labels, doc_offset in zip(labels, encodings.offset_mapping):
        # create an empty array of -100
        doc_enc_labels = np.ones(len(doc_offset),dtype=int) * -100
        arr_offset = np.array(doc_offset)

        # set labels whose first offset position is 0 and the second is not 0
        doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
        encoded_labels.append(doc_enc_labels.tolist())

    return encoded_labels

train_labels = encode_tags(train_tags, train_encodings)
val_labels = encode_tags(val_tags, val_encodings)

import torch
import os

#os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
torch.backends.cudnn.enabled = False
# check if CUDA is available
train_on_gpu = torch.cuda.is_available()
# torch.backends.cudnn.enabled

if not train_on_gpu:
    print('CUDA is not available.  Training on CPU ...')
else:
    print('CUDA is available!  Training on GPU ...')

class WNUTDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)

train_encodings.pop("offset_mapping") # we don't want to pass this to the model
val_encodings.pop("offset_mapping")
train_dataset = WNUTDataset(train_encodings, train_labels)
val_dataset = WNUTDataset(val_encodings, val_labels)

from transformers import DistilBertForTokenClassification
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased', num_labels=len(unique_tags))

from transformers import DistilBertForSequenceClassification, Trainer, TrainingArguments, DistilBertForTokenClassification
from sklearn.metrics import precision_recall_fscore_support
import tensorflow as tf

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='binary')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

model = DistilBertForTokenClassification.from_pretrained("distilbert-base-uncased")

trainer = Trainer(
    model=model,                         # the instantiated  Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset,             # evaluation dataset
    compute_metrics=compute_metrics
)

trainer.train()

What I did
I checked cuda and GPU related settings.
#os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
torch.backends.cudnn.enabled = False
# check if CUDA is available
train_on_gpu = torch.cuda.is_available()
# torch.backends.cudnn.enabled

if not train_on_gpu:
    print('CUDA is not available.  Training on CPU ...')
else:
    print('CUDA is available!  Training on GPU ...')

#output
CUDA is available!  Training on GPU ...

training_args.device

#output
device(type='cuda', index=0)

Responce to an answer
When I comment out the part,
#os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
#torch.backends.cudnn.enabled = False

The error message changed to the below when I didn't reset runtime.
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in _make_grads(outputs, grads)
     49                 if out.numel() != 1:
     50                     raise RuntimeError("grad can be implicitly created only for scalar outputs")
---> 51                 new_grads.append(torch.ones_like(out, memory_format=torch.preserve_format))
     52             else:
     53                 new_grads.append(None)

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

If I reset runtime, the message was the same.
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorMath.cu:29

Output:
Maybe the problem comes from this line:
torch.backends.cudnn.enabled = False
You might comment or remove it and try again.


Instruction: How to deploy PyTorch in Centos6?

Input:
Recently, I want to run some pytorch codes on centos6. However, no matter I perform either "pip install torch" or "conda install torch", the prompt shows:
>>>  import torch
Traceback (most recent call last):
File "", line 1, in
File "XXX/anaconda3/envs/XXX/lib/python3.6/site-packages/torch/init.py", line 56, in
from torch._C import *
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by XXX/anaconda3/envs/XXX/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so)

My enviroment:
OS: CentOS release 6.7
How you installed PyTorch (conda, pip, source): pip & conda
Python version: 3.6.5

I tried mannually compiling glibc-2.19, but when I put the library path into LD_LIBRARY_PATH, I can't use commands like "ls", "make", etc. And I got "segment fault". I also tried just copying the *.so to the lib directory under the python virtual environment, no wonder, python crashed with "segment fault". I understand plenty of components rely on a specific glibc library. I just wonder how I can run pytorch on centos6 without switching to a new linux distribution.
PS: I need to deploy such python environments on the product environment, whose OS is centos6.

Output:
It's a tough one. You can either downgrade to a very old version of torch ( v0.3.1 as I remember was running ok on Centos 6.5 ), or upgrade to Centos 7. Having 2 version of glibc is hell.
If you really need Centos 6 to live with the latest version of torch, try installing glibc into non standard location and compiling both Python and pytorch from source.
update
You can't replace system's glibc and but you can install it somewhere else, like /opt/myglibc.
Pytorch stopped supporting Centos 6 since v0.4.1. So you will have to build it using gcc v5+ and linking it to your glibc version. Here are the instructions. But since you don't invoke pytorch directly, you need to build Python also. Then you can run your program by setting glibc path specifically for your program.
LD_LIBRARY_PATH=/opt/myglibc python my_program.py


Instruction: How to update the weights of a model only if loss for current batch is smaller than previous

Input:
I'm trying to update the weights of a model during training only for those batches in which the loss is smaller than that obtained in the previous batch.
So, in the batches loop, I store the loss obtained at each iteration, and then I have tried evaluating a condition: if loss at time t-1 is smaller that that a time t, then I proceed as follows:
if loss[t-1] <= loss[t]:
  loss.backward()
  optimizer.step()
else:
  #do nothing  or what ?

Then, nothing should be done in the else part. Nonetheless, I get an error saying CUDA is running out of memory.
Of course, before computing the loss, I perform an optimizer.zero_grad() sentence.
The for loop that runs over batches seems to be running fine, but memory usage blows up. I read that maybe setting gradients to None would prevent the weights update process but I have tried many sentences (output.clone().detach() also optimizer.zero_grad(set_to_none=True)) but I'm not sure they work. I think they did not. Nonetheless, the memory usage explosion still occurs.
Is there a way to get this done?

Output:
This is a common problem when storing losses from consecutive steps.
The out-of-memory error is caused because you are storing the losses in a list. The computational graphs will still remain and will stay in memory as long as you keep a reference to your losses. An easy fix is to detach the tensor when you append it to the list:
# loss = loss_fn(...)
losses.append(loss.detach())

Then you can work with
if losses[t] <= losses[t-1]: # current loss is smaller
    losses[t].backward()
    optimizer.step()
else:
    pass


Instruction: How to automatically judge whether the training process of the deep learning model is converged?

Input:
When training a deep learning model, I have to look at the loss curve and performance curve to judge whether the training process of the deep learning model is converged.
This has cost me a lot of time. Sometimes, the time of convergence judged by the naked eye is not accurate.
Therefore, I'd like to know whether there exists an algorithm or a package that can automatically judge whether the training process of the deep learning model is converged.
Can anyone help me?
Thanks a lot.

Output:
To the risk of disappointing you, I believe there is no such universal algorithm. In my experience, it depends on what you want to achieve, which metrics are important to you and how much time you are willing to let the training go on for.

I have already seen validation losses dramatically go up (a sign of overfitting) while other metrics (mIoU in this case) were still improving on the validation set. In these cases, you need to know what your target is.

It is possible (although it is very rare) that your loss goes up for a substantial amount of time before going down again and reach better levels than before. There is no way to anticipate this.

Finally, and this is arguably a common case if you have tons of training data, your validation loss may continually go down, but do so slower and slower. In this case, the best strategy if you had an infinite amount of time would be to let it keep the training going indefinitely. In practice, this is impossible, and you would need to find the right balance between performance and training time.

If you really need an algorithm, I would suggest this quite simple one :

Compute a validation metric M(i) after each ith epoch on a fixed subset of your validation set or the whole validation set. Let's suppose that the higher M(i)is, the better. Fix k an integer depending on the duration of one training epoch (k~3 should do the trick)
If for some n you have M(n) > max(M(n+1), ..., M(n+k)), stop and keep the network you had at epoch n.

It's far from perfect, but should be enough for simple tasks.
[Edit] If you're not using it yet, I invite you to use TensorBoard to visualize the evolution of your metrics throughout the training. Once set up, it is a huge gain of time.


Instruction: Torch.gather from 1D array using 2D indices

Input:
I have an nx1 tensor and an nxm tensor. I want to gather values from the nx1 tensor using the nxm tensor.
For example
for input tensor([1, 2, 3, 4]) and
index tensor([[0, 3], [2, 1],[1, 3], [2,3]])
output should be
tensor([[1, 4], [3, 2], [2,4], [3,4])

The indices are in the 2D matrix and the values are to be gathered from the 1D list.
How to use torch.gather/ or any torch tensor function for this purpose?
My following code gives error
t = torch.tensor([[1, 2, 3, 4]])
ind = torch.tensor([[0, 3], [2, 1],[1, 3], [2,3]])
torch.gather(t, 0, ind)

RuntimeError: index 2 is out of bounds for dimension 0 with size 1

Edit:
You can do simple indexing to achieve this output.
t[ind]

Is this the best way to do this. I assume this involves broadcasting the input array.
Edit
Using t[ind] in forward pass is resulting in the error
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [430,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

When I try to print tensors in the forward pass, no output is shown after the t[ind] operation. Which makes sense as getitem is not a differentiable operation for the loss to propagate.
So there is a valid use case in using gather over getitem.

Output:
If you want to use torch.gather:
torch.gather(t.expand(4, -1), 1, ind)


Instruction: Building a non-image classifier with ground truths

Input:
I have a dataset that looks like this:

The labels are basically a list of items (let's say, cars in a parking lot) I am given, where there are 10 of them in total, labeled from 0 to 10. I have 14 classes (let's say, 14 different car brands). Every float point value is just percentage of which class that particular item belongs to. For example, Item 2 is likely class 2 with a probability of 0.995275:
print(set(list(df['label'])))
> {0, 1, 2, 3, 4, 5, 6, 7, 9}

My goal is to train a classifier to output an integer from 0 to 14 to predict what class label x belongs to.
I am trying to build a feedforward NN with 3 hidden layers (+ input and output layers) and takes 15 inputs and outputs a prediction from 0 to 14. This is what I've designed so far:
class NNO(nn.Module):
  def __init__(self):
    super(NNO, self).__init__()
    h= [2,1]
    self.hidden = nn.Linear(h[0], h[1])
    self.hidden = nn.Linear(2,20)
    self.hidden = nn.Linear(20,20)
    self.output = nn.Linear(20,15)

    self.sigmoid = nn.Sigmoid()
    self.softmax = nn.Softmax(dim = 1)

  def forward(self, y):
    x = self.hidden(x)
    x = self.sigmoid(x)
    x = self.output(x)
    x = self.softmax(x)

My question is this. How do I feed the dataset to my NN to start training the epochs? I couldn't find any resource that pertains to a dataset like this.

Output:
Here is the answer:
# First I create some dummy data
label = np.random.randint(0, 14, 1000)
random = np.random.random((1000, 14))
total = pd.DataFrame(data=random, columns=[f'{i}_col' for i in range(14)])
total['label'] = label

'''
From what I understood you need 1 class in output that has the highest probability and hence this is a multi-class classification problem. In my case, I will just use the highest value from `random` as the target class.
'''
class TDataset(torch.utils.data.Dataset):
    def __init__(self, df):
        self.inputs = df[[f'{i}_col' for i in range(14)] + ['label']].values
        self.outputs = df[[f'{i}_col' for i in range(14)]].values

    def __len__(self):
        return len(self.inputs)

    def __getitem__(self, idx):
        x = torch.tensor(self.inputs[idx], dtype=torch.float)
        y = torch.tensor(np.argmax(self.outputs[idx]))
        return x, y

ds = TDataset(total)
dl = torch.utils.data.DataLoader(ds, batch_size=64)

# After doing this I will create a model which takes 15 inputs and
# Give 14 outputs in my case which represent the logits

class NNO(nn.Module):
    def __init__(self):
        super(NNO, self).__init__()
        self.hidden = nn.Linear(15, 20)
        self.relu = nn.ReLU()
        self.output = nn.Linear(20, 14)

    def forward(self, x):
        x = self.hidden(x)
        x = self.relu(x)
        x = self.output(x)
        return x

# Now we create the model object
m = NNO()

sample = None
for i in dl:
    sample = i
    break

print(m(sample[0]).shape) # shape = [64, 14] as desired.

# Now we define the loss function and then the optimizer
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(m.parameters())

# Now we define the training loop
for i in range(500): # for 500 epochs
    epoch_loss = 0
    for idx, data in enumerate(dl):
        inputs = data[0]
        targets = data[1] # change accordingly for your data
        preds = m(inputs)
        optimizer.zero_grad()
        loss = loss_fn(preds, targets)
        epoch_loss += loss
        loss.backward()
        optimizer.step()

    if (i%50 == 0):
        print('loss: ', epoch_loss.item() / len(dl))

'''
Now at the time of inference, you just need to apply softmax on the results of your model and select the most probable output.
'''

preds = m(sample[0])
predicted_classes = torch.argmax(torch.nn.functional.softmax(preds), axis=1)
# Here the predicted classes are the desired final output.


Instruction: Pytorch Assigning fixed parameter to the model

Input:
I am only interest in the feature map after 2 convolution layers with specific weights.
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2, 2), stride=1, padding=1, bias=False),
            nn.AvgPool2d(kernel_size=(2, 2), stride=(2, 2))
        )
        with torch.no_grad():
            weights1 = torch.tensor([[0.2390, 0.1593], [0.5377, 0]])
            self.layer1.weight = nn.Parameter(weights1, requires_grad=False)

        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2, 2), stride=1, padding=1, bias=False),
            nn.AvgPool2d(kernel_size=(2, 2), stride=(2, 2))
        )
        with torch.no_grad():
            weights2 = torch.tensor([[-0.2390, -0.3585], [-0.5377, 0.2390]])
            self.layer2.weight = nn.Parameter(weights2, requires_grad=False)

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        return x

Due to the limitation of real sensor implementation, I have to use fixed weights as above.
Problem is that the output is not constant.
>>>list(model.parameters())
[Parameter containing:
tensor([[0.2390, 0.1593],
        [0.5377, 0.0000]]), Parameter containing:
tensor([[[[-0.2701,  0.1602],
          [-0.0056, -0.0924]]]], requires_grad=True), Parameter containing:
tensor([[-0.2390, -0.3585],
        [-0.5377,  0.2390]]), Parameter containing:
tensor([[[[-0.0287,  0.2864],
          [ 0.3319, -0.3913]]]], requires_grad=True)]

Above is the result of model's parameter and you can see there are other parameters.
Do you know how to fix the paramteres?

Output:
You are not accessing the weight property on the incorrect object: self.layer1 and self.layer2 are not nn.Conv2d instances, they are nn.Sequential layers. Doing so you are essentially registering two new tensors (the fixed ones) to your module, added the two tensor parameters instantiated by the nn.Conv2d layers.
You should assign the fixed parameters to self.layer1[0] and (self.layer2[0] respectively):
self.layer1[0].weight = nn.Parameter(weights1, requires_grad=False)
# and
self.layer2[0].weight = nn.Parameter(weights2, requires_grad=False)

Then .parameters() will generate two tensor parameters:
>>> list
[Parameter containing:
 tensor([[0.2390, 0.1593],
         [0.5377, 0.0000]]), Parameter containing:
 tensor([[-0.2390, -0.3585],
         [-0.5377,  0.2390]])]


Instruction: ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content

Input:
I have installed many many packages including torch, gpytorch, ... in the past in Windows, Ubuntu and Mac following this scenario:
conda create -n env_name
conda activate env_name
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

However, this time on Ubuntu, I interfered the following error on downloading the package which apparently after downloading when checking the checksum, it sees a mismatch. I also tried removing those *.bz2 files just in case if there is a pre-downloaded file, it didn't work.
ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/torchaudio-0.9.0-py39.tar.bz2'.
  download saved to: /home/amin/anaconda3/pkgs/torchaudio-0.9.0-py39.tar.bz2
  expected md5: 7224453f68125005e034cb6646f2f0a3
  actual md5: 6bbb8056603453427bbe4cca4b033361

ChecksumMismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://conda.anaconda.org/pytorch/linux-64/torchvision-0.10.0-py39_cu111.tar.bz2'.
  download saved to: /home/amin/anaconda3/pkgs/torchvision-0.10.0-py39_cu111.tar.bz2
  expected md5: 78b4c927e54b06d7a6d18eec8b3f2d18
  actual md5: 69dd8411c573903db293535017742bd9

My system information:
Linux SPOT-Server 5.8.0-63-generic #71~20.04.1-Ubuntu SMP Thu Jul 15 17:46:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

My conda --version is also 4.8.2.
I also should add that I have same issue on Windows having conda --version equal to 4.10.1.

Output: The PyTorch channel maintainers had an issue when uploading some new package builds, which has since been resolved (see GitHub Issue).  The technical cause was uploading new builds with identical versions and build numbers as before, without replacing the previous build. This caused the expected MD5 checksum to correspond to the new upload, but the tarball that was ultimately downloaded still corresponded to the previous upload, leading to a checksum mismatch.


Instruction: Setting learning rate for Stochastic Weight Averaging in PyTorch

Input:
Following is a small working code for Stochastic Weight Averaging in Pytorch taken from here.
loader, optimizer, model, loss_fn = ...
swa_model = torch.optim.swa_utils.AveragedModel(model)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=300)
swa_start = 160
swa_scheduler = SWALR(optimizer, swa_lr=0.05)

for epoch in range(300):
    for input, target in loader:
        optimizer.zero_grad()
        loss_fn(model(input), target).backward()
        optimizer.step()
        if epoch > swa_start:
            swa_model.update_parameters(model)
            swa_scheduler.step()
        else:
            scheduler.step()

    # Update bn statistics for the swa_model at the end
    torch.optim.swa_utils.update_bn(loader, swa_model)
    # Use swa_model to make predictions on test data
    preds = swa_model(test_input)

In this code after 160th epoch the swa_scheduler is used instead of the usual scheduler. What does swa_lr signify? The documentation says,

Typically, in SWA the learning rate is set to a high constant value. SWALR is a learning rate scheduler that anneals the learning rate to a fixed value, and then keeps it constant.

So what happens to the learning rate of the optimizer after 160th epoch?
Does swa_lr affect the optimizer learning rate?

Suppose that at the beginning of the code the optimizer was ADAM initialized with a learning rate of 1e-4. Then does the above code imply that for the first 160 epochs the learning rate for training will be 1e-4 and then for the remaining number of epochs it will be swa_lr=0.05? If yes, is it a good idea to define swa_lr also to 1e-4?

Output:
does the above code imply that for the first 160 epochs the learning rate for training will be 1e-4

No it won't be equal to 1e-4, during the first 160 epochs the learning rate is managed by the first scheduler scheduler. This one is a initialize as a torch.optim.lr_scheduler.CosineAnnealingLR. The learning rate will follow this curve:

for the remaining number of epochs it will be swa_lr=0.05

This is partially true, during the second part - from epoch 160 - the optimizer's learning rate will be handled by the second scheduler swa_scheduler. This one is initialized as a torch.optim.swa_utils.SWALR. You can read on the documentation page:

SWALR is a learning rate scheduler that anneals the learning rate to a fixed value [swa_lr], and then keeps it constant.

By default (cf. source code), the number of epochs before annealing is equal to 10. Therefore the learning rate from epoch 170 to epoch 300 will be equal to swa_lr and will stay this way. The second part will be:

This complete profile, i.e. both parts:

If yes, is it a good idea to define swa_lr also to 1e-4

It is mentioned in the docs:

Typically, in SWA the learning rate is set to a high constant value.

Setting swa_lr to 1e-4 would result in the following learning-rate profile:


Instruction: cross entropy loss with weight manual calculation

Input:
Hi just playing around with code, I got the unexpected result of cross entropy loss weight implementation.
pred=torch.tensor([[8,5,3,2,6,1,6,8,4],[2,5,1,3,4,6,2,2,6],[1,1,5,8,9,2,5,2,8],[2,2,6,4,1,1,7,8,3],[2,2,2,7,1,7,3,4,9]]).float()
label=torch.tensor([[3],[7],[8],[2],[5]],dtype=torch.int64)
weights=torch.tensor([1,1,1,10,1,6,1,1,1],dtype=torch.float32)

with this kind of sample variables, pytorch's cross entropy loss gives out 4.7894
loss = F.cross_entropy(pred, label, weight=weights,reduction='mean')
> 4.7894

I manually implemented the cross entropy loss code as below
one_hot = torch.zeros_like(pred).scatter(1, label.view(-1, 1), 1)
log_prb = F.log_softmax(pred, dim=1)
loss = -(one_hot * log_prb).sum(dim=1).mean()

this kind of implementation gives same result with pytorch's cross entropy function if given without weight value. However with weight value
one_hot = torch.zeros_like(pred).scatter(1, label.view(-1, 1), 1)
log_prb = F.log_softmax(pred, dim=1)
loss = -(one_hot * log_prb)*weights.sum(dim=1).sum()/weights.sum()
> 3.9564

it gives out different loss value with pytorch module(4.7894).
I can roughly estimate that my understanding of loss's weight have some problem here, but I can't find out the exact reason for this kind of discrepancy.
Can anybody help me handling this issue?

Output:
I found out the problem. It was quite simple...
I shouldn't have divided with the whole sum of weights.
Instead with dividing with wt.sum() (wt=one_hot*weight) got me 4.7894.
>>> wt = one_hot*weights
>>> loss = -(one_hot * log_prb * weights).sum(dim=1).sum() / wt.sum()
4.7894

The denominator was only with 'related' weight value, not whole.


Instruction: Pytorch model output - only keep scores above 0.3

Input:
I have a large list (actually only one element with 3 dictionaries) as shown below. It is the output of a pretrained pytorch model for one instance of the test set. There are three attributes in the list (boxes, labels, scores), all of type tensor. Each box has a corresponding score and label. In total there are 100 boxes. Is there any quick way to only keep the boxes, labels and scores where the score is greater than 0.3? So for this example, there should only be 5 boxes with their respective score and label.
   output = [{'boxes': tensor([[0.0000e+00, 2.9095e+01, 7.3249e+01, 1.1387e+02],
              [7.8610e+01, 1.9392e+01, 1.6580e+02, 1.0291e+02],
              [3.6086e-01, 2.9609e+01, 1.0292e+02, 2.0285e+02],
              [1.8569e+02, 2.3418e+01, 2.4397e+02, 1.4092e+02],
              [1.9678e-03, 0.0000e+00, 5.8328e+01, 1.7467e+02],
              [1.4161e+02, 1.5196e+02, 2.2797e+02, 2.3690e+02],
              [1.5630e+02, 5.4246e+01, 2.1178e+02, 1.7170e+02],
              [5.3407e+01, 6.4962e+01, 1.0892e+02, 1.8180e+02],
              [1.0011e+02, 1.5188e+02, 1.8732e+02, 2.3737e+02],
              [1.5080e+02, 3.9219e+01, 2.3776e+02, 1.2494e+02],
              [8.9806e+01, 1.3143e+02, 1.7610e+02, 2.1669e+02],
              [1.3518e+02, 1.2713e+02, 1.9257e+02, 2.4350e+02],
              [1.1423e+02, 1.4989e+01, 1.7153e+02, 1.3093e+02],
              [7.9036e+01, 1.1927e+00, 1.9153e+02, 1.7694e+02],
              [8.4356e+01, 2.3523e+01, 1.4035e+02, 1.4181e+02],
              [6.9645e+01, 1.5251e+02, 1.5582e+02, 2.3697e+02],
              [1.4163e+02, 1.2086e+02, 2.2753e+02, 2.0553e+02],
              [8.3618e+01, 1.0583e+02, 1.4110e+02, 2.2334e+02],
              [3.2450e-01, 7.1444e+01, 7.1565e+01, 1.5488e+02],
              [7.2167e+00, 4.9198e+01, 9.3541e+01, 1.3515e+02],
              [3.8690e+01, 3.7546e+01, 1.2640e+02, 1.2457e+02],
              [1.0393e+02, 8.4865e+01, 1.6160e+02, 2.0193e+02],
              [9.6637e+00, 1.2074e+02, 9.1465e+01, 2.0829e+02],
              [2.6140e+00, 8.6522e+01, 5.9267e+01, 2.0357e+02],
              [1.6260e+02, 6.0580e+01, 2.4744e+02, 1.4646e+02],
              [1.7624e+02, 7.5614e+01, 2.3260e+02, 1.9287e+02],
              [1.2096e+02, 2.8686e+01, 2.0757e+02, 1.1476e+02],
              [1.0993e+02, 1.1107e+02, 1.9594e+02, 1.9697e+02],
              [3.8821e+01, 1.6499e+00, 1.5277e+02, 1.7680e+02],
              [1.3592e+02, 1.7006e+00, 2.5177e+02, 1.7528e+02],
              [4.3270e+01, 8.5363e+01, 9.8090e+01, 2.0313e+02],
              [3.9082e+01, 1.6281e+02, 1.2582e+02, 2.4565e+02],
              [1.0941e+02, 9.5967e+00, 1.9760e+02, 9.2006e+01],
              [9.4279e+01, 5.4012e+01, 1.5129e+02, 1.7273e+02],
              [1.7610e+02, 1.1657e+02, 2.3257e+02, 2.3306e+02],
              [1.8356e+02, 1.2347e+02, 2.5438e+02, 2.0546e+02],
              [1.4145e+00, 7.8904e+01, 7.9495e+01, 2.5600e+02],
              [5.7602e+01, 9.8933e+01, 1.7596e+02, 2.5600e+02],
              [1.3184e+02, 9.0243e+01, 2.1412e+02, 1.7617e+02],
              [1.3507e+02, 4.4525e+01, 1.9094e+02, 1.6303e+02],
              [1.0465e+01, 1.5353e+02, 9.2553e+01, 2.3779e+02],
              [1.8336e+02, 1.5361e+02, 2.5428e+02, 2.3656e+02],
              [1.9591e+02, 7.6679e+01, 2.5259e+02, 1.9309e+02],
              [9.8446e+01, 7.9284e+01, 2.1062e+02, 2.5600e+02],
              [1.3960e+02, 9.7938e+00, 2.2880e+02, 9.1881e+01],
              [9.0553e+01, 0.0000e+00, 1.7578e+02, 7.1238e+01],
              [1.8702e+00, 1.2331e+02, 4.8407e+01, 2.4824e+02],
              [0.0000e+00, 4.9428e-01, 1.2664e+02, 1.4013e+02],
              [7.9054e+01, 1.5236e+00, 2.3079e+02, 1.0113e+02],
              [1.6006e+02, 6.4527e+01, 2.5467e+02, 2.5600e+02],
              [0.0000e+00, 1.7543e+02, 1.8282e+02, 2.5560e+02],
              [1.7264e+00, 1.7961e+02, 7.0644e+01, 2.5600e+02],
              [1.8063e+02, 9.7504e+00, 2.5516e+02, 9.3108e+01],
              [5.0636e+01, 1.3299e+02, 1.3221e+02, 2.1604e+02],
              [3.1850e+01, 5.4289e+01, 8.8847e+01, 1.7312e+02],
              [2.0640e+02, 3.2171e+01, 2.5485e+02, 1.5160e+02],
              [1.9062e+01, 1.2459e+00, 1.7152e+02, 1.0271e+02],
              [8.0108e+01, 1.8195e+02, 1.6510e+02, 2.5523e+02],
              [6.4087e+00, 1.3263e+00, 9.6138e+01, 7.0220e+01],
              [2.1170e+01, 2.3619e+00, 7.9999e+01, 1.0748e+02],
              [5.7921e+01, 6.5922e-01, 1.4736e+02, 8.1241e+01],
              [1.1025e+02, 1.8136e+02, 1.9421e+02, 2.5513e+02],
              [6.1567e+01, 1.7640e+02, 2.5535e+02, 2.5600e+02],
              [3.9355e+01, 3.4047e+00, 1.0319e+02, 8.7065e+01],
              [5.0878e+01, 1.0217e+02, 1.3312e+02, 1.8664e+02],
              [7.4605e+01, 5.4398e+01, 1.2882e+02, 1.7320e+02],
              [1.7292e+02, 1.8397e+02, 2.5249e+02, 2.5558e+02],
              [1.8037e+01, 9.5900e+01, 1.3505e+02, 2.5211e+02],
              [1.4013e+02, 1.9098e+02, 2.2696e+02, 2.5415e+02],
              [6.2275e+01, 8.4387e+01, 9.0523e+01, 1.4154e+02],
              [1.7307e+01, 1.9287e+02, 1.1007e+02, 2.5532e+02],
              [7.2651e+01, 6.8909e+01, 1.0101e+02, 1.2587e+02],
              [1.6461e+02, 7.4065e+01, 1.9301e+02, 1.3071e+02],
              [7.7585e+01, 5.3052e+01, 1.0652e+02, 1.0954e+02],
              [1.6948e+02, 6.2818e+01, 1.9857e+02, 1.2058e+02],
              [5.7015e+01, 1.0044e+02, 8.5367e+01, 1.5666e+02],
              [7.7270e+01, 8.4819e+01, 1.0621e+02, 1.4148e+02],
              [6.7998e-01, 5.9344e+01, 3.2433e+01, 1.0408e+02],
              [5.7399e+01, 5.8315e+01, 8.5744e+01, 1.1517e+02],
              [1.5450e+02, 5.2688e+01, 1.8301e+02, 1.1024e+02],
              [6.7396e+01, 5.3385e+01, 9.6198e+01, 1.0940e+02],
              [5.2431e+01, 1.1456e+02, 8.0534e+01, 1.7151e+02],
              [5.2424e+01, 7.2901e+01, 8.0602e+01, 1.3118e+02],
              [5.4905e+01, 7.4931e+01, 9.8882e+01, 1.1913e+02],
              [1.7986e+02, 6.2481e+01, 2.0913e+02, 1.2018e+02],
              [6.7338e+01, 1.0491e+02, 9.5506e+01, 1.6148e+02],
              [1.7451e+02, 8.3800e+01, 2.0362e+02, 1.4150e+02],
              [4.9071e+01, 4.8894e+01, 9.4031e+01, 9.3116e+01],
              [1.0840e+00, 1.1331e+01, 3.7614e+01, 1.2899e+02],
              [8.2344e+01, 6.8916e+01, 1.1157e+02, 1.2634e+02],
              [1.6138e+02, 5.9034e+01, 2.0742e+02, 1.0366e+02],
              [1.4473e+01, 2.2719e+01, 2.1446e+02, 1.3569e+02],
              [4.7128e+01, 8.9411e+01, 7.5240e+01, 1.4684e+02],
              [1.8501e+02, 1.1383e+02, 2.1348e+02, 1.7167e+02],
              [5.4657e+01, 1.2145e+02, 9.8843e+01, 1.6540e+02],
              [3.7407e+00, 2.7927e+01, 4.8678e+01, 7.2360e+01],
              [1.6647e+02, 8.0292e+01, 2.1177e+02, 1.2410e+02],
              [3.4396e+01, 6.4177e+01, 7.8244e+01, 1.0873e+02],
              [8.7888e+01, 5.2212e+01, 1.1652e+02, 1.1079e+02],
              [1.7443e+02, 1.1404e+02, 2.0353e+02, 1.7198e+02]], device='cuda:0'),
      'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
              1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
              1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
              1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
              1, 1, 1, 1], device='cuda:0'),
      'scores': tensor([0.3317, 0.3235, 0.3208, 0.3157, 0.3108, 0.2977, 0.2974, 0.2955, 0.2938,
              0.2904, 0.2902, 0.2836, 0.2797, 0.2794, 0.2787, 0.2784, 0.2782, 0.2746,
              0.2739, 0.2695, 0.2658, 0.2655, 0.2647, 0.2628, 0.2622, 0.2596, 0.2596,
              0.2593, 0.2591, 0.2584, 0.2574, 0.2559, 0.2550, 0.2529, 0.2526, 0.2429,
              0.2428, 0.2408, 0.2397, 0.2381, 0.2370, 0.2344, 0.2302, 0.2296, 0.2292,
              0.2260, 0.2258, 0.2252, 0.2201, 0.2166, 0.2125, 0.2063, 0.2056, 0.2054,
              0.2050, 0.2032, 0.2023, 0.2021, 0.1985, 0.1956, 0.1943, 0.1776, 0.1739,
              0.1708, 0.1700, 0.1665, 0.1657, 0.1595, 0.1588, 0.1561, 0.1553, 0.1553,
              0.1484, 0.1426, 0.1419, 0.1416, 0.1289, 0.1265, 0.1250, 0.1248, 0.1226,
              0.1219, 0.1216, 0.1208, 0.1197, 0.1186, 0.1182, 0.1164, 0.1164, 0.1157,
              0.1133, 0.1109, 0.1097, 0.1086, 0.1055, 0.1055, 0.1054, 0.1047, 0.1026,
              0.1020], device='cuda:0')}]

Output:
So you're looking for something like this?
mask = output[0]['scores'] > 0.3
for key,val in output[0].items():
    output[0][key] = val[mask]
output[0]

{'boxes': tensor([[0.0000e+00, 2.9095e+01, 7.3249e+01, 1.1387e+02],
         [7.8610e+01, 1.9392e+01, 1.6580e+02, 1.0291e+02],
         [3.6086e-01, 2.9609e+01, 1.0292e+02, 2.0285e+02],
         [1.8569e+02, 2.3418e+01, 2.4397e+02, 1.4092e+02],
         [1.9678e-03, 0.0000e+00, 5.8328e+01, 1.7467e+02]]),
 'labels': tensor([1, 1, 1, 1, 1]),
 'scores': tensor([0.3317, 0.3235, 0.3208, 0.3157, 0.3108])}


Instruction: RuntimeError: CUDA error: device-side assert triggered - BART model

Input:
I am trying to run BART language model for a text generation task.
My code was working fine when I used for another encoder-decoder model (T5), but with bart I am getting this error:
File "train_bart.py", line 89, in train
    outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)                                                     cs-lab-host1" 12:39 10-Aug-21
  File ".../venv/tf_23/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File ".../venv/tf_23/lib/python3.6/site-packages/transformers/models/bart/modeling_bart.py", line 1308, in forward
    return_dict=return_dict,
  File ".../venv/tf_23/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File ".../venv/tf_23/lib/python3.6/site-packages/transformers/models/bart/modeling_bart.py", line 1196, in forward
    return_dict=return_dict,
  File ".../venv/tf_23/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File ".../venv/tf_23/lib/python3.6/site-packages/transformers/models/bart/modeling_bart.py", line 985, in forward
    attention_mask, input_shape, inputs_embeds, past_key_values_length
  File ".../venv/tf_23/lib/python3.6/site-packages/transformers/models/bart/modeling_bart.py", line 866, in _prepare_decoder_attent
ion_mask
    ).to(self.device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

And this is where error happens:
for _, data in tqdm(enumerate(loader, 0), total=len(loader), desc='Processing batches..'):
    y = data['target_ids'].to(device, dtype = torch.long)
    y_ids = y[:, :-1].contiguous()
    lm_labels = y[:, 1:].clone().detach()
    lm_labels[y[:, 1:] == tokenizer.pad_token_id] = -100
    ids = data['source_ids'].to(device, dtype = torch.long)
    mask = data['source_mask'].to(device, dtype = torch.long)

    outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)
    loss = outputs[0]

loader is the tokenized and processed data.

Output:
After fighting for many hours, I found that the error was due to adding new tokens to the Bart tokenizer. Thus I needed to resize the model input embeddings matrix:
model.resize_token_embeddings(len(tokenizer))

The point that is still not clear to me is that, without resizing the embeddings matrix, I was able to fine-tune T5 model without any problem, but not Bart.
Maybe this is because Bart is sharing weights between the input and the output layers (I am not sure of this either).


Instruction: Mixed Precision(Pytorch Autocast) Slows Down the Code

Input:
I have RTX 3070. Somehow using autocast slows down my code.
torch.version.cuda prints 11.1, torch.backends.cudnn.version() prints 8005 and my PyTorch version is 1.9.0. I’m using Ubuntu 20.04 with Kernel 5.11.0-25-generic.
That’s the code I’ve been using:
torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for epoch in range(10):

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):

        inputs, labels = data

        optimizer.zero_grad()

        with torch.cuda.amp.autocast():
             outputs = net(inputs)
             oss = criterion(outputs, labels)

         scaler.scale(loss).backward()
         scaler.step(optimizer)
         scaler.update()

end.record()

torch.cuda.synchronize()

print(start.elapsed_time(end))

Without torch.cuda.amp.autocast(), 1 epoch takes 22 seconds, whereas with autocast() 1 epoch takes 30 seconds.

Output: It turns out, my model was not big enough to utilize mixed precision. When I increased the in/out channels of convolutional layer, it finally worked as expected.


Instruction: Do I need to load the weights of another class I use in my NN class?

Input:
I have a model that needs to implement self-attention and this is how I wrote my code:
class SelfAttention(nn.Module):
    def __init__(self, args):
        self.multihead_attn = torch.nn.MultiheadAttention(args)

    def foward(self, x):
        return self.multihead_attn.forward(x, x, x)

class ActualModel(nn.Module):
    def __init__(self):
        self.inp_layer = nn.Linear(arg1, arg2)
        self.self_attention = SelfAttention(some_args)
        self.out_layer = nn.Linear(arg2, 1)

    def forward(self, x):
        x = self.inp_layer(x)
        x = self.self_attention(x)
        x = self.out_layer(x)
        return x

After loading a checkpoint of ActualModel, in ActualModel.__init__ during continuing-training or during prediction time should I load a saved model checkpoint of class SelfAttention?
If I create an instance of class SelfAttention, would the trained weights corresponding to SelfAttention.multihead_attn be loaded if I do torch.load(actual_model.pth) or would be they be reinitialized?
In other words, is this necessary?
class ActualModel(nn.Module):

    def __init__(self):
        self.inp_layer = nn.Linear(arg1, arg2)
        self.self_attention = SelfAttention(some_args)
        self.out_layer = nn.Linear(arg2, 1)

    def pred_or_continue_train(self):
        self.self_attention = torch.load('self_attention.pth')

actual_model = torch.load('actual_model.pth')
actual_model.pred_or_continue_training()
actual_model.eval()

Output:
In other words, is this necessary?

In short, No.
The SelfAttention class will be automatically loaded if it has been registered as a nn.module, nn.Parameters, or manually registered buffers.
A quick example:
import torch
import torch.nn as nn

class SelfAttention(nn.Module):
    def __init__(self, fin, n_h):
        super(SelfAttention, self).__init__()
        self.multihead_attn = torch.nn.MultiheadAttention(fin, n_h)

    def foward(self, x):
        return self.multihead_attn.forward(x, x, x)

class ActualModel(nn.Module):
    def __init__(self):
        super(ActualModel, self).__init__()
        self.inp_layer = nn.Linear(10, 20)
        self.self_attention = SelfAttention(20, 1)
        self.out_layer = nn.Linear(20, 1)

    def forward(self, x):
        x = self.inp_layer(x)
        x = self.self_attention(x)
        x = self.out_layer(x)
        return x

m = ActualModel()
for k, v in m.named_parameters():
    print(k)

You will get as follows, where self_attention is successfully registered.
inp_layer.weight
inp_layer.bias
self_attention.multihead_attn.in_proj_weight
self_attention.multihead_attn.in_proj_bias
self_attention.multihead_attn.out_proj.weight
self_attention.multihead_attn.out_proj.bias
out_layer.weight
out_layer.bias


Instruction: How to avoid two variables refering to the same data? #Pytorch

Input:
During initializing, I tried to reduce the repeats in my code, so instead of:
output=  (torch.zeros(2, 3),
          torch.zeros(2, 3))

I wrote:
z = torch.zeros(2, 3)
output=  (z,z)

However, I find that the second method is wrong.
If I assign the data to variables h,c, any change on h would also be applied to c
h,c = output
print(h,c)
h +=torch.ones(2,3)
print('-----------------')
print(h,c)

results of the test above:
tensor([[0., 0., 0.],
        [0., 0., 0.]]) tensor([[0., 0., 0.],
        [0., 0., 0.]])
-----------------
tensor([[1., 1., 1.],
        [1., 1., 1.]]) tensor([[1., 1., 1.],
        [1., 1., 1.]])

Is there a more elegent way to initialize two indenpendent variables?

Output:
I agree that your initial line needs no modification but if you do want an alternative, consider:
z = torch.zeros(2, 3)
output=  (z,z.clone())

The reason the other one (output = (z,z)) doesn't work, as you've correctly discovered is that no copy is made. You're only passing the same reference in each entry of the tuple to z


Instruction: Error while trying to fine-tune the ReformerModelWithLMHead (google/reformer-enwik8) for NER

Input:
I'm trying to fine-tune the ReformerModelWithLMHead (google/reformer-enwik8) for NER. I used the padding sequence length same as in the encode method (max_length = max([len(string) for string in list_of_strings])) along with attention_masks. And I got this error:
ValueError: If training, make sure that config.axial_pos_shape factors: (128, 512) multiply to sequence length. Got prod((128, 512)) != sequence_length: 2248. You might want to consider padding your sequence length to 65536 or changing config.axial_pos_shape.

When I changed the sequence length to 65536, my colab session crashed by getting all the inputs of 65536 lengths.
According to the second option(changing config.axial_pos_shape), I cannot change it.

I would like to know, Is there any chance to change config.axial_pos_shape while fine-tuning the model? Or I'm missing something in encoding the input strings for reformer-enwik8?
Thanks!
Question Update: I have tried the following methods:

By giving paramteres at the time of model instantiation:

model = transformers.ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8", num_labels=9, max_position_embeddings=1024, axial_pos_shape=[16,64], axial_pos_embds_dim=[32,96],hidden_size=128)

It gives me the following error:

RuntimeError: Error(s) in loading state_dict for ReformerModelWithLMHead:
size mismatch for reformer.embeddings.word_embeddings.weight: copying a param with shape torch.Size([258, 1024]) from checkpoint, the shape in current model is torch.Size([258, 128]).
size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([128, 1, 256]) from checkpoint, the shape in current model is torch.Size([16, 1, 32]).

This is quite a long error.

Then I tried this code to update the config:

model1 = transformers.ReformerModelWithLMHead.from_pretrained('google/reformer-enwik8', num_labels = 9)

Reshape Axial Position Embeddings layer to match desired max seq length
model1.reformer.embeddings.position_embeddings.weights[1] = torch.nn.Parameter(model1.reformer.embeddings.position_embeddings.weights[1][0][:128])

Update the config file to match custom max seq length
model1.config.axial_pos_shape = 16,128
model1.config.max_position_embeddings = 16*128 #2048
model1.config.axial_pos_embds_dim= 32,96
model1.config.hidden_size = 128
output_model_path = "model"
model1.save_pretrained(output_model_path)

By this implementation, I am getting this error:

RuntimeError: The expanded size of the tensor (512) must match the existing size (128) at non-singleton dimension 2.  Target sizes: [1, 128, 512, 768].  Tensor sizes: [128, 768]

Because updated size/shape doesn't match with the original config parameters of pretrained model. The original parameters are: axial_pos_shape = 128,512 max_position_embeddings = 128*512 #65536 axial_pos_embds_dim= 256,768 hidden_size = 1024
Is it the right way I'm changing the config parameters or do I have to do something else?
Is there any example where ReformerModelWithLMHead('google/reformer-enwik8') model fine-tuned.
My main code implementation is as follow:
class REFORMER(torch.nn.Module):
def __init__(self):
    super(REFORMER, self).__init__()
    self.l1 = transformers.ReformerModelWithLMHead.from_pretrained("google/reformer-enwik8", num_labels=9)

def forward(self, input_ids, attention_masks, labels):
    output_1= self.l1(input_ids, attention_masks, labels = labels)
    return output_1

model = REFORMER()

def train(epoch):
    model.train()
    for _, data in enumerate(training_loader,0):
        ids = data['input_ids'][0]   # input_ids from encode method of the model https://huggingface.co/google/reformer-enwik8#:~:text=import%20torch%0A%0A%23%20Encoding-,def%20encode,-(list_of_strings%2C%20pad_token_id%3D0
        input_shape = ids.size()
        targets = data['tags']
        print("tags: ", targets, targets.size())
        least_common_mult_chunk_length = 65536
        padding_length = least_common_mult_chunk_length - input_shape[-1] % least_common_mult_chunk_length
        #pad input
        input_ids, inputs_embeds, attention_mask, position_ids, input_shape = _pad_to_mult_of_chunk_length(self=model.l1,
                input_ids=ids,
                inputs_embeds=None,
                attention_mask=None,
                position_ids=None,
                input_shape=input_shape,
                padding_length=padding_length,
                padded_seq_length=None,
                device=None,
            )
        outputs = model(input_ids, attention_mask, labels=targets) # sending inputs to the forward method
        print(outputs)
        loss = outputs.loss
        logits = outputs.logits
        if _%500==0:
           print(f'Epoch: {epoch}, Loss:  {loss}')

for epoch in range(1):
    train(epoch)

Output:
First of all, you should note that google/reformer-enwik8 is not a properly trained language model and that you will probably not get decent results from fine-tuning it. enwik8 is a compression challenge and the reformer authors used this dataset for exactly that purpose:

To verify that the Reformer can indeed fit large models on a single
core and train fast on long sequences, we train up to 20-layer big
Reformers on enwik8 and imagenet64...

This is also the reason why they haven't trained a sub-word tokenizer and operate on character level.
You should also note that the LMHead is usually used for predicting the next token of a sequence (CLM). You probably want to use a token classification head (i.e. use an encoder ReformerModel and add a linear layer with 9 classes on top+maybe a dropout layer).
Anyway, in case you want to try it still, you can do the following to reduce the memory footprint of the google/reformer-enwik8 reformer:

Reduce the number of hashes during training:

from transformers import ReformerConfig, ReformerModel
conf = ReformerConfig.from_pretrained('google/reformer-enwik8')
conf.num_hashes = 2 # or maybe even to 1
model = transformers.ReformerModel.from_pretrained("google/reformer-enwik8", config =conf)

After you have finetuned your model, you can increase the number of hashes again to increase the performance (compare Table 2 of the reformer paper).

Replace axial-position embeddings:

from transformers import ReformerConfig, ReformerModel
conf = ReformerConfig.from_pretrained('google/reformer-enwik8')
conf.axial_pos_embds = False
model = transformers.ReformerModel.from_pretrained("google/reformer-enwik8", config =conf)

This will replace the learned axial positional embeddings with learnable position embeddings like Bert's and do not require the full sequence length of 65536. They are untrained and randomly initialized (i.e. consider a longer training).


Instruction: Full C++ LibTorch API on Moblie [Android]

Input:
I'm trying to compile a programme which uses LibTorch for android.
But the libraries built by script/build_android.sh in the pytorch repo, contain only the torch.jit Module. I need the torch.nn Module.
Is it possible to access the full C++ LibTorch API ? If so how can I build it ?

Output: It would seem that by setting NO_API to OFF and BUILD_MOBILE_AUTOGRAD to ON in the root CMake file, your can build the api and thus torch.nn.


Instruction: 'Vocab' object has no attribute 'itos'

Input:
I am building ML models for NLP with pytorch, but as I define vocabulary for tekenized words in my text with "vacab" and try to use vocab.itos I get: 'Vocab' object has no attribute 'itos' error.
This is my vocab:
vocab = torchtext.vocab.vocab(counter, min_freq=1)

How can I solve this problem?

Output:
You should access torchtext.vocab.Vocab.get_itos to get the indices->tokens mapping.
>>> itos = vocab.get_itos()


Instruction: Reproducibility in PyTorch with K-Fold Cross Validation

Input:
I have recently started a new project using PyTorch and I am still new in AI. In order to perform  better on my dataset during training process I used cross-validation technique. Everyone seems to work fine but I am struggling with reproducibility. I even tried to set SEED number for each k-fold iteration but it does not seem to work at all. Changes in loss and accuracy are insignificant but they are. Before using cross-validation everything worked perfect. Thank you in advance.
Here is a for loop for my k-fold. I used a solution from:
k-fold cross validation using DataLoaders in PyTorch
K_FOLD = 5
fraction = 1 / K_FOLD
unit = int(dataset_length * fraction)

for i in range(K_FOLD):

    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    torch.cuda.manual_seed_all(SEED)            # if you are using multi-GPU.
    np.random.seed(SEED)                        # Numpy module.
    random.seed(SEED)                           # Python random module.
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

    print("-----------K-FOLD {}------------".format(i+1))
    tr_ll = 0
    print("Train left begin:", tr_ll)
    tr_lr = i * unit
    print("Train left end:", tr_lr)
    val_l = tr_lr
    print("Validation begin:", val_l)
    val_r = i * unit + unit
    print("Validation end:", val_r)
    tr_rl = val_r
    print("Train right begin:", tr_rl)
    tr_rr = dataset_length
    print("Train right end:", tr_rr)
    # msg
    #         print("train indices: [%d,%d),[%d,%d), test indices: [%d,%d)"
    #               % (tr_ll,tr_lr,tr_rl,tr_rr,val_l,val_r))

    train_left_indices = list(range(tr_ll, tr_lr))
    train_right_indices = list(range(tr_rl, tr_rr))

    train_indices = train_left_indices + train_right_indices
    val_indices = list(range(val_l, val_r))
    # print("TRAIN Indices:", train_indices, "VAL Indices:", val_indices)

    train_set = torch.utils.data.dataset.Subset(DATASET, train_indices)
    val_set = torch.utils.data.dataset.Subset(DATASET, val_indices)
    # print("Length of train set:", len(train_set), "Length of val set:", len(val_set))

    image_datasets = {"train": train_set, "val": val_set}
    loader = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=10, shuffle=True)
                     for x in sets}
    dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}

    # training
    trained_model = train_model(AlexNet, CRITERION, OPTIMIZER,
                            dataloader=loader, dataset_sizes=dataset_sizes, num_epochs=EPOCHS, k_fold=i)

Output:
According to the latest docs, looks like you'll also need:
torch.use_deterministic_algorithms(True)


Instruction: Copying model output to a torch.Tensor where requires_grad is True

Input:
I have a model that outputs a sequence of vectors for each element in the batch, e.g., [Batch size, Sequence Length, Hidden size]. Then, I want to select a variable number of vectors for each element in the batch, and copy these vectors to a tensor where requires_grad = True. A sample code is bellow:

from torch import nn
from typing import List

class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(8,8)

    def forward(self, x: torch.Tensor, indices: List[torch.Tensor]):
        # Example indices: [torch.tensor([0,1]), torch.tensor([2,3,4])]
        out = self.fc(x)
        batch_size, _, hidden_size = out.size()
        max_num_hidden_states = max([ind.size(0) for ind in indices])
        selected_hidden_states = torch.zeros(batch_size, max_num_hidden_states, hidden_size, requires_grad=True)
        for i in range(batch_size):
            selected_hidden_states.data[i, :indices[i].size(0)] = out[i, indices[i]]
        return selected_hidden_states

model = MyModel()
with torch.no_grad():
    output = model(torch.rand(2, 5, 8), [torch.tensor([0,1]), torch.tensor([2,3,4])])

The questions I have w.r.t. this are:

If I train such model, would the gradients be backpropagated in the rest of the model parameters?
Why does output.requires_grad = True, when I explicitly state torch.no_grad()?
The way I'm doing this (which it doesn't seem to work as expected as of now) seems too hacky and wrong. What is the proper way to achieve what I want?

I'm aware of this answer, which approves my way doing it (at least it seems like it), but still it looks hacky to me.
Cheers!

Output:
Copy-paste an answer from the PyTorch forum.
That is from a time long gone and answering a different question.

No. Between creating a new tensor requiring grad and using .data, which you never should these days, you created a new leaf which will accumulate .grad.
Because you requested it. no_grad signals that you do not need the grad, it does not include guarantees about the requires_grad of the result.
If the utility function does not work for you, dropping the requires_grad and the .data should do the trick.


Instruction: What is the mechanism of "torch.Tensor in torch.Tensor" in python and why is there such a confusing phenomenon?

Input:
environment:
google colab
Python 3.7.11
torch 1.9.0+cu102
code and output
import torch
b = torch.tensor([[1,1,1],[4,5,6]])
print(b.T)

print(torch.tensor([1,4]) in b.T) #
print(torch.tensor([2,1]) in b.T) #
print(torch.tensor([1,2]) in b.T) # Not as expected
print(torch.tensor([2,5]) in b.T) # Not as expected

----------------------------------------------------------
tensor([[1, 4],
        [1, 5],
        [1, 6]])
True
False
True
True

probleam
I want to judge whether one tensor is in another.But the result above is confusing.
What is the mechanism of in? How should I use it to avoid the above unexpected output?
And is it the problem of torch.tensor.T? (When .T is not used and initial b = torch.tensor([[1,4],[1,5],[1,6]]), there also can be no expected output)

Output:
The source code is as follows (edit: source code snippet found by @Rune):
def __contains__(self, element):
    r"""Check if `element` is present in tensor
    Args:
        element (Tensor or scalar): element to be checked
            for presence in current tensor"
    """
    if has_torch_function_unary(self):
        return handle_torch_function(Tensor.__contains__, (self,), self, element)
    if isinstance(element, (torch.Tensor, Number)):
        # type hint doesn't understand the __contains__ result array
        return (element == self).any().item()  # type: ignore[union-attr]

    raise RuntimeError(
        "Tensor.__contains__ only supports Tensor or scalar, but you passed in a %s." %
        type(element)
    )

The __contains__ (used by the 'in' syntax: x in b) operator is equivalent to applying torch.any on the x == b boolean condition:
>>> b = tensor([[1, 1, 1],
                [4, 5, 6]])

>>> check_in = lambda x: torch.any(x == b.T)

Then
>>> check_in(torch.tensor([1,4]))
tensor(True)

>>> check_in(torch.tensor([2,1]))
tensor(False)

>>> check_in(torch.tensor([1,2]))
tensor(True)

>>> check_in(torch.tensor([2,5]))
tensor(True)

It is the position of the elements in the columns that matters not the exact match of the whole column.

.T reverses the order of the dimension: equivalent to b.permute(1, 0) and has no effect on the results. The only constraint you have when using in is x's size needs to match the shape of b[1]. If you're working with b.T then it will be b[0].
>>> check_in = lambda x: torch.any(x == b)

>>> check_in(torch.tensor([1,1,1]))
tensor(True)

>>> check_in(torch.tensor([5,4,2]))
tensor(False)


Instruction: Pytorch Problem: My jupyter stuck when num_workers > 0

Input:
This is a snippet of my code in PyTorch, my jupiter notebook stuck when I used num_workers > 0, I spent a lot on this problem without any answer. I do not have a GPU and I work only with a CPU.
class IndexedDataset(Dataset):

def __init__(self,data,targets, test=False):
    self.dataset = data
    if not test:
        self.labels = targets.numpy()
        self.mask =  np.concatenate((np.zeros(NUM_LABELED), np.ones(NUM_UNLABELED)))

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, idx):
        image = self.dataset[idx]
        return image, self.labels[idx]

    def display(self, idx):
        plt.imshow(self.dataset[idx], cmap='gray')
        plt.show()

train_set = IndexedDataset(train_data, train_target, test = False)

test_set = IndexedDataset(test_data, test_target, test = True)

train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, num_workers=2)

test_loader = DataLoader(test_set, batch_size=BATCH_SIZE, num_workers=2)

Any help, appreciated.

Output:
When num_workers is greater than 0, PyTorch uses multiple processes for data loading.
Jupyter notebooks have known issues with multiprocessing.
One way to resolve this is not to use Jupyter notebooks - just write a normal .py file and run it via command-line.
Or try use what's suggested here: Jupyter notebook never finishes processing using multiprocessing (Python 3).


Instruction: Is there a way to use a pre-trained transformers model without the configuration file?

Input:
I would like to fine-tune a pre-trained transformers model on Question Answering. The model was pre-trained on large engineering & science related corpora.
I have been provided a "checkpoint.pt" file containing the weights of the model. They have also provided me with a "bert_config.json" file but I am not sure if this is the correct configuration file.
from transformers import AutoModel, AutoTokenizer, AutoConfig

MODEL_PATH = "./checkpoint.pt"
config = AutoConfig.from_pretrained("./bert_config.json")
model = AutoModel.from_pretrained(MODEL_PATH, config=config)

The reason I believe that bert_config.json doesn't match "./checkpoint.pt" file is that,  when I load the model with the code above, I get the error that goes as below.

Some weights of the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt were not used when initializing BertModel: ['files', 'optimizer', 'model', 'master params']

This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt and are newly initialized: ['encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.21.attention.self.value.bias', 'encoder.layer.11.attention.self.value.bias', ............

If I am correct in assuming that "bert_config.json" is not the correct one, is there a way to load this model correctly without the config.json file?
Is there a way to see the model architecture from the saved weights of checkpoint.pt file?

Output:
This is a warning message instead of a error.
It means that the pretrained model is pretrained in some task (such as Question Answering, MLM etc), if your own fine tune task is the same as those pretrained task, then this IS NOT expected; unless this IS expected because some pooler of pretrained model will not be used in fine tune.
But this message doesn't mean that the bert_config.json isn't the right one. You can test it on huggingface's official colab notebook

You can find more information in this issue.


Instruction: Print input / output / grad / loss at every step/epoch when training Transformers HuggingFace model

Input:
I'm working on HuggingFace Transformers and using toy example from here:
https://huggingface.co/transformers/custom_datasets.html#fine-tuning-with-trainer
What I actually need: ability to print input, output, grad and loss at every step.
It is trivial using Pytorch training loop, but it is not obvious using HuggingFace Trainer.
At the current moment I have next idea: create a CustomCallback like this:
class MyCallback(TrainerCallback):
    "A callback that prints a grad at every step"

    def on_step_begin(self, args, state, control, **kwargs):
        print("next step")
        print(kwargs['model'].classifier.out_proj.weight.grad.norm())

args = TrainingArguments(
    output_dir='test_dir',
    overwrite_output_dir=True,
    num_train_epochs=1,
    logging_steps=100,
    report_to="none",
    fp16=True,
    disable_tqdm=True,
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    callbacks=[MyCallback],
)

trainer.train()

This way I can print grad and weights for any model layer.
But I still can't figure out how to print input/output (for example, I want to check them on nan) and loss?
P.S. I also read something about forward_hook but still can't find good code examples for it.

Output:
While using hooks and custom callbacks is the right way to solve the problem I find better solution - use built-in utility for finding nan/Inf in losses / weights / inputs / outputs:
https://huggingface.co/transformers/internal/trainer_utils.html#transformers.debug_utils.DebugUnderflowOverflow
since the 4.6.0 transformers has such option.
You can use it manually in forward function or just use additional option for TrainingArguments like this:
args = TrainingArguments(
    output_dir='test_dir',
    overwrite_output_dir=True,
    num_train_epochs=1,
    logging_steps=100,
    report_to="none",
    fp16=True,
    disable_tqdm=True,
    debug="debug underflow_overflow"
)


Instruction: Pytorch Scheduler: how to get decreasing LR epochs

Input:
I'm training a network in pytorch and using ReduceLROnPlateau as scheduler.
I set verbose=True in the parameteres and my scheduler prints something like:

Epoch   159: reducing learning rate to 6.0000e-04.
Epoch   169: reducing learning rate to 3.0000e-04.
Epoch   178: reducing learning rate to 1.5000e-04.
Epoch   187: reducing learning rate to 7.5000e-05.

I would like to get the epochs in some way, in order to obtain a list with all the epochs in which the scheduler reduced the learning rate.
Something like: lr_decrease_epochs = ['159', '169', '178', '187']
Which is the simplest way to do that ?

Output:
I think the scheduler doesn't take track of this (at least I didn't see anything like this in the source code), but you can just keep track of this in your training loop.
Whenever the learning rate changes (scheduler.get_lr()) you simply record the current epoch.


Instruction: Extremely low accuracy for IRIS dataset using CNN

Input:
Being a beginner, I am trying to implement my CNN on IRIS dataset with only 2 labels considered:

Iris-setosa: 0
Iris-versicolor: 1

I am using 90% data for training and 10% for testing with 1D CNN and Adam optimization and learning rate of 0.001. The accuracy achieved is around 40-50% which also is changing with every execution. Please suggest what should be done.
DATA LOADING TO DATALOADERS:
#Training data
class IrisDataset(T.utils.data.Dataset):

  def __init__(self, Iris):
    sc = StandardScaler()
    X_tr = sc.fit_transform(trainX)
    Y_tr = trainY
    self.X_tr = torch.tensor(X_tr, dtype = torch.float32)
    self.Y_tr = torch.tensor(Y_tr, dtype = torch.float32)

  def __len__(self):
    return len(self.Y_tr)

  def __getitem__(self, idx):
    return self.X_tr[idx], self.Y_tr[idx]

train_ds = IrisDataset(Iris)

bat_size = 1
# Leaving only labels 0 and 1
idx = np.append(np.where(train_ds.Y_tr == 0)[0],
                np.where(train_ds.Y_tr == 1)[0])
train_ds.X_tr = train_ds.X_tr[idx]
train_ds.Y_tr = train_ds.Y_tr[idx]

#len(train_ds)
train_ldr = T.utils.data.DataLoader(train_ds,
    batch_size=bat_size, shuffle=True)
batch = next(iter(train_ldr))

# and in the same way test data

#NETWORK CLASS

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv1d(1, 6, kernel_size=1)
        self.conv2 = nn.Conv1d(6, 16, kernel_size=1)
        self.dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(64, 16)
        self.fc2 = nn.Linear(16, 1)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = self.dropout(x)
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return T.cat((x, 1 - x), -1)

# MODEL TRAINING
model = Net()
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_func = nn.NLLLoss()
epochs = 2
loss_list = []

model.train()

for epoch in range(epochs):
    total_loss = []

    for X_tr, Y_tr in train_ldr:
       X_tr = X_tr.unsqueeze(0)
       optimizer.zero_grad()

       output = model(X_tr)
       pred = output.argmax(dim=1, keepdim=True)
       Y_tr = torch.tensor(Y_tr, dtype=torch.long)
       loss = loss_func(output, Y_tr.squeeze(1))
        # Backward pass
       loss.backward()
        # Optimize the weights
       optimizer.step()

       total_loss.append(loss.item())

    loss_list.append(sum(total_loss)/len(total_loss))
    print('Training [{:.0f}%]\tLoss: {:.4f}'.format(
         100. * (epoch + 1) / epochs, loss_list[-1]))

Output:
The mistake is here
def forward(self, x):
   x = F.relu(self.conv1(x))
   .
   .
   x = self.fc2(x)
   return T.cat((x, 1 - x), -1)
'''
The thing is that the output from the dense layer is not the probability and hence
you subtracting it form 1 also don't make any sense. It will become probability if
you use sigmoid activation after it.
'''
def forward(self, x):
   x = F.relu(self.conv1(x))
   .
   .
   x = self.fc2(x)
   x = torch.nn.functional.sigmoid(x)
   return T.cat((x, 1 - x), -1)

But I would recommend that you change your loss function to BCE as below. This is less prone to error and is same as above in terms of results. Also you can read docs for NLL and see that it says "Obtaining log-probabilities in a neural network is easily achieved by adding a LogSoftmax layer in the last layer of your network. You may use CrossEntropyLoss instead, if you prefer not to add an extra layer." So you can either add LogSoftmax or use CrossEntropy.
# LogSoftmax
def forward(self, x):
   x = F.relu(self.conv1(x))
   .
   .
   x = self.fc2(x)
   x = torch.nn.functional.log_softmax(x, dim=1)
   return x

loss_func = nn.NLLLoss()

# BCE
def forward(self, x):
   x = F.relu(self.conv1(x))
   .
   .
   x = self.fc2(x)
   return x

loss_func = torch.nn.BCEWithLogitsLoss()


Instruction: Issue with viewing data transformation with PyTorch

Input:
I am currently trying to process data for a simple network. This is the code I entered:
Screenshot here
I keep getting this error message but can't find any syntax problems or anyone else with this issue, I'm guessing it's something to do with my vienv because I've seen tutorials of people with no issues and that exact code. It's possible I haven't imported a package into my IDE and I am using anaconda and PyCharm if that helps.
Anyway, this is the error message I keep getting.
Error Message

Output: You need to use transforms.ToTensor() instead of transforms.ToTensor when passing to transforms.Compose.