Instruction: UNET with CrossEntropy Loss Function

Input:
I was trying to train UNET with input size as [3,128,128] and the corresponding mask is [1,128,128] which contains classes directly(instead of pixels it will contain class numbers - 1,2). I am trying for a two-class problem hence my mask contains 1,2 as labels. Now I send my images to the model and the dimension of the predicted masks are [2,128,128]. Now to train a model I choose 16 as batch size. So, now I have input as [16,3,128,128] so the predicted dimension is [16,2,128,128]. But I have ground-truth masks as [16,1,128,128]. Now how can I apply Cross entropy loss in Pytorch? I have tried as follows and getting the following error. Could you please help? Thanks in advance.
lr = 0.1  # 0.1
criterion = nn.CrossEntropyLoss() #nn.L1Loss()
optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9, nesterov=True, weight_decay=0.0001)
is_train = True
is_pretrain = False
acc_best = 0
total_epoch = 30
if is_train is True:
    # Training
    for epoch in range(total_epoch):
        model.train()
        tims = time.time()
        for i, (images, labels) in enumerate(train_loader):
            images = Variable(images.permute(0,3,1,2).float().cuda())
            labels = Variable(labels.type(torch.LongTensor).cuda())
            # Forward + Backward + Optimize
            optimizer.zero_grad()
            outputs = model(images)
            outputs = outputs.type(torch.float)
            print('predictedLabelsType:',outputs[0].type())
            print('ActualLabelsType:',labels[0].type())
            print('shape of predicted outputs:',outputs.shape)
            print('shape of groundtruth masks:',labels.shape)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

My output is as follows:
predictedLabelsType: torch.cuda.FloatTensor
ActualLabelsType: torch.cuda.LongTensor
shape of predicted outputs: torch.Size([16, 2, 128, 128])
shape of groundtruth masks: torch.Size([16, 1, 128, 128])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-b692a8d536a9> in <module>()
     52             print('shape of predicted outputs:',outputs.shape)
     53             print('shape of groundtruth masks:',labels.shape)
---> 54             loss = criterion(outputs, labels)
     55             loss.backward()
     56             optimizer.step()

3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2385         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2386     elif dim == 4:
-> 2387         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2388     else:
   2389         # dim == 3 or dim > 4

RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [16, 1, 128, 128]

Could you please suggest where is the mistake and how this CrossEntropyLoss() will work in Pytorch for image segmentation? What I am missing here? I have tried reshaping the target size as [16,128,128] which is leads to another error. Thanks a lot!

Output:
The documentation specifies that if the input is shape (N, C, d1, d2) then the target must be shape (N, d1, d2). Instead, your targets are shape (N, 1, d1, d2) so you need to remove the unnecessary unitary dimension.
loss = criterion(output, labels.squeeze(1))

If you're getting another error from this change then there's another issue with your code, but this is the correct tensor shape for CrossEntropyLoss.


Instruction: AttributeError: 'collections.OrderedDict' object has no attribute 'predict'

Input: Being a new guy and a beginner to deep learning and pytorch I am not sure what all inputs should I give you guys to answer my question. But I will try my best to make you guys understand my problem. I have loaded a model in pytorch using 'model= torch.load('model/resnet18-5c106cde.pth')'. But it is showing an AttributeError: 'collections.OrderedDict' object has no attribute 'predict', when I used the command 'prediction = model.predict(test_image)'. Hope you guys understood my problem and Thanks in advance...

Output:
I'd guess that the checkpoint you are loading stores a model state dict (the model's parameters) rather than a model (the structure of the model plus its parameters). Try:
model = resnet18(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()

where PATH is the path to the model checkpoint. You need to declare model as an instance of the object class (declare the model structure) so that you can load the checkpoint (parameters only, no structure). So you'll need to find the appropriate class to import for the resnet18, probably something along the lines of:
from torchvision.models import resnet18


Instruction: how to convert the output of a nural network to long type while maintaining the trainability

Input:
The output of my pytorch neural network is a float64 type of data. This variable has to be used as a pixel offset and as such I need to convert it to long type.
However I have just discovered that a conversion out=out.long() switches the variable attribute ".requires_grad" to False.
How can I convert it to long maintaining ".requires_grad" to true?

Output:
In general, you cannot convert a tensor to an integer-based type while maintaining it's gradient properties since converting to an integer is a non-differentiable operation. Thus, you essentially have two options:

If the data is only required as type long for inference operations that need not maintain their gradient,you can back-propagate loss before converting to long type, sequentially. You could also make a copy or use torch.detach().

Change the input-output structure of your model such that integer outputs are not needed. One way to do this might be to output a pixel-map with one value for each value in the original tensor which you are trying to index. This would be similar to NNs that output masks for segmentation.

Without more detail on what you're trying to accomplish, it's difficult to say what your best path forward is. Please add more code so the context of this operation is visible.


Instruction: Why is my loss not decreasing over training 10 epochs?

Input:
My hardware is a Ryzen 5000 series cpu with an nvidia rtx 3060 gpu. I'm currently working on a school assignment involving using a deep learning model (implemented in PyTorch) to predict COVID diagnosis from CT slice images. The dataset can be found at this url on GitHub: https://github.com/UCSD-AI4H/COVID-CT
I've written a custom dataset that takes the images from the dataset and resizes them to 224x224. I've also converted all rgba or grayscale images to rgb using skimage.color. Other transforms include random horizontal and vertical flipping, as well as ToTensor(). To evaluate the model I've used sklearn.metrics to compute the AUC, F1 score, and accuracy of the model.
My trouble is that I can't get the model to train. After 10 epochs the loss has not decreased. I've tried adjusting the learning rate of my optimizer but it hasn't helped. Any recommendations/thoughts would be greatly appreciated. Thanks!
class RONANet(nn.Module):
  def __init__(self, classifier_type=None):
    super(RONANet, self).__init__()
    self.classifier_type = classifier_type
    self.relu = nn.ReLU()
    self.maxpool = nn.MaxPool2d(kernel_size=2, stride=2)
    self.classifier = self.compose_classifier()
    self.conv_layers = nn.Sequential(
        nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(32),
        self.relu,
        self.maxpool,
        nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        self.relu,
        self.maxpool,
        nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        self.relu,
        self.maxpool,
        nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
        self.relu,
        self.maxpool,
        nn.AdaptiveAvgPool2d(output_size=(1,1)),
    )
  def compose_classifier(self):
    if 'fc' in self.classifier_type:
            classifier = nn.Sequential(
                nn.Flatten(),
                nn.Linear(14**2*256, 256),
                self.relu,
                nn.Linear(256, 128),
                self.relu,
                nn.Linear(128, 2))

    elif 'conv'in self.classifier_type:
        classifier = nn.Sequential(
            nn.Conv2d(256, 1, kernel_size=1, stride=1))
        return classifier

  def forward(self, x):
    features = self.conv_layers(x)
    out = self.classifier(features)
    if 'conv' in self.classifier_type:
        out = out.reshape([-1,])
    return out

RONANetv1 = RONANet(classifier_type='conv')
RONANetv1 = RONANetv1.cuda()
RONANetv2 = RONANet(classifier_type='fc')
RONANetv2 = RONANetv2.cuda()

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(RONANetv1.parameters(), lr=0.1)
num_epochs = 100
best_auc = 0.5 # set threshold to random model performance
scores = {}

for epoch in range(num_epochs):
RONANetv1.train()
print(f'Current Epoch: {epoch+1}')
epoch_loss = 0

for images, labels in train_dataloader:
    batch_loss = 0
    optimizer.zero_grad()
    with torch.set_grad_enabled(True):
        images = images.cuda()
        labels = labels.cuda()
        out = RONANetv1(images)
        loss = criterion(out, labels)
        batch_loss += loss.item()
        loss.backward()
        optimizer.step()
        epoch_loss += batch_loss
print(f'Loss this epoch: {epoch_loss}\n')

current_val_auc, current_val_f1, current_val_acc = get_scores(RONANetv1, val_dataloader)
if current_val_auc > best_auc:
    best_auc = current_val_auc
    torch.save(RONANetv1.state_dict(), 'RONANetv1.pth')
    scores['AUC'] = current_val_auc
    scores['f1'] = current_val_f1
    scores['Accuracy'] = current_val_acc
    print(scores)

.
Output:

Current Epoch: 1
Loss this epoch: 38.038745045661926

{'AUC': 0.6632183908045978, 'f1': 0.0, 'Accuracy': 0.4915254237288136}

Current Epoch: 2
Loss this epoch: 37.96312761306763

Current Epoch: 3
Loss this epoch: 37.93656861782074

Current Epoch: 4
Loss this epoch: 38.045261442661285

Current Epoch: 5
Loss this epoch: 38.01626980304718

Current Epoch: 6
Loss this epoch: 37.93017905950546

Current Epoch: 7
Loss this epoch: 37.913547694683075

Current Epoch: 8
Loss this epoch: 38.049841582775116

Current Epoch: 9
Loss this epoch: 37.95650988817215

Output:
So the issue is you're only training the first part of the classifier and not the second
# this
optimizer = torch.optim.Adam(RONANetv1.parameters(), lr=0.1)
# needs to become this
from itertools import chain
optimizer = torch.optim.Adam(chain(RONANetv1.parameters(), RONANetv2.parameters()))

and you need to incorportate the other cnn in training too
intermediate_out = RONANetv1(images)
out = RONANetv2(intermediate_out)
loss = criterion(out, labels)
batch_loss += loss.item()
loss.backward()
optimizer.step()

Hope that helps best of luck!


Instruction: How to add training end callback to AllenNLP config file?

Input:
Currently training models using AllenNLP 1.2 and the commands api:
allennlp train -f --include-package custom-exp /usr/training_config/mock_model_config.jsonnet -s test-mock-out
I'm trying to execute a forward pass on a test dataset after training is completed. I know how to add an epoch_callback, but am not sure about the syntax for the end_callback.
In my config.json, I have the following:
{
...
"trainer": {
...
"epoch_callbacks": [{"type": 'log_metrics_to_wandb',},]
}
...
}

I've tried:
              "end_callback": [{"type": 'my_custom_function',},]

but got an illegal argument error. Also, I am not sure how I would accurately specify the exact custom function and communicate it to the trainer.

Output: I think you can create a new callback function/object that inherits from TrainerCallback and override the on_end method, and then it should work as expected if you register it the same way as you did log_metrics_to_wandb above.


Instruction: PyTorch can't use a float type but only long

Input:
I am trying to run this very basic neural network:
import os; os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import time

#####################################################
#             Create the neural network             #
#####################################################

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 10)
        self.fc2 = nn.Linear(10, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

#####################################################
#                   Create the datasets             #
#####################################################

trainset = [torch.tensor([1., 1.**2]), torch.tensor([2., 2.**2]), torch.tensor([3., 3.**2]), torch.tensor([4., 4.**2]), torch.tensor([5., 5.**2]), torch.tensor([6., 6.**2]), torch.tensor([7., 7.**2]), torch.tensor([8., 8.**2])]

testset = [torch.tensor([1.1, 1.1**2]), torch.tensor([2.3, 2.3**2]), torch.tensor([3.1, 3.1**2]), torch.tensor([4.5, 4.5**2]), torch.tensor([5.9, 5.9**2]), torch.tensor([6.1, 6.1**2]), torch.tensor([7.3, 7.3**2]), torch.tensor([8.01, 8.01**2])]

#####################################################
#               Optimize the parameters             #
#####################################################

optimizer = torch.optim.Adam(net.parameters(), lr=0.001)

EPOCHS = 3

for epoch in range(EPOCHS):
    for data in trainset:
        x, y = data
        net.zero_grad()
        output = net(x.view(-1,1))
        loss = F.nll_loss(output, y.view(-1,1)[0])
        loss.backward()
        optimizer.step()
    print(loss)

#####################################################
#           Calculate the accuracy rate             #
#####################################################

correct = 0
total = 0

with torch.no_grad():
    for data in trainset:
        x, y = data
        output = net(x)
        if y - 0.01 < output < y + 0.01:
            correct += 1
        total += 1

print("Accuracy: %.2f" % (correct / total))

but I get the following error:

Traceback (most recent call last):   File
"C:\Users\Andrea\Desktop\pythonProject\main.py", line 52, in
loss = F.nll_loss(output, y.view(-1,1)[0])   File "C:\WinPython\python-3.9.1.amd64\lib\site-packages\torch\nn\functional.py",
line 2235, in nll_loss
ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index) RuntimeError: expected scalar type Long but found Float

Why can't I use a float type?

Output:
The negative log likelihood loss (NLLLoss) is suitable for classification problems, where the output is one out of C classes. Since the classes are discrete, your labels need to be of the long type.
In your case, in a comment, you say:

I want to create a network that simulates a quadratic function with x as input and sth similar to x**2 as output.

This is a regression problem, where the output can have a real, continuous value. For this, you should use a suitable loss function such as the mean squared error loss (MSELoss). So, one way to fix would be changing F.nll_loss in your code to F.mse_loss.


Instruction: Tensorflow and Torch on the same environment

Input:
I am working on a goat detection problem. The main model identifies "goat" (single class) on an image and each goat then cropped from the original image. The cropped image then passes through a tensorflow model (trained tensorflow.keras.applications.InceptionV3) to find the current posture of the goat (sitting and standing - two classes).
The version of the torch should be 1.7+ and I am trying to use any version of the tensorflow (1.15.1/1.13.0 preferred). If I install tensorflow while torch is already installed, both tensorflow and torch disappear from the environment (pip freeze). Are there any compatible versions out there? Any specific version of TensorFlow and torch can be on the same environment?
Here are the commands -
conda create -n tst2 python=3.7
conda activate tst2

# install torch 1.7 gpu with cuda 10.1
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch

if I import torch, it works fine.
Then I Installed tensorflow 2.1.0 gpu
conda install -c anaconda tensorflow-gpu

Now, I can import tensorflow but no torch
>>> import tensorflow
2021-05-02 17:47:23.158416: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'

Output:
Install tensorflow-gpu=2.6.0 and PyTorch with cudatoolkit=11.3 and its working fine with anaconda.
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
conda install tensorflow-gpu


Instruction: Do we have lower performance and accuracy than when not using `pytorch.nn.Sequnetial` and if yes, why?

Input:
I was checking out this video where Phil points out to this fact that using torch.nn.Sequential is faster than not using it. I did quick google and came across this post which is not answered satisfactorily, so I am replicating it here.
Here is the code from the post with Sequential:
class net2(nn.Module):
    def __init__(self):
        super(net2, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=1,
                padding=2,
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.out = nn.Linear(32 * 7 * 7, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output

And here is the net without Sequential:
class net1(nn.Module):
    def __init__(self):
        super(net1, self).__init__()
        self.conv1 = nn.Conv2d(1,16,5,1,2)
        self.conv2 = nn.Conv2d(16,32,5,1,2)
        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(32*7*7,10)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = self.pool1(out)
        out = F.relu(self.conv2(out))
        out = self.pool2(out)

        out = out.view(-1,32*7*7)
        out = F.relu(self.fc1(out))
        return out

In the comment, the author also states that the accuracy is better with Sequential than without it. One of the comment states that speed is more with Sequential because loop unrolling results in faster execution. So here are my related questions:

Is loop unrolling only reason why Sequntial implementation faster than the other one?

Can someone explain how non-Sequential does not result in loop unrolling and Sequntial results in loop-unrolling, perhaps by pointing me to pytorch source lines on github

Does Sequential really result in more accuracy, if yes why?

Output:
I'm not familiar with what kind of optimizations the python interpreter does, but I'd guess it is very limited.
But the claim that one method is more accurate than another is completely nonsense. If you look at the implementation of nn.Sequential you'll see that it does exactly the same thing you'd do anyway, you just iterate over the modules and pass the output of one to the input of the next:
def forward(self, input):
    for module in self:
        input = module(input)
    return input

There might be a tiny difference in speed due to the overhead of all the additional things that nn.Sequential might do, but this is negligible.


Instruction: only first gpu is allocated (eventhough I make other gpus visible, in pytorch cuda framework)

Input:
I am using cuda in pytorch framwework in linux server with multiple cuda devices.
The problem is that
eventhough I specified certain gpus that can be shown,
the program keeps using only first gpu.
(But other program works fine and other specified gpus are allocated well.
because of that, I think it is not nvidia or system problem.
nvidia-smi shows all gpus well and there's no problem.
I didn't have problem with allocating gpus with below codes before (except when the system is not working)
)
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBILE_DEVICES"] = str(args.gpu)

I wrote that before running main function.
and it works fine for other programs in same system.
I printed that args.gpu variable, and could see that the value is not "0".

Output:
Have you tried something like this?
device = torch.device("cuda:0,1" if torch.cuda.is_available() else "cpu") ## specify the GPU id's, GPU id's start from 0.

model = CreateModel()

model= nn.DataParallel(model,device_ids = [0, 1])
model.to(device)

let me know about this


Instruction: CUDA version of package not importing?

Input:
Firstly, I installed torch 1.1.0, and then I installed its' dependencies. So, I can import torch_scatter 1.2.0 however I get this error when importing torch_scatter.scatter_cuda:
 import torch_scatter.scatter_cuda
ModuleNotFoundError: No module named 'torch_scatter.scatter_cuda'

I have Cuda v10 installed and I have a GPU. All of the requirements for this code were installed together through pip in one go on my virtual environment.

Output:
As pointed out by phd - it looks like the setup.py file of pytorch_scatter checks for and uses an available cuda installation automatically.
Also in the version you are using as seen here:
...
if CUDA_HOME is not None:
    ext_modules += [
                   CUDAExtension('torch_scatter.scatter_cuda',
                  ['cuda/scatter.cpp', 'cuda/scatter_kernel.cu'])
    ]
...

Might be a question of whether CUDA_HOME is available.
Installing from source might give you more information as suggested here.


Instruction: ImageNet pretrained ResNet50 backbones are different between Pytorch and TensorFlow

Input:
"Obviously!", you might say... But there's one significant difference that I have trouble explaining by the difference in random initialization.
Take the two pre-trained basenets (before the average pooling layer) and feed them with the same image, you will notice that the output features don't follow the same distribution. Specifically, TensorFlow's backbone has more inhibited features by the ReLU compared to Pytorch's backbone. Additionally, as shows in the third figure, the dynamic range is different between the two frameworks.

Of course, this difference is absorbed by the dense layer addressing the classification task, but: Can that difference be explained by randomness in the training process? Or training time? Or is there something else that would explain the difference?
Code to reproduce:
import imageio
import numpy as np
image = imageio.imread("/tmp/image.png").astype(np.float32)/255

import tensorflow as tf
inputs = image[np.newaxis]
model = tf.keras.applications.ResNet50(include_top=False, input_shape=(None, None, 3))
output = model(inputs).numpy()
print(f"TensorFlow features range: [{np.min(output):.02f};{np.max(output):.02f}]")

import torchvision
import torch
model = torch.nn.Sequential(*list(torchvision.models.resnet50(pretrained=True).children())[0:8])
inputs = torch.tensor(image).permute(2,0,1).unsqueeze(0)
output = model(inputs).detach().permute(0,2,3,1).numpy()
print(f"Pytorch features range: [{np.min(output):.02f};{np.max(output):.02f}]")

Outputting
TensorFlow features range: [0.00;25.98]
Pytorch features range: [0.00;12.00]

Note: it's similar to any image.

Output:
There are 2 things that differ in the implementations of ResNet50 in TensorFlow and PyTorch that I could notice and might explain your observation.

The batch normalization does not have the same momentum in both. It's 0.1 in PyTorch and 0.01 in TensorFlow (although it is reported as 0.99 I am writing it down in PyTorch's convention for comparison here). This might affect training and therefore the weights.

TensorFlow's implementation uses biases in convolutions while PyTorch's one doesn't (as can be seen in the conv3x3 and conv1x1 definitions). Because the batch normalization layers are affine, the biases are not needed, and are spurious. I think this is truly what explains the difference in your case since they can be compensated by the batch norm, and therefore be arbitrarily large, which would be why you observe a bigger range  for TF.
Another way to see this is to compare the summaries as I did in this colab.

I currently have a PR that should fix the bias part (at least provide the possibility to train a resnet without conv bias in TF), and plan on submitting one for BN soon.
EDIT
I have actually found out more differences, that I listed in a paper I recently wrote. You can check them in Table 3 of the F appendix.
I list here for completeness of the answer, those that might have an impact on the output features statistics:

the variance estimation in the batch norm is different
the convolution weights and classification head weights and bias initialization are not the same


Instruction: Why some weights of GPT2Model are not initialized?

Input:
I am using the GPT2 pre-trained model for a research project and when I load the pre-trained model with the following code,
from transformers.models.gpt2.modeling_gpt2 import GPT2Model
gpt2 = GPT2Model.from_pretrained('gpt2')

I get the following warning message:

Some weights of GPT2Model were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

From my understanding, it says that the weights of the above layers are not initialized from the pre-trained model. But we all know that attention layers ('attn') are so important in GPT2 and if we can not have their actual weights from the pre-trained model, then what is the point of using a pre-trained model?
I really appreciate it if someone could explain this to me and tell me how I can fix this.

Output: The masked_bias was added but the huggingface community as a speed improvement compared to the original implementation. It should not negatively impact the performance as the original weights are loaded properly. Check this PR for further information.


Instruction: What is the TensorFlow/Keras equivalent of PyTorch's `no_grad` function?

Input:
When writing machine learning models, I find myself needing to compute metrics, or run additional forward-passes in callbacks for visualization purposes. In PyTorch, I do this with torch.no_grad(), and this prevents gradients from being computed and these operations, therefore, do not influence the optimization.

How does this mechanism work in TensorFlow/Keras?
Keras models are callable. So, something like model(x) is possible. But, it is also possible to say model.predict(x), which also seems to invoke the call. Is there a difference between the two?

Output:
The tensorflow equivalent would be tf.stop_gradient
Also don't forget, that Keras does not compute gradients when using predict (or just calling the model via __call__).


Instruction: How to merge sequential integer values into intervals in PyTorch?

Input:
I have a 1D array:
[3, 4, 5, 6, 7, 20, 31, 32, 33, 34]

which I want to turn into a 2D interval array by merging every consecutive values into intervals:
[
  [ 3,  7],
  [20, 20],
  [32, 34]
]

What would be a decent, possibly GPU-friendly, way to do it?

Output:
Not sure if this is ideal, but you can try cumsum() on the differences compared to 1. Then use that to slice the original data:
# mark the consecutive blocks
blocks = torch.cat([torch.tensor([0]),(t[1:] - t[:-1]) != 1]).cumsum(dim=0)

# where the blocks shift
mask =  blocks[1:] != blocks[:-1]

out = torch.cat([t[:1], t[:-1][mask],t[1:][mask], t[-1:]]).reshape(-1,2)

Output:
tensor([[ 3,  7],
        [20, 20],
        [31, 34]])


Instruction: Preprocessing a video in android for pytorch

Input:
What is the best way to preprocess video data in Android Kotlin, in preparation to feed into a PyTorch Android model? Specifically, I have a ready-made model in PyTorch, and I've converted it to be ready for PyTorch Mobile.
During training, the model takes in raw footage from phones and is preprocessed to (1) be greyscale, and (2) compressed to a specific smaller resolution that I have specified, (3) converted to a Tensor to be fed into a neural net (or potentially send the compressed video to a remote server). I use OpenCV for this, but I'm wondering what the easiest way to do this would be in Android Kotlin?
Python code for reference:

def save_video(filename):

    frames = []

    cap = cv2.VideoCapture(filename)
    frameCount = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    frameWidth = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    frameHeight = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    buf_c = np.empty((frameCount, frameHeight, frameWidth, 3), np.dtype('uint8'))
    buf = np.empty((frameCount, frameHeight, frameWidth), np.dtype('uint8'))

    fc = 0
    ret = True

    # 9:16 ratio
    width = 121
    height = 216
    dim = (width, height)

    # Loop until the end of the video
    while fc < frameCount and ret:
        ret, buf_c[fc] = cap.read()

        # convert to greyscale
        buf[fc] = cv2.cvtColor(buf_c[fc], cv2.COLOR_BGR2GRAY)

        # reduce resolution
        resized = cv2.resize(buf[fc], dim, interpolation = cv2.INTER_AREA)

        frames.append(resized)
        fc += 1

    # release the video capture object
    cap.release()

    # Closes all the windows currently opened.
    cv2.destroyAllWindows()

    return frames

Output:
You said that your model was converted to be ready for PyTorch Mobile so I will assume that you scripted your model with TorchScript.
With TorchScript, you can write preprocessing logic using Torch operation and keep it inside the scripted model like this:
import torch
import torch.nn.functional as F

@torch.jit.script_method
def preprocess(self,
               image: torch.Tensor, # This should have format HxWx3
               height: int,
               width: int) -> torch.Tensor:
    img = image.to(self.device)

    # (1) Convert to Grayscale
    img = ((img[:, :, 0] + img[:, :, 1] + img[:, :, 2]) / 3).unsqueeze(-1)

    # (2) Resize to specified resolution
    # Mimic torchvision.transforms.ToTensor to use interpolate
    img = img.float()
    img = img.permute(2, 0, 1).unsqueeze(0)
    img = F.interpolate(img, size=(
        height, width), mode="bicubic", align_corners=False)
    img = img.squeeze(0).permute(1, 2, 0)
    # Then turn it back to normal image tensor

    # (3) Other normalization like mean substraction and convert to BxCxHxW format
    img -= self.mean_tensor  # mean substraction
    img = img.permute(2, 0, 1).unsqueeze(0)
    return img

So all the preprocess will be done by libtorch, not opencv.


Instruction: Use Adam optimizer for LSTM network vs LBGFS

Input:
I have modified pytorch tutorial on LSTM (sine-wave prediction: given [0:N] sine-values -> [N:2N] values) to use Adam optimizer instead of LBFGS optimizer. However, the model does not train well and cannot predict sine-wave correctly. Since in most cases we use Adam optimizer for RNN training, I wonder how this issue can be resolved. I also wonder if the code segment regarding sequence-in-sequence-out (done with a loop: for input_t in input.split(1, dim=1)), can be done by a pytorch module or function.
from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib
#matplotlib.use('Agg')
import matplotlib.pyplot as plt

class Sequence(nn.Module):
    def __init__(self):
        super(Sequence, self).__init__()
        self.lstm1 = nn.LSTMCell(1, 51)
        self.lstm2 = nn.LSTMCell(51, 51)
        self.linear = nn.Linear(51, 1)

    def forward(self, input, future = 0):
        outputs = []
        h_t = torch.zeros(input.size(0), 51, dtype=torch.double)
        c_t = torch.zeros(input.size(0), 51, dtype=torch.double)
        h_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)
        c_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)

        for input_t in input.split(1, dim=1):
            h_t, c_t = self.lstm1(input_t, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
            output = self.linear(h_t2)
            outputs += [output]
        for i in range(future):# if we should predict the future
            h_t, c_t = self.lstm1(output, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
            output = self.linear(h_t2)
            outputs += [output]
        outputs = torch.cat(outputs, dim=1)
        return outputs

if __name__ == '__main__':
    # set random seed to 0
    np.random.seed(0)
    torch.manual_seed(0)
    # load data and make training set
    data = torch.load('traindata.pt')
    input = torch.from_numpy(data[3:, :-1])
    target = torch.from_numpy(data[3:, 1:])
    test_input = torch.from_numpy(data[:3, :-1])
    test_target = torch.from_numpy(data[:3, 1:])
    print("input.size", input.size())
    print("target.size", target.size())
    # build the model
    seq = Sequence()
    seq.double()
    criterion = nn.MSELoss()
    # use LBFGS as optimizer since we can load the whole data to train
    optimizer = optim.Adam(seq.parameters(), lr=0.005)
    #begin to train
    for i in range(15):
        print('STEP: ', i)
        seq.train()
        def run1step():
            optimizer.zero_grad()
            out = seq(input)
            loss = criterion(out, target)
            print('train loss:', loss.item())
            loss.backward()
            return loss
        run1step()
        optimizer.step()
        # begin to predict, no need to track gradient here
        seq.eval()
        with torch.no_grad():
            future = 1000
            pred = seq(test_input, future=future)
            loss = criterion(pred[:, :-future], test_target)
            print('test  loss:', loss.item())
            y = pred.detach().numpy()
        # draw the result
        def draw(yi, color):
            plt.figure(figsize=(30,10))
            plt.title('Predict future values for time sequences\n(Dashlines are predicted values)', fontsize=30)
            plt.xlabel('x', fontsize=20)
            plt.ylabel('y', fontsize=20)
            plt.xticks(fontsize=20)
            plt.yticks(fontsize=20)
            plt.plot(np.arange(input.size(1)), yi[:input.size(1)], color, linewidth = 2.0)
            plt.plot(np.arange(input.size(1), input.size(1) + future), yi[input.size(1):], color + ':', linewidth = 2.0)
            plt.show()
        if i == 14:
          draw(y[0], 'r')
          draw(y[1], 'g')
          draw(y[2], 'b')
          plt.savefig('predict_LSTM%d.pdf'%i)
          #plt.close()

Output:
I've just executed your code and the original code. I think the problem is you didn't train your code with ADAM long enough. You can see your training loss is still getting smaller at step 15. So I changed the number of steps from 15 to 45 and this is the figure generated after step 40:

The original code reached 4e-05 loss after step 4. But after that, the loss somehow exploded. Your code with ADAM can reduce the loss across all 45 steps, but the final loss is around 0.001. I hope I run both programs correctly.
Oh, regarding your second question.

also wonder if the code segment regarding sequence-in-sequence-out

Yes, you can write a function or define a module with two LSTMs to do that. But it doesn't make sense since your network contains only two LSTMs. After all, you have to do this “wiring“ work at some point.
If your network contains several such blocks, you can write a module with two LSTMs and use it as a primitive module, e.g. self.BigLSTM = BigLSTM(...) , just like you define self.lstm1 = nn.LSTMCell(...).


Instruction: Does pytorch Dataset.__getitem__ have to return a dict?

Input:
EDIT:  This is not about the general __getitem__ method but the usage of __getitem__ in the Pytorch Dataset-subclass, as @dataista correctly states.
I'm trying to implement the usage of Pytorchs Dataset-class.
The guide e.g here is really good, but I struggle to figure out Pytorch requirements for the return value of __getitem__. In the Pytorch documentation I cannot find anything about what it should return; is it any object which is iterable with size 2 e.g [sample,target], (sample,target)? In some guides they return a dict, but they do not specify if it has to be a dict which is returned.

Output:
PyTorch has no requirements on the return value of a DataSet's __getitem__ method. It can be anything, but you will commonly encounter a tensor, a tuple of tensors, a dictionary (e.g. {'features':..., 'label':...}) etc.
It is usual in 2d data to return a single tensor whose final column are the target values, but equally you may see tuples/dicts of the features and targets explicitly separated.
Note there is no requirement that you return two values - in many unsupervised contexts (e.g. autoencoders) there is only a set of features, with no distinct target.


Instruction: "Numpy not Available" After installing Pytorch XLA

Input:
I am just getting started with using TPUs on kaggle with Pytorch and install it as follows -
!pip3 install mkl

!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py

!python3 pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev

However, after installing Pytorch XLA, I am not able to use NumPy at all. Whenever I call functions like np.uint8 or even NumPy based functions from like torch.from_numpy I get an error whose bottom line says - NumPy not Available. Please note that I am able to import Numpy.
The whole stack trace is as follows -
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-abfcbbc939b0> in <module>
   1026                                  segmentation_Maps='/kaggle/input/pascal-voc/VOC2012/SegmentationClass/')
   1027 dataloader = DataLoader(dataset, batch_size=5)
-> 1028 for _, data in enumerate(dataloader):
   1029     i = data['image']
   1030     gt = data['ground_truth']

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-1-abfcbbc939b0> in __getitem__(self, item)
    939         print(mask.shape)
    940         image = Image.fromarray(np.uint8(image)).convert('RGB')
--> 941         mask = torch.from_numpy(np.uint8(mask))
    942
    943         image = self.transforms(image)

RuntimeError: Numpy is not available

I have no clue what is going on. Could someone please Help.
PS - please note that pytorch xla updates pytorch to a nightly 1.9 version.
TIA

Output:
At least in Google Colab I was able to solve this issue by running (after installing xla):
!pip install -U numpy

Not completely sure it will help in any context


Instruction: Use PyTorch to speed up linear least squares optimization with bounds?

Input:
I'm using scipy.optimize.lsq_linear to run some linear least squares optimizations and all is well, but a little slow.  My A matrix is typically about 100 x 10,000 in size and sparse (sparsity usually ~50%).  The bounds on the solution are critical.  Given my tolerance lsq_linear typically solves the problems in about 10 seconds and speeding this up would be very helpful for running many optimizations.
I've read about speeding up linear algebra operations using GPU acceleration in PyTorch.  It looks like PyTorch handles sparse arrays (torch calls them tensors), which is good.  However, I've been digging through the PyTorch documentation, particularly the torch.optim and torch.linalg packages, and I haven't found anything that appears to be able to do a linear least squares optimization with bounds.

Is there a torch method that can do linear least squares optimization with bounds like scipy.optimize.lsq_linear?
Is there another way to speed up lsq_linear or to perform the optimization in a faster way?

For what it's worth, I think I've pushed lsq_linear pretty far.  I don't think I can decrease the number of matrix elements, increase sparsity or decrease optimiation tolerances much farther without sacrificing the results.

Output:
Not easily, no.
I'd try to profile lsq_linear on your problem to see if it's pure python overhead (which can probably be trimmed some) or linear algebra. In the latter case, I'd start with vendoring the lsq_linear code and swapping relevant linear algebra routines. YMMV though.


Instruction: How do I proceed to load a ga_instance as ".pkl" format in PyGad?

Input:
I have been trying to load the PyGad trained instance in another file, in order to make some prediction. But I have been having some problems in the loading process.
After the training phase, I saved the instance like this:
The saving function:
filename = 'GNN_CPTNet' #GNN_CPTNet.pkl
ga_instance.save(filename=filename)
The loading function:
loaded_ga_instance = pygad.load(filename=filename)
loaded_ga_instance.plot_result()
But, when I tried to load the instance in a new notebook or script, I could not load the instance, especially the "GNN_CPT Net.pkl" file.

Output:
In the new script, you should define the the fitness function and all the callback functions you used in the original script.
For example, if you used only the on_generation (callback_generation) parameter, then the following functions should be defined:
def fitness_func(solution, solution_idx):
    ...

def callback_generation(ga_instance):
    ...

This way, the saved instance will be loaded correctly.
Anyway, it is better to post the sample codes you used to give more accurate answer.
Thanks for using PyGAD :)


Instruction: Is it advisable to use the same torch Dataset class for training and predicting?

Input:
I have recently started using PyTorch and I liked it for its object-oriented style. However, I wonder what’s the best and advised workflow when predicting the model. I wanted to use a custom Dataset class I wrote and which I use for training and validating my model. This class is a map-style dataset, therefore I implement __getitem__ method to return image and target:
class CustomDataset:

    def __init__(self, ...):
        ...

    def __getitem__(self, image_id):
        ....
        return (
            torch.tensor(image, dtype=torch.float),
            torch.tensor(target, dtype=torch.long),
       )

However, when I’m using this class for predicting I don’t have any targets to return. My current workaround is something like
def __getitem__(self, image_id):
    ....
    if predict:
        return (
            torch.tensor(image, dtype=torch.float),
            np.nan,
       )
   else:
        return (
             torch.tensor(image, dtype=torch.float),
             torch.tensor(target, dtype=torch.long),
       )

However, I wonder if there’s a better way to do it. And at the same time, as it feels a bit unnatural, I started wondering if it is even advisable to use the same class for training and predicting (it should be, but the clunkiness of my solutions makes me wonder). Of course, I could not return a tuple at all, but only a first element, but this still needs if-else.

Output:
PyTorch's DataSet class is really simple. So, do not overthink it. It's not much more than a wrapper for accessing your data.
You don't have to return a tuple, not even Tensors. You can return whatever data you want. Commonly, it will be in one of those styles:

For unsupervised data: Sample or (Sample, None)
For supervised data: (Sample, Label)
For supervised data with multiple targets, e.g. object detection: (Sample, [Label1, Label2, ...]) or (Sample, Label1, Label2, ...)

It is also common to use the same DataSet class for train / test.
So, in your case, simply return the sample or a tuple (sample, None) as done in torchvision and adjust your pipeline accordingly. I'd not suggest using np.nan as it would fail a simple None check (np.nan == None). Also, I'd encourage you to inherit from torch.data.Dataset.
If however, your pipeline forces you to use a tuple or has other constraints I'd suggest to rephrase your question.


Instruction: TypeError when using torch.autograd.profiler.profile

Input:
I am trying to analyze memory consumption for my model as described here:
https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html
using these lines:
with profiler.profile(profile_memory=True, record_shapes=True) as prof:
    tubes, _, _ = zip(*model(imgs, img_metas, return_loss=False))
print(prof.key_averages().table(sort_by="self_cpu_memory_usage", row_limit=10))

but I get this error

TypeError: init() got an unexpected keyword argument
'profile_memory'

So, what would be the reason and/or the solution for this error?

Output:
You need to upgrade your version of PyTorch. Looking at the code, one can see that the profile_memory argument was added to the function signature first in PyTorch v1.6.0.
You can also see this through the documentation of torch.autograd. You can see that the argument is not present in PyTorch v1.1.0.


Instruction: what does pytorch do for creating tensor from numpy

Input:
I am interested in what torch have done when I call torch.from_numpy. As the name indicates, it seems that PyTorch creates a Tensor instance and allocates the memory for copying the content from numpy ndarray to itself. But how does PyTorch do the memcpy work and what else has PyTorch done in the background? It seems the implementation of tensor is in autograd. But I have no idea which part should I look for.
I have the question because I found it is really fast for constructing a tensor from numpy. And it is even fast than creating a tensor directly
a = np.random.randn(100,100)
%timeit torch.from_numpy(a)
759 ns ± 7.53 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
%timeit torch.randn(100,100)
61 µs ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit torch.zeros(100,100)
3.1 µs ± 136 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Output:
The documentation explains that

The returned tensor and ndarray share the same memory. Modifications to the tensor will be reflected in the ndarray and vice versa. The returned tensor is not resizable.

These sentences implies that there is no memcopy involved (otherwise modifications would not be reflected in one aonther). That is why the operation is so fast : pytorch merely creates a pointer to the numpy array underlying data, and "assigns" this pointer to a tensor. This function does not allocate or copy any memory at all. Therefore, from_numpy is just duplicating a pointer (which is an integer number) and probably performing a few checks.
What is important to remember is really that the underlying memories are shared and therefore the tensor and the numpy array modify one another, and you should use clone or copy to perform a clean deep copy and get rid of this behavior (if you need to), like
b = torch.from_numpy(a).clone()


Instruction: Custom Pytorch layer to apply LSTM on each group

Input:
I have a N × F tensor with features and a N × 1 tensor with group index. I want to design a custom pytorch layer which will apply LSTM on each group with sorted features. I have mentioned LSTM with sorted group features as an example, hypothetically it can be anything which supports variable length input or sequence. Please refer to the image below for visual interpretation of the problem.

The obvious approach would be calling a LSTM layer for each unique group but that would be inefficient. Is there any better way to do it?

Output:
You can certainly parallelize the LSTM application -- the problem is indexing the feature tensor efficiently.
The best thing I could come up with (I use something similar for my own stuff) would be to list comprehend over the unique group ids to make a list of variable-length tensors, then pad them over and run the LSTM on top.
In code:
import torch
from torch import Tensor
from torch.nn.utils.rnn import pad_sequence

n = 13
f = 77
n_groups = 3

xs = torch.rand(n, f)
ids = torch.randint(low=0, high=n_groups, size=(n,))

def groupbyid(xs: Tensor, ids: Tensor, batch_first: bool,
              padding_value: int = 0) -> Tensor:
   return pad_sequence([xs[ids==idx] for idx in ids.unique()],
                       batch_first=batch_first,
                       padding_value=padding_value)

grouped = groupbyid(xs, ids)
print(grouped.shape)
# torch.Size([3, 5, 77])

You can then apply your LSTM in parallel over the n_groups dimension on the grouped Tensor.
Note that you will also need to inspect the content of ids.unique() to assign each LSTM output to its corresponding group id, but this is easy to write and depends on your application.


Instruction: Is a .pth file a security risk, and how can we sanitise it?

Input:
It's well-established that pickled files are  [unsafe][1] to simply load directly. However, the advice on that SE post concludes that basically one should not use a pickled file if they are not sure of its provenance.
What about PyTorch machine-learning models that are stored as .pth files on, say, public repos on Github, which we want to use for inference? For example, if I have a model.pth which I plan to load with torch.load(model.pth), is it necessary to check it's safe to do so? Assuming I have no choice but to use the model, how should one go about checking it?
Given these models are ultimately just weights, could we do something like make a minimal Docker container with PyTorch, load the model inside there, and then resave the weights? Is this necessary, and what sort of checking should we do (i.e. assuming it is safe to load within the container, what sort of code treatment should be applied to sanitise the model for shipping?).
EDIT: response to asking for clarification: say I have a model.pth - (1) do I need to be careful with it, like I would with a .pkl, given a .pth is meant to contain only model weights? Or can I just go ahead and throw it into torch.load(model.pth)? (2) If I can't, what can I do before torch.load() to provide some peace-of-mind?
EDIT 2: An answer should show some focus on ML pretrained models in particular. An example: see the model https://download.pytorch.org/models/resnet34-333f7ec4.pth (warning: 80 MB download from TorchHub - feel free to use another Resnet .pth model, but any cleaning does need to be reasonably fast). Currently I would download this and then load it in PyTorch using load_state_dict (explained in detail [here][2] for example). If I didn't know this was a safe model, how could I try to sanitise it first before loading it in load_state_dict?

  [1]: https://stackoverflow.com/questions/25353753/python-can-i-safely-unpickle-untrusted-data
  [2]: https://www.programmersought.com/article/95324315915/

Output:
As pointed out by @MobeusZoom, this is answer is about Pickle and not PyTorch format. Anyway as PyTorch load mechanism relies on Pickle behind the scene observations drawn in this answer still apply.
TL;DR;

Don't try to sanitize pickle. Trust or reject.

Quoted from Marco Slaviero in his presentation Sour Pickle at the Black Hat USA 2011.
Real solution is:

Don't setup exchange with unequally trusted parties;
Setup a secure transport layer for exchange;
Sign exchanged files;

Also be aware that there are new kind of AI based attacks, this even if the pickle is shellcode free, you still may have other issues to address when loading pre-trained networks from untrusted sources.
Important notes
From the presentation linked above we can draw several important notes:

Pickle use a Virtual Machine to reconstruct live data (PVM happens alongside the python process), this virtual machine is not Turing complete but has: an instruction set (opcodes), a stack for the execution and a memo to host object data. This is enough for attacker to create exploits.
Pickle mechanism is backward compatible, it means latest Python can unpickle the very first version of its protocol.
Pickle can (re)construct any object as long as PVM does not crash, there is no consistency check in this mechanism to enforce object integrity.
In broad outline, Pickle allows attacker to execute shellcode in any language (including python) and those code can even persist after the victim program exits.
Attacker will generally forge their own pickles because it offers more flexibility than naively using pickle mechanism. Off course they can use pickle as an helper to write opcode sequences. Attacker can craft the malicious pickle payload in two significant ways:

to prepend the shellcode to be executed first and leave the PVM stack clean. Then you probably get a normal object after unpickling;
to insert the shellcode into the payload, so it gets executed while unpickling and may interact with the memo. Then unpickled object may have extra capabilities.

Attackers are aware of "safe unpickler" and know how to circumvent them.

MCVE
Find below a very naive MCVE to evaluate your suggestion to encapsulate cleaning of suspect pickled files in Docker container. We will use it to assess main associated risks. Be aware, real exploit will be more advanced and complexer.
Consider the two classes below, Normal is what you expect to unpickle:
# normal.py
class Normal:

    def __init__(self, config):
        self.__dict__.update(config)

    def __str__(self):
        return "<Normal %s>" % self.__dict__

And Exploit is the attacker vessel for its shellcode:
# exploit.py
class Exploit(object):

    def __reduce__(self):
        return (eval, ("print('P@wn%d!')",))

Then, the attacker can use pickle as an helper to produce intermediate payloads in order to forge the final exploit payload:
import pickle
from normal import Normal
from exploit import Exploit

host = Normal({"hello": "world"})
evil = Exploit()

host_payload = pickle.dumps(host, protocol=0) # b'c__builtin__\neval\np0\n(S"print(\'P@wn%d!\')"\np1\ntp2\nRp3\n.'
evil_payload = pickle.dumps(evil, protocol=0) # b'(i__main__\nNormal\np0\n(dp1\nS"hello"\np2\nS"world"\np3\nsb.'

At this point the attacker can craft a specific payload to both inject its shellcode and returns the data.
with open("inject.pickle", "wb") as handler:
    handler.write(b'c__builtin__\neval\np0\n(S"print(\'P@wn%d!\')"\np1\ntp2\nRp3\n(i__main__\nNormal\np0\n(dp1\nS"hello"\np2\nS"world"\np3\nsb.')

Now, when victim will deserialize the malicious pickle file, the exploit is executed and a valid object is returned as expected:
from normal import Normal
with open("inject.pickle", "rb") as handler:
     data = pickle.load(handler)
print(data)

Execution returns:
P@wn%d!
<Normal {'hello': 'world'}>

Off course, shellcode is not intended to be so obvious, you may not notice it has been executed.
Containerized cleaner
Now, lets try to clean this pickle as you suggested. We will encapsulate the following cleaning code:
# cleaner.py
import pickle
from normal import Normal

with open("inject.pickle", "rb") as handler:
    data = pickle.load(handler)
print(data)

cleaned = Normal(data.__dict__)
with open("cleaned.pickle", "wb") as handler:
    pickle.dump(cleaned, handler)

with open("cleaned.pickle", "rb") as handler:
    recovered = pickle.load(handler)
print(recovered)

Into a Docker image to try to contain its execution. As a baseline, we could do something like this:
FROM python:3.9

ADD ./exploit ./
RUN chown 1001:1001 inject.pickle

USER 1001:1001

CMD ["python3", "./cleaner.py"]

Then we build the image and execute it:
docker build -t jlandercy/doclean:1.0 .
docker run -v /home/jlandercy/exploit:/exploit jlandercy/doclean:1.0

Also ensure the mounted folder containing the exploit has restrictive ad hoc permissions.
P@wn%d!
<Normal {'hello': 'world'}> # <-- Shellcode has been executed
<Normal {'hello': 'world'}> # <-- Shellcode has been removed

Now the cleaned.pickle is shellcode free. Off course you need to carefully check this assumption before releasing the cleaned pickle.
Observations
As you can see, Docker image does not prevent the exploit to be executed when unpickling but it may help to contain the exploit in some extent.
Points of attention are (not exhaustive):

Having a recent pickle file with the original protocol is a hint but not an evidence of something suspicious.
Be aware even if containerized, you still are running attacker code on your host;
Additionally, attacker may have designed its exploit to break a Docker container, use unprivileged user to reduce the risk;
Don't bind any network to this container as attacker can start a terminal and expose it over a network interface (and potentially to the web);
Depending on how the attacker has designed its exploit data may not be available at all. For the instance, if __reduce__ method actually returns the exploit instead of a recipe to recreate the desired instance. After all the main purpose of this is to make you unpickling it nothing more;
If you intend to dump raw data after loading the suspicious pickle archive you need a strict procedure to detach data from the exploit;
The cleaning step can be a limitation. It relies on your ability to recreate the intended object from the malicious payload. It will depends on what is really reconstructed from the pickle file and how the desired object constructor needs to be parametrized;
Finally, if you are confident in your cleaning procedure, you can mount a volume to access the result after the container exits.


Instruction: from keras.preprocessing.text import one_hot equivalent in pytorch?

Input:
I just started using pytorch for NLP. I found a tutorial that uses from keras.preprocessing.text import one_hot and converts text to one_hot representation given a vocabulary size.
For example:
The input is
vocab_size = 10000
sentence = ['the glass of milk',
            'the cup of tea',
            'I am a good boy']

onehot_repr = [one_hot(words, vocab_size) for words in sentence]

The output is"
[[6654, 998, 8896, 1609], [6654, 998, 1345, 879], [123, 7653, 1, 5678,7890]]

how can i perform the same procedure in pytorch and get the output like above.

Output:
PyTorch fundamentally works with Tensors, and is not designed to work with strings. You can use SK Learn's LabelEncoder to encode your words however:
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit([w for s in sentence for w in s.split()])

onehot_repr = [le.transform(s.split()) for s in sentence]

>>> [array([10,  5,  8,  7]), array([10,  4,  8,  9]), array([0, 2, 1, 6, 3])]


Instruction: How to make VScode launch.json for a Python module

Input:
I'm reseaching self-supervised muchine learning code.
And I have wanted to debug the code with python debugger not pdb.set_trace().
This is python command for ubuntu terminal.
python -m torch.distributed.launch --nproc_per_node=1 main_swav.py \
--data_path /dataset/imagenet/train \
--epochs 400 \
--base_lr 0.6 \
--final_lr 0.0006 \
--warmup_epochs 0 \
--batch_size 8 \
--size_crops 224 96 \
--nmb_crops 2 6 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--use_fp16 true \
--freeze_prototypes_niters 5005 \
--queue_length 380 \
--epoch_queue_starts 15\
--workers 10

In order to debug the code with VScode, I tried to revise launch.json like below as referring stackoverflow -question
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "module": "torch.distributed.launch --nproc_per_node=1 main_swav.py",
            "request": "launch",
            "console": "integratedTerminal",
            "args": ["--data_path", "/dataset/imagenet/train"]
        }
    ]
}

I knew this would not work... TT
Could you give me some advice?
Thank you for your time.

Output:
Specify the module you want to run with "module": "torch.distributed.launch"
You can ignore the -m flag. Put everything else under the args key.
Note: Make sure to include --nproc_per_node and the name of file (main_swav.py) in the list of arguments
{
            "version": "0.2.0",
            "configurations": [
                {
                    "name": "Python: Current File",
                    "type": "python",
                    "module": "torch.distributed.launch",
                    "request": "launch",
                    "console": "integratedTerminal",
                    "args": [
                        "--nproc_per_node", "1",
                        "main_swav.py",
                        "--data_path", "/dataset/imagenet/train",
                    ]
                }
            ]
        }

Read more here: https://code.visualstudio.com/docs/python/debugging#_module


Instruction: PyTorch DataLoader Error: object of type 'type' has no len()

Input:
I'm quite new to programming and have now clue where my error comes from.
I got the following code to set up my dataset for training my classifier:
class cows_train(Dataset):

    def __init__(self, folder_path):
        self.image_list = glob.glob(folder_path+'/content/cows/train')
        self.data_len = len(self.image_list)

    def __getitem__(self, index):
        single_image_path = self.image_list[index]
        im_as_im = Image.open(single_image_path)
        im_as_np = np.asarray(im_as_im)/255
        im_as_np = np.expand_dims(im_as_np, 0)
        im_as_ten = torch.from_numpy(im_as_np).float()
        class_indicator_location = single_image_path.rfind('/content/cows/train/_annotations.csv')
        label = int(single_image_path[class_indicator_location+2:class_indicator_location+3])
        return (im_as_ten, label)

    def __len__(self):
        return self.data_len

And this for the DataLoader:
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = cows_train
trainloader =    torch.utils.data.DataLoader(dataset = trainset, batch_size=10,

shuffle=True, num_workers=2)

classes = ('cow_left', 'cow_other')

As Output I receive:
TypeError Traceback (most recent call last)
<ipython-input-6-54702f98a725> in <module>()
      6
      7 trainset = cows_train
----> 8 trainloader = torch.utils.data.DataLoader(dataset = trainset, batch_size=10, shuffle=True, num_workers=2)
      9
     10 testset = cows_test

2 frames
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context, generator, prefetch_factor, persistent_workers)
    264                     # Cannot statically verify that dataset is Sized
    265                     # Somewhat related: see NOTE [ Lack of Default `__len__` in Python Abstract Base Classes ]
--> 266                     sampler = RandomSampler(dataset, generator=generator)  # type: ignore
    267                 else:
    268                     sampler = SequentialSampler(dataset)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py in __init__(self, data_source, replacement, num_samples, generator)
    100                              "since a random permute will be performed.")
    101
--> 102         if not isinstance(self.num_samples, int) or self.num_samples <= 0:
    103             raise ValueError("num_samples should be a positive integer "
    104                              "value, but got num_samples={}".format(self.num_samples))

/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py in num_samples(self)
    108         # dataset size might change at runtime
    109         if self._num_samples is None:
--> 110             return len(self.data_source)
    111         return self._num_samples
    112

TypeError: object of type 'type' has no len()

Problem is: I don't understand why typ has no length, in my eyes it's defined... Someone please help?
Add: This is where "return len(self.data_source)" shows up in the code

def num_samples(self) -> int:
    if self._num_samples is None:
        return len(self.data_source)
    return self._num_samples

Output:
You are not creating your dataset object correctly. Currently, you do:
trainset = cows_train

This only assigns the class type to trainset. To create an object of the class, you need to use:
folder_path = '/path/to/dataset/'
trainset = cows_train(folder_path)


Instruction: How to make code run on GPU on Windows 10?

Input: I want to run my code run on gpu in windows 10, like for google colab, we can just change the runtime option which is pretty easy to do to shift to gpu. Is there a possibility to do the same for jupyter notebook in windows.

Output:
You will actually need to use tensorflow-gpu to run your jupyter notebook on a gpu.
The best way to achieve this would be

Install Anaconda on your system

Download cuDNN & Cuda Toolkit 11.3 .

Add cuDNN and Cuda Toolkit to your PATH.

Create an environment in Anaconda

pip install tensorflow-gpu

pip install [jupyter-notebook/jupyterlab]

Import tensorflow-gpu in your notebook

Enjoy. You can now run your notebook on your GPU


Instruction: how to use collate_fn properly in the code below?

Input:
My code is:
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))
data_loader = torch.utils.data.DataLoader(
dataset, batch_size=2, shuffle=True, num_workers=4,
collate_fn=utils.collate_fn)
# For Training
images,targets = next(iter(data_loader))
images = list(image for image in images)
targets = [{k: v for k, v in t.items()} for t in targets]
output = model(images,targets)   # Returns losses and detections
# For inference
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)           # Returns predictions

I get the error:

"collate_fn=utils.collate_fn" shows error"name 'utils' is not defined". "module 'torch.utils' has no attribute 'collate_fn' error after adding torch.

Output:
Ok so I read the tutorial and it seems that it wants you to use the helper files in this repository: https://github.com/pytorch/vision/tree/master/references/detection .
In there is the utils.py which contains the collate_fn function.
So it seems that you dont have downloaded/copied this repository to integrate it into your project, right?
To solve just that error, you could just copy the collate_fn in utils.py
def collate_fn(batch):
    return tuple(zip(*batch))

and paste it into your project. But since this tutorial probably wants you to use other util functions of utils.py too, you might want to download this directory and put it into your project directory so you can access it.


Instruction: Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel

Input:
I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:
Error:
Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel:
['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight',   'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias',
 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight',
 'cls.predictions.bias']
    - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I have downloaded the bert model from here and the additional files from here
Code
Following is the code for my model:
import config
import torch
import transformers
import torch.nn as nn

def loss_fn(output, target, mask, num_labels):

    lfn = nn.CrossEntropyLoss()
    active_loss = mask.view(-1) == 1
    active_logits = output.view(-1, num_labels)
    active_labels = torch.where(
        active_loss,
        target.view(-1),
        torch.tensor(lfn.ignore_index).type_as(target)
    )
    loss = lfn(active_logits, active_labels)
    return loss

class EntityModel(nn.Module):
    def __init__(self, num_tag, num_pos):
        super(EntityModel, self).__init__()

        self.num_tag = num_tag
        self.num_pos = num_pos
        self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(p = 0.3)
        self.bert_drop_2 = nn.Dropout(p = 0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
        self.out_pos = nn.Linear(768, self.num_pos)

    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
        o1, _ = self.bert(ids,
                          attention_mask = mask,
                          token_type_ids = token_type_ids)

        bo_tag = self.bert_drop_1(o1)
        bo_pos = self.bert_drop_2(o1)

        tag = self.out_tag(bo_tag)
        pos = self.out_pos(bo_pos)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
        loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)

        loss = (loss_tag + loss_pos) / 2

        return tag, pos, loss

print("model.py run success!")

Output:
As R. Marolahy suggests, if you don't want to see this every time, I know I don't, add the following:
from transformers import logging
logging.set_verbosity_error()


Instruction: TypeError: new(): argument 'size' must be tuple of ints, but found element of type NoneType at pos 2 when using pytorch, using nn.linear

Input:
File "C:\Users\J2\Desktop\Pytorchseries\thenn.py", line 50, in
net = Net()
TypeError: new(): argument 'size' must be tuple of ints, but found element of type NoneType at pos 2
If it helps I was following the sentdex pytorch tutorial. Any help would be appreciated. I am new to machine learning, and I was hoping that this would work. Please help me out!

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import tqdm

training_data = np.load('training_data.npy', allow_pickle=True)
print(len(training_data))

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

plt.imshow(X[0], cmap='gray')
print(y[0])

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50,50).view(-1,1,50,50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

net = Net()
print(net)

import torch.optim as optim

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1  # lets reserve 10% of our data for validation
val_size = int(len(X)*VAL_PCT)
print(val_size)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

print(len(train_X), len(test_X))

BATCH_SIZE = 100
EPOCHS = 1

for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
        #print(f"{i}:{i+BATCH_SIZE}")
        batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)
        batch_y = train_y[i:i+BATCH_SIZE]

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()    # Does the update

    print(f"Epoch: {epoch}. Loss: {loss}")

correct = 0
total = 0
with torch.no_grad():
    for i in tqdm(range(len(test_X))):
        real_class = torch.argmax(test_y[i])
        net_out = net(test_X[i].view(-1, 1, 50, 50))[0]  # returns a list,
        predicted_class = torch.argmax(net_out)

        if predicted_class == real_class:
            correct += 1
        total += 1
print("Accuracy: ", round(correct/total, 3))

Output:
The issue is with self._to_linear. You use it in __init__ as:
self._to_linear = None

self.convs(x)
self.fc1 = nn.Linear(self._to_linear, 512) #flattening.

The call to nn.Linear has it as a parameter. This parameter should equal the number of input features in the linear layer, and cannot be None, since the value will determine the shape of the layer (number of weights and biases). How to fix this depends on what you're trying to achieve.


Instruction: A Problem aboud using multi-gpu with a two-stage CNN model

Input:
I design a CNN model which has two stages. First stage is generating proposals like RPN in Faster RCNN and the second feeds these proposals into the following part.
It causes error in the second step.
Accroding the below error information, it seems like the second input is not correctly assigned to the multi GPU.
However, The model works file with single gpu.
  File "/home/f523/guazai/sdb/rsy/cornerPoject/myCornerNet6/exp/train.py", line 212, in run_epoch
    cls, rgr = self.model([proposal, fm], stage='two')
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 156, in forward
    return self.gather(outputs, self.output_device)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 168, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather
    res = gather_map(outputs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 55, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 68, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/cuda/comm.py", line 166, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: CUDA error: an illegal memory access was encountered

PS
my model sciprt shows below. I want my two-stage model can support for multi batch. e.g. the batch size is 4 and every img output 128 proposal, so the proposal size in here is (4*128, 5)
def _stage2(self, xs):
    proposal, fm = xs
    if proposal.dim()==2 and proposal.size(1) == 5:
        # train mode
        roi = roi_align(fm, proposal, output_size=[15, 15])
    elif proposal.dim()==3 and proposal.size(2) == 4:
        # eval mode
        roi = roi_align(fm, [proposal[0]], output_size=[15, 15])
    else:
        assert AssertionError(" The boxes tensor shape should be Tensor[K, 5] in train or Tensor[N, 4] in eval")
    x = self.big_kernel(roi)
    cls = self.cls_fm(x)
    rgr = self.rgr_fm(x)
    return cls, rgr

Output:
I know where I am wrong. Here’s my second stage to feed input
cls, offset = self.model([proposal, fm], stage='two')

proposal is the ROI whose shape is [N, 5], the 1th dim is the batch index. e.g. The batch size is 4, the range of index is [0,1,2,3]. And fm is the feature map.
When I use the mult-gpu like 2 gpu. the proposal and fm will be split into two branch and feed into two gpu. However the batch index range still be [0,1,2,3], then cause a index error and raise gpu error.
What I do is add a line before roi_align like below:
from torchvision.ops import roi_align
proposal[:, 0] = proposal[:, 0] % fm.size(0) # this make multi-gpu work
roi = roi_align(fm, proposal, output_size=[15, 15])


Instruction: Python - How to output a numpy array of probabilities with certain precision and remain its sum

Input:
I have a numpy array of 6 probabilities which come from pytorch softmax function.
[0.055709425,0.04365404,0.008613999,0.0022386343,0.0037478858,0.88603604]

I want to convert all 6 float numbers to string to represent a score output,
and all of them need to be rounded to a certain precision, say 4.
I used the following code to get the output text:
','.join(f'{x:.4f}' for x in scores)  # scores is the array above

and the output is
0.0557,0.0437,0.0086,0.0022,0.0037,0.8860

which sums up to 0.9999 instead of 1.0. And I have a bunch of arrays like this one but sums up to either 0.9999 or 1.0001.
So my question is, how do I get the output that sums up exactly to 1.0?
I know it's a floating point computation problem. What am I missing, some rounding operation or some adjustment?
Thank you very much.

Output:
You can round off to 2 decimal places, to reduce that error:
For example:
import numpy as np
a = np.array([0.055709425,0.04365404,0.008613999,0.0022386343,0.0037478858,0.88603604])
print(sum(a))

Output:
1.0000000241

Now:
new_array = [round(x,2) for x in a]
print(sum(new_array))

Output:
1.0


Instruction: FileNotFoundError: [Errno 2] :No such file or directory: 'C:/Users/My_computer/Desktop/Compare/MHAN-master/AID_train/AID_train_LR/x4\\9.png'

Input:
I'm very new to python environment. I have tried to compile a super-resolution code for upscaling factor 4 using my own dataset. The low resolution RGB images are kept in "C:/Users/My_computer/Desktop/Compare/MHAN-master/AID_train/AID_train_LR/x4". The code used for image load is shown in below:
def load_img(filepath):
img = Image.open(filepath).convert('RGB')
#img = Image.open(filepath, 'rb')

#y, _, _ = img.split()
return img

class DatasetFromFolder(data.Dataset):
def __init__(self, image_dir, lr_dir, patch_size, upscale_factor, data_augmentation, transform=None):
    super(DatasetFromFolder, self).__init__()
    self.image_filenames = [join(image_dir, x) for x in listdir(image_dir) if is_image_file(x)]
    self.patch_size = patch_size
    self.upscale_factor = upscale_factor
    self.transform = transform
    self.data_augmentation = data_augmentation
    self.HR ='C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_HR'
    self.LR ='C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_LR//x4'

def __getitem__(self, index):
    target = load_img(self.image_filenames[index])
    input = load_img(os.path.join(self.LR, file))
    input, target, _ = get_patch(input,target,self.patch_size, self.upscale_factor)
    return input, target

But I am getting the following error while the training code is compiled:
File "main_x4.py", line 185, in <module>
    train(model, epoch)
  File "main_x4.py", line 60, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\My_computer\Desktop\Compare\MHAN-master\dataset_x4.py", line 91, in __getitem__
    input = load_img(os.path.join(self.LR, file))

File "C:\Users\My_computer\Desktop\Compare\MHAN-master\dataset_x4.py",
line 16, in load_img

img = Image.open(filepath).convert('RGB')

  File "C:\Users\My_computer\anaconda3\envs\MHAN\lib\site-packages\PIL\Image.py",

line 2912, in open
fp = builtins.open(filename, "rb")

FileNotFoundError: [Errno 2] No such file or directory: 'C://Users//My_computer//Desktop//Compare//MHAN-master//AID_train//AID_train_LR//x4\\9.png'

As LR images are already in RGB format, so is it necessary to convert to RGB again?
Please help me to fix this error

Output:
'C:/Users/My_computer/Desktop/Compare/MHAN-master/AID_train/AID_train_LR/x4\\9.png'
Your string contains a double backslash at the end of the path, that's why you can't access the directory
use a raw string like
r'yourString'

or review your os.path.join
EDIT:
Try to convert every string into a raw-String, like mentioned above. You are still getting double backslashes, because certain \character combinations are escaped.
These are the escaped characters:

Edit your code to:
self.HR =r'C:/Users/My_computer/Desktop/Compare/MHAN-
master/AID_train/AID_train_HR'
self.LR =r'C:/Users/My_computer/Desktop/Compare/MHAN-
master/AID_train/AID_train_LR/x4'

Please notice the "r" in front of the string to convert them into raw-Strings.


Instruction: After installing Pytorch cuda , torch.cuda.is_available() show false. What to do?

Input:
I have installed pytorch cuda by running this command:
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

My cuda version is 11.2 . I am using windows 10 .
Pytorch cuda 11.2 is not available right now . (pytorch.org)
So I have install 11.1 version .
(using nvidia-smi)
Cuda version
But it show false.
torch.cuda.is_available() >>> False

I have tried both 10.2 and 11.1 version.
As far as I know, I do not need to install cuda toolkit for pytorch

Output:
You should not install package to your base environment. Create a separate environment with necessary tools.
Example: create env called dlearn with Python v3.7 and torch packages
conda create -n dlearn python=3.7 pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

Activate and use your dlearn environment
conda activate dlearn
python -c "import torch;print(torch.cuda.is_available())"
# this should echo True if all is well

At the moment the supported  cudatoolkit is 11.1 which works fine with 11.2 driver. They will update it sooner or later. You can build from GitHub PyTorch to get the latest if you want. The steps are more complex than above.


Instruction: How to save training weight checkpoint of model and continue training from last point in PyTorch?

Input:
I'm trying to save checkpoint weights of the trained model after a certain number of epochs and continue to train from that last checkpoint to another number of epochs using PyTorch
To achieve this I've written a script like below
To train the model:
def create_model():
  # load model from package
  model = smp.Unet(
      encoder_name="resnet152",        # choose encoder, e.g. mobilenet_v2 or efficientnet-b7
      encoder_weights='imagenet',           # use `imagenet` pre-trained weights for encoder initialization
      in_channels=3,                  # model input channels (1 for gray-scale images, 3 for RGB, etc.)
      classes=2,                      # model output channels (number of classes in your dataset)
  )
  return model

model = create_model()
model.to(device)
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
epochs = 5

for epoch in range(epochs):
    print('Epoch: [{}/{}]'.format(epoch+1, epochs))

    # train set
    pbar = tqdm(train_loader)
    model.train()
    iou_logger = iouTracker()
    for batch in pbar:
        # load image and mask into device memory
        image = batch['image'].to(device)
        mask = batch['mask'].to(device)

        # pass images into model
        pred = model(image)
        # pred = checkpoint['model_state_dict']

        # get loss
        loss = criteria(pred, mask)

        # update the model
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # compute and display progress
        iou_logger.update(pred, mask)
        mIoU = iou_logger.get_mean()
        pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

    # development set
    pbar = tqdm(development_loader)

    model.eval()
    iou_logger = iouTracker()
    with torch.no_grad():
        for batch in pbar:
            # load image and mask into device memory
            image = batch['image'].to(device)
            mask = batch['mask'].to(device)

            # pass images into model
            pred = model(image)

            # get loss
            loss = criteria(pred, mask)

            # compute and display progress
            iou_logger.update(pred, mask)
            mIoU = iou_logger.get_mean()
            pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

# save model
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,}, '/content/drive/MyDrive/checkpoint.pt')

from this, I can save the model checkpoint file as checkpoint.pt for 5 epochs
To continue the training using the saved checkpoint weight file for another I wrote below script:
epochs = 5
for epoch in range(epochs):
    print('Epoch: [{}/{}]'.format(epoch+1, epochs))

    # train set
    pbar = tqdm(train_loader)

    checkpoint = torch.load( '/content/drive/MyDrive/checkpoint.pt')
    print(checkpoint)

    model.load_state_dict(checkpoint['model_state_dict'])
    model.to(device)

    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    epoch = checkpoint['epoch']
    loss = checkpoint['loss']
    model.train()
    iou_logger = iouTracker()
    for batch in pbar:
        # load image and mask into device memory
        image = batch['image'].to(device)
        mask = batch['mask'].to(device)

        # pass images into model
        pred = model(image)
        # pred = checkpoint['model_state_dict']

        # get loss
        loss = criteria(pred, mask)

        # update the model
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # compute and display progress
        iou_logger.update(pred, mask)
        mIoU = iou_logger.get_mean()
        pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

    # development set
    pbar = tqdm(development_loader)

    model.eval()
    iou_logger = iouTracker()
    with torch.no_grad():
        for batch in pbar:
            # load image and mask into device memory
            image = batch['image'].to(device)
            mask = batch['mask'].to(device)

            # pass images into model
            pred = model(image)

            # get loss
            loss = criteria(pred, mask)

            # compute and display progress
            iou_logger.update(pred, mask)
            mIoU = iou_logger.get_mean()
            pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

# save model
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,}, 'checkpoint.pt')

This throws error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-54f48c10531a> in <module>()

---> 14     model.load_state_dict(checkpoint['model_state_dict'])

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1222         if len(error_msgs) > 0:
   1223             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1224                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1225         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1226

RuntimeError: Error(s) in loading state_dict for DataParallel:
    Missing key(s) in state_dict: "module.encoder.conv1.weight", "module.encoder.bn1.weight", "module.encoder.bn1.bias", "module.encoder.bn1.running_mean", "module.encoder.bn1.running_var", "module.encoder.layer1.0.conv1.weight", "module.encoder.layer1.0.bn1.weight", "module.encoder.layer1.0.bn1.bias", "module.encoder.layer1.0.bn1.running_mean", "module.encoder.layer1.0.bn1.running_var", "module.encoder.layer1.0.conv2.weight", "module.encoder.layer1.0.bn2.weight", "module.encoder.layer1.0.bn2.bias", "module.encoder.layer1.0.bn2.running_mean", "module.encoder.layer1.0.bn2.running_var", "module.encoder.layer1.0.conv3.weight", "module.encoder.layer1.0.bn3.weight", "module.encoder.layer1.0.bn3.bias", "module.encoder.layer1.0.bn3.running_mean", "module.encoder.layer1.0.bn3.running_var", "module.encoder.layer1.0.downsample.0.weight", "module.encoder.layer1.0.downsample.1.weight", "module.encoder.layer1.0.downsample.1.bias", "module.encoder.layer1.0.downsample.1.running_mean", "module.encoder.layer1.0.downsample.1.running_var", "module.encoder.layer1.1.conv1.weight", "module.encoder.layer1.1.bn1.weight", "module.encoder.layer1.1.bn1.bias", "module.encoder.layer1.1.bn1.running_mean", "module.encoder.layer1.1.bn1.running_var", "module.encoder.layer1.1.conv2.weight", "module.encoder.layer1.1.bn2.weight", "module.encoder.layer1.1.bn2.bias", "module.encoder.layer1.1.bn2.running_mean", "module.encoder.layer1.1.bn2.running_var", "module.encoder.layer1.1.conv3.weight", "module.encoder.layer...
    Unexpected key(s) in state_dict: "encoder.conv1.weight", "encoder.bn1.weight", "encoder.bn1.bias", "encoder.bn1.running_mean", "encoder.bn1.running_var", "encoder.bn1.num_batches_tracked", "encoder.layer1.0.conv1.weight", "encoder.layer1.0.bn1.weight", "encoder.layer1.0.bn1.bias", "encoder.layer1.0.bn1.running_mean", "encoder.layer1.0.bn1.running_var", "encoder.layer1.0.bn1.num_batches_tracked", "encoder.layer1.0.conv2.weight", "encoder.layer1.0.bn2.weight", "encoder.layer1.0.bn2.bias", "encoder.layer1.0.bn2.running_mean", "encoder.layer1.0.bn2.running_var", "encoder.layer1.0.bn2.num_batches_tracked", "encoder.layer1.1.conv1.weight", "encoder.layer1.1.bn1.weight", "encoder.layer1.1.bn1.bias", "encoder.layer1.1.bn1.running_mean", "encoder.layer1.1.bn1.running_var", "encoder.layer1.1.bn1.num_batches_tracked", "encoder.layer1.1.conv2.weight", "encoder.layer1.1.bn2.weight", "encoder.layer1.1.bn2.bias", "encoder.layer1.1.bn2.running_mean", "encoder.layer1.1.bn2.running_var", "encoder.layer1.1.bn2.num_batches_tracked", "encoder.layer1.2.conv1.weight", "encoder.layer1.2.bn1.weight", "encoder.layer1.2.bn1.bias", "encoder.layer1.2.bn1.running_mean", "encoder.layer1.2.bn1.running_var", "encoder.layer1.2.bn1.num_batches_tracked", "encoder.layer1.2.conv2.weight", "encoder.layer1.2.bn2.weight", "encoder.layer1.2.bn2.bias", "encoder.layer1.2.bn2.running_mean", "encoder.layer1.2.bn2.running_var", "encoder.layer1.2.bn2.num_batches_tracked", "encoder.layer2.0.conv1.weight", "encoder.layer...

What am I doing wrong? How can I fix this? Any help on this will be helpful.

Output:
this line:
model.load_state_dict(checkpoint['model_state_dict'])

should be like this:
model.load_state_dict(checkpoint)


Instruction: How to update tensors matching dimensionwise vectors

Input:
Let there be two 2D tensors, A (m × c) and B (n × c). Each row vector which belongs to B also belongs to A i.e. . Additionally, row vectors in A are not unique i.e. A may have duplicate rows. However, row vectors in B are unique.
There another pair of tensors P (m × f) and Q (n × f). I am trying to do the following
for i in range(B.shape[0]):
    rv = B[i, :]
    fv = Q[i, :]
    # P[<row indexes of A matching rv>, :] = fv

How to do this correctly?
Is it possible to get rid of the for loop?

Output:
You can use the following mask:
for i in range(B.shape[0]):
    rv = B[i]
    fv = Q[i]
    mask = torch.where((A == rv).all(dim=1))[0]
    P[mask] = fv