Instruction: detectron2 - CUDA is not available

Input:
I am trying out detectron2 and want to train the sample model.
When running the following code I get (<class 'RuntimeError'>, RuntimeError('No CUDA GPUs are available'), <traceback object at 0x7f42b094ebc0>). Find below the code:
import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

# import some common libraries
import matplotlib.pyplot as plt
import numpy as np
import cv2

# import some common detectron2 utilities
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog

from detectron2.data.datasets import register_coco_instances
import random
from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
import os

# To verify the data loading is correct, let's visualize the annotations of randomly selected samples in the training set:
register_coco_instances("fruits_nuts", {}, "../data/trainval.json", "../data/images")

fruits_nuts_metadata = MetadataCatalog.get("fruits_nuts")
dataset_dicts = DatasetCatalog.get("fruits_nuts")
'''
for d in random.sample(dataset_dicts, 3):
    img = cv2.imread(d["file_name"])
    visualizer = Visualizer(img[:, :, ::-1], metadata=fruits_nuts_metadata, scale=0.5)
    vis = visualizer.draw_dataset_dict(d)
    cv2.imshow('new', vis.get_image()[:, :, ::-1])
    cv2.waitKey(0)
'''
# train model
cfg = get_cfg()
cfg.merge_from_file("../detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.DATASETS.TRAIN = ("fruits_nuts",)
cfg.DATASETS.TEST = ()   # no metrics implemented for this dataset
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"  # initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.02
cfg.SOLVER.MAX_ITER = 300    # 300 iterations seems good enough, but you can certainly train longer
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128   # faster, and good enough for this toy dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 3  # 3 classes (data, fig, hazelnut)

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

I ran the script collect_env.py from torch:
/home/project/.venv/bin/python /home/project/src/collect_env.py
Collecting environment information...
PyTorch version: 1.10.2+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0] (64-bit runtime)
Python platform: Linux-5.13.0-27-generic-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: 10.1.243
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.22.1
[pip3] torch==1.10.2
[pip3] torchvision==0.11.3
[conda] Could not collect

Process finished with exit code 0

I am having on the system a RTX3080 graphic card. However, it seems to me that its not found.
Any suggestions why?
Is there a way to run the training without CUDA?
I appreciate your replies!

Output:
I'm not sure if this works for you. But let's see from a Windows user perspective.
I'm using Detectron2 on Windows 10 with RTX3060 Laptop GPU CUDA enabled.
The first thing you should check is the CUDA. You can check by using the command:
nvcc -V
It should be shown this message:
C:\Users\User>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:41:42_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

And to check if your Pytorch is installed with CUDA enabled, use this command (reference from their website):
import torch
torch.cuda.is_available()

As on your system info shared in this question, you haven't installed CUDA on your system. And your system doesn't detect any GPU (driver) available on your system.
As far as I know, they recommended installing Pytorch CUDA to run Detectron2 by (Nvidia) GPU.
(you can check on Pytorch website and Detectron2 GitHub repo for more details).
Or, you can use this option:
Add this line of code to your python program (as reference of this issues#300):
cfg.MODEL.DEVICE = "cpu"

I hope it helps. Cheers.


Instruction: LSTM always predicts the same class

Input:
I’m trying to solve an nlp classification problem with a LSTM. The code for the model is defined here:
class LSTM(nn.Module):

  def __init__(self, hidden_size, embedding_size=66 ):

      super().__init__()
      self.lstm = nn.LSTM(embedding_size, hidden_size, batch_first = True, bidirectional = True)
      self.fc = nn.Linear(2*hidden_size,2)

  def forward(self, input_seq):

      output, (hidden_state, cell_state) = self.lstm(input_seq)

      hidden_state = torch.cat((hidden_state[-1,:], hidden_state[-2,:]), -1)

      logits = self.fc(hidden_state)

      return nn.LogSoftmax(dim=1)(logits)

And the function I’m using to train this model is here:
def train_loop(dataloader, model, loss_fn, optimizer):

    loss_fn = loss_fn
    size = len(dataloader.dataset)
    model.train()
    zeros = 0
    for batch, (X, y) in enumerate(dataloader):

        # Transform string into tensor
        tensor = torch.zeros(1,len(X[0]),66)
        for i in range(len(X[0])):
            tensor[0][i][ctoi[X[0][i]]] = 1

        pred = model(tensor)

        target = torch.zeros(2, dtype=torch.long)
        target[y] = 1

        if batch % 100 == 0:
            print(pred.squeeze(), target)
        loss = loss_fn(pred.squeeze(), target)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if pred.squeeze().argmax() == 0:
            zeros += 1

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

    print(f'In trainning predicted {zeros} zeroes out of {size} samples')

The X’s are still strings, that’s why I need to convert them to tensors before running it through the model. The y’s are either a 0 or 1 (since its a binary classification problem), that I need to convert to a tensor of shape (2,) to run through the loss function.
For some reason I keep getting the same class predicted for every input. The classes are not even that unbalanced (~45% to 55%), and I’ve tried changing the weights of the classes in the loss function with no improvements, it either converges to predicting always a 0 or always a 1. Most of the time it it converges to predicting always a 0, which makes even less sense because what happens usually is that the class 0 has less samples than class 1.

Output:
Since you're training a binary classification model, your output dim should be 1 (corresponding to a single probability P(y|x)). This means that the y you're retrieving from your dataloader should be the y used in your loss function (assuming a cross-entropy loss). The predicted class is therefore y_hat = round(pred) (i.e., is the prediction >= 0.5).
As a point of clarity, it would be much easier to follow your logic if the one-hot encoding happened within your dataset (either in __getitem__ or __iter__). It's also worth noting that you don't use embeddings, so the code of your classifier is a bit misleading.


Instruction: Using older torch version in conda environment not working

Input:
Can anyone please help me?
I am trying to run a .py script for which I need an older pytorch version, because a function I am using is deprecated in later torch versions. But I seem not to be able to install it correctly.
I installed torch into my virtual environment using
conda create -n my_env python=3.6.2
source activate my_env
conda install pytorch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0 cudatoolkit=10.2 -c pytorch
Then I have a python file (myfile.py) that I start using a shell file (start.sh). The files are on a SLURM-cluster, so I start start.sh with sbatch start.sh.
start.sh
source activate my_env
srun --unbuffered python myfile.py
myfile.py
import torch
print(torch.__version__)
The print command from myfile.py still returns "1.8.0", but using conda list in my environment shows "1.7.0" for pytorch version.
Even when I type python -c "import torch; print(torch.__version__)" directly into terminal, it will return "1.8.0" (rather than "1.7.0" from conda list)
Am I doing something very obvious wrong possibly? Did I install in a wrong way, or is somehow my environment not properly loaded in the python file?
Best regards and thanks a lot in advance

Output:
It turned out that installing the environment as described added a link to another python installation to my PYTHONPATH (a link to /.local/python) and that directory was added to PYTHONPATH in a higher order than the python used in my environment (/anaconda/env/my_env/python/...) .
Therefore, the local version of python was used instead.
I could not delete it from PYTHONPATH either, but changing the directory name to /.local/_python did the trick.
It's not pretty, but it works.
Thanks everyone for the contributions!


Instruction: Pytorch/YOLOv5 - Compare detected Object if it's the same

Input:
I am trying to use Pytorch and YOLOv5 to detect objects in multiple images and count them. My problem now is that if I have for example a frame rate of 15fps, the same objects can be recognized in the image, but they were only recognized for example a little bit in the front of the image (other coordinates) or the Objects have the same coordinates as before. Currently, only how many objects are detected in an image is counted. How can I exclude these objects or compare whether the objects have already been detected?
My Code so far:
counts = {"cars" : 0 , "trucks" : 0}
class_mappings = {2.0: "cars", 7.0: "trucks"}

def predict():
    img = Image.open("test.jpeg")
    result = model(img)
    labels = dict(Counter(result[:, -1].tolist()))
    for k, v in class_mappings.items():
        counts[v] += labels.get(k, 0)

This Code above is extracting the labels of detected objects from the Tensor and count them in a counter variable.

Output:
Most of the sorting algorithms/models will work out for you like a charm.
i.e. what you need is to track each box step by step after inferencing on each frame and assigning id/count to them based on some distance function to determine object's id after it has moved.
It's commonly referenced as MOT (Multiple Object Tracking).
You can meet two versions of MOT: statistical approach & DL + statistical on top.
DL version is more useful if you're working on certain environments with lot of noise ofc. But it comes with downside - you've to run feature extractor in real-time.
Although, I've worked with statistical approaches (based on Kalman Filter) in a very demanding production setting and after some tweaking it worked unbelievably well on very dense MOT task.
You can try out: https://github.com/wmuron/motpy
It'll be easy to integrate it to YoloV5.


Instruction: Transfer OpenGL image on GPU from C++ to Python for deep learning

Input:
I built a simulator in C++ with a pybind11 interface to run deep learning in Python using PyTorch. At each time step, I draw certain things from the simulator's scene using the SFML library (wrapper around openGL). I draw that on a texture, then get the pixels from that texture as follows:
glBindTexture(GL_TEXTURE_2D, imageTexture.getTexture().getNativeHandle());
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE, img.data());

I then move the pixels vector img from C++ to Python using the pybind11 interface. Problem is that this GPU-to-CPU operation is very slow. Since the vector in Python is then transferred back to the GPU for fast deep learning (CNN) operations, I was wondering how I could avoid that step.
My best guess so far is that I should at each step bind the texture in C++ (as in the code above), then right after that in the Python get the bound texture using CUDA, while keeping it on the GPU. However I couldn't figure out how to do that, I don't know much about GPUs and how CUDA/OpenGL work. Pointers to the right direction would be very appreciated!

Output:
You should use PBO(Pixel Buffer Object) for this operation.
Data transferring operation is very fast using PBO
https://www.khronos.org/opengl/wiki/Pixel_Buffer_Object
GLuint w_pbo[2];

 // Create pbo objects and than

 // Do your drawings.

int w_readIndex = 0;
int w_writeIndex = 1;
glReadBuffer(GL_COLOR_ATTACHMENT0);
w_writeIndex = (w_writeIndex + 1) % 2;
w_readIndex = (w_readIndex + 1) % 2;
glBindBuffer(GL_PIXEL_PACK_BUFFER, w_pbo[w_writeIndex]);
// copy from framebuffer to PBO asynchronously. it will be ready in the NEXT frame
glReadPixels(0, 0, SCR_WIDTH, SCR_HEIGHT, GL_RGBA, GL_UNSIGNED_BYTE, nullptr);
// now read other PBO which should be already in CPU memory
glBindBuffer(GL_PIXEL_PACK_BUFFER, w_pbo[w_readIndex]);
unsigned char* downsampleData = (unsigned char*)glMapBuffer(GL_PIXEL_PACK_BUFFER, GL_READ_ONLY);

Now you can use unsigned char* downsampleData to build the texture memory


Instruction: AssertionError in torch_geometric.nn.GATConv

Input:
I am trying to use graph attention network (GAT) module in torch_geometric but keep running into AssertionError: Static graphs not supported in 'GATConv' with the following code.
class GraphConv_sum(nn.Module):
    def __init__(self, in_ch, out_ch, num_layers, block, adj):
        super(GraphConv_sum, self).__init__()
        adj_coo = coo_matrix(adj) # convert the adjacency matrix to COO format for Pytorch Geometric
        self.edge_index = torch.tensor([adj_coo.row, adj_coo.col], dtype=torch.long)
        self.g_conv = nn.ModuleList()

        self.act = nn.LeakyReLU()

        for n in range(num_layers):
            if n == 0:
                self.g_conv.append(block(in_ch, 16))
            elif n > 0 and n < num_layers - 1:
                self.g_conv.append(block(16, 16))
            else:
                self.g_conv.append(block(16, out_ch))

    def forward(self, x):
        for layer in self.g_conv:
            x = layer(x=x, edge_index=self.edge_index)
            x = self.act(x)
            print(x.shape)
        return x[:, 0, :]

When I replace block with GATConv followed by a standard training loop, this error happens (other conv layers such as GCNConv or SAGEConv didn't have any problems). I checked the documentation and made sure the input shape was correct (same for other conv layers).
In the source code, there is this assert x.dim() == 2, "Static graphs not supported in 'GATConv'" part in the forward method but apparently the batch dimension will come into play in the forward pass and x.dim() would be 3. The input shape with batch dimension is [1024, 6, 200]. However, if I manually change the assert condition to x.dim() == 3 the same error will still be raised as if the condition is not satisfied. I only have a high-level grasp of GAT so there might be something I am missing. Anyways, I have a few questions from this

Are there any implementation errors possible from my side that caused this error?
What is this assertion condition for? What is the static graph in this case?

I would appreciate any insights and help!! Thanks!

Output: It turns out due to the attention weight calculation, GATConv doesn't support multiple feature matrices and single edge_index. More info: https://github.com/pyg-team/pytorch_geometric/issues/2844


Instruction: Applying transformation to data set in pytorch and add them to the data

Input:
I want to load fashion-mnist (or any other data set) using
torchvision.datasets.FashionMNIST(data_dir, train=True, download=True)
and then apply some image transformation such as cropping or adding noise, etc and finally add the transformed data to the original data set.
The only way I found is torchvision.transform but it changes the original dataset and do not augment the data set.
How can I augment the data set?

Output:
As @Ivan already pointed out in the comments, when accessing an image, PyTorch always loads its original dataset version. Then, transform applies online your transformation of choice to the data.
In general, setting a transform to augment the data without touching the original dataset is the common practice when training neural models.
That said, if you need to mix an augmented dataset with the original one you can, for example, stack two datasets with torch.utils.data.ConcatDataset, as follows:
dset1 = torchvision.datasets.FashionMNIST(data_dir, train=True, download=True)
dset2 = torchvision.datasets.FashionMNIST(data_dir, train=True, transform=my_transform)

dset = torch.utils.data.ConcatDataset([dset1, dset2])

Have a look at this page for more alternatives.
Finally, if you need, you can also save your dataset for future use (thus freezing the random transform applied to your data):
for idx, (img, target) in enumerate(dset):
    torch.save(img, f"mydset/{fname}_img_{idx}.pt")
    torch.save(target, f"mydset/{fname}_tgt_{idx}.pt")


Instruction: How to change the threshold of a prediction of multi-label classification using FASTAI library

Input:
I have a multi-label dataset that I'm using to train my model using fast-ai library for Python, using as metrics an accuracy function such as:
def accuracy_multi1(inp, targ, thresh=0.5, sigmoid=True):
    "Compute accuracy when 'inp' and 'targ' are the same size"
    if sigmoid: inp=inp.sigmoid()
    return ((inp>thresh) == targ.bool()).float().mean()

And my learner is like:
learn = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi1,thresh=0.1))
learn.fine_tune(2,base_lr=3e-2,freeze_epochs=2)

After training my model, I want to predict an image considering the threshold I used a argument, but the method learn.predict('img.jpg') only considers the default thres=0.5. In the following example, my predidction should return True for  'red, 'shirt' and 'shoes', as their probabilities are above 0.1 (but shoes is below 0.5, so it is not considered as True):
def printclasses(prediction,classes):
    print('Prediction:',prediction[0])
    for i in range(len(classes)):
        print(classes[i],':',bool(prediction[1][i]),'|',float(prediction[2][i]))

printclasses(learn.predict('rose.jpg'),dls.vocab)

Output:
Prediction: ['red', 'shirt']
black : False | 0.007274294272065163
blue : False | 0.0019288889598101377
brown : False | 0.005750810727477074
dress : False | 0.0028723080176860094
green : False | 0.005523672327399254
hoodie : False | 0.1325301229953766
pants : False | 0.009496113285422325
pink : False | 0.0037188702262938023
red : True | 0.9839697480201721
shirt : True | 0.5762518644332886
shoes : False | 0.2752271890640259
shorts : False | 0.0020902694668620825
silver : False | 0.0009014935349114239
skirt : False | 0.0030087409541010857
suit : False | 0.0006510693347081542
white : False | 0.001247694599442184
yellow : False | 0.0015280473744496703

Is there a way to impose the threshold when I'm doing a prediction of a image I'm referencing? Something that would look like:
learn.predict('img.jpg',thresh=0.1)

Output:
I've encountered the same problem. I remain interested in a better solution, but since accuracy_mult only seems to provide user-friendly evaluation of the model during the training process (and is not involved in the prediction), I created a work-around for my data.
The basic idea is to take the tensor with the actual predictions (which is the third entry in the 3-tuple returned by the predict() function), apply the threshold & get the corresponding labels from the vocab.
def predict_labels(x, model, thresh=0.5):
  '''
  function to predict multi-labels in text (x)

  arguments:
  ----------
  x: the text to predict
  model: the trained learner
  thresh: thresh to indicate which labels should be included, fastai default is 0.5

  return:
  -------
  (str) predictions separated by blankspace
  '''

  # getting categories according to threshold
  preds = model.predict(x)[2] > thresh
  labels = model.dls.multi_categorize.vocab[preds]

  return ' '.join(labels)


Instruction: Shall I use grad.zero_() in PyTorch with or without gradient tracking?

Input:
I'm quite new to PyTorch, and I have a question about zeroing the gradients after an epoch. Suppose I have the following training loop:
for epoch in range(n_iters):
  y_hat = forward(X)
  l = loss(y, y_hat)
  with torch.no_grad():
    l.backward()
    w -= lr * w.grad

It is clear that in order not to have the gradients accumulated I need to zero the .grad attribute of w. However, I'm unsure about where to call w.grad.zero_(). I both found on internet tutorials where it was called in the no_grad() section and also where it was called out of it. So I tested them and they both worked fine for a simple linear regression.
Is any difference between the two? If there is, which is better to use?

Output:
In your snippet that doesn't really matter. The underscore in the name of zero_() means it is an inplace function, and since w.grad.requires_grad == False we know that there won't be any gradient computation with respect to w.grad happening anyway. The only important thing is that it happens before the loss.backward() call.
I would recommend though to use different names for your loss function and the actuall loss tensor it computes, otherwise you're overwriting one with the other.


Instruction: Why is a 1x1 pytorch convolution changing the data?

Input:
I am debugging an issue I have using torch::nn:Conv2d.  Here is a simple script which demonstrates the unexpected behavour
import torch

use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0" if use_cuda else "cpu")

t = torch.ones([1,1,3,3]).to(device)

print(t)

kernel_size=[1,1]
t2 = torch.nn.Conv2d(in_channels=1,out_channels=1,kernel_size=kernel_size, stride=kernel_size, padding=0).to(device)(t)

print(t2)

Running this results in:
tensor([[[[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]]]], device='cuda:0')
tensor([[[[-1.7190, -1.7190, -1.7190],
          [-1.7190, -1.7190, -1.7190],
          [-1.7190, -1.7190, -1.7190]]]], device='cuda:0',
       grad_fn=<AddBackward0>)

Why is t2 different to t1?  I would expect a 1x1 convolution to leave the input unchanged.

Output:
Thanks to Michael Szczesny's comment, I replace the conv2d with;
t2 = torch.nn.AvgPool2d(1, stride=1)(t)

And all is well:
tensor([[[[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]]]], device='cuda:0')
tensor([[[[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]]]], device='cuda:0')

Basically, I was using the wrong function.


Instruction: How to retrieve PyTorch tensor from queue in multiprocessing

Input:
I am simply trying to retrieve a tensor that I put into a queue in another process, but I get a 'Connection Refused' error whenever I do. Please point me to any documentation that may help or give me some suggestions please.
import torch
import torch.multiprocessing as mp

def test(q):
    x = torch.normal(mean=0.0, std=1.0, size=(2, 3))
    x.share_memory_()
    q.put(x)

if __name__ == "__main__":
    mp.set_start_method("spawn")

    q = mp.Queue()
    p = mp.Process(target=test, args=(q,))
    p.start()
    p.join()

    print(q.get())

Output:
You should use Manager() to get rid of this error. So working code example should look like below
import torch
import torch.multiprocessing as mp

def test(q):
    x = torch.normal(mean=0.0, std=1.0, size=(2, 3))
    x.share_memory_()
    q.put(x)

if __name__ == "__main__":
    #mp.set_start_method("spawn")
    manager = mp.Manager()
    q = manager.Queue()
    p = mp.Process(target=test, args=(q,))
    p.start()
    p.join()

    print(q.get())


Instruction: Correct way of freezing layers

Input:
I have a model M and I am cloning it M.clone()
Now, I want to freeze certain layers of M.clone(). When I set requires_grad=False, it results in this error:
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
How to freeze the layers of M.clone() in that case? I want to ensure that when I backpropagate using the loss computed on a batch using M.clone(), I compute the gradients of M
A small script:
model = ResNet()
optimizer = Adam(model.parameters())
cloned_model = model.clone()
for p in cloned_model.features.parameters():
     p.require_grad = False
error = loss(cloned_model(data), labels)
error.backward()
optimizer.step()

P.S. I am not sure if I can use .detach() as I don't want to break the graph. Do correct me if I am wrong.
Thank you!

Output:
You can use the in-place requires_grad_ function either on a nn.Module or on a torch.Tensor directly. Here you could do:
cloned_model = copy.deepcopy(model)
cloned_model.requires_grad_(False)

Where deepcopy is from copy.
You should copy your optimizer as well otherwise optimizer will be updating model, not cloned_model... resulting in no changes at all since you are not back-propagating on model.


Instruction: How to save and load only particular layers of a neural network with PyTorch?

Input:
Bare Problem Statement:
I have trained a Model A, that consists of a feature Extractor FE and a classification head ACH.
I want to train a model B, that uses A's feature extractor FE and retrains it's own classification head BCH.
So far it's easy. Now I don't want to save the entire model B since the FE part of it is already saved in the model A. I only want to dump the BCH, and during inference

Load model A - do it's prediction
Load B's classification head BCH.
Swap the classification head ACH with BCH
Run prediction using this swapped state.

Reading pyTorches documentation it only talks about saving entire models. How can I achieve this?
End of problem statement
More details on the motivation of the problem:
I have a dataset of images that I want to classify, these images have can have several classes given to them. For example the same image can have the class of "Land Vehicle" (supercategory) and a class of "Car" (category) or a "Truck". Another image might have the class "Aerial Vehicle" and it can be a "Helicopter" or a "Plane".
Since the images and therefore most of the features should be the same, I wish to train one classifier for the supercategories, then freeze it's feature-extractor, and sort of transfer learn the same model for the categories using the pretrained feature extractor.
Since the weights of the feature extracting backbone is the same, I only want to save the weights of the classification head of the categories model, and thus save some precious computational resources.

Output:
In general, it's something usual to only want an access to the backbone of a model in order to reuse it for others purposes. You have several ways to perform this. But mostly, having in mind that saving a model checkpoint and loading it later means saving weights and biases and being able to load them correctly to the corresponding layers, you first need to know, from your model, what part do you want to save.
When you get the state of a model, you will obtain a dictionary. The keys will be the layers names and the values will be the weights and the biases. Let's see an example with an efficientnet classifier on how to only save the backbone of a model. Basically, an efficientnet, as in your example, is a backbone and a fully connected layer as a head, if you only want the backbone, you want every single layers, except the head that you'll fine tune later.
import torch
import torch.nn as nn
from efficientnet_pytorch import EfficientNet

model = EfficientNet.from_name("efficientnet-b0")

print(model)

It will print the model layers and some features, basic stuff.
    EfficientNet(
  (_conv_stem): Conv2dStaticSamePadding(
    3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False
    (static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
  )
  (_bn0): BatchNorm2d(32, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
  (_blocks): ModuleList(
    (0): MBConvBlock(
      (_depthwise_conv): Conv2dStaticSamePadding(
        32, 32, kernel_size=(3, 3), stride=[1, 1], groups=32, bias=False
        (static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
      )
      (_bn1): BatchNorm2d(32, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
      (_se_reduce): Conv2dStaticSamePadding(
        32, 8, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      (_se_expand): Conv2dStaticSamePadding(
        8, 32, kernel_size=(1, 1), stride=(1, 1)
        (static_padding): Identity()
      )
      ...

Now what is interesting is the final layers of this model :
  ...
  (_bn1): BatchNorm2d(1280, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
  (_avg_pooling): AdaptiveAvgPool2d(output_size=1)
  (_dropout): Dropout(p=0.2, inplace=False)
  (_fc): Linear(in_features=1280, out_features=1000, bias=True)
  (_swish): MemoryEfficientSwish()

Let's say we want to reuse this model backbone, except _fcsince we would like to use the weights on another model having the same backbone but a different head, not pre-trained. In this example I'll take the same backbone and add 3 heads :
class ThreeHeadEfficientNet(torch.nn.Module):

def __init__(self,nbClasses1,nbClasses2,nbClasses3,model="efficientnet-b0",dropout_p=0.2):
    super(ThreeHeadEfficientNet, self).__init__()
    self.NBC1 = nbClasses1
    self.NBC2 = nbClasses2
    self.NBC3 = nbClasses3
    self.dropout_p = dropout_p

    self._dropout_layer = torch.nn.Dropout(p=self.dropout_p)
    self._head1 = torch.nn.Linear(1280,self.NBC1)
    self._head2 = torch.nn.Linear(1280,self.NBC2)
    self._head3 = torch.nn.Linear(1280,self.NBC3)

    self.model = EfficientNet.from_name(model,include_top=False) #you can notice here, I'm not loading the head, only the backbone

def forward(self,x):
    features = self.model(x)
    res = features.flatten(start_dim=1)
    res = self._dropout_layer(res)
    res1 = self._head1(res)
    res2 = self._head2(res)
    res3 = self._head3(res)

    return res1,res2,res3

You'll notice now, if you print this ThreeHeadsModel layers, the layers name have slightly changed from _conv_stem.weight to model._conv_stem.weight since the backbone is now stored in a attribute variable model. We'll thus have to process that otherwise the keys will mismatch, create a new state dictionary that matches the expected keys of this new model and containing the pretrained weights and biases :
pretrained_dict = model.state_dict() #pretrained model keys
model_dict = new_model.state_dict() #new model keys

processed_dict = {}

for k in model_dict.keys():
    decomposed_key = k.split(".")
    if("model" in decomposed_key):
        pretrained_key = ".".join(decomposed_key[1:])
        processed_dict[k] = pretrained_dict[pretrained_key] #Here we are creating the new state dict to make our new model able to load the pretrained parameters without the head.

new_model.load_state_dict(processed_dict, strict=False) #strict here is important since the heads layers are missing from the state, we don't want this line to raise an error but load the present keys anyway.

And finally, in new_model you should have your new model with a pretrained backbone and heads to fine tune.
Now you should be able to fix your issues :)
For more pytorch information, please also check the forum.


Instruction: Shall I use transformations on PIL Image or rather on PyTorch tensor?

Input:
My inputs are PIL images. Suppose I have the following transformation composition:
 transforms.Compose([
            transforms.RandomResizedCrop(size=224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor()])

As most of the transforms in PyTorch can work on both PIL images and tensors, I wonder in which order I should use them. Shall I first use all the transformations possible to use on PIL images and then transform to tensor, or shall I first transform to tensor and then apply the other transformations on tensors? Is the one more effective than the other?

Output: There's no real advantage in general to changing the order. However, there can be advantages to moving the ToTensor out of the transforms chain. Specifically, you cannot JIT transformations operating on PIL images which may have optimization impact. For this reason, it may be better to convert PIL images to tensors in your data loading code and then transform as needed. I refer you to the documentation to read more about this.


Instruction: How to I specify model.learn() to end within a certain episodes of stable baselines 3?

Input:
I know specifying that total_timesteps= is a require parameter, but how to I end model.learn() within a certain episodes? Forgive me for I'm still new to stables_baselines3 and pytorch still not how to implement it in code.
import gym
import numpy as np
from stable_baselines3 import DDPG
from stable_baselines3.common.noise import NormalActionNoise

env = gym.make('NeuralTraffic-v1')

n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))
model = DDPG("MlpPolicy", env, action_noise=action_noise, verbose=1)
model.learn(total_timesteps=60, log_interval=1)
model.save("ddpg")
env = model.get_env()

I wanted to ended the episode on 60 instead my rollout was:
----------------------------------
| rollout/           |           |
|    ep_len_mean     | 94        |
|    ep_rew_mean     | -2.36e+04 |
| time/              |           |
|    episodes        | 1         |
|    fps             | 0         |
|    time_elapsed    | 452       |
|    total_timesteps | 94        |
----------------------------------

I don't understand why is it only 1 episode? I'd like to learn how to implement to restrict learning to specified episodes.

Output: Generic Box-2D and classic control environments have 1000 timesteps within one episode but this is not constant as the agent can do some weird thing in the beginning and the environment can reset itself (resulting in uneven timesteps per episode). So it's the norm to keep a specific timestep in mind while benchmarking (1e6 in most research papers in model-free RL) on contrary to specifying a certain number of episodes. As you can see in SB3 Docs DDPG.learn method that they don't provide a specific argument to set the number of episodes and it is actually best to keep a specific number of timesteps in mind. I see that you have written 60 in place of total_timesteps. It's way too little to train an RL agent. Try keeping something like 1e5 or 1e6 and you might see good results. Good Luck!


Instruction: No module named ‘torchvision.models.utils‘

Input:
When I use the environment of pytorch=1.10.0, torchvision=0.11.1 to run the code, I run to the statement from torchvision.models.utils import load_state_dict_from_url. The following error will appear when:
>>> from torchvision.models.utils import load_state_dict_from_url
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torchvision.models.utils'

Output:
After consulting torchvision's code repository, there is a solution:
Note that this syntax is only for higher versions of PyTorch.
The original code from .utils import load_state_dict_from_url is not applicable.
you connot import load_state_dict_from_url from .utils.
change .utils to torch.hub can fix the problem.
from torch.hub import load_state_dict_from_url

This worked for me.


Instruction: different method of running pytorch on gpu

Input:
See the code block below (the source of the code can be found here, also you don't need to read the whole block, I will explain and highlight the important part)
def train(data_loader, model, optimizer, scheduler, total_epochs, save_interval, save_folder, sets):
    # settings
    batches_per_epoch = len(data_loader) #
    log.info('{} epochs in total, {} batches per epoch'.format(total_epochs, batches_per_epoch))
    loss_seg = nn.CrossEntropyLoss(ignore_index=-1)

    print("Current setting is:")
    print(sets)
    print("\n\n")
    if not sets.no_cuda:
        loss_seg = loss_seg.cuda()

    # change model in training mode, enable batch normalization etc
    model.train()

    # record train time
    train_time_sp = time.time()

    # loop to train the model
    for epoch in range(total_epochs):
        log.info('Start epoch {}'.format(epoch))

        scheduler.step()
        log.info('lr = {}'.format(scheduler.get_lr()))

        for batch_id, batch_data in enumerate(data_loader):
            # getting data batch
            batch_id_sp = epoch * batches_per_epoch
            volumes, label_masks = batch_data

            if not sets.no_cuda:
                volumes = volumes.cuda()

            optimizer.zero_grad()
            out_masks = model(volumes)
            # resize label
            [n, _, d, h, w] = out_masks.shape
            new_label_masks = np.zeros([n, d, h, w])
            for label_id in range(n):
                label_mask = label_masks[label_id]
                [ori_c, ori_d, ori_h, ori_w] = label_mask.shape
                label_mask = np.reshape(label_mask, [ori_d, ori_h, ori_w])
                scale = [d*1.0/ori_d, h*1.0/ori_h, w*1.0/ori_w]
                label_mask = ndimage.interpolation.zoom(label_mask, scale, order=0)
                new_label_masks[label_id] = label_mask

            new_label_masks = torch.tensor(new_label_masks).to(torch.int64)
            if not sets.no_cuda:
                new_label_masks = new_label_masks.cuda()

            # calculating loss
            loss_value_seg = loss_seg(out_masks, new_label_masks)
            loss = loss_value_seg
            loss.backward()
            optimizer.step()

            avg_batch_time = (time.time() - train_time_sp) / (1 + batch_id_sp)
            log.info(
                    'Batch: {}-{} ({}), loss = {:.3f}, loss_seg = {:.3f}, avg_batch_time = {:.3f}'\
                    .format(epoch, batch_id, batch_id_sp, loss.item(), loss_value_seg.item(), avg_batch_time))

            if not sets.ci_test:
                # save model
                if batch_id == 0 and batch_id_sp != 0 and batch_id_sp % save_interval == 0:
                #if batch_id_sp != 0 and batch_id_sp % save_interval == 0:
                    model_save_path = '{}_epoch_{}_batch_{}.pth.tar'.format(save_folder, epoch, batch_id)
                    model_save_dir = os.path.dirname(model_save_path)
                    if not os.path.exists(model_save_dir):
                        os.makedirs(model_save_dir)

                    log.info('Save checkpoints: epoch = {}, batch_id = {}'.format(epoch, batch_id))
                    torch.save({
                                'ecpoch': epoch,
                                'batch_id': batch_id,
                                'state_dict': model.state_dict(),
                                'optimizer': optimizer.state_dict()},
                                model_save_path)

    print('Finished training')
    if sets.ci_test:
        exit()

This is a customised training function, the author achieve GPU running by using code loss_seg = loss_seg.cuda() where loss_seg is a instance of torch.optim.SGD. This part confused me, because according to the official documentation in pytorch, I only need to move my model and my input data to gpu is enough. I want to know why the author who write the code below also move the optimizer to the gpu and more details about running pytorch on gpu.

Output:
In general, in order to harness the full power of GPUs, every stateful Module should be sent to a cuda device before the forward step. A stateful Module has an internal state, e.g. Parameter (weights).
This is not usually the case of Loss, which, in general, applies just a functional that has been already implemented for cuda devices.
In conclusion, if the loss is stateful, then it makes sense to send it to a cuda device, otherwise, it is not necessary.
Have also a look at this question in the PyTorch forum.


Instruction: indices in MaxPool2d in pytorch

Input:
I am studying the documentation at https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html.
In the parameters section, it states

return_indices – if True, will return the max indices along with the outputs. Useful for torch.nn.MaxUnpool2d later

Could someone explain to me what max indices mean here? I believe it is the indices corresponding to the maximal value. If the maximal value is unique, does that mean only 1 index will be returned?

Output:
I assume you already know how max pooling works.
Then, let's print some results to get more insights.
import torch
import torch.nn as nn

pool = nn.MaxPool2d(kernel_size=2, return_indices=True)
input = torch.zeros(1, 1, 4, 4)
input[..., 0, 1] = input[..., 1, 3] = input[..., 2, 2] = input[..., 3, 0] = 1.
print(input)

output
tensor([[[[0., 1., 0., 0.],
          [0., 0., 0., 1.],
          [0., 0., 1., 0.],
          [1., 0., 0., 0.]]]])

output, indices = pool(input)
print(output)

output
tensor([[[[1., 1.],
          [1., 1.]]]])

print(indices)

output
tensor([[[[ 1,  7],
          [12, 10]]]])

If you stretch the input tensor and make it 1d, you can see that indices contains the positions of each 1 value (the maximum for each window of MaxPool2d). As written in the documentation of torch.nn.MaxPool2d, indices is required for the torch.nn.MaxUnpool2d module:

MaxUnpool2d takes in as input the output of MaxPool2d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero.


Instruction: how to create empty parameter in in pytorch tensor variable

Input:
from torch import FloatTensor

def new_parameter(*size): #1024
    out = torch.nn.Parameter(FloatTensor(*size), requires_grad=True)
    torch.nn.init.xavier_normal_(out)
    return out

at = new_parameter(1024, 1)
output is
Parameter containing:
tensor([[ 0.0203],
        [-0.0043],
        [-0.0386],
        ...,
        [-0.0084],
        [-0.0289],
        [-0.0188]], requires_grad=True)

similarway we can create
bt=torch.randn((1024,1),requires_grad=True)

output also same
tensor([[-1.5478],
        [ 1.5060],
        [ 0.1580],
        ...,
        [ 0.9754],
        [ 0.1699],
        [ 0.2062]], requires_grad=True)

are there any differences in a tensor variable the above two ways? please explain the above code in simply

Output:
The first method will initialize a random float tensor, then wrap it with nn.Parameter. Which is generally used to register than tensor as a parameter to a nn.Module (not seen here). A utility function nn.init.xavier_normal_ is then applied on that parameter to initialize its values.
The second method only initializes a random float tensor.


Instruction: Pytorch CNN script training, but not getting results

Input:
I’m just getting started with pytorch. I am trying to do a simple binary classification project with the cats and dogs dataset. After much fumbling around, I was able to get the model to train, but I’m not getting the expected results.
First, the loss starts out way too low. To me, that seems to indicate I’m not measuring loss correctly.
Second, the model just predicts everything as 0.
I’m sure there are many mistakes here, but I would appreciate it if someone could take a look and let me know what I’m doing wrong. Thank you!
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import torch.nn as nn
import torch.nn.functional as F
from torchvision.io import read_image
from torch.utils.data import Dataset, DataLoader
from torchvision.utils import make_grid
from torchvision.utils import save_image
from sklearn.model_selection import train_test_split
import os
import numpy as np
from sklearn import preprocessing
import glob
import cv2

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)

IMAGE_SIZE = 64
DATA_DIR = "C:\\Users\\user\\source\\repos\\pytorch-youtube\\data\\catsdogs\\PetImages\\"
LABELS = ('cat', 'dog')

# custom dataset class
# expects the root folder to have sub folders with class names
# and pictures of classes inside folder
class CustomImageDataset(Dataset):
    def __init__(self):
        self.imgs_path = DATA_DIR
        file_list = glob.glob(self.imgs_path + "*")
        self.data = []
        for class_path in file_list:
            class_name = class_path.split("\\")[-1]
            for img_path in glob.glob(class_path + "\\*.jpg"):
                self.data.append([img_path, class_name])
        self.class_map = {"Dog": 0, "Cat": 1}
        self.img_dim = (IMAGE_SIZE, IMAGE_SIZE)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        img_path, class_name = self.data[idx]
        # this is to handle corrupt images in the dataset
        # could probably be handled better
        try:
            img = cv2.imread(img_path)
            img = cv2.resize(img, self.img_dim)
        except:
            img_path, class_name = self.data[idx+1]
            img = cv2.imread(img_path)
            img = cv2.resize(img, self.img_dim)
        class_id = self.class_map[class_name]
        img_tensor = torch.from_numpy(img)
        img_tensor = img_tensor.permute(2, 0, 1) # not exactly sure what/why for this line
        class_id = torch.tensor([class_id])
        return img_tensor, class_id

# as is, we aren't using these
transform = transforms.Compose(
    [transforms.Resize((64, 64)),
    transforms.ConvertImageDtype(torch.float32),
    transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5)),

    ]
)

dataset = CustomImageDataset()
dataloader = DataLoader(dataset, batch_size=4, shuffle=True)

dataiter = iter(dataloader)
train_features, train_labels = dataiter.next()

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(2704, 128) # only way I got input size was by running code
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
# net.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.001)

for epoch in range(2):
    running_loss = 0.0
    for i, data in enumerate(dataloader, 0):

        inputs, labels = data

        # this is the fix for "expected scalar type Byte but found Float"
        # this seems to completely destroy the features in the image to just white
        inputs = inputs.float()
        optimizer.zero_grad()

        outputs = net(inputs)

        loss = criterion(outputs, torch.max(labels,1)[1])
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.10f}')
            running_loss = 0.0

print("finished")

# save the model
PATH = './custom_trained_model_dogs_cats.pth'
torch.save(net.state_dict(), PATH)

Output:
It seems I was passing in the wrong thing to my loss function. I changed this line
loss = criterion(outputs, torch.max(labels,1)[1])

to this
loss = criterion(outputs, torch.max(labels,1)[0])

and everything seems to be working. I'm able to correctly classify the cats and dogs.


Instruction: randomized data trasnformation in pytorch

Input:
I want to rotate all the images in my Dataset with a random degree between [0,180]. If I compose a transformation function and pass my images to this function in the __getitem__ function of my Dataset class. Does this mean:

every single image is randomly rotated?
images in each batch get rotated with an identical degree but this degree randomly changes across batches (calls)?

I would appreciate it if you could clarify this for me.

Output:
In mapped datasets, __getitem__ is used to select a single element from the dataset.
The way random transformations work in PyTorch/Torchvision is they apply a unique random transformation each time the transform is called. This means:

Every single image in your dataset is indeed randomly rotated but not by the same amount.

Additionally images in a batch get different transformations. In other words, elements in the batch won't share the same transformation parameters.

Here is a minimal example with a dummy dataset:
class D(Dataset):
    def __init__(self, n):
        super().__init__()
        self.n = n
        self.transforms = T.Lambda(lambda x: x*randint(0,10))

    def __len__(self):
        return self.n

    def __getitem__(self, index):
        x = self.transforms(index)
        return x

Here you can see the random transformer inter and intra batches:
>>> dl = DataLoader(D(10), batch_size=2)

>>> for i, x  in enumerate(dl):
...     print(f'batch {i}: elements {2*i} and {2*i+1} = {x.tolist()}')
batch 0: elements 0 and 1 = [0, 2]
batch 1: elements 2 and 3 = [14, 27]
batch 2: elements 4 and 5 = [32, 40]
batch 3: elements 6 and 7 = [60, 0]
batch 4: elements 8 and 9 = [80, 27]


Instruction: What is the impact of `pos_weight` argument in `BCEWithLogitsLoss`?

Input: According to the pytorch doc of nn.BCEWithLogitsLoss, pos_weight is an optional argument  a that takes the weight of positive examples. I don't fully understand the statement "pos_weight > 1 increases recall and pos_weight < 1 increases precision" in that page. How do you guys understand this statement?

Output:
The binary cross-entropy with logits loss (nn.BCEWithLogitsLoss, equivalent to F.binary_cross_entropy_with_logits) is a sigmoid layer (nn.Sigmoid) followed with a binary cross-entropy loss (nn.BCELoss). The general case assumes you are in a multi-label classification task i.e. a single input can be labeled with multiple classes. One common sub-case is to have a single class: the binary classification task. If you define q as your tensor of predicted classes and p the ground-truth [0,1] corresponding to the true probabilities for each class.
The explicit formulation for the binary cross-entropy would be:
z = torch.sigmoid(q)
loss = -(w_p*p*torch.log(z) + (1-p)*torch.log(1-z))

introducing the w_p, the weight associated with the true label for each class. Read this post for more details on the weighting scheme used by the BCELoss.
For a given class:
precision =  TP / (TP + FP)
recall = TP / (TP + FN)

Then if w_p > 1, it increases the weight on the positive classification (classifying as true). This will tend to increase false positives (FP), thus decreasing the precision. Similarly if if w_p < 1, we are decreasing the weight on the true class which means it will tend to increase false negatives (FN), which decreases recall.


Instruction: Is it possible to get dataset file infromation at the time of test a model?

Input:
My dataset code is like the below one; Here, X_test is a list[list] and y_test is list[Path]
The first.py file
self.test_dataset = LongDataset(
                X_path=X_test,
                y_path=y_test,
                transform=val_transforms,
            )

The rest of the part is as usual (dataloader)
def test_dataloader(self):
        return DataLoader(self.test_dataset, batch_size=1, num_workers=8)

In the second.py file
The DataModule
data_module = DataModuleLong(batch_size=3,)

The Trainer
trainer = Trainer(gpus=1)
    trainer.test(
        model=model,
        ckpt_path=ckpt_path,
        datamodule=data_module,
    )

The  train_step() in the third.py file
def test_step(self, batch, batch_idx: int):
        inputs, targets = batch
        logits = self(inputs)
        ...
        ...
        ...

Now, is it possible to print (in the train_step()) the (inputs, targets) filename (or the full path) I am sending from test_dataset as (X_path, y_path)?

Output:
Essentially, what you want to do is get the index of each batch element in the batch returned by the dataloader object (from there it is trivial to index the dataset to get the desired data elements (in this case file paths).
Now the short answer is that there is no directly implemented way to return this data using the dataloader. However, there are a few workarounds:

Pass your own BatchSampler or Sampler object to the DataLoader constructor. Unfortunately there's not a simple way to query the Sampler for the current batch because it relies on generators (where yielding the next sample clears it and loades the next one. This is the same reason why you can't directly access the batch indices of the Dataloader. So to use this method, you'd have to pass a sampler wherein you know a priori which indices will be returned on the i-th query to the sampler. Not an ideal solution.

Create a custom dataset object - this is actually extremely easy to do, simply inherit from the torch.data object and implement the __init__, __len__ and __getitem__ methods. The __getitem__ method takes an index (let's say idx) as input and returns that index of the dataset. You can essentially copy the code for the existing LongDataset line for line, but simply append idx to the returned values from the __getitem__ method. I would demonstrate but you don't indicate where the LongDataset code comes from.
def __getitem__(self,idx):
    ... #load files, preprocess, etc.
    return data, idx

Now dataloader will automatically zip the idx values for each, so you can simply replace the existing line with:
inputs, targets, indices = batch
data_paths = [self.test_dataset.file_paths[idx] for idx in indices]

The second solution is by far preferable as it is more transparently easy to understand.


Instruction: std::vector, how does it store values with libtorch tensors?

Input:
When I was collecting trainable parameters as vector<torch::tensor>, I've realized that it is type cast to torch::autograd::VariableList.
With this structure, how does the vector access its element? Does it store the value's memory space even without explicitly having to call them by pointer or reference?
So I've tested with some simple codes like this.
With regular int data type:
int a = 10;
std::vector<int> b;
b.push_back(a);
b[0] += 10;
cout << b << endl;
cout << a << endl;

As expected, this produces 20 for b (only one element), and 10 for a (original int data)
However, for the torch::Tensor with the same style of codes:
torch::Tensor t = torch::ones({ 1 });
std::vector<torch::Tensor> tv;
tv.push_back(t);
tv[0] += 10;
cout << t << endl;
cout << tv << endl;

Just like the int vectors, I thought tv will produce 11 (one element vector), and v is just 1 (shape 1)
However, the results for both tv and v are updated to 11.
Although the operation is done on the vector, the original tensor value is also updated. Why does this happen?
My guess is torch::autograd::Variable list stores its element by memory address...?
Also, when you do,
torch::Tensor t = torch::ones({ 1 });
std::vector<torch::Tensor> tv;
tv.push_back(t);
tv[0] = tv[0] + 10;
cout << t << endl;
cout << tv << endl;

Only tv value is updated to 11 and original tensor t is the same 1.
I mean this makes collecting trainable parameters and passing them to the optimizer much easier, but I am really not sure about how this happens.
Could you please kindly explain to me why these cases are all different and how the vector stores the elements in each case?
Thank you very much for your help in advance!

Output:
This is not strange behaviour of std::vector, it is strange behaviour of torch::Tensor. The following should also exhibit it.
int a = 10;
int b = a;
b += 10;
std::cout << b << std::endl;
std::cout << a << std::endl;

torch::Tensor c = torch::ones({ 1 });
torch::Tensor d = c;
d += 10;
std::cout << d << std::endl;
std::cout << c << std::endl;

torch::Tensor e = torch::ones({ 1 });
torch::Tensor f = e;
f = f + 10;
std::cout << f << std::endl;
std::cout << e << std::endl;

A std::vector<T> allocates some space, and constructs T instances in that space. The particular constructor used depends on how you insert. In your case push_back uses the copy constructor of the type (it would use the move constructor if given an rvalue).


Instruction: How do I solve error: Tensor object has not atrribute 'fold'

Input:
I have a method that divides an image into patches and change the colour of a specified patch. I tried merging the patches together  after the manipulation and I got the error: AttributeError: 'Tensor' object has no attribute 'fold'
def perturb_patch(img, patch_idx, patch_size, stride):
    img = img.unsqueeze(0)
    patches = img.unfold(2, patch_size, stride).unfold(3, patch_size, stride)
    patches = patches.reshape(1, 3, -1, patch_size, patch_size)
    patches = patches.squeeze(0).permute(1, 0, 2, 3)
    patches[patch_idx][0, :, :] = 0.09803922
    patches[patch_idx][1,:, :] = 0.21333333
    patches[patch_idx][2,:, :] = 0.61176471

    merged_patches = patches.fold(img.shape[-2:], kernel_size=16, stride=16, padding=0)
    return merged_patches

When I tried returning patches instead of merged_patches with  new_img = perturb_patch(img, 6, 16, 16), I could visualize the patches and the manipulated patch was noticeable. How do I merge this patches together to form the original image which of size (3, 224, 224)?

Output:
So, I was able to find an alternative to merge the patches together with the pytorch's view method here.
updated code:
def perturb_patch(img, patch_idx, patch_size, stride):

    img = img.unsqueeze(0)
    patches = img.unfold(2, patch_size, stride).unfold(3, patch_size, stride)
    patches = patches.reshape(1, 3, -1, patch_size, patch_size)
    patches = patches.squeeze(0).permute(1, 0, 2, 3)

    patches[patch_idx][0, :, :] = 0.09803922
    patches[patch_idx][1,:, :] = 0.21333333
    patches[patch_idx][2,:, :] = 0.61176471

    unsqueezed_patch = patches.unsqueeze(0)
    grid_size = (14, 14)
    batch_size, num_patches, c, height, width = unsqueezed_patch.size()
    image = unsqueezed_patch.view(batch_size, grid_size[0], grid_size[1], c, height, width)
    output_height = grid_size[0] * height
    output_width = grid_size[1] * width

    image = image.permute(0, 3, 1, 4, 2, 5).contiguous()
    image = image.view(batch_size, c, output_height, output_width)

    return image


Instruction: Unknown category '2' encountered. Set `add_nan=True` to allow unknown categories pytorch_forecasting

Input:
error: "Unknown category '2' encountered. Set add_nan=True to allow unknown categories" while creating time series dataset in pytorch forecasting.
training = TimeSeriesDataSet(
train,
time_idx="index",
target=dni,
group_ids=["Solar Zenith Angle", "Relative Humidity","Dew
Point","Temperature","Precipitable Water", "Wind Speed"],
min_encoder_length=max_encoder_length // 2,  # keep encoder length long (as it is in the
validation set)
max_encoder_length=max_encoder_length,
min_prediction_length=1,
max_prediction_length=max_prediction_length,
static_reals=["Wind Direction"],
time_varying_known_reals=["index", "Solar Zenith Angle", "Relative Humidity","Dew
Point","Temperature","Precipitable Water"],
#     time_varying_unknown_categoricals=[],
time_varying_unknown_reals=[dhi,dni,ghi],
categorical_encoders={data.columns[2]: NaNLabelEncoder(add_nan=True)},
target_normalizer=GroupNormalizer(
    groups=["Solar Zenith Angle", "Relative Humidity","Dew
Point","Temperature","Precipitable Water", "Wind Speed"], transformation="softplus"
),  # use softplus and normalize by group
add_relative_time_idx=True,
add_target_scales=True,
add_encoder_length=True,

)

Output:
Try adding pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True), as in this example:
max_prediction_length = 1
max_encoder_length = 27

training = TimeSeriesDataSet(
    sales_train,
    time_idx='dayofyear',
    target="QTT",
    group_ids=['S100','I100','C100','C101'],
    min_encoder_length=0,
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=[],
    static_reals=['S100','I100','C100','C101'],
    time_varying_known_categoricals=[],
    time_varying_known_reals=['DATE'],
    time_varying_unknown_categoricals=[],
    time_varying_unknown_reals=['DATE'],
    categorical_encoders={
        'S100': *pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),*
        'I100':pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),
        'C100':pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),
        'C101':pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True)
    },
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_length=True,
    allow_missing_timesteps=True

)
print ('Executado')


Instruction: Input 1D array with float datatype in C++

Input:
I would like to input row = [0.160625, 0.967468297, 3.520480583, 0.862454481, -0.341933766] as entry which is float type and pass it to forward module. I used python trying to translate to C++,
I got syntax error. Support needed. Thanks!
    // run not okay
    // Create a vector of inputs.
    std::vector<torch::jit::IValue> inputs;
    row = [0.190625, 0.957468297, 4.520480583, 0.962454481, -0.241933766]
    inputs.push_back(torch::tensor(row));

    // Execute the model and turn its output into a tensor.
    at::Tensor output = module.forward(inputs).toTensor();
    std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/4) << '\n';

I would like to use row as instance and get the output.
When I use dummy values such as torch::one({1, 5}), the app run ok.
However, when I passed the real value as row - the float array, the app is aborted.
    // run ok for this case
    // Create a vector of inputs.
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(torch::ones({ 1, 5}));

    // Execute the model and turn its output into a tensor.
    at::Tensor output = module.forward(inputs).toTensor();
    std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/4) << '\n';

Output:
Did you try replacing [] with {} as mentioned before?
float row[] = { 0.190625, 0.957468297, 4.520480583, 0.962454481, -0.241933766 };


Instruction: stylegan3 stylegan2-ada tensor mismatch error for every 256 or 512 flickr related model

Input:
Anyone having the same tensor size mismatch when trying finetuning on ffhq,ffhqu or celebahq models with stylegan3 (and with --cfg=stylegan2)?
With afhqv2  and metfaces I had no problems at 512 and 1048 sizes.
Error:
...
File "/home/ubuntu/stylegan3/training/training_loop.py", line 162, in training_loop
    misc.copy_params_and_buffers(resume_data[name], module, require_all=False)
  File "/home/ubuntu/stylegan3/torch_utils/misc.py", line 162, in copy_params_and_buffers
    tensor.copy_(src_tensors[name].detach()).requires_grad_(tensor.requires_grad)

RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 0

example command:
python "train.py" --outdir=training-runs --cfg=stylegan3-r --data="datasets/256.zip" --gpus=1 --batch=16 --batch-gpu=16 --gamma=6.6 --mirror=1 --kimg=2 --snap=5  --resume=https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-ffhqu-256x256.pkl

I've verified I was passing 256 images built with the tool:
python dataset_tool.py --source="img/" --dest="datasets/256.zip" --resolution='256x256'

Note: I was able to finetune only with this version stylegan2-ffhq-512x512.pkl

Output:
If you are trying to do transfer learning on "stylegan3-r-ffhqu-256x256.pkl", you should add
--cbase=16384

in your python "train.py" ...
command line


Instruction: Intermediate layer outputs pytorch

Input:
I have Alexnet neural network:
class AlexNet(nn.Module):
def __init__(self, num_classes=100):
    super(AlexNet, self).__init__()
    self.features = nn.Sequential(
        nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(64, 192, kernel_size=5, padding=2),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(192, 384, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(384, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(256, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
    )
    self.classifier = nn.Sequential(
        nn.Dropout(),
        nn.Linear(256 * 6 * 6, 4096),
        nn.ReLU(inplace=True),
        nn.Dropout(),
        nn.Linear(4096, 4096),
        nn.ReLU(inplace=True),
        nn.Linear(4096, num_classes),
    )

def forward(self, x):
    x = self.features(x)
    x = x.view(x.size(0), 256 * 6 * 6)
    x = self.classifier(x)
    return x

I am trying to get the information of the intermediate layers (for example the penultimate layer ) with backward hook but I couldn't get it

Output:
According to this answer
You have to split your model in different parts and create methods to access them parts such as :
class AlexNet(nn.Module):
    def __init__(self, num_classes=100):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def getFeatures(self,x):
        x = self.features(x)
        return x.view(x.size(0), 256 * 6 * 6)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x

This way is quite common and you can find plenty of examples.


Instruction: Pytorch Geometric Datasets

Input:
I need your help. I have two set of graph structured data, one from Open Graph Benchmark (OGB) and another created with torch_geometric.data.Dataset from my own data . The data looks like:

Data(edge_index=[2, 88], edge_attr=[88, 3], x=[39, 9], y=[1, 1])  #OGB

Data(x=[23, 9], edge_index=[2, 48], edge_attr=[48, 2], y=[1]) #PyG

I am trying to use a framework developed using OGB functions, this doesn't work with data created using PyG. For example: the first part of the framework load and split the dataset into train, val and test:
# Set the random seed
random.seed(random_seed)
np.random.seed(random_seed)

# Create data loaders
split_idx = dataset.get_idx_split() # train/val/test split
loader_dict = {}
for phase in split_idx:
    batch_size = 32
    loader_dict[phase] = DataLoader(dataset[split_idx[phase]], batch_size=batch_size, shuffle=False)

When I run this code with a native ogb dataset I have no problems, when I use the PyG data return the error:

AttributeError

This is strange because they are both Pytorch objects, the only difference is that the OGB dataset is an InMemoryDataset and the PyG one is a 'Larger' dataset (https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html). Is there any way to fix this without having to change the source code?
Thanks!

Output:
If you want to use the same code, you need to implement get_idx_split for your own dataset.
You can find the desired return structure in the OGB GitHub, e.g. here:
def get_idx_split(self):
    < ... do something to retrieve train/test/validation set>
    return {'train': train_idx, 'valid': valid_idx, 'test': test_idx}


Instruction: Extracting weights from SGD algorithm

Input:
So I am implementing SGD for a binary classification problem. There are 2 classes of points and I want to plot the decision boundary but I'm not sure how to extract the weights from the code to plot it.
Here is the code:
def train_model(train_dl, model):
    # define the optimization
    criterion = nn.BCELoss(reduction='none')
    optimizer = torch.optim.SGD(net.parameters(), lr=0.1)
    # enumerate epochs
    for epoch in range(10):
        # enumerate mini batches
        for i, (inputs, targets) in enumerate(train_dl):
            # clear the gradients
            optimizer.zero_grad()
            # compute the model output
            yhat = model(inputs)
            # calculate loss
            loss = criterion(yhat, targets)
            # credit assignment
            loss.backward()
            # update model weights
            optimizer.step()

Any help would be much appreciated! Thanks.

Output:
You do not extract this information from the SGD optimizer, this information is part of your model.
What you can do, at test time, is generate a grid of points, compute their prediction using the trained model and then plot the grid points coloring them according to the prediction.


Instruction: How to correct when Accuracy equals F1 in Torch Lightning for binary classification?

Input:
I understand that with multi-class, F1 (micro) is the same as Accuracy. I aim to test a binary classification in Torch Lightning but always get identical F1, and Accuracy.
To get more detail, I shared my code at GIST, where I used the MUTAG dataset. Below are some important parts I would like to bring up for discussion
The function where I compute Accuracy and F1 (line #28-40)
def evaluate(self, batch, stage=None):
        y_hat = self(batch.x, batch.edge_index, batch.batch)
        loss = self.criterion(y_hat, batch.y)
        preds = torch.argmax(y_hat.softmax(dim=1), dim=1)
        acc = accuracy(preds, batch.y)
        f1_score = f1(preds, batch.y)

        if stage:
            self.log(f"{stage}_loss", loss, on_step=True, on_epoch=True, logger=True)
            self.log(f"{stage}_acc", acc, on_step=True, on_epoch=True, logger=True)
            self.log(f"{stage}_f1", f1_score, on_step=True, on_epoch=True, logger=True)

        return loss

To inspect, I put a checkpoint at line #35, and got acc=0.5, f1_score=0.5, while prediction and label respectively are
preds = tensor([1, 1, 1, 0, 1, 1, 1, 1, 0, 0])
batch.y = tensor([1, 0, 1, 1, 0, 1, 0, 1, 1, 0])

Using these values, I run a notebook to double-check with scikit-learn
from sklearn.metrics import f1_score
y_hat = [1, 1, 1, 0, 1, 1, 1, 1, 0, 0]
y = [1, 0, 1, 1, 0, 1, 0, 1, 1, 0]
f1_score(y_hat, y, average='binary') # got 0.6153846153846153
accuracy_score(y_hat, y) # 0.5

I obtained a different result compared to evaluation's code. Besides, I verified again with torch, interestingly, I got a correct result
from torchmetrics.functional import accuracy, f1
import torch
f1(torch.Tensor(y_hat), torch.LongTensor(y)) # tensor(0.6154)
accuracy(torch.Tensor(pred), torch.LongTensor(true)) # tensor(0.5000)

I guess somehow the torch-lightning treats my calculation as a multiclass task. My question is how to correct its behavior?

Output:
You can pass multiclass=False in case your dataset is binary.
This will give you the result which matches the Sklearn F1 score output where average="binary" (default) is passed.

We can set multiclass=False to treat the inputs as binary - which is the same as converting the predictions to float beforehand.

Sklearn results
from sklearn.metrics import f1_score, accuracy_score

y_hat = [1, 1, 1, 0, 1, 1, 1, 1, 0, 0]
y =     [1, 0, 1, 1, 0, 1, 0, 1, 1, 0]

print("Binary f1: ", f1_score(y, y_hat, average="binary"))  # default
print("Micro f1:", f1_score(y, y_hat, average="micro"))  # this is same as accuracy
print("accuracy_score", accuracy_score(y, y_hat))

>>> Binary f1: 0.6153846153846153
>>> Micro f1: 0.5
>>> accuracy_score: 0.5

Pytorch-Lightning Results
import torchmetrics.functional as F
import torch

y_hat = [1, 1, 1, 0, 1, 1, 1, 1, 0, 0]
y =     [1, 0, 1, 1, 0, 1, 0, 1, 1, 0]

# torchmetrics
print("Non-Multiclass f1: ", F.f1_score(torch.tensor(y), torch.tensor(y_hat), multiclass=False))
print("Multiclass f1:", F.f1_score(torch.tensor(y), torch.tensor(y_hat))) # same as accuracy
print("accuracy_score", F.accuracy(torch.tensor(y), torch.tensor(y_hat)))

>>> Non-Multiclass f1: tensor(0.6154)
>>> Multiclass f1: tensor(0.5000)
>>> accuracy_score tensor(0.5000)


Instruction: One hot Encoding text data in pytorch

Input:
I am wondering how to one hot encode text data in pytorch?
For numeric data you could do this
import torch
import torch.functional as F

t = torch.tensor([6,6,7,8,6,1,7], dtype = torch.int64)
one_hot_vector = F.one_hot(x = t, num_classes=9)
print(one_hot_vector.shape)
# Out > torch.Size([7, 9])

But what if you have text data instead
from torchtext.data.utils import get_tokenizer
corpus = ["The cat sat the mat", "The dog ate my homework"]
tokenizer = get_tokenizer("basic_english")
tokens = [tokenizer(doc) for doc in corpus]

But how do I one hot encode this vocab using Pytorch?
With something like Scikit Learn I could do this, is there a similar way to do in pytorch
import spacy
from spacy.lang.en import English
from sklearn.preprocessing import OneHotEncoder

corpus = ["The cat sat the mat", "The dog ate my homework"]
nlp = English()
tokenizer = spacy.tokenizer.Tokenizer(nlp.vocab)
tokens = np.array([[token for token in tokenizer(doc)] for doc in corpus])
one_hot_encoder = OneHotEncoder(sparse = False)
one_hot_encoded = one_hot_encoder.fit_transform(tokens)

Output:
You can do the following:
from typing import Union, Iterable
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

corpus = ["The cat sat the mat", "The dog ate my homework"]
tokenizer = get_tokenizer("basic_english")
tokens = [tokenizer(doc) for doc in corpus]

voc = build_vocab_from_iterator(tokens)

def my_one_hot(voc, keys: Union[str, Iterable]):
    if isinstance(keys, str):
        keys = [keys]
    return F.one_hot(torch.tensor(voc(keys)), num_classes=len(voc))


Instruction: How to customize threshold PyTorch

Input:
I have trained ResNet50 for binary image classification.
I want to descrease FalseNegatives by reducing threshold value.
How can I do that?

Output:
To decrease the number of false negatives (FN) i.e. increase the recall (since recall = TP / (TP + FN)) you should increase the positive weight (the weight of the occurrence of that class) above 1. For example nn.BCEWithLogitsLoss allows you to provide the pos_weight option:

pos_weight > 1 increases the recall, pos_weight < 1 increases the precision.
For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100 = 3. The loss would act as if the dataset contains 3*100 = 300 positive examples.

As a side note, the explicit expression for the binary cross entropy with logits (where "with logits" should rather be understood as "from logits") is:
>>> z = torch.sigmoid(q)
>>> loss = -(w_p*p*torch.log(z) + (1-p)*torch.log(1-z))

Above q are the raw logit values while w_p is the weight of the positive instance.


Instruction: How can Python execute a line of code, then display an error indicating a crash at the previous line?

Input:
I have a script which contains (among other things) these three lines of code:
(line 138) pdb.set_trace()

(line 140) training_start_time = datetime.now()
(line 141) print(f'Network training beginning at {training_start_time}.')

Here is the output I'm seeing:
> c:\vtcproject\yolov5\roadometry_train.py(140)train()
-> training_start_time = datetime.now()
(Pdb) >? continue
Network training beginning at 2022-02-17 11:23:04.340499.
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/VTCProject/yolov5/roadometry_train.py", line 316, in <module>
    main(opt)
  File "C:/VTCProject/yolov5/roadometry_train.py", line 304, in main
    train(data, cfg, hyp, opt, device)
  File "C:/VTCProject/yolov5/roadometry_train.py", line 140, in train
    training_start_time = datetime.now()
TypeError: 'module' object is not callable

There are a few things I don't understand here.

How is it possible to see this print output (network training beginning at...) if the code is crashing at datetime.now()?

Why is it that if I manually execute dt = datetime.now() in pdb, then it works fine, but I'm seeing TypeError: 'module' object is not callable if I type continue into pdb or just execute the script without pdb?

The library is imported like this:
(line 13) from datetime import datetime

Output:
Well, this turned out to be somewhat complicated.
I removed code from my script until I found something very strange. Further down my script, below the lines listed above, I was eventually entering a loop which created an instance of tqdm for progress updates:
for epoch in range(start_epoch,
                   opt.epochs):
    pbar = enumerate(train_loader)
    pbar = tqdm(pbar, total=nb)  # progress bar

When I commented out the tqdm usage in the loop, the TypeError above (at line 140, according to the exception) disappeared.
Paring everything down reinforced that simply calling tqdm caused this error. Searching the tqdm Github issues, I found this:
https://github.com/tqdm/tqdm/issues/611
There is a post in this discussion indicating that TypeError: 'module' object is not callable can occur if one uses:
import tqdm

instead of:
from tqdm import tqdm

So my analysis of what happened here is that tqdm was simply imported as a module, and obviously calling a module as a new object instance isn't going to work. The only confusing part is why the stated line-number is wrong.
The issue raised on Github corresponds closely to my scenario: creating a PyTorch dataloader which is then passed to tqdm.
I was basing my code on the Ultralytics Yolov5 repo and it appears that I changed from tqdm import tqdm to just import tqdm mistakenly. Specifically importing the class and not just the module causes the TypeError to disappear.
It seems that this error has nothing to do with datetime.now(). After commenting this out, I still get a TypeError, but pointing at a different line - now it blames the actual line trying to create a tqdm instance, which is what I would have expected in the first place.
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 41, in train
    for epoch in range(start_epoch,
TypeError: 'module' object is not callable

In the above output, line 41 of roadometry_debug.py is:
    pbar = tqdm(pbar, total=nb)  # progress bar

While the line number being blamed appears correct, it seems that the error printout is still printing out the wrong line: for epoch in range....
This explains why pdb allows me to manually execute the next line, and the next print-out: because they're not the issue!
I still don't understand why the first error text blames the wrong line of code, or why the 2nd error text prints out the wrong text but the correct line-number.

Update: it seems that pdb is causing the reported line numbers in the error message to be incorrect.
Here is a minimal example:
import pdb
import tqdm
from datetime import datetime

def train():

    pdb.set_trace()

    training_start_time = datetime.now()
    print(f'Network training beginning at {training_start_time}.')

    for epoch in range(0,
                       10):  # epoch ------------------------------------------------------------------
        pbar = enumerate([1, 2, 3])
        pbar = tqdm(pbar, total=3)  # progress bar

def main():
    train()

if __name__ == "__main__":
    main()

The error message printed out from the above blames the wrong line:
c:\vtcproject\yolov5\roadometry_debug.py(10)train()
-> training_start_time = datetime.now()
(Pdb) >? continue
Network training beginning at 2022-02-17 18:32:13.892776.
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 23, in <module>
    main()
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 20, in main
    train()
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 10, in train
    training_start_time = datetime.now()
TypeError: 'module' object is not callable

If the call to pdb.set_trace() is commented out, the error message blames the correct line.
Script after commenting out relevant line:
import pdb
import tqdm
from datetime import datetime

def train():

    #pdb.set_trace()

    training_start_time = datetime.now()
    print(f'Network training beginning at {training_start_time}.')

    for epoch in range(0,
                       10):  # epoch ------------------------------------------------------------------
        pbar = enumerate([1, 2, 3])
        pbar = tqdm(pbar, total=3)  # progress bar

def main():
    train()

if __name__ == "__main__":
    main()

Output:
Network training beginning at 2022-02-17 18:33:31.278133.
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Users\Alexander Farley\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\212.5457.59\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 23, in <module>
    main()
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 20, in main
    train()
  File "C:/VTCProject/yolov5/roadometry_debug.py", line 16, in train
    pbar = tqdm(pbar, total=3)  # progress bar
TypeError: 'module' object is not callable

Notice that the final line-number in the error printout has now changed from 10 (incorrect) to 16 (correct).
It's a reverse heisenbug - only appears if observed!


Instruction: Problem in passing value to parser argument

Input:
Running my code including this line:
def parse_args():
    parser = argparse.ArgumentParser(description='test with parser')
    parser.add_argument("--model", type=str, default= "E:\Script\weights\resnext101.pth")

I got this error:
OSError: [Errno 22] Invalid argument: 'E:\\Script\\weights\resnext101.pth'

What is the error for and how I can fix it?

Output:
You aren't passing a path ending with the name resnext101.pth; you are passing a path ending with the name weights␍esnext101.pth, which contains a literal carriage return.
Use a raw string literal to protect all backslashes from expansion, regardless of the character that follows the backslash.
parser.add_argument("--model", type=str, default= r"E:\Script\weights\resnext101.pth")


Instruction: No module named 'model'

Input:
import numpy as np
import random
import json

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

from nltk_utils import bag_of_words, tokenize, stem
from model import NeuralNet

i keep trying to pip install NeuralNet but I keep getting

ModuleNotFoundError: No module named 'model'

I have Neuralnet successfully installed on my pc, and I have have tried what you said I should try and its still not working, can I send you the project on linkedin so you would check it out

Output:
I think you are supposed to import neuralnet by itself:
import neuralnet

or import model from neuralnet:
from neuralnet import model

Since model seems to be a part of the NeuralNet module rather than the other way around.


Instruction: Setting Picture Colormap in Tensorboard with PyTorch

Input:
I'm pretty new to using Python, PyTorch and Tensorboard - moved from MATLAB due to the lacking automatic differentiation.
I am trying to use the above stated tools since I'm running an optimization problem - simple gradient descent for reconstructing distorted images. No machine learning or deep learning.
The point is that I need to see the images every few iterations of the algorithm, so I was told that tensorboard would be great for it. The only problem is that these images are shown in grayscale and I need to see it in a different colormap. Is there any way to change the colormap in tensorboard?
Thanks!

Output:
You can colorize your tensor shape using tensorflow gather function. Following is a simple script for doing this. You may use other maps rather than 'Spectral':
import matplotlib
import matplotlib.cm
import tensorflow as tf

def colormap(shape):
    min = tf.reduce_min(shape)
    max = tf.reduce_max(shape)
    shape = (shape - min) / (max - min)
    gatherIndices = tf.to_int32(tf.round(shape * 255))
    map = matplotlib.cm.get_cmap('cividis')
    colors = tf.constant(map.colors, dtype=tf.float32)
    return tf.gather(colors, gatherIndices)


Instruction: Multiplying PyTorch tensors of different shape

Input: I have a torch tensor of shape (32, 100, 50) and another of shape (32,100). Call these A and B respectively. I want to element-wise multiply A and B, such that each of the 50 elements at A[i, j, :] get multiplied by B[i, j], i.e.like multiplying a vector with a scalar. How can I do this via broadcasting rules?

Output:
Just add a singleton dimension to the second tensor, for example:
a = torch.randn([32,100,50])
b = torch.randint(10,[32,100])
b = b[:,:,None] #or .unsqueeze(-1)
c = a * b
assert (c[0,0,0]/a[0,0,0]).int() == b[0,0] and (c[0,0,1]/a[0,0,1]).int() == b[0,0]

The assert on the end is just to prove that adjacent elements in the last dimension is multiplied by the same element of b.


Instruction: PyTorch Binary classification not learning

Input:
I state that I am new on PyTorch. I wrote this simple program for binary classification. I also created the CSV with two columns of random values, with the "ok" column whose value is 1 only if the other two values are included between two values I decided at the same time. Example:
diam_int,diam_est,ok
37.782,125.507,0
41.278,115.15,1
42.248,115.489,1
29.582,113.141,0
37.428,107.247,0
32.947,123.233,0
37.146,121.537,0
38.537,110.032,0
26.553,113.752,0
27.369,121.144,0
41.632,108.178,0
27.655,111.279,0
29.779,109.268,0
43.695,115.649,1
44.587,116.126,0

It seems to me all done correctly, loss actually lowers (it comes back up slightly after many epochs but I don't think it's a problem), but when I try to test my Net after the training, with a sample batch of the trainset, what I got is always a prediction below 0.5 (so always 0 as estimated output) with a completely random trend.
with torch.no_grad():
        pred = net(trainSet[10])
        trueVal = ySet[10]
        for i in range(len(trueVal)):
            print(trueVal[i], pred[i])

Here is my Net class:
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self) :
        super().__init__()
        self.fc1 = nn.Linear(2, 32)
        self.fc2 = nn.Linear(32, 64)
        self.fc3 = nn.Linear(64, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return torch.sigmoid(x)

Here is my Main class:
import torch
import torch.optim as optim
import torch.nn.functional as F
import pandas as pd

from net import Net

df = pd.read_csv("test.csv")
y = torch.Tensor(df["ok"])
ySet = torch.split(y, 32)
df.drop(["ok"], axis=1, inplace=True)
data = F.normalize(torch.Tensor(df.values), dim=1)
trainSet = torch.split(data, 32)

net = Net()
optimizer = optim.Adam(net.parameters(), lr=0.001)
lossFunction = torch.nn.BCELoss()
EPOCHS = 300

for epoch in range(EPOCHS):
    for i, X in enumerate(trainSet):
        optimizer.zero_grad()
        output = net(X)
        target = ySet[i].reshape(-1, 1)
        loss = lossFunction(output, target)
        loss.backward()
        optimizer.step()

    if epoch % 20 == 0:
        print(loss)

What am I doing wrong? Thanks in advance for the help

Output:
Your model is underfit. Increasing the number of epochs to (say) 3000 makes the model predict perfectly on the examples you showed.
However after this many epochs the model may be overfit. A good practice is to use validation data (separate the generated data into train and validation sets), and check the validation loss in each epoch. When the validation loss starts increasing you start overfitting and stop the training.


Instruction: Convert Keras (TensorFlow) MaxPooling3d to PyTorch MaxPool3d

Input:
I'm Trying to convert some Keras (TensorFlow) code to Pytorch, and I'm unable to reproduce the MaxPooling3d in Keras (TensorFlow) as MaxPool3d in PyTorch.
The following code:
import torch
import torch.nn as nn
import tensorflow.keras.layers as layers
import matplotlib.pyplot as plt

kernel_size = (10, 10, 2)
strides = (32, 32, 2)
in_tensor = torch.randn(1, 1, 256, 256, 64)

tf_out = layers.MaxPooling3D(data_format='channels_first', pool_size=kernel_size,
                                                    strides=strides, padding='same')(in_tensor.detach().numpy())
pt_out = nn.MaxPool3d(kernel_size=kernel_size, stride=strides)(in_tensor)

fig = plt.figure(figsize=(10, 5))
axs = fig.subplots(1,2)
axs[0].matshow(pt_out[0,0,:,:,0].detach().numpy())
axs[0].set_title('PyTorch')
axs[1].matshow(tf_out.numpy()[0,0,:,:,0])
axs[1].set_title('TensorFlow')

Gives very different results:

What could be the problem?
Is the padding in the PyTorch version inorrect?

Output:
The padding is not the same in both layers, that's why you're not getting the same results.
You set padding='same' in  tensorflow MaxPooling3D layer, but there is no padding set in pytorch MaxPool3d layer.
Unfortunately, in Pytorch, there is no option for 'same' padding for MaxPool3d as in tensorflow. So, you will need to manually pad the tensor before passing it to the pytorch MaxPool3d layer.
Try this code:
import torch
import torch.nn as nn
import torch.nn.functional as F
import tensorflow.keras.layers as layers
import matplotlib.pyplot as plt

kernel_size = (10, 10, 2)
strides = (32, 32, 2)
in_tensor = torch.randn(1, 1, 256, 256, 64)

tf_out = layers.MaxPooling3D(data_format='channels_first', pool_size=kernel_size,
                                                    strides=strides)(in_tensor.detach().numpy())
in_tensor = F.pad(in_tensor, (0, 0, 0, 0))
pt_out = nn.MaxPool3d(kernel_size=kernel_size, stride=strides)(in_tensor)

fig = plt.figure(figsize=(10, 5))
axs = fig.subplots(1,2)
axs[0].matshow(pt_out[0,0,:,:,0].detach().numpy())
axs[0].set_title('PyTorch')
axs[1].matshow(tf_out.numpy()[0,0,:,:,0])
axs[1].set_title('TensorFlow')

Output:


Instruction: error then import pytorch-lightning, azure notebook

Input:
i am use microsoft azure (for students) ML servise. Then i work with notebook i can not import pytorch-lightning libary.
!pip install pytorch-lightning==0.9.0
import pytorch_lightning as pl

Here i have error:
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <module>
----> 2 import pytorch_lightning as pl

ModuleNotFoundError: No module named 'pytorch_lightning'

This is unbearably weird. someone faced such a problem?

Output:
This is rather strange but could be related to that your installation is in another location, so let's:

try where is PL installed with find -name "lightning"
also, check what is the loaded package locations python -c "import sys; print(sys.path)"

I guess that the problem will be in What's the difference between dist-packages and site-packages?


Instruction: Having problems with Pandas when storing the results of a CNN

Input:
I have a CNN that runs well, but when I'm trying to store the error, loss and accuracy training and validation with Pandas, for some reason the Data Frame that I created has more rows than necesary (173 to be exact) and it looks like it trains more than I ask to, but while the CNN is training and validating the results it gives are according to what I ask of it. I will lay all of my code here by parts.
This is how I define my neural network
class Network(nn.Module):
    def __init__(self, p=0.1):
        super().__init__()

        # define layers
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)

        self.fc1 = nn.Linear(in_features=12 * 4 * 4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)

        self.dropout = nn.Dropout(p)

    # define forward function
    def forward(self, t):
        # conv 1
        t = self.conv1(t)
        t = F.relu(t)
        t = F.max_pool2d(t, kernel_size=2, stride=2)

        # conv 2
        t = self.conv2(t)
        t = F.relu(t)
        t = F.max_pool2d(t, kernel_size=2, stride=2)

        # fc1
        t = t.reshape(-1, 12 * 4 * 4)
        t = self.fc1(t)
        t = self.dropout(t)
        t = F.relu(t)

        # fc2
        t = self.fc2(t)
        t = self.dropout(t)
        t = F.relu(t)

        # output
        t = self.out(t)
        # don't need softmax here since we'll use cross-entropy as activation.

        return t

I call my network as my model and move it to the CPU
model = Network().to(device)

This is how I define my test and training loops
def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)

    for batch, (X, y) in enumerate(dataloader):

        pred = model(X)
        loss = loss_fn(pred, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)

def test_loop(dataloader,model,loss_fn):
    num_samples  = 0
    num_batches  = 0
    avrg_loss    = 0
    frac_correct = 0
    model.eval()

    model = model.to(device)

    with torch.no_grad():

        for X,y in dataloader:
            X = X.to(device)
            y = y.to(device)
            pred = model(X)
            num_batches += 1
            avrg_loss += loss_fn(pred,y).item()
            num_samples += y.size(0)
            frac_correct += (pred.argmax(1)==y).type(torch.float).sum().item()

    avrg_loss    /= num_batches
    frac_correct /= num_samples
    return avrg_loss,frac_correct

And here is where I call my CNN and start training it
learning_rate = 1e-3
batch_size = 1000
num_epochs = 100
num_k = 1
n=10
dropouts=[0.1,0.3,0.5]

loss_fn = nn.CrossEntropyLoss()

df = pd.DataFrame()

for p in dropouts:
    model = Network(p)
    train_dataloader = DataLoader(train_dataset,batch_size=batch_size)
    valid_dataloader = DataLoader(test_dataset,batch_size=batch_size)
    optimizer = torch.optim.Adam(model.parameters(),lr=learning_rate,eps=1e-08,weight_decay=0,amsgrad=False)
    min_valid_loss = float("inf")
    for epoch in range(num_epochs):
        train_loop(train_dataloader,model,loss_fn,optimizer)
        train_loss,train_accu = test_loop(train_dataloader,model,loss_fn)
        valid_loss,valid_accu = test_loop(valid_dataloader,model,loss_fn)
        print(f"n={n} p={p} epoch={epoch} train_loss={train_loss} train_accu={train_accu} valid_loss={valid_loss} valid_accu={valid_accu}")
        df = df.append({"n":n,
                        "p":p,
                        "epoch":epoch,
                        "train_loss":train_loss,
                        "train_accu":train_accu,
                        "valid_loss":valid_loss,
                        "valid_accu":valid_accu}
                        ,ignore_index=True)
json_fname = "simulation-results-"+datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")+".json"
df.to_json(json_fname)
if COLAB:
    files.download(json_fname)

The results I get while is training look like this
n=10 p=0.1 epoch=0 train_loss=0.8500287334124247 train_accu=0.6877666666666666 valid_loss=0.864807802438736 valid_accu=0.684

so I can sort of infere that it's doing a good job. But when I ask Pandas ifnormation about the DataFrame df with df.info() I get this
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 474 entries, 0 to 473
Data columns (total 8 columns):
 #   Column      Non-Null Count  Dtype
---  ------      --------------  -----
 0   n           425 non-null    float64
 1   k           174 non-null    float64
 2   epoch       474 non-null    int64
 3   train_loss  474 non-null    float64
 4   train_accu  474 non-null    float64
 5   valid_loss  474 non-null    float64
 6   valid_accu  474 non-null    float64
 7   p           300 non-null    float64
dtypes: float64(7), int64(1)
memory usage: 29.8 KB

So Pandas is,for some reason, adding 173 rows with information that makes no sense (for instance my dropout p is NaN in those 173 rows) and I don't know what's going on.

Output:
Hmm, you should try wrapping your dictionary with brackets like so,
data = {'n': n, 'n': p, 'epoch': epoch} # etc...
pd.DataFrame([data])

If this doesn't work, you should consider converting your dictionary to a pandas dataframe using this function.
data = {'n': n, 'p': p, 'epoch': epoch} # etc...
pd.DataFrame.from_dict(data)

You can also specify your own columns to make the dataframe more readable.
data = {'n': n, 'p': p, 'epoch': epoch} # etc...
columns = ['N', 'P', 'Epoch'] # etc...
pd.DataFrame.from_dict(data, columns=columns)

Obviously, the column size must match the data.