Instruction: PyTorch BatchNorm2d Calculation

Input:
I am trying to understand the mechanics of PyTorch BatchNorm2d through calculation. My example code:
import torch
from torch import nn

torch.manual_seed(123)

a = torch.rand(3,2,3,3)
print(a)

print(nn.BatchNorm2d(2)(a))
#print(a[:,0,:,:])
mean_by_plane_feature = torch.mean(a,dim=0)
std_by_plane_feature = torch.std(a,dim=0)
print(mean_by_plane_feature)
print(std_by_plane_feature)

Output:
tensor([[[[0.2961, 0.5166, 0.2517],
          [0.6886, 0.0740, 0.8665],
          [0.1366, 0.1025, 0.1841]],

         [[0.7264, 0.3153, 0.6871],
          [0.0756, 0.1966, 0.3164],
          [0.4017, 0.1186, 0.8274]]],

        [[[0.3821, 0.6605, 0.8536],
          [0.5932, 0.6367, 0.9826],
          [0.2745, 0.6584, 0.2775]],

         [[0.8573, 0.8993, 0.0390],
          [0.9268, 0.7388, 0.7179],
          [0.7058, 0.9156, 0.4340]]],

        [[[0.0772, 0.3565, 0.1479],
          [0.5331, 0.4066, 0.2318],
          [0.4545, 0.9737, 0.4606]],

         [[0.5159, 0.4220, 0.5786],
          [0.9455, 0.8057, 0.6775],
          [0.6087, 0.6179, 0.6932]]]])
tensor([[[[-0.5621,  0.2574, -0.7273],
          [ 0.8968, -1.3879,  1.5584],
          [-1.1552, -1.2819, -0.9787]],

         [[ 0.5369, -1.0117,  0.3888],
          [-1.9141, -1.4584, -1.0073],
          [-0.6859, -1.7524,  0.9171]]],

        [[[-0.2425,  0.7925,  1.5103],
          [ 0.5422,  0.7042,  1.9901],
          [-0.6425,  0.7846, -0.6311]],

         [[ 1.0298,  1.1880, -2.0520],
          [ 1.2915,  0.5833,  0.5047],
          [ 0.4593,  1.2495, -0.5645]]],

        [[[-1.3761, -0.3375, -1.1132],
          [ 0.3187, -0.1512, -0.8011],
          [ 0.0269,  1.9569,  0.0493]],

         [[-0.2561, -0.6096, -0.0199],
          [ 1.3619,  0.8356,  0.3525],
          [ 0.0933,  0.1281,  0.4116]]]], grad_fn=<NativeBatchNormBackward>)
tensor([[[0.2518, 0.5112, 0.4177],
         [0.6049, 0.3724, 0.6937],
         [0.2885, 0.5782, 0.3074]],

        [[0.6999, 0.5455, 0.4349],
         [0.6493, 0.5804, 0.5706],
         [0.5721, 0.5507, 0.6515]]])
tensor([[[0.1572, 0.1521, 0.3810],
         [0.0784, 0.2829, 0.4042],
         [0.1594, 0.4411, 0.1406]],

        [[0.1723, 0.3110, 0.3471],
         [0.4969, 0.3340, 0.2211],
         [0.1553, 0.4028, 0.2000]]])

I found that the output of BatchNorm is not what I expected to be. For example, the mean across batch for first plane, first feature = 0.2518 and the std is 0.1572. The normalized value for the first value = (0.2961-0.2518)/0.1572 = 0.2818 != -0.5621.
My questions:

Am I correct to calculate the means in this way (across batch, per plane and feature)? as I understand batchnorm is used to treat the issue of having different scales for different feature, so it should at least be per feature dimension, however I am not sure whether to sum across the "plane dimension" as well.

Any other modifications I need to do to get the same output from BatchNorm2d?

Output:
This is the implementation of BatchNorm2d in pytorch (source1, source2). Using this, you can verify the operations you performed.
class MyBatchNorm2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1,
                 affine=True, track_running_stats=True):
        super(MyBatchNorm2d, self).__init__(
            num_features, eps, momentum, affine, track_running_stats)

    def forward(self, input):
        self._check_input_dim(input)

        exponential_average_factor = 0.0

        if self.training and self.track_running_stats:
            if self.num_batches_tracked is not None:
                self.num_batches_tracked += 1
                if self.momentum is None:  # use cumulative moving average
                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)
                else:  # use exponential moving average
                    exponential_average_factor = self.momentum

        # calculate running estimates
        if self.training:
            mean = input.mean([0, 2, 3])
            # use biased var in train
            var = input.var([0, 2, 3], unbiased=False)
            n = input.numel() / input.size(1)
            with torch.no_grad():
                self.running_mean = exponential_average_factor * mean\
                    + (1 - exponential_average_factor) * self.running_mean
                # update running_var with unbiased var
                self.running_var = exponential_average_factor * var * n / (n - 1)\
                    + (1 - exponential_average_factor) * self.running_var
        else:
            mean = self.running_mean
            var = self.running_var

        input = (input - mean[None, :, None, None]) / (torch.sqrt(var[None, :, None, None] + self.eps))
        if self.affine:
            input = input * self.weight[None, :, None, None] + self.bias[None, :, None, None]

        return input

The outputs of nn.BatchNorm2d(2)(a) and MyBatchNorm2d(2)(a) are same.


Instruction: CUDA out of memory error, cannot reduce batch size

Input:
I want to run some experiments on my GPU device, but I get this error:

RuntimeError: CUDA out of memory. Tried to allocate 3.63 GiB (GPU 0;
15.90 GiB total capacity; 13.65 GiB already allocated; 1.57 GiB free; 13.68 GiB reserved in total by PyTorch)

I read about possible solutions here, and the common solution is this:

It is because of mini-batch of data does not fit onto GPU memory.
Just decrease the batch size. When I set batch size = 256 for cifar10
dataset I got the same error; Then I set the batch size = 128, it is
solved.

But in my case, it is a research project, and I want to have specific hyper-parameters and I can not reduce anything such as batch size.
Does anyone have a solution for this?

Output:
As long as a single sample can fit into GPU memory, you do not have to reduce the effective batch size: you can do gradient accumulation.
Instead of updating the weights after every iteration (based on gradients computed from a too-small mini-batch) you can accumulate the gradients for several mini-batches and only when seeing enough examples, only then updating the weights.
This is nicely explained in this video.
Effectively, your training code would look something like this.
Suppose your large batch size is large_batch, but can only fit small_batch into GPU memory, such that large_batch = small_batch * k.
Then you want to update the weights every k iterations:
train_data = DataLoader(train_set, batch_size=small_batch, ...)

opt.zero_grad()  # this signifies the start of a large_batch
for i, (x, y) in train_data:
  pred = model(x)
  loss = criterion(pred, y)
  loss.backward()  # gradeints computed for small_batch
  if (i+1) % k == 0 or (i+1) == len(train_data):
    opt.step()  # update the weights only after accumulating k small batches
    opt.zero_grad()  # reset gradients for accumulation for the next large_batch


Instruction: How can I separate the last layer of deep network in pytorch?

Input:
My deep network is:
self.actor = nn.Sequential(
   nn.Linear(state_dim, 256),
   nn.Softplus(),
   nn.Linear(256, 256),
   nn.Softplus(),
   nn.Linear(256, action_dim),
   nn.Softplus())

Now, I would like the network to give two separate outputs, like this:

That's to say, only the last layer of the network is different and maybe the last layer has different activation functions.
How should I change my code above?

Output:
The model you want to build is not sequential anymore, since there are two parallel branches at the end. You can keep the common trunk and separate with two additional separate layers. Something like:
class Model(nn.Module):
    def __init__(self):
        super.__init__()

        self.actor = nn.Sequential(
            nn.Linear(state_dim, 256),
            nn.Softplus(),
            nn.Linear(256, 256),
            nn.Softplus())

        self.outA = nn.Sequential(
            nn.Linear(256, action_dim),
            nn.Softplus())

        self.outB = nn.Sequential(
            nn.Linear(256, action_dim),
            nn.Softplus())

    def forward(self, x):
      features = self.actor(x)
      return self.outA(features), self.outB(features)


Instruction: PyTorch: Dogs vs Cat dataset with datasets.ImageFolder

Input:
I am new on PyTorch trying to create a TransferLearning model.
I am using dogs vs cats dataset from Kaggle.
I am using ImageFolder to load the data and it requires a folder for each classes. But the photos in the test folder are mixed.So I'm not able to separate the images on the test folder. What can I do to solve the problem apart from labeling all the test data with my hand?

Output:
You can create a custom Dataset class and wrap it inside a dataloader in Pytorch.
This link has great information on this topic
An overall structure to follow is
class Dog_and_Cat():
  def __init__(self, ...):
    ... # replace with a zipped list of image paths and labels (Cat or Dog)
    # You can use glob.glob
    # Overall ask how do I know the label of the image and install that reasoning in code
    # result is a zipped list like [("img1.jpg", 0), ("img2.jpg", 1)] where 0 and 1 represent Cat and Dog

  def __getitem__(self, idx):
    #describe the func when indexing your class. This is where you open your image and do transforms and return it.

  def __len__(self):
   # This is where you describe the length of the dataset.
   # Ideally should return len of zipped list mentioned above.

You can init this class you just made instead of Image Folder and then put an object of that class inside the dataloader. This will solve your task
Sarthak


Instruction: How to adjust the learning rate after N number of epochs?

Input:
I am using Hugginface's Trainer.
How to adjust the learning rate after N number of epochs?
For example, I have an initial learning rate set to lr=2e-6, and I would like to change the learning rate to lr=1e-6 after the first epoch and stay on it the rest of the training.
I tried this so far:
optimizer = AdamW(model.parameters(),
              lr = 2e-5,
              eps = 1e-8
            )

epochs = 5
batch_number = len(small_train_dataset) / 8
total_steps = batch_number * epochs

scheduler = get_linear_schedule_with_warmup(optimizer,
                                            num_warmup_steps = 0,
                                            num_training_steps = total_steps,
                                            last_epoch=-1
                                            )

I know that there is https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LambdaLR.html#torch.optim.lr_scheduler.LambdaLR but here it drops learning rate every epoch but that is not what i want to do. I want it to drop after 1 epoch and then stay on it rest of the training process.

Output:
You could train in two steps,
first, train with desired initial learning rate then create a second optimizer with the final learning rate. It is equivalent.


Instruction: Pytorch cuda is unavailable even installed CUDA and pytorch with cuda. How to fix?

Input:
I'm trying to use pytorch with my GPU (RTX 3070) on my Windows machine using WSL2, but I couldn't get it work even though I followed the Nvidia guide (https://docs.nvidia.com/cuda/wsl-user-guide/index.html#abstract).
nvidia-smi.exe output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 471.41       Driver Version: 471.41       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:0A:00.0  On |                  N/A |
|  0%   40C    P5    12W / 220W |   1815MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1384    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      1628    C+G   ...dows\System32\WWAHost.exe    N/A      |
|    0   N/A  N/A      3172    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A      5940    C+G   ...lPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A      6360    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A      7280    C+G   ...artMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A      7732    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A      8256    C+G   ...ekyb3d8bbwe\YourPhone.exe    N/A      |
|    0   N/A  N/A      8780    C+G   ...nputApp\TextInputHost.exe    N/A      |
|    0   N/A  N/A     10032    C+G   ...perience\NVIDIA Share.exe    N/A      |
|    0   N/A  N/A     10732    C+G   ...hyper\app-3.0.2\Hyper.exe    N/A      |
|    0   N/A  N/A     10804    C+G   ...kyb3d8bbwe\Calculator.exe    N/A      |
|    0   N/A  N/A     10852    C+G   ...in7x64\steamwebhelper.exe    N/A      |
|    0   N/A  N/A     12180    C+G   ...ge\Application\msedge.exe    N/A      |
|    0   N/A  N/A     13880    C+G   ...icrosoft VS Code\Code.exe    N/A      |
|    0   N/A  N/A     14724    C+G   ...b3d8bbwe\WinStore.App.exe    N/A      |
+-----------------------------------------------------------------------------+

nvcc -v output:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:15:15_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0

dpkg -l | grep nvidia output:
ii  libnvidia-cfg1-470:amd64             470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-470                 470.57.02-0ubuntu0.20.04.1        all          Shared files used by the NVIDIA libraries
rc  libnvidia-compute-450:amd64          450.51.05-0ubuntu1                amd64        NVIDIA libcompute package
rc  libnvidia-compute-465:amd64          465.27-0ubuntu0.20.04.2           amd64        NVIDIA libcompute package
ii  libnvidia-compute-470:amd64          470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA libcompute package
ii  libnvidia-decode-470:amd64           470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-470:amd64           470.57.02-0ubuntu0.20.04.1        amd64        NVENC Video Encoding runtime library
ii  libnvidia-extra-470:amd64            470.57.02-0ubuntu0.20.04.1        amd64        Extra libraries for the NVIDIA driver
ii  libnvidia-fbc1-470:amd64             470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-470:amd64               470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries
and Vulkan ICD
ii  libnvidia-ifr1-470:amd64             470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  libnvidia-ml-dev                     10.1.243-3                        amd64        NVIDIA Management Library (NVML) development files
rc  nvidia-compute-utils-450             450.51.05-0ubuntu1                amd64        NVIDIA compute utilities
ii  nvidia-compute-utils-470             470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA compute utilities
ii  nvidia-cuda-dev                      10.1.243-3                        amd64        NVIDIA CUDA development files
ii  nvidia-cuda-doc                      10.1.243-3                        all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                      10.1.243-3                        amd64        NVIDIA CUDA Debugger (GDB)
ii  nvidia-cuda-toolkit                  10.1.243-3                        amd64        NVIDIA CUDA development toolkit
rc  nvidia-dkms-450                      450.51.05-0ubuntu1                amd64        NVIDIA DKMS package
ii  nvidia-dkms-470                      470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA DKMS package
ii  nvidia-driver-470                    470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA driver metapackage
rc  nvidia-kernel-common-450             450.51.05-0ubuntu1                amd64        Shared files used with the kernel module
ii  nvidia-kernel-common-470             470.57.02-0ubuntu0.20.04.1        amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-470             470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA kernel source package
ii  nvidia-opencl-dev:amd64              10.1.243-3                        amd64        NVIDIA OpenCL development files
ii  nvidia-prime                         0.8.16~0.20.04.1                  all          Tools to enable NVIDIA's Prime
ii  nvidia-profiler                      10.1.243-3                        amd64        NVIDIA Profiler for CUDA and OpenCL
ii  nvidia-settings                      450.51.05-0ubuntu1                amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-470                     470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA driver support binaries
ii  nvidia-visual-profiler               10.1.243-3                        amd64        NVIDIA Visual Profiler for CUDA and OpenCL
ii  screen-resolution-extra              0.18build1                        all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia-470        470.57.02-0ubuntu0.20.04.1        amd64        NVIDIA binary Xorg driver

dpkg -l | grep cuda output:
ii  cuda                                 11.4.0-1                          amd64        CUDA meta-package
ii  cuda-11-4                            11.4.0-1                          amd64        CUDA 11.4 meta-package
ii  cuda-cccl-11-4                       11.4.43-1                         amd64        CUDA CCCL
ii  cuda-command-line-tools-11-4         11.4.0-1                          amd64        CUDA command-line tools
ii  cuda-compiler-11-4                   11.4.0-1                          amd64        CUDA compiler
rc  cuda-cudart-11-0                     11.0.194-1                        amd64        CUDA Runtime native Libraries
ii  cuda-cudart-11-4                     11.4.43-1                         amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-11-4                 11.4.43-1                         amd64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-11-4                  11.4.43-1                         amd64        CUDA cuobjdump
ii  cuda-cupti-11-4                      11.4.65-1                         amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-11-4                  11.4.65-1                         amd64        CUDA profiling tools interface.
ii  cuda-cuxxfilt-11-4                   11.4.43-1                         amd64        CUDA cuxxfilt
ii  cuda-demo-suite-11-4                 11.4.43-1                         amd64        Demo suite for CUDA
ii  cuda-documentation-11-4              11.4.43-1                         amd64        CUDA documentation
ii  cuda-driver-dev-11-4                 11.4.43-1                         amd64        CUDA Driver native dev stub library
ii  cuda-gdb-11-4                        11.4.55-1                         amd64        CUDA-GDB
ii  cuda-libraries-11-4                  11.4.0-1                          amd64        CUDA Libraries 11.4 meta-package
ii  cuda-libraries-dev-11-4              11.4.0-1                          amd64        CUDA Libraries 11.4 development meta-package
ii  cuda-memcheck-11-4                   11.4.43-1                         amd64        CUDA-MEMCHECK
ii  cuda-nsight-11-4                     11.4.43-1                         amd64        CUDA nsight
ii  cuda-nsight-compute-11-4             11.4.0-1                          amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-11-4             11.4.0-1                          amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-11-4                       11.4.48-1                         amd64        CUDA nvcc
ii  cuda-nvdisasm-11-4                   11.4.43-1                         amd64        CUDA disassembler
ii  cuda-nvml-dev-11-4                   11.4.43-1                         amd64        NVML native dev links, headers
ii  cuda-nvprof-11-4                     11.4.43-1                         amd64        CUDA Profiler tools
ii  cuda-nvprune-11-4                    11.4.43-1                         amd64        CUDA nvprune
ii  cuda-nvrtc-11-4                      11.4.50-1                         amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-11-4                  11.4.50-1                         amd64        NVRTC native dev links, headers
ii  cuda-nvtx-11-4                       11.4.43-1                         amd64        NVIDIA Tools Extension
ii  cuda-nvvp-11-4                       11.4.43-1                         amd64        CUDA Profiler tools
ii  cuda-repo-ubuntu2004-11-0-local      11.0.2-450.51.05-1                amd64        cuda repository configuration files
ii  cuda-repo-wsl-ubuntu-11-4-local      11.4.0-1                          amd64        cuda repository configuration files
ii  cuda-runtime-11-4                    11.4.0-1                          amd64        CUDA Runtime 11.4 meta-package
ii  cuda-samples-11-4                    11.4.43-1                         amd64        CUDA example applications
ii  cuda-sanitizer-11-4                  11.4.54-1                         amd64        CUDA Sanitizer
rc  cuda-toolkit-11-0                    11.0.2-1                          amd64        CUDA Toolkit 11.0 meta-package
ii  cuda-toolkit-11-4                    11.4.0-1                          amd64        CUDA Toolkit 11.4 meta-package
ii  cuda-toolkit-11-4-config-common      11.4.43-1                         all          Common config package for CUDA Toolkit 11.4.
ii  cuda-toolkit-11-config-common        11.4.43-1                         all          Common config package for CUDA Toolkit 11.
ii  cuda-toolkit-config-common           11.4.43-1                         all          Common config package for CUDA Toolkit.
ii  cuda-tools-11-4                      11.4.0-1                          amd64        CUDA Tools meta-package
rc  cuda-visual-tools-11-0               11.0.2-1                          amd64        CUDA visual tools
ii  cuda-visual-tools-11-4               11.4.0-1                          amd64        CUDA visual tools
ii  libcudart10.1:amd64                  10.1.243-3                        amd64        NVIDIA CUDA Runtime Library
ii  nvidia-cuda-dev                      10.1.243-3                        amd64        NVIDIA CUDA development files
ii  nvidia-cuda-doc                      10.1.243-3                        all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                      10.1.243-3                        amd64        NVIDIA CUDA Debugger (GDB)
ii  nvidia-cuda-toolkit                  10.1.243-3                        amd64        NVIDIA CUDA development toolkit

The pytorch version I installed:
torch             1.9.0+cu111
torchaudio        0.9.0
torchvision       0.10.0+cu111

But pytorch tells me that cuda is not available:
Python 3.8.10 (default, Jun  2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False

Does anyone know if I installed all the packages I need? I tried many solutions posted on the internet but still doesn't work.

Output:
My environment is (Ubuntu 20.04 with NVIDIA GTX 1080Ti):
$ nvidia-smi | grep CUDA
| NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Aug_15_21:14:11_PDT_2021
Cuda compilation tools, release 11.4, V11.4.120
Build cuda_11.4.r11.4/compiler.30300941_0

After the installation of CUDA and cuDNN, I run this followwing command getting from https://pytorch.org/get-started/locally/ with options (PyTorch Stable 1.9.1, Linux, Pip, Python, CUDA 11.1).
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

Find packages:
(test-py38) ➜  ~ conda list | grep torch
_pytorch_select           0.1                       cpu_0    defaults
pytorch                   1.8.1           cpu_py38h60491be_0    defaults
torch                     1.9.1+cu111               <pip>
torchaudio                0.9.1                     <pip>
torchvision               0.10.1+cu111              <pip>

Then:
$ python
Python 3.8.11 (default, Aug  3 2021, 15:09:35)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True


Instruction: AttributeError: 'LSTMClassifier' object has no attribute 'log_softmax'

Input:
While making predictions from my LSTM model, I am getting the error :: AttributeError: 'LSTMClassifier' object has no attribute 'log_softmax'.Can anyone explain me what I am doing wrong?
class LSTMClassifier(nn.Module):

    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.batch_size = None
        self.hidden = None

    def forward(self, x):
        h0, c0 = self.init_hidden(x)
        out, (hn, cn) = self.lstm(x, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        print(x.size(0))
        print(layer_dim)
        return [t.to(device) for t in (h0, c0)]

test_dl = DataLoader(tst_data, batch_size=64, shuffle=False)
test = []
print('Predicting on test dataset')
for batch, _ in tst_data:
    batch=batch.to(device)
    print(batch.shape)
    out = model.to(device)
    y_hat = F.log_softmax(out, dim=1).argmax(dim=1) ### at this line I am getting error
    test += y_hat.tolist()

Thank you in advance!
Error :: AttributeError: 'LSTMClassifier' object has no attribute 'log_softmax'
Traceback:::
AttributeError                            Traceback (most recent call last)
<ipython-input-74-df6f970f9b87> in <module>()
      8     print(batch.shape)
      9     out = model.to(device)
---> 10     y_hat = F.log_softmax(out, dim=1).argmax(dim=1)
     11
     12     test += y_hat.tolist()

1 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1129                 return modules[name]
   1130         raise AttributeError("'{}' object has no attribute '{}'".format(
-> 1131             type(self).__name__, name))
   1132
   1133     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'LSTMClassifier' object has no attribute 'log_softmax'

Output:
Your train loop does not work. You never pass the input batch to the model therefore out is not a output tensor but a model object which of course can't be passed into an activation function.
You have to do this:
model = model.to(device)
for batch, _ in tst_data:
    batch = batch.to(device)

    # pass your input batch to the model like this
    out = model.train()(batch)

    # now you can calculate the log-softmax for out
    y_hat = F.log_softmax(out, dim=1).argmax(dim=1)
    test += y_hat.tolist()


Instruction: How can we use Pytorch Autograd for sequence optimization (in a for loop)?

Input:
I want to optimize a sequence in a for loop using Pytorch Autograd. I am using LBFGS.
loss = 0.0
for i in range(10):
     x = f(x,z[i])
     loss = loss + mse_loss(x,x_GT)

Say the sequence length is 10. I want to optimize x as well as z(z is a tensor array), these are learnable parameters.  Note the x will be updated in the loop.
x_GT is ground truth data.
To run this, I have to open:
loss.backward(retain_graph=True)

Is there a better way to do so (To make it run faster)?

Output:
The code you provided is actually perfectly fine:
loss = torch.zeros(1)
for i in range(10):
     x = f(x, z[i])
     loss += mse_loss(x, x_GT)

It will accumulate the loss over the loop steps. The backward pass only needs to be called once, though, so you are not required to retain the graph on it:
>>> loss.backward()

I don't believe not retaining the graph will make your code run any faster. It only adds to the memory load since it has to save all activations on the graph, expecting a second backward pass to come.


Instruction: Use MS-COCO format as input to PyTorch MASKRCNN

Input:
I am trying to train a MaskRCNN Image Segmentation model with my custom dataset in MS-COCO format.
I am trying to use the polygon masks as the input but cannot get it to fit the format for my model.
My data looks like this:
{"id": 145010,
"image_id": 101953,
"category_id": 1040,
"segmentation": [[140.0, 352.5, 131.0, 351.5, 118.0, 344.5, 101.50000000000001, 323.0, 94.5, 303.0, 86.5, 292.0, 52.0, 263.5, 35.0, 255.5, 20.5, 240.0, 11.5, 214.0, 14.5, 190.0, 22.0, 179.5, 53.99999999999999, 170.5, 76.0, 158.5, 88.5, 129.0, 100.5, 111.0, 152.0, 70.5, 175.0, 65.5, 217.0, 64.5, 272.0, 48.5, 296.0, 56.49999999999999, 320.5, 82.0, 350.5, 135.0, 374.5, 163.0, 382.5, 190.0, 381.5, 205.99999999999997, 376.5, 217.0, 371.0, 221.5, 330.0, 229.50000000000003, 312.5, 240.0, 310.5, 291.0, 302.5, 310.0, 288.0, 326.5, 259.0, 337.5, 208.0, 339.5, 171.0, 349.5]],
"area": 73578.0,
"bbox": [11.5, 11.5, 341.0, 371.0],
"iscrowd": 0}
I have one object in this image, hence one item for segmentation and bbox. Segmentation values are the pixels of the polygon, hence have different sizes for different objects.
Could anyone help me with this?

Output:
To manage COCO formated datasets you can use this repo. It gives classes which you can instantiate from you annotation's file making it really easy to use and to access the data.
I don't know which implementation you are using, but if it's something like this tutorial, this piece of code might give you at least some ideas on how to solve your problem:
class CocoDataset(torch.utils.data.Dataset):
def __init__(self, dataset_dir, subset, transforms):
    dataset_path = os.path.join(dataset_dir, subset)
    ann_file = os.path.join(dataset_path, "annotation.json")
    self.imgs_dir = os.path.join(dataset_path, "images")
    self.coco = COCO(ann_file)
    self.img_ids = self.coco.getImgIds()

    self.transforms = transforms

def __getitem__(self, idx):
    '''
    Args:
        idx: index of sample to be fed
    return:
        dict containing:
        - PIL Image of shape (H, W)
        - target (dict) containing:
            - boxes:    FloatTensor[N, 4], N being the n° of instances and it's bounding
            boxe coordinates in [x0, y0, x1, y1] format, ranging from 0 to W and 0 to H;
            - labels:   Int64Tensor[N], class label (0 is background);
            - image_id: Int64Tensor[1], unique id for each image;
            - area:     Tensor[N], area of bbox;
            - iscrowd:  UInt8Tensor[N], True or False;
            - masks:    UInt8Tensor[N, H, W], segmantation maps;
    '''
    img_id = self.img_ids[idx]
    img_obj = self.coco.loadImgs(img_id)[0]
    anns_obj = self.coco.loadAnns(self.coco.getAnnIds(img_id))

    img = Image.open(os.path.join(self.imgs_dir, img_obj['file_name']))

    # list comprhenssion is too slow, might be better changing it
    bboxes = [ann['bbox'] for ann in anns_obj]
    # bboxes = ? from [x, y, w, h] to [x0, y0, x1, y1]
    masks = [self.coco.annToMask(ann) for ann in anns_obj]
    areas = [ann['area'] for ann in anns_obj]

    boxes = torch.as_tensor(bboxes, dtype=torch.float32)
    labels = torch.ones(len(anns_obj), dtype=torch.int64)
    masks = torch.as_tensor(masks, dtype=torch.uint8)
    image_id = torch.tensor([idx])
    area = torch.as_tensor(areas)
    iscrowd = torch.zeros(len(anns_obj), dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["masks"] = masks
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
        img, target = self.transforms(img, target)
    return img, target

def __len__(self):
    return len(self.img_ids)

Once again, this is just a draft and meant to give tips.


Instruction: Parameters for LSTM with CNN in a sequential data

Input:
I am doing a classification problem with ECG data. I built a LSTM model but the accuracy of the model is not quiet good. Hence, I am thinking to implement it with CNN. I am planning to pass the data from CNN, then passing the output from CNN to LSTM. Howver, I have noticed that CNN is mostly used in Image classifications. I have sequential data with 4000 time steps. Could you please help me to define the parameters of the CNN model.
Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)

Can someone explain me what would be the in_channels, out_channels, kernel_size and stride for a sequence data having 4000 time steps?

Output:
Well, that's yours to define, it's the actual architectural decision your need to take to construct your model. The following is not a solution to your question, however, this might give you some ideas.

You could pass each timestep through the CNN and retrieve a sequence of feature vectors corresponding to the CNN's outputs at consecutive time steps. Your CNN input would be shaped as (batch_size, channel, height, width) and output something like (batch_size, feature_length). Stacking the timesteps results would give you (batch_size, sequence_length, feature_length).

You could use a 3D convolutional layer, in that case, you can work straight away with shape (batch_size, sequence_length, channel, height, width). This is much more computation-intensive, since you are already planning on using an LSTM, it might be a little over-complex.

The number of channels, kernel sizes, number of filters in each convolutional layer isn't really an obvious question. You need to decide on that based on your setup: how large is your dataset, how many classes you have, how complex is the task (if not a classification problem).
My best advice is you start by using a well-known CNN architecture such as VGG or ResNet and work from there. Better yet, look at the literature and see if some else has ever faced that problem, you will most likely find interesting ideas that will help shape your project.


Instruction: Error with pytorch compilation: LAPACK library not found in compilation. How to solve?

Input:
I am already desperate about this problem that I am having.
RuntimeError: inverse: LAPACK library not found in compilation

The easiest way to reproduce it is:
import torch
A = torch.rand(5,5)
torch.inverse(A)

I run this inside a docker container. The part of the dockerfile that compiles pytorch is:
#PyTorch
RUN pip3 install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses

ENV PYTORCH_INST_VERSION="v1.8.1"
RUN git clone --recursive --branch ${PYTORCH_INST_VERSION} https://github.com/pytorch/pytorch pytorch-src && \
    cd pytorch-src && \
    export MAX_JOBS=$((`nproc` - 2)) && \
    export TORCH_CUDA_ARCH_LIST=${CUDA_ARCH} && \
    python3 setup.py install --prefix=/opt/pytorch && \
    cp -r /opt/pytorch/lib/python3.8/site-packages/* /usr/lib/python3/dist-packages/ && \
    cd /opt && \
    rm -rf /opt/pytorch-src

I am not super experienced so I don't know if I need to provide additional details. Please tell me if so.

Output:
I solved my own problem.
I added apt-get liblapack-dev on the dockerfile before the torch compilation. Then I runned the docker container again and it worked.


Instruction: How to fix input and parameter tensors are not at the same device?

Input:
I have seen other people have this error and I try to follow the steps to resolve, but continue to receive this error. "RuntimeError: Input and parameter tensors are not at the same device, found input tensor at cpu and parameter tensor at cuda:0"
I run both model.to(device) and input_seq.to(device). Error says it found an input tensor on CPU, but all input data should be on GPU with input_seq.to(device). Below is fill code
text = ['hey how are you','good i am fine','have a nice day']

# Join all the sentences together and extract the unique characters from the combined sentences
chars = set(''.join(text))

# Creating a dictionary that maps integers to the characters
int2char = dict(enumerate(chars))

# Creating another dictionary that maps characters to integers
char2int = {char: ind for ind, char in int2char.items()}

# Finding the length of the longest string in our data
maxlen = len(max(text, key=len))

# Padding

# A simple loop that loops through the list of sentences and adds a ' ' whitespace until the length of
# the sentence matches the length of the longest sentence
for i in range(len(text)):
  while len(text[i])<maxlen:
      text[i] += ' '

# Creating lists that will hold our input and target sequences
input_seq = []
target_seq = []

for i in range(len(text)):
    # Remove last character for input sequence
  input_seq.append(text[i][:-1])

    # Remove first character for target sequence
  target_seq.append(text[i][1:])
  print("Input Sequence: {}\nTarget Sequence: {}".format(input_seq[i], target_seq[i]))

for i in range(len(text)):
    input_seq[i] = [char2int[character] for character in input_seq[i]]
    target_seq[i] = [char2int[character] for character in target_seq[i]]

dict_size = len(char2int)
seq_len = maxlen - 1
batch_size = len(text)

def one_hot_encode(sequence, dict_size, seq_len, batch_size):
    # Creating a multi-dimensional array of zeros with the desired output shape
    features = np.zeros((batch_size, seq_len, dict_size), dtype=np.float32)

    # Replacing the 0 at the relevant character index with a 1 to represent that character
    for i in range(batch_size):
        for u in range(seq_len):
            features[i, u, sequence[i][u]] = 1
    return features

# Input shape --> (Batch Size, Sequence Length, One-Hot Encoding Size)
input_seq = one_hot_encode(input_seq, dict_size, seq_len, batch_size)

input_seq = torch.from_numpy(input_seq)
target_seq = torch.Tensor(target_seq)

# torch.cuda.is_available() checks and returns a Boolean True if a GPU is available, else it'll return False
is_cuda = torch.cuda.is_available()

# If we have a GPU available, we'll set our device to GPU. We'll use this device variable later in our code.
if is_cuda:
    device = torch.device("cuda")
    print("GPU is available")
else:
    device = torch.device("cpu")
    print("GPU not available, CPU used")

class Model(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(Model, self).__init__()

        # Defining some parameters
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        #Defining the layers
        # RNN Layer
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        # Fully connected layer
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x):

        batch_size = x.size(0)

        # Initializing hidden state for first input using method defined below
        hidden = self.init_hidden(batch_size)

        # Passing in the input and hidden state into the model and obtaining outputs
        out, hidden = self.rnn(x, hidden)

        # Reshaping the outputs such that it can be fit into the fully connected layer
        out = out.contiguous().view(-1, self.hidden_dim)
        out = self.fc(out)

        return out, hidden

    def init_hidden(self, batch_size):
        # This method generates the first hidden state of zeros which we'll use in the forward pass
        # We'll send the tensor holding the hidden state to the device we specified earlier as well
        hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
        return hidden

# Instantiate the model with hyperparameters
model = Model(input_size=dict_size, output_size=dict_size, hidden_dim=12, n_layers=1)
# We'll also set the model to the device that we defined earlier (default is CPU)
model.to(device)

# Define hyperparameters
n_epochs = 100
lr=0.01

# Define Loss, Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

# Training Run
for epoch in range(1, n_epochs + 1):
    optimizer.zero_grad() # Clears existing gradients from previous epoch
    input_seq.to(device)
    target_seq.to(device)
    output, hidden = model(input_seq)
    loss = criterion(output, target_seq.view(-1).long())
    loss.backward() # Does backpropagation and calculates gradients
    optimizer.step() # Updates the weights accordingly

    if epoch%10 == 0:
        print('Epoch: {}/{}.............'.format(epoch, n_epochs), end=' ')
        print("Loss: {:.4f}".format(loss.item()))

Output:
Unlike the to method available on nn.Modules such as your model. The to method on Tensors is not an in-place operation! As stated on the documentation page:

This method [nn.Module.to] modifies the module in-place.

vs for Tensor.to:

[...] the returned tensor is a copy of self with the desired [...] torch.device.

In other words, you need to reassign the tensors in order to effectively send them to the device.
input_seq = input_seq.to(device)
target_seq = target_seq.to(device)

While an nn.Module won't need this treatment:
model.to(device)

To clearly understand what happens here, take this example:
>>> x = torch.zeros(1)  # on cpu
>>> y = x.cuda()        # y is a copy of x

>>> y.device            # placed on cuda device
'cuda:0'

>>> x.device            # but x remains on the original device
'cpu'


Instruction: Difference between transformers schedulers and Pytorch schedulers

Input:
Transformers also provide their own schedulers for learning rates like get_constant_schedule, get_constant_schedule_with_warmup, etc. They are again returning torch.optim.lr_scheduler.LambdaLR (torch scheduler). Is the warmup_steps the only difference between the two?
How can we create a custom transformer-based scheduler similar to other torch schedulers like lr_scheduler.MultiplicativeLR, lr_scheduler.StepLR, lr_scheduler.ExponentialLR?

Output:
You can create a custom scheduler by just creating a function in a class that takes in an optimizer and its state dicts and edits the values in its param_groups.
To understand how to structure this in a class, just take a look at how Pytorch creates its schedulers and use the same functions just change the functionality to your liking.
The Permalink I found that will be a good reference is over here
EDIT After comments:
This is like a template you can use
from torch.optim import lr_scheduler

class MyScheduler(lr_scheduler._LRScheduler # Optional inheritance):
    def __init__(self, # optimizer, epoch, step size, whatever you need as input to lr scheduler, you can even use vars from LRShceduler Class that you can inherit from etc.):
      super(MyScheduler, self).__init__(optimizer, last_epoch, verbose)
      # Put variables that you will need for updating scheduler like gamma, optimizer, or step size etc.
      self.optimizer = optimizer

    def get_lr(self):
       # How will you use the above variables to update the optimizer
       for group in self.optimizer.param_groups:
          group["lr"] = # Fill this out with updated optimizer

       return self.optimizer

You can add more functions for increased functionality. Or you can just use a function to update your learning rate. This will take in the optimizer and change it optimizer.param_groups[0]["lr"] and return the new optimizer.


Instruction: Summing vector pairs efficiently in pytorch

Input:
I'm trying to calculate the summation of each pair of rows in a matrix. Suppose I have an m x n matrix, say one like
[[1,2,3],
 [4,5,6],
 [7,8,9]]

and I want to create a matrix of the summations of all pairs of rows. So, for the above matrix, we would want
[[5,7,9],
 [8,10,12],
 [11,13,15]]

In general, I think the new matrix will be (m choose 2) x n. For the above example in pytorch, I ran
import torch

x = torch.tensor([[1,2,3], [4,5,6], [7,8,9]])

y = x[None] + x[:, None]

torch.cat((y[0, 1:3, :], y[1, 2:3, :]))

which manually creates the matrix I am looking for. However, I am struggling to think of a way to create the output without manually specifying indices and without using a for-loop. Is there even a way to create such a matrix for an arbitrary matrix without the use of a for-loop?

Output:
You can try using this function:
def sum_rows(x):
   y = x[None] + x[:, None]
   ind = torch.tril_indices(x.shape[0], x.shape[0], offset=-1)
   return y[ind[0], ind[1]]

Because you know you want pairs with the constraints of  sum_matrix[i,j], where i<j (but i>j would also work), you can just specify that you want the lower/upper triangle indices of your 3D matrix. This still uses a for loop, AFAIK, but should do the job for variable-sized inputs.


Instruction: RTX 3070 compatibility with Pytorch

Input:
NVIDIA GeForce RTX 3070 with CUDA capability sm_86 is not compatible
with the current PyTorch installation. The current PyTorch install
supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

So I'm currently trying to train a neural network but I'm getting this issue. It seems that the GPU model I have is not compatible with the version of PyTorch that I have.
The output of my nvcc -V is:
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0

and my PyTorch version is 1.9.0.
I've tried changing the CUDA version from what was initially 10 to 11.4 and there was no change. Any help would be hugely appreciated.

Output:
It might be because you have installed a torch package with cuda==10.* (e.g. torch==1.9.0+cu102) . I'd suggest trying:
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Forcing Ration on Batches in PyTorch DataLoader

Input:
I came across a binary classifcation task where the data is heavily imbalanced. (I'm looking at 80:1)
Through undersampling, the data ratio is at now 20:1.\
Now, the undersampled/treated data is loaded in to dataloader as below. (this is an nlp task)
train_inputs = torch.tensor(input_ids)
train_labels = torch.tensor(labels)
train_masks = torch.tensor(attention_masks)

train_data = TensorDataset(train_inputs, train_masks, train_labels)
if is_distributed:
    train_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
else:
    train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

For each batch, I want to make sure/enforce that each batch has the ration of 20:1 in terms of each classifications. Is there a pytorch builtin method that allows me to enforce such conditions ?

Output:
You can use WeightedRandomSampler with replacement set to true.
Just multiply the weight of the positive examples by 20 to bias the ratio towards 20:1. (assuming positives are 20 times more than negatives here)
# labels is a numpy array of shape n,1 containing 1 and 0 for each datapoint
weights = np.ones(labels.shape)
weights[labels==1] *= 20
# now the samples will be drawn with a 20:1 ratio
sampler = WeightedRandomSampler(weights=weights, num_samples=len(labels), replacement=True)

Note that setting replacement to True is necessary to enforce each batch to be sampled with the ratio but you might see an example twice during training.
However if the ratio is naturally occurring in your dataset you can leave it to False.
IMHO:
Random sampling do not ensure a 20:1 ratio for each batch but only for an expected value of the ratio (on average the ratio of the batch(sample) will tend to the ratio of the dataset(population)) so it should not impact the average gradient as well during training (in theory). But I see how in practice you might want to have more control.


Instruction: "SyntaxError: Can't assign to operator"; "ipykernel_launcher.py: error: unrecognized arguments"

Input:
I want to execute a train.py script inside a colab or jupyter notebook.
Before running I have to set some variables, e.g. dataset-type.
I did as I would type into a terminal command but I get a SyntaxError.
--dataset-type=voc

        --dataset-type='voc'
                            ^
    SyntaxError: can't assign to operator

Executing from a terminal works fine but how do I declare the variables correctly?
Here is some code of train.py:
parser = argparse.ArgumentParser(description='Single Shot MultiBox Detector Training With PyTorch')

# Params for datasets
parser.add_argument('--dataset-type', default="voc", type=str,
                    help='Specify dataset type. Currently supports voc and open_images.')
parser.add_argument('--datasets', '--data', nargs='+', default=["data"], help='Dataset directory path')
parser.add_argument('--balance-data', action='store_true',
                    help="Balance training data by down-sampling more frequent labels.")

I tried:
from sys import argv
argv.append('--dataset-type=voc')

This solved the SyntaxError.
But I get following error in the end. There are more variables to set, but --dataset-type is still listed.
usage: ipykernel_launcher.py [-h] [--dataset-type DATASET_TYPE]
                             [--datasets DATASETS [DATASETS ...]]
                             [--balance-data] [--net NET] [--freeze-base-net]
                             [--freeze-net] [--mb2-width-mult MB2_WIDTH_MULT]
                             [--base-net BASE_NET]
                             [--pretrained-ssd PRETRAINED_SSD]
                             [--resume RESUME] [--lr LR] [--momentum MOMENTUM]
                             [--weight-decay WEIGHT_DECAY] [--gamma GAMMA]
                             [--base-net-lr BASE_NET_LR]
                             [--extra-layers-lr EXTRA_LAYERS_LR]
                             [--scheduler SCHEDULER] [--milestones MILESTONES]
                             [--t-max T_MAX] [--batch-size BATCH_SIZE]
                             [--num-epochs NUM_EPOCHS]
                             [--num-workers NUM_WORKERS]
                             [--validation-epochs VALIDATION_EPOCHS]
                             [--debug-steps DEBUG_STEPS] [--use-cuda USE_CUDA]
                             [--checkpoint-folder CHECKPOINT_FOLDER]
ipykernel_launcher.py: error: unrecognized arguments: -f /root/.local/share/jupyter/runtime/kernel-f89b4ea1-0c84-4617-af88-0191a91639c0.json
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2
/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py:2890: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

Output:
It seems the model uses sys.argv, to assign them in a notebook:
from sys import argv
argv.append('--dataset-type=voc')

That should work the same as adding --dataset-type=voc in a terminal.


Instruction: How can I add multiple Metadata in Torch Tensorboard Embedding?

Input:
I am using torch==1.9.0 and tensorboard==2.5.0. I would like to track data with tensorbaord as an embedding, so I am something like this:
data = np.random.poisson(lam=10.0, size=(4,4))
labels = ["A","A","B","B"]
ids = [1,2,3,4]
writer = SummaryWriter("/runs/")
writer.add_embedding(data,
                    metadata=labels)
writer.close()

But I only can add lables or ids as metadeta and not a combined dictionaty {'ids':ids, 'lables':labels}. Any idea, how to solve this? Thanks!
FYI: Tensorboard docs just descirbes metadata as a list:
https://pytorch.org/docs/stable/tensorboard.html

Output:
Found the answer. You could get multiple field by adding a metadata header and give metadat as list of lists:
writer.add_embedding(data.values,
                    metadata=metadata,
                    metadata_header=["Name","Labels"])

Reference: https://github.com/tensorflow/tensorboard/issues/61


Instruction: Simple Adjacency matric creation with Pytorch Tensors

Input:
I was trying to write a simple function to create a random adjacency matrix in the following way :
def create_adj(a):
    a[a>0.5] = 1
    a[a<=0.5] = 0
    return a

given that a is assumed to be a torch.Tensor() as input, but I get the following error:
TypeError: 'int' object does not support item assignment

If I do things separately (i.e. not inside a function), I simply do:
>> a = torch.rand(3,3)
>> a[a>0.5] = 1
>> a[a<=0.5] = 0

>> a
 tensor([[1., 1., 1.],
         [0., 0., 0.],
         [1., 0., 0.]])

But I don't understand what I'm doing wrong in the function.

Output:
I would assume you are not passing the correct variable your create_adj function. As long as a is a torch.tensor, then it should work.
Alternatively, you can directly use the mask as result:
def create_adj(x):
  return (a > .5).float()


Instruction: How to Fix "AssertionError: CUDA unavailable, invalid device 0 requested"

Input:
I'm trying to use my GPU to run the YOLOR model, and I keep getting the error:
Traceback (most recent call last):
  File "D:\yolor\detect.py", line 198, in <module>
    detect()
  File "D:\yolor\detect.py", line 41, in detect
    device = select_device(opt.device)
  File "D:\yolor\utils\torch_utils.py", line 47, in select_device
    assert torch.cuda.is_available(), 'CUDA unavailable, invalid device %s requested' % device  # check availablity
AssertionError: CUDA unavailable, invalid device 0 requested

When I try to check if CUDA is available with the following:
python3
>>import torch
>>print(torch.cuda.is_available())

I get False, which explains the problem. I tried running the command
py -m pip install torch1.9.0+cu111 torchvision0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

I get the error: ERROR: Invalid requirement: 'torch1.9.0+cu111'
Running nvcc --version, I get:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:41:42_Pacific_Daylight_Time_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

Thus, I'm not really sure what the issue is, or how to fix it.
EDIT: As @Ivan pointed out, I added the == sign, but still get False when checking if CUDA is available.

Output:
You forgot to put the == signs between the packages and the version number. According to the PyTorch installation page:
py -m pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Pytorch unable to export trained model as ONNX

Input:
I have been training a model in the Pytorch framework using multiple convolutional layers (3x3, stride 1, padding same). The model performs well and I want to use it in Matlab for inference. For that, the ONNX format for NN exchange between frameworks seems to be the (only?) solution. The model can be exported using the following command:
torch.onnx.export(net.to('cpu'), test_input,'onnxfile.onnx')

Here is my CNN architecture definition:
class Encoder_decoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
        nn.Conv2d(2,8, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(8,8, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(8,16, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(16,16, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(16,32, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(32,32, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(32,64, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(64,64, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(64,128, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(128,128, (3, 3),stride = 1, padding='same'),
        nn.ReLU(),
        nn.Conv2d(128,1, (1, 1))
        )

    def forward(self, x):
        x = self.model(x)

        return x

However, when I run the torch.onnx.export command I get the following error:
RuntimeError: Exporting the operator _convolution_mode to ONNX opset version 9 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.

I have tried changing the opset, but that doesn't solve the problem. ONNX has full support for convolutional neural networks. Also, I am training the network in google colab.
Do you know other methods to transfer the model to matlab?

Output:
Currently, _convolution_mode operator isn't supported in pytorch. This is due to the use of padding='same'.
You need to change padding to an integer value or change it to its equivalent. Consult Same padding equivalent in Pytorch.


Instruction: How to fix RuntimeError CUDA error CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm`?

Input:
When training some models on a working cuda environment, you can get the error RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
What does it means and how to fix it?

Output:
It may be an incomplete error reporting of a shape error:
A mismatch in dimension of a nn.Linear module and its inpput, for example x.shape == [a, b] going into a nn.Linear(c, c, bias=False) with c not matching the shape of x, will result in this error message.
See the Pytorch forum conversation.


Instruction: Converting From .pt model to .h5 model

Input:
I am using Google colab. I want to convert .pt model from my google drive to .h5 model. I follow link https://github.com/gmalivenko/pytorch2keras and https://www.programmersought.com/article/57938937172/ and install libraries and also write code as below:
%pip install pytorch2keras
%pip install onnx==1.8.1

import numpy as np
from numpy import random
from random import uniform
import torch
from torch.autograd import Variable
input_np = np.random.uniform(0, 1, (1, 10, 32, 32))
input_var = Variable(torch.FloatTensor(input_np))
model='/content/gdrive/MyDrive/model.pt'
pytorch_to_keras(model,input_var,input_shapes= [(10, 32, 32,)], verbose=True)

but it gives me error like:
WARNING:pytorch2keras:Custom shapes isn't supported now.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-53-eef217a11c8a> in <module>()
      8 input_var = Variable(torch.FloatTensor(input_np))
      9 model='/content/gdrive/MyDrive/model.pt'
---> 10 pytorch_to_keras(model,input_np,input_shapes= [(10, 32, 32,)], verbose=True)
     11
     12

/usr/local/lib/python3.7/dist-packages/pytorch2keras/converter.py in pytorch_to_keras(model, args, input_shapes, change_ordering, verbose, name_policy, use_optimizer, do_constant_folding)
     51     args = tuple(args)
     52
---> 53     dummy_output = model(*args)
     54
     55     if isinstance(dummy_output, torch.autograd.Variable):

TypeError: 'str' object is not callable

Output:
Ah, the classic problem of PyTorch to Tensorflow. Many libraries have come and gone over the years, but I've found ONNX to work the most consistent. You could try something like this.
Specific to PyTorch is a Dynamic Computational Graph. A dynamic computational graph means that PyTorch models can dynamically adapt to different input sizes. You can specify which axes need dynamic sizing as such.
Here is some minimal code to convert a CNN from PyTorch to ONNX.
import onnx
import torch

model = get_model()
model.eval()

# Test model on sample image size
example_input = torch.randn((1, 3, img_size, img_size), requires_grad=True)
model(example_input)

# Set input and output names, include more names in the list if your model has more than 1 input/output
input_names = ["input0"]
output_names = ["output0"]

# Set dynamic axes (in this case, make the batch a dynamic dimension)
dynamic_axes = {'input0': {0: 'batch'}, 'output0': {0: 'batch'}}

# Export model with the above parameters
torch_out = torch.onnx.export(
    model, example_input, 'model.onnx', export_params=True, input_names=input_names, output_names=output_names,
    dynamic_axes=dynamic_axes, operator_export_type=torch.onnx.OperatorExportTypes.ONNX
)

# Use ONNX checker to verify integrity of model
onnx_model = onnx.load("model.onnx")
onnx.checker.check_model(onnx_model)

One could also set the height and width of the model as a dynamic input size with
dynamic_axes['input0'][2] = 'height'
dynamic_axes['input0'][3] = 'width'

Next, we convert our ONNX model to a Tensorflow SavedModel.
from onnx_tf.backend import prepare
import onnx

onnx_model = onnx.load('model.onnx')
tf_model = prepare(onnx_model)
tf_model.export_graph('tf_model')

tf_model is now a Tensorflow SavedModel.


Instruction: Loading PyTorch Lightning Trained checkpoint

Input:
I am using PyTorch Lightning version 1.4.0 and have defined the following class for the dataset:
class CustomTrainDataset(Dataset):
    '''
    Custom PyTorch Dataset for training

    Args:
        data (pd.DataFrame) - DF containing product info (and maybe also ratings)
        all_itemIds (list) - Python3 list containing all Item IDs
    '''

    def __init__(self, data, all_orderIds):
        self.users, self.items, self.labels = self.get_dataset(data, all_orderIds)

    def __len__(self):
        return len(self.users)

    def __getitem__(self, idx):
        return self.users[idx], self.items[idx], self.labels[idx]

    def get_dataset(self, data, all_orderIds):
        users, items, labels = [], [], []
        user_item_set = set(zip(train_ratings['CustomerID'], train_ratings['ItemCode']))

        num_negatives = 7
        for u, i in user_item_set:
            users.append(u)
            items.append(i)
            labels.append(1)
            for _ in range(num_negatives):
                negative_item = np.random.choice(all_itemIds)
                while (u, negative_item) in user_item_set:
                    negative_item = np.random.choice(all_itemIds)
                users.append(u)
                items.append(negative_item)
                labels.append(0)

        return torch.tensor(users), torch.tensor(items), torch.tensor(labels)

followed by the PL class:
class NCF(pl.LightningModule):
    '''
    Neural Collaborative Filtering (NCF)

    Args:
        num_users (int): Number of unique users
        num_items (int): Number of unique items
        data (pd.DataFrame): Dataframe containing the food ratings for training
        all_orderIds (list): List containing all orderIds (train + test)
    '''

    def __init__(self, num_users, num_items, data, all_itemIds):
    # def __init__(self, num_users, num_items, ratings, all_movieIds):
        super().__init__()
        self.user_embedding = nn.Embedding(num_embeddings = num_users, embedding_dim = 8)
        # self.user_embedding = nn.Embedding(num_embeddings = num_users, embedding_dim = 10)
        self.item_embedding = nn.Embedding(num_embeddings = num_items, embedding_dim = 8)
        # self.item_embedding = nn.Embedding(num_embeddings = num_items, embedding_dim = 10)
        self.fc1 = nn.Linear(in_features = 16, out_features = 64)
        # self.fc1 = nn.Linear(in_features = 20, out_features = 64)
        self.fc2 = nn.Linear(in_features = 64, out_features = 64)
        self.fc3 = nn.Linear(in_features = 64, out_features = 32)
        self.output = nn.Linear(in_features = 32, out_features = 1)
        self.data = data
        # self.ratings = ratings
        # self.all_movieIds = all_movieIds
        self.all_orderIds = all_orderIds

    def forward(self, user_input, item_input):

        # Pass through embedding layers
        user_embedded = self.user_embedding(user_input)
        item_embedded = self.item_embedding(item_input)

        # Concat the two embedding layers
        vector = torch.cat([user_embedded, item_embedded], dim = -1)

        # Pass through dense layer
        vector = nn.ReLU()(self.fc1(vector))
        vector = nn.ReLU()(self.fc2(vector))
        vector = nn.ReLU()(self.fc3(vector))

        # Output layer
        pred = nn.Sigmoid()(self.output(vector))

        return pred

    def training_step(self, batch, batch_idx):
        user_input, item_input, labels = batch
        predicted_labels = self(user_input, item_input)
        loss = nn.BCELoss()(predicted_labels, labels.view(-1, 1).float())
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())

    def train_dataloader(self):
        return DataLoader(
            ChupsTrainDataset(
                self.data, self.all_orderIds
            ),
            batch_size = 32, num_workers = 2
            # Google Colab's suggested max number of worker in current
            # system is 2 and not 4.
        )

print(f"num_users = {num_users}, num_items = {num_items} & all_itemIds = {len(all_itemIds)}")
# num_users = 12958, num_items = 511238 & all_itemIds = 9114

# Initialize NCF model-
model = NCF(num_users, num_items, train_ratings, all_itemIds)

trainer = pl.Trainer(
    max_epochs = 75, gpus = 1,
    # max_epochs = 5,
    reload_dataloaders_every_n_epochs = True,
    # reload_dataloaders_every_epoch = True,   # deprecated!
    progress_bar_refresh_rate = 50,
    logger = False, checkpoint_callback = False)

trainer.fit(model)

# Save trained model as a checkpoint-
trainer.save_checkpoint("NCF_Trained.ckpt")

To load the saved checkpoint, I have tried:
trained_model = NCF.load_from_checkpoint(
    "NCF_Trained.ckpt", num_users = num_users,
    num_items = train_ratings, data = train_ratings,
    all_itemIds = all_itemIds)

trained_model = NCF(num_users, num_items, train_ratings, all_orderIds).load_from_checkpoint(checkpoint_path = "NCF_Trained.ckpt")

But these don't seem to work. How do I load this saved checkpoint?
Thanks!

Output: As shown in here, load_from_checkpoint is a primary way to load weights in pytorch-lightning and it automatically load hyperparameter used in training. So you do not need to pass params except for overwriting existing ones. My suggestion is to try trained_model = NCF.load_from_checkpoint("NCF_Trained.ckpt")


Instruction: pytorch make_grid (from torchvision.utils import make_grid) behaves different then I expect

Input:
trying to run the visualization utils tutorial from pytorch, I tried it with some images of dogs found on the internet. the images used in the tutorial are not distributed for use.. making the gris and showing the result behaves funny - it shows each channel as a separate image (I guess this is what I see)
so -  from the tutorial

but here is what I get from the images I got:

I was expecting to see the two images in their original colors in a grid.
Another step I tried following Ivan's comment:

tutorial: https://pytorch.org/vision/master/auto_examples/plot_visualization_utils.html
I would like to know how to fix this (and use make_grid correctly)

Output:
For the output you got, I would assume the correct shape is (height, width, channels) instead of (channels, height, width). You can correct this with torch.permute. The following should provide the desired result:
>>> grid = make_grid(torch.stack([transformed_dog1, transformed_dog2]).permute(0,3,1,2))
>>> show(grid)


Instruction: How can I solve this issue? input must have 3 dimensions, got 4

Input:
The Below is data which I passed to the Data Loader,
train_path='/content/drive/MyDrive/Dataset_manual_pytorch/train'
test_path='/content/drive/MyDrive/Dataset_manual_pytorch/test'

train = torchvision.datasets.ImageFolder(train_path,transform=transformations)
test = torchvision.datasets.ImageFolder(test_path,transform=transformations)

train_loader = torch.utils.data.DataLoader(train, batch_size=64, shuffle=True)
test_loader = torch.utils.data.DataLoader(test, batch_size =32, shuffle=True)

This is my Recurrent Neural Network Model,
hidden_size = 256
sequence_length = 28
num_classes = 2
num_layers = 2
input_size = 32
learning_rate = 0.001
num_epochs = 3

class RNN(nn.Module):
  def __init__(self, input_size, hidden_size, num_layers, num_classes):
      super(RNN, self).__init__()
      self.hidden_size = hidden_size
      self.num_layers = num_layers

      self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first = True)
      self.fc = nn.Linear(hidden_size*sequence_length, num_classes)

  def forward(self, x):
    h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

    #Forward Prop
    out,_ = self.rnn(x, h0)
    out = out.reshape(out.shape[0], -1)
    out = self.fc(out)
    return out

model_rnn = RNN(input_size, hidden_size, num_layers, num_classes).to(device)
When I train this model for the particular epochs and for the training data it gives me the following error;
RuntimeError: input must have 3 dimensions, got 4

The shape of data is: torch.Size([64, 3, 32, 32])
I think the error is because I am feeding the data of 4 dimensional, in which I am passing three channels (RGB) as well, to solve this issue I need to reshape; torch.Size([64, 3, 32, 32]) --> torch.Size([64, 32, 32])) But I am unable to do this.
The Training code is;
@torch.no_grad()
def Validation_phase(model, val_loader):
    model.eval()
    for data, labels in val_loader:
      out = model(data)
      val_loss = F.cross_entropy(out, labels)
      val_acc = accuracy(out, labels)

    return val_loss.detach(), val_acc

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        model.train()
        train_losses = []
        train_accuracy = []
        for data, labels in train_loader:
            #forward

            print(data.shape)
            out = model(data)
            #loss calculate
            train_loss = F.cross_entropy(out, labels)

            #Accuracy
            train_acc = accuracy(out, labels)

            train_accuracy.append(train_acc)
            train_losses.append(train_loss.item())

            #back_propagate
            train_loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        train_accuracy = np.mean(torch.stack(train_accuracy).numpy())
        train_losses = np.mean(train_losses)

        #Validation phase
        val_losses, val_accuracy = Validation_phase(model, val_loader)

        print("Epoch [{}], train_loss: {:.4f}, train_accuracy: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
            epoch, train_losses*100 , train_accuracy*100 , val_losses.item()*100, val_accuracy.item()*100))
        # history.append(result)
    # return history

fit(5, 0.001, model_rnn, train_loader, test_loader, torch.optim.Adam)

Output:
You can do the size conversion of torch.Size([64, 3, 32, 32]) to torch.size([64, 32, 32]) by following the bottom code:
x = torch.ones((64, 3, 32, 32))
x = x[:, 0, :, :]

#Check code:
print(x.size())


Instruction: Pytorch fasterrcnn resnet50 fpn loss functions

Input:
I am using a pretrained model from this tutorial. https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-your-model
The model is pytorch's Faster RCNN ResNet 50 FPN model. Does anyone know what the classification loss, loss, and objectness loss functions are (i.e. Cross Entropy or?). Thanks in advance,
Sriram A.

Output:
Objectness is a binary cross entropy loss term over 2 classes (object/not object) associated with each anchor box in the first stage (RPN), and classication loss is normal cross-entropy term over C classes. Both first stage region proposals and second stage bounding boxes are also penalized with a smooth L1 loss term.
It should also be noted that the authors train the first and second stage alternately since both rely on the same features computed with convolutional layers + FPN to aid in training convergence.
Not a very clear description? I'd recommend reading the original Faster-RCNN paper as it is pretty foundational and will probably do a better job describing the loss terms than me.


Instruction: ImportError: cannot import name 'load_mnist' from 'pytorchcv'

Input:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-2cacdf187bba> in <module>
      6 import numpy as np
      7
----> 8 from pytorchcv import load_mnist, train, plot_results, plot_convolution, display_dataset
      9 load_mnist(batch_size=128)

ImportError: cannot import name 'load_mnist' from 'pytorchcv' (/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/pytorchcv/__init__.py)

How can fix this bug?
I use python 3.7 and Jupiter notebook. The code in Pytorch Fundamental of Microsoft: Link https://learn.microsoft.com/en-gb/learn/modules/intro-computer-vision-pytorch/5-convolutional-networks

import torch
import torch.nn as nn
import torchvision
import matplotlib.pyplot as plt
from torchinfo import summary
import numpy as np

from pytorchcv import load_mnist, train, plot_results, plot_convolution, display_dataset
load_mnist(batch_size=128)

I installed PyTorch by command: pip install pytorchcv

Output:
I assume you might have the wrong pytorchcv package. The one in pypy does not contain load_mnist
Starting from scratch you could download mnist as such:

data_train = torchvision.datasets.MNIST('./data',
download=True,train=True,transform=ToTensor()) data_test = torchvision.datasets.MNIST('./data',
download=True,train=False,transform=ToTensor())


Instruction: The best method for normalizing dataset of images

Input:
I have a dataset of images consisting of three splits - the training, validation and test splits, and want to normalize the dataset to make training easier. Hence I want to find the mean and standard deviation of RGB values from the available data.
The doubt I have is - should I consider all the splits for normalizing?
My personal thought is that only the training split should be used since it is assumed to be the only data that we have to train the model. Hence the model is provided inputs from the distribution of the training data, leaving errors that can be picked by evaluation on the validation split. If I provide the distribution to a network from data outside what is provided for training, would it not be feeding the network extra information than what it is supposed to learn from?
Any other way to do this would also be of help. For example, is it just better to use standard values for RGB?
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))

(Soure: Pytorch Torchvision Transforms)

Output:
The doubt I have is - should I consider all the splits for normalizing?

As you said, in theory you should only make use of training data for anything, even for normalization.

Any other way to do this would also be of help. For example, is it just better to use standard values for RGB?

In practice, probably yes. In fact, it shouldn't really matter how you normalize your data, you could even go for mean=0.5, std=0.5 for each channel. Or even adopt a -127/+127 range, the network should adapt to whatever input you provide during training.
What you should probably bear in mind is practical use and application: if you're dealing with pretrained networks, they are usually provided with ImageNet normalization (the one you suggested). This is common practice since:

They are widely used
They are indeed a good approximation of the "real" RGB means and stds for natural images.

TLDR: the choice on custom or "standard" normalization depends on the task itself. In practice, normalization shouldn't matter very much, you should be fine in both cases. You have a decently sized set and time to compute some statistics? Go for custom values. Not so much time for statistics or the dataset is quite small? Probably better to go for the safe ImageNet approach.


Instruction: How to maintain state in a DataLoader's Dataset

Input:
I have something like: (see self.cache for the bit that's interesting).
class DescriptorDataset(torch.utils.data.Dataset):
    def __init__(self, descriptor_dir):
        super().__init__()
        self.file_paths = glob(osp.join(descriptor_dir, '*'))
        self.image_ids = [Path(fp).stem for fp in self.file_paths]
        self.cache = {}

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, ix):
        file_path = self.file_paths[ix]
        descriptor = self.get_descriptor(file_path)
        return descriptor, Path(file_path).stem

    def get_descriptor(self, file_path):
        descriptor = self.cache.get(file_path, torch.load(file_path))
        self.cache[file_path] = descriptor
        return descriptor

query_loader = torch.utils.data.DataLoader(
    DescriptorDataset(query_dir), batch_size=1, num_workers=0

I noticed that the caching mechanism works when num_workers == 0 but not for num_workers > 0. Does PyTorch have an inbuilt way to handle this?

Output:
When I have come across this situation, I have filled the cache during initialisation. In that case it remains fixed during training/inference and can be reloaded the next time you instantiate:
class DescriptorDataset(torch.utils.data.Dataset):
    def __init__(self, descriptor_dir, cache_loc=None):
        super().__init__()
        self.file_paths = glob(osp.join(descriptor_dir, '*'))
        self.image_ids = [Path(fp).stem for fp in self.file_paths]
        self.cache = self.make_cache(cache_loc)

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, ix):
        file_path = self.file_paths[ix]
        descriptor = self.get_descriptor(file_path)
        return descriptor, Path(file_path).stem

    def get_descriptor(self, file_path):
        descriptor = self.cache.get(file_path, torch.load(file_path))
        self.cache[file_path] = descriptor
        return descriptor

    def make_cache(self, cache_loc):
        if os.path.exists(cache_loc):
            return joblib.load(cache_loc)
        else:
            cache = {}
            for p in self.file_paths:
                descriptor = torch.load(p)
                cache[p] = descriptor
        return cache


Instruction: RuntimeError: "reflection_pad2d" not implemented for 'Byte'

Input:
padding = (2, 2, 2, 2)
img = torch.nn.functional.pad(img, padding, mode='reflect')
out = torch.nn.functional.conv2d(img, kernel, groups=img.shape[1])

Here is the complete Error:
  File "/home/amir/PycharmProjects/LPTN/loadPretrainedModel.py", line 57, in conv_gauss
    img = torch.nn.functional.pad(img, padding, mode='reflect')
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py", line 4017, in _pad
    return torch._C._nn.reflection_pad2d(input, pad)
RuntimeError: "reflection_pad2d" not implemented for 'Byte'

What do you think is the problem? I can't figure it out. Thanks in advance.

Output: You need to change data type of your img to float e.g. img.float(). Many operations such as reflection_pad2d are implemented only for float tensors.


Instruction: RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [4]

Input:
How to get around the following error with nn.CrossEntrophyLoss() ?
Note: I tried using nn.BCELoss(), but it resulted in different error: ValueError: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 3, 32, 32])) is deprecated. Please ensure they have the same size.

Output:
The error message is pretty clear: you are using a one-dimensional target tensor while your output prediction has spatial dimensions (a three-channel map).
When using nn.CrossEntropyLoss, your target must be dense (each element is a label id): something with a shape of (batch_size,), where each element in the target belongs to [0, num_classes[. While the output consists of logits: (batch_size, num_classes,), i.e. each class from each batch is assigned a value (this is not yet a probability distribution). In the spatial setting, you will have two additional dimensions (height and width), this is the case for dense predictions such as semantic segmentation. This will make your target have a shape of (batch_size, height, width), and your output prediction (batch_size, num_classes, height, width).


Instruction: Methods for increasing accuracying of a CNN for image classification

Input:
I'm currently working on a image classification task, involving a large datasets of grayscale images of cartoons and my CNN needs to classify them. Atm my model has a test accuracy of about 88% but I know a higher accuracy is possible.
I've tried:

improving / changing the actual model / architecture
using different meta parameters
different loss functions from the pytorch libraries
a bunch of  different transforms
different optimizes from torch.optim

I've also tried a bunch of the standard models included in torchvision.models and am still getting sub 90% accuracy on the test set.
Do I just need to keep trying the above things to squeeze out better accuracy or are there any other avenues I can try? Would really appreciate any suggestions, the only other thing I can think of would be making my own custom loss function specific for the data set but I'm not exactly sure how much that would help?

Output:
From what you've described, it sounds like it might be worth spending some time on the data preparation. Here is a good article on how to do that for images. Some ideas you could try are:

Resizing all your images to a fixed size
Subtracting mean pixel values, i.e. normalizing the dataset

I don't really know the context of what you're doing but I would also consider adding additional features that may be relevant and seeing if that helps.


Instruction: Why network with linear layers can't learn anything

Input:
I am trying to build a neural network for binary classification, unfortunately it always predicts the value 0, even though one fifth of the training set data is 1. I have no idea why it is so. My dataset looks as this, so there are a couple of categorical variables and a couple of continuous, (target is the one we predict):

Here one can download the data:
https://drive.google.com/drive/folders/1PsG2rRdbxyocyqvLSa7zSy_aVDMRJ2Ug?usp=sharing
You can read it with
df = pd.read_csv("train.csv", index_col=0)

Now I am preparing the data for neural network:
x_train=df.drop(labels=['target'], axis=1).values
y_train=df['target'].values
X_train, X_val, y_train, y_val = train_test_split(
x_train,
y_train,
test_size=0.2)

LR = 0.001
EPOCH = 50
BATCH_SIZE = 64

torch_X_train = torch.tensor(X_train)
torch_y_train = torch.tensor(y_train)

torch_X_val = torch.tensor(X_val)
torch_y_val = torch.tensor(y_val)

train = torch.utils.data.TensorDataset(torch_X_train,torch_y_train)
validate = torch.utils.data.TensorDataset(torch_X_val,torch_y_val)

train_loader = torch.utils.data.DataLoader(train, batch_size =
BATCH_SIZE, shuffle = True)
val_loader = torch.utils.data.DataLoader(validate, batch_size =
BATCH_SIZE, shuffle = False)

And define a simple network with just linear layers:
class NN(nn.Module):
def __init__(self, input_size):
    super().__init__()
    self.to_class=nn.Sequential(
    nn.Linear(input_size,512),
    nn.ReLU(),
    nn.Linear(512,256),
    nn.ReLU(),
    nn.Linear(256,32),
    nn.ReLU(),
    nn.Linear(32,2)
    )
def forward(self,inputs):
    pred= self.to_class(inputs)
    return F.softmax(pred, dim=1)

And lastly I train it
net=NN(7)

loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=LR)

train_loss = np.zeros(EPOCH)
val_loss = np.zeros(EPOCH)
acc_train = []
acc_val=[]

for epoch in range(EPOCH):
    correct = 0
    total = 0
    for data in train_loader:
        X, y = data
        optimizer.zero_grad()
        output=net(X.float())
        total += output.size(0)
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(y.view_as(pred)).sum().item()
        loss = loss_func(output.squeeze(), y)
        loss.backward()
        optimizer.step()
        train_loss[epoch] += loss
    train_loss[epoch] /= len(train_loader)
    acc_train.append(correct/total*100)
print('epoch %d:\t train_accuracy %.5f\ttrain loss: %.5f'%(epoch,acc_train[epoch], train_loss[epoch]))

But the train loss is not going anywhere and the predictions are always one class! Can someone explain this phenomenon and hint how I can improve it?

Output:
Please see the doc
This is the standard training loop
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

It seems you are missing optimizer.zero_grad().
Please try to add it and see how it does.

If this doesn't work, please use the same net on a different dataset, for example MNIST, directly downloaded  from pytorch. This net should be able to solve MNIST, so this way we can debug if the problem is the net or something related to the data or the labels.

Another thing is balancing the labels: if 80% are 0 and 20% are 1 (or vice versa) you may want to use weighted cross entropy

Another thing you may want is leaky relu instead of relu. This prevents vanishing gradients for some problems.


Instruction: Action-selection for dqn with pytorch

Input:
I’m a newbie in DQN and try to understand its coding. I am trying the code below as epsilon greedy action selection but I am not sure how it works

    if sample > eps_threshold:
        with torch.no_grad():
           # t.max(1) will return largest      column value of each row.
            # second column on max result is index of where max element was
            # found, so we pick action with the larger expected reward.
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return   torch.tensor([[random.randrange(n_actions)]], device=device, dtype=torch.long)

Could you please let me know what are indices in max(1)[1] and what is view(1, 1) and it’s indices. Also why “with torch.no_grad():” has been used

Output:
When you train a model, torch has to store all the tensors involved in computing the output into a graph, to then be able to make a backward pass during training; this is computationally expensive, and considering that after selecting the action you don't have to train the network, because your only goal here it to pick one using the current weights, then it's just better to use torch.no_grad(). Note that without that part the code would still work the same way, maybe just a bit slower.
About the max(1)[1] part, I'm not really sure how the inputs and outputs are taken considering that there's only a small portion of code here, but I guess that the model takes as input batches of data and outputs a Q-value for each action; then, for each of this outputs you have to take the action that gives you the highest value, so you basically need a max at each row, and that's done by specifying as axis (or dim as torch calls it) the first one, which represents the columns (at every row you take the max of the corresponding columns, which are the actions in this case).


Instruction: Normalizing pixel Values in PyTorch

Input:
I am currently working with the CORnet-Z neural network and I am training it on an alternative version of the ImageNet image dataset.
I looked through the code and noticed this image value normalization method:
normalize = torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                             std=[0.229, 0.224, 0.225])
...

And then implemented later in the train, val, and test class:

torchvision.transforms.Compose([
    torchvision.transforms.RandomResizedCrop(256),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    normalize]))

I was wondering if I could rewrite this to just take the RGB pixel values and divide them by 255 to have a scale of 0-1 to work with.
According to the torchvision.transforms documentation this is not an implemented approach.

Output:
What you found in the code is statistics standardization, you're looking to normalize the input. These are two different operations but can be carried out with the same operator: under torchvision.transforms by the name of Normalize. It applies a shift-scale on the input:

Normalize a tensor image with mean and standard deviation. This transform does not support PIL Image. Given mean: (mean[1],...,mean[n]) and std: (std[1],..,std[n]) for n channels, this transform will normalize each channel of the input torch.*Tensor i.e., output[channel] = (input[channel] - mean[channel]) / std[channel].

>>> normalize = T.Normalize(mean=0, std=255)

Your transformation pipeline is then:
>>> T.Compose([T.RandomResizedCrop(256),
               T.RandomHorizontalFlip(),
               T.ToTensor(),
               normalize]))

where torchvivision.transforms is imported as T.


Instruction: How to convert the below Tensorflow code to Pytorch (transfer learning)?

Input:
I want to know how to convert the below codes(Tensorflow) to Pytorch.
I've wanted to use DataLoader but I couldn't. Is it possible to use DataLoader for converting? or Can you tell me any other ways to convert?
Thanks a lot :)

from tensorflow.keras.preprocessing import image as image_utils
from tensorflow.keras.applications.vgg16 import preprocess_input

def load_and_process_image(image_path):
# Print image's original shape, for reference
print('Original image shape: ', mpimg.imread(image_path).shape)

    # Load in the image with a target size of 224, 224
    image = image_utils.load_img(image_path, target_size=(224, 224))
    # Convert the image from a PIL format to a numpy array
    image = image_utils.img_to_array(image)
    # Add a dimension for number of images, in our case 1
    image = image.reshape(1,224,224,3)
    # Preprocess image to align with original ImageNet dataset
    image = preprocess_input(image)
    # Print image's shape after processing
    print('Processed image shape: ', image.shape)
    return image

Output:
import os
from PIL import Image
import torch
from torch.utils.data import DataLoader, Dataset
from torchvision import transforms

class MyData(Dataset):
    def __init__(self, data_path):
        #path of the folder where your images are located
        self.data_path = data_path
        #transforms to perform on image. In general, these are the default normalization used. you can change std, mean values about three channels according to your requirement
        #when ToTensor() is used it automatically permutes the dimensions according to the torch layers
        self.transforms = transforms.Compose([
                              transforms.Resize((224, 224)),
                              transforms.ToTensor(),
                              transforms.Normalize((mean=[0.485, 0.456, 0.406],
                                                    std=[0.229, 0.224, 0.225])

        self.image_path_list = sorted(os.listdir(self.data_path))

    def __len__(self):
        #returns the length of your dataset
        return len(self.image_path_list)

    def __getitem__(self, idx):
        #pytorch accepts PIL images, use PIL.Image to load images
        image = Image.open(self.image_path_list[idx])
        image = self.transform(image)
        return image

Above is a small snippet based on my assumptions from your post. I assumed you would need to resize, permute and normalize about the given mean values. DataLoader is iterate-able. It yields single image at a time.
For example,
#instantiate your loader, with the desired parameters. checkout the pytorch documentation for other arguments
myloader = DataLoader(MyData, batch_size = 32, num_workers = 10)
myloader = iter(myloader)

for i in range(0, 10):
#this yields first 10 batches of your dataset
img = next(myloader)

Hopefully, this is what you are looking for. Please feel free to comment your quires for any further clarification.


Instruction: Best way to detect Vanishing/Exploding gradient in Pytorch via Tensorboard

Input:
I suspect my Pytorch model has vanishing gradients. I know I can track the gradients of each layer and record them with writer.add_scalar or writer.add_histogram. However, with a model with a relatively large number of layers, having all these histograms and graphs on the TensorBoard log becomes a bit of a nuisance. I'm not saying it doesn't work, it's just a bit inconvenient to have different graphs and histograms for each layer and scroll through them.
I'm looking for a graph where the y axis (vertical) represents the gradient value (mean of gradient of a specific layer), the x axis (horizontal) shows the layer number (e.g. the value at x=1 is the gradient value for 1st layer), and the z axis (depth) is the epoch number.
This would look like a histogram, but of course, it would be essentially different from a histogram since the x axis does not represent beans. One can write a dirty code that would create a histogram where instead of beans there would be layer numbers, something like (this is a pseudo-code, obviously):
fake_distribution = []
for i, layer in enumerate(model.layers):
   fake_distribution += [i for j in range(int(layer.grad.mean()))]
writer.add_histogram('gradients', fake_distribution)

I was wondering if there is a better way for this.

Output:
This is a minimal example of how you could go about evaluating the norm of a particular layer in your model. Taking a simple model for illustration purposes:
class ConvNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, 5)
        self.conv2 = nn.Conv2d(10, 20, 5)
        self.fc1 = nn.Linear(8000, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, input):
        x = F.relu(self.conv1(input))
        x = F.relu(self.conv2(x))
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

net = ConvNet()
net(torch.rand(5,1,28,28)).mean().backward()

Looking at clip_grad_norm_ as reference. To measure the magnitude of the gradient on layer conv1 you could: compute the L2-norm of the vector comprised of the L2-gradient-norms of parameters belonging to that layer. This is done with the following code:
parameters = net.conv1.parameters()
norm_type = 2
total_norm = torch.norm(
    torch.stack([torch.norm(p.grad.detach(), norm_type) for p in parameters]), norm_type)

Alternatively, you can take the maximum of maximum gradient component on that layer i.e. the inf-norm:
total_norm = torch.max(
     torch.stack([p.grad.detach().abs().max() for p in parameters]))

To log them onto your TensorBoard, you can use add_scalar on your SummaryWriter:
for name, module in net.named_children():
    norm = torch.norm(
        torch.stack([torch.norm(p.grad.detach(), 2) for p in parameters]), 2)
    writer.add_scalar(f'check_info/{name}', norm, iter)