Instruction: What is the appropriate way to use BCE loss with ResNet outputs?

Input:
The title explains the overall problem, but for some elaboration:

I'm using torchvision.models.resnet18() to run an anomaly detection scheme. I initialize the model by doing:

net = torchvision.models.resnet18(num_classes=2)

since in my particular setting 0 equals normal samples and 1 equals anomalous samples.

The output from my model is of shape (16, 2) (batch size is 16) and labels are of size (16, 1). This gives me the error that the two input tensors are of inappropriate shape.

In order to solve this, I tried something like:

>>> new_output = torch.argmax(output, dim=1)

Which gives me the appropriate shape, but running loss = nn.BCELoss(new_output, labels) gives me the error:

RuntimeError: bool value of Tensor with more than one value is ambiguous

What is the appropriate way for me to approach this issue? Thanks.

Edit

I've also tried using nn.CrossEntropyLoss as well, but am getting the same RuntimeError.

More specifically, I tried nn.CrossEntropyLoss(output, label) and nn.CrossEntropyLoss(output, label.flatten()).

Output:
If you want to use BCELoss, the output shape should be (16, 1) instead of (16, 2) even though you have two classes. You may consider reading this excellent writing to under binary cross-entropy loss.

Since you are getting output from resnet18 with shape (16, 2), you should rather use CrossEntropyLoss where you can give (16, 2) output and label of shape (16).

You should use CrossEntropyLoss as follows.

loss_crit = nn.CrossEntropyLoss()
loss = loss_crit(output, label)

where output = (16, 2) and label = (16). Please note, the label should contain either 0 or 1.

Please see the example provided (copied below) in the official documentation.

>>> loss = nn.CrossEntropyLoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.empty(3, dtype=torch.long).random_(5)
>>> output = loss(input, target)
>>> output.backward()


Instruction: .eq() method is not giving same result as [ == ]

Input:
I am having a hard time understanding why the results are not the same for following code.

I am trying to find the accuracy of a model but the first item gives a result of tensor(66.), and second item gives a result of tensor(105).

(y_test[y_test==y_predicted_cls].sum(), y_predicted_cls.eq(y_test).sum())

Both of the tensors (y_test and y_predicted_cls) have the same data type, torch.float32.
Output:

(tensor(66.), tensor(105))

I thought that first one would be equal to the second one.

Output:
The statement y_test[y_test==y_predicted_cls].sum() gives the sum for the y_test list/array while, y_predicted_cls.eq(y_test).sum() gives the sum for y_predicted_cls, and in the first case, if both the arrays are same, it yields:

y_test[1].sum()

else :

y_test[0].sum()


Instruction: Pytorch says that CUDA is not available (on Ubuntu)

Input:
I'm trying to run Pytorch on a laptop that I have. It's an older model but it does have an Nvidia graphics card. I realize it is probably not going to be sufficient for real machine learning but I am trying to do it so I can learn the process of getting CUDA installed.

I have followed the steps on the installation guide for Ubuntu 18.04 (my specific distribution is Xubuntu).

My graphics card is a GeForce 845M, verified by lspci | grep nvidia:

01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce 845M] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)

I also have gcc 7.5 installed, verified by gcc --version

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And I have the correct headers installed, verified by trying to install them with sudo apt-get install linux-headers-$(uname -r):

Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-headers-4.15.0-106-generic is already the newest version (4.15.0-106.107).

I then followed the installation instructions using a local .deb for version 10.1.

Npw, when I run nvidia-smi, I get:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 845M        On   | 00000000:01:00.0 Off |                  N/A |
| N/A   40C    P0    N/A /  N/A |     88MiB /  2004MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       982      G   /usr/lib/xorg/Xorg                            87MiB |
+-----------------------------------------------------------------------------+

and I run nvcc -V I get:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

I then performed the post-installation instructions from section 6.1, and so as a result, echo $PATH looks like this:

/home/isaek/anaconda3/envs/stylegan2_pytorch/bin:/home/isaek/anaconda3/bin:/home/isaek/anaconda3/condabin:/usr/local/cuda-10.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

echo $LD_LIBRARY_PATH looks like this:

/usr/local/cuda-10.1/lib64

and my /etc/udev/rules.d/40-vm-hotadd.rules file looks like this:

# On Hyper-V and Xen Virtual Machines we want to add memory and cpus as soon as they appear
ATTR{[dmi/id]sys_vendor}=="Microsoft Corporation", ATTR{[dmi/id]product_name}=="Virtual Machine", GOTO="vm_hotadd_apply"
ATTR{[dmi/id]sys_vendor}=="Xen", GOTO="vm_hotadd_apply"
GOTO="vm_hotadd_end"

LABEL="vm_hotadd_apply"

# Memory hotadd request

# CPU hotadd request
SUBSYSTEM=="cpu", ACTION=="add", DEVPATH=="/devices/system/cpu/cpu[0-9]*", TEST=="online", ATTR{online}="1"

LABEL="vm_hotadd_end"

After all of this, I even compiled and ran the samples. ./deviceQuery returns:

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce 845M"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 2004 MBytes (2101870592 bytes)
  ( 4) Multiprocessors, (128) CUDA Cores/MP:     512 CUDA Cores
  GPU Max Clock rate:                            863 MHz (0.86 GHz)
  Memory Clock rate:                             1001 Mhz
  Memory Bus Width:                              64-bit
  L2 Cache Size:                                 1048576 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS

and ./bandwidthTest returns:

[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce 845M
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         11.7

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         11.8

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         14.5

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

But after all of this, this Python snippet (in a conda environment with all dependencies installed):

import torch
torch.cuda.is_available()

returns False

Does anybody have any idea about how to resolve this? I've tried to add /usr/local/cuda-10.1/bin to etc/environment like this:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
PATH=$PATH:/usr/local/cuda-10.1/bin

And restarting the terminal, but that didn't fix it. I really don't know what else to try.

EDIT - Results of collect_env for @kHarshit

Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce 845M
Nvidia driver version: 418.87.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.5
[pip] pytorch-ranger==0.1.1
[pip] stylegan2-pytorch==0.12.0
[pip] torch==1.5.0
[pip] torch-optimizer==0.0.1a12
[pip] torchvision==0.6.0
[pip] vector-quantize-pytorch==0.0.2
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] stylegan2-pytorch         0.12.0                   pypi_0    pypi
[conda] torch                     1.5.0                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a12                 pypi_0    pypi
[conda] torchvision               0.6.0                    pypi_0    pypi
[conda] vector-quantize-pytorch   0.0.2                    pypi_0    pypi

Output:
PyTorch doesn't use the system's CUDA library. When you install PyTorch using the precompiled binaries using either pip or conda it is shipped with a copy of the specified version of the CUDA library which is installed locally. In fact, you don't even need to install CUDA on your system to use PyTorch with CUDA support.
There are two scenarios which could have caused your issue.

You installed the CPU only version of PyTorch. In this case PyTorch wasn't compiled with CUDA support so it didn't support CUDA.

You installed the CUDA 10.2 version of PyTorch. In this case the problem is that your graphics card currently uses the 418.87 drivers, which only support up to CUDA 10.1. The two potential fixes in this case would be to either install updated drivers (version >= 440.33 according to Table 2) or to install a version of PyTorch compiled against CUDA 10.1.

To determine the appropriate command to use when installing PyTorch you can use the handy widget in the "Install PyTorch" section at pytorch.org. Just select the appropriate operating system, package manager, and CUDA version then run the recommended command.
In your case one solution was to use
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

which explicitly specifies to conda that you want to install the version of PyTorch compiled against CUDA 10.1.
For more information about PyTorch CUDA compatibility with respect drivers and hardware see this answer.

Edit After you added the output of collect_env we can see that the problem was that you had the CUDA 10.2 version of PyTorch installed. Based on that an alternative solution would have been to update the graphics driver as elaborated in item 2 and the linked answer.


Instruction: Why do 'loss.backward()' and 'weight.grad' return a tensor containing all zeros?

Input:
When I run 'loss.backward()' and 'weight.grad' I get a tensor containing all zeros. Also, 'weight.grad_fn' retruns NONE.

However, it all seems to return the correct result for the second layer 'w2'.
If I play with simple operations such as x*2 or x**2 'backward()' and '.grad' return correct results

Here's my code:

import torch
from torch import nn
import torch.nn.functional as F
from torchvision import datasets, transforms

# Getting MNIST data
num_workers = 0
batch_size = 64
transform = transforms.ToTensor()
train_data = datasets.MNIST(root='data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers)
dataiter = iter(train_loader)
images, labels = dataiter.next()

#####################################
#####################################
#### NN Part

def activation(x):
    return 1/(1+torch.exp(-x))

inputs = torch.from_numpy(images.view())
# Flatten the inputs format from (64,1,28,28) into (64,784)
inputs = inputs.reshape(images.shape[0], int(images.shape[1]*images.shape[2]*images.shape[3]))

w1 = torch.randn(784, 256, requires_grad=True)# n_input, n_hidden
b1 = torch.randn(256)# n_hidden

w2 = torch.randn(256, 10, requires_grad=True)# n_hidden, n_output
b2 = torch.randn(10)# n_output

h = activation(torch.mm(inputs, w1) + b1)
y = torch.mm(h, w2) + b2

#print(h)
#print(y)

y.sum().backward()
print(w1.grad)
print(w1.grad_fn)
#print(w2.grad)
#print(w2.grad_fn)

By the way it gives me the same problem if I try to run it this way also:

images = images.reshape(images.shape[0], -1)

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

logits = model(images)
criterion = nn.NLLLoss()

loss = criterion(logits, labels)
print(loss)
print(loss.grad_fn)

print('Before backward pass: ', model[0].weight.grad)
loss.backward()
print('After: ', model[0].weight.grad)
#print('After: ', model[2].weight.grad)
#print('After: ', model[4].weight.grad)

Output:
The gradients of w1 are not all zero, there are simply a lot of zeros, especially around the border, because the MNIST images have a lot of black pixels (zeros). When multiplying with zero, the resulting gradients are also zero.

By printing w1.grad you only see a very small part of the values (borders), and you just can't see the non-zero values.

w1.grad
# => tensor([[0., 0., 0.,  ..., 0., 0., 0.],
#            [0., 0., 0.,  ..., 0., 0., 0.],
#            [0., 0., 0.,  ..., 0., 0., 0.],
#            ...,
#            [0., 0., 0.,  ..., 0., 0., 0.],
#            [0., 0., 0.,  ..., 0., 0., 0.],
#            [0., 0., 0.,  ..., 0., 0., 0.]])

# Indices of non-zero elements
w1.grad.nonzero()
# => tensor([[ 71,   0],
#            [ 71,   1],
#            [ 71,   2],
#            ...,
#            [746, 253],
#            [746, 254],
#            [746, 255]])


Instruction: Get probability from predicted class pytorch

Input:
I have a network like this for image classification, for now i have 2 classes:

class ActionNet(Module):

    def __init__(self, num_class=4):

        super(ActionNet, self).__init__()

        self.cnn_layer = Sequential(

            #conv1

            Conv2d(in_channels=1, out_channels=32, kernel_size=1, bias=False),

            BatchNorm2d(32),

            PReLU(num_parameters=32),

            MaxPool2d(kernel_size=3),

            #conv2

            Conv2d(in_channels=32, out_channels=64, kernel_size=1, bias=False),

            BatchNorm2d(64),

            PReLU(num_parameters=64),

            MaxPool2d(kernel_size=3),

            #flatten

            Flatten(),

            Linear(576, 128),

            BatchNorm1d(128),

            ReLU(inplace=True),

            Dropout(0.5),

            Linear(128, num_class)

        )

    def forward(self, x):

        x = self.cnn_layer(x)

        return x

then after i training my network, i predict the image using this code:

def predict_image(image):

    input = torch.from_numpy(image)

    input = input.unsqueeze(1)

    input = input.to(device)

    output = model(input)

    index = output.data.cpu().numpy().argmax()

    return index

how do i get all class probability of prediction image? so the result would be array of index with probaility like 0=0.1, 1=0.7

Output:
To get probability from model output here you can use softmax function.

Try this

import torch.nn.functional as F
...
prob = F.softmax(output, dim=1)
...


Instruction: PyTorch demands image-like dimensionality in the DataLoader even though no images are used

Input:
I cannot get a simple neural network to run with a custom dummy dataset. You can find the error message at the very bottom of this question.

I would like to deeply understand how PyTorch processes data inputs and therefore built a simple dataset that gets two series of boolean values (encoded as 0s and 1s) as inputs, with OR, AND, and XOR series as targets. Everything should work with a custom Dataset and a DataLoader (for learning purposes). The input data looks as follows (naturally, only one target column is used at a time):

      column_1  column_2  or  and  xor
0            1         1   1    1    0
1            0         1   1    0    1
2            0         1   1    0    1
3            0         1   1    0    1
4            1         0   1    0    1
...        ...       ...  ..  ...  ...
9995         1         1   1    1    0
9996         1         0   1    0    1
9997         1         0   1    0    1
9998         0         1   1    0    1
9999         0         1   1    0    1

So I would like to build a neural network that represents either an OR gate, an AND gate, or an XOR gate.

Can someone shed some light on why one-dimensional data does not seem to be accepted in the iterator? It seems like image-like data is assumed? Is it even possible to solve this problem with a custom Dataset and with a DataLoader or will I have to compromise to not use a DataLoader (like in this intro to logic gates in PyTorch)?

Minimum working example with XOR as the target column:

# Imports
from pandas import read_csv, DataFrame
import torch
from torch.utils.data import Dataset, DataLoader
import torch.nn as torch_nn
import torch.nn.functional as torch_functional
import torchvision
import random

# Classes
class CustomDataset(Dataset):
    def __init__(self, data, target_column_name, transform=None):
        self.dataframe = data
        self.x = data[['column_1', 'column_2']].values
        self.y = data[[target_column_name]].values
        self.n_samples = len(data)
        self.transform = transform

    def __len__(self):
        return self.n_samples

    def __getitem__(self, index):
        x = self.x[index]
        y = self.y[index]

        if not self.transform == None:
            return (self.transform(x), self.transform(y))
        return (x, y)

class CustomNet(torch_nn.Module):
    def __init__(self):
        super().__init__()
        self.fully_connected_input_layer = torch_nn.Linear(2, 5)
        self.fully_connected_hidden_layer_1 = torch_nn.Linear(5, 5)
        self.fully_connected_hidden_layer_2 = torch_nn.Linear(5, 5)
        self.fully_connected_output_layer = torch_nn.Linear(5, 2)

    def forward(self, data):
        data = torch_functional.relu(self.fully_connected_input_layer(data))
        data = torch_functional.relu(self.fully_connected_hidden_layer_1(data))
        data = torch_functional.relu(self.fully_connected_hidden_layer_2(data))
        data = torch_functional.log_softmax(
            self.fully_connected_output_layer(data),
            dim=1
        )
        # data = data.squeeze(1)

        return data

# Global variables
NUMBER_OF_OBSERVATIONS = 10000
N_TRAIN_OBSERVATIONS = 7000
BATCH_SIZE = 4
N_EPOCHS = 3
random.seed(42)

# Generating logical gate data
## Generating two columns with random 0s and 1s.
df_data = DataFrame({
    'column_1': random.choices([0, 1], k=NUMBER_OF_OBSERVATIONS),
    'column_2': random.choices([0, 1], k=NUMBER_OF_OBSERVATIONS)
})
## Adding the logic gate results of the previously generated two columns.
df_data.loc[:,'or'] = (df_data['column_1'] == 1) | (df_data['column_2'] == 1)
df_data.loc[:,'or'] = df_data.loc[:,'or'].astype(int)
df_data.loc[:,'and'] = (df_data['column_1'] == 1) & (df_data['column_2'] == 1).astype(int)
df_data.loc[:,'and'] = df_data.loc[:,'and'].astype(int)
df_data.loc[:,'xor'] = ((df_data['column_1'] == 1) & (df_data['column_2'] != 1)) | ((df_data['column_1'] != 1) & (df_data['column_2'] == 1)).astype(int)
df_data.loc[:,'xor'] = df_data.loc[:,'xor'].astype(int)
print(df_data.info())
print(df_data)

# Instantiating a CustomDataSet object and a DataLoader object.
dataset_data = CustomDataset(df_data, target_column_name='xor', transform=torchvision.transforms.ToTensor())
dataset_data_train, dataset_data_test = torch.utils.data.random_split(
    dataset_data,
    [N_TRAIN_OBSERVATIONS, len(dataset_data)-N_TRAIN_OBSERVATIONS]
)
dataloader_data_train = DataLoader(
    dataset=dataset_data_train,
    batch_size=BATCH_SIZE,
    shuffle=True
)
dataloader_data_test = DataLoader(
    dataset=dataset_data_test,
    batch_size=BATCH_SIZE,
    shuffle=True
)

# Instantiating the neural network
custom_net = CustomNet()

# Running one epoch
for epoch in range(N_EPOCHS):
    for data in dataloader_data_train:
        X, y = data[0].float(), data[1].float()
        net.zero_grad()
        output = net(X)
        print(output)
        print(y)
        loss = torch_functional.nll_loss(output, y)
        loss.backward()
        optimizer.step()
        break

Error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-f2833deb4b11> in <module>
     95 # Running one epoch
     96 for epoch in range(N_EPOCHS):
---> 97     for data in dataloader_data_train:
     98         X, y = data[0].float(), data[1].float()
     99         net.zero_grad()

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataset.py in __getitem__(self, idx)
    255
    256     def __getitem__(self, idx):
--> 257         return self.dataset[self.indices[idx]]
    258
    259     def __len__(self):

<ipython-input-2-f2833deb4b11> in __getitem__(self, index)
     26
     27         if not self.transform == None:
---> 28             return (self.transform(x), self.transform(y))
     29         return (x, y)
     30

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, pic)
     90             Tensor: Converted image.
     91         """
---> 92         return F.to_tensor(pic)
     93
     94     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in to_tensor(pic)
     43
     44     if _is_numpy(pic) and not _is_numpy_image(pic):
---> 45         raise ValueError('pic should be 2/3 dimensional. Got {} dimensions.'.format(pic.ndim))
     46
     47     if isinstance(pic, np.ndarray):

ValueError: pic should be 2/3 dimensional. Got 1 dimensions.

Output:
The problem is that you're using a specialized conversion routine from the Torchvision library, torchvision.transforms.ToTensor. You should just use torch.from_numpy instead.

Also note that .values on Pandas objects is deprecated. You should use .to_numpy instead:

import pandas as pd
import torch

x_pandas = pd.Series([0.0, 0.5, 1.0])
x_numpy = x_pandas.to_numpy()
x_torch = torch.from_numpy(x_numpy)


Instruction: Delete an element from torch.Tensor

Input:
I'm trying to delete an item from a tensor.

In the example below, How can I remove the third item from the tensor ?

tensor([[-5.1949, -6.2621, -6.2051, -5.8983, -6.3586, -6.2434, -5.8923, -6.1901,
         -6.5713, -6.2396, -6.1227, -6.4196, -3.4311, -6.8903, -6.1248, -6.3813,
         -6.0152, -6.7449, -6.0523, -6.4341, -6.8579, -6.1961, -6.5564, -6.6520,
         -5.9976, -6.3637, -5.7560, -6.7946, -5.4101, -6.1310, -3.3249, -6.4584,
         -6.2202, -6.3663, -6.9293, -6.9262]], grad_fn=<SqueezeBackward1>)

Output:
You can first filter array through indices and then concat both

t.shape
torch.Size([1, 36])

t = torch.cat((t[:,:3], t[:,4:]), axis = 1)

t.shape
torch.Size([1, 35])


Instruction: Understanding input shape to PyTorch conv1D?

Input:
This seems to be one of the common questions on here (1, 2, 3), but I am still struggling to define the right shape for input to PyTorch conv1D.
I have text sequences of length 512 (number of tokens per sequence) with each token being represented by a vector of length 768 (embedding). The batch size I am using is 6.
So my input tensor to conv1D is of shape [6, 512, 768].
input = torch.randn(6, 512, 768)

Now, I want to convolve over the length of my sequence (512) with a kernel size of 2 using the conv1D layer from PyTorch.
Understanding 1:
I assumed that "in_channels" are the embedding dimension of the conv1D layer. If so, then a conv1D layer will be defined in this way where
in_channels = embedding dimension (768)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(768, 100, 2)
feature_map = convolution_layer(input)

But with this assumption, I get the following error:
RuntimeError: Given groups=1, weight of size 100 768 2, expected input `[4, 512, 768]` to have 768 channels, but got 512 channels instead

Understanding 2:
Then I assumed that "in_channels" is the sequence length of the input sequence. If so, then a conv1D layer will be defined in this way where
in_channels = sequence length (512)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(512, 100, 2)
feature_map = convolution_layer(input)

This works fine and I get an output feature map of dimension [batch_size, 100, 767]. However, I am confused. Shouldn't the convolutional layer convolve over the sequence length of 512 and output a feature map of dimension [batch_size, 100, 511]?
I will be really grateful for your help.

Output:
In pytorch your input shape of [6, 512, 768] should actually be [6, 768, 512] where the feature length is represented by the channel dimension and sequence length is the length dimension. Then you can define your conv1d with in/out channels of 768 and 100 respectively to get an output of [6, 100, 511].

Given an input of shape [6, 512, 768] you can convert it to the correct shape with Tensor.transpose.

input = input.transpose(1, 2).contiguous()

The .contiguous() ensures the memory of the tensor is stored contiguously which helps avoid potential issues during processing.


Instruction: Why does the output from VQ-Wav2Vec from FairSeq missing frames?

Input:
I am using the fairseq library to run an example code for feature extraction with the VQ-Wav2Vec code as written below:

In [6]: import torch
   ...: from fairseq.models.wav2vec import Wav2VecModel

In [7]: cp = torch.load('wav2vec_models/checkpoint_best.pt')
   ...: model = Wav2VecModel.build_model(cp['args'], task=None)
   ...: model.load_state_dict(cp['model'])
   ...: model.eval()

In [9]: wav_input_16khz = torch.randn(1,10000)
   ...: z = model.feature_extractor(wav_input_16khz)
   ...: f, idxs = model.vector_quantizer.forward_idx(z)
   ...: print(idxs.shape, f.shape)

>>>> torch.Size([1, 60, 4]) torch.Size([1, 512, 60])

My understanding is that the vq-wav2vec processes every 10ms of input speech (assumed to be sampled at 16K samples / sec) samples and outputs a feature vector of size [512] samples for each of these 10ms of speech. So given that the input speech is 10000 samples, we are supposed to get 62 frames ( 62 * 160 = 9920 samples).

Why do I see only 60 frames?

Output:
From the article (arxiv.org/pdf/1904.05862.pdf): "The output of the encoder is a low frequency feature representation zi ∈Z which encodes about 30 ms of 16 kHz of audio and the striding results in representations zi every 10ms." => The windows are overlapping and this explains why you are getting 2 frames fewer.
Indeed we are moving a 30 ms window by 10ms steps. In your example, the 30 ms window takes 60 different positions.


Instruction: Add values to a tensor element based on its position in pytorch

Input:
In pytorch, I want to add values to elements in the tensor based on their position. For example consider,

Input = torch.tensor([1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0])

Between several offsets of the Input array, Offsets = [0,5,10,15,20], I want to add different values, ValuesToAdd = [10,100,1000,10000]

I expect the output to be

Output = torch.tensor([11,12,13,14,15,106,107,108,109,100,1001,1002,1003,1004,1005,10006,10007,10008,10009,10000])

Here, between indices Offsets[i] and Offsets[i+1] in Input array, ValuesToAdd[i] is added. For example, for indices 10,11,12,13 and 14 (Offsets[2] = 10 to Offsets[3]=15) in Input array, 1000 (ValuesToAdd[2]) is added.

How can I achieve this? Instead of looping over Offsets array, I am looking for a more efficient way.

Output:
You can use torch.repeat_interleave

Offsets = torch.tensor(Offsets)
shifts = Offsets[1:] - Offsets[:-1]
output = Input.clone()
output[Offsets[0]:Offsets[-1]] += torch.tensor(ValuesToAdd).repeat_interleave(shifts)
print(torch.all(output == Output))
# True


Instruction: How to obtain sequence of submodules from a pytorch module?

Input:
For a pytorch module, I suppose I could use .named_children, .named_modules, etc. to obtain a list of the submodules. However, I suppose the list is not given in order, right? An example:

In [19]: import transformers

In [20]: model = transformers.DistilBertForSequenceClassification.from_pretrained('distilb
    ...: ert-base-cased')

In [21]: [name for name, _ in model.named_children()]
Out[21]: ['distilbert', 'pre_classifier', 'classifier', 'dropout']

The order of .named_children() in the above model is given as distilbert, pre_classifier, classifier, and dropout. However, if you examine the code, it is evident that dropout happens before classifier. So how do I get the order of these submodules?

Output:
In Pytorch, the results of print(model) or .named_children(), etc are listed based on the order they are declared in __init__ of the model's class e.g.

Case 1

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)
        self.conv2_drop = nn.Dropout2d()

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, p=0.6)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

model = Model()
print(model)
[name for name, _ in model.named_children()]
# output
['conv1', 'conv2', 'fc1', 'fc2', 'conv2_drop']

Case 2

Changed order of fc1 and fc2 layers in constructor.

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.fc2 = nn.Linear(50, 10)
        self.fc1 = nn.Linear(320, 50)
        self.conv2_drop = nn.Dropout2d()

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, p=0.6)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

model = Model()
print(model)
[name for name, _ in model.named_children()]
# output
['conv1', 'conv2', 'fc2', 'fc1', 'conv2_drop']

That's why classifier is printed before dropout as it's declared so in constructor:

class DistilBertForSequenceClassification(DistilBertPreTrainedModel):
        ...
        self.distilbert = DistilBertModel(config)
        self.pre_classifier = nn.Linear(config.dim, config.dim)
        self.classifier = nn.Linear(config.dim, config.num_labels)
        self.dropout = nn.Dropout(config.seq_classif_dropout)

Nevertheless, you can play with model's submodules using .modules(), etc. but they'll be listed only in the order they are declared in __init__. If you only want to print structure based on forward method, you may try using pytorch-summary.


Instruction: How to improve code to speed up word embedding with transformer models?

Input:
I need to compute words embeddings for a bunch of documents with different language models.
No problem with that, the script is doing fine, except I'm working on a notebook, without GPU and each text needs around 1.5s to be processed which is by far too long (I have thousands of texts to process).

Here is how I'm doing it with pytorch and transformers lib:

import torch
from transformers import CamembertModel, CamembertTokenizer

docs = [text1, text2, ..., text20000]
tok = CamembertTokenizer.from_pretrained('camembert-base')
model = CamembertModel.from_pretrained('camembert-base', output_hidden_states=True)
# let try with a batch size of 64 documents
docids = [tok.encode(
  doc, max_length=512, return_tensors='pt', pad_to_max_length=True) for doc in docs[:64]]
ids=torch.cat(tuple(docids))
device = 'cuda' if torch.cuda.is_available() else 'cpu' # cpu in my case...
model = model.to(device)
ids = ids.to(device)
model.eval()
with torch.no_grad():
    out = model(input_ids=ids)
# 103s later...

Do someone has any idea or suggestions to improve speed?

Output:
I don't think that there is a trivial way to significantly improve the speed, without using a GPU.

Some of the ways I could think of include smart batching, which is used by Sentence-Transformers, where you basically sort inputs of similar length together, to avoid padding to the full 512 token limit. I'm not sure how much of a speedup this is going to get you, but the only way that you can improve it significantly in a short period of time.

Otherwise, if you have access to Google colab, you can also utilize their GPU environment, if the processing can be completed in reasonable time.


Instruction: Cannot import BertModel from transformers

Input:
I am trying to import BertModel from transformers, but it fails. This is code I am using

from transformers import BertModel, BertForMaskedLM

This is the error I get

ImportError: cannot import name 'BertModel' from 'transformers'

Can anyone help me fix this?

Output:
Fixed the error. This is the code

from transformers.modeling_bert import BertModel, BertForMaskedLM


Instruction: Why not accumulate query loss and then take derivative in MAML with Pytorch and Higher?

Input:
when doing MAML (Model agnostic meta-learning) there are two ways to do the inner loop:

def inner_loop1():
    n_inner_iter = 5
    inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

    qry_losses = []
    qry_accs = []
    meta_opt.zero_grad()
    for i in range(task_num):
        with higher.innerloop_ctx(
            net, inner_opt, copy_initial_weights=False
        ) as (fnet, diffopt):
            # Optimize the likelihood of the support set by taking
            # gradient steps w.r.t. the model's parameters.
            # This adapts the model's meta-parameters to the task.
            # higher is able to automatically keep copies of
            # your network's parameters as they are being updated.
            for _ in range(n_inner_iter):
                spt_logits = fnet(x_spt[i])
                spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                diffopt.step(spt_loss)

            # The final set of adapted parameters will induce some
            # final loss and accuracy on the query dataset.
            # These will be used to update the model's meta-parameters.
            qry_logits = fnet(x_qry[i])
            qry_loss = F.cross_entropy(qry_logits, y_qry[i])
            qry_losses.append(qry_loss.detach())
            qry_acc = (qry_logits.argmax(
                dim=1) == y_qry[i]).sum().item() / querysz
            qry_accs.append(qry_acc)

            # Update the model's meta-parameters to optimize the query
            # losses across all of the tasks sampled in this batch.
            # This unrolls through the gradient steps.
            qry_loss.backward()

    meta_opt.step()
    qry_losses = sum(qry_losses) / task_num
    qry_accs = 100. * sum(qry_accs) / task_num
    i = epoch + float(batch_idx) / n_train_iter
    iter_time = time.time() - start_time

def inner_loop2():
    n_inner_iter = 5
    inner_opt = torch.optim.SGD(net.parameters(), lr=1e-1)

    qry_losses = []
    qry_accs = []
    meta_opt.zero_grad()
    meta_loss = 0
    for i in range(task_num):
        with higher.innerloop_ctx(
            net, inner_opt, copy_initial_weights=False
        ) as (fnet, diffopt):
            # Optimize the likelihood of the support set by taking
            # gradient steps w.r.t. the model's parameters.
            # This adapts the model's meta-parameters to the task.
            # higher is able to automatically keep copies of
            # your network's parameters as they are being updated.
            for _ in range(n_inner_iter):
                spt_logits = fnet(x_spt[i])
                spt_loss = F.cross_entropy(spt_logits, y_spt[i])
                diffopt.step(spt_loss)

            # The final set of adapted parameters will induce some
            # final loss and accuracy on the query dataset.
            # These will be used to update the model's meta-parameters.
            qry_logits = fnet(x_qry[i])
            qry_loss = F.cross_entropy(qry_logits, y_qry[i])
            qry_losses.append(qry_loss.detach())
            qry_acc = (qry_logits.argmax(
                dim=1) == y_qry[i]).sum().item() / querysz
            qry_accs.append(qry_acc)

            # Update the model's meta-parameters to optimize the query
            # losses across all of the tasks sampled in this batch.
            # This unrolls through the gradient steps.
            #qry_loss.backward()
            meta_loss += qry_loss

    meta_loss.backward()
    meta_opt.step()
    qry_accs = 100. * sum(qry_accs) / task_num
    i = epoch + float(batch_idx) / n_train_iter
    iter_time = time.time() - start_time

are they truly equivalent?

cross-posted:

git issue: https://github.com/facebookresearch/higher/issues/60

Output:
The only difference is that in second approach you'll have to keep much more stuff in memory - until you call backward you'll have all unrolled parameters fnet.parameters(time=T) (along with intermediate computation tensors) for each of task_num iterations as part of the graph for the aggregated meta_loss. If you call backward on every task then you only need to keep full set of unrolled parameters (and other pieces of the graph) for one task.
So to answer your question's title: because in this case the memory footprint is task_num times bigger.
In a nutshell what you're doing is similar to comparing loopA(N) and loopB(N) in the following code. Here loopA will get as much memory as it can and OOM with sufficiently large N, while loopB will use about same amount of memory for any large N:
import torch
import numpy as np
a = 0
np.random.seed(1)
v = torch.tensor(np.random.randn(1000000))
y = torch.tensor(np.random.randn(1000000))
x = torch.zeros(1000000, requires_grad=True)

def loopA(N=1000):
    a = 0
    for i in range(N):
        a += ((x * v - y)**2).sum()
    a.backward()

def loopB(N=1000):
    for i in range(N):
        a = ((x * v - y)**2).sum()
        a.backward()

Regarding the normalization - two approaches are equivalent (up to numerical precision maybe): if you first sum up individual losses, then divide by task_num, then finally call backward then you'll effectively compute d((Loss_1 + ... + Loss_{task_num})/task_num) / dw (where w is one of the weights meta-optimizer is fitting). On the other hand if you call backward for each loss divided by task_num you'll get d(Loss_1/task_num)/dw + ... + d(Loss_{task_num}/task_num)/dw which is the same because taking gradient operation is linear. So in both cases your meta-optimizer step will start with pretty much same gradients.


Instruction: Why does 'dimension' mean several different things in the machine-learning world?

Input:
I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a single datapoint. e.g. in 512-d word-embeddings means 512 length vector of floats used to represent 1 english-word e.g. https://medium.com/@jonathan_hui/nlp-word-embedding-glove-5e7f523999f6

But it isn't 512 different dimensions, it's only 1 dimensional vector? Why is the term dimension used in such a different manner than usual?

When we use the term conv1d or conv2d which are convolutions over 1-dimension and 2-dimensions, a dimension is used in the typical way it's used in math/sciences but in the word-embedding context, a 1-d vector is said to be a 512-d vector, or am I missing something?

Why is this overloaded use of the term dimension? What context determines what dimension means in machine-learning as the term seems overloaded?

Output:
In the context of word embeddings in neural networks, dimensionality reduction, and many other machine learning areas, it is indeed correct to call the vector (which is typically, an 1D array or tensor) as n-dimensional where n is usually greater than 2. This is because we usually work in the Euclidean space where a (data) point in a certain dimensional (Euclidean) space is represented as an n-tuple of real numbers (i.e. real n-space ℝn).

Below is an exampleref of a (data) point in a 3D (Euclidean) space. To represent any point in this space, say d1, we need a tuple of three real numbers (x1, y1, z1).

Now, your confusion arises why this point d1 is called as 3 dimensional instead of 1 dimensional array. The reason is because it lies or lives in this 3D space. The same argument can be extended to all points in any n-dimensional real space, as it is done in the case of embeddings with 300d, 512d, 1024d vector etc.

However, in all nD array compute frameworks such as NumPy, PyTorch, TensorFlow etc, these are still 1D arrays because the length of the above said vectors can be represented using a single number.

But, what if you have more than 1 data point? Then, you have to stack them in some (unique) way. And this is where the need for a second dimension arises. So, let's say you stack 4 of these 512d vectors vertically, then you'd end up with a 2D array/tensor of shape (4, 512). Note that here we call the array as 2D because two integer numbers are required to represent the extent/length along each axis.

To understand this better, please refer my other answer on axis parameter visualization for nD arrays, the visual representation of which I will include it below.

ref: Euclidean space wiki


Instruction: Size mismatch error in converting TensorFlow model to Pytorch

Input:
I´m trying to convert a TensorFlow model to Pytorch but got stuck in this error. Can anyone help me?

#getting weights and biases from tensorflow model
weights, biases = model.layers[0].get_weights()
#[1] is the dropout layer
weights2, biases2 = model.layers[2].get_weights()

#initializing pytorch
class TwoLayerNet(torch.nn.Module):
def __init__(self, weights, biases, weights2, biases2):

    super(TwoLayerNet, self).__init__()
    #created the model in the same dimensions as tensorflow´s model
    self.linear1 = torch.nn.Linear(9, 2048)
    self.hidden1 = nn.Dropout(0.2)
    self.linear2 = torch.nn.Linear(2048,5)

    weights = torch.from_numpy(weights)
    biases = torch.from_numpy(biases)
    weights2 = torch.from_numpy(weights2)
    biases2 = torch.from_numpy(biases2)

    self.linear1.weight = torch.nn.Parameter(weights)
    self.linear1.bias = torch.nn.Parameter(biases)
    self.linear2.weight.data = weights2
    self.linear2.bias.data = biases2
    #in this print the dimensions are ok (Linear(in_features=9, out_features=2048, bias=True))
    print(self.linear1)

def forward(self, x):
  print(self.linear1)
  x = self.linear1(x)
  x = self.hidden1(x)
  x = self.linear2(x)
  return x

model_pytorch = TwoLayerNet(weights, biases, weights2, biases2)

model_pytorch.eval()
exemplo_input_torch = torch.from_numpy(exemplo_input)
exemplo_input_torch = exemplo_input_torch.float()
print(exemplo_input_torch)
result = model_pytorch(exemplo_input_torch)

The error is:

RuntimeError: size mismatch, m1: [1 x 9], m2: [2048 x 9] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:41

Output:
You need to transpose the weights and biases:

weights = torch.from_numpy(weights).T
biases = torch.from_numpy(biases).T
weights2 = torch.from_numpy(weights2).T
biases2 = torch.from_numpy(biases2).T


Instruction: How to remove showing renderings in the Cartpole game from OpenAI in Pytorch

Input:
My current code is based off of Pytorch's example on their website where they use env.render() to make the next state. That makes the game run very slow and would like it to run much quicker without the renderings. Here is the the class that uses the render function and at the bottom is the full code. Any ideas how I can get this to work?

class CartPoleEnvManager(): #manage cartpole env
    def __init__(self, device):
        self.device = device
        self.env = gym.make('CartPole-v0').unwrapped #unwrapped so we can see other dynamics that there is no access to otherwise
        self.env.reset() #reset to starting state
        self.current_screen = None #screen initialization
        self.done = False #if episode is finished

    def reset(self):
        self.env.reset()
        self.current_screen = None

    def close(self): #close env
        self.env.close()

    def render(self, mode='human'): #render the current state to screen
        return self.env.render(mode)

    def num_actions_available(self): #returns # actions available to agent (2)
        return self.env.action_space.n

    def take_action(self, action):# step returns tuple containing env observation, reward and diagnostic info -- all from taking a certain action
        _, reward, self.done, _ = self.env.step(action.item()) # only reward and done status are of importance
        return torch.tensor([reward], device=self.device)
    #####action is a tensor, action.item() gives a number, what step wants

    def just_starting(self):
        return self.current_screen is None

    def get_state(self): #return to the current state of env in the form of a processed image of the screen
        if self.just_starting() or self.done:
            self.current_screen = self.get_processed_screen() #state = processed image of diff of 2 separate screens
            black_screen = torch.zeros_like(self.current_screen)
            return black_screen
        else:
            s1 = self.current_screen
            s2 = self.get_processed_screen() ####what is get_processed_screen?
            self.current_screen = s2
            return s2 - s1 # this represents a single state

    def get_screen_height(self):
        screen = self.get_processed_screen()
        return screen.shape[2]

    def get_screen_width(self):
        screen = self.get_processed_screen()
        return screen.shape[3]

    def get_processed_screen(self):
        screen = self.env.render(mode='rgb_array').transpose((2, 0, 1)) # PyTorch expects CHW
        screen = self.crop_screen(screen)
        return self.transform_screen_data(screen)

    def crop_screen(self, screen):
        screen_height = screen.shape[1]

        # Strip off top and bottom
        top = int(screen_height * 0.4)
        bottom = int(screen_height * 0.8)
        screen = screen[:, top:bottom, :]
        return screen

    def transform_screen_data(self, screen):
        # Convert to float, rescale, convert to tensor
        screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
        screen = torch.from_numpy(screen)

        # Use torchvision package to compose image transforms
        resize = T.Compose([
            T.ToPILImage()
            ,T.Resize((40,90))
            ,T.ToTensor()
        ])

        return resize(screen).unsqueeze(0).to(self.device) # add a batch dimension (BCHW)

And the full code:

import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
from itertools import count
from PIL import Image
import torch
import torch.nn as nnfr
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T
import pennylane as qml
import numpy as np
import torch
import torch.nn as nn
from torch.nn.functional import relu, sigmoid

class DQN(nn.Module):
    def __init__(self, img_height, img_width):
        super().__init__()

        self.fc1 = nn.Linear(in_features=img_height*img_width*3, out_features=64)
        self.fc2 = nn.Linear(in_features=64, out_features=48)
        self.fc3 = nn.Linear(in_features=48, out_features=32)
        self.out = nn.Linear(in_features=32, out_features=2)

    def forward(self, b):
        b = b.flatten(start_dim=1)
        #t = F.relu(clayer_out)
        b = F.relu(self.fc1(b))
        b = F.relu(self.fc2(b))
        b = F.relu(self.fc3(b))
        b = self.out(b)
        return b

is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython: from IPython import display

Experience = namedtuple(
    'Experience',
    ('state', 'action', 'next_state', 'reward')
) # use experiences to train network

class ReplayMemory():
    def __init__(self, capacity): # replay memory has set capacity, only need to
        self.capacity = capacity # initialize capacity for a ReplayMemory object
        self.memory = [] #memory will hold the experiences
        self.push_count = 0 #to show how many experiences we've added to memory

    def push(self, experience): # to add experiences in memory
        if len(self.memory) < self.capacity: # ensuring that the length of memory doesn't exceed set capacity
            self.memory.append(experience)
        else:
            self.memory[self.push_count % self.capacity] = experience # if memory greater than capacity,
         # then the new experiences will be put to the front of memory, erasing the
                            # oldest experiences in the memory array
        self.push_count += 1
    def sample(self, batch_size): # returns a randome sample of experiences, to later use for
        return random.sample(self.memory, batch_size) #random.sample(sequence, k), of the sequence, gives k randomly chosen experiences

    def can_provide_sample(self, batch_size): #to sample, batch size needs to be bigger than memory -- this is important at the beginning
        return len(self.memory) >= batch_size

class EpsilonGreedyStrategy(): #explor vs. exploitation
    def __init__(self, start, end, decay):
        self.start = start
        self.end = end
        self.decay = decay

    def get_exploration_rate(self, current_step):  ####this was not explained in the video, woher kommt
        return self.end + self.start*(1/(1+self.decay*current_step))#self.end + (self.start - self.end) * \
            #math.exp(-1. * current_step * self.decay)

class LearningRate():
    def __init__(self, start, end, decay, current_step):
        self.start = start
        self.end = end
        self.decay = decay
        self.current_step = current_step
    def get_learning_rate(self, current_step):
        self.current_step += 1
        return self.end + self.start*(1/(1+self.decay*current_step))

class lr(): # learning rate class needed. Left for possible future use, need to update things beforehand
    def __init__(self, learning_rate):
        self.current_step = 0
        self.learning_rate = learning_rate
    def update_lr(self):
        lrrate = learning_rate.get_learning_rate(self.current_step)
        self.current_step +=1
class Agent():
    def __init__(self, strategy, num_actions, device): # when we later create an agent object, need to get strategy from epsilon, num_actions = how many actions from a given state (2 for this game), device is the device in pytorch for tensor calculations CPU or GPU
        self.current_step = 0 # current step number in the environment
        self.strategy = strategy
        self.num_actions = num_actions
        self.device = device

    def select_action(self, state, policy_net): #policy_net is the policy trained by DQN
        rate = strategy.get_exploration_rate(self.current_step)
        self.current_step += 1

        if rate > random.random():
            action = random.randrange(self.num_actions)
            return torch.tensor([action]).to(self.device) # explore
        else:
            with torch.no_grad(): #turn off gradient tracking
                return policy_net(state).argmax(dim=1).to(self.device) # exploit

class CartPoleEnvManager(): #manage cartpole env
    def __init__(self, device):
        self.device = device
        self.env = gym.make('CartPole-v0').unwrapped #unwrapped so we can see other dynamics that there is no access to otherwise
        self.env.reset() #reset to starting state
        self.current_screen = None #screen initialization
        self.done = False #if episode is finished

    def reset(self):
        self.env.reset()
        self.current_screen = None

    def close(self): #close env
        self.env.close()

    def render(self, mode='human'): #render the current state to screen
        return self.env.render(mode)

    def num_actions_available(self): #returns # actions available to agent (2)
        return self.env.action_space.n

    def take_action(self, action):# step returns tuple containing env observation, reward and diagnostic info -- all from taking a certain action
        _, reward, self.done, _ = self.env.step(action.item()) # only reward and done status are of importance
        return torch.tensor([reward], device=self.device)
    #####action is a tensor, action.item() gives a number, what step wants

    def just_starting(self):
        return self.current_screen is None

    def get_state(self): #return to the current state of env in the form of a processed image of the screen
        if self.just_starting() or self.done:
            self.current_screen = self.get_processed_screen() #state = processed image of diff of 2 separate screens
            black_screen = torch.zeros_like(self.current_screen)
            return black_screen
        else:
            s1 = self.current_screen
            s2 = self.get_processed_screen() ####what is get_processed_screen?
            self.current_screen = s2
            return s2 - s1 # this represents a single state

    def get_screen_height(self):
        screen = self.get_processed_screen()
        return screen.shape[2]

    def get_screen_width(self):
        screen = self.get_processed_screen()
        return screen.shape[3]

    def get_processed_screen(self):
        screen = self.env.render(mode='rgb_array').transpose((2, 0, 1)) # PyTorch expects CHW
        screen = self.crop_screen(screen)
        return self.transform_screen_data(screen)

    def crop_screen(self, screen):
        screen_height = screen.shape[1]

        # Strip off top and bottom
        top = int(screen_height * 0.4)
        bottom = int(screen_height * 0.8)
        screen = screen[:, top:bottom, :]
        return screen

    def transform_screen_data(self, screen):
        # Convert to float, rescale, convert to tensor
        screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
        screen = torch.from_numpy(screen)

        # Use torchvision package to compose image transforms
        resize = T.Compose([
            T.ToPILImage()
            ,T.Resize((40,90))
            ,T.ToTensor()
        ])

        return resize(screen).unsqueeze(0).to(self.device) # add a batch dimension (BCHW)

def plot(values, moving_avg_period):
    plt.figure(2)
    plt.clf()
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(values)

    moving_avg = get_moving_average(moving_avg_period, values)
    plt.plot(moving_avg)
    plt.pause(0.001)
    print("Episode", len(values), "\n", \
          moving_avg_period, "episode moving avg:", moving_avg[-1])
    if is_ipython: display.clear_output(wait=True)

def get_moving_average(period, values):
    values = torch.tensor(values, dtype=torch.float)
    if len(values) >= period:
        moving_avg = values.unfold(dimension=0, size=period, step=1) \
            .mean(dim=1).flatten(start_dim=0)
        moving_avg = torch.cat((torch.zeros(period-1), moving_avg))
        return moving_avg.numpy()
    else:
        moving_avg = torch.zeros(len(values))
        return moving_avg.numpy()

def extract_tensors(experiences):
    # Convert batch of Experiences to Experience of batches
    batch = Experience(*zip(*experiences))

    t1 = torch.cat(batch.state)
    t2 = torch.cat(batch.action)
    t3 = torch.cat(batch.reward)
    t4 = torch.cat(batch.next_state)

    return (t1,t2,t3,t4)

class QValues():
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    @staticmethod
    def get_current(policy_net, states, actions):
        return policy_net(states).gather(dim=1, index=actions.unsqueeze(-1))

    @staticmethod
    def get_next(target_net, next_states):
        final_state_locations = next_states.flatten(start_dim=1) \
            .max(dim=1)[0].eq(0).type(torch.bool)
        non_final_state_locations = (final_state_locations == False)
        non_final_states = next_states[non_final_state_locations]
        batch_size = next_states.shape[0]
        values = torch.zeros(batch_size).to(QValues.device)
        values[non_final_state_locations] = target_net(non_final_states).max(dim=1)[0].detach()
        return values

batch_size = 128
gamma = 0.999
eps_start = 1
eps_end = 0.01
eps_decay = 0.0005
target_update = 10
memory_size = 500000
lr_start = 0.01
lr_end = 0.00001
lr_decay = 0.00009
num_episodes = 1000 # run for more episodes for better results

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
em = CartPoleEnvManager(device)
strategy = EpsilonGreedyStrategy(eps_start, eps_end, eps_decay)
agent = Agent(strategy, em.num_actions_available(), device)
memory = ReplayMemory(memory_size)

policy_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net.load_state_dict(policy_net.state_dict())

target_net.eval() #tells pytorch that target_net is only used for inference, not training
optimizer = optim.Adam(params=policy_net.parameters(), lr=0.01)

i = 0
episode_durations = []
for episode in range(num_episodes): #iterate over each episode
    em.reset()
    state = em.get_state()

    for timestep in count():
        action = agent.select_action(state, policy_net)
        reward = em.take_action(action)
        next_state = em.get_state()
        memory.push(Experience(state, action, next_state, reward))
        state = next_state
        i = 0
        if memory.can_provide_sample(batch_size):
            scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)
            experiences = memory.sample(batch_size)
            states, actions, rewards, next_states = extract_tensors(experiences)

            current_q_values = QValues.get_current(policy_net, states, actions)
            next_q_values = QValues.get_next(target_net, next_states) #will get the max qvalues of the next state, q values of next state are used via next state
            target_q_values = (next_q_values * gamma) + rewards

            loss = F.mse_loss(current_q_values, target_q_values.unsqueeze(1))
            optimizer.zero_grad() # sets the gradiesnt of all weights n biases in policy_net to zero
            loss.backward() #computes gradient of loss with respect to all weights n biases in the policy net
            optimizer.step() # updates the weights n biases with the gradients that were computed form loss.backwards
            scheduler.step()

        if em.done:
            episode_durations.append(timestep)
            plot(episode_durations, 100)
            break
    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())

em.close()

Output:
You're using a "hacked" (or patched if you will) version of  CartPole environment which in effect replacing real state CartPole-v0 returns with rendered image. So your code is trying to train a policy which is taking image as an input rather than 4-values feature array original CartPole-v0 is returning.

If you look closer when you call

 _, reward, self.done, _ = self.env.step(action.item())

the first element _ is actual state of original CartPole-v0 env.

Then instead of using that the class you have is doing rendering and returning image as input for training.

So for the existing task (effectively state is an image) you can't really skip rendering since it is a part of preparing inputs for the policy.


Instruction: TransformerEncoder with a padding mask

Input:
I'm trying to implement torch.nn.TransformerEncoder with a src_key_padding_mask not equal to none. Imagine the input is of the shape src = [20, 95] and the binary padding mask has the shape src_mask = [20, 95], 1 in the position of padded tokens and 0 for other positions. I make a transformer encoder with 8 layers, each of which contain an attention with 8 heads and hidden dimension 256:

layer=torch.nn.TransformerEncoderLayer(256, 8, 256, 0.1)
encoder=torch.nn.TransformerEncoder(layer, 6)
embed=torch.nn.Embedding(80000, 256)
src=torch.randint(0, 1000, (20, 95))
src = emb(src)
src_mask = torch.randint(0,2,(20, 95))
output =  encoder(src, src_mask)

But I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-107-31bf7ab8384b> in <module>
----> 1 output =  encoder(src, src_mask)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/transformer.py in forward(self, src, mask, src_key_padding_mask)
    165         for i in range(self.num_layers):
    166             output = self.layers[i](output, src_mask=mask,
--> 167                                     src_key_padding_mask=src_key_padding_mask)
    168
    169         if self.norm:

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/transformer.py in forward(self, src, src_mask, src_key_padding_mask)
    264         """
    265         src2 = self.self_attn(src, src, src, attn_mask=src_mask,
--> 266                               key_padding_mask=src_key_padding_mask)[0]
    267         src = src + self.dropout1(src2)
    268         src = self.norm1(src)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/activation.py in forward(self, query, key, value, key_padding_mask, need_weights, attn_mask)
    781                 training=self.training,
    782                 key_padding_mask=key_padding_mask, need_weights=need_weights,
--> 783                 attn_mask=attn_mask)
    784
    785

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v)
   3250     if attn_mask is not None:
   3251         attn_mask = attn_mask.unsqueeze(0)
-> 3252         attn_output_weights += attn_mask
   3253
   3254     if key_padding_mask is not None:

RuntimeError: The size of tensor a (20) must match the size of tensor b (95) at non-singleton dimension 2

I was wondering if somebody could help me figure out this problem.

Thanks

Output:
The required shapes are shown in nn.Transformer.forward - Shape (all building blocks of the transformer refer to it). The relevant ones for the encoder are:

src: (S, N, E)
src_mask: (S, S)
src_key_padding_mask: (N, S)

where S is the sequence length, N the batch size and E the embedding dimension (number of features).

The padding mask should have shape [95, 20], not [20, 95]. This assumes that your batch size is 95 and the sequence length is 20, but if that is the other way around, you would have to transpose the src instead.

Furthermore, when calling the encoder, you are not specifying the src_key_padding_mask, but rather the src_mask, as the signature of torch.nn.TransformerEncoder.forward is:

forward(src, mask=None, src_key_padding_mask=None)

The padding mask must be specified as the keyword argument src_key_padding_mask not as the second positional argument. And to avoid confusion, your src_mask should be renamed to src_key_padding_mask.

src_key_padding_mask = torch.randint(0,2,(95, 20))
output =  encoder(src, src_key_padding_mask=src_key_padding_mask)


Instruction: tf.test.is_gpu_available() is False in subprocess but True in main process

Input:
I'm currently running a pytorch model which periodically calls out to a tensorflow model for benchmarking purposes. I'd like both of these models to be GPU-enabled and to run in the same script. Since tensorflow benchmarking code claims GPU memory til the end of the process, I've elected to run the benchmarking code in a multiprocessing.Process so that my pytorch model can use the full GPU's memory after the benchmarking script has run.

During this, I've stumbled across an unusual bug (?) in tensorflow's gpu utilization. It seems that tensorflow run in a subprocess doesn't want to use a GPU which has been used ~at all~ by a parent process. I can have tensorflow models and pytorch models in the same GPU and process with no problems, but when I introduce subprocesses tensorflow is ill-behaved.

I'm running
tensorflow-gpu==1.14.0
torch==1.1.0
cudatoolkit=10.0 on an NVIDIA 2080-Ti.

Below is a minimal code snipped to reproduce:

import torch
import tensorflow as tf
from multiprocessing import Process

def f():
    print(tf.test.is_gpu_available())

pa = Process(target=f, args=())
pa.start()
pa.join()

torch.ones(1).cuda()

pb = Process(target=f, args=())
pb.start()
pb.join()

>>> True
>>> False

Output: To anyone running into this problem, you need to call multiprocessing.set_start_method('spawn'). Tensorflow is not fork-safe and some weirdness can happen with global variables/modules that is probably very hard to reason about. Remember to call it only once, inside a if __name__ == '__main__': check.


Instruction: Runtime error while running PyTorch model on local machine

Input:
I'm running this notebook locally

https://github.com/udacity/deep-learning-v2-pytorch/blob/master/sentiment-rnn/Sentiment_RNN_Solution.ipynb
everything was working just until I started training the model

# training params

epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing

counter = 0
print_every = 100
clip=5 # gradient clipping

# move model to GPU, if available
if(train_on_gpu):
    net.cuda()

net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        if(train_on_gpu):
            inputs, labels = inputs.cuda(), labels.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])

                if(train_on_gpu):
                    inputs, labels = inputs.cuda(), labels.cuda()

                output, val_h = net(inputs, val_h)
                val_loss = criterion(output.squeeze(), labels.float())

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

the error that occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-31-9f7dea11cb7b> in <module>
     32
     33         # get the output from the model
---> 34         output, h = net(inputs, h)
     35
     36         # calculate the loss and perform backprop

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-16-b99cefc1dc61> in forward(self, x, hidden)
     36
     37         # embeddings and lstm_out
---> 38         embeds = self.embedding(x)
     39         lstm_out, hidden = self.lstm(embeds, hidden)
     40

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    110
    111     def forward(self, input):
--> 112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
    114             self.norm_type, self.scale_grad_by_freq, self.sparse)

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722         # remove once script supports set_grad_enabled
   1723         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1724     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1725
   1726

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)

I don't understand why it's happening. I tried to find a solution online. which said I need transfer my model and data to GPU. I did but the problem still persists.

Output:
You are trying to embed the inputs, which are given as ints (torch.int). Only integers (torch.long) can be embedded, since they need to be indices, which cannot be float.

inputs need to be converted to torch.long:

inputs = inputs.to(torch.long)

It seems like you removed the conversion to long, because in the notebook it's done within the model:

# embeddings and lstm_out
x = x.long()
embeds = self.embedding(x)

Whereas, in your stack trace, the line x = x.long() (same as using .to(torch.long)) is missing.


Instruction: ValueError: too many values to unpack while using torch tensors

Input:
For a project on neural networks, I am using Pytorch and am working with the EMNIST dataset.

The code that is already given loads in the dataset:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

And prepares it:

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

Then, when all the configurations of the network are defined, there is a for loop to train the model per epoch:

 for i, (images, labels) in enumerate(train_loader):

In the example code this works fine.

For my task, I am given a dataset that I load as follows:

emnist = scipy.io.loadmat("DIRECTORY/emnist-letters.mat")

data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
y_train = data ['train'][0, 0]['labels'][0, 0]

Then, I create the train_dataset as follows:

train_dataset = np.concatenate((X_train, y_train), axis = 1)
train_dataset = torch.from_numpy(train_dataset)

And use the same step to prepare it:

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

However, when I try to use the same loop as before:

for i, (images, labels) in enumerate(train_loader):

I get the following error:

ValueError: too many values to unpack (expected 2)

Who knows what I can do so that I can train my dataset with this loop?

Output:
The dataset you created from the EMNIST data is a single tensor, and therefore, the data loader will also produce a single tensor, where the first dimension is the batch dimensions. This results in trying to unpack that tensor across the batch dimension, which doesn't work because your batch size is greater than two, but is also not what you want to happen.

You can use torch.utils.data.TensorDataset to easily create a dataset, which produces a tuple of images and their respective labels, just like the MNIST dataset does.

train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))


Instruction: Cannot obtain same output tensor values in C++ side

Input:
I try to make predictions for a 4-class classification project in C++ side using libtorch 1.4. However, I cannot obtain same predictions compared to Python side. Firstly, I obtain same input tensor values just before prediction. When I compare the output tensor values, I noticed that they are different. You can find those values in that picture:

Left side includes Python output tensor values and the prediction results for each input picture.

Right side includes C++ output tensor values and the prediction results for each input picture.

Could you offer a solution to obtain same output tensor values and prediction results?

Output:
I noticed that I used opencv functions to apply normalization using this code:

subtract(image, Scalar(0.485, 0.456, 0.406), temp);
divide(temp, Scalar(0.229, 0.224, 0.225), image);

This operation changes only the first channel and does not change other channels. For this reason, in fact, input tensor values are different. I applied normalization directly on tensor values writing this code:

tensor_image = tensor_image.permute({ 2,0,1 });//chw
tensor_image = tensor_image.toType(torch::kFloat);
tensor_image = tensor_image.div(255.0);
//normalize
tensor_image[0] = tensor_image[0].sub_(0.485).div_(0.229);
tensor_image[1] = tensor_image[1].sub_(0.456).div_(0.224);
tensor_image[2] = tensor_image[2].sub_(0.406).div_(0.225);

Thus, I obtained the same input tensor values compared to Python side. After prediction, I obtained the same output tensor values. My problem solved.


Instruction: Train inactive class in classification AI problem

Input:
I'm facing an issue. I want to classify data in n classes. But, I also want to say that, the answer is none of these n classes.

For example, my classes are : HORSE, CAT, DOG. So I will train with data related to HORSE, CAT and DOG.

But if I give my model something else, just like a car, I would like my model to tell that it is not HORSE or DOG or CAT.
So maybe I have to train a model with these classes : HORSE, CAT, DOG, OTHER ?

But If I train OTHER class with data, how can I be sure that if I give it something new like a spacerocket, the prediction will be active for the OTHER class ?

In other words, I'm in trouble with inactive class. I don't want my model to give "the best prediction" beetween my desired classes, I also want it to tell me if it is none of them.

Thank you !

Output:
I had some answer, and it seems that there are 2 main solutions for this problem :

As I thought, I can train my classes and 1 more for the other cases. To be robust, I have to create data for this class that my model may have in input to cover as much as cases as possible !
Train n classifier : 1 binary classifier for 1 class. Each one have these outputs : Active / Not Active.

For each solution, I think it is necessary to create dataset for the "other cases", even in the second one to avoid uncontained prediction. In Solution 1, you can choose if you want 1 active class (softmax) or several active classes (sigmoid), in solution 2, it seems harder to control this behavior as each classifier is independent and several classes could be active.

source


Instruction: PyTorch: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True

Input:
This is an error message I get as I work with some synthetic data. I am a bit puzzled because the error persists although I do what I am advised to do.  Could it somehow be related to the fact that I do not specify a batch?  Would the use of PyTorch Dataset relieve the problem?

Here is my code (I am new in PyTorch, just learning it now) -- it should be reproducible:

Data Creation:

x, y = np.meshgrid(np.random.randn(100) , np.random.randn(100))
z = 2 * x + 3 * y + 1.5 * x * y - x ** 2 - y**2
X = x.ravel().reshape(-1, 1)
Y = y.ravel().reshape(-1, 1)
Z = z.ravel().reshape(-1, 1)
U = np.concatenate([X, Y], axis = 1)
U = torch.tensor(U, requires_grad=True)
Z = torch.tensor(Z, requires_grad=True)
V = []

for i in range(U.shape[0]):
    u = U[i, :]
    u1 = u.view(-1, 1) @ u.view(1, -1)
    u1 = u1.triu()
    ones = torch.ones_like(u1)
    mask = ones.triu()
    mask = (mask == 1)
    u2 = torch.masked_select(u1, mask)
    u3 = torch.cat([u, u2])
    u3 = u3.view(1, -1)
    V.append(u3)

V = torch.cat(V, dim = 0)

Training a Model

from torch import nn
from torch import optim
net = nn.Sequential(nn.Linear(V.shape[1], 1))
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(50):  # loop over the dataset multiple times
    running_loss = 0.0
    i = 0
    for inputs , labels in zip(V, Z):
        # get the inputs; data is a list of [inputs, labels]

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)

        loss.backward(retain_graph = True)

        optimizer.step()

        # print statistics
        running_loss += loss.item()

        i += 1

        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

Error Message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-143-2454f4bb70a5> in <module>
     25
     26
---> 27         loss.backward(retain_graph = True)
     28
     29         optimizer.step()

~\Anaconda3\envs\torch\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--> 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196
    197     def register_hook(self, hook):

~\Anaconda3\envs\torch\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Could you explain the error and fix the code?

Output:
Presumably, you haven't re-run the data creation code again after setting retain_graph=True, as it is in a IPython REPL. It would work with that, but in nearly all cases, setting retain_graph=True is not the appropriate solution.

The problem in your case, is that you have set requires_grad=True for U, which means that everything that involved U in the data creation, will be recorded in the computational graph and when loss.backward() is called, the gradients will propagate through all of these up to U. After the first time, all buffers for the gradients of those will have been freed, and a second backward will fail.

Neither U nor Z should have requires_grad=True, as they are not being optimised/learned. Only the learned parameters (parameters given to the optimiser) should have requires_grad=True and generally, you don't have to set it manually either, since nn.Parameter handles that automatically.

You should also ensure that the tensors you create from NumPy data have type torch.float (float32) as NumPy's float array will usually be float64, which are mostly unnecessary and just slower compared to float32, especially on the GPU.

U = torch.tensor(U, dtype=torch.float)

Z = torch.tensor(Z, dtype=torch.float)

And remove the retain_graph=True from the backward call:

loss.backward()


Instruction: Writing tensor to a file in a visually readable manner

Input: In pytorch, I want to write a tensor to a file and visually read the file contents. For example, consider T = torch.tensor([3,4,5,6]). I want to write the tensor T to a file, say file_T.txt, and want to visually read the contents of the file_T.txt, which will be 3,4,5 and 6. How can I achieve this?

Output:
You can use numpy:

import numpy as np
np.savetxt('my_file.txt', torch.Tensor([3,4,5,6]).numpy())


Instruction: Why is a simple Binary classification failing in a feedforward neural network?

Input:
I am new to Pytorch. I was trying to model a binary classifier on the Kepler dataset. The following was my dataset class.

class KeplerDataset(Dataset):
    def __init__(self, test=False):
        self.dataframe_orig = pd.read_csv(koi_cumm_path)

        if (test == False):
            self.data = df_numeric[( df_numeric.koi_disposition == 1 ) | ( df_numeric.koi_disposition == 0 )].values
        else:
            self.data = df_numeric[~(( df_numeric.koi_disposition == 1 ) | ( df_numeric.koi_disposition == 0 ))].values

        self.X_data = torch.FloatTensor(self.data[:, 1:])
        self.y_data = torch.FloatTensor(self.data[:, 0])

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]

Here, I created a custom classifier class with one hidden layer and a single output unit that produces sigmoidal probability of being in class 1 (planet).

class KOIClassifier(nn.Module):
    def __init__(self, input_dim, out_dim):
        super(KOIClassifier, self).__init__()
        self.linear1 = nn.Linear(input_dim, 32)

        self.linear2 = nn.Linear(32, 32)

        self.linear3 = nn.Linear(32, out_dim)

    def forward(self, xb):
        out = self.linear1(xb)
        out = F.relu(out)
        out = self.linear2(out)
        out = F.relu(out)
        out = self.linear3(out)
        out = torch.sigmoid(out)

        return out

I then created a train_model function to optimize the loss using SGD.

def train_model(X, y):
    criterion = nn.BCELoss()

    optim = torch.optim.SGD(model.parameters(), lr=0.001)

    n_epochs = 100
    losses = []

    for epoch in range(n_epochs):
        y_pred = model.forward(X)
        loss = criterion(y_pred, y)
        losses.append(loss.item())
        optim.zero_grad()
        loss.backward()
        optim.step()

losses = []
for X, y in train_loader:
    losses.append(train_model(X, y))

But after performing the optimization over the train_loader, When I try predicting on the trainn_loader itself, the prediction values are so much worse.

for features, y in train_loader:
    y_pred = model.predict(features)
    break

y_pred

> tensor([[4.5436e-02],
        [1.5024e-02],
        [2.2579e-01],
        [4.2279e-01],
        [6.0811e-02],
        .....

Why is my model not working properly? Is it the problem with the dataset or am I doing something wrong with implementing the Neural net? I will link my Kaggle notebook because more context might be helpful. Please help.

Output:
You are optimizing many times (100 steps) on the first batch (first samples), then moving to the next samples. It means that your model will overfit your few samples before going to the next batch. Then, your training will be very non smooth, diverge and go far from your global optimum.

Usually, in a training loop you should:

go over all samples (this is one epoch)
shuffle your dataset in order to visit your samples in a different order (set your pytorch training loader accordingly)
go back to 1. until you reach the max number of epochs

Also you should not define your optimizer each time (nor your criterion).

Your training loop should look like this:

criterion = nn.BCELoss()
optim = torch.optim.SGD(model.parameters(), lr=0.001)
n_epochs = 100

def train_model():
    for X, y in train_loader:
        optim.zero_grad()
        y_pred = model.forward(X)
        loss = criterion(y_pred, y)
        loss.backward()
        optim.step()

for epoch in range(n_epochs):
    train_model()


Instruction: pytorch 4d numpy array applying transfroms inside custom dataset

Input:
Inside my custom dataset, I want to apply transforms.Compose() to a NumPy array.

My images are in a NumPy array format with shape (num_samples, width, height, channels).

How can I apply the follwoing transforms to the full numpy array?

img_transform = transforms.Compose([
        transforms.Scale((224,224)),
        transforms.ToTensor(),
        transforms.Normalize([0.46, 0.48, 0.51], [0.32, 0.32, 0.32])
    ])

My attempts are ending in multiple errors as the transforms accept a PIL image not a 4-d NumPy array.

from torchvision import transforms
import numpy as np
import torch

img_transform = transforms.Compose([
        transforms.Scale((224,224)),
        transforms.ToTensor(),
        transforms.Normalize([0.46, 0.48, 0.51], [0.32, 0.32, 0.32])
    ])

a = np.random.randint(0,256, (299,299,3))
print(a.shape)

img_transform(a)

Output:
All torchvision transforms operate on single images, not batches of images, hence a 4D array cannot be used.

Single images given as NumPy arrays, like in your code example, can be used by converting them to a PIL image. You can simply add transforms.ToPILImage to the beginning of the transformation pipeline, as it converts either a tensor or a NumPy array to a PIL image.

img_transform = transforms.Compose([
        transforms.ToPILImage(),
        transforms.Resize((224,224)),
        transforms.ToTensor(),
        transforms.Normalize([0.46, 0.48, 0.51], [0.32, 0.32, 0.32])
    ])

Note: transforms.Scale is deprecated in favour of transforms.Resize.

In your example you used np.random.randint, which by default uses type int64, but images have to be uint8. Libraries such as OpenCV return uint8 arrays when loading an image.

a = np.random.randint(0,256, (299,299,3), dtype=np.uint8)


Instruction: How do I clamp the magnitude of a PyTorch tensor?

Input:
I know I can use torch.clamp to clamp a tensor's values within some min / max, but how can I do this if I want to clamp by the magnitude (absolute value)? Example:

import torch
t = torch.tensor([-5.0, -250, -1, 0.003, 7, 1238])
min_mag = 1 / 10
max_mag = 100

# desired output:
tensor([  -5.0000, -100.0000,   -1.0000,    0.1000,    7.0000,  100.0000])

Output:
Here's one method:

sign = t.sign()
t = t.abs_().clamp_(min_mag, max_mag)
t *= sign

(note: this is using in-place operations)


Instruction: Why am I able to change the value of a tensor without the computation graph knowing about it in Pytorch with detach?

Input:
I can change the value of a tensor that requires grad without autograd knowing about it:

def error_unexpected_way_to_by_pass_safety():
    import torch
    a = torch.tensor([1,2,3.], requires_grad=True)
    # are detached tensor's leafs? yes they are
    a_detached = a.detach()
    #a.fill_(2) # illegal, warns you that a tensor which requires grads is used in an inplace op (so it won't be recorded in computation graph so it wont take the right derivative of the forward path as this op won't be in it)
    a_detached.fill_(2) # weird that this one is allowed, seems to allow me to bypass the error check from the previous comment...?!
    print(f'a = {a}')
    print(f'a_detached = {a_detached}')
    a.sum().backward()

this throws no errors. Though, I am able to change the contents of a which is a tensor that requires grad without autograd knowing about. Which means the computation graph does not know about this op (filling with 2). This seems wrong. Can anyone shed light what is going on?

Output:
.detach gives you a view on the same data, so modifying the data of the detached tensor modifies the data of the original. You can check this like so:

a.data_ptr() == a_detached.data_ptr() # True

As for why this is how .detach is implemented (as opposed to doing a defensive copy), that's a design question that only the PyTorch authors know the answer to. I assume that it's to save unnecessary copies, but users then need to be aware that they have to copy the tensors themselves if they want to modify the detached ones in-place.

Note that you can also alter the non-detached tensor if you really want to:

a.data.fill_(2)

PyTorch isn't trying to stop you from "hacking" autograd; users still have to be aware of how to use tensors properly so that gradients will be tracked correctly.


Instruction: Updating learning rate with Libtorch 1.5 and optimiser options in C++

Input:
With the release of the 1.5 stable version of the C++ API for PyTorch, there are some changes in some of the object interfaces. For instance, now

optimizer.options.learning_rate();

won't work (here the optimiser being used is Adam) since learning_rate has changed to lr (see https://github.com/pytorch/pytorch/releases) but moreover the optimiser no longer has options (no member named 'options' in 'torch::optim::Adam'). So my question is: how would one run

optimizer.options.learning_rate();

or update the learning rate

optimizer.options.learning_rate(updatedlearningrate);

with the new release? Any help will be appreciated! Thank you

Output:
The optimisers now behave like their Python counterparts and the learning rates need to be set per parameter group.

for (auto param_group : optimizer.param_groups()) {
  # Static cast needed as options() returns OptimizerOptions (base class)
  static_cast<torch::optim::AdamOptions &>(param_group.options()).lr(new_lr);
}

If you didn't specify separate parameter groups, there will be only a single group and you could directly set its learning rate as suggested in Issue #35640 - How do you change Adam learning rate since the latest commits?:

static_cast<torch::optim::AdamOptions &>(optimizer.param_groups()[0].options()).lr(new_lr)


Instruction: RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead

Input:
RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead

This is the portion of code I think where the problem is.

def __init__(self):
        super(Lightning_CNNModel, self).__init__()

        self.conv_layer1 = self._conv_layer_set(3, 32)
        self.conv_layer2 = self._conv_layer_set(32, 64)
        self.fc1 = nn.Linear(2**3*64, 128)
        self.fc2 = nn.Linear(128, 10)   # num_classes = 10
        self.relu = nn.LeakyReLU()
        self.batch=nn.BatchNorm1d(128)
        self.drop=nn.Dropout(p=0.15)

    def _conv_layer_set(self, in_c, out_c):
        conv_layer = nn.Sequential(
            nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0),
            nn.LeakyReLU(),
            nn.MaxPool3d((2, 2, 2)),
        )

        return conv_layer

    def forward(self, x):
        out = self.conv_layer1(x)
        out = self.conv_layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.relu(out)
        out = self.batch(out)
        out = self.drop(out)
        out = self.fc2(out)

        return out

This is the code I am working on

Output:
nn.Conv3d expects the input to have size [batch_size, channels, depth, height, width]. The first convolution expects 3 channels, but with your input having size [100, 16, 16, 16, 3], that would be 16 channels.

Assuming that your data is given as [batch_size, depth, height, width, channels], you need to swap the dimensions around, which can be done with torch.Tensor.permute:

# From: [batch_size, depth, height, width, channels]
# To: [batch_size, channels, depth, height, width]
input = input.permute(0, 4, 1, 2, 3)