Instruction: Specify GCC version for nvcc without root priviledges

Input:
I am using a GPU cluster where the submitted jobs are managed by Slurm. I don't have admin / root priviledges on that server. I am currently trying to build a project that contains .cpp and .cu files. I do that by calling TORCH_CUDA_ARCH_LIST=7.2 CC=gcc-7 CXX=g++-7 python setup.py install, as the cluster uses CUDA 10.1 and runs V100 GPUs (hence the gencode is sm_70).
However, the build crashes with the following error message:
building <filename> extension
gcc-7 -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes (...):
error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
error: command '/<somepath>/anaconda3/envs/pytorch14/bin/nvcc' failed with exit status 1

So, as one can see by the gcc-7 call in the 2nd line, the python script is using the right compiler, but unfortunately, the nvcc call uses the system-wide gcc symlink, which is: /usr/bin/gcc: symbolic link to gcc-9. I have found a couple of answers online (including this and this) and have tried the suggested steps. But: as I don't have root access, I cannot create a new symlink / change the existing symlink to another installed gcc version, e.g. /usr/bin/gcc-7: doing ln -s /usr/bin/gcc-7 /usr/bin/gcc gives me a ln: failed to create symbolic link '/usr/bin/gcc': File exists error, and copying the files into /usr/local/bin, as suggested in other answers on SO, wont work either because of the missing priviledges.
I'm really at a loss here and feel that this might be a dead end. Does anybody have any suggestions?
For reference, this is what my setup.py looks like:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='noise_cuda',
    ext_modules=[
        CUDAExtension('noise_cuda', [
            'noise_cuda.cpp',
            'noise_cuda_kernel.cu',
        ]),
    ],
    cmdclass={
        'build_ext': BuildExtension
    })

Output:
I'm not a pytorch user, but if I read the docs right, this should work:

import sysconfig
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

setup(
    name='noise_cuda',
    ext_modules=[
        CUDAExtension('noise_cuda', [
            'noise_cuda.cpp',
            'noise_cuda_kernel.cu',
        ], extra_compile_args={'cxx': sysconfig.get_config_var('CFLAGS').split(),
                               'nvcc': ['-ccbin=/usr/bin/gcc-7']}),
    ],
    cmdclass={
        'build_ext': BuildExtension
    })


Instruction: How do I find the number of elements taking up memory for an expanded view of a pytorch tensor?

Input:
Tensor.expand() returns a new view of an underlying tensor, but doesn't actually allocate more memory for the expanded view.
If I have a tensor that is the result of calling expand() (some unknown number of times), how can I tell how many cells are actually allocated in memory for the tensor (in my actual use-case, I really just care about knowing whether or not that number is 1)?
Is there something like what I'm calling elements_in_memory as used in the following?:
import torch

t = torch.tensor(4.0)
t2 = t.expand(3, 4)
t3 = t2.unsqueeze(0).expand(5, 3, 4)

# I'm looking for something like this (which doesn't work)
assert t.elements_in_memory == 1
assert t2.elements_in_memory == 1
assert t3.elements_in_memory == 1

Some things I've tried:

t.data_ptr refers to the first element of the underlying tensor in memory, so t.data_ptr == t2.data_ptr, but that doesn't tell me how many elements.

Output:
It looks like t.storage().size() is what I'm after.
From the torch.Tensor documentation:

Each tensor has an associated torch.Storage, which holds its data. The tensor class also provides multi-dimensional, strided view of a storage and defines numeric operations on it.

Tensor.storage() returns a reference to the storage used for the tensor:
import torch

t = torch.tensor(4.0)
t2 = t.expand(3, 4)
t3 = t2.unsqueeze(0).expand(5, 3, 4)

assert t.storage().size() == 1
assert t2.storage().size() == 1
assert t3.storage().size() == 1

t4 = torch.ones(3, 4)
t5 = t4.unsqueeze(0).expand(5, 3, 4)

assert t4.storage().size() == 12
assert t5.storage().size() == 12

Note that the underlying storage might also include more elements than are exposed by some particular view (this wasn't relevant to my use-case).  For example torch.ones(10)[3:6].storage().size() == 10.


Instruction: Reduce torch tensor

Input:
For a boolean tensor of shape (15,10), I want to perform bitwise_or along axis 0 so that the resulting tensor would be of shape 10. torch.bitwise_or does not support this.
I know it can done in numpy using np.bitwise_or.reduce(x,axis=0). I did not find something similar in torch. How to reduce torch tensor?

Output:
Hi figured out the problem here if you look at the docstring for the reduce function it's essentially just a for loop adding itself from 0
# ufunc docstring
# op.identiy is 0
r = op.identity # op = ufunc
for i in range(len(A)):
  r = op(r, A[i])
return r

So to solve and fix your problem
import numpy as np
import torch

bool_arr = np.random.randint(0, 2, (15, 10), dtype=np.bool) # create a bool arr
tensor_bool_arr = torch.tensor(bool_arr) # Create torch version
output_np = np.bitwise_or.reduce(bool_arr, axis=0)
# array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True])

# Create a pytorch equivalent of bitwise reduce
r = torch.tensor(0)
for i in range(len(tensor_bool_arr)):
    r = torch.bitwise_or(r, tensor_bool_arr[i])

torch_output = r.type(torch.bool)
# tensor([True, True, True, True, True, True, True, True, True, True])
assert torch_output.shape[0] == np_output.shape[0]


Instruction: I get this error using PyTorch: RuntimeError: gather_out_cpu(): Expected dtype int64 for index

Input:
I'm trying to make an AI with PyTorch, but I get this error:
RuntimeError: gather_out_cpu(): Expected dtype int64 for index

And this is my function:
def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
    outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
    next_outputs = self.model(batch_next_state).detach().max(1)[0]
    target = self.gamma * next_outputs + batch_reward
    td_loss = F.smooth_l1_loss(outputs, target)
    self.optimizer.zero_grad()
    td_loss.backward(retain_variables = True)
    self.optimizer.step()

Output:
You need to change the data type of your batch_action tensor before passing it to torch.gather.
def learn(...):
    batch_action = batch_action.type(torch.int64)
    outputs = ...
    ...

# or
outputs = self.model(batch_state).gather(1, batch_action.type(torch.int64).unsqueeze(1)).squeeze(1)


Instruction: Torch mv behavior not understandable

Input:
The following screenshots show that torch.mv is unusable in a situation that obviously seem to be correct... how is this possible, any idea what can be the problem?

this first image shows the correct situation, where the vector has 10 rows for a matrix of 10 columns, but I showed the other also just in case. Also swapping w.mv(x) for x.mv(w)  does not make a difference.

However, the @ operator works... the thing is that for my own reasons I want to use mv, so I would like to know what the problem is.

Output:
According to documentation:

torch.mv(input, vec, *, out=None) → Tensor
If input is a (n×m) tensor, vec is a 1-D tensor of size m, out will be 1-D of size n.

The x here should be 1-D, but in your case it's 10x1 (2D). You can remove extra dimension (or create a single dimension x)
>>> w.mv(x.squeeze())

tensor([ 0.1432, -2.0639, -2.1871, -1.8837,  0.7333, -0.4000,  0.4023, -1.1318,
         0.0423, -1.2136])

>>> w @ x

tensor([[ 0.1432],
        [-2.0639],
        [-2.1871],
        [-1.8837],
        [ 0.7333],
        [-0.4000],
        [ 0.4023],
        [-1.1318],
        [ 0.0423],
        [-1.2136]])


Instruction: Need to implement Deep Learning architecture quite similar to Siamese Network

Input:
I must implement this network:

Similar to a siamese network with a contrastive loss. My problem is S1/F1. The paper tells this:

"F1 and S1 are neural networks that we use to learn the unit-normalized embeddings for the face and speech modalities, respectively. In Figure 1, we depict F1 and S1 in both training and testing routines. They are composed of 2D convolutional layers (purple), max-pooling layers (yellow), and fully connected layers (green). ReLU non-linearity is used between all layers. The last layer is a unit-normalization layer (blue). For both face and speech modalities, F1 and S1 return 250-dimensional unit-normalized embeddings".

My question is:

How can apply a 2D convolutional layer (purple) to input with shape (number of videos, number of frames, features)?
What is the last layer? Batch norm? F.normalize?

Output:
I will give an answer to your two questions without going too much into details:

If you're working with a CNN, you're most likely having spatial information in your input, that is your input is a two dimensional multi-channel tensor (*, channels, height, width), not a feature vector (*, features). You simply won't be able to apply a convolution on your input (at least a 2D conv), if you don't retain two-dimensionality.

The last layer is described as a "unit-normalization" layer. This is merely the operation of making the vector's norm unit (equal to 1). You can do this by dividing the said vector by its norm.


Instruction: RuntimeError: Given groups=721, expected weight to be at least 721 at dimension 0, but got weight of size [3, 1, 5, 5] instead

Input:
Here is the complete error: It is worth mentioning that the input image is of size 480 by 721.
Traceback (most recent call last):
  File "/home/amir/PycharmProjects/LPTN/loadPretrainedModel.py", line 222, in <module>
    output = model(images)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amir/PycharmProjects/LPTN/loadPretrainedModel.py", line 180, in forward
    pyr_A = self.lap_pyramid.pyramid_decom(img=real_A_full)
  File "/home/amir/PycharmProjects/LPTN/loadPretrainedModel.py", line 65, in pyramid_decom
    filtered = self.conv_gauss(current, self.kernel)
  File "/home/amir/PycharmProjects/LPTN/loadPretrainedModel.py", line 58, in conv_gauss
    out = torch.nn.functional.conv2d(img, kernel, groups=img.shape[1])
RuntimeError: Given groups=721, expected weight to be at least 721 at dimension 0, but got weight of size [3, 1, 5, 5] instead

I am trying to run inference on the LPTN (Laplacian Pyramid Translation Network) model.
model = LPTN()

state_dict = torch.load('/home/amir/PycharmProjects/LPTN/experiments/pretrained_models/net_g_FiveK_numhigh3.pth',
                        map_location='cpu')

model.load_state_dict(state_dict, strict=False)
model.eval()

img = cv2.imread("/home/amir/PycharmProjects/LPTN/scripts/data_preparation/datasets/FiveK/FiveK_480p/train/A/2.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

images = torch.from_numpy(np.asarray([img])).float()
output = model(images)
prediction = torch.argmax(output)

Here is the function where the error occurs:
def conv_gauss(self, img, kernel):
     padding = (2, 2, 2, 2)
     img = torch.nn.functional.pad(img, padding, mode='reflect')
     out = torch.nn.functional.conv2d(img, kernel, groups=img.shape[1])

Output:
Well i dont really have the code or anything but i am making a guess based what you have showed
import cv2
import torch

img = cv2.imread("/home/amir/PycharmProjects/LPTN/scripts/data_preparation/datasets/FiveK/FiveK_480p/train/A/2.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # H, W, C

img_tensor = torch.tensor(img, dtype=torch.float32) # Convert to torch tensor
img_tensor = img_tensor / 255. # Normalize [0 - 1] range (but depends on the model)
img_tensor = img_tensor.permute(2, 1, 0) # Reorder to C, H, W (torch requires this format)
img_tensor = img_tensor.unsqueeze(0) # Becomes this format B, C, H, W

# Set model to eval mode
model.eval()

# Run forward pass
with torch.no_grad(): # Dont run your gradients, speeds up inference
    predictions = model(img_tensor) # Get back predictions from model


Instruction: AttributeError: module transformers has no attribute TFGPTNeoForCausalLM

Input:
I cloned this repository/documentation https://huggingface.co/EleutherAI/gpt-neo-125M
I get the below error whether I run it on google collab or locally. I also installed transformers using this
pip install git+https://github.com/huggingface/transformers

and made sure the configuration file is named as config.json
      5 tokenizer = AutoTokenizer.from_pretrained("gpt-neo-125M/",from_tf=True)
----> 6 model = AutoModelForCausalLM.from_pretrained("gpt-neo-125M",from_tf=True)
      7
      8

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py in __getattr__(self, name)

AttributeError: module transformers has no attribute TFGPTNeoForCausalLM

Full code:
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-125M",from_tf=True)

model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-125M",from_tf=True)

transformers-cli env results:

transformers version: 4.10.0.dev0
Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.29
Python version: 3.8.5
PyTorch version (GPU?): 1.9.0+cpu (False)
Tensorflow version (GPU?): 2.5.0 (False)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?:
Using distributed or parallel set-up in script?:

Both collab and locally have TensorFlow 2.5.0 version

Output:
My solution was to first edit the source code to remove the line that adds "TF" in front of the package as the correct transformers module is GPTNeoForCausalLM
, but somewhere in the source code it manually added a "TF" in front of it.
Secondly, before cloning the repository it is a must to run
 git lfs install.

This link helped me install git lfs properly https://askubuntu.com/questions/799341/how-to-install-git-lfs-on-ubuntu-16-04


Instruction: How to efficiently calculate distance matrix in pytorch for two sets 3D tensors with different sizes?

Input:
I have tensors X of shape BxCxHxW and Y of shape NxCxHxW. B is batch size, C is channels, H is height, W is width, and N will be constant for any batch. Basically I want the BxN distance matrix of distances between a set of B images and another set of N images.
I tried using torch.cdist by reshaping X as 1xBx(C*H*W) and Y as 1xNx(C*H*W) by unsqueezing a dimension and flattening the last 3 channels, but I did a sanity check and got wrong answers with this method.
I want L2 distance.

Output:
According to the documentation page for torch.cdist, the two inputs and outputs are shaped in the following manner: x1: (B, P, M), x2: (B, R, M), and output: (B, P, R).
To match your case: B=1, P=B, R=N, while M=C*H*W (i.e. flattened). As you just explained.
So you are basically going for:
>>> torch.cdist(X[None].flatten(2), Y[None].flatten(2))

If you're not convinced, you can check with the following method:
>>> dist = []
>>> for x in X:
...    for y in Y:
...       dist.append((x-y).norm())

And compare the torch.cdist result with torch.tensor(dist).reshape(len(X), len(Y)).


Instruction: What is difference between nn.Module and nn.Sequential

Input:
I am just learning to use PyTorch as a beginner. If anyone is familiar with PyTorch, would you tell me the difference between nn.Module and nn.Sequential?
My questions are

What is the advantage to use nn.Module instead of nn.Sequential?

Which is regularly utilised to build the model?

How we should select nn.Module or nn.Sequential?

Output:
TLDR; answering your questions

What is the advantage to use nn.Module instead of nn.Sequential?

While nn.Module is the base class to implement PyTorch models, nn.Sequential is a quick way to define a sequential neural network structures inside or outside an existing nn.Module.

Which is regularly utilized to build the model?

Both are widely used.

How we should select nn.Module or nn.Sequential?

All neural networks are implemented with nn.Module. If the layers are sequentially used (self.layer3(self.layer2(self.layer1(x))), you can leverage nn.Sequential to not have to define the forward function of the model.

I should start by mentioning that nn.Module is the base class for all neural network modules in PyTorch. As such nn.Sequential is actually a direct subclass of nn.Module, you can look for yourself on this line.
When creating a new neural network, you would usually go about creating a new class and inheriting from nn.Module, and defining two methods: __init__ (the initializer, where you define your layers) and forward (the inference code of your module, where you use your layers). That's all you need, since
PyTorch will handle backward pass with Autograd. Here is an example of a module:
class NN(nn.Module):
    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(10, 4)
        self.fc2 = nn.Linear(4, 2)

    def forward(self, x)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

If the model you are defining is sequential, i.e. the layers are called sequentially on the input, one by one. Then, you can simply use a nn.Sequential. As I explained earlier, nn.Sequential is a special kind of nn.Module made for this particular widespread type of neural network. The equivalent here is:
class NN(nn.Sequential):
    def __init__(self):
        super().__init__(
           nn.Linear(10, 4),
           nn.ReLU(),
           nn.Linear(4, 2),
           nn.ReLU())

Or a simpler way of putting it is:
NN = Sequential(
   nn.Linear(10, 4),
   nn.ReLU(),
   nn.Linear(4, 2),
   nn.Linear())

The objective of nn.Sequential is to quickly implement sequential modules such that you are not required to write the forward definition, it being implicitly known because the layers are sequentially called on the outputs.
In a more complicated module though, you might need to use multiple sequential submodules. For instance, take a CNN classifier, you could define a nn.Sequential for the CNN part, then define another nn.Sequential for the fully connected classifier section of the model.


Instruction: How to calculate correct Cross Entropy between 2 tensors in Pytorch when target is not one-hot?

Input:
I am confused about the calculation of cross entropy in Pytorch. If I want to calculate the cross entropy between 2 tensors and the target tensor is not a one-hot label, which loss should I use? It is quite common to calculate the cross entropy between 2 probability distributions instead of the predicted result and a determined one-hot label.
The basic loss function CrossEntropyLoss forces the target as the index integer and it is not eligible in this case. BCELoss seems to work but it gives an unexpected result. The expected formula to calculate the cross entropy is

But BCELoss calculates the BCE of each dimension, which is expressed as
-yi*log(pi)-(1-yi)*log(1-pi)

Compared with the first equation, the term -(1-yi)*log(1-pi) should not be involved. Here is an example using BCELoss and we can see the second term is involved in each dimension's result. And that make the result different from the correct one.
import torch.nn as nn
import torch
from math import log

a = torch.Tensor([0.1,0.2,0.7])
y = torch.Tensor([0.2,0.2,0.6])
L = nn.BCELoss(reduction='none')
y1 = -0.2 * log(0.1) - 0.8 * log(0.9)
print(L(a, y))
print(y1)

And the result is
tensor([0.5448, 0.5004, 0.6956])
0.5448054311250702

If we sum the results of all the dimensions, the final cross entropy doesn't correspond to the expected one. Because each one of these dimensions involves the -(1-yi)*log(1-pi) term. In constrast, Tensorflow can calculate the correct cross entropy value with CategoricalCrossentropy. Here is the example with the same setting and we can see the cross entropy is calculated in the same way as the first formula.
import tensorflow as tf
from math import log
L = tf.losses.CategoricalCrossentropy()
a = tf.convert_to_tensor([0.1,0.2,0.7])
y = tf.convert_to_tensor([0.2,0.2,0.6])
y_ = -0.2* log(0.1) - 0.2 * log(0.2) - 0.6 * log(0.7)

print(L(y,a), y_)

tf.Tensor(0.9964096, shape=(), dtype=float32) 0.9964095674488687

Is there any function can calculate the correct cross entropy in Pytorch, using the first formula, just like CategoricalCrossentropy in Tensorflow?

Output:
The fundamental problem is that you are incorrectly using the BCELoss function.
Cross-entropy loss is what you want. It is used to compute the loss between two arbitrary probability distributions. Indeed, its definition is exactly the equation that you provided:

where p is the target distribution and q is your predicted distribution. See this StackOverflow post for more information.
In your example where you provide the line
y = tf.convert_to_tensor([0.2, 0.2, 0.6])

you are implicitly modeling a multi-class classification problem where the target class can be one of three classes (the length of that tensor). More specifically, that line is saying that for this one data instance, class 0 has probably 0.2, class 1 has probability 0.2, and class 2 has probability 0.6.
The problem you are having is that PyTorch's BCELoss computes the binary cross-entropy loss, which is formulated differently. Binary cross-entropy loss computes the cross-entropy for classification problems where the target class can be only 0 or 1.
In binary cross-entropy, you only need one probability, e.g. 0.2, meaning that the probability of the instance being class 1 is 0.2. Correspondingly, class 0 has probability 0.8.
If you give the same tensor [0.2, 0.2, 0.6] to BCELoss, you are modeling a situation where there are three data instances, where data instance 0 has probability 0.2 of being class 1, data instance 1 has probability 0.2 of being class 1, and data instance 2 has probability 0.6 of being class 1.
Now, to your original question:

If I want to calculate the cross entropy between 2 tensors and the target tensor is not a one-hot label, which loss should I use?

Unfortunately, PyTorch does not have a cross-entropy function that takes in two probability distributions. See this question:
https://discuss.pytorch.org/t/how-should-i-implement-cross-entropy-loss-with-continuous-target-outputs/10720
The recommendation is to implement your own function using its equation definition. Here is code that works:
def cross_entropy(input, target):
    return torch.mean(-torch.sum(target * torch.log(input), 1))

y = torch.Tensor([[0.2, 0.2, 0.6]])
yhat = torch.Tensor([[0.1, 0.2, 0.7]])
cross_entropy(yhat, y)
# tensor(0.9964)

It provides the answer that you wanted.


Instruction: How do I insert a 1D Torch tensor into an existing 2D Torch tensor into a specific row?

Input:
I want to insert a 1D Torch tensor into a specific row number into a 2D Torch tensor (Using Pytorch).
The 1D tensor and the 2D tensor will always have the same length, so you can easily visualize this as a table with rows and columns.
The 2D tensor is the existing table and I would like to be able to specify the the row number the 1D tensor (or row) will be inserted.
When I say I'd like to use Pytorch, I don't want to turn anything into a non-Pytorch list and send the computations back and forth over the CPU and GPU. The tensors are all already on my CUDA device and I would like to keep them there for the sake of time.
The 2D tensor three_by_four
tensor([[0.7421, 0.1584, 0.3231, 0.4840],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.5226, 0.6216, 0.9420, 0.0605]], device='cuda:1')

The 1D tensor one_by_three
tensor([[0.3095, 0.8460, 0.2900, 0.9683]], device='cuda:1')

The best I was able to do was get the new row (the 1D tensor) appended to the bottom or top of the 2D tensor with torch.cat depending on the order.
The 1D tensor added to the top.
torch.cat([one_by_three, three_by_four])

tensor([[0.3095, 0.8460, 0.2900, 0.9683],
        [0.7421, 0.1584, 0.3231, 0.4840],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.5226, 0.6216, 0.9420, 0.0605]], device='cuda:1')

The 1D tensor added to the bottom
torch.cat([three_by_four, one_by_three])
tensor([[0.7421, 0.1584, 0.3231, 0.4840],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.5226, 0.6216, 0.9420, 0.0605],
        [0.3095, 0.8460, 0.2900, 0.9683]], device='cuda:1')

What I would like, for example, if I could put it in position 1, or 2 in this example.
tensor([[0.7421, 0.1584, 0.3231, 0.4840],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.3095, 0.8460, 0.2900, 0.9683]
        [0.5226, 0.6216, 0.9420, 0.0605]], device='cuda:1')

Output:
As of now the best I could find
from torch import tensor, cat
x = tensor([[0.7421, 0.1584, 0.3231, 0.4840],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.5226, 0.6216, 0.9420, 0.0605]])
y = tensor([[0.3095, 0.8460, 0.2900, 0.9683]])
cat([x[:1], y, x[1:]])
'''
tensor([[0.7421, 0.1584, 0.3231, 0.4840],
        [0.3095, 0.8460, 0.2900, 0.9683],
        [0.4065, 0.7646, 0.9677, 0.4537],
        [0.5226, 0.6216, 0.9420, 0.0605]])
'''

https://discuss.pytorch.org/t/is-there-a-way-to-insert-a-tensor-into-an-existing-tensor/14642


Instruction: pos_weight in binary cross entropy calculation

Input:
When we deal with imbalanced training data (there are more negative samples and less positive samples), usually pos_weight parameter will be used.
The expectation of pos_weight is that the model will get higher loss when the positive sample gets the wrong label than the negative sample.
When I use the binary_cross_entropy_with_logits function, I found:
bce = torch.nn.functional.binary_cross_entropy_with_logits

pos_weight = torch.FloatTensor([5])
preds_pos_wrong =  torch.FloatTensor([0.5, 1.5])
label_pos = torch.FloatTensor([1, 0])
loss_pos_wrong = bce(preds_pos_wrong, label_pos, pos_weight=pos_weight)

preds_neg_wrong =  torch.FloatTensor([1.5, 0.5])
label_neg = torch.FloatTensor([0, 1])
loss_neg_wrong = bce(preds_neg_wrong, label_neg, pos_weight=pos_weight)

However:
>>> loss_pos_wrong
tensor(2.0359)

>>> loss_neg_wrong
tensor(2.0359)

The losses derived from wrong positive samples and negative samples are the same, so how does pos_weight work in the imbalanced data loss calculation?

Output:
TLDR; both losses are identical because you are computing the same quantity: both inputs are identical, the two batch elements and labels are just switched.

Why are you getting the same loss?
I think you got confused in the usage of F.binary_cross_entropy_with_logits (you can find a more detailed documentation page with nn.BCEWithLogitsLoss). In your case your input shape (aka the output of your model) is one-dimensional, which means you only have a single logit x, not two).
In your example you have
preds_pos_wrong = torch.FloatTensor([0.5, 1.5])
label_pos = torch.FloatTensor([1, 0])

This means your batch size is 2, and since by default the function is averaging the losses of the batch elements, you end up with the same result for BCE(preds_pos_wrong, label_pos) and BCE(preds_neg_wrong, label_neg). The two elements of your batch are just switched.
You can verify this very easily by not averaging the loss over the batch-elements with the reduction='none' option:
>>> F.binary_cross_entropy_with_logits(preds_pos_wrong, label_pos,
       pos_weight=pos_weight, reduction='none')
tensor([2.3704, 1.7014])

>>> F.binary_cross_entropy_with_logits(preds_pos_wrong, label_pos,
       pos_weight=pos_weight, reduction='none')
tensor([1.7014, 2.3704])

Looking into F.binary_cross_entropy_with_logits:
That being said the formula for the binary cross-entropy is:
bce = -[y*log(sigmoid(x)) + (1-y)*log(1- sigmoid(x))]

Where y (respectively sigmoid(x) is for the positive class associated with that logit, and 1 - y (resp. 1 - sigmoid(x)) is the negative class.
The documentation could be more precise on the weighting scheme for pos_weight (not to be confused with weight, which is the weighting of the different logits output). The idea with pos_weight as you said, is to weigh the positive term, not the whole term.
bce = -[w_p*y*log(sigmoid(x)) + (1-y)*log(1- sigmoid(x))]

Where w_p is the weight for the positive term, to compensate for the positive to negative sample imbalance. In practice, this should be w_p = #negative/#positive.
Therefore:
>>> w_p = torch.FloatTensor([5])
>>> preds = torch.FloatTensor([0.5, 1.5])
>>> label = torch.FloatTensor([1, 0])

With the builtin loss function,
>>> F.binary_cross_entropy_with_logits(preds, label, pos_weight=w_p, reduction='none')
tensor([2.3704, 1.7014])

Compared with the manual computation:
>>> z = torch.sigmoid(preds)
>>> -(w_p*label*torch.log(z) + (1-label)*torch.log(1-z))
tensor([2.3704, 1.7014])


Instruction: Pytorch: best practice to save list of tensors?

Input:
I use tensors to do transformation then I save it in a list. Later, I will make it a dataset using Dataset, then finally DataLoader to train my model. To do it, I can simply use:
l = [tensor1, tensor2, tensor3,...]
dataset = Dataset.TensorDataset(l)
dataloader = DataLoader(dataset)

I wonder what is the best practice doing so, to avoid RAM overflow if the size of l grows? Can something like Iterator avoid it?

Output:
Save tensors
for idx, tensor in enumerate(dataloader0):
    torch.save(tensor, f"{my_folder}/tensor{idx}.pt")

Create dataset
class FolderDataset(Dataset):
   def __init__(self, folder):
       self.files = os.listdir(folder)
       self.folder = folder
   def __len__(self):
       return len(self.files)
   def __getitem__(self, idx):
       return torch.load(f"{self.folder}/{self.files[idx]}")

And then you can implement your own dataloader. If you can't hold the whole dataset in memory, some file system loading is required.


Instruction: Roll rows with variable step value on a 2D tensor

Input:
I have a tensor a that I would like to first mask using mask and then discard the remaining frames. To ensure the output tensor is of the correct shape, padding should fill in the remaining values at the end. I can assume there is only a single continuous sequence of True's in each row of the mask.
e.g.
a = torch.arange(1,17).reshape(4,4)
# tensor([[ 1,  2,  3,  4],
#         [ 5,  6,  7,  8],
#         [ 9, 10, 11, 12],
#         [13, 14, 15, 16]])

mask = torch.tensor([[False,  True,  True, False],
                     [False,  True,  True,  True],
                     [ True, False, False, False],
                     [ True,  True,  True,  True]])

# desired output (assuming padding value is 0):
# tensor([[ 2,  3,  0,  0],
#         [ 6,  7,  8,  0],
#         [ 9,  0,  0,  0],
#         [13, 14, 15, 16]])

I can achieve the desired output by applying torch.masked_select followed by torch.nn.functional.pad on each row in a loop but I am struggling to think of a way to do this more efficiently in batches.
I have also looked into starting by using torch.roll and zeroing after appropriate indexes, but this function can only be applied across an entire dimension and not a custom amount of roll per row.

Output:
By applying torch.sort on the mask itself you can achieve the desired result. Indeed if your sort the boolean values you can manage to move the False values at the end of the stack, and let the True values at the beginning.
Do note this might vary depending on the sorting algorithm, there might be some shuffling for certain algorithms.... As @Seraf Fej pointed out: you can use the stable=True option on torch.stable such that the order of equivalent items is preserved.
Then use the indices of the sorting to gather the values on a with torch.gather. Finally, you will need to mask the resulting matrix to replace the discarded values with the appropriate padding.
>>> a
tensor([[ 1,  2,  3,  4],
        [ 5,  6,  7,  8],
        [ 9, 10, 11, 12],
        [13, 14, 15, 16]])

>>> mask
tensor([[False,  True,  True, False],
        [False,  True,  True,  True],
        [ True, False, False, False],
        [ True,  True,  True,  True]])

Sort the mask:
>>> values, indices = mask.sort(1, descending=True, stable=True)

>>> values
tensor([[ True,  True, False, False],
        [ True,  True,  True, False],
        [ True, False, False, False],
        [ True,  True,  True,  True]])

>>> indices
tensor([[1, 2, 0, 3],
        [1, 2, 3, 0],
        [0, 1, 2, 3],
        [0, 1, 2, 3]])

Gather from indices and mask with values:
>>> a.gather(1, indices)*values
tensor([[ 2,  3,  0,  0],
        [ 6,  7,  8,  0],
        [ 9,  0,  0,  0],
        [13, 14, 15, 16]])

You can easily extend to any padding value using torch.where:
>>> torch.where(values, a.gather(1, indices), -1)
tensor([[ 2,  3, -1, -1],
        [ 6,  7,  8, -1],
        [ 9, -1, -1, -1],
        [13, 14, 15, 16]])

Or using the inverse mask ~values, weighted by the padding value:
>>> a.gather(1, indices)*values -1*~values
tensor([[ 2,  3, -1, -1],
        [ 6,  7,  8, -1],
        [ 9, -1, -1, -1],
        [13, 14, 15, 16]])


Instruction: RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Input:
I am trying to train PeleeNet pytorch and got the following error

train.py line 80

pelee_voc train configuration

Output:
Turning the shuffle parameter off in the dataloader solved it.
Got the answer form here.


Instruction: Create custom gradient descent in pytorch

Input:
I am trying to use PyTorch autograd to implement my own batch gradient descent algorithm. I want to create a simple one-layer neural net with a linear activation function and the mean squared error as the loss function. I can't seem to get my head around what exactly is happening in the backward pass and how PyTorch understands my outputs. I have coded one class specifying the linear function in the forward pass, and in the backward pass, I calculated the gradients with respect to each variable. I also coded a class for the MSE function and specified the gradients with respect to ITS variables in the backward pass. When I run a simple gradient descent algorithm, I get no errors, but the MSE only goes down in the first iteration, and after that, it continually goes up. This leads me to believe that I have made a mistake, but I am not sure, where. Does anybody see the error in my code? Also, if somebody could explain to me what exactly the grad_output stands for, that would be amazing.
Here are the functions:
import torch
from torch.autograd import Function
from torch.autograd import gradcheck

class Context:
    def __init__(self):
        self._saved_tensors = ()

    def save_for_backward(self, *args):
        self._saved_tensors = args

    @property
    def saved_tensors(self):
        return self._saved_tensors

class MSE(Function):
    @staticmethod
    def forward(ctx, yhat, y):
        ctx.save_for_backward(yhat, y)
        q = yhat.size()[0]
        mse = torch.sum((yhat-y)**2)/q
        return mse

    @staticmethod
    def backward(ctx, grad_output):
        yhat, y = ctx.saved_tensors
        q = yhat.size()[0]
        return 2*grad_output*(yhat-y)/q, -2*grad_output*(yhat-y)/q

class Linear(Function):
    @staticmethod
    def forward(ctx, X, W, b):
        rows = X.size()[0]
        yhat = torch.mm(X,W) + b.repeat(rows,1)
        ctx.save_for_backward(yhat, X, W)
        return yhat

    @staticmethod
    def backward(ctx, grad_output):
        yhat, X, W = ctx.saved_tensors
        q = yhat.size()[0]
        p = yhat.size()[1]
        return torch.transpose(X, 0, 1), W, torch.ones(p)

And here is my gradient descent:
import torch
from torch.utils.tensorboard import SummaryWriter
from tp1moi import MSE, Linear, Context

x = torch.randn(50, 13)
y = torch.randn(50, 3)

w = torch.randn(13, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)

epsilon = 0.05

writer = SummaryWriter()
for n_iter in range(100):
    linear = Linear.apply
    mse = MSE.apply
    loss = mse(linear(x, w, b), y)

    writer.add_scalar('Loss/train', loss, n_iter)

    print(f"Itérations {n_iter}: loss {loss}")

    loss.backward()

    with torch.no_grad():
        w -= epsilon*w.grad
        b -= epsilon*b.grad
        w.grad.zero_()
        b.grad.zero_()

Here is one output I got (they all look similar to this one):
Itérations 0: loss 72.99712371826172
Itérations 1: loss 7.509067535400391
Itérations 2: loss 7.309497833251953
Itérations 3: loss 7.124927997589111
Itérations 4: loss 6.955358982086182
Itérations 5: loss 6.800788402557373
Itérations 6: loss 6.661219596862793
Itérations 7: loss 6.536648750305176
Itérations 8: loss 6.427078723907471
Itérations 9: loss 6.3325090408325195
Itérations 10: loss 6.252938747406006
Itérations 11: loss 6.188369274139404
Itérations 12: loss 6.138798713684082
Itérations 13: loss 6.104228973388672
Itérations 14: loss 6.084658145904541
Itérations 15: loss 6.0800886154174805
Itérations 16: loss 6.090517520904541
Itérations 17: loss 6.115947723388672
Itérations 18: loss 6.156377792358398
Itérations 19: loss 6.2118072509765625
Itérations 20: loss 6.2822370529174805
Itérations 21: loss 6.367666721343994
Itérations 22: loss 6.468096733093262
Itérations 23: loss 6.583526611328125
Itérations 24: loss 6.713956356048584
Itérations 25: loss 6.859385967254639
Itérations 26: loss 7.019815444946289
Itérations 27: loss 7.195245742797852
Itérations 28: loss 7.385674953460693
Itérations 29: loss 7.591104507446289
Itérations 30: loss 7.811534881591797
Itérations 31: loss 8.046965599060059
Itérations 32: loss 8.297393798828125
Itérations 33: loss 8.562823295593262
Itérations 34: loss 8.843254089355469
Itérations 35: loss 9.138683319091797
Itérations 36: loss 9.449112892150879
Itérations 37: loss 9.774543762207031
Itérations 38: loss 10.114972114562988
Itérations 39: loss 10.470401763916016
Itérations 40: loss 10.840831756591797
Itérations 41: loss 11.226261138916016
Itérations 42: loss 11.626690864562988
Itérations 43: loss 12.042119979858398
Itérations 44: loss 12.472548484802246
Itérations 45: loss 12.917980194091797
Itérations 46: loss 13.378408432006836
Itérations 47: loss 13.853838920593262
Itérations 48: loss 14.344267845153809
Itérations 49: loss 14.849695205688477
Itérations 50: loss 15.370124816894531
Itérations 51: loss 15.905555725097656
Itérations 52: loss 16.455984115600586
Itérations 53: loss 17.02141571044922
Itérations 54: loss 17.601844787597656
Itérations 55: loss 18.19727325439453
Itérations 56: loss 18.807701110839844
Itérations 57: loss 19.43313217163086
Itérations 58: loss 20.07356071472168
Itérations 59: loss 20.728988647460938
Itérations 60: loss 21.3994197845459
Itérations 61: loss 22.084848403930664
Itérations 62: loss 22.7852783203125
Itérations 63: loss 23.50070571899414
Itérations 64: loss 24.23113441467285
Itérations 65: loss 24.9765625
Itérations 66: loss 25.73699188232422
Itérations 67: loss 26.512422561645508
Itérations 68: loss 27.302854537963867
Itérations 69: loss 28.108285903930664
Itérations 70: loss 28.9287166595459
Itérations 71: loss 29.764144897460938
Itérations 72: loss 30.614578247070312
Itérations 73: loss 31.48000717163086
Itérations 74: loss 32.36043930053711
Itérations 75: loss 33.2558708190918
Itérations 76: loss 34.16630172729492
Itérations 77: loss 35.091732025146484
Itérations 78: loss 36.032161712646484
Itérations 79: loss 36.98759460449219
Itérations 80: loss 37.95802307128906
Itérations 81: loss 38.943458557128906
Itérations 82: loss 39.943885803222656
Itérations 83: loss 40.959320068359375
Itérations 84: loss 41.98974609375
Itérations 85: loss 43.03517532348633
Itérations 86: loss 44.09561538696289
Itérations 87: loss 45.171043395996094
Itérations 88: loss 46.261474609375
Itérations 89: loss 47.366905212402344
Itérations 90: loss 48.487335205078125
Itérations 91: loss 49.62276840209961
Itérations 92: loss 50.773197174072266
Itérations 93: loss 51.93863296508789
Itérations 94: loss 53.11906433105469
Itérations 95: loss 54.31448745727539
Itérations 96: loss 55.524925231933594
Itérations 97: loss 56.75035095214844
Itérations 98: loss 57.990787506103516
Itérations 99: loss 59.2462158203125```

Output:
Let's take a look at the implementation of MSE, the forward pass will be MSE(y, y_hat) = (y_hat-y)² which is straightforward. For the backward pass, we are looking to compute the derivative of the output with regards to the input, as well as the derivative with regards to each of the parameters. Here MSE does not have any learned parameters, so we just want to compute dMSE/dy*dz/dMSE using the chain rule, which is d(y_hat-y)²/dy*dz/dMSE, i.e. -2(y_hat-y)*dz/dMSE. Not to confuse you here: I wrote dz/dMSEas the incoming gradient. It corresponds to the gradient following backward towards the MSE layer. From your notation grad_output is dz/dMSE. Therefore the backward pass is simply -2*(y_hat-y)*grad_output. Then normalized by the batch size q, retrieved from y_hat.size(0).
The same thing goes with the Linear layer. It will involve some more computation since, this time, the layer is parametrized by w and b. The forward pass is essentially x@w + b. While the backward pass, consists in calculating dz/dx, dz/dw, and dz/db. Writing f as x@w + b. After some work you can find that that:

dz/dx = d(x@w + b)/dx * dz/df = dz/df*W.T,
dz/dw = d(x@w + b)/dw * dz/df = X.T*dz/df,
dz/db = d(x@w + b)/db * dz/df = 1.

In terms of implementation this would look like:

output_grad@w.T for the gradient w.r.t x,
x.T@output_grad for the gradient w.r.t w,
torch.ones_like(b) for the gradient w.r.t b.


Instruction: Running out of memory with pytorch

Input:
I am trying to train a model using huggingface's wav2vec for audio classification. I keep getting this error:
The following columns in the training set  don't have a corresponding argument in `Wav2Vec2ForSpeechClassification.forward` and have been ignored: name, emotion, path.
***** Running training *****
  Num examples = 2708
  Num Epochs = 1
  Instantaneous batch size per device = 4
  Total train batch size (w. parallel, distributed & accumulation) = 64
  Gradient Accumulation steps = 2
  Total optimization steps = 42
 [ 2/42 : < :, Epoch 0.02/1]
Step    Training Loss   Validation Loss

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "<ipython-input-81-dd9fe3ea0f13>", line 77, in forward
    return_dict=return_dict,
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1073, in forward
    return_dict=return_dict,
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 732, in forward
    hidden_states, attention_mask=attention_mask, output_attentions=output_attentions
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 574, in forward
    hidden_states = hidden_states + self.feed_forward(self.final_layer_norm(hidden_states))
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 510, in forward
    hidden_states = self.intermediate_act_fn(hidden_states)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/functional.py", line 1555, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.49 GiB already allocated; 11.44 MiB free; 10.68 GiB reserved in total by PyTorch)

I'm on an AWS ubuntu deep learning AMI ec2.
I've been researching this a lot. I've already tried:

reducing the batch size (I want 4, but I've gone down to 1 with no change in error)
adding:
import gc
gc.collect()
torch.cuda.empty_cache()

removing all wav files in my dataset that are longer than 6 seconds

Is there anything else I can do? I'm on a p2.8xlarge dataset with 105 GiB mounted.
Running torch.cuda.memory_summary(device=None, abbreviated=False)  gives me:
|===========================================================================|\n|                  PyTorch CUDA memory summary, device ID 0                 |\n|---------------------------------------------------------------------------|\n|            CUDA OOMs: 3            |        cudaMalloc retries: 4         |\n|===========================================================================|\n|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |\n|---------------------------------------------------------------------------|\n| Allocated memory      |    7550 MB |   10852 MB |  209624 MB |  202073 MB |\n|       from large pool |    7544 MB |   10781 MB |  209325 MB |  201780 MB |\n|       from small pool |       5 MB |      87 MB |     298 MB |     293 MB |\n|---------------------------------------------------------------------------|\n| Active memory         |    7550 MB |   10852 MB |  209624 MB |  202073 MB |\n|       from large pool |    7544 MB |   10781 MB |  209325 MB |  201780 MB |\n|       from small pool |       5 MB |      87 MB |     298 MB |     293 MB |\n|---------------------------------------------------------------------------|\n| GPU reserved memory   |   10936 MB |   10960 MB |   63236 MB |   52300 MB |\n|       from large pool |   10928 MB |   10954 MB |   63124 MB |   52196 MB |\n|       from small pool |       8 MB |      98 MB |     112 MB |     104 MB |\n|---------------------------------------------------------------------------|\n| Non-releasable memory |  443755 KB |    1309 MB |  155426 MB |  154992 MB |\n|       from large pool |  443551 KB |    1306 MB |  155081 MB |  154648 MB |\n|       from small pool |     204 KB |      12 MB |     344 MB |     344 MB |\n|---------------------------------------------------------------------------|\n| Allocations           |    1940    |    2622    |   32288    |   30348    |\n|       from large pool |    1036    |    1618    |   21855    |   20819    |\n|       from small pool |     904    |    1203    |   10433    |    9529    |\n|---------------------------------------------------------------------------|\n| Active allocs         |    1940    |    2622    |   32288    |   30348    |\n|       from large pool |    1036    |    1618    |   21855    |   20819    |\n|       from small pool |     904    |    1203    |   10433    |    9529    |\n|---------------------------------------------------------------------------|\n| GPU reserved segments |     495    |     495    |    2169    |    1674    |\n|       from large pool |     491    |     491    |    2113    |    1622    |\n|       from small pool |       4    |      49    |      56    |      52    |\n|---------------------------------------------------------------------------|\n| Non-releasable allocs |     179    |     335    |   15998    |   15819    |\n|       from large pool |     165    |     272    |   12420    |   12255    |\n|       from small pool |      14    |      63    |    3578    |    3564    |\n|===========================================================================|\n'

After reducing data only to inputs that are less tahn 2 seconds in length, it trains a lot further but still errors with this:
The following columns in the training set  don't have a corresponding argument in `Wav2Vec2ForSpeechClassification.forward` and have been ignored: path, emotion, name.
***** Running training *****
  Num examples = 1411
  Num Epochs = 1
  Instantaneous batch size per device = 4
  Total train batch size (w. parallel, distributed & accumulation) = 64
  Gradient Accumulation steps = 2
  Total optimization steps = 22
/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
/home/ubuntu/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
 [11/22 01:12 < 01:28, 0.12 it/s, Epoch 0.44/1]
Step    Training Loss   Validation Loss Accuracy
10  2.428100    2.257138    0.300283
The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForSpeechClassification.forward` and have been ignored: path, emotion, name.
***** Running Evaluation *****
  Num examples = 353
  Batch size = 32
Saving model checkpoint to trainingArgs/checkpoint-10
Configuration saved in trainingArgs/checkpoint-10/config.json
Model weights saved in trainingArgs/checkpoint-10/pytorch_model.bin
Configuration saved in trainingArgs/checkpoint-10/preprocessor_config.json
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    378             with _open_zipfile_writer(opened_file) as opened_zipfile:
--> 379                 _save(obj, opened_zipfile, pickle_module, pickle_protocol)
    380                 return

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/serialization.py in _save(obj, zip_file, pickle_module, pickle_protocol)
    498         num_bytes = storage.size() * storage.element_size()
--> 499         zip_file.write_record(name, storage.data_ptr(), num_bytes)
    500

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-25-3435b262f1ae> in <module>
----> 1 trainer.train()

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1334                     self.control = self.callback_handler.on_step_end(args, self.state, self.control)
   1335
-> 1336                     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1337                 else:
   1338                     self.control = self.callback_handler.on_substep_end(args, self.state, self.control)

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/trainer.py in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1441
   1442         if self.control.should_save:
-> 1443             self._save_checkpoint(model, trial, metrics=metrics)
   1444             self.control = self.callback_handler.on_save(self.args, self.state, self.control)
   1445

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/trainer.py in _save_checkpoint(self, model, trial, metrics)
   1531         elif self.args.should_save and not self.deepspeed:
   1532             # deepspeed.save_checkpoint above saves model/optim/sched
-> 1533             torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
   1534             with warnings.catch_warnings(record=True) as caught_warnings:
   1535                 torch.save(self.lr_scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    378             with _open_zipfile_writer(opened_file) as opened_zipfile:
    379                 _save(obj, opened_zipfile, pickle_module, pickle_protocol)
--> 380                 return
    381         _legacy_save(obj, opened_file, pickle_module, pickle_protocol)
    382

~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/serialization.py in __exit__(self, *args)
    257
    258     def __exit__(self, *args) -> None:
--> 259         self.file_like.write_end_of_file()
    260         self.buffer.flush()
    261

RuntimeError: [enforce fail at inline_container.cc:298] . unexpected pos 1849920000 vs 1849919888

When I run !free in the notebook, I get:
The history saving thread hit an unexpected error (OperationalError('database or disk is full')).History will not be written to the database.
              total        used        free      shared  buff/cache   available
Mem:      503392908     6223452   478499292      346492    18670164   492641984
Swap:             0           0           0

For training code, I am essentially running this colab notebook as an example:
https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb#scrollTo=6M8bNvLLJnG1
All that I am changing is the incoming data/labels, which I have intentionally fit into the same directory structure used in the tutorial notebook. The tutorial notebook runs fine for some reason, even though my data has comparable size/num classes.

Output:
You might use the DataParallel or DistributedDataParallel framework in Pytorch
model = Model(input_size, output_size)
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model = nn.DataParallel(model)

model.to(device)

In this approach the model get replicated on each device (gpu) and the data is distributed across devices

DataParallel splits your data automatically and sends job orders to
multiple models on several GPUs. After each model finishes their job,
DataParallel collects and merges the results before returning it to
you.

Further examples here https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html.
If the model does not fit in the memory of one gpu, then a model parallel approach should be resorted to.
From your existing model you might tell which layer sits on which gpu with .to('cuda:0'), .to('cuda:1') etc.
class ModelParallelResNet50(ResNet):
    def __init__(self, *args, **kwargs):
        super(ModelParallelResNet50, self).__init__(
            Bottleneck, [3, 4, 6, 3], num_classes=num_classes, *args, **kwargs)

        self.seq1 = nn.Sequential(
            self.conv1,
            self.bn1,
            self.relu,
            self.maxpool,

            self.layer1,
            self.layer2
        ).to('cuda:0')

        self.seq2 = nn.Sequential(
            self.layer3,
            self.layer4,
            self.avgpool,
        ).to('cuda:1')

        self.fc.to('cuda:1')

    def forward(self, x):
        x = self.seq2(self.seq1(x).to('cuda:1'))
        return self.fc(x.view(x.size(0), -1))

Since you might lose performance, a pipelining approach might be of use, i.e. further chunking input data into batches which are run in parallel on the different devices.


Instruction: How I can give input to the 3d Convolutional Neural Network?

Input:
The 3d CNN works with the videos, MRI, and scan datasets. Can you tell me If I have to feed the input (video) to the proposed 3d CNN network, and train it's weights, how can I able to do that? as 3d CNN expect 5 dimensional inputs;
[batch size, channels, depth, height, weight]

how can I extract depth from the videos?
If I have 10 video of 10 different classes. The duration of each video is 6 seconds. I extract 2 frames for each second and it goes around 12 frames for each video.
Size of RGB videos is 112x112 --> Height = 112, Width=112, and Channels=3
If I keep the batch size equals 2
1 video --> 6 seconds --> 12 frames (1sec == 2frames) [each frame (3,112,112)]
10 videos (10 classes) --> 60 seconds --> 120 frames
So the 5 dimensions will be something like this; [2, 3, 12, 112, 112]
2 --> Two videos will be processed for each batch size.
3 --> RGB channels
12 --> each video contains 12 frames
112 --> Height of each video
112 --> Width of each video
Am I right?

Output:
Yes, that seems to make sense if you're looking to use a 3D CNN. You're essentially adding a dimension to your input which is the temporal one, it is logical to use the depth dimension for it. This way you keep the channel axis as the feature channel (i.e. not a spatial-temporal dimension).
Keep in mind 3D CNNs are really memory intensive. There exist other methods to work with temporal dependent input. Here you are not really dealing with a third dimension (a 'spatial' dimension that is), so you're not required to use a 3D CNN.

Edit:

If I give the input of the above dimension to the 3d CNN, will it learn both features (spatial and temporal)? [...] Can you make me understand, spatial and temporal features?

If you use a 3D CNN then your filters will have a 3D kernel, and the convolution will be three dimensional: along the two spatial dimensions (width and height) as well as the depth dimensions (here corresponding to the temporal dimensions, since you're using depth dimension for the sequence of videos frames. A 3D CNN will allow you to capture local ('local' because the perception field is limited by the sizes of the kernels and the overall number of layers in the CNN) spatial and temporal information.


Instruction: How can I plot pytorch tensor?

Input:
I would like to plot pytorch gpu tensor:
input= torch.randn(100).to(device)
output = torch.where(input>=0, input, -input)

input = input.('cpu').detach().numpy().copy()
output = output.('cpu').detach().numpy().copy()

plt.plot(input,out)

However I try to convert those tensors into cpu, numpy, it does not work. How can I plot the tensors ?

Output:
Does this work?
plt.plot(input.cpu().numpy(),output.cpu().numpy())

Alternatively you can try,
plt.plot(input.to('cpu').numpy(),output.to('cpu').numpy())


Instruction: compress two consecutive dimension in PyTorch

Input:
I'm looking for such a function Tensor.compress(*dims) where dims=(int ...) is a consecutive sequence of integer:
a = torch.rand(2,2,3)
b = a.compress(0,1)

b.size()
>>> (4,3)

I know view would work, however, in case I don't know the shape of a in advance, I have to do an extra operation to acquire its size and then do view, which is not what I want.

Output:
You do not have to explicitly "do the math", torch.view can do some of it for you if you use -1 as the shape of one of the dimensions:
b = a.view(-1, *a.shape[2:])
b.shape
>>> torch.Size([4, 3])


Instruction: Implementing dropout with pytorch

Input:
I wonder if I want to implement dropout by myself, is something like the following sufficient (taken from Implementing dropout from scratch):
class MyDropout(nn.Module):
    def __init__(self, p: float = 0.5):
        super(MyDropout, self).__init__()
        if p < 0 or p > 1:
            raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
        self.p = p

    def forward(self, X):
        if self.training:
            binomial = torch.distributions.binomial.Binomial(probs=1-self.p)
            return X * binomial.sample(X.size()) * (1.0/(1-self.p))
        return X

My concern is even if the unwanted weights are masked out (either through this way or by using a mask tensor), there can still be gradient flow through the 0 weights (https://discuss.pytorch.org/t/custom-connections-in-neural-network-layers/3027/9). Is my concern valid?

Output:
DropOut does not mask the weights - it masks the features.
For linear layers implementing y = <w, x> the gradient w.r.t the parameters w is x. Therefore, if you set entries in x to zero - it will amount to no update for the corresponding weight in the adjacent linear layer.


Instruction: How to test a trained model saved in .pth.tar files?

Input:
I am working with CORnet-Z and I am building a separate test file.
The model seems to be saved as .pth.tar files
            if FLAGS.output_path is not None:
            records.append(results)
            if len(results) > 1:
                pickle.dump(records, open(os.path.join(FLAGS.output_path, 'results.pkl'),
                                          'wb'))

            ckpt_data = {}
            ckpt_data['flags'] = FLAGS.__dict__.copy()
            ckpt_data['epoch'] = epoch
            ckpt_data['state_dict'] = model.state_dict()
            ckpt_data['optimizer'] = trainer.optimizer.state_dict()

            if save_model_secs is not None:
                if time.time() - recent_time > save_model_secs:
                    torch.save(ckpt_data, os.path.join(FLAGS.output_path,
                                                       'latest_checkpoint.pth.tar'))
                    recent_time = time.time()

What would be the best approach to load this model and run evaluation and testing?

Output:
def load_checkpoint(checkpoint, model, optimizer = None):

    if not os.path.exists(checkpoint):
        raise("File does not exists {}".format(checkpoint))

    checkpoint = torch.load(checkpoint)
    model.load_state_dict(checkpoint['state_dict'])

    if optimizer:
        optimizer.load_state_dict(checkpoint['optim_dict'])

    return checkpoint

To test a model you need to load the state dictionary of your trained model and optimizer (if applicable). But, if you are resuming training from a point and you are using any sort of scheduler you need to load the scheduler state too.


Instruction: How to load two neural networks in pytorch

Input:
Let's say I have two jupyter notebooks, one for each neural network. They both are for binary classification. I want to combine the results of two networks, for example I want the probability of x belonging to a class to be 0.2model1(x)+0.8model2(x).
I saved both models with
torch.save(model.state_dict(), 'saved_networks/model1.pt')

Now I saw that in order to load a model I first have to create an object of this class
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))

But I have these classes in two different notebooks, so my question is how does one usually handles such situations?

Output:
I would suggest you write the two classes properly in .py files. This way you can import those classes anywhere you want (let it be a notebook or another python file). For instance, if you have Model1 and Model2 classes defined in models.py you can then import them, initialize separate models and load their respective state dictionaries:
from models import Model1, Model2

model1 = Model1(*args, **kwargs)
model2 = Model2(*args, **kwargs)

model1.load_state_dict(torch.load(PATH1))
model2.load_state_dict(torch.load(PATH2))


Instruction: Gradient Matrix (NxWxEPOCH) using Pytorch

Input:
I'm trying to create a matrix of gradient with the gradient of each observation by parameters and EPOCH. If my model has 100 obs, 1000 params and 10 Epoch, my matrix should be (100,1000,10).
The problem is that I'm not able to get those gradient. The parameters and the observation are set at required_gradient=True.
I've tried to run this after each observation pass thru the net:
 for p in net.parameters():
     paramgradlist.append(p.grad)

But the gradient stays the same of each params stays the same for all observations.
Thank you

Output:
You are not copying your data and instead of storing a reference to the gradients. In the end, this means all your observations will be the same (i.e. the gradients' final value).
Instead, you could clone the gradients before appending them to the list:
for p in net.parameters():
    paramgradlist.append(p.grad.clone())


Instruction: issue with arcface ( 0 accuracy)

Input:
Hello guys I've joined a university-level image recognition competition.

In the test, they will give two images (people face) and my model need to detect pair of the image is the same person or not

My model is resnet18 with IR block and SE block. and it will use Arcface loss.

I can use only the MS1M dataset with a total of 86876 classes

The problem is that loss is getting better, but accuracy is 0 and not changing.
Here's part of code I'm working on.
Train
def train_model(model, net, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        for phase in ['train']:
            if phase == 'train':
                model.train()  # Set model to training mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in notebook.tqdm(dataloader):
                inputs = inputs.to(device)
                labels = labels.to(device).long()

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    features = model(inputs)
                    outputs = net(features, labels)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / len(dataloader)
            epoch_acc = running_corrects.double() / len(dataloader)

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'train' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
        torch.save({'epoch': epoch,
                    'mode_state_dict': model.state_dict(),
                    'fc_state_dict': net.state_dict(),
                    'optimizer_state_dict': optimizer.state_dict(),
                    'scheduler': scheduler.state_dict(),    # HERE IS THE CHANGE
                    }, f'/content/drive/MyDrive/inha_data/training_saver/training_stat{epoch}.pth')
        print(f'finished {epoch} and saved model_save_{epoch}.pt')
        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best train Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    torch.save(model.state_dict(), 'model_save.pt')
    return model

Parameters
train_dataset = MS1MDataset('train')
dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True,num_workers=4)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") # 디바이스 설정

num_classes = 86876
# normal classifier
# net = nn.Sequential(nn.Linear(512, num_classes))
# Feature extractor backbone, input is 112x112 image output is 512 feature vector
model_ft = resnet18(True)
#set metric
metric_fc = metrics.ArcMarginProduct(512, num_classes, s = 30.0, m = 0.50, easy_margin = False)
metric_fc.to(device)
# net = net.to(device)
model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()
# Observe that all parameters are being optimized
optimizer_ft = torch.optim.Adam([{'params': model_ft.parameters()}, {'params': metric_fc.parameters()}],
                                     lr=0.1)
# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=4, gamma=0.1)

Arcface
from __future__ import print_function
from __future__ import division
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import Parameter
import math

class ArcMarginProduct(nn.Module):
    r"""Implement of large margin arc distance: :
        Args:
            in_features: size of each input sample
            out_features: size of each output sample
            s: norm of input feature
            m: margin
            cos(theta + m)
        """
    def __init__(self, in_features, out_features, s=30.0, m=0.50, easy_margin=False):
        super(ArcMarginProduct, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.s = s
        self.m = m
        self.weight = Parameter(torch.FloatTensor(out_features, in_features))
        nn.init.xavier_uniform_(self.weight)

        self.easy_margin = easy_margin
        self.cos_m = math.cos(m)
        self.sin_m = math.sin(m)
        self.th = math.cos(math.pi - m)
        self.mm = math.sin(math.pi - m) * m

    def forward(self, input, label):
        # --------------------------- cos(theta) & phi(theta) ---------------------------
        cosine = F.linear(F.normalize(input), F.normalize(self.weight))
        sine = torch.sqrt((1.0 - torch.pow(cosine, 2)).clamp(0, 1))
        phi = cosine * self.cos_m - sine * self.sin_m
        if self.easy_margin:
            phi = torch.where(cosine > 0, phi, cosine)
        else:
            phi = torch.where(cosine > self.th, phi, cosine - self.mm)
        # --------------------------- convert label to one-hot ---------------------------
        # one_hot = torch.zeros(cosine.size(), requires_grad=True, device='cuda')
        one_hot = torch.zeros(cosine.size(), device='cuda')
        one_hot.scatter_(1, label.view(-1, 1).long(), 1)
        # -------------torch.where(out_i = {x_i if condition_i else y_i) -------------
        output = (one_hot * phi) + ((1.0 - one_hot) * cosine)  # you can use torch.where if your torch.__version__ is 0.4
        output *= self.s
        # print(output)

        return output

dataset
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomHorizontalFlip(),
        transforms.ColorJitter(brightness=0.125, contrast=0.125, saturation=0.125),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}
#train_ms1_data = torchvision.datasets.ImageFolder('/content/drive/MyDrive/inha_data/train', transform = data_transforms)

class MS1MDataset(Dataset):

    def __init__(self,split):

        self.file_list = '/content/drive/MyDrive/inha_data/ID_List.txt'
        self.images = []
        self.labels = []
        self.transformer = data_transforms['train']
        with open(self.file_list) as f:
            files = f.read().splitlines()
        for i, fi in enumerate(files):
            fi = fi.split()
            image = "/content/" + fi[1]
            label = int(fi[0])
            self.images.append(image)
            self.labels.append(label)

    def __getitem__(self, index):

        img = Image.open(self.images[index])
        img = self.transformer(img)

        label = self.labels[index]
        return img, label

    def __len__(self):
        return len(self.images)

Output: You can try to use a smaller m in ArcFace, even a minus value.


Instruction: Difference between freezing layer with requires_grad and not passing params to optim in PyTorch

Input:
Let's say I train an autoencoder.
I want to freeze the parameters of the encoder for the training, so only the decoder trains.
I can do this using:
# assuming it's a single layer called 'encoder'
model.encoder.weights.data.requers_grad = False

Or I can pass only the decoder's parameters to the optimizer. Is there a difference?

Output:
The most practical way is to iterate through all parameters of the module you want to freeze and set required_grad to False. This gives you the flexibility to switch your modules on and off without having to initialize a new optimizer each time. You can do this using the parameters generator available on all nn.Modules:
for param in module.parameters():
    param.requires_grad = False

This method is model agnostic since you don't have to worry whether your module contains multiple layers or sub-modules.

Alternatively, you can call the function nn.Module.requires_grad_ once as:
module.requires_grad_(False)


Instruction: How to check torch gpu compatibility without initializing CUDA?

Input:
Older GPUs don't seem to support torch in spite of recent cuda versions.
In my case the crash has the following error:
/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning:
    Found GPU%d %s which is of cuda capability %d.%d.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is %d.%d.

  warnings.warn(old_gpu_warn.format(d, name, major, minor, min_arch // 10, min_arch % 10))
WARNING:lightwood-16979:Exception: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1. when training model: <lightwood.model.neural.Neural object at 0x7f9c34df1e80>
Process LearnProcess-1:13:
Traceback (most recent call last):
  File "/home/maxs/dev/mdb/venv38/sources/lightwood/lightwood/model/helpers/default_net.py", line 59, in forward
    output = self.net(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

This happens in spite of:
assert torch.cuda.is_available() == True
torch.version.cuda == '10.2'

How can I check for an older GPU that doesn't support torch without actually try/catching a tensor-to-gpu transfer? The transfer initializes cuda, which wastes like 2GB of memory, something I can't afford since I'd be running this check in dozens of processes, all of which would then waste 2GB of memory extra due to the initialization.

Output:
Based on the code in torch.cuda.__init__ that was actually throwing the error the following check seems to work:
import torch
from torch.cuda import device_count, get_device_capability

def is_cuda_compatible():
    compatible_device_count = 0
    if torch.version.cuda is not None:
        for d in range(device_count()):
            capability = get_device_capability(d)
            major = capability[0]
            minor = capability[1]
            current_arch = major * 10 + minor
            min_arch = min((int(arch.split("_")[1]) for arch in torch.cuda.get_arch_list()), default=35)
            if (not current_arch < min_arch
                    and not torch._C._cuda_getCompiledVersion() <= 9000):
                compatible_device_count += 1

    if compatible_device_count > 0:
        return True
    return False

Not sure if it's 100% correct but putting it out here for feedback and in case anybody else needs it.


Instruction: ValueError: expected sequence of length 0 at dim 2 (got 1)

Input:
I've recently started a tutorial about Neural Networks with Python.
I am working on a cat/dog classification task with a CNN. However even though I thought I've done exactly what the tutorial told me to do, I somehow ended up with a dim error.
This is the tutorial.
I believe he uses Python 3.7, I'm using Python 3.9(64-bit).

The Error: ValueError: expected sequence of length 0 at dim 2 (got 1)
 The line of code: y = torch.Tensor([i[1] for i in training_data])

It sounds like I might have made a mistake in preparing the training data, but I'm not sure. Here is the code for that:
class DogsVSCats:
    IMG_SIZE = 50
    CATS = '[Path]'
    DOGS = '[Path]'

    LABELS = {CATS: 0, DOGS: 1}

    training_data = []

    catcount = 0
    dogcount = 0

    def make_training_data(self):
        for label in self.LABELS:
            print label

            for f in tqdm(os.listdir(label)):
                try:
                    path = os.path.join(label, f)
                    img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                    img = cv2.resize(img, (self.IMG_SIZE,
                            self.IMG_SIZE))

                    self.training_data.append([np.array(img), np.eye(2,
                            self.LABELS[label])])

                    if label == self.CATS:
                        self.catcount += 1
                    elif label == self.DOGS:

                        self.dogcount += 1
                except Exception, e:

                    pass

            np.random.shuffle(self.training_data)
            np.save('training_data.npy', self.training_data)

            print ('Cats: ', self.catcount)
            print ('Dogs: ', self.dogcount)

    if REBUILD_DATA:
        dogsvcats = DogsVSCats()
        dogsvcats.make_training_data()

print 'Nothing found!!'

That all seemed to work as it did in the tutorial, with no errors and showing the same amount of pictures for each category. Here is also the problematic line:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50, 50).view(-1, 1, 50, 50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512)
        self.fc2 = nn.Linear(512, 2)

    def convs(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0] * x[0].shape[1] * x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)

        return F.softmax(x, dim = 1)

net = Net()

optimizer = optim.Adam(net.parameters(), lr = 1e-3)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1, 50, 50)
X = X / 255.0
y = torch.Tensor([i[1] for i in training_data]) !!!Error Line!!!

VAL_PCT = 0.1
val_size = int(len(X) * VAL_PCT)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

print(val_size)

Output:
You didn't define the labels properly, it shouldn't be
np.eye(2, self.LABELS[label])

but instead:
np.eye(2)[self.LABELS[label]]


Instruction: How to convert numpy array(float data) to torch tensor?

Input:
test = ['0.01171875', '0.01757812', '0.02929688']
test = np.array(test).astype(float)
print(test)
->[0.01171875 0.01757812 0.02929688]

test_torch = torch.from_numpy(test)
test_torch
->tensor([0.0117, 0.0176, 0.0293], dtype=torch.float64)

It looks like from_numpy() loses some precision there... If I want to convert this float data exactly the same, what kind of functions do I use?

Output:
The data precision is the same, it's just that the format used by PyTorch to print the values is different, it will round the floats down:
>>> test_torch = torch.from_numpy(test)
>>> test_torch
tensor([0.0117, 0.0176, 0.0293], dtype=torch.float64)

You can check that it matches your original input by converting to a list with tolist:
>>> test_torch.tolist()
[0.01171875, 0.01757812, 0.02929688]


Instruction: what does reshape`(1, 1, 28, 28)` mean

Input:
I'm trying to reproduce the results of this code:
https://github.com/vjayd/Image-Alignment-using-CNN
the first problem I've faced is that, as far as I know, MNIST data are gray images and not color images
so why he converted them to grayscale images using rgb2gray function
for img_train in glob.glob(trdata):
    n = io.imread(img_train)
    n = rgb2gray(n)
    n= resize(n,(28,28))
    train_x.append(n.reshape(1, 28, 28))

and what does (1, 1, 28, 28) mean in this line
test_x = test_x.reshape(1, 1, 28, 28)

Output: A Pytorch model mostly requires the first dimension of the input to be the batch size. So the shape of the image is (1, 28, 28). If you want to feed only one image to the model you still have to specify the batch size, which is of course 1 for one image. Therefore he adds the batch size dimension to the image by "reshaping" it to (1, 1, 28, 28).


Instruction: Modify training function of 3 nets with shared classifier

Input:
I have 3 VGG: VGGA, VGGB and VGG*, trained with the following training function:
def train(nets, loaders, optimizer, criterion, epochs=20, dev=None, save_param=False, model_name="valerio"):
    # try:
      nets = [n.to(dev) for n in nets]

      model_a = module_unwrap(nets[0], True)
      model_b = module_unwrap(nets[1], True)
      model_c = module_unwrap(nets[2], True)

      reg_loss = nn.MSELoss()

      criterion.to(dev)
      reg_loss.to(dev)

      # Initialize history
      history_loss = {"train": [], "val": [], "test": []}
      history_accuracy = {"train": [], "val": [], "test": []}
      # Store the best val accuracy
      best_val_accuracy = 0

      # Process each epoch
      for epoch in range(epochs):
        # Initialize epoch variables
        sum_loss = {"train": 0, "val": 0, "test": 0}
        sum_accuracy = {"train": [0,0,0], "val": [0,0,0], "test": [0,0,0]}

        progbar = None
        # Process each split
        for split in ["train", "val", "test"]:
          if split == "train":
            for n in nets:
              n.train()
            widgets = [
              ' [', pb.Timer(), '] ',
              pb.Bar(),
              ' [', pb.ETA(), '] ', pb.Variable('ta','[Train Acc: {formatted_value}]')
            ]

            progbar = pb.ProgressBar(max_value=len(loaders[split][0]),widgets=widgets,redirect_stdout=True)

          else:
            for n in nets:
              n.eval()
          # Process each batch
          for j,((input_a, labels_a),(input_b, labels_b)) in enumerate(zip(loaders[split][0],loaders[split][1])):

            input_a = input_a.to(dev)
            input_b = input_b.to(dev)

            labels_a = labels_a.long().to(dev)
            labels_b = labels_b.long().to(dev)
            #print(labels_a.shape)
            #labels_a = labels_a.squeeze()
            #labels_b = labels_b.squeeze()

            #labels_a = labels_a.unsqueeze(1)
            #labels_b = labels_b.unsqueeze(1)
            #print(labels_a.shape)
            #labels_a = labels_a.argmax(-1)
            #labels_b = labels_b.argmax(-1)

            inputs = torch.cat([input_a,input_b],axis=0)
            labels = torch.cat([labels_a, labels_b])

            #labels  = labels.squeeze()
            #print(labels.shape)
            #labels = labels.argmax(-1)

            # Reset gradients
            optimizer.zero_grad()
            # Compute output
            features_a = nets[0](input_a)
            features_b = nets[1](input_b)
            features_c = nets[2](inputs)

            pred_a = torch.squeeze(nets[3](features_a))
            pred_b = torch.squeeze(nets[3](features_b))
            pred_c = torch.squeeze(nets[3](features_c))

            loss = criterion(pred_a, labels_a) + criterion(pred_b, labels_b) + criterion(pred_c, labels)

            for n in model_a:
              layer_a = model_a[n]
              layer_b = model_b[n]
              layer_c = model_c[n]
              if (isinstance(layer_a,nn.Conv2d)):
                loss += lambda_reg * reg_loss(combo_fn(layer_a.weight,layer_b.weight),layer_c.weight)
                if (layer_a.bias is not None):
                  loss += lambda_reg * reg_loss(combo_fn(layer_a.bias, layer_b.bias), layer_c.bias)

            # Update loss
            sum_loss[split] += loss.item()
            # Check parameter update
            if split == "train":
              # Compute gradients
              loss.backward()
              # Optimize
              optimizer.step()

            # Compute accuracy

            #https://discuss.pytorch.org/t/bcewithlogitsloss-and-model-accuracy-calculation/59293/ 2
            #pred_labels_a = (pred_a >= 0.0).long()  # Binarize predictions to 0 and 1
            #pred_labels_b = (pred_b >= 0.0).long()  # Binarize predictions to 0 and 1
            #pred_labels_c = (pred_c >= 0.0).long()  # Binarize predictions to 0 and 1

            #print(pred_a.shape)

            _,pred_label_a = torch.max(pred_a, dim = 1)
            pred_labels_a = (pred_label_a == labels_a).float()

            _,pred_label_b = torch.max(pred_b, dim = 1)
            pred_labels_b = (pred_label_b == labels_b).float()

            _,pred_label_c = torch.max(pred_c, dim = 1)
            pred_labels_c = (pred_label_c == labels).float()

            batch_accuracy_a = pred_labels_a.sum().item() / len(labels_a)
            batch_accuracy_b = pred_labels_b.sum().item() / len(labels_b)
            batch_accuracy_c = pred_labels_c.sum().item() / len(labels)

            # Update accuracy
            sum_accuracy[split][0] += batch_accuracy_a
            sum_accuracy[split][1] += batch_accuracy_b
            sum_accuracy[split][2] += batch_accuracy_c

            if (split=='train'):
              progbar.update(j, ta=batch_accuracy_c)

        if (progbar is not None):
          progbar.finish()
        # Compute epoch loss/accuracy
        epoch_loss = {split: sum_loss[split] / len(loaders[split][0]) for split in ["train", "val", "test"]}
        epoch_accuracy = {split: [sum_accuracy[split][i] / len(loaders[split][0]) for i in range(len(sum_accuracy[split])) ] for split in ["train", "val", "test"]}

        # # Store params at the best validation accuracy
        # if save_param and epoch_accuracy["val"] > best_val_accuracy:
        #   # torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
        #   torch.save(net.state_dict(), f"{model_name}_best_val.pth")
        #   best_val_accuracy = epoch_accuracy["val"]

        print(f"Epoch {epoch + 1}:")
        # Update history
        for split in ["train", "val", "test"]:
          history_loss[split].append(epoch_loss[split])
          history_accuracy[split].append(epoch_accuracy[split])
          # Print info
          print(f"\t{split}\tLoss: {epoch_loss[split]:0.5}\tVGG 1:{epoch_accuracy[split][0]:0.5}"
                f"\tVGG 2:{epoch_accuracy[split][1]:0.5}\tVGG *:{epoch_accuracy[split][2]:0.5}")

      if save_param:
        torch.save({'vgg_a':nets[0].state_dict(),'vgg_b':nets[1].state_dict(),'vgg_star':nets[2].state_dict(),'classifier':nets[3].state_dict()},f'{model_name}.pth')

For each epoch of training the result is this:

Then, I have a combined model which sums the weights of VGGA and VGGB:
DO = 'TEST'
if (DO=='TRAIN'):
  train(nets, loaders, optimizer, criterion, epochs=50, dev=dev,save_param=True)
else:
  state_dicts = torch.load('valerio.pth')
  model1.load_state_dict(state_dicts['vgg_a']) #questi state_dict vengono dalla funzione di training
  model2.load_state_dict(state_dicts['vgg_b'])
  model3.load_state_dict(state_dicts['vgg_star'])
  classifier.load_state_dict(state_dicts['classifier'])

  test(model1,classifier,test_loader_all)
  test(model2, classifier, test_loader_all)
  test(model3, classifier, test_loader_all)

  summed_state_dict = OrderedDict()

  for key in state_dicts['vgg_star']:
    if key.find('conv') >=0:
      print(key)
      summed_state_dict[key] = combo_fn(state_dicts['vgg_a'][key],state_dicts['vgg_b'][key])
    else:
      summed_state_dict[key] = state_dicts['vgg_star'][key]

  model3.load_state_dict(summed_state_dict)
  test(model3, classifier, test_loader_all)

where the test function is this:
def test(net,classifier, loader):

      net.to(dev)
      classifier.to(dev)

      net.eval()

      sum_accuracy = 0

      # Process each batch
      for j, (input, labels) in enumerate(loader):

        input = input.to(dev)
        labels = labels.float().to(dev)

        features = net(input)

        pred = torch.squeeze(classifier(features))

        # https://discuss.pytorch.org/t/bcewithlogitsloss-and-model-accuracy-calculation/59293/ 2
        #pred_labels = (pred >= 0.0).long()  # Binarize predictions to 0 and 1
        _,pred_label = torch.max(pred, dim = 1)
        pred_labels = (pred_label == labels).float()

        batch_accuracy = pred_labels.sum().item() / len(labels)

        # Update accuracy
        sum_accuracy += batch_accuracy

      epoch_accuracy = sum_accuracy / len(loader)

      print(f"Accuracy after sum: {epoch_accuracy:0.5}")

And the result of this aggregation is the following:

I want to modify my training function in order to print the same things of the first image, plus the accuracy of the aggregated model (the highlighted part in red of the second picture). So basically, for each epoch, accuracies of VGGA, VGGB, VGG* and combined VGG, print these accuracies and continue with the training. I tried to add this model combo but I failed, because I did not able to insert into each epoch, but only at the end of the training. I was trying to add in the training function, between print(f"Epoch {epoch + 1}:")and
# Update history
for split in ["train", "val", "test"]:

the code with the part of state_dict, but i am doing something wrong, i do not know what.
Can I reuse the code of the test, or I have to write new code?
Do you think i have to save the state_dict for each epoch, or i can do something else? Like model_c.parameters()=model_a.parameters()+model_b.parameters() (which does not work, already tried)

Output:
I solved, here is the solution of how I modified my training function:
def train(nets, loaders, optimizer, criterion, epochs=20, dev=None, save_param=False, model_name="valerio"):
  # try:
  nets = [n.to(dev) for n in nets]

  model_a = module_unwrap(nets[0], True)
  model_b = module_unwrap(nets[1], True)
  model_c = module_unwrap(nets[2], True)

  reg_loss = nn.MSELoss()

  criterion.to(dev)
  reg_loss.to(dev)

  # Initialize history
  history_loss = {"train": [], "val": [], "test": []}
  history_accuracy = {"train": [], "val": [], "test": []}
  history_test = 0
  # Store the best val accuracy
  best_val_accuracy = 0

  # Process each epoch
  for epoch in range(epochs):
    # Initialize epoch variables
    sum_loss = {"train": 0, "val": 0, "test": 0}
    sum_accuracy = {"train": [0,0,0], "val": [0,0,0], "test": [0,0,0]}

    progbar = None
    # Process each split
    for split in ["train", "val", "test"]:
      if split == "train":
        for n in nets:
          n.train()
        widgets = [
          ' [', pb.Timer(), '] ',
          pb.Bar(),
          ' [', pb.ETA(), '] ', pb.Variable('ta','[Train Acc: {formatted_value}]')
        ]

        progbar = pb.ProgressBar(max_value=len(loaders[split][0]),widgets=widgets,redirect_stdout=True)

      else:
        for n in nets:
          n.eval()
      # Process each batch
      for j,((input_a, labels_a),(input_b, labels_b)) in enumerate(zip(loaders[split][0],loaders[split][1])):

        input_a = input_a.to(dev)
        input_b = input_b.to(dev)

        labels_a = labels_a.long().to(dev)
        labels_b = labels_b.long().to(dev)
        #print(labels_a.shape)
        #labels_a = labels_a.squeeze()
        #labels_b = labels_b.squeeze()

        #labels_a = labels_a.unsqueeze(1)
        #labels_b = labels_b.unsqueeze(1)
        #print(labels_a.shape)
        #labels_a = labels_a.argmax(-1)
        #labels_b = labels_b.argmax(-1)

        inputs = torch.cat([input_a,input_b],axis=0)
        labels = torch.cat([labels_a, labels_b])

        #labels  = labels.squeeze()
        #print(labels.shape)
        #labels = labels.argmax(-1)

        # Reset gradients
        optimizer.zero_grad()
        # Compute output
        features_a = nets[0](input_a)
        features_b = nets[1](input_b)
        features_c = nets[2](inputs)

        pred_a = torch.squeeze(nets[3](features_a))
        pred_b = torch.squeeze(nets[3](features_b))
        pred_c = torch.squeeze(nets[3](features_c))

        loss = criterion(pred_a, labels_a) + criterion(pred_b, labels_b) + criterion(pred_c, labels)

        for n in model_a:
          layer_a = model_a[n]
          layer_b = model_b[n]
          layer_c = model_c[n]
          if (isinstance(layer_a,nn.Conv2d)):
            loss += lambda_reg * reg_loss(combo_fn(layer_a.weight,layer_b.weight),layer_c.weight)
            if (layer_a.bias is not None):
              loss += lambda_reg * reg_loss(combo_fn(layer_a.bias, layer_b.bias), layer_c.bias)

        # Update loss
        sum_loss[split] += loss.item()
        # Check parameter update
        if split == "train":
          # Compute gradients
          loss.backward()
          # Optimize
          optimizer.step()

        # Compute accuracy

        #https://discuss.pytorch.org/t/bcewithlogitsloss-and-model-accuracy-calculation/59293/ 2
        #pred_labels_a = (pred_a >= 0.0).long()  # Binarize predictions to 0 and 1
        #pred_labels_b = (pred_b >= 0.0).long()  # Binarize predictions to 0 and 1
        #pred_labels_c = (pred_c >= 0.0).long()  # Binarize predictions to 0 and 1

        #print(pred_a.shape)

        _,pred_label_a = torch.max(pred_a, dim = 1)
        pred_labels_a = (pred_label_a == labels_a).float()

        _,pred_label_b = torch.max(pred_b, dim = 1)
        pred_labels_b = (pred_label_b == labels_b).float()

        _,pred_label_c = torch.max(pred_c, dim = 1)
        pred_labels_c = (pred_label_c == labels).float()

        batch_accuracy_a = pred_labels_a.sum().item() / len(labels_a)
        batch_accuracy_b = pred_labels_b.sum().item() / len(labels_b)
        batch_accuracy_c = pred_labels_c.sum().item() / len(labels)

        # Update accuracy
        sum_accuracy[split][0] += batch_accuracy_a
        sum_accuracy[split][1] += batch_accuracy_b
        sum_accuracy[split][2] += batch_accuracy_c

        if (split=='train'):
          progbar.update(j, ta=batch_accuracy_c)

    if (progbar is not None):
      progbar.finish()
    # Compute epoch loss/accuracy
    epoch_loss = {split: sum_loss[split] / len(loaders[split][0]) for split in ["train", "val", "test"]}
    epoch_accuracy = {split: [sum_accuracy[split][i] / len(loaders[split][0]) for i in range(len(sum_accuracy[split])) ] for split in ["train", "val", "test"]}

    # # Store params at the best validation accuracy
    # if save_param and epoch_accuracy["val"] > best_val_accuracy:
    #   # torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
    #   torch.save(net.state_dict(), f"{model_name}_best_val.pth")
    #   best_val_accuracy = epoch_accuracy["val"]

    print(f"Epoch {epoch + 1}:")
    # Update history
    for split in ["train", "val", "test"]:
      history_loss[split].append(epoch_loss[split])
      history_accuracy[split].append(epoch_accuracy[split])
      # Print info
      print(f"\t{split}\tLoss: {epoch_loss[split]:0.5}\tVGG 1:{epoch_accuracy[split][0]:0.5}"
            f"\tVGG 2:{epoch_accuracy[split][1]:0.5}\tVGG *:{epoch_accuracy[split][2]:0.5}")

    if save_param:
      torch.save({'vgg_a':nets[0].state_dict(),'vgg_b':nets[1].state_dict(),'vgg_star':nets[2].state_dict(),'classifier':nets[3].state_dict()},f'{model_name}.pth')

    test(nets[0], nets[3], test_loader_all)
    test(nets[1], nets[3], test_loader_all)
    test(nets[2], nets[3], test_loader_all)

    summed_state_dict = OrderedDict()

    for key in nets[2].state_dict():
      if key.find('conv') >=0:
        #print(key)
        summed_state_dict[key] = combo_fn(nets[0].state_dict()[key],nets[1].state_dict()[key])
      else:
        summed_state_dict[key] = nets[2].state_dict()[key]

    nets[2].load_state_dict(summed_state_dict)
    test(nets[2], nets[3], test_loader_all)

The edited parts are the last rows.


Instruction: Pytorch Lightning limit_val_batches and val_check_interval behavior

Input:
I'm setting limit_val_batches=10 and val_check_interval=1000 so that I'm validating on 10 validation batches every 1000 training steps. Is it guaranteed that Trainer will use the same 10 batches every time validation is called?
I tried search the source code for limit_val_batches but couldn't figure out how it was being used to obtain the validation batches.

Output:
The answer doesn't have much to do with PyTorch Lightning and its flags (--limit_val_batches and --val_check_interval). The exact batches of data provided by Lightning inside any of the
def *_step(self, batch, ...):
    ...

methods (* is training/validation/test) is determined by the underlying PyTorch DataLoaders returned by
def *_dataloder(...):
    return DataLoader(dataset, shuffle=..., sampler=..., batch_sampler=...)

If the dataloader returned by these functions DO NOT have shuffle=True or any randomized Samplers, the batches will be same.
As far as --limit_val_batches=N is concerned, it fetches first N batches from the underlying dataloader. Lightning doesn't do any data selection by itself. It is confirmed by a core developer here.