Instruction: Cannot perform reduction function min on tensor with no elements because the operation does not have an identity at THCTensorMathReduce.cu:64

Input:
I am configuring a GitHub repo in which the author stated that you have to install pytorch=0.4 and python=3.7. Now, I have CUDA 11.0 and the Pytorch version is conflicting with CUDA. After installing the Pytorch it gives the below error. Any hint?
My Conda List
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_pytorch_select           0.1                       cpu_0
blas                      1.0                         mkl
ca-certificates           2021.1.19            h06a4308_1
certifi                   2020.12.5        py37h06a4308_0
cffi                      1.14.0           py37h2e261b9_0
cuda100                   1.0                           0    pytorch
cudatoolkit               9.2                           0
freetype                  2.10.4               h5ab3b9f_0
intel-openmp              2019.4                      243
jpeg                      9b                   h024ee3a_2
lcms2                     2.11                 h396b838_0
libedit                   3.1.20210216         h27cfd23_1
libffi                    3.2.1             hf484d3e_1007
libgcc-ng                 9.1.0                hdf63c60_0
libmklml                  2019.0.5                      0
libpng                    1.6.37               hbc83047_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.2.0                h3942068_0
libwebp-base              1.2.0                h27cfd23_0
lz4-c                     1.9.3                h2531618_0
mkl                       2020.2                      256
mkl-service               2.3.0            py37he8ac12f_0
mkl_fft                   1.3.0            py37h54f3939_0
mkl_random                1.1.1            py37h0573a6f_0
ncurses                   6.2                  he6710b0_1
ninja                     1.10.2           py37hff7bd54_0
numpy                     1.20.1                   pypi_0    pypi
numpy-base                1.19.2           py37hfa32c7d_0
olefile                   0.46                     py37_0
openssl                   1.1.1j               h27cfd23_0
pillow                    8.1.2            py37he98fc37_0
pip                       21.0.1           py37h06a4308_0
pycparser                 2.20                       py_2
python                    3.7.1                h0371630_7
pytorch                   1.2.0           py3.7_cuda9.2.148_cudnn7.6.2_0    pytorch
readline                  7.0                  h7b6447c_5
scipy                     1.6.1                    pypi_0    pypi
setuptools                52.0.0           py37h06a4308_0
six                       1.15.0           py37h06a4308_0
sqlite                    3.33.0               h62c20be_0
tk                        8.6.10               hbc83047_0
torchaudio                0.8.0                    pypi_0    pypi
torchvision               0.4.0                 py37_cu92    pytorch
typing                    3.7.4.3          py37h06a4308_0
typing-extensions         3.7.4.3              hd3eb1b0_0
typing_extensions         3.7.4.3            pyh06a4308_0
wheel                     0.36.2             pyhd3eb1b0_0
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

Error
Model size: 44.17957M
==> Epoch 1/800  lr:0.0003
Traceback (most recent call last):
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 369, in <module>
    main()
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 230, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 279, in train
    htri_loss = criterion_htri(features, pids)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/losses.py", line 86, in forward
    dist_an.append(dist[i][mask[i] == 0].min())
RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /opt/conda/conda-bld/pytorch_1565287025495/work/aten/src/THC/generic/THCTensorMathReduce.cu:64

Output:
As the error message suggests, the argument for min function is empty.
The behavior of torch.min([]) is undefined.
Check that dist[i][mask[i] == 0] is not empty, before taking min of it.


Instruction: Is torch.float32 different from numpy's float32?

Input:
Setting precision as 30 in PyTorch shows:
>>> torch.set_printoptions(precision=30)
>>> y
tensor([[-0.388252139091491699218750000000, -0.610148549079895019531250000000,
         -1.333969473838806152343750000000, -1.027917861938476562500000000000,
         -0.498563587665557861328125000000, -0.096793495118618011474609375000,
         -0.895992159843444824218750000000, -0.752071321010589599609375000000,
         -0.879653215408325195312500000000],
        [ 1.960780262947082519531250000000,  0.290681242942810058593750000000,
          0.111534759402275085449218750000, -1.412155270576477050781250000000,
          1.015806078910827636718750000000,  0.201809123158454895019531250000,
          0.131465137004852294921875000000, -1.262379050254821777343750000000,
         -0.480409622192382812500000000000],
        [ 0.111068181693553924560546875000, -2.129202365875244140625000000000,
          0.538800299167633056640625000000,  1.165832757949829101562500000000,
          0.194993987679481506347656250000, -1.110693812370300292968750000000,
         -1.451576709747314453125000000000, -3.398952484130859375000000000000,
          2.022404193878173828125000000000],
        [-2.374018669128417968750000000000, -1.442466974258422851562500000000,
         -0.406166225671768188476562500000,  0.045908458530902862548828125000,
         -0.835370421409606933593750000000, -0.302138328552246093750000000000,
         -0.421340197324752807617187500000,  0.931307554244995117187500000000,
          1.061386585235595703125000000000],
        [ 1.166660070419311523437500000000,  0.710260510444641113281250000000,
          1.008558034896850585937500000000,  1.957847237586975097656250000000,
         -1.070753335952758789062500000000,  0.319442749023437500000000000000,
         -1.140496969223022460937500000000, -1.723430752754211425781250000000,
          0.109533369541168212890625000000],
        [ 0.501820147037506103515625000000,  0.349833250045776367187500000000,
         -0.073374643921852111816406250000, -2.459295272827148437500000000000,
         -1.853959321975708007812500000000,  0.153838425874710083007812500000,
         -1.860147237777709960937500000000, -0.880943417549133300781250000000,
         -1.352821707725524902343750000000]])

but in numpy setting precision as 30 shows:
>>> np.set_printoptions(precision=30)
>>> y.numpy()
array([[-0.38825214 , -0.61014855 , -1.3339695  , -1.0279179  ,
        -0.4985636  , -0.096793495, -0.89599216 , -0.7520713  ,
        -0.8796532  ],
       [ 1.9607803  ,  0.29068124 ,  0.11153476 , -1.4121553  ,
         1.0158061  ,  0.20180912 ,  0.13146514 , -1.262379   ,
        -0.48040962 ],
       [ 0.11106818 , -2.1292024  ,  0.5388003  ,  1.1658328  ,
         0.19499399 , -1.1106938  , -1.4515767  , -3.3989525  ,
         2.0224042  ],
       [-2.3740187  , -1.442467   , -0.40616623 ,  0.04590846 ,
        -0.8353704  , -0.30213833 , -0.4213402  ,  0.93130755 ,
         1.0613866  ],
       [ 1.1666601  ,  0.7102605  ,  1.008558   ,  1.9578472  ,
        -1.0707533  ,  0.31944275 , -1.140497   , -1.7234308  ,
         0.10953337 ],
       [ 0.50182015 ,  0.34983325 , -0.073374644, -2.4592953  ,
        -1.8539593  ,  0.15383843 , -1.8601472  , -0.8809434  ,
        -1.3528217  ]], dtype=float32)e

Why are the results different?

Output:
By default, if it takes less digits than the configured value of precision to distinguish a floating-point value from other values of the same dtype, NumPy will only print as many digits as necessary for that. You have to set the floatmode option to 'fixed' to get the behavior you were expecting:
numpy.set_printoptions(precision=30, floatmode='fixed')

Note that even if you print 30 decimal digits, 32-bit floats don't have anywhere near that level of precision.


Instruction: Split a torch tensor using a same-sized tensor of indices

Input:
Let's say that I have tensor
t = torch.tensor([1,2,3,4,5])

I want to split it using a same-sized tensor of indices that tells me for each element, in which split it should go.
indices = torch.tensor([0,1,1,0,2])

So that the final result is
splits
[tensor([1,4]), tensor([2,3]), tensor([5])]

Is there a neat way to do this in Pytorch?
EDIT : In general there will be more than 2 or 3 splits.

Output:
One could do it using argsort for general case:
def mask_split(tensor, indices):
    sorter = torch.argsort(indices)
    _, counts = torch.unique(indices, return_counts=True)
    return torch.split(t[sorter], counts.tolist())

mask_split(t, indices)

Though it might be better to use @flawr answer if this is your real use case (also list comprehension might also be faster as it does not require sorting), something like this:
def mask_split(tensor, indices):
    unique = torch.unique(indices)
    return [tensor[indices == i] for i in unique]


Instruction: Installing PyTorch with CUDA in setup.py

Input:
I'm trying to specify PyTorch with CUDA in install_requires. The command to install with pip is
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio===0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
How do I do that in the setup.py install_requires?

Output:
I also faced same problem later I fixed it but using this in setup.py files and it worked, just add these lines as in your setup.py file.
"torch@https://download.pytorch.org/whl/cu111/torch-1.8.0%2Bcu111-cp37-cp37m-linux_x86_64.whl",
"torchvision@https://download.pytorch.org/whl/cu111/torchvision-0.9.0%2Bcu111-cp37-cp37m-linux_x86_64.wh",
"torchaudio@https://download.pytorch.org/whl/torchaudio-0.8.0-cp36-cp36m-linux_x86_64.whl"

All these are for linux version if you wantr any MACOSX or windows just change the link after @ in each line you can get link of your desired version from  https://download.pytorch.org/whl/torch_stable.html


Instruction: with torch.no_grad: AttributeError: __enter__

Input:
with torch.no_grad:AttributeError: __enter__

I got this error while running pytorch code.
I have torch==0.4.1 torchvision==0.3.0, I run the code in google colab.

Output:
torch.no_grad is a contextmanager it really has __enter__ and __exit__.
You should use it with with statement, like this
with context_manager():
    pass

Thus, simply replace with torch.no_grad: (accessing the attribute) with with torch.no_grad(): (calling a method) to use contextmanager properly.


Instruction: Same weights, implementation but different results n Keras and Pytorch

Input:
I have an encoder and a decoder model (monodepth2). I try convert them from Pytorch to Keras using Onnx2Keras, but :

Encoder(ResNet-18) succeeds
I build the decoder myself in Keras (with TF2.3), and copy the weights (numpy array, including weight and bias) for each layer from Pytorch to Keras, without any modification.

But it turns out both Onnx2Keras-converted Encoder and self-built Decoder fails to reproduce the same results. The cross-comparison pictures are below, but I'd first introduce the code of Decoder.
First the core Layer, all the conv2d layer (Conv3x3, ConvBlock) is based on this, but different dims or add an activation:
# Conv3x3 (normal conv2d without BN nor activation)
# There's also a ConvBlock, which is just "Conv3x3 + ELU activation", so I don't list it here.
def TF_Conv3x3(input_channel, filter_num, pad_mode='reflect', activate_type=None):

    # Actually it's 'reflect, but I implement it with tf.pad() outside this
    padding = 'valid'

    # if TF_ConvBlock, then activate_type=='elu
    conv = tf.keras.layers.Conv2D(filters=filter_num, kernel_size=3, activation=activate_type,
                                  strides=1, padding=padding)
    return conv

Then the structure. Note that the definition is EXACTLY the same as the original code. I think it must be some details about the implementation.
def DepthDecoder_keras(num_ch_enc=np.array([64, 64, 128, 256, 512]), channel_first=False,
                       scales=range(4), num_output_channels=1):
    num_ch_dec = np.array([16, 32, 64, 128, 256])
    convs = OrderedDict()
    for i in range(4, -1, -1):
        # upconv_0
        num_ch_in = num_ch_enc[-1] if i == 4 else num_ch_dec[i + 1]
        num_ch_out = num_ch_dec[i]

        # convs[("upconv", i, 0)] = ConvBlock(num_ch_in, num_ch_out)
        convs[("upconv", i, 0)] = TF_ConvBlock(num_ch_in, num_ch_out, pad_mode='reflect')

        # upconv_1
        num_ch_in = num_ch_dec[i]
        if i > 0:
            num_ch_in += num_ch_enc[i - 1]
        num_ch_out = num_ch_dec[i]
        convs[("upconv", i, 1)] = TF_ConvBlock(num_ch_in, num_ch_out, pad_mode='reflect')  # Just Conv3x3 with ELU-activation

    for s in scales:
        convs[("dispconv", s)] = TF_Conv3x3(num_ch_dec[s], num_output_channels, pad_mode='reflect')

    """
    Input_layer dims: (64, 96, 320), (64, 48, 160),  (128, 24, 80), (256, 12, 40), (512, 6, 20)
    """
    x0 = tf.keras.layers.Input(shape=(96, 320, 64))
    # then define the the rest input layers
    input_features = [x0, x1, x2, x3, x4]

    """
    # connect layers
    """
    outputs = []
    ch = 1 if channel_first else 3
    x = input_features[-1]
    for i in range(4, -1, -1):
        x = tf.pad(x, paddings=[[0, 0], [1, 1], [1, 1], [0, 0]], mode='REFLECT')
        x = convs[("upconv", i, 0)](x)
        x = [tf.keras.layers.UpSampling2D()(x)]
        if i > 0:
            x += [input_features[i - 1]]
        x = tf.concat(x, ch)
        x = tf.pad(x, paddings=[[0, 0], [1, 1], [1, 1], [0, 0]], mode='REFLECT')
        x = convs[("upconv", i, 1)](x)
    x = TF_ReflectPad2D_1()(x)
    x = convs[("dispconv", 0)](x)
    disp0 = tf.math.sigmoid(x)

    """
    build keras Model ([input0, ...], [output0, ...])
    """
    # decoder = tf.keras.Model(input_features, outputs)
    decoder = tf.keras.Model(input_features, disp0)

    return decoder

The cross-comparison is as follows... I would really appreciate it if anyone could offer some insights. Thanks!!!
Original results:

Original Encoder + Self-build Decoder:

ONNX-converted Enc + Original Dec (Texture is good, but the contrast is not enough, the car should be very close, i.e. very bright color):

ONNX-converted Enc + Self-built Dec:

Output:
Solved!
It turns out there's indeed no problem with implementation (at least not significant ones). It's the problem with weights copying.
The original weights has (H, W, 3, 3),  but TF-model requires dim of (3, 3, W, H), so I permuted it by [3,2,1,0], overlooking the (3, 3) also have their own sequence.
So it should be weights.permute([2,3,1,0]), and all is well!


Instruction: Torch JIT Trace = TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect

Input:
I am following this tutorial: https://huggingface.co/transformers/torchscript.html
to create a trace of my custom BERT model, however when running the exact same dummy_input I receive an error:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect.
We cant record the data flow of Python values, so this value will be treated as a constant in the future.

Having loaded in my model and tokenizer, the code to create the trace is the following:
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

traced_model = torch.jit.trace(model, dummy_input)

The dummy_input is a list of tensors so I'm not sure where the Boolean type is coming into play here. Does anyone understand why this error is occurring and whether the Boolean conversion is happening?
Many Thanks

Output:
What this error means
This warning occurs, when one tries to torch.jit.trace models which have data dependent control flow.
This simple example should make it clearer:
import torch

class Foo(torch.nn.Module):
    def forward(self, tensor):
        # It is data dependent
        # Trace will only work with one path
        if tensor.max() > 0.5:
            return tensor ** 2
        return tensor

model = Foo()
traced = torch.jit.script(model) # No warnings
traced = torch.jit.trace(model, torch.randn(10)) # Warning

In essence, BERT model has some control flow (like if, for loop) dependent on the data, hence you get the warning.
Warning itself
You can see BERT forward code here.
You are fine if:

arguments do not change (like None values passed to forward) and it will stay that way after script (e.g. during inference calls)
if there is control flow based on data gathered inside __init__ (like configs), because this will not change

For example:
elif input_ids is not None:
    input_shape = input_ids.size()
    batch_size, seq_length = input_shape

Will only run as one branch with torch.jit.trace, as it just traces operations on tensor and is unaware of control flow like this.
HuggingFace teams is probably aware of that and this warning is not an issue (though you might double check with your use case or try to go with torch.jit.script)
Going with torch.jit.script
This one would be hard as the whole model has to be torchscript compatible (torchscript has a subset of Python available and more than likely will not work out of the box with BERT).
Do it only when necessary (probably not).


Instruction: RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces)

Input: I am using Pytorch. I got this RuntimeError while evaluating a model. Any idea how to solve this?

Output:
SOLUTION: Just replace the view() function with reshape() function as suggested in the error and it works.
I guess this has to do with how the tensor is stored in memory.


Instruction: Pytorch linear regression loss increase

Input:
I tried to implement a simple demo that gets a polynomial regression, but the linear model's loss fails to decrease.
I am confused about where I went wrong.
If I trained the model one sample(batch size = 1) each time, it works fine. but when I feed the model with many samples a time, the loss increase and get inf.
import numpy as np

import torch
import math
from matplotlib import pyplot as plt

def rand_series(size):
    x = np.linspace(-100, 100, size)
    np.random.shuffle(x)
    base_y = 20 * np.sin(2 * math.pi / 200 * x)
    y = base_y + 10 * np.random.rand(size)
    return x, y

def rescale_vec(vector):
    vec_as_tensor = torch.tensor(vector, dtype=torch.float32)
    max_in_vec = torch.max(vec_as_tensor)
    min_in_vec = torch.min(vec_as_tensor)
    if max_in_vec - min_in_vec == 0:
        return torch.ones(vec_as_tensor.size(), dtype=torch.float32)
    else:
        return (vec_as_tensor - min_in_vec) / (max_in_vec - min_in_vec)

def rescale(vectors):
    if len(vectors.shape) == 1:
        return rescale_vec(vectors)
    nor_vecs = torch.empty(vectors.shape)
    for i in range(vectors.shape[0]):
        nor_vecs[i] = rescale_vec(vectors[i])
    return nor_vecs

class LinearRegression (torch.nn.Module):
    def __init__ (self, power=4):
        super().__init__()
        self.layer = torch.nn.Linear(power, 1)

    def forward(self, x):
        return self.layer(x)

def regression(x_, y_, learning_rate):
    x = torch.t(torch.tensor(x_, dtype=torch.float32))
    y = torch.tensor(y_, dtype=torch.float32)
    dim_size = x.size()[1]
    print(dim_size, x.size())

    model = LinearRegression(dim_size)
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
    loss_func = torch.nn.MSELoss(reduction='sum')
    batch_size = 400
    for round in range(50):
        sample_indices = torch.randint(0, len(x), (batch_size, ))
        x_samples = torch.index_select(x, 0, sample_indices)
        y_samples = torch.index_select(y, 0, sample_indices)
        optimizer.zero_grad()

        y_hat = model(x_samples.view(-1, dim_size))
        loss = loss_func(y_hat, y_samples)
        print(loss.item())
        loss.backward()

        optimizer.step()

    return model

x_one, y = rand_series(1000)
b = np.ones(len(x_one))
x = np.array([b, x_one, x_one ** 2, x_one ** 3, x_one ** 4, x_one ** 5])

model = regression(rescale(x), torch.tensor(y, dtype=torch.float32), 0.002)

nor_x = rescale(x)

y_hat = model(torch.t(torch.tensor(x, dtype=torch.float32)))
plt.scatter(x_one, y)
plt.scatter(x_one, y_hat.data, c='red')
plt.show()

the loss:
4.7375866968775066e+19
1.6979300048622735e+26
6.0214270068868396e+32
inf
inf
inf

Output: You need to use loss_func = torch.nn.MSELoss(reduction='mean') to solve the NaN problem. A batch of one or two seems to work because the loss was small enough. By adding more epochs, you should see that your loss tend exponentially to infinity.


Instruction: CNN Classifier only guesses one thing - PyTorch

Input:
I'm trying to make a model predict the race of a 75x75 image's ethnicity, but when ever I train the model, the accuracy always stays completely still at 53.2%. I didn't realize why until I actually ran it on some of photos. It turned out, that no matter what the photo was, it would always predict 'other'. I'm not entirely sure why though.
I copied the code over from the official PyTorch Quickstart tutorial, and in that dataset or the standard MNIST data, it worked fine. I changed the dataset to the UTKFace, and then it started only predicting one label, all the time.
Here's my code:
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
import torch.nn.functional as F

training_data = ImageFolder(
    root = "data_training/",
    transform = ToTensor(),
)

testing_data = ImageFolder(
    root = "data_testing/",
    transform = ToTensor()
)

training_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)
test_dataloader = DataLoader(testing_data, batch_size=64, shuffle=True)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fc1 = nn.Linear(1296, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 512)
        self.fc4 = nn.Linear(512, 84)
        self.fc5 = nn.Linear(84, 5)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))

        x = self.fc5(x)
        return x

model = NeuralNetwork().to("cpu")
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to("cpu"), y.to("cpu")
        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def tests(dataloader, model):
    size = len(dataloader.dataset)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to("cpu"), y.to("cpu")
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= size
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

epochs = 10
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(training_dataloader, model, loss_fn, optimizer)
    tests(test_dataloader, model)

torch.save(model.state_dict(), "model.pth")

The training logs:
Epoch 1
-------------------------------
loss: 1.628994  [    0/23705]
loss: 1.620698  [ 6400/23705]
loss: 1.615423  [12800/23705]
loss: 1.596390  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.024725

Epoch 2
-------------------------------
loss: 1.593613  [    0/23705]
loss: 1.581375  [ 6400/23705]
loss: 1.583656  [12800/23705]
loss: 1.591942  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.024165

Epoch 3
-------------------------------
loss: 1.541260  [    0/23705]
loss: 1.592345  [ 6400/23705]
loss: 1.540908  [12800/23705]
loss: 1.540741  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.023705

Epoch 4
-------------------------------
loss: 1.566888  [    0/23705]
loss: 1.524875  [ 6400/23705]
loss: 1.540764  [12800/23705]
loss: 1.510044  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.023323

Epoch 5
-------------------------------
loss: 1.530084  [    0/23705]
loss: 1.498773  [ 6400/23705]
loss: 1.537755  [12800/23705]
loss: 1.508989  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.022993
....

No matter how many epochs I set it to, or how many layers I add in to try to get it to overfit, it always just seems to guess the same thing over and over again, with no signs of improvement.
I separated the UTKFace dataset into folders based on the ethnicity category of the name. There are 23705 images in the training data and 10134 in the testing.
I'm not sure why this is happening. Is my dataset not large enough? Are there not enough layers?

Output:
The number of layers and the dataset size don't explain this behavior for this example. Your CNN is behaving as a constant function, so far I don't know why, but these might be some clues:

Since you have separated your data by label into folders, if you are training your model using only one of those folders you will obtain a constant function.
The last layer of your neural network has no activation function! This is, in method forward you are doing x = self.fc5(x) instead of x = F.<function>(self.fc5(x)).
Where do you indicate, when loading the training data, which is the label for each image? Are you sure that training_dataloader is loading the images with their correct label?


Instruction: How to create a graph neural network dataset? (pytorch geometric)

Input:
How can I convert my own dataset to be usable by pytorch geometric for a graph neural network?
All the tutorials use existing dataset already converted to be usable by pytorch. For example if I have my own pointcloud dataset how can i use it to train for classification with graph neural network? What about my own image dataset for classification?

Output:
How you need to transform your data depends on what format your model expects.
Graph neural networks typically expect (a subset of):

node features
edges
edge attributes
node targets

depending on the problem. You can create an object with tensors of these values (and extend the attributes as you need) in PyTorch Geometric wth a Data object like so:
data = Data(x=x, edge_index=edge_index, y=y)
data.train_idx = torch.tensor([...], dtype=torch.long)
data.test_mask = torch.tensor([...], dtype=torch.bool)


Instruction: Why putting value on GPU slow the calculation?

Input:
I'm a new one in GPU parallezation. I found putting values on GPU in advance slows the calculations and indexing. My code is as follows:
import torch

A = torch.rand(600, 600, device='cuda:0')
row0 = torch.tensor(100, device='cuda:0')
col0 = torch.tensor(100, device='cuda:0')
row1 = torch.tensor(356, device='cuda:0')
col1 = torch.tensor(356, device='cuda:0')
B = torch.rand(256, 256, device='cuda:0')
a = 10

%timeit B[:] = A[row0:row1, col0:col1]
# 395 µs ± 4.01 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit a*A + a**2
# 17 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

A = torch.rand(600, 600, device='cuda:0')
row0 = 100
col0 = 100
row1 = 356
col1 = 356
B = torch.rand(256, 256, device='cuda:0')
a1 = torch.as_tensor(a).cuda()

%timeit B[:] = A[row0:row1, col0:col1]
# 10.6 µs ± 141 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit a1*A + a1**2
# 30.2 µs ± 584 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Could anyone explain the mechanism behind it?

Output:
Your code is slow because there is nothing to parallelize, and you're just taking unnecessary GPU overheads.
GPU parallelism works by launching a large number of threads, and simultaneously computing chunks of some operation. Things like matrix multiplication and convolution are very GPU-friendly, because you can break them down into a lot of similar smaller operations.
However, there is also an overhead when doing things on the GPU.
Only when sufficient amount of threads have been launched to beat the CUDA overhead, we observe a speedup. Let's see an example:
import torch

device = torch.device('cuda:0')

A = torch.randn(5, 10, device=device)
B = torch.randn(10, 5, device=device)
A_ = torch.randn(5, 10)
B_ = torch.randn(10, 5)

%timeit A @ B
# 10.5 µs ± 745 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit A_ @ B_
# 5.21 µs ± 120 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

You may think this is going against common sense -- how can CPU matrix multiplication be faster than its GPU counterpart? It's simply because we haven't had a big enough operation to parallelize. Let's use retry the same thing, but on bigger inputs:
A = torch.randn(100, 200, device=device)
B = torch.randn(200, 100, device=device)
A_ = torch.randn(100, 200)
B_ = torch.randn(200, 100)

%timeit A @ B
# 10.4 µs ± 333 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit A_ @ B_
# 45.3 µs ± 647 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

We increased the input size by a factor of 20. GPU's still basically showing us the same time, the overhead, while CPU time has blown up. Because the input was bigger, GPU parallelism can show its magic.
In your case, you're not doing any parallelization at all. You're simply trying to slice your tensor with a GPU scalar, thus getting some sort of overhead but no benefits. Similar case in the other operation: there is nothing to parallelize.
%timeit a**2
# 200 ns ± 11.9 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
%timeit a1**2
# 16.8 µs ± 1.43 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

There is no way to break the operation a1 ** 2 into smaller repeatable chunks. Knowing when and when not to use the GPU is quite important. This can also be a useful starting point to get a glimpse of how CUDA works under the hood.


Instruction: How to install the module pytorch_lightning.metrics in Raspberry pi3

Input:
I am trying to execute a python file which has pytorch with lightning and torchvision modules. But after I downloaded and successfully installed whl file of pytorch in pi3 I am getting same error again and again.
The error is
 ModuleNotFoundError: No module named 'pytorch_lightning.metrics'

Help would be highly appreciated as I am stuck for more than 3 days.
I have installed the modules using pip.

Output:
Use instead:
from torchmetrics.functional import accuracy


Instruction: How to solve the famous `unhandled cuda error, NCCL version 2.7.8` error?

Input:
I've seen multiple issue about the:
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378083779/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.

but none seem to fix it for me:

https://github.com/pytorch/pytorch/issues/54550
https://github.com/pytorch/pytorch/issues/47885
https://github.com/pytorch/pytorch/issues/50921
https://github.com/pytorch/pytorch/issues/54823

I've tried to do torch.cuda.set_device(device) manually at the beginning of every script. That didn't seem to work for me. I've tried different GPUS. I've tried downgrading pytorch version and cuda version. Different combinations of 1.6.0, 1.7.1, 1.8.0 and cuda 10.2, 11.0, 11.1. I am unsure what else to do. What did people do to solve this issue?

very related perhaps?

Pytorch "NCCL error": unhandled system error, NCCL version 2.4.8"

More complete error message:
('jobid', 4852)
('slurm_jobid', -1)
('slurm_array_task_id', -1)
('condor_jobid', 4852)
('current_time', 'Mar25_16-27-35')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar25_16-27-35_jobid_4852/tb'))
('gpu_name', 'GeForce GTX TITAN X')
('PID', '30688')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 0
rank=0
mp.current_process()=<SpawnProcess name='SpawnProcess-1' parent=30688 started>
os.getpid()=30704
setting up rank=0 (with world_size=2)
MASTER_ADDR='127.0.0.1'
59264
backend='nccl'
--> done setting up rank=0
setup process done for rank=0
Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 279, in <module>
    main_distributed()
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 188, in main_distributed
    spawn_return = mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 212, in train
    tactic_predictor = move_to_ddp(rank, opts, tactic_predictor)
  File "/home/miranda9/ultimate-utils/ultimate-utils-project/uutils/torch/distributed.py", line 162, in move_to_ddp
    model = DistributedDataParallel(model, find_unused_parameters=True, device_ids=[opts.gpu])
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 446, in __init__
    self._sync_params_and_buffers(authoritative_rank=0)
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 457, in _sync_params_and_buffers
    self._distributed_broadcast_coalesced(
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1155, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1616554793803/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.

Bonus 1:
I still have errors:
ncclSystemError: System call (socket, malloc, munmap, etc) failed.
Traceback (most recent call last):
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1423, in <module>
    main()
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1365, in main
    train(args=args)
  File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 1385, in train
    args.opt = move_opt_to_cherry_opt_and_sync_params(args) if is_running_parallel(args.rank) else args.opt
  File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/distributed.py", line 456, in move_opt_to_cherry_opt_and_sync_params
    args.opt = cherry.optim.Distributed(args.model.parameters(), opt=args.opt, sync=syn)
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py", line 62, in __init__
    self.sync_parameters()
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/cherry/optim.py", line 78, in sync_parameters
    dist.broadcast(p.data, src=root)
  File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1090, in broadcast
    work = default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8

one of the answers suggested to have nvcca & pytorch.version.cuda to match but they do not:
(meta_learning_a100) [miranda9@hal-dgx ~]$ python -c "import torch;print(torch.version.cuda)"

11.1
(meta_learning_a100) [miranda9@hal-dgx ~]$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

How do I match them?

Output:
This is not a very satisfactory answer but this seems to be what ended up working for me. I simply used pytorch 1.7.1 and it's cuda version 10.2. As long as cuda 11.0 is loaded it seems to be working. To install that version do:
conda install -y pytorch==1.7.1 torchvision torchaudio cudatoolkit=10.2 -c pytorch -c conda-forge

if your are in an HPC do module avail to make sure the right cuda version is loaded. Perhaps you need to source bash and other things for the submission job to work. My setup looks as follows:
#!/bin/bash

echo JOB STARTED

# a submission job is usually empty and has the root of the submission so you probably need your HOME env var
export HOME=/home/miranda9
# to have modules work and the conda command work
source /etc/bashrc
source /etc/profile
source /etc/profile.d/modules.sh
source ~/.bashrc
source ~/.bash_profile

conda activate metalearningpy1.7.1c10.2
#conda activate metalearning1.7.1c11.1
#conda activate metalearning11.1

#module load cuda-toolkit/10.2
module load cuda-toolkit/11.1

#nvidia-smi
nvcc --version
#conda list
hostname
echo $PATH
which python

# - run script
python -u ~/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py

I also echo other useful things like the nvcc version to make sure load worked (note the top of nvidia-smi doesn't show the right cuda version).
Note I think this is probably just a bug since cuda 11.1 + pytorch 1.8.1 are new as of this writing. I did try
            torch.cuda.set_device(opts.gpu)  # https://github.com/pytorch/pytorch/issues/54550

but I can't say that it always works or why it doesn't. I do have it in my current code but I think I still get error with pytorch 1.8.x + cuda 11.x.
see my conda list in case it helps:
$ conda list

# packages in environment at /home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
absl-py                   0.12.0           py38h06a4308_0
aioconsole                0.3.1                    pypi_0    pypi
aiohttp                   3.7.4            py38h27cfd23_1
anatome                   0.0.1                    pypi_0    pypi
argcomplete               1.12.2                   pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
async-timeout             3.0.1            py38h06a4308_0
attrs                     20.3.0             pyhd3eb1b0_0
beautifulsoup4            4.9.3              pyha847dfd_0
blas                      1.0                         mkl
blinker                   1.4              py38h06a4308_0
boto                      2.49.0                   pypi_0    pypi
brotlipy                  0.7.0           py38h27cfd23_1003
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.17.1               h27cfd23_0
ca-certificates           2021.1.19            h06a4308_1
cachetools                4.2.1              pyhd3eb1b0_0
cairo                     1.14.12              h8948797_3
certifi                   2020.12.5        py38h06a4308_0
cffi                      1.14.0           py38h2e261b9_0
chardet                   3.0.4           py38h06a4308_1003
click                     7.1.2              pyhd3eb1b0_0
cloudpickle               1.6.0                    pypi_0    pypi
conda                     4.9.2            py38h06a4308_0
conda-build               3.21.4           py38h06a4308_0
conda-package-handling    1.7.2            py38h03888b9_0
coverage                  5.5              py38h27cfd23_2
crcmod                    1.7                      pypi_0    pypi
cryptography              3.4.7            py38hd23ed53_0
cudatoolkit               10.2.89              hfd86e86_1
cycler                    0.10.0                   py38_0
cython                    0.29.22          py38h2531618_0
dbus                      1.13.18              hb2f20db_0
decorator                 5.0.3              pyhd3eb1b0_0
dgl-cuda10.2              0.6.0post1               py38_0    dglteam
dill                      0.3.3              pyhd3eb1b0_0
expat                     2.3.0                h2531618_2
fasteners                 0.16                     pypi_0    pypi
filelock                  3.0.12             pyhd3eb1b0_1
flatbuffers               1.12                     pypi_0    pypi
fontconfig                2.13.1               h6c09931_0
freetype                  2.10.4               h7ca028e_0    conda-forge
fribidi                   1.0.10               h7b6447c_0
future                    0.18.2                   pypi_0    pypi
gast                      0.3.3                    pypi_0    pypi
gcs-oauth2-boto-plugin    2.7                      pypi_0    pypi
glib                      2.63.1               h5a9c865_0
glob2                     0.7                pyhd3eb1b0_0
google-apitools           0.5.31                   pypi_0    pypi
google-auth               1.28.0             pyhd3eb1b0_0
google-auth-oauthlib      0.4.3              pyhd3eb1b0_0
google-pasta              0.2.0                    pypi_0    pypi
google-reauth             0.1.1                    pypi_0    pypi
graphite2                 1.3.14               h23475e2_0
graphviz                  2.40.1               h21bd128_2
grpcio                    1.32.0                   pypi_0    pypi
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
gsutil                    4.60                     pypi_0    pypi
gym                       0.18.0                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
harfbuzz                  1.8.8                hffaf4a1_0
higher                    0.2.1                    pypi_0    pypi
httplib2                  0.19.0                   pypi_0    pypi
icu                       58.2                 he6710b0_3
idna                      2.10               pyhd3eb1b0_0
importlib-metadata        3.7.3            py38h06a4308_1
intel-openmp              2020.2                      254
jinja2                    2.11.3             pyhd3eb1b0_0
joblib                    1.0.1              pyhd3eb1b0_0
jpeg                      9b                   h024ee3a_2
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.3.1            py38h2531618_0
lark-parser               0.6.5                    pypi_0    pypi
lcms2                     2.11                 h396b838_0
ld_impl_linux-64          2.33.1               h53a641e_7
learn2learn               0.1.5                    pypi_0    pypi
libarchive                3.4.2                h62408e4_0
libffi                    3.2.1             hf484d3e_1007
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
liblief                   0.10.1               he6710b0_0
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.14.0               h8c45485_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_1
libuuid                   1.0.3                h1bed415_2
libuv                     1.40.0               h7b6447c_0
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.10               hb55368b_3
lmdb                      0.94                     pypi_0    pypi
lz4-c                     1.9.2                he1b5a44_3    conda-forge
markdown                  3.3.4            py38h06a4308_0
markupsafe                1.1.1            py38h7b6447c_0
matplotlib                3.3.4            py38h06a4308_0
matplotlib-base           3.3.4            py38h62a2d02_0
memory-profiler           0.58.0                   pypi_0    pypi
mkl                       2020.2                      256
mkl-service               2.3.0            py38h1e0a361_2    conda-forge
mkl_fft                   1.3.0            py38h54f3939_0
mkl_random                1.2.0            py38hc5bc63f_1    conda-forge
mock                      2.0.0                    pypi_0    pypi
monotonic                 1.5                      pypi_0    pypi
multidict                 5.1.0            py38h27cfd23_2
ncurses                   6.2                  he6710b0_1
networkx                  2.5                        py_0
ninja                     1.10.2           py38hff7bd54_0
numpy                     1.19.2           py38h54aff64_0
numpy-base                1.19.2           py38hfa32c7d_0
oauth2client              4.1.3                    pypi_0    pypi
oauthlib                  3.1.0                      py_0
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openssl                   1.1.1k               h27cfd23_0
opt-einsum                3.3.0                    pypi_0    pypi
ordered-set               4.0.2                    pypi_0    pypi
pandas                    1.2.3            py38ha9443f7_0
pango                     1.42.4               h049681c_0
patchelf                  0.12                 h2531618_1
pbr                       5.5.1                    pypi_0    pypi
pcre                      8.44                 he6710b0_0
pexpect                   4.6.0                    pypi_0    pypi
pillow                    7.2.0                    pypi_0    pypi
pip                       21.0.1           py38h06a4308_0
pixman                    0.40.0               h7b6447c_0
pkginfo                   1.7.0            py38h06a4308_0
progressbar2              3.39.3                   pypi_0    pypi
protobuf                  3.14.0           py38h2531618_1
psutil                    5.8.0            py38h27cfd23_1
ptyprocess                0.7.0                    pypi_0    pypi
py-lief                   0.10.1           py38h403a769_0
pyasn1                    0.4.8                      py_0
pyasn1-modules            0.2.8                      py_0
pycapnp                   1.0.0                    pypi_0    pypi
pycosat                   0.6.3            py38h7b6447c_1
pycparser                 2.20                       py_2
pyglet                    1.5.0                    pypi_0    pypi
pyjwt                     1.7.1                    py38_0
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7              pyhd3eb1b0_0
pyqt                      5.9.2            py38h05f1152_4
pysocks                   1.7.1            py38h06a4308_0
python                    3.8.2                hcf32534_0
python-dateutil           2.8.1              pyhd3eb1b0_0
python-libarchive-c       2.9                pyhd3eb1b0_0
python-utils              2.5.6                    pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.7.1           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
pytz                      2021.1             pyhd3eb1b0_0
pyu2f                     0.1.5                    pypi_0    pypi
pyyaml                    5.4.1            py38h27cfd23_1
qt                        5.9.7                h5867ecd_1
readline                  8.1                  h27cfd23_0
requests                  2.25.1             pyhd3eb1b0_0
requests-oauthlib         1.3.0                      py_0
retry-decorator           1.1.1                    pypi_0    pypi
ripgrep                   12.1.1                        0
rsa                       4.7.2              pyhd3eb1b0_1
ruamel_yaml               0.15.100         py38h27cfd23_0
scikit-learn              0.24.1           py38ha9443f7_0
scipy                     1.6.2            py38h91f5cce_0
setuptools                52.0.0           py38h06a4308_0
sexpdata                  0.0.3                    pypi_0    pypi
sip                       4.19.13          py38he6710b0_0
six                       1.15.0             pyh9f0ad1d_0    conda-forge
soupsieve                 2.2.1              pyhd3eb1b0_0
sqlite                    3.35.2               hdfb4753_0
tensorboard               2.4.0              pyhc547734_0
tensorboard-plugin-wit    1.6.0                      py_0
tensorflow                2.4.1                    pypi_0    pypi
tensorflow-estimator      2.4.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.1.0              pyh5ca1d4c_0
tk                        8.6.10               hbc83047_0
torchaudio                0.7.2                      py38    pytorch
torchmeta                 1.7.0                    pypi_0    pypi
torchtext                 0.8.1                      py38    pytorch
torchvision               0.8.2                py38_cu102    pytorch
tornado                   6.1              py38h27cfd23_0
tqdm                      4.56.0                   pypi_0    pypi
typing-extensions         3.7.4.3                       0
typing_extensions         3.7.4.3                    py_0    conda-forge
urllib3                   1.26.4             pyhd3eb1b0_0
werkzeug                  1.0.1              pyhd3eb1b0_0
wheel                     0.36.2             pyhd3eb1b0_0
wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h7b6447c_0
yarl                      1.6.3            py38h27cfd23_0
zipp                      3.4.1              pyhd3eb1b0_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

For a100s this seemed to work at some point:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Slow execution time for CUDA initialization in Azure Batch VM

Input:
I have an issue of slow initialization time for running some CUDA program in one of the VM for Azure Batch.
After some troubleshooting, I made a simple test running this call as shown in the below code.
#include <stdio.h>
#include <cuda.h>
#include <cuda_runtime_api.h>
#include <time.h>

clock_t start, end;
double cpu_time_used;

int main()
{
    CUresult result;
    printf("CUDA version %d \n", CUDA_VERSION);
    start = clock();
    result = cuInit(0);
    if (result != CUDA_SUCCESS) {
        printf("cuInit failed with error code %d: %s\n", result, cudaGetErrorString(result));
        return 1;
    }
    end = clock();
    cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;
    printf("cuInit took %f seconds to execute \n", cpu_time_used);
    return 0;
}

It takes about 1.9 seconds in average.
Some specs:

NVidia driver: 460.32.03
CUDAToolkit: 10.2
Azure Batch: nc6, Tesla K80

As a comparisons, the same code was running on my desktop (Windows) as well another custom Azure VM (nc6, not Azure Batch) giving similar result of 0.03 seconds. (cudatoolkit 10.2)
-- update 1 --
Calling CUDA initialization via Torch also shows a significant lag (for the first call) as shown from this test:
run: 0, import torch: 0.430, cuda_available: 4.921
run: 1, import torch: 0.000, cuda_available: 0.000
run: 2, import torch: 0.000, cuda_available: 0.000
run: 3, import torch: 0.000, cuda_available: 0.000
max time for import torch: 0.43 s,  max time for cuda_available: 4.92 s
torch.version 1.7.1+cu101 torch.version.cuda: 10.1

The import torch code is:
import torch

and the cuda_available code is:
torch.cuda.is_available()

My question is the time taken by the Azure Batch for CUDA initialization normal behavior ?

Output:
Azure Batch in VirtualMachineConfiguration mode allocates Virtual Machine Scale Sets internally. There is no difference in the underlying hardware Azure Batch allocates from. For further investigation:

How big is your sample set? Is your start time reproducible between different VMs within different Batch pools? Perhaps you got a bad or degraded VM or GPU.
What does nvidia-smi tell you on the compute node in question? Are there any potential hardware faults?
Are you using an Azure Batch task or are you remoting into the VM and executing your task interactively?


Instruction: Why do we need to pass the gradient parameter to the backward function in PyTorch?

Input:
According to the docs, when we call the backward function to the tensor if the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying gradient.
import torch
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([1.,1.]))

print(a.grad)

Output: tensor([20., 20.])
Now scaling the external gradient:
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([2.,2.])) #modified

print(a.grad)

Output: tensor([40., 40.])
So, passing the gradient argument to backward seems to scale the gradients.Also, by default F.backward() is F.backward(gradient=torch.Tensor([1.]))
Apart from scaling the grad value how does the gradient parameter passed to the backward function helps to compute the derivatives when we have a non-scalar tensor?Why can't PyTorch calculate the derivative implicitly without asking explicit gradient parameter as it did for the scalar tensor?

Output:
It's because PyTorch is calculating the jacobian product. In case of scalar value, .backward() w/o parameters is equivalent to .backward(torch.tensor(1.0)).
That's why you need to provide the tensor with which you want to calculate the product. Read more about automatic differentiation.


Instruction: Logging training metrics to a csv file

Input:
I want to log all training metrics to a csv file while it is training on YOLOV5 which is written with pytorch but the problem is that I don't want to use tensorboard.
To achieve this goal I tried some techniques like below:
-First log it into tensorboard and then try to convert it to a csv file (failed)
-Extract log files from Weights & Biases (failed)
-Write to a csv file directly during training (failed)
Here I tried opening a csv file and transforming tensor data into string but couldn't succeed.
So I really wonder how could I achieve this goal because YOLOV5 is not using pytorch utilites like model.fit which we can use callbacks with.
Thanks in advance.

Output:
You might try using the Weights and Biases YOLOv5 integration.
Here is the link: https://docs.wandb.ai/guides/integrations/yolov5
The link has more details, but here are some quotes that convey the basic idea:

Simply by installing wandb, you'll activate the built-in W&B logging features: system metrics, model metrics, and media logged to interactive Dashboards.

pip install wandb
git clone https://github.com/ultralytics/yolov5.git
python yolov5/train.py  # train a small network on a small dataset

Just follow the links printed to the standard out by wandb.pip install wandb git clone

You can also do model versioning & dataset visualization, which is explained more in the w&b yolov5 integration docs page (linked above). You can also watch this YouTube video for a guide: https://youtu.be/yyecuhBmLxE


Instruction: PyTorch CPU memory leak but only when running on a specific machine

Input:
I'm running a model and I've noticed that the RAM usage slowly increases during the training of the model. It's around 200mb-400mb per epoch, but over time it fills up all the RAM on my machine which eventually leads the OS to kill the job. However, the strange thing about this is it's only when running on a specific machine. I've run the same code on a different machine and there's no memory leak whatsoever.
The difference between the two machines is one is running PyTorch 1.7.1 with cuda 10.2 (the machine without the memory leak) and the other machine (the one with the memory leak) is running PyTorch 1.8.1· Both are run with conda and only on the CPU. I have tried using older versions of PyTorch on the machine with the memory leak, but the memory leak still exists so I doubt it's due to a PyTorch version.
I've been using psutil to monitor the RAM usage on the CPU and I've been using the tracemalloc package to print out snapshots during the training loop to see how the memory usage changes from one epoch to the next. Yet when printing out these differences there isn't any within my code that matches anywhere near the 200mb-400mb RAM increase...
An example of this would something like...
Epoch     25/  1000 Loss 1428.8508 RAM:  9.910GB
~/Model.py:46: size=1136 B (-96 B), count=2 (+0), average=568 B
~/utils.py:14: size=1040 B (+24 B), count=8 (+1), average=130 B
~/calc_loss.py:79: size=2128 B (+0 B), count=24 (+0), average=89 B
~/calc_loss.py:78: size=2056 B (+0 B), count=23 (+0), average=89 B
~/utils.py:6: size=1920 B (+0 B), count=21 (+0), average=91 B
Epoch     26/  1000 Loss 1426.0033 RAM: 10.254GB
~/Model.py:46: size=1232 B (+96 B), count=2 (+0), average=616 B
~/utils.py:14: size=1016 B (-24 B), count=7 (-1), average=145 B
~/calc_loss.py:79: size=2128 B (+0 B), count=24 (+0), average=89 B
~/calc_loss.py:78: size=2056 B (+0 B), count=23 (+0), average=89 B
~/utils.py:6: size=1920 B (+0 B), count=21 (+0), average=91 B
~/Layers.py:71: size=992 B (+0 B), count=11 (+0), average=90 B
Epoch     27/  1000 Loss 1436.8241 RAM: 10.606GB
~/utils.py:14: size=1040 B (+24 B), count=8 (+1), average=130 B
~/calc_loss.py:79: size=2128 B (+0 B), count=24 (+0), average=89 B
~/calc_loss.py:78: size=2056 B (+0 B), count=23 (+0), average=89 B
~/utils.py:6: size=1920 B (+0 B), count=21 (+0), average=91 B
~/Model.py:46: size=1232 B (+0 B), count=2 (+0), average=616 B
~/Layers.py:71: size=992 B (+0 B), count=11 (+0), average=90 B
Epoch     28/  1000 Loss 1428.6560 RAM: 10.968GB
~/calc_loss.py:79: size=2128 B (+0 B), count=24 (+0), average=89 B
~/calc_loss.py:78: size=2056 B (+0 B), count=23 (+0), average=89 B
~/utils.py:6: size=1920 B (+0 B), count=21 (+0), average=91 B
~/Model.py:46: size=1232 B (+0 B), count=2 (+0), average=616 B
~/utils.py:14: size=1040 B (+0 B), count=8 (+0), average=130 B
Epoch     29/  1000 Loss 1435.2988 RAM: 11.321GB
~/calc_loss.py:79: size=2128 B (+0 B), count=24 (+0), average=89 B
~/calc_loss.py:78: size=2056 B (+0 B), count=23 (+0), average=89 B
~/utils.py:6: size=1920 B (+0 B), count=21 (+0), average=91 B
~/Model.py:46: size=1232 B (+0 B), count=2 (+0), average=616 B
~/utils.py:14: size=1040 B (+0 B), count=8 (+0), average=130 B
~/Layers.py:71: size=992 B (+0 B), count=11 (+0), average=90 B

The information printed out in between the lines showing the current Epoch are created via the use of this function
def compare_snaps(snap1, snap2, limit=50):
  top_stats=snap1.compare_to(snap2, "lineno")

  for stat in top_stats[:limit]:
    line=str(stat)
    if("~/" in line): #filter only lines from my own code
      print(line)

this function takes two snapshots from  tracemalloc.take_snapshot() from the current epoch and the previous epoch and compares how the memory usage changes. It takes the top 50 memory intensive operations, and filters only the ones that I've written (i.e. it excludes any changes from within anaconda3) and prints these changes to the screen. As can be seen, the changes in memory are negligible. In fact, when comparing the snap shotoutput from both machines, they're near identical.
It seems really weird that PyTorch code would have a memory leak on one machine and not on another... Could this perhaps be a conda environemnt issue? I have tried running this from a pip install and the leak still persists. I understand that this a bit vague (as there's no source code) but are there any ways to check where the leaks coming from? Things like Valgrind comes to mind for compile code but not much for interpretative code. This memory leak issue seems to be outside of my skill-set, so any help would be appreciated.
Thank you! :)

Output:
Try incorporating this in your process:
import gc

# add this after computing one complete operation
gc.collect()


Instruction: how to save a Pytorch model?

Input: I am new to Deep learning and I want to know, how can I save the final model in Pytorch? I tried some things that were mentioned but I got confused with, how to save the model and how to load it back?

Output:
to save:
# save the weights of the model to a .pt file
torch.save(model.state_dict(), "your_model_path.pt")

to load:
# load your model architecture/module
model = YourModel()
# fill your architecture with the trained weights
model.load_state_dict(torch.load("your_model_path.pt"))


Instruction: Memory leak with en_core_web_trf model, Spacy

Input:
there is a Memory leak when using pipe of en_core_web_trf model, I run the model using GPU with 16GB RAM, here is a sample of the code.
!python -m spacy download en_core_web_trf

import en_core_web_trf
nlp = en_core_web_trf.load()

#it's just an array of 100K sentences.
data = dataload()

for index, review in enumerate( nlp.pipe(data, batch_size=100) ):
    #doing some processing here
    if index % 1000: print(index)

this code cracks when reaching 31K, and raises OOM error.
CUDA out of memory. Tried to allocate 46.00 MiB (GPU 0; 11.17 GiB total capacity; 10.44 GiB already allocated; 832.00 KiB free; 10.72 GiB reserved in total by PyTorch)

I just use the pipeline to predict, not train any data or other stuff and tried with different batch sizes, but nothing happened,
still, crash.
Your Environment

spaCy version: 3.0.5
Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.10
Pipelines: en_core_web_trf (3.0.0)

Output:
Lucky you with GPU - I am still trying to get thru the (torch GPU) DLL Hell on Windows :-). But it looks like Spacy 3 uses more GPU memory than Spacy 2 did - my 6GB GPU may have become useless.
That said, have you tried running your case without the GPU (and watching memory usage)?
Spacy 2 'leak' on large datasets is (mainly) due to growing vocabulary - each data row may add couple more words, and the suggested 'solution' is reloading the model and/or just the vocabulary every nnn rows. The GPU usage may have the same issue...


Instruction: Huggingface error during training: AttributeError: 'str' object has no attribute 'size'

Input:
While trying to finetune a Huggingface GPT2LMHeadModel model for casual language modeling (given a sequence of words, predict the next word) using Pytorch Lightning, I am getting an error during training:

AttributeError: 'str' object has no attribute 'size'

What went wrong with our training code? Is this due to the incorrect use of DataCollatorForLanguageModeling in a Pytorch DataLoader?
Reproducible Example:
import os
from pathlib import Path
import torch
import pytorch_lightning as pl
from transformers import (
    GPT2Config,
    GPT2LMHeadModel,
    GPT2Tokenizer,
    DataCollatorForLanguageModeling,
)
from transformers.optimization import AdamW
from tokenizers import ByteLevelBPETokenizer
from torch.utils.data import (
    DataLoader,
    Dataset,
)

TOKENIZER_DIRPATH = os.path.join("..", "data")

def tokenize_data():
    tokenizer = ByteLevelBPETokenizer()
    tokenizer.train(
        files=os.path.join(TOKENIZER_DIRPATH, "words.txt"),
        vocab_size=50000,
        min_frequency=2,
        special_tokens=["<s>", "</s>", "<unk>", "<mask>", "<pad>",],
    )
    tokenizer.save_model("../data")

class MyDataset(Dataset):
    def __init__(self):
        tokenizer = GPT2Tokenizer(
            os.path.join(TOKENIZER_DIRPATH, "vocab.json"),
            os.path.join(TOKENIZER_DIRPATH, "merges.txt"),
        )

        src_file = Path(os.path.join(TOKENIZER_DIRPATH, "words.txt"))
        lines = src_file.read_text(encoding="utf-8").splitlines()
        self.examples = [tokenizer.encode(line) for line in lines]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        return torch.tensor(self.examples[i])

class MyDataModule(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.tokenizer = GPT2Tokenizer(
            os.path.join(TOKENIZER_DIRPATH, "vocab.json"),
            os.path.join(TOKENIZER_DIRPATH, "merges.txt"),
        )

    def setup(self, stage):
        self.train_dataset = MyDataset()

    def train_dataloader(self):
        data_collator = DataCollatorForLanguageModeling(
            tokenizer=self.tokenizer, mlm=False
        )
        train_dataloader = DataLoader(self.train_dataset, collate_fn=data_collator)
        return train_dataloader

class MyModel(pl.LightningModule):
    def __init__(self, learning_rate, adam_beta1, adam_beta2, adam_epsilon):
        super().__init__()
        self.save_hyperparameters()
        config = GPT2Config()
        self.model = GPT2LMHeadModel(config)

    def forward(self, x):
        return self.model(x).logits

    def training_step(self, batch, batch_idx):
        input_ids, labels = batch
        loss = self.model(input_ids, labels=labels).loss
        self.log("train_loss", loss, on_epoch=True)
        return loss

    def configure_optimizers(self):
        optimizer = AdamW(
            self.parameters(),
            self.hparams.learning_rate,
            betas=(self.hparams.adam_beta1, self.hparams.adam_beta2),
            eps=self.hparams.adam_epsilon,
        )
        return optimizer

tokenize_data()
dm = MyDataModule()
model = MyModel(
    learning_rate=5e-5, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-8,
)

trainer = pl.Trainer()
trainer.fit(model, dm)

Error Traceback:
Epoch 0:   0%|                                                                                | 0/9 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "test_gpt.py", line 102, in <module>
    trainer.fit(model, dm)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 493, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 655, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 426, in optimizer_step
    model_ref.optimizer_step(
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1387, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 277, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 282, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 163, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/transformers/optimization.py", line 318, in step
    loss = closure()
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 649, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 743, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 293, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 156, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 125, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "test_gpt.py", line 81, in training_step
    loss = self.model(input_ids, labels=labels).loss
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 904, in forward
    transformer_outputs = self.transformer(
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/envs/test_huggingface/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 633, in forward
    input_shape = input_ids.size()
AttributeError: 'str' object has no attribute 'size'

Conda Packages:
pytorch                   1.7.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
pytorch-lightning         1.2.5              pyhd8ed1ab_0    conda-forge
tokenizers                0.10.1                   pypi_0    pypi
transformers              4.4.2                    pypi_0    pypi

Output:
Here transformer recent version may occur this
pip install transformers==2.11.0

In my case it's work!! then restart your kernel


Instruction: When using torch.backward() for a GANs generator, why doesn't discriminator losses change in Pytorch?

Input:
My understanding of GANs is:

When training your generator, you need to back-propagate through the discriminator first so you can follow the chain rule. As a result, we can't use a .detach()  when working on our generators loss calculation.

When updating discriminator, since your generator weight update doesn't affect discriminator weight updates, we can .detach() the generator output from your computation, my understanding tells me that since this generator is no longer a part of computation graph, we don't update it during back prop any longer.

So when we update your discriminator losses with:
disc_loss.backward(retain_graph=True) at each mini-batch, we don't have to worry about your generator being part of the pipeline thanks to the .detach() function call.
But what about when we work on our generator? What is keeping our model from constantly changing our discriminator weights in accordance to generator? That isn't what we want after all right? The discriminator shouldn't learn when it is told that fake samples are real.
Why does such a model work in the first place

Output: backward doesn't update the weights, it updates the gradients of the weights. Updating weights is the responsibility of the optimizer(s). There are different ways to implement GANs, but often you would have two optimizers, one that is responsible for updating the weights (and resetting the gradients) of the generator and one that is responsible for updating the weights (and resetting the gradients) of the discriminator. Upon initialization, each optimizer is provided only the weights of the model it will update. Therefore, when you call an optimizer's step method it only updates those weights. Using separate optimizers is what prevents the discriminator weights from being updated while minimizing the generator's loss function.


Instruction: I can't figure out why the size of the tensors doesn't match in Pytorch

Input:
Some context:
I have been studying AI and ML for the last couple of month now and finally I am studying neural nets. Great! The problem is that when I follow a tutorial everything seems to be OK, but when I try to implement a NN by my self I always face issues related to the size of the tensors.
I have seem the answer to other questions (like this one) but they face the exact problem of the post. I am not looking for a code to just copy and paste. I want to understand why I am facing this problem, how to handle it and avoid it.
The error message:

/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([16, 2])) that is different to the input size (torch.Size([9, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
Traceback (most recent call last):
File "nn_conv.py", line 195, in
loss = loss_function(outputs, targets)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 528, in forward
return F.mse_loss(input, target, reduction=self.reduction)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2928, in mse_loss
expanded_input, expanded_target = torch.broadcast_tensors(input, target)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/functional.py", line 74, in broadcast_tensors
return _VF.broadcast_tensors(tensors)  # type: ignore

RuntimeError: The size of tensor a (9) must match the size of tensor b (16) at non-singleton dimension 0

This is my code:
import os

import cv2
import numpy as np
from tqdm import tqdm
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class DogsVSCats():

    IMG_SIZE = 50
    CATS = 'PetImages/Cat'
    DOGS = 'PetImages/Dog'
    LABELS = {CATS: 0, DOGS: 1}

    training_data = []
    cats_count = 0
    dogs_count = 0

    def make_training_data(self):
        for label in self.LABELS.keys():

            for f in tqdm(os.listdir(label)):
                try:
                    path = os.path.join(label, f)
                    # convert image to grayscale
                    img = cv2.imread(path)
                    if img is not None:
                        height, width = img.shape[:2]
                        if width > height:
                            height = round((height * self.IMG_SIZE) / width)
                            width = self.IMG_SIZE
                            right = 0
                            bottom = self.IMG_SIZE - height
                        else:
                            width = round((width * self.IMG_SIZE) / height)
                            height = self.IMG_SIZE
                            right = self.IMG_SIZE - width
                            bottom = 0

                        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                        img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
                        img = cv2.resize(img, (width, height))
                        img = cv2.copyMakeBorder(img,
                                                 top=0,
                                                 bottom=bottom,
                                                 left=0,
                                                 right=right,
                                                 borderType=cv2.BORDER_CONSTANT)

                        # Add a One-hot-vector of label of the image to self.training_data
                        self.training_data.append([np.array(img), np.eye(len(self.LABELS))[self.LABELS[label]]])

                        if label == self.CATS:
                            self.cats_count += 1
                        elif label == self.DOGS:
                            self.dogs_count += 1
                except cv2.error as e:
                    pass

        np.random.shuffle(self.training_data)
        np.save("PetImages/training_data.npy", self.training_data)

        print("Cats:", self.cats_count)
        print("Dogs:", self.dogs_count)

        training_data = np.load('PetImages/training_data.npy', allow_pickle=True)
        plt.imsave('PetImages/trained_example.png', training_data[1][0])

class RunningMetrics():

    def __init__(self):
        self._sum = 0
        self._count = 0

    def __call__(self):
        return self._sum/float(self._count)

    def update(self, val, size):
        self._sum += val
        self._count += size

class Net(nn.Module):

    def __init__(self, num_channels, conv_kernel_size=3, stride=1, padding=1, max_pool_kernel_size=2):
        super(Net, self).__init__()

        self._num_channels = num_channels
        self._max_pool_kernel_size = max_pool_kernel_size

        self.conv1 = nn.Conv2d(1, self._num_channels, conv_kernel_size, stride, padding)
        self.conv2 = nn.Conv2d(self._num_channels, self._num_channels*2, conv_kernel_size, stride, padding)
        self.conv3 = nn.Conv2d(self._num_channels*2, self._num_channels*4, conv_kernel_size, stride, padding)

        # Calc input of first
        self.fc1 = nn.Linear(self._num_channels*4*8*8, self._num_channels*8)
        self.fc2 = nn.Linear(self._num_channels*8, 2)

    def forward(self, x):

        # Conv
        x = self.conv1(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))
        x = self.conv2(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))
        x = self.conv3(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))

        # Flatten
        x = x.view(-1, self._num_channels*4*8*8)

        # Fully Connected
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        # return F.log_softmax(x, dim=1)
        return F.softmax(x, dim=1)

    def save_model(path):
        torch.save(save, path)

    def load_model(path):
        self = torch.load(PATH)
        self.eval()

if __name__ == '__main__':

    print('Loading dataset')
    if not os.path.exists("PetImages/training_data.npy"):
        dogsvcats = DogsVSCats()
        dogsvcats.make_training_data()

    training_data = np.load('PetImages/training_data.npy', allow_pickle=True)

    print('Loading Net')
    net = Net(num_channels=32)
    # net = net.to(device)

    # optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9  )
    optimizer = optim.Adam(net.parameters(), lr=0.001)

    # loss_function = nn.NLLLoss()
    loss_function = nn.MSELoss()

    print('Converting X tensor')
    X = torch.Tensor([i[0] for i in training_data]).view(-1, 50, 50)
    X = X/255.0

    print('Converting Y tensor')
    y = torch.Tensor([i[1] for i in training_data])

    # Validation data
    VAL_PERCENT = 0.1
    val_size = int(len(X)*VAL_PERCENT)

    X_train = X[:-val_size]
    y_train = y[:-val_size]

    X_test = X[-val_size:]
    y_test = y[-val_size:]

    print('Training Set:', len(X_train))
    print('Testing Set:', len(X_test))

    BATCH_SIZE = 16
    EPOCHS = 2
    IMG_SIZE=50
    for epoch in range(EPOCHS):
        print(f'Epoch {epoch+1}/{EPOCHS}')

        running_loss = RunningMetrics()
        running_acc = RunningMetrics()

        for i in tqdm(range(0, len(X_train), BATCH_SIZE)):
            inputs = X_train[i:i+BATCH_SIZE].view(-1,1, IMG_SIZE, IMG_SIZE)
            targets = y_train[i:i+BATCH_SIZE]
            # inputs, targets = inputs.to(device), targets.to(device)

            optimizer.zero_grad()
            outputs = net(inputs)
            _, preds = torch.max(outputs, 1)

            loss = loss_function(outputs, targets)
            loss.backward()

            optimizer.step()

            running_loss.update(loss.item()*BATCH_SIZE,
                                BATCH_SIZE)

            running_acc.update(toch.sum(preds == targets).float(),
                                BATCH_SIZE)

        print(f'Loss: {running_loss:.4f}, Acc: {running_acc:.4f}')
        print('-'*10)

Dataset:
I am using the Microsoft's dataset of cats and dogs images
EDIT:
The error previous message has been solved following Anonymous' advice but now I am getting another error:

Traceback (most recent call last):
File "nn_conv.py", line 203, in
running_acc.update(torch.sum(preds == targets).float(),

RuntimeError: The size of tensor a (16) must match the size of tensor b (2) at non-singleton dimension 1

Output:
Input : 16 x 1 x 50 x 50
After conv1/maxpool1 : 16 x 32 x 25 x 25
After conv2/maxpool2 : 16 x 64 x 12 x 12 (no padding so taking floor)
After conv3/maxpool3 : 16 x 128 x 6 x 6 (=73 728 neurons here is your error)
Flattening : you specified a view like -1 x 32 * 4 * 8 * 8 = 9 x 8192
The correct flattening is -1 x 32 * 4 * 6 * 6
Few tips :

as you begin pytorch, you should go see how to use a dataloader/dataset
the binary cross entropy is more commonly used for classification (though MSE is still possible)


Instruction: The input tensor should have dimensions 1 x height x width x 3. Got 1 x 3 x 224 x 224

Input:
I want to convert the Pytorch-trained model to the tensorflow model and use the model on mobile devices. For this, I follow these steps; First I convert the pytorch trained model to onnx format. Then I convert the onnx format to the tensorflow model.
Firstly pytorch trained model to onnx;
import torch
import torch.onnx
from detectron2.modeling import build_model
from detectron2.modeling import build_backbone
from torch.autograd import Variable

model= build_backbone(cfg)
model.eval()

dummy_input = torch.randn(1,3,224, 224,requires_grad=True)

torch.onnx.export(model,dummy_input,"drive/Detectron2/model_final.onnx")

Then onnx to tflite model;
import onnx
import warnings
from onnx_tf.backend import prepare

model = onnx.load("drive/Detectron2/model_final.onnx")
tf_rep = prepare(model)
tf_rep.export_graph("drive/Detectron2/tf_rep.pb")

import tensorflow as tf

## TFLite Conversion
# Before conversion, fix the model input size
model = tf.saved_model.load("drive/Detectron2/tf_rep.pb")
model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs[0].set_shape([1, 3,224, 224])
tf.saved_model.save(model, "saved_model_updated", signatures=model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY])
# Convert
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir='saved_model_updated', signature_keys=['serving_default'])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()

# Save the model.
with open('drive/Detectron2/model.tflite', 'wb') as f:
  f.write(tflite_model)

## TFLite Interpreter to check input shape
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test the model on random input data.
input_shape = input_details[0]['shape']
print(input_shape)

But when I use the model on mobile devices, I get the following error;
java.lang.AssertionError: Error occurred when initializing ImageSegmenter: The input tensor should have dimensions 1 x height x width x 3. Got 1 x 3 x 224 x 224.

Where am I doing wrong?

Output:
Maybe you could try einops for tensor transformations. It's elegant and powerful.
In your case, the code should be

import einops
input_tensor = einops.rearrange(input_tensor,'b c w h -> b w h c')


Instruction: TypeError: linear(): argument 'input' (position 1) must be Tensor, not str

Input:
so ive been trying to work on some example of bert that i found on github as its the first time im trying to use bert and see how it works. The respiratory im working with is the following: https://github.com/prateekjoshi565/Fine-Tuning-BERT/blob/master/Fine_Tuning_BERT_for_Spam_Classification.ipynb
im using a different dataset however im getting the issue TypeError: linear(): argument 'input' (position 1) must be Tensor, not str" and honestly i dont know what im doing wrong. is there anyone that could help me?
the code ive been using is the following:
# convert class weights to tensor
weights= torch.tensor(class_wts,dtype=torch.float)
weights = weights.to(device)

# loss function
cross_entropy  = nn.NLLLoss(weight=weights)

# number of training epochs
epochs = 10

def train():

  model.train()

  total_loss, total_accuracy = 0, 0

  # empty list to save model predictions
  total_preds=[]

  # iterate over batches
  for step,batch in enumerate(train_dataloader):

    # progress update after every 50 batches.
    if step % 50 == 0 and not step == 0:
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

    # push the batch to gpu
    batch = [r.to(device) for r in batch]

    sent_id, mask, labels = batch

    # clear previously calculated gradients
    model.zero_grad()

    # get model predictions for the current batch
    preds = model(sent_id, mask)

    # compute the loss between actual and predicted values
    loss = cross_entropy(preds, labels)

    # add on to the total loss
    total_loss = total_loss + loss.item()

    # backward pass to calculate the gradients
    loss.backward()

    # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # update parameters
    optimizer.step()

    # model predictions are stored on GPU. So, push it to CPU
    preds=preds.detach().cpu().numpy()

    # append the model predictions
    total_preds.append(preds)

  # compute the training loss of the epoch
  avg_loss = total_loss / len(train_dataloader)

  # predictions are in the form of (no. of batches, size of batch, no. of classes).
  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  #returns the loss and predictions
  return avg_loss, total_preds

def evaluate():

  print("\nEvaluating...")

  # deactivate dropout layers
  model.eval()

  total_loss, total_accuracy = 0, 0

  # empty list to save the model predictions
  total_preds = []

  # iterate over batches
  for step,batch in enumerate(val_dataloader):

    # Progress update every 50 batches.
    if step % 50 == 0 and not step == 0:

      # Calculate elapsed time in minutes.
      elapsed = format_time(time.time() - t0)

      # Report progress.
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(val_dataloader)))

    # push the batch to gpu
    batch = [t.to(device) for t in batch]

    sent_id, mask, labels = batch

    # deactivate autograd
    with torch.no_grad():

      # model predictions
      preds = model(sent_id, mask)

      # compute the validation loss between actual and predicted values
      loss = cross_entropy(preds,labels)

      total_loss = total_loss + loss.item()

      preds = preds.detach().cpu().numpy()

      total_preds.append(preds)

  # compute the validation loss of the epoch
  avg_loss = total_loss / len(val_dataloader)

  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  return avg_loss, total_preds

# set initial loss to infinite
best_valid_loss = float('inf')

# empty lists to store training and validation loss of each epoch
train_losses=[]
valid_losses=[]

#for each epoch
for epoch in range(epochs):

    print('\n Epoch {:} / {:}'.format(epoch + 1, epochs))

    #train model
    train_loss, _ = train()

    #evaluate model
    valid_loss, _ = evaluate()

    #save the best model
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'saved_weights.pt')

    # append training and validation loss
    train_losses.append(train_loss)
    valid_losses.append(valid_loss)

    print(f'\nTraining Loss: {train_loss:.3f}')
    print(f'Validation Loss: {valid_loss:.3f}')

the traceback i receive is:
Epoch 1 / 10
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-105-c5138ddf6b25> in <module>()
     12
     13     #train model
---> 14     train_loss, _ = train()
     15
     16     #evaluate model

5 frames
<ipython-input-103-3236a6e339dd> in train()
     24
     25     # get model predictions for the current batch
---> 26     preds = model(sent_id, mask)
     27
     28     # compute the loss between actual and predicted values

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-99-9ebdcf410f97> in forward(self, sent_id, mask)
     28       _, cls_hs = self.bert(sent_id, attention_mask=mask)
     29
---> 30       x = self.fc1(cls_hs)
     31
     32       x = self.relu(x)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

TypeError: linear(): argument 'input' (position 1) must be Tensor, not str

Output:
I've been working on this repo too.
Motivated by the answer provided on this link. There is a class probably named Bert_Arch that inherits the nn.Module and this class has a overriden method named forward. Inside forward method just add the parameter 'return_dict=False' to the self.bert() method call. Like so:
_, cls_hs = self.bert(sent_id, attention_mask=mask, return_dict=False)

This worked for me.


Instruction: model.eval for class with field of network - pytorch

Input:
I have a class model with field of pre-trained resnet
something like:
class A(nn.Module):
    def __init__(self, **kwargs):
        super(A, self).__init__()
        self.resnet = get_resnet()

    def forward(self, x):
        return self.resnet(x)

...

now Im doing
model = A()
...
model.eval()

Is it ok or shuld I overwrite the eval, train functions?

Output:
Short answer
It's OK.
Long answer
As the nn.Module.train() runs recursively like this.
self.training = mode
for module in self.children():
    module.train(mode)
return self

And the nn.Module.eval() is just calling self.train(False)
So as long as self.resnet is an nn.Module subclass. You don't need to bother about it and practically every method in nn.Module except forward will affect all the sub modules.
You can test this by
model = A()
...
model.eval()
print(model.resnet.training)  # should be False

If you get False then everything is fine. If you get something else then there's something wrong with the get_resnet().