Instruction: Is there a element-map function in pytorch?

Input:
I'm new in PyTorch and I come from functional programming languages(where map function is used everywhere). The problem is that I have a tensor and I want to do some operations on each element of the tensor. The operation may be various so I need a function like this:
map : (Numeric -> Numeric) -> Tensor -> Tensor

e.g. map(lambda x: x if x < 255 else -1, tensor) # the example is simple, but the lambda may be very complex

Is there such a function in PyTorch? How should I implement such function?

Output:
Most mathematical operations that are implemented for tensors (and similarly for ndarrays in numpy) are actually applied element wise, so you could write for instance
mask = tensor < 255
result = tensor * mask + (-1) * ~mask

This is a quite general appraoch. For the case that you have right now where you only want to modify certain elements, you can also apply "logical indexing" that let's you overwrite the current tensor:
tensor[mask < 255] = -1

So in python there actually is a map() function but usually there are better ways to do it (better in python; in other languages - like Haskell -  map/fmap is obviously prefered in most contexts).
So the key take-away here is that the preferred method is taking advantage of the vectorization. This also makes the code more efficient as those tensor operations are implemented in a low level language, while map() is nothing but a python-for loop that is a lot slower.


Instruction: HTTPS POST to query FastAPI using python requests

Input:
I am trying to serve a Neural Network using FastAPI.
from fastapi import Depends, FastAPI
from pydantic import BaseModel
from typing import Dict

class iRequest(BaseModel):
    arg1: str
    arg2: str

class iResponse(BaseModel):
    pred: str
    probs: Dict[str, float]

@app.post("/predict", response_model=iResponse)
def predict(request: iRequest, model: Model = Depends(get_model)):
    pred, probs = model.predict(request.arg1, request.arg2)
    return iResponse(pred = pred, probs = probs)

The manual site http://localhost:8000/docs#/default/predict_predict_post works fine and translates into the following curl command:
curl -X POST "http://localhost:8000/predict" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"arg1\":\"I am the King\",\"arg2\":\"You are not my King\"}"

which also works.
When I try to query the API using python requests:
import requests
data = {"arg1": "I am the King",
        "arg2": "You are not my King"}
r = requests.post("http://localhost:8000/predict", data=data)

I only get the "422 Unprocessable Entity" Errors. Where am I going wrong here?

Output:
You provide a data argument to requests.post, which does a POST with Content-Type: application/x-www-form-urlencoded, which is not JSON.
Consider using requests.post(url, json=data) and you should be fine.


Instruction: torchvision MNIST HTTPError: HTTP Error 403: Forbidden

Input:
I am trying to replicate this experiment presented in this webpage https://adversarial-ml-tutorial.org/adversarial_examples/
I got the jupyter notebook and loaded in my localhost and open it using Jupiter notebook. When I run the following code to get the dataset using the following code:
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

mnist_train = datasets.MNIST("../data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("../data", train=False, download=True, transform=transforms.ToTensor())
train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 100, shuffle=False)

and I get the following error:
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ../data\MNIST\raw\train-images-idx3-ubyte.gz
0/? [00:00<?, ?it/s]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-15-e6f62798f426> in <module>
      2 from torch.utils.data import DataLoader
      3
----> 4 mnist_train = datasets.MNIST("../data", train=True, download=True, transform=transforms.ToTensor())
      5 mnist_test = datasets.MNIST("../data", train=False, download=True, transform=transforms.ToTensor())
      6 train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)

~\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root, train, transform, target_transform, download)
     77
     78         if download:
---> 79             self.download()
     80
     81         if not self._check_exists():

~\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in download(self)
    144         for url, md5 in self.resources:
    145             filename = url.rpartition('/')[2]
--> 146             download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
    147
    148         # process and save as torch files

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
    254         filename = os.path.basename(url)
    255
--> 256     download_url(url, download_root, filename, md5)
    257
    258     archive = os.path.join(download_root, filename)

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5)
     82                 )
     83             else:
---> 84                 raise e
     85         # check integrity of downloaded file
     86         if not check_integrity(fpath, md5):

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5)
     70             urllib.request.urlretrieve(
     71                 url, fpath,
---> 72                 reporthook=gen_bar_updater()
     73             )
     74         except (urllib.error.URLError, IOError) as e:  # type: ignore[attr-defined]

~\Anaconda3\lib\urllib\request.py in urlretrieve(url, filename, reporthook, data)
    245     url_type, path = splittype(url)
    246
--> 247     with contextlib.closing(urlopen(url, data)) as fp:
    248         headers = fp.info()
    249

~\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223
    224 def install_opener(opener):

~\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
    529         for processor in self.process_response.get(protocol, []):
    530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
    532
    533         return response

~\Anaconda3\lib\urllib\request.py in http_response(self, request, response)
    639         if not (200 <= code < 300):
    640             response = self.parent.error(
--> 641                 'http', request, response, code, msg, hdrs)
    642
    643         return response

~\Anaconda3\lib\urllib\request.py in error(self, proto, *args)
    567         if http_err:
    568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
    570
    571 # XXX probably also want an abstract factory that knows when it makes

~\Anaconda3\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    501         for handler in handlers:
    502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
    504             if result is not None:
    505                 return result

~\Anaconda3\lib\urllib\request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

Any help solving this issue is much appreciated.
I also can download the dataset directly from the link but then I don't know how to use that!

Output:
Yes it's a known bug: https://github.com/pytorch/vision/issues/3500
The possible solution can be to patch MNIST download method.
But it requires wget to be installed.
For Linux:
sudo apt install wget

For Windows:
choco install wget

import os
import subprocess as sp
from torchvision.datasets.mnist import MNIST, read_image_file, read_label_file
from torchvision.datasets.utils import extract_archive

def patched_download(self):
    """wget patched download method.
    """
    if self._check_exists():
        return

    os.makedirs(self.raw_folder, exist_ok=True)
    os.makedirs(self.processed_folder, exist_ok=True)

    # download files
    for url, md5 in self.resources:
        filename = url.rpartition('/')[2]
        download_root = os.path.expanduser(self.raw_folder)
        extract_root = None
        remove_finished = False

        if extract_root is None:
            extract_root = download_root
        if not filename:
            filename = os.path.basename(url)

        # Use wget to download archives
        sp.run(["wget", url, "-P", download_root])

        archive = os.path.join(download_root, filename)
        print("Extracting {} to {}".format(archive, extract_root))
        extract_archive(archive, extract_root, remove_finished)

    # process and save as torch files
    print('Processing...')

    training_set = (
        read_image_file(os.path.join(self.raw_folder, 'train-images-idx3-ubyte')),
        read_label_file(os.path.join(self.raw_folder, 'train-labels-idx1-ubyte'))
    )
    test_set = (
        read_image_file(os.path.join(self.raw_folder, 't10k-images-idx3-ubyte')),
        read_label_file(os.path.join(self.raw_folder, 't10k-labels-idx1-ubyte'))
    )
    with open(os.path.join(self.processed_folder, self.training_file), 'wb') as f:
        torch.save(training_set, f)
    with open(os.path.join(self.processed_folder, self.test_file), 'wb') as f:
        torch.save(test_set, f)

    print('Done!')

MNIST.download = patched_download

mnist_train = MNIST("../data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = MNIST("../data", train=False, download=True, transform=transforms.ToTensor())
train_loader = DataLoader(mnist_train, batch_size=1, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size=1, shuffle=False)


Instruction: Creating an annotation list using time stamps

Input:
I have ECG data of length 230897 and sampling frequency=100hz. I would like to create an annotation list (of length 230897) corresponding to the timestamps and duration (in seconds) of the arrhythmia event (sequence of 1's):
timestamps                                  duration
2014-09-10T22:10:20.000000000                 3.5
2014-09-10T23:10:10.000000000                 4
2014-09-10T23:50:20.000000000                 6

FOr example: the annotation list should be of length 230897 and at '2014-09-10T22:10:20.000000000' there should be 1's for 3.5 seconds; at'2014-09-10T23:10:10.000000000' 1's for 4 seconds and so on. This is what I have tried so far and unfortunately, it is not working:
start_time= '2014-09-10T21:01:10.000000000'
data_length=230897
annot_list = np.zeros(data_length)

prev=0, fs=100
for t in timestamps:
        time_diff = t - prev
        seconds = time_diff.astype('timedelta64[s]').astype(np.int32)
        total_cov = seconds*fs
        i1 = i0 + total_cov
        i0, i1 = int(i0), int(i1)
        annot_list[i0:i1] = 1.0
        i0 = i1
        prev = t

Can someone please help me here?

Output:
Assuming that your data looks like that:
timestamps = [np.datetime64("2014-09-10T22:10:20.000000000"),
    np.datetime64("2014-09-10T23:10:10.000000000"),
    np.datetime64("2014-09-10T23:50:20.000000000")]

durations = [3.5,4,6]

Then, you can simply iterate over your list and caluclate your start index and your stop index from your start time, your timestamp and your duration, like so:
start_time = np.datetime64("2014-09-10T21:01:10.000000000")
data_length = 2308970
annot_list = np.zeros(data_length)
freq = 100

for timestamp, duration in zip(timestamps, durations):
    time_from_start = int(
        freq * (np.timedelta64(timestamp - start_time, "s").astype("float32"))
    )
    duration = int(freq * duration)
    annot_list[time_from_start : time_from_start + duration] = 1


Instruction: Pytorch lightning metrics: ValueError: preds and target must have same number of dimensions, or one additional dimension for preds

Input:
Googling this gets you no where, so I decided to help future me and others by posting this as a searchable question.

def __init__():
    ...
    self.val_acc = pl.metrics.Accuracy()

def validation_step(self, batch, batch_index):
    ...
    self.val_acc.update(log_probs, label_batch)

gives
ValueError: preds and target must have same number of dimensions, or one additional dimension for preds

for log_probs.shape == (16, 4) and for label_batch.shape == (16, 4)
What's the issue?

Output:
pl.metrics.Accuracy() expects a batch of dtype=torch.long labels, not one-hot encoded labels.
Thus, it should be fed
self.val_acc.update(log_probs, torch.argmax(label_batch.squeeze(), dim=1))

This is just the same as torch.nn.CrossEntropyLoss


Instruction: UserWarning: torchaudio C++ extension is not available

Input:
can someone please help me out with this UserWarning in torchaudio?
ErrorMessage:
C:\Users\anaconda3\lib\site-packages\torchaudio\extension\extension.py:14:
UserWarning: torchaudio C++ extension is not available.
  warnings.warn('torchaudio C++ extension is not available.')

Thank's in advance!

Output:
https://pytorch.org/audio/stable/backend.html says like this:
Availability
"sox" and "sox_io" backends require C++ extension module, which is included in Linux/macOS binary distributions. These backends are not available on Windows.
So you have a wrong operating system or backend selected. Depends point of view :)


Instruction: Converting 1D tensor into a 1D array using Fastai

Input:
I have a question, about hoy to convert a 1D Tensor to a 1D Array, using Fastai.
For example, I have this tensor:
tensor([2.2097e-05, 2.6679e-04, 4.6098e-05, 5.5458e-01, 4.4509e-01])

Output:
I'm assuming when you say tensor you are talking about Pytorch and when you say array you are talking about Numpy.
Then this should do it.
tensor.numpy()


Instruction: Is there a better way to multiply & sum two Pytorch tensors along the first dimension?

Input:
I have two Pytorch tensors, a & b, of shape (S, M) and (S, M, H) respectively. M is my batch dimension. I want to multiply & sum the two tensors such that the output is of shape (M, H). That is, I want to compute the sum over s of a[s] * b[s].
For example, for S=2, M=2, H=3:
>>> import torch
>>> S, M, H = 2, 2, 3
>>> a = torch.arange(S*M).view((S,M))
tensor([[0, 1],
        [2, 3]])
>>> b = torch.arange(S*M*H).view((S,M,H))
tensor([[[ 0,  1,  2],
         [ 3,  4,  5]],

        [[ 6,  7,  8],
         [ 9, 10, 11]]])

'''
DESIRED OUTPUT:
= [[0*[0, 1, 2] + 2*[6, 7, 8]],
   [1*[3, 4, 5] + 3*[9, 10, 11]]]

= [[12, 14, 16],
   [30, 34, 38]]

note: shape is (2, 3) = (M, H)
'''

I've found one way that sort of works, using torch.tensordot:
>>> output = torch.tensordot(a, b, ([0], [0]))
tensor([[[12, 14, 16],
         [18, 20, 22]],

        [[18, 22, 26],
         [30, 34, 38]]])
>>> output.shape
torch.Size([2, 2, 3]) # always (M, M, H)
>>> output = output[torch.arange(M), torch.arange(M), :]
tensor([[12, 14, 16],
        [30, 34, 38]])

But as you can see, it makes a lot of unnecessary computations and I have to slice the ones that are relevant for me.
Is there a better way to do this that doesn't involve the unnecessary computations?

Output:
This should work:
(torch.unsqueeze(a, 2)*b).sum(axis=0)
>>> tensor([[12, 14, 16],
            [30, 34, 38]])


Instruction: PyTorch model input shape

Input:
I loaded a custom PyTorch model and I want to find out its input shape. Something like this:
model.input_shape

Is it possible to get this information?

Update: print() and summary() don't show this model's input shape, so they are not what I'm looking for.

Output:
PyTorch flexibility
PyTorch models are very flexible objects, to the point where they do not enforce or generally expect a fixed input shape for data.
If you have certain layers there may be constraints e.g:

a flatten followed by a fully connected layer of width N would enforce the dimensions of your original input (M1 x M2 x ... Mn) to have a product equal to N
a 2d convolution of N input channels would enforce the data to be 3 dimensionsal, with the first dimension having size N

But as you can see neither of these enforce the total shape of the data.

We might not realize it right now, but in more complex models, getting the size of the first linear layer right is sometimes a source of frustration. We’ve heard stories of famous practitioners putting in arbitrary numbers and then relying on error messages from PyTorch to backtrack the correct sizes for their linear layers. Lame, eh? Nah, it’s all legit!

Deep Learning with PyTorch

Investigation
Simple case: First layer is Fully Connected
If your model's first layer is a fully connected one, then the first layer in print(model) will detail the expected dimensionality of a single sample.
Ambiguous case: CNN
If it is a convolutional layer however, since these are dynamic and will stride as long/wide as the input permits, there is no simple way to retrieve this info from the model itself.1 This flexibility means that for many architectures multiple compatible input sizes2 will all be acceptable by the network.
This is a feature of PyTorch's Dynamic computational graph.
Manual inspection
What you will need to do is investigate the network architecture, and once you've found an interpretable layer (if one is present e.g. fully connected) "work backwards" with its dimensions, determining how the previous layers (e.g. poolings and convolutions) have compressed/modified it.
Example
e.g. in the following model from Deep Learning with PyTorch (8.5.1):
class NetWidth(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 16, kernel_size=3, padding=1)
        self.fc1 = nn.Linear(16 * 8 * 8, 32)
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
        out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
        out = out.view(-1, 16 * 8 * 8)
        out = torch.tanh(self.fc1(out))
        out = self.fc2(out)
        return out

We see the model takes an input 2.d. image with 3 channels and:

Conv2d -> sends it to an image of the same size with 32 channels
max_pool2d(,2) -> halves the size of the image in each dimension
Conv2d -> sends it to an image of the same size with 16 channels
max_pool2d(,2) -> halves the size of the image in each dimension
view -> reshapes the image
Linear -> takes a tensor of size 16 * 8 * 8 and sends to size 32
...

So working backwards, we have:

a tensor of shape 16 * 8 * 8
un-reshaped into shape (channels x height x width)
un-max_pooled in 2d with factor 2, so height and width un-halved
un-convolved from 16 channels to 32
Hypothesis: It is likely 16 in the product thus refers to the number of channels, and that the image seen by view was of shape (channels, 8,8), and currently is (channels, 16,16)2
un-max_pooled in 2d with factor 2, so height and width un-halved again (channels, 32,32)
un-convolved from 32 channels to 3

So assuming the kernel_size and padding are sufficient that the convolutions themselves maintain image dimensions, it is likely that the input image is of shape (3,32,32) i.e. RGB 32x32 pixel square images.

Notes:

Even the external package pytorch-summary requires you provide the input shape in order to display the shape of the output of each layer.

It could however be any 2 numbers whose produce equals 8*8 e.g. (64,1), (32,2), (16,4) etc however since the code is written as 8*8 it is likely the authors used the actual dimensions.


Instruction: TypeError: __init__() got an unexpected keyword argument 'filepath'

Input:
I don't know how to solve this error but I hope some of you guys know how to solve this issue.
Error:TypeError: __init__() got an unexpected keyword argument 'filepath'
Full error message:
  File "train.py", line 167, in <module>
    main(args)
  File "train.py", line 113, in main
    checkpoint_callback=checkpoint_callback(),
  File "train.py", line 86, in checkpoint_callback
    return ModelCheckpoint(
TypeError: __init__() got an unexpected keyword argument 'filepath'

from pytorch_lightning.callbacks import ModelCheckpoint

save_model_path = path/to/your/dir
def checkpoint_callback():
    return ModelCheckpoint(
        filepath= save_model_path,
        save_top_k=True,
        verbose=True,
        monitor='val_loss',
        mode='min',
        prefix=''
    )

Output:
ModelCheckpoint doesn't have a filepath keyword, it does however have a dirpath keyword (as you can see in the documentation), replace filepath with dirpath, like this:
from pytorch_lightning.callbacks import ModelCheckpoint

save_model_path = path/to/your/dir
def checkpoint_callback():
    return ModelCheckpoint(
        dirpath=save_model_path, # changed line
        save_top_k=True,
        verbose=True,
        monitor='val_loss',
        mode='min',
        prefix=''
    )


Instruction: Pytorch: split a tensor by column

Input:
How can I split a tensor by column (axis = 1). For example
"""
input:              result:
tensor([[1, 1],     (tensor([1, 2, 3, 1, 2, 3]),
        [2, 1],      tensor([1, 1, 2, 2, 3, 3]))
        [3, 2],
        [1, 2],
        [2, 3],
        [3, 3]])
"""

The solution I came out with is first transpose the input tensor, split it and then flatten each of the split tensor. However, is there a simpler and more effective way on doing this? Thank you
import torch
x = torch.LongTensor([[1,1],[2,1],[3,2],[1,2],[2,3],[3,3]])
x1, x2 = torch.split(x.T, 1)
x1 = torch.flatten(x1)
x2 = torch.flatten(x2)
x1, x2 # output

Output:
Simply do:
x1 = x[:, 0]
x2 = x[:, 1]
# x1: (tensor([1, 2, 3, 1, 2, 3]), x2: tensor([1, 1, 2, 2, 3, 3]))


Instruction: CUDA_OUT_OF_MEMORY in PyTorch head2head model

Input:
I am executing the head2head model presented in the Github repo here.
When I am running the code using the following command:
./scripts/train/train_on_target.sh Obama head2headDataset

with contents of the train_on_target.sh file as:
target_name=$1
dataset_name=$2

python train.py --checkpoints_dir checkpoints/$dataset_name \
                --target_name $target_name \
                --name head2head_$target_name \
                --dataroot datasets/$dataset_name/dataset \
                --serial_batches

Then I am getting the following error:
Traceback (most recent call last):
  File "train.py", line 108, in <module>
    flow_ref, conf_ref, t_scales, n_frames_D)
  File "/home/nitin/head2head/util/util.py", line 48, in get_skipped_flows
    flow_ref_skipped[s], conf_ref_skipped[s] = flowNet(real_B[s][:,1:], real_B[s][:,:-1])
  File "/home/nitin/anaconda3/envs/head2head/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nitin/anaconda3/envs/head2head/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/nitin/anaconda3/envs/head2head/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nitin/head2head/models/flownet.py", line 38, in forward
    flow, conf = self.compute_flow_and_conf(input_A, input_B)
  File "/home/nitin/head2head/models/flownet.py", line 55, in compute_flow_and_conf
    flow1 = self.flowNet(data1)
  File "/home/nitin/anaconda3/envs/head2head/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nitin/head2head/models/flownet2_pytorch/models.py", line 156, in forward
    flownetfusion_flow = self.flownetfusion(concat3)
  File "/home/nitin/anaconda3/envs/head2head/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nitin/head2head/models/flownet2_pytorch/networks/FlowNetFusion.py", line 62, in forward
    concat0 = torch.cat((out_conv0,out_deconv0,flow1_up),1)
RuntimeError: CUDA out of memory. Tried to allocate 82.00 MiB (GPU 0; 5.80 GiB total capacity; 4.77 GiB already allocated; 73.56 MiB free; 4.88 GiB reserved in total by PyTorch)

I have checked the batch size in the file options/base_options.py. It is already set to 1. How can I solve the above mentioned exception. My system has 6 GB NVIDIA GTX 1660 Super GPU.

Output:
Data management:
You can try reducing the dataset used for training to check if is a hardware limitation.
Moreover, if it is an image dataset, you can reduce the dimensions of the images by reducing the dpi.
Model parameters management:
Another approach is to reduce the number of parameters of your model. The first suggestion would be to change the Dense layer size and then the other neural network hyperparameters.


Instruction: Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU

Input:
Following my previous question , I have written this code to train an autoencoder and then extract the features.
(There might be some changes in the variable names)
# Autoencoder class
#https://medium.com/pytorch/implementing-an-autoencoder-in-pytorch-19baa22647d1
class AE_class(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=128
        )
        self.encoder_output_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_output_layer = nn.Linear(
            in_features=128, out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        #print("in forward")
        #print(type(features))
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)
        code = self.encoder_output_layer(activation)
        code = torch.relu(code)
        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)
        activation = self.decoder_output_layer(activation)
        reconstructed = torch.relu(activation)
        return reconstructed

    def encode(self, features_h):
        activation_h = self.encoder_hidden_layer(features_h)
        activation_h = torch.relu(activation_h)
        code_h = self.encoder_output_layer(activation_h)
        code_h = torch.relu(code_h)
        return code_h

And then, for training:
def retrieve_AE_features(X_before, n_voxel_region):

        #  use gpu if available
    #https://discuss.pytorch.org/t/runtimeerror-tensor-for-out-is-on-cpu-tensor-for-argument-1-self-is-on-cpu-but-expected-them-to-be-on-gpu-while-checking-arguments-for-addmm/105453
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # create a model from `AE` autoencoder class
    # load it to the specified device, either gpu or cpu
    model_AE = AE_class(input_shape=n_voxel_region).to(device)

    # create an optimizer object
    # Adam optimizer with learning rate 1e-3
    optimizer = optim.Adam(model_AE.parameters(), lr=1e-3)

    # mean-squared error loss
    criterion = nn.MSELoss()

    X_tensor = torch.tensor(X_before, dtype=torch.float32)

    print(type(X_tensor))

    train_loader = torch.utils.data.DataLoader(
        X_tensor, batch_size=64, shuffle=True, num_workers=2, pin_memory=True
    )

    test_loader = torch.utils.data.DataLoader(
        X_tensor, batch_size=32, shuffle=False, num_workers=2
    )

    print(type(train_loader))

    for epoch in range(epochs_AE):
        loss = 0

        for batch_features in train_loader:
            # reshape mini-batch data to [N, 784] matrix
            # load it to the active device
            #batch_features = batch_features.view(-1, 784).to(device)

            #print(batch_features.shape)

            # reset the gradients back to zero
            # PyTorch accumulates gradients on subsequent backward passes
            optimizer.zero_grad()

            # compute reconstructions
            outputs = model_AE(batch_features)

            # compute training reconstruction loss
            train_loss = criterion(outputs, batch_features)

            # compute accumulated gradients
            train_loss.backward()

            # perform parameter update based on current gradients
            optimizer.step()

            # add the mini-batch training loss to epoch loss
            loss += train_loss.item()

        # compute the epoch training loss
        loss = loss / len(train_loader)

        # display the epoch training loss

        print("AE, epoch : {}/{}, loss = {:.6f}".format(epoch + 1, epochs_AE, loss))

    #After training
    hidden_features = model_AE.encode(X_before)
    return hidden_features

However, I received the following error:

Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU
(while checking arguments for addmm)

It seems some of my variables should be defined in another way to be able to be executed on GPU.
My questions:

How can I understand which variables will be executed on GPU and which ones on CPU?
How to fix it? In other words, how to define a variable executable on GPU?

Thanks in advance

Output:
I see that Your model is moved to device which is decided by this line device = torch.device("cuda" if torch.cuda.is_available() else "cpu") This can be is either cpu or cuda.
So adding this line batch_features = batch_features.to(device) will actually move your input data to device.
Since your model is moved to device , You should also move your input to the device.
Below code has that change
for epoch in range(epochs_AE):
    loss = 0

    for batch_features in train_loader:
        batch_features = batch_features.to(device)  #this will move inout to your device
        optimizer.zero_grad()
        outputs = model_AE(batch_features)
        train_loss = criterion(outputs, batch_features)
        ...

coming to your question : Calling .to(device) can directly move the tensor to your specified device
And if you want it it to be hard coded then do .to('cpu') or .to('cuda') on your torch tensor


Instruction: Pytorch pairwise concatenation of tensors

Input:
I'd like to compute a pairwise concatenation over a specific dimension in a batched manner.
For instance,
x = torch.tensor([[[0],[1],[2]],[[3],[4],[5]]])
x.shape = torch.Size([2, 3, 1])

I would like to get y such that y is the concatenation of all pairs of vectors across one dimension, ie:
y = torch.tensor([[[[0,0],[0,1],[0,2]],[[1,0],[1,1],[1,2]], [[2,0], [2,1], [2,2]]],
                 [[[3,3],[3,4],[3,5]],[[4,3],[4,4],[4,5]], [[5,3],[5,4],[5,5]]]])

y.shape = torch.Size([2, 3, 3, 2])

So essentially, for each x[i,:], you generate all pairs of vectors and you concatenate them on the last dimension.
Is there a straightforward way of doing that?

Output:
One possible way to do that would be:
    all_ordered_idx_pairs = torch.cartesian_prod(torch.tensor(range(x.shape[1])),torch.tensor(range(x.shape[1])))
    y = torch.stack([x[i][all_ordered_idx_pairs] for i in range(x.shape[0])])

After reshaping the tensor:
y = y.view(x.shape[0], x.shape[1], x.shape[1], -1)

you get:
y = torch.tensor([[[[0,0],[0,1],[0,2]],[[1,0],[1,1],[1,2]], [[2,0], [2,1], [2,2]]],
                 [[[3,3],[3,4],[3,5]],[[4,3],[4,4],[4,5]], [[5,3],[5,4],[5,5]]]])


Instruction: TorchServe MAR multiple Python files per model

Input:
Is there a way to split a model between multiple files before calling MAR?
For example, if I have the file base_model.py:
import torch
from torch import nn

class BaseModel(nn.Module):
    ...

And a model that inherits from BaseModel:
from base_model import BaseModel

class FullModel(BaseModel):
    ...

The same goes for the handler. I don't know how to split the code between multiple files.

Output:
Short answer: Yes
You can pass main model file and remaining files as extra-files.
For example: full_model.py using  --model-file ./xxx/aaa/full_model.py and all the other files using --extra-files ./xxx/bbb/base_model_1.py,./xxx/ccc/base_model_2.py,./xxx/ddd/mini_handler.py,./xxx/eee/mini_handler_2.py.
All the files will be in a single folder before packaging as .mar file. With this in mind you might need to change imports in your python files a bit.
All in all, it should be similar to
torch-model-archiver \
    --model-name recommendation \
    --version 1.0 \
    --model-file src/model/full_model.py \
    --serialized-file model/model.pt \
    --handle model_handler.py \
    --extra-files \
"dd/utils/utils.py,\
ee/network_config.json,\
ff/utils/preprocess.py,\
gg/utils/data_loader.py\

Now you can use functions from preprocess or dataloader in model handler by normal imports


Instruction: Error in running backwards() function in PyTorch

Input:
The code:
 import numpy as np
 predictors = np.array([[73,67,43],[91,88,64],[87,134,58],[102,43,37],[69,96,70]],dtype='float32')
 outputs = np.array([[56,70],[81,101],[119,133],[22,37],[103,119]],dtype='float32')

 inputs = torch.from_numpy(predictors)
 targets = torch.from_numpy(outputs)

 weights = torch.randn(2,3,requires_grad=True)
 biases = torch.randn(2,requires_grad=True)

 def loss_mse(x,y):
  d = x-y
  return torch.sum(d*d)/d.numel()

 def model(w,b,x):
  return x @ w.t() +b

 def train(x,y,w,b,lr,e):
  w = torch.tensor(w,requires_grad=True)
  b = torch.tensor(b,requires_grad=True)
  for epoch in range(e):
    preds = model(w,b,x)
    loss = loss_mse(y,preds)
    if epoch%5 == 0:
      print("Loss at Epoch [{}/{}] is {}".format(epoch,e,loss))
    #loss.requires_grad=True
    loss.backward()
    with torch.no_grad():
      w = w - lr*w.grad
      b = b - lr*b.grad
      w.grad.zero_()
      b.grad.zero_()

 train(inputs,targets,weights,biases,1e-5,100)

Running this gives different errors. Once it gave the error that loss was of size 0. Then it gave the error in the update line w = w-lr*w.grad that float can't be subtracted from NoneType.

Output:
First, why do you wrap your weights and biases as Tensor twice?
weights = torch.randn(2,3,requires_grad=True)
biases = torch.randn(2,requires_grad=True)de here

then inside the train function you use:
w = torch.tensor(w,requires_grad=True)
b = torch.tensor(b,requires_grad=True)

Second, in the part of updating your weights change it to:
  with torch.no_grad():
   w_new = w - lr*w.grad
   b_new = b - lr*b.grad
   w.copy_(w_new)
   b.copy_(b_new)
   w.grad.zero_()
   b.grad.zero_()

you can check this discussion for a more comprehensive explanation:
https://discuss.pytorch.org/t/updatation-of-parameters-without-using-optimizer-step/34244/20


Instruction: Purpose of '*' in front of Python3 function

Input:
I saw the code for a ResNet CNN in Python3 and PyTorch here as follows:
def resnet_block(input_channels, num_channels, num_residuals,
                 first_block=False):
    blk = []
    for i in range(num_residuals):
        if i == 0 and not first_block:
            blk.append(Residual(input_channels, num_channels,
                                use_1x1conv=True, strides=2))
        else:
            blk.append(Residual(num_channels, num_channels))
    return blk

To add the modules, the following code is used-
b2 = nn.Sequential(*resnet_block(64, 64, 2, first_block=True))
b3 = nn.Sequential(*resnet_block(64, 128, 2))
b4 = nn.Sequential(*resnet_block(128, 256, 2))
b5 = nn.Sequential(*resnet_block(256, 512, 2))

What does "*resnet_block()" mean/do?

Output: Basically *iterable is used to unpack the items of an iterable object as positional arguments. In your question resnet_block returns a list, and the items of that list are passed to nn.Sequential rather than the list itself.


Instruction: Pytorch only sees 15GB memory when the device should have more

Input:
I just kicked off a g4dn.8xlarge instance on AWS. According to their website, this instance should have 128GB memory, as shown in the screenshot below:

However, I notice my model keeps running out of memory. When I investigated, I see this:
>>> print(torch.cuda.get_device_properties('cuda:0'))
_CudaDeviceProperties(name='Tesla T4', major=7, minor=5, total_memory=15109MB, multi_processor_count=40)

It turns out the device can only use about 15GB of memory! Could someone please explain why this might be? Do I need to use a different AWS instance to get more memory, or is PyTorch just not able to use the rest of the memory that's available?

Output: The g4dn.8xlarge use NVIDIA T4 GPU which have 16GB of RAM. So I think PyTorch shows the memory of the GPU, not the instance.


Instruction: Is pytorch 1.7 officially enabled for cuda 10.0?

Input:
I had to stay on CUDA 10.0 for personal projects.
Rather than installing Pytorch with versions appropriate for CUDA 10.0, I accidentally installed Pytorch 1.7 supported with CUDA 10.1. In particular, I installed by
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Surprisingly, everything works fine so far although the CUDA versions do not match.
To verify my installation, I've run the code given in collect_env.py, and it was fine.
I am just wondering few things.

Did Pytorch team officially comment Pytorch 1.7 is compatible with CUDA 10.0?
Would there be more rigorous ways to verify my Pytorch installation?

Output:
Surprisingly, everything works fine so far although the CUDA versions
do not match.

Changes between minor versions should work (mismatch like this worked in my case), although there is no promise of compatibility in 10.x release (source), only since 11.x there will be binary compatibility.

Did Pytorch team officially commented Pytorch 1.7 is compatible with
CUDA 10.0?

Not that I'm aware of, but listed wheels do not include 10.0 CUDA and PyTorch 1.7.0 (latest with 10.0 support seems to be 1.4.0).

Would there be more rigorous way to verify my Pytorch installation?

As above, maybe cloning PyTorch's github repo, reverting to tagged release and running tests (folder here, one of cuda test files here), but for personal projects might be excessive.


Instruction: AttributeError: module 'torchtext.data' has no attribute 'Field'

Input:
I want to run a git project used pytorch and torchtext but when I run it, it raise error:
  File "main.py", line 60, in <module>
    main()
  File "main.py", line 50, in main
    train_iters, dev_iters, test_iters, vocab = load_dataset(config)
  File "/home/esmailza/style transfer/style-transformer/data.py", line 23, in load_dataset
    TEXT = data.Field(batch_first=True, eos_token='<eos>')
AttributeError: module 'torchtext.data' has no attribute 'Field'

torch version  = 1.8.0
torchtext version = 0.9

def load_dataset(config, train_pos='train.pos', train_neg='train.neg',
                 dev_pos='dev.pos', dev_neg='dev.neg',
                 test_pos='test.pos', test_neg='test.neg'):

    root = config.data_path
    TEXT = data.Field(batch_first=True, eos_token='<eos>')

    dataset_fn = lambda name: data.TabularDataset(
        path=root + name,
        format='tsv',
        fields=[('text', TEXT)]
    )

Output:
From TorchText 0.9.0 Release Notes

torchtext.data.Field -> torchtext.legacy.data.Field
This means, all features are still available, but within torchtext.legacy instead of torchtext.

torchtext.data.Field has been moved to torchtext.legacy.data.Field

And the imports would change this way:
from torchtext.legacy import data


Instruction: Does the PyTorch Lightning average metrics over the whole epoch?

Input:
I am looking at the example provided on PyTorch-Lightning official documentation https://pytorch-lightning.readthedocs.io/en/0.9.0/lightning-module.html.
Here the loss and metric is calculated on the concrete batch. But when logging one is not interested in the accuracy for a particular batch, which can be rather small and not representative, but the averaged over all epoch. Do I understand correctly, that there is some code performing the averaging over all batches, passed through the epoch?
 import pytorch_lightning as pl
 from pytorch_lightning.metrics import functional as FM

 class ClassificationTask(pl.LightningModule):

 def __init__(self, model):
     super().__init__()
     self.model = model

 def training_step(self, batch, batch_idx):
     x, y = batch
     y_hat = self.model(x)
     loss = F.cross_entropy(y_hat, y)
     return pl.TrainResult(loss)

 def validation_step(self, batch, batch_idx):
    x, y = batch
    y_hat = self.model(x)
    loss = F.cross_entropy(y_hat, y)
    acc = FM.accuracy(y_hat, y)
    result = pl.EvalResult(checkpoint_on=loss)
    result.log_dict({'val_acc': acc, 'val_loss': loss})
    return result

 def test_step(self, batch, batch_idx):
    result = self.validation_step(batch, batch_idx)
    result.rename_keys({'val_acc': 'test_acc', 'val_loss': 'test_loss'})
    return result

 def configure_optimizers(self):
     return torch.optim.Adam(self.model.parameters(), lr=0.02)

Output:
If you want to average metrics over the epoch, you'll need to tell the LightningModule you've subclassed to do so. There are a few different ways to do this such as:

Call result.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True, logger=True) as shown in the docs with on_epoch=True so that the training loss is averaged across the epoch. I.e.:

 def training_step(self, batch, batch_idx):
     x, y = batch
     y_hat = self.model(x)
     loss = F.cross_entropy(y_hat, y)
     result = pl.TrainResult(loss)
     result.log('train_loss', loss, on_step=True, on_epoch=True, prog_bar=True, logger=True)
     return result

Alternatively, you can call the log method on the LightningModule itself: self.log("train_loss", loss, on_epoch=True, sync_dist=True) (Optionally passing sync_dist=True to reduce across accelerators).

You'll want to do something similar in validation_step to get aggregated val-set metrics or implement the aggregation yourself in the validation_epoch_end method.


Instruction: PyTorch: load weights from another model without saving

Input:
Assume that I have two models in PyTorch, how can I load the weights of model 1 by weights of model 2 without saving the weights?
Like this:
model1.weights = model2.weights

In TensorFlow I can do this:
variables1 = model1.trainable_variables
variables2 = model2.trainable_variables
for v1, v2 in zip(variables1, variables2):
    v1.assign(v2.numpy())

Output:
Assuming you have two instances of the same model (must subclass nn.Module), then you can use nn.Module.state_dict() and nn.Module.load_state_dict(). You can find a brief introduction to state dictionaries here.
model1.load_state_dict(model2.state_dict())


Instruction: Facing Broken Pipe error when trying to run the next(iter(train_data_loader)). I'm running the code in local jupyter notebook

Input:
I'm facing BrokenPipeError when I'm trying to run sentiment analysis with hugging face. It's returning [Error No] 32 Broken Pipe. Is there any way to rewrite the next(iter(train_data_loader)) code?
Link with total code 'https://colab.research.google.com/drive/1wBXKa-gkbSPPk-o7XdwixcGk7gSHRMas?usp=sharing'
The code is
def create_data_loader(df, tokenizer, max_len, batch_size):
  ds = GPReviewDataset(
    reviews=df.content.to_numpy(),
    targets=df.sentiment.to_numpy(),
    tokenizer=tokenizer,
    max_len=max_len
  )
  return DataLoader(
    ds,
    batch_size=batch_size,
    num_workers=4
  )

Followed by below code
BATCH_SIZE = 16
train_data_loader = create_data_loader(df_train, tokenizer, MAX_LEN, BATCH_SIZE)
val_data_loader = create_data_loader(df_val, tokenizer, MAX_LEN, BATCH_SIZE)
test_data_loader = create_data_loader(df_test, tokenizer, MAX_LEN, BATCH_SIZE)

Followed by
data = next(iter(train_data_loader))
data.keys()

I'm facing error with this 'data = next(iter(train_data_loader))' code
Error is BrokenPipeError: [Errno 32] Broken pipe

Output:
One of the reason of this issue might be the OS. When you're using Windows, you should not define num_worker, because PyTorch dataloader does not support multi-processing on Windows. By default num_workers is 0 and works on Windows.
DataLoader(
    ds,
    batch_size=batch_size,
    num_workers=0 # should be zero on Windows
  )


Instruction: PyTorch Tensors - vectorized slicing with given list of end indices

Input:
Suppose I have a 1D PyTorch tensor end_index of length L.
I want to construct a 2D PyTorch tensor T with L lines where
T[i,j] = 2 when j < end_index[i] and T[i,j] = 1 otherwise.
The following works:
T = torch.ones([4,3], dtype=torch.long)
for element in end_index:
    T[:, :element] = 2

Is there a vectorizd way to do this?

Output:
You can construct such a tensor using broadcast semantics
# sample inputs
L, C = 4, 3
end_index = torch.tensor([0, 2, 2, 1])

# Construct tensor of shape [L, C] such that for all (i, j)
#     T[i, j] = 2 if j < end_index[i] else 1
j_range = torch.arange(C, device=end_index.device)
T = (j_range[None, :] < end_index[:, None]).long() + 1

which results in
T =
tensor([[1, 1, 1],
        [2, 2, 1],
        [2, 2, 1],
        [2, 1, 1]])


Instruction: How to replace NaN values in the weights of neural network?

Input:
I am trying to replace NaN values in the weights with some constant. However I am unaware on how one can do so.
I am aware that to access weights, I have to write.
for i in model.params:
    arr = i.clone()
    arr[torch.isnan(arr)] = 0
    param_list.append(nn.Parameter(arr))

model.params= param_list

But this breaks the computation graph, any better way you can suggest me for the same?

Output:
How about
with torch.no_grad():
  for p in model.parameters():
    p.data[torch.isnan(p.data)] = 0


Instruction: AttributeError: 'str' object has no attribute 'shape' while encoding tensor using BertModel with PyTorch (Hugging Face)

Input:
AttributeError: 'str' object has no attribute 'shape' while encoding tensor using BertModel with PyTorch (Hugging Face). Below is the code
bert_model = BertModel.from_pretrained(r'downloads\bert-pretrained-model')
input_ids

Output is:
tensor([[  101,   156, 13329,  ...,     0,     0,     0],
        [  101,   156, 13329,  ...,     0,     0,     0],
        [  101,  1302,  1251,  ...,     0,     0,     0],
        ...,
        [  101, 25456,  1200,  ...,     0,     0,     0],
        [  101,   143,  9664,  ...,     0,     0,     0],
        [  101,  2586,  7340,  ...,     0,     0,     0]])

Followed by code below
last_hidden_state, pooled_output = bert_model(
  input_ids=encoding['input_ids'],
  attention_mask=encoding['attention_mask']
)

Followed by code below
last_hidden_state.shape

Output is
AttributeError                            Traceback (most recent call last)
<ipython-input-70-9628339f425d> in <module>
----> 1 last_hidden_state.shape

AttributeError: 'str' object has no attribute 'shape'

Complete Code link is 'https://colab.research.google.com/drive/1FY4WtqCi2CQ9RjHj4slZwtdMhwaWv2-2?usp=sharing'

Output:
The issue is that the return type has changed since 3.xx version of transformers. So, we have explicitly ask for a tuple of tensors.
So, we can pass an additional kwarg return_dict = False when we call the bert_model() to get an actual tensor that corresponds to the last_hidden_state.
last_hidden_state, pooled_output = bert_model(
  input_ids=encoding['input_ids'],
  attention_mask=encoding['attention_mask'],
  return_dict = False   # this is needed to get a tensor as result
)

In case you do not like the previous approach, then you can resort to:
In [13]: bm = bert_model(
    ...:   encoding_sample['input_ids'],
    ...:   encoding_sample['attention_mask']
    ...: )

In [14]: bm.keys()
Out[14]: odict_keys(['last_hidden_state', 'pooler_output'])

# accessing last_hidden_state
In [15]: bm['last_hidden_state']

In [16]: bm['last_hidden_state'].shape
Out[16]: torch.Size([1, 17, 768])


Instruction: PyTorch for Object detection - Image augmentation

Input:
I am using PyTorch for object detection and refining an existing model (transfer learning) as described in the following link -
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
While different transformations are used for image augmentation (horizontal flip in this tutorial), the tutorial doesnt mention anything on transforming the bounding box/annotation to ensure they are in line with the transformed image. Am I missing something basic here?

Output:
In the training phase, the transforms are indeed applied on both images and targets, while loading the data. In the PennFudanDataset class, we have these two lines:
if self.transforms is not None:
    img, target = self.transforms(img, target)

where target is a dictionary containing:
target = {}
target["boxes"] = boxes
target["labels"] = labels
target["masks"] = masks
target["image_id"] = image_id
target["area"] = area
target["iscrowd"] = iscrowd

self.transforms() in PennFudanDataset class is set to a list of transforms comprising [transforms.ToTensor(), transforms.Compose()], the return value from get_transform() while instantiating the dataset with:
dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))

The transforms transforms.Compose() comes from T, a custom transform written for object detection task. Specifically, in the __call__ of RandomHorizontalFlip(), we process both the image and target (e.g., mask, keypoints):
For the sake of completeness, I borrow the code from the github repo:
def __call__(self, image, target):
        if random.random() < self.prob:
            height, width = image.shape[-2:]
            image = image.flip(-1)
            bbox = target["boxes"]
            bbox[:, [0, 2]] = width - bbox[:, [2, 0]]
            target["boxes"] = bbox
            if "masks" in target:
                target["masks"] = target["masks"].flip(-1)
            if "keypoints" in target:
                keypoints = target["keypoints"]
                keypoints = _flip_coco_person_keypoints(keypoints, width)
                target["keypoints"] = keypoints
        return image, target

Here, we can understand how they perform the flipping on the masks and keypoints in accordance with the image.


Instruction: Training yolov5 causes a CUDNN_STATUS_NOT_INITIALIZED error

Input:
I'm following this guide without changing anything. I'm using an aws server with deep learning ami: Deep Learning AMI (Ubuntu 18.04) Version 40.0
I've tried to change my custom dataset to the coco dataset and to a small subset of the custom one.
batch size doesn't seems to matter, CUDA and other drivers seems to work.
The exception is thrown when the batch starts the training process. This is the full stack trace:
Logging results to runs/train/exp66
Starting training for 5 epochs...

     Epoch   gpu_mem       box       obj       cls     total   targets  img_size
  0%|                                                                                | 0/22 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "train.py", line 533, in <module>
    train(hyp, opt, device, tb_writer, wandb)
  File "train.py", line 298, in train
    pred = model(imgs)  # forward
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/yolov5/models/yolo.py", line 121, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "/home/ubuntu/yolov5/models/yolo.py", line 137, in forward_once
    x = m(x)  # run
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/yolov5/models/common.py", line 113, in forward
    return self.conv(torch.cat([x[..., ::2, ::2], x[..., 1::2, ::2], x[..., ::2, 1::2], x[..., 1::2, 1::2]], 1))
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/yolov5/models/common.py", line 38, in forward
    return self.act(self.bn(self.conv(x)))
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Output: I fixed it using conda, I cloned the pytorch environment one which came with the image and it works perfectly. I still don't know the cause though.


Instruction: Wild discrepancies between training DeepLab ResNet V3 on Google Colab versus on local machine

Input:
I am attempting to train Deeplab Resnet V3 to perform semantic segmentation on a custom dataset. I had been working on my local machine however my GPU is just a small Quadro T1000 so I decided to move my model onto Google Colab to take advantage of their GPU instances and get better results.
Whilst I get the speed increase I was hoping for, I am getting wildly different training losses on colab compared to my local machine. I have copied and pasted the exact same code so the only difference I can find would be in the dataset. I am using the exact same dataset except for the one on colab is a copy of the local dataset on Google Drive. I have noticed that Drive orders file differently on Windows but I can't see how this is a problem since I randomly shuffle the dataset. I understand that these random splitting can cause small differences in the outputs however a difference of about 10x in the training losses doesn't make sense.
I have also tried running the version on colab with different random seeds, different batch sizes, different train_test_split parameters, and changing the optimizer from SGD to Adam, however, this still causes the model to converge very early at a loss of around 0.5.
Here is my code:
import torch
from torch.utils import data
from torchvision import transforms
from customdatasets import SegmentationDataSet
import pathlib
from sklearn.model_selection import train_test_split
from customtransforms import Compose, AlbuSeg2d, DenseTarget
from customtransforms import MoveAxis, Normalize01, Resize
import albumentations
import matplotlib.pyplot as plt
import time
import GPUtil

def get_filenames_of_path(path: pathlib.Path, ext: str = '*'):
    """Returns a list of files in a directory/path. Uses pathlib."""
    filenames = [file for file in path.glob(ext) if file.is_file()]
    return filenames

if __name__ == '__main__':

    root = pathlib.Path.cwd() / 'train'
    inputs = get_filenames_of_path(root / 'input')
    targets = get_filenames_of_path(root / 'target')

# training transformations and augmentations
    transforms_training = Compose([
        Resize(input_size=(128, 128, 3), target_size=(128, 128)),
        AlbuSeg2d(albu=albumentations.HorizontalFlip(p=0.5)),
        MoveAxis(),
        Normalize01()
    ])
# validation transformations
    transforms_validation = Compose([
        Resize(input_size=(128, 128, 3), target_size=(128, 128)),
        MoveAxis(),
        Normalize01()
    ])
    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    random_seed = 142
    train_size = 0.8

    inputs_train, inputs_valid = train_test_split(
        inputs,
        random_state=random_seed,
        train_size=train_size,
        shuffle=True)
    targets_train, targets_valid = train_test_split(
        targets,
        random_state=random_seed,
        train_size=train_size,
        shuffle=True)

    dataset_train = SegmentationDataSet(inputs=inputs_train,
                                    targets=targets_train,
                                    transform=transforms_training,
                                    device=device)

    dataset_valid = SegmentationDataSet(inputs=inputs_valid,
                                    targets=targets_valid,
                                    transform=transforms_validation,
                                    device=device)

    dataloader_training = data.DataLoader(dataset=dataset_train,
                                      batch_size=15,
                                      shuffle=True,
                                      num_workers=4,
                                      pin_memory=True)

    dataloader_validation = data.DataLoader(dataset=dataset_valid,
                                        batch_size=15,
                                        shuffle=True,
                                        num_workers=4,
                                        pin_memory=True)

    model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=False)

    criterion = torch.nn.CrossEntropyLoss()

    model = model.to(device)

    optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.99)

    epochs = 10
    steps = 0
    running_loss = 0
    print_every = 10
    train_losses, valid_losses = [], []

    start_time = time.time()
    prev_time = time.time()

    for epoch in range(epochs):
        #Training
        for inputs, labels in dataloader_training:
            steps += 1
            inputs, labels = inputs.to(device, non_blocking=True), labels.to(device,non_blocking=True)
            optimizer.zero_grad()
            logps = model(inputs)
            loss = criterion(logps['out'], labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

            if steps % print_every == 0:
                train_losses.append(running_loss / len(dataloader_training))
                epoch_time = time.time()
                elasped_time = epoch_time - prev_time
                prev_time = epoch_time
                print(f"Epoch {epoch + 1}/{epochs}.. "
                    f"Train loss: {running_loss / print_every:.3f}.. "
                    f"Elapsed time: {elasped_time}")

                running_loss = 0
                model.train()
        # Evaluation
        valid_loss = 0
        accuracy = 0
        model.eval()
        with torch.no_grad():
            for inputs, labels in dataloader_validation:
                inputs, labels = inputs.to(device, non_blocking=True), labels.to(device, non_blocking=True)
                logps = model.forward(inputs)
                batch_loss = criterion(logps['out'], labels)
                valid_loss += batch_loss.item()

                ps = torch.exp(logps['out'])
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
        valid_losses.append(valid_loss / len(dataloader_validation))
        print(f"Epoch {epoch + 1}/{epochs}.. "
            f"Validation loss: {valid_loss / len(dataloader_training):.3f}.. "
            f"Validation accuracy: {accuracy / len(dataloader_training):.3f} ")
        model.train()
    torch.save(model, 'model.pth')

    end_time = time.time()
    total_time = end_time - start_time
    print("Total Time: ", total_time)
    plt.plot(train_losses, label='Training loss')
    plt.plot(valid_losses, label='Validation loss')
    plt.legend(frameon=False)
    plt.show()

This is the output for one epoch on Colab:
Epoch 1/10.. Train loss: 2.080.. Elapsed time: 12.156640768051147
Epoch 1/10.. Train loss: 1.231.. Elapsed time: 8.76858925819397
Epoch 1/10.. Train loss: 1.051.. Elapsed time: 8.315532445907593
Epoch 1/10.. Train loss: 0.890.. Elapsed time: 8.249168634414673
Epoch 1/10.. Train loss: 0.839.. Elapsed time: 8.248667478561401
Epoch 1/10.. Train loss: 0.807.. Elapsed time: 8.120820999145508
Epoch 1/10.. Train loss: 0.742.. Elapsed time: 8.298616886138916
Epoch 1/10.. Train loss: 0.726.. Elapsed time: 8.170734167098999
Epoch 1/10.. Train loss: 0.677.. Elapsed time: 8.221246004104614
Epoch 1/10.. Train loss: 0.698.. Elapsed time: 8.124614000320435
Epoch 1/10.. Train loss: 0.675.. Elapsed time: 8.197462558746338
Epoch 1/10.. Train loss: 0.682.. Elapsed time: 8.263437509536743
Epoch 1/10.. Train loss: 0.626.. Elapsed time: 8.156179189682007
Epoch 1/10.. Train loss: 0.632.. Elapsed time: 8.268096446990967
Epoch 1/10.. Train loss: 0.616.. Elapsed time: 8.214547872543335
Epoch 1/10.. Train loss: 0.585.. Elapsed time: 8.31475019454956
Epoch 1/10.. Train loss: 0.598.. Elapsed time: 8.388074398040771
Epoch 1/10.. Train loss: 0.626.. Elapsed time: 8.179292440414429
Epoch 1/10.. Train loss: 0.612.. Elapsed time: 8.252359390258789
Epoch 1/10.. Train loss: 0.592.. Elapsed time: 8.284745693206787
Epoch 1/10.. Train loss: 0.597.. Elapsed time: 8.31213927268982
Epoch 1/10.. Train loss: 0.566.. Elapsed time: 8.164374113082886
Epoch 1/10.. Train loss: 0.556.. Elapsed time: 8.300082206726074
Epoch 1/10.. Train loss: 0.568.. Elapsed time: 8.26304841041565
Epoch 1/10.. Train loss: 0.572.. Elapsed time: 8.309881448745728
Epoch 1/10.. Train loss: 0.586.. Elapsed time: 8.211671352386475
Epoch 1/10.. Train loss: 0.586.. Elapsed time: 8.321797609329224
Epoch 1/10.. Train loss: 0.535.. Elapsed time: 8.318871021270752
Epoch 1/10.. Train loss: 0.543.. Elapsed time: 8.152915239334106
Epoch 1/10.. Train loss: 0.569.. Elapsed time: 8.251380205154419
Epoch 1/10.. Train loss: 0.526.. Elapsed time: 8.29153847694397
Epoch 1/10.. Train loss: 0.565.. Elapsed time: 8.15071702003479
Epoch 1/10.. Train loss: 0.542.. Elapsed time: 8.253364562988281
Epoch 1/10.. Validation loss: 0.182.. Validation accuracy: 0.271

And here is the output on my local machine:
Epoch 1/10.. Train loss: 2.932.. Elapsed time: 32.148621797561646
Epoch 1/10.. Train loss: 1.852.. Elapsed time: 14.120505809783936
Epoch 1/10.. Train loss: 0.887.. Elapsed time: 14.210048198699951
Epoch 1/10.. Train loss: 0.618.. Elapsed time: 14.23294186592102
Epoch 1/10.. Train loss: 0.549.. Elapsed time: 14.212541103363037
Epoch 1/10.. Train loss: 0.519.. Elapsed time: 14.047481775283813
Epoch 1/10.. Train loss: 0.506.. Elapsed time: 14.060708045959473
Epoch 1/10.. Train loss: 0.347.. Elapsed time: 14.301624059677124
Epoch 1/10.. Train loss: 0.399.. Elapsed time: 13.9844491481781
Epoch 1/10.. Train loss: 0.361.. Elapsed time: 13.957871913909912
Epoch 1/10.. Train loss: 0.305.. Elapsed time: 14.164010763168335
Epoch 1/10.. Train loss: 0.296.. Elapsed time: 14.001536846160889
Epoch 1/10.. Train loss: 0.298.. Elapsed time: 14.019971132278442
Epoch 1/10.. Train loss: 0.271.. Elapsed time: 13.951345443725586
Epoch 1/10.. Train loss: 0.252.. Elapsed time: 14.037938594818115
Epoch 1/10.. Train loss: 0.283.. Elapsed time: 13.944657564163208
Epoch 1/10.. Train loss: 0.299.. Elapsed time: 13.977224826812744
Epoch 1/10.. Train loss: 0.219.. Elapsed time: 13.941975355148315
Epoch 1/10.. Train loss: 0.242.. Elapsed time: 13.936140060424805
Epoch 1/10.. Train loss: 0.244.. Elapsed time: 13.942122459411621
Epoch 1/10.. Train loss: 0.216.. Elapsed time: 13.960899114608765
Epoch 1/10.. Train loss: 0.186.. Elapsed time: 13.956881523132324
Epoch 1/10.. Train loss: 0.241.. Elapsed time: 13.944581985473633
Epoch 1/10.. Train loss: 0.203.. Elapsed time: 13.934357404708862
Epoch 1/10.. Train loss: 0.189.. Elapsed time: 13.938358306884766
Epoch 1/10.. Train loss: 0.181.. Elapsed time: 13.944468021392822
Epoch 1/10.. Train loss: 0.186.. Elapsed time: 13.946297407150269
Epoch 1/10.. Train loss: 0.164.. Elapsed time: 13.940366744995117
Epoch 1/10.. Train loss: 0.165.. Elapsed time: 13.938241720199585
Epoch 1/10.. Train loss: 0.176.. Elapsed time: 14.015569925308228
Epoch 1/10.. Train loss: 0.165.. Elapsed time: 14.019208669662476
Epoch 1/10.. Train loss: 0.175.. Elapsed time: 14.149503469467163
Epoch 1/10.. Train loss: 0.159.. Elapsed time: 14.128302097320557
Epoch 1/10.. Train loss: 0.155.. Elapsed time: 13.935027837753296
Epoch 1/10.. Train loss: 0.137.. Elapsed time: 13.937382221221924
Epoch 1/10.. Train loss: 0.127.. Elapsed time: 13.929635524749756
Epoch 1/10.. Train loss: 0.133.. Elapsed time: 13.935472011566162
Epoch 1/10.. Train loss: 0.152.. Elapsed time: 13.922808647155762
Epoch 1/10.. Validation loss: 0.032.. Validation accuracy: 0.239

I won't paste more than this since it's long and takes a while to run but by the end of the 3rd epoch, the loss on the Colab model is still bouncing around 0.5 whereas locally it reaches 0.02.
If anyone could help me resolve this issue it would be greatly appreciated.

Output:
I fixed this problem by unzipping the training data to Google Drive and reading the files from there instead of using the Colab command to unzip the folder to my workspace directly. I have absolutely no idea why this was causing the problem; a quick visual inspection at the images and their corresponding tensors looks fine, but I can't go through each of the 6,000 or so images to check every one.
If anyone knows why this was causing a problem, please let me know!


Instruction: How to get confusion matrix in pytorch?

Input:
There are 15 classes. I have two tensors, one has the target classes and other has the predicted classes.
How can I get a confusion matrix?

Output:
You can try using the sklearn package, given you have your target class y_test and predicted class y_pred.
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, y_pred))


Instruction: Which PyTorch modules are affected by model.eval() and model.train()?

Input:
The model.eval() method modifies certain modules (layers) which are required to behave differently during training and inference. Some examples are listed in the docs:

This has [an] effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Is there an exhaustive list of which modules are affected?

Output:
In addition to info provided by @iacob:

Base class
Module
Criteria

RNNBase
RNN   LSTM  GRU
dropout > 0 (default: 0)

Transformer layers
Transformer  TransformerEncoder  TransformerDecoder
dropout > 0 (Transformer default: 0.1)

Lazy variants
LazyBatchNorm  currently nightly  merged PR
track_running_stats=True


Instruction: Add column and row to multidimensional torch.tensor (kind of wrap-up or padding)

Input:
The tensor should be updated with additional row-zeros (bottom) and column-zeros (on the right side).
My solution will be provided below. Is there any better (actually simpler) one?
input: («ones» are just for clarification - figures might be different, because in my case there is a tensor exactly the same size but with real values in it)
tensor([[[[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]],

         [[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]],

         [[1., 1., 1.],
          [1., 1., 1.],
          [1., 1., 1.]]]])

output:
tensor([[[[1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [0., 0., 0., 0.]],

         [[1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [0., 0., 0., 0.]],

         [[1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [1., 1., 1., 0.],
          [0., 0., 0., 0.]]]])

possible solution:
x1 = torch.ones(1, 3, 3, 3)

z2 = torch.cat((torch.cat((x1[0, 0, :], torch.zeros(1, 3)), 0), torch.zeros(4, 1)), 1)
z3 = torch.cat((torch.cat((x1[0, 1, :], torch.zeros(1, 3)), 0), torch.zeros(4, 1)), 1)
z4 = torch.cat((torch.cat((x1[0, 2, :], torch.zeros(1, 3)), 0), torch.zeros(4, 1)), 1)

output_t = torch.zeros(1, 3, 4, 4)

output_t[0, 0, :] = z2
output_t[0, 1, :] = z3
output_t[0, 2, :] = z4

output_t

Output:
You can do this with pytorch's torch.nn.ConstantPad?d functions.
from torch import nn

x1 = torch.ones(1, 3, 3, 3)

pad_value = 0
pad_func = nn.ConstantPad1d((0, 1, 0, 1), pad_value)

output_t = pad_func(x1)

You could also exchange nn.ConstantPad1d with nn.ConstantPad2d or nn.ConstantPad3d. All did what you want with the same settings.
Then there is also numpy's np.pad.
x1 = torch.ones(1, 3, 3, 3)

pad_value = 0

output_n = np.pad(x1.numpy(), (0, 0), (0, 0), (0, 1), (0, 1)), "constant", constant_values=pad_value)
output_t = torch.from_numpy(output_n)


Instruction: Cross entropy IndexError Dimension out of range

Input:
I'm trying to train a GAN in some images, I followed the tutorial on pytorch's page and got to the following code, but when the crossentropy function is applyed during the training it returns the error below the code:
import random
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as torch_dataset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

seed = 1
print("Random Seed: ", seed)
random.seed(seed)
torch.manual_seed(seed)
images_folder_path = "./images/"

batch_size = 128
image_size = 256
n_channels = 1
z_vector = 100
n_features_generator = 64
n_features_discriminator = 64
num_epochs = 5
lr = 0.0002
beta1 = 0.5

dataset = torch_dataset.ImageFolder(
    root=images_folder_path, transform=transforms.Compose(
        [
            transforms.Grayscale(num_output_channels=1),
            transforms.Resize(image_size),
            transforms.CenterCrop(image_size),
            transforms.ToTensor(),
            transforms.Normalize(0.5, 0.5)
         ]
    )
)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0)

device = torch.device("cuda:0" if (torch.cuda.is_available()) else "cpu")

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            nn.ConvTranspose2d(z_vector, n_features_generator * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(n_features_generator * 8),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 8, n_features_generator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 4),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 4, n_features_generator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 2),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 2, n_features_generator, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator, n_channels, 4, 2, 1, bias=False),
            nn.Tanh()
        )

    def forward(self, inputs):
        return self.main(inputs)

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(n_channels, n_features_discriminator, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator, n_features_discriminator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 2),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 2, n_features_discriminator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 4),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 4, n_features_discriminator * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, inputs):
        return self.main(inputs)

netG = Generator().to(device)
if device.type == 'cuda':
    netG = nn.DataParallel(netG)
netG.apply(weights_init)
print(netG)

netD = Discriminator().to(device)
if device.type == 'cuda':
    netD = nn.DataParallel(netD)
netD.apply(weights_init)
print(netD)

criterion = nn.CrossEntropyLoss()

fixed_noise = torch.randn(64, z_vector, 1, 1, device=device)

real_label = 1.
fake_label = 0.

optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
for epoch in range(num_epochs):
    for i, data in enumerate(dataloader, 0):
        netD.zero_grad()
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        output = netD(real_cpu).view(-1)
# ----------------------------------------------------------------------------------
        errD_real = criterion(output, label) # ERROR HAPPENS HERE
# ----------------------------------------------------------------------------------
        errD_real.backward()
        D_x = output.mean().item()

        noise = torch.randn(b_size, z_vector, 1, 1, device=device)
        fake = netG(noise)
        label.fill_(fake_label)
        output = netD(fake.detach()).view(-1)
        errD_fake = criterion(output, label)
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        optimizerD.step()

        netG.zero_grad()
        label.fill_(real_label)
        output = netD(fake).view(-1)
        errG = criterion(output, label)
        errG.backward()
        D_G_z2 = output.mean().item()
        optimizerG.step()

        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        G_losses.append(errG.item())
        D_losses.append(errD.item())

        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

Error:
Traceback (most recent call last):
File "G:/Pastas Estruturadas/Conhecimento/CEFET/IA/SpectroGAN/dcgan.py", line 137, in
errD_real = criterion(output, label)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\loss.py", line 948, in forward
ignore_index=self.ignore_index, reduction=self.reduction)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\functional.py", line 2422, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\functional.py", line 1591, in log_softmax
ret = input.log_softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
Process finished with exit code 1

Output:
Your model's output is not consistent with your criterion.
If you want to keep the model and change the criterion:
Use BCELoss instead of CrossEntropyLoss. Note: You will need to cast your labels to float before passing them in. Also consider removing the Sigmoid() from the model and using BCEWithLogitsLoss.
If you want to keep the criterion and change the model:
CrossEntropyLoss expects the shape (..., num_classes). So for your 2 class case (real & fake), you will have to predict 2 values for each image in the batch which means you will need to alter the output channels of the last layer in your model. It also expects the raw logits, so you should remove the Sigmoid().


Instruction: How to include a OneHot in an ONNX coming from PyTorch

Input:
I'm using PyTorch to train neural-net and output them into ONNX. I use these models in a Vespa index, which loads ONNXs through TensorRT. I need one-hot-encoding for some features but this is really hard to achieve within the Vespa framework.
Is it possible to embed a one-hot-encoding for some given features inside my ONNX net (e.g. before the network's representation) ?
If so, how should I achieve this based on a PyTorch model ?
I already noticed two things:

ONNX format includes the OneHot operator : see ONNX doc
PyTorch built-in ONNX exporting system not not support OneHot operator : see torch.onnx doc

EDIT 2021/03/11:
Here is my workflow:

training learning-to-rank models via PyTorch
exporting them as ONNX
importing these ONNX into my Vespa index in order to rank any query's results thanks to the ONNX model. Under the hood, Vespa uses TensorRT for inference (so I use Vespa's ONNX model evaluation)

Output:
So, according to my testing, PyTorch does support one-hot encoding export to ONNX. With the following model:
#! /usr/bin/env python3

import torch
import torch.onnx
import torch.nn.functional as F

class MyModel(torch.nn.Module):
    def __init__(self, classes=5):
        super(MyModel, self).__init__()
        self._classes = classes
        self.linear = torch.nn.Linear(in_features=self._classes, out_features=1)
        self.logistic = torch.nn.Sigmoid()

    def forward(self, input):
        one_hot = F.one_hot(input, num_classes=self._classes).float()
        return self.logistic(self.linear(one_hot))

def main():
    model = MyModel()

    # training omitted

    data = torch.tensor([0, 4, 2])
    torch.onnx.export(model, data, "test.onnx",
        input_names=["input"], output_names=["output"])

    result = model.forward(data)
    print(result)

if __name__ == "__main__":
    main()

This model doesn't do any training, just takes a vector of indices in, one-hot encodes them using PyTorch's one_hot and sends that to the simple NN layer. The weights are randomly initialised, and the output here for me was:
tensor([[0.5749],
        [0.5081],
        [0.5581]], grad_fn=<SigmoidBackward>)

This model is exported to ONNX to the "test.onnx" file. Testing this model using ONNX Runtime (which is what Vespa uses in the backend, not TensorRT):
In [1]: import onnxruntime as ort
In [2]: m = ort.InferenceSession("test.onnx")
In [3]: m.run(input_feed={"input":[0,4,2]}, output_names=["output"])
Out[3]:
[array([[0.57486993],
        [0.5081395 ],
        [0.5580716 ]], dtype=float32)]

Which is the same output as given from PyTorch with the same input. So PyTorch does export the OneHot ONNX operator. This was for PyTorch 1.7.1.
If the input to the one-hot encoding is indexed in Vespa as integers, you can then just use these directly as inputs.


Instruction: AttributeError: 'dict' object has no attribute 'step'

Input:
How to solve this problem?
  File "train.py", line 98, in validation_epoch_end
    self.scheduler.step(avg_loss)
AttributeError: 'dict' object has no attribute 'step'

My source code:
import os
import ast
import torch
import torchaudio
import torch.nn as nn
from torch.nn import functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from pytorch_lightning.core.lightning import LightningModule
from pytorch_lightning import Trainer
from argparse import ArgumentParser
from model import SpeechRecognition
from dataset import Data, collate_fn_padd
from pytorch_lightning.callbacks import ModelCheckpoint

nodes = int(1)
gpus = int(1)
data_workers = int(0)
train_file  = 'path/to/your/train.json'
valid_file = 'path/to/your/test.json'
valid_every = int(1000)
save_model_path = 'path/where/you/want/to/save/your/model'
logdir = 'path/where/you/want/to/save/your/logs'
epochs = int(10)
batch_size = int(64)
pct_start = float(0.3)
div_factor = int(100)
dparams_override = str("{}")
hparams_override = str("{}")
load_model_from = None
resume_from_checkpoint = None

class SpeechModule(LightningModule):
    def __init__(self, model, args):
        super(SpeechModule, self).__init__()
        self.model = model
        self.criterion = nn.CTCLoss(blank=28, zero_infinity=True)
        self.args = args

    def forward(self, x, hidden):
        return self.model(x, hidden)

    def configure_optimizers(self):
        self.optimizer = optim.AdamW(self.model.parameters(), self.args.learning_rate)
        lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(
            self.optimizer, mode='min',
            factor=0.50, patience=6
        )
        self.scheduler = {
            'scheduler': lr_scheduler,
            'reduce_on_plateau': True,
            'monitor': 'val_checkpoint_on'
        }

        return [self.optimizer], [self.scheduler]

    def step(self, batch):
        spectrograms, labels, input_lengths, label_lengths = batch
        bs = spectrograms.shape[0]
        hidden = self.model._init_hidden(bs)
        hn, c0 = hidden[0].to(self.device), hidden[1].to(self.device)
        output, _ = self(spectrograms, (hn, c0))
        output = F.log_softmax(output, dim=2)
        loss = self.criterion(output, labels, input_lengths, label_lengths)
        return loss

    def training_step(self, batch, batch_idx):
        loss = self.step(batch)
        logs = {'loss': loss, 'lr': self.optimizer.param_groups[0]['lr'] }
        return {'loss': loss, 'log': logs}

    def train_dataloader(self):
        d_params = Data.parameters
        d_params.update(dparams_override)
        train_dataset = Data(json_path=self.train_file, **d_params)
        return DataLoader(dataset=train_dataset,
                            batch_size= batch_size,
                            num_workers= data_workers,
                            pin_memory=True,
                            collate_fn=collate_fn_padd)

    def validation_step(self, batch, batch_idx):
        loss = self.step(batch)
        return {'val_loss': loss}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        self.scheduler.step(avg_loss)
        tensorboard_logs = {'val_loss': avg_loss}
        return {'val_loss': avg_loss, 'log': tensorboard_logs}

    def val_dataloader(self):
        d_params = Data.parameters
        d_params.update(dparams_override)
        test_dataset = Data(json_path=valid_file, **d_params, valid=True)
        return DataLoader(dataset=test_dataset,
                            batch_size= batch_size,
                            num_workers= data_workers,
                            collate_fn=collate_fn_padd,
                            pin_memory=True)

def checkpoint_callback():
    return ModelCheckpoint(
        filepath= save_model_path,
        save_top_k=True,
        verbose=True,
        monitor='val_loss',
        mode='min',
        prefix=''
    )

def main(args):
    h_params = SpeechRecognition.hyper_parameters
    h_params.update(hparams_override)
    model = SpeechRecognition(**h_params)
    if load_model_from:
        speech_module = SpeechModule.load_from_checkpoint(load_model_from, model=model)
    else:
        speech_module = SpeechModule(model, args)
    trainer = Trainer()

    trainer = Trainer(
        max_epochs= epochs, gpus= gpus,
        num_nodes= nodes, distributed_backend=None,
        gradient_clip_val=1.0,
        val_check_interval= valid_every,
        checkpoint_callback=checkpoint_callback,
        resume_from_checkpoint= resume_from_checkpoint
    )
    trainer.fit(speech_module)

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument('--learning_rate', default=1e-3, type=float, help='learning rate')

    args = parser.parse_args()
    hparams_override = ast.literal_eval(hparams_override)
    dparams_override = ast.literal_eval(dparams_override)
    if save_model_path:
        if not os.path.isdir(os.path.dirname(save_model_path)):
            raise Exception("the directory for path {} does not exist".format(save_model_path))

    main(args)

Output:
self.scheduler.step(avg_loss)

should be
self.step(avg_loss)


Instruction: Confusion about the output channels of convolution neural network

Input:
I'm confused about the multi-channel scenario in convolution neural network.
Say I have a 10(width) * 5(height) * 6(channels) image, and I feed it into a default 2-D convolution layer with stride=1 and padding=0 and expect the output to be 8(width) * 3(height) * 16(channels).
I know the size of the kernel is 3(width) * 3(height), but I don't know how many kernels are there exactly, and how the are applied to the input data to give the final 16 channels.
Someone can help me please.

Output:
A 2D convolution layer contains one kernel per input channel, per output channel. So in your case, this will be 6*16=96 kernels. For 3x3 kernels, this corresponds to 3*3*96 = 864 parameters.
>>> import torch

>>> conv = torch.nn.Conv2d(6, 16, (3, 3))
>>> torch.numel(conv.weight)
864

For one image, one kernel per input channel is first applied. In your case, this results in 6 features maps, that are summed together (+ a possible bias) to form 1 of the output channel. Then, you repeat this 15 times to form the 15 other output channels.


Instruction: How does the output of the Discriminator of a Convolutional Generative Adversarial Network work, can it have a Fully Connected Layer?

Input:
I'm building a DCGAN, and I am having a problem with the shape of the output, it is not matching the shape of the labels when I try calculating the BCELoss.
To generate the discriminator output, do I have to use convolutions all the way down or can I add a Linear layer at some point to match the shape I want?
I mean, do I have to reduce the shape by adding more convolutional layers or can I add a fully connected one? I thought it should have a fully connected layer, but on every tutorial I checked the discriminator had no fully connected layer.
import random
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as torch_dataset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

seed = 1
print("Random Seed: ", seed)
random.seed(seed)
torch.manual_seed(seed)
images_folder_path = "./spectrograms/"

batch_size = 1
image_size = 256
n_channels = 1
z_vector = 100
n_features_generator = 32
n_features_discriminator = 32
num_epochs = 5
lr = 0.0002
beta1 = 0.5

dataset = torch_dataset.ImageFolder(
    root=images_folder_path, transform=transforms.Compose(
        [
            transforms.Grayscale(num_output_channels=1),
            transforms.Resize(image_size),
            transforms.CenterCrop(image_size),
            transforms.ToTensor(),
            transforms.Normalize(0.5, 0.5)
         ]
    )
)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0)

device = torch.device("cuda:0" if (torch.cuda.is_available()) else "cpu")

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            nn.ConvTranspose2d(z_vector, n_features_generator * 8, 4, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 8),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 8, n_features_generator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 4),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 4, n_features_generator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 2),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 2, n_features_generator, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator, n_channels, 4, 2, 1, bias=False),
            nn.Tanh()
        )

    def forward(self, inputs):
        return self.main(inputs)

# Convolutional Layer Output Shape = [(W−K+2P)/S]+1
# W is the input volume
# K is the Kernel size
# P is the padding
# S is the stride
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(n_channels, n_features_discriminator, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator, n_features_discriminator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 2),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 2, n_features_discriminator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 4),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 4, n_features_discriminator * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 8, 1, 4, 1, bias=False),
        )

    def forward(self, inputs):
        return self.main(inputs)

netG = Generator().to(device)
if device.type == 'cuda':
    netG = nn.DataParallel(netG)
netG.apply(weights_init)
print(netG)

netD = Discriminator().to(device)
if device.type == 'cuda':
    netD = nn.DataParallel(netD)
netD.apply(weights_init)
print(netD)

criterion = nn.BCEWithLogitsLoss()

fixed_noise = torch.randn(64, z_vector, 1, 1, device=device)

real_label = 1.
fake_label = 0.

optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
for epoch in range(num_epochs):
    for i, data in enumerate(dataloader, 0):
        netD.zero_grad()
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        output = netD(real_cpu)
        print(output.shape)
        print(label.shape)
        output = output.view(-1)
        errD_real = criterion(output, label)
        errD_real.backward()
        D_x = output.mean().item()

        noise = torch.randn(b_size, z_vector, 1, 1, device=device)
        fake = netG(noise)
        label.fill_(fake_label)
        output = netD(fake.detach()).view(-1)
        errD_fake = criterion(output, label)
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        optimizerD.step()

        netG.zero_grad()
        label.fill_(real_label)
        output = netD(fake).view(-1)
        errG = criterion(output, label)
        errG.backward()
        D_G_z2 = output.mean().item()
        optimizerG.step()

        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        G_losses.append(errG.item())
        D_losses.append(errD.item())

        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

The error I'm getting:
Traceback (most recent call last):
  File "G:/Pastas Estruturadas/Conhecimento/CEFET/IA/SpectroGAN/dcgan.py", line 140, in <module>
    errD_real = criterion(output, label)
  File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\loss.py", line 631, in forward
    reduction=self.reduction)
  File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\functional.py", line 2538, in binary_cross_entropy_with_logits
    raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
ValueError: Target size (torch.Size([1])) must be the same as input size (torch.Size([169]))

The shape of output: torch.Size([1, 1, 13, 13]), and shape of label: torch.Size([1]).

Output:
The DCGAN described a concrete architecture where Conv layers were used for the downsampling of the feature maps. If you carefully design your Conv layers, you can do without a Linear layer but that does not mean that it will not work when you use a Linear layer to downsample (especially as the very last layer). The DCGAN paper just found out it worked better to use Conv layers instead of Linear to downsample.
If you want to maintain this architecture, you can change the kernel size or padding or stride to give you exactly a single value in the last layer. Refer to the Pytorch documentation on Conv layers to see what the output size should be, given an input size


Instruction: Appling sliding window to torch.tensor and adjusting tensor initial size

Input:
Looking for a simpler way of torch.tensor modification. Probably there is a way to apply Unfold to initial tensor directly.
input:
tensor([[0., 1., 2.],
        [3., 4., 5.],
        [6., 7., 8.]])

output:
tensor([[0., 1., 3., 4.],
        [1., 2., 4., 5.],
        [3., 4., 6., 7.],
        [4., 5., 7., 8.]])

possible solution:
import torch

t = torch.linspace(0., 8., steps=9)

t1 = t.reshape(3,3) # starting point

t2 = torch.flatten(t1)

t3 = t2.reshape(1, 1, 1, -1) # unfold works with 4D only

unfold = torch.nn.Unfold(kernel_size=(1, 5), dilation=1)

t4 = unfold(t3)

indices = torch.tensor([0, 1, 3, 4]) # deleting 3d (or middle) row and 3d (middle) column

t5 = torch.index_select(torch.index_select(t4.squeeze(), 0, indices), 1, indices)

t5

Output:
You can use unfold, but in a simpler manner:
import torch
import torch.nn.functional as nnf

t1 = torch.arange(9.).reshape(3,3)  # initial tensor
out = nnf.unfold(t1[None, None, ...], kernel_size=2, padding=0)  # that's it. done.


Instruction: How to make a list of pytorch tensors in python?

Input:
How do I make a list of tensors in Pytorch.
The list should look like this:
mylist = [tensor1, tensor2, tensor3]

where all the tensors have different shapes

Output:
You can instantiate each tensor using pytorch inline or append to a list in a loop.
Inline:
mylist = [torch.rand(2), torch.rand(5), torch.rand(1)]

In a loop:
mylist = [torch.rand(i) for i in range(1, 5)]

To create a custom tensor, use torch.tensor([[1., -1.], [1., -1.]]) for example.
https://pytorch.org/docs/stable/tensors.html


Instruction: Fixing Python Dependencies The Right Way

Input:
I'm just getting my first Python environment setup. All have gone well and it seems to be GPU enabled and all that good stuff.
However, I have one issue and no idea how to fix. After getting the correct install command for torch it informed of this issue:
Installing collected packages: torch
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.8.2 requires torch==1.7.1, but you have torch 1.8.0+cu111 which is incompatible.
Successfully installed torch-1.8.0+cu111

As far as I can tell torchvision 0.8.2 is the latest version.
The environment seems happy at the moment as all these commands return expected things:
import torch
print(torch.__version__)
torch.cuda.get_device_name(0)

I've seen some people talking about "patching requirements files" or updating dependencies. But I'm not sure of the best way to tackle this.

Output:
You can lock the version of a package in a requirements file. This file has the appropriate values.
requirements.txt:
torch==1.7.1
torchvision==0.8.2

The packages are installed via pip like so:
pip install -r requirements.txt

You may have other dependencies for this project. In that case, you can generate a requirements.txt file with pip as well:
pip freeze > requirements.txt

Check out the documentation on managing dependencies with pip