Instruction: fastapi could not find model defintion when run with uvicorn

Input:
I want to host a pytorch model in a fastapi backend. When I run the code with python it is working fine. the depickled model can use the defined class. When the same file is started with uvicorn it cannot find the class definition.
Sourcecode looks like this:
import uvicorn
import json
from typing import List
from fastapi import Body, FastAPI
from fastapi.encoders import jsonable_encoder
import requests
from pydantic import BaseModel

#from model_ii import Model_II_b

import dill as pickle
import torch as T
import sys

app = FastAPI()
current_model = 'model_v2b_c2_small_ep15.pkl'
verbose_model = False  # for model v2

class Model_II_b(T.nn.Module):
[...]
@app.post('/function')
def API_call(req_json: dict = Body(...)):
    try:
        # load model...
        model = pickle.load(open('models/' + current_model, 'rb'))
        result = model.dosomething_with(req_json)

        return result

    except Exception as e:
        raise e
        return {"error" : str(e)}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

When I run this with python main.py it is working fine and I am gettings results. When I run it with  uvicorn main:app and send a request I get the following error:
AttributeError: Can't get attribute 'Model_II_b' on <module '__mp_main__' from '/opt/webapp/env/bin/uvicorn'>

both should be using the same python env as I use the uvicorn from within the env.
I hope someone has an idea what is wrong with my setup or code.
Update Stacktrace:
(model_2) root@machinelearning-01:/opt/apps# uvicorn main:app --env-file /opt/apps/env/pyvenv.cfg --reload
INFO:     Loading environment from '/opt/apps/env/pyvenv.cfg'
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [164777] using statreload
INFO:     Started server process [164779]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:33872 - "POST /ml/v2/predict HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/apps/env/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/apps/env/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/applications.py", line 183, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/applications.py", line 102, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 550, in __call__
    await route.handle(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/routing.py", line 197, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./main.py", line 155, in API_call
    raise e
  File "./main.py", line 129, in API_call
    model = pickle.load(open('models/' + current_model, 'rb'))
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 473, in load
    obj = StockUnpickler.load(self)
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 463, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'Model_II_b' on <module '__mp_main__' from '/opt/apps/env/bin/uvicorn'>
enter code here

Output:
With the help from @lsabi I found the solution here https://stackoverflow.com/a/51397373/13947506
With the custom unpickler my problem was solved:
class CustomUnpickler(pickle.Unpickler):

    def find_class(self, module, name):
        if name == 'Model_II_b':
            from model_ii_b import Model_II_b
            return Model_II_b
        return super().find_class(module, name)

current_model = 'model_v2b_c2_small_ep24.pkl'

model = CustomUnpickler(open('models/' + current_model, 'rb')).load()


Instruction: Broadcasting element wise multiplication in pytorch

Input:
I have a tensor in pytorch with size torch.Size([1443747, 128]). Let's name it tensor A. In this tensor, 128 represents a batch size. I have another 1D tensor with size torch.Size([1443747]). Let's call it B. I want to do element wise multiplication of B with A, such that B is multiplied with all 128 columns of tensor A (obviously in an element wise manner). In other words, I want to broadcast the element wise multiplication along dimension=1.
How can I achieve this in pytorch?
It I didn't have a batch size involved in the tensor A (batch size = 1), then normal * operator would do the multiplication easily. A*B then would have generated resultant tensor of size torch.Size([1443747]). However, I don't understand why pytorch is not broadcasting the tensor multiplication along dimension 1? Is there any way to do this?
What I want is, B should be multiplied with all 128 columns of A in an element wise manner. So, the resultant tensors' size would be torch.Size([1443747, 128]).

Output:
The dimensions should match, it should work if you transpose A or unsqueeze B:
C = A.transpose(1,0) * B    # shape: [128, 1443747]

or
C = A * B.unsqueeze(dim=1)  # shape: [1443747, 128]

Note that the shapes of the two solutions are different.


Instruction: calculate accuracy for each class using CNN and pytorch

Input:
I Can calculate accuracy after each epoch using this code . But, I want to calculate the accuracy for each class at the end . how can i do that?
I have two folders train and val . each folder has 7 folders of 7 different classes. the train folder is used for training .otherwise val folder is used for testing
  def train_model(model, criterion, optimizer, lr_scheduler, num_epochs=25):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                mode='train'
                optimizer = lr_scheduler(optimizer, epoch)
                model.train()  # Set model to training mode
            else:
                model.eval()
                mode='val'

            running_loss = 0.0
            running_corrects = 0

            counter=0
            # Iterate over data.
            for data in dset_loaders[phase]:
                inputs, labels = data
                print(inputs.size())
                # wrap them in Variable
                if use_gpu:
                    try:
                        inputs, labels = Variable(inputs.float().cuda()),
                        Variable(labels.long().cuda())
                    except:
                        print(inputs,labels)
                else:
                    inputs, labels = Variable(inputs), Variable(labels)

                # Set gradient to zero to delete history of computations in previous epoch. Track operations so that differentiation can be done automatically.
                optimizer.zero_grad()
                outputs = model(inputs)
                _, preds = torch.max(outputs.data, 1)

                loss = criterion(outputs, labels)
                # print('loss done')
                # Just so that you can keep track that something's happening and don't feel like the program isn't running.
                # if counter%10==0:
                #     print("Reached iteration ",counter)
                counter+=1

                # backward + optimize only if in training phase
                if phase == 'train':
                    # print('loss backward')
                    loss.backward()
                    # print('done loss backward')
                    optimizer.step()
                    # print('done optim')
                # print evaluation statistics
                try:
                    # running_loss += loss.data[0]
                    running_loss += loss.item()
                    # print(labels.data)
                    # print(preds)
                    running_corrects += torch.sum(preds == labels.data)
                    # print('running correct =',running_corrects)
                except:
                    print('unexpected error, could not calculate loss or do a sum.')
            print('trying epoch loss')
            epoch_loss = running_loss / dset_sizes[phase]
            epoch_acc = running_corrects.item() / float(dset_sizes[phase])
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val':
                if USE_TENSORBOARD:
                    foo.add_scalar_value('epoch_loss',epoch_loss,step=epoch)
                    foo.add_scalar_value('epoch_acc',epoch_acc,step=epoch)
                if epoch_acc > best_acc:
                    best_acc = epoch_acc
                    best_model = copy.deepcopy(model)
                    print('new best accuracy = ',best_acc)
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    print('returning and looping back')
    return best_model

def exp_lr_scheduler(optimizer, epoch, init_lr=BASE_LR, lr_decay_epoch=EPOCH_DECAY):
    """Decay learning rate by a factor of DECAY_WEIGHT every lr_decay_epoch epochs."""
    lr = init_lr * (DECAY_WEIGHT**(epoch // lr_decay_epoch))

    if epoch % lr_decay_epoch == 0:
        print('LR is set to {}'.format(lr))

    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

    return optimizer

Output:
Calculating overall accuracy is rather straight forward:
outputs = model(inputs)
_, preds = torch.max(outputs.data, 1)

acc_all = (preds == labels).float().mean()

To calculate it per class requires a few more lines of code:
acc = [0 for c in list_of_classes]
for c in list_of_classes:
    acc[c] = ((preds == labels) * (labels == c)).float() / (max(labels == c).sum(), 1))


Instruction: Why do I get the error 'The ordinal 242 could not be located in the dynamic link library' when trying to import torch?

Input:
I am currently encountering some issues when trying to import PyTorch on my computer. I am working from my own local Windows laptop (which doesn't have any GPU) and installed Python 3.6 from python.org. I don't have the Anaconda distribution and usually install any new package by opening a Windows Command Prompt and using this command: pip install package.
Usually that's enough for me to be able to use the package right away (either via a Jupyter Notebook or by writing and running jupya .py script in Sublime Text). But PyTorch seem to be a little less straight forward.
I followed the installation instruction copied below (from the PyTorch website):

pip
No CUDA
To install PyTorch via pip, and do not have a CUDA-capable system or do not require CUDA, in the above selector, choose OS: Windows,
Package: Pip and CUDA: None. Then, run the command that is presented
to you.

I opened my Windows Command Prompt and simply ran this command:
pip install torch==1.5.1+cpu torchvision==0.6.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

The installation was successful and no error showed up during the process.
Now, when I open a Jupyter Notebook and run the command: import torch
I get the following error:
python.exe - Ordinal Not Found

The ordinal 242 could not be located in the dynamic link library

c:\users\bdour\appdata\local\programs\python\python36\lib\site-packages\torch\lib\torch_cpu.dll

I checked and the torch_cpu.dll file DOES exist at the path mentioned in the error.
I tried to understand what that error meant but could not find much help. It seems like many people have issues with PyTorch, often due to some issues with their conda environment, but I am not using any environment. I am just trying to import and run the package locally.
And like I mentioned above, I usually encounter no problem with any other library when simply using a pip install command.
Does anyone know where that error is coming from and how to fix it?
Thank you in advance for your time and help.

Output:
I was actually able to find a solution on my own and thought I would post it here in case someone else struggled with the same error.
I found this helpful link and the suggested solution worked for me: https://kittaiwong.wordpress.com/2019/11/04/how-to-fix-the-ordinal-242-could-not-be-located-in-the-dynamic-link-library-mkl_intel_thread-dll/
In short, the problem seems to originate from a file named libiomp5md.dll present in the C:\Windows\System32 folder that is simply incompatible with numpy.
To fix it, I just looked for the file by copying and pasting it in Windows search, Opening file location (which should be C:\Windows\System32) and then renaming it to: libiomp5md.dll.bak
Now I can import torch without getting any error.
Hope that will help others who struggled with getting torch to run!


Instruction: OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index']

Input: When I load the BERT pretrained model online I get this error OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory uncased_L-12_H-768_A-12 or 'from_tf' set to False what should I do?

Output:
Here is what I found. Go to the following link, and click the circled to download, rename it to pytorch_model.bin, and drop it to the directory of biobert-nli,  then the issue is resolved. Didn't figure out how to clone from the link.
https://huggingface.co/gsarti/biobert-nli/tree/main


Instruction: Best way to convert a tensor from a condensed representation

Input:
I have a Tensor that is in a condensed format representing a sparse 3-D matrix. I need to convert it to a normal matrix (the one that it is actually representing).
So, in my case, each row of any 2-D slice of my matrix can only contain one non-zero element. As data, then, I have for each of these rows, the value, and the index where it appears. For example, the tensor
inp = torch.tensor([[ 1,  2],
 [ 3,  4],
 [-1,  0],
 [45,  1]])

represents a 4x5 matrix (first dimension comes from the first dimension of the tensor, second comes from the metadata) A, where A[0][2] = 1, A[1][4] = 3, A[2][0] = -1, A[3][1] = 45.
This is just one 2-D slice of my Matrix, and I have a variable number of these.
I was able to do this for a 2-D slice as described above in the following way using sparse_coo_tensor:
>>> torch.sparse_coo_tensor(torch.stack([torch.arange(0, 4), inp.t()[1]]), inp.t()[0], [4,5]).to_dense()
tensor([[ 0,  0,  1,  0,  0],
        [ 0,  0,  0,  0,  3],
        [-1,  0,  0,  0,  0],
        [ 0, 45,  0,  0,  0]])

Is this the best way to accomplish this? Is there a simpler, more readable alternative?
How do I extend this to a 3-D matrix without looping?
For a 3-D matrix, you can imagine the input to be something like
inp_list = torch.stack([inp, inp, inp, inp])
and the desired output would be the above output stacked 4 times.
I feel like I should be able to do something if I create an index array correctly, but I cannot think of a way to do this without using some kind of looping.

Output:
OK, after a lot of experiments with different types of indexing, I got this to work. Turns out, the answer was in Advanced Indexing. Unfortunately, PyTorch documentation doesn't go in the details of Advanced Indexing. Here is a link for it in the Numpy documentation.
For the problem described above, this command did the trick:
>>> k_lst = torch.zeros([4,4,5])
>>> k_lst[torch.arange(4).unsqueeze(1), torch.arange(4), inp_list[:,:,1]] = inp_list[:,:,0].float()
>>> k_lst
tensor([[[ 0.,  0.,  1.,  0.,  0.],
     [ 0.,  0.,  0.,  0.,  3.],
     [-1.,  0.,  0.,  0.,  0.],
     [ 0., 45.,  0.,  0.,  0.]],
    [[ 0.,  0.,  1.,  0.,  0.],
     [ 0.,  0.,  0.,  0.,  3.],
     [-1.,  0.,  0.,  0.,  0.],
     [ 0., 45.,  0.,  0.,  0.]],
    [[ 0.,  0.,  1.,  0.,  0.],
     [ 0.,  0.,  0.,  0.,  3.],
     [-1.,  0.,  0.,  0.,  0.],
     [ 0., 45.,  0.,  0.,  0.]],
    [[ 0.,  0.,  1.,  0.,  0.],
     [ 0.,  0.,  0.,  0.,  3.],
     [-1.,  0.,  0.,  0.,  0.],
     [ 0., 45.,  0.,  0.,  0.]]])

Which is exactly what I wanted.
I learned quite a few things searching for this, and I want to share this for anyone who stumbles on this question. So, why does this work? The answer lies in the way Broadcasting works. If you look at the shapes of the different index tensors involved, you'd see that they are (of necessity) broadcastable.
>>> torch.arange(4).unsqueeze(1).shape, torch.arange(4).shape, inp_list[:,:,1].shape
(torch.Size([4, 1]), torch.Size([4]), torch.Size([4, 4]))

Clearly, to access an element of a 3-D tensor such as k_lst here, we need 3 indexes - one for each dimension. If you give 3 tensors of same shapes to the [] operator, it can get a bunch of legal indexes by matching corresponding elements from the 3 tensors.
If the 3 tensors are of different shapes, but broadcastable (as is the case here), it copies the relevant rows/columns of the lacking tensors the requisite number of times to get tensors with the same shapes.
Ultimately, in my case, if we go into how the different values got assigned, this would be equivalent to doing
k_lst[0,0,inp_list[0,0,1]] = inp_list[0,0,0].float()
k_lst[0,1,inp_list[0,1,1]] = inp_list[0,1,0].float()
k_lst[0,2,inp_list[0,2,1]] = inp_list[0,2,0].float()
k_lst[0,3,inp_list[0,3,1]] = inp_list[0,3,0].float()
k_lst[1,0,inp_list[1,0,1]] = inp_list[1,0,0].float()
k_lst[1,1,inp_list[1,1,1]] = inp_list[1,1,0].float()
.
.
.
k_lst[3,3,inp_list[3,3,1]] = inp_list[3,3,0].float()

This format reminds me of torch.Tensor.scatter(), but if it can be used to solve this problem, I haven't figured out how yet.


Instruction: pytorch "trying to backward through the graph a second time" error with chracter level RNN

Input:
I am training a character level GRU with pytorch, while dividing the text into batches of a certain chunk length.
This is the trainning loop:
for e in range(self.epochs):
  self.model.train()
  h = self.get_init_state(self.batch_size)

  for batch_num in range(self.num_batch_runs):
    batch = self.generate_batch(batch_num).to(device)

    inp_batch = batch[:-1,:]
    tar_batch = batch[1:,:]

    self.model.zero_grad()
    loss = 0

    for i in range(inp_batch.shape[0]):
      out, h = self.model(inp_batch[i:i+1,:],h)

      loss += loss_fn(out[0],tar_batch[i].view(-1))

    loss.backward()

    nn.utils.clip_grad_norm_(self.model.parameters(), 5.0)

    optimizer.step()

    if not (batch_num % 5):
      print("epoch: {}, loss: {}".format(e,loss.data.item()/inp_batch.shape[0]))

Still, I am getting this error after the first batch:
Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Thanks in advance..

Output:
I found the answer myself, the hidden state of the GRU was still attached to the last batch run, so it had to be detached using
h.detach_()


Instruction: How to slice 3D torch tensor into 2D slices

Input:
I'm working with 3D CT medical data, and I'm trying to slice it into 2D slices that I can input into a UNet model.
I've loaded the data into a torch dataloader, and each iteration currently produces a 4D tensor:
for batch_index, batch_samples in enumerate(train_loader):
    data, target = batch_samples['image'].float().cuda(), batch_samples['label'].float().cuda()
    print(data.size())
torch.Size([1, 333, 512, 512])
torch.Size([1, 356, 512, 512])

such as this one. I want to iterate over the 333 slices, and then the 356 slices, such that the model receives torch sizes [1, 1, 512, 512] each time.
I was hoping something like :
for x in (data[:,x,:,:]):

would work but it says I need to define x first. How can I iterate over a specific dimension in a torch tensor?

Output:
Simply specify the dimension:
for i in range(data.shape[1]):  # dim=1
    x = data[:, i, :, :]
    # [...]

If you really need that extra dimension, simply add .unsqueeze():
d = 1
for i in range(data.shape[d]):         # dim=1
    x = data[:, i, :, :].unsqueeze(d)  # same dim=1
    # [...]


Instruction: How to create a custom data loader in Pytorch?

Input:
I have a file containing paths to images I would like to load into Pytorch, while utilizing the built-in dataloader features (multiprocess loading pipeline, data augmentations, and so on).
def create_links():
    data_dir = "/myfolder"

    full_path_list = []
    assert os.path.isdir(data_dir)
    for _, _, filenames in os.walk(data_dir):
        for filename in filenames:
            full_path_list.append(os.path.join(data_dir, filename))

    with open(config.data.links_file, 'w+') as links_file:
        for full_path in full_path_list:
            links_file.write(f"{full_path}\n")

def read_links_file_to_list():
    config = ConfigProvider.config()
    links_file_path = config.data.links_file
    if not os.path.isfile(links_file_path):
        raise RuntimeError("did you forget to create a file with links to images? Try using 'create_links()'")
    with open(links_file_path, 'r') as links_file:
        return links_file.readlines()

So I have a list of files (or a generator, or whatever works), file_list = read_links_file_to_list().
How can I build a Pytorch dataloader around it, and how would I use it?

Output:
What you want is a Custom Dataset. The __getitem__ method is where you would apply transforms such as data-augmentation etc. To give you an idea of what it looks like in practice you can take a look at this Custom Dataset I wrote the other day:
class GTSR43Dataset(Dataset):
    """German Traffic Sign Recognition dataset."""
    def __init__(self, root_dir, train_file, transform=None):
        self.root_dir = root_dir
        self.train_file_path = train_file
        self.label_df = pd.read_csv(os.path.join(self.root_dir, self.train_file_path))
        self.transform = transform
        self.classes = list(self.label_df['ClassId'].unique())

    def __getitem__(self, idx):
        """Return (image, target) after resize and preprocessing."""
        img = os.path.join(self.root_dir, self.label_df.iloc[idx, 7])

        X = Image.open(img)
        y = self.class_to_index(self.label_df.iloc[idx, 6])

        if self.transform:
            X = self.transform(X)

        return X, y

    def class_to_index(self, class_name):
        """Returns the index of a given class."""
        return self.classes.index(class_name)

    def index_to_class(self, class_index):
        """Returns the class of a given index."""
        return self.classes[class_index]

    def get_class_count(self):
        """Return a list of label occurences"""
        cls_count = dict(self.label_df.ClassId.value_counts())
#         cls_percent = list(map(lambda x: (1 - x / sum(cls_count)), cls_count))
        return cls_count

    def __len__(self):
        """Returns the length of the dataset."""
        return len(self.label_df)


Instruction: Pytorch: explain torch.argmax

Input:
Hello I have the following code:
import torch
x = torch.zeros(1,8,4,576) # create a 4 dimensional tensor
x[0,4,2,333] = 1.0 # put on 1 on a random spot

# I want to find the index of the highest value (0,4,2,333)
print(x.argmax()) # this should return the index

This returns

tensor(10701)

How does this 10701 make sense?
How do I get the actual indices 0,4,2,333 ?

Output:
The data in the 4-dimensional array is stored linearly in memory, and argmax() returns the corresponding index of this flat representation.
Numpy has a function for unraveling the index (converting from the flat array index to the corresponding multi-dimensional indices).
import numpy as np
np.unravel_index(10701, (1,8,4,576))


Instruction: ValueError: only one element tensors can be converted to Python scalars

Input:
I'm following this tutorial.
I'm at the last part where we combine the models in a regression.
I'm coding this in jupyter as follows:
import shutil
import os
import time
from datetime import datetime
import argparse
import pandas
import numpy as np
from tqdm import tqdm
from tqdm import tqdm_notebook

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
from torchsample.transforms import RandomRotate, RandomTranslate, RandomFlip, ToTensor, Compose, RandomAffine
from torchvision import transforms
import torch.nn.functional as F
from tensorboardX import SummaryWriter

import dataloader
from dataloader import MRDataset
import model

from sklearn import metrics

def extract_predictions(task, plane, train=True):
    assert task in ['acl', 'meniscus', 'abnormal']
    assert plane in ['axial', 'coronal', 'sagittal']

    models = os.listdir('models/')

    model_name = list(filter(lambda name: task in name and plane in name, models))[0]
    model_path = f'models/{model_name}'

    mrnet = torch.load(model_path)
    _ = mrnet.eval()

    train_dataset = MRDataset('data/',
                              task,
                              plane,
                              transform=None,
                              train=train,
                              )

    train_loader = torch.utils.data.DataLoader(train_dataset,
                                               batch_size=1,
                                               shuffle=False,
                                               num_workers=10,
                                               drop_last=False)
    predictions = []
    labels = []
    with torch.no_grad():
        for image, label, _ in tqdm_notebook(train_loader):
            logit = mrnet(image.cuda())
            prediction = torch.sigmoid(logit)
            predictions.append(prediction.item())
            labels.append(label.item())

    return predictions, labels

task = 'acl'
results = {}

for plane in ['axial', 'coronal', 'sagittal']:
    predictions, labels = extract_predictions(task, plane)
    results['labels'] = labels
    results[plane] = predictions

X = np.zeros((len(predictions), 3))
X[:, 0] = results['axial']
X[:, 1] = results['coronal']
X[:, 2] = results['sagittal']

y = np.array(labels)

logreg = LogisticRegression(solver='lbfgs')
logreg.fit(X, y)

task = 'acl'
results_val = {}

for plane in ['axial', 'coronal', 'sagittal']:
    predictions, labels = extract_predictions(task, plane, train=False)
    results_val['labels'] = labels
    results_val[plane] = predictions

y_pred = logreg.predict_proba(X_val)[:, 1]
metrics.roc_auc_score(y_val, y_pred)

However I get this error:
ValueError                                Traceback (most recent call last)
<ipython-input-2-979acb314bc5> in <module>
      3
      4 for plane in ['axial', 'coronal', 'sagittal']:
----> 5     predictions, labels = extract_predictions(task, plane)
      6     results['labels'] = labels
      7     results[plane] = predictions

<ipython-input-1-647731b6b5c8> in extract_predictions(task, plane, train)
     54             logit = mrnet(image.cuda())
     55             prediction = torch.sigmoid(logit)
---> 56             predictions.append(prediction.item())
     57             labels.append(label.item())
     58

ValueError: only one element tensors can be converted to Python scalars

Here's the MRDataset code in case:
class MRDataset(data.Dataset):
    def __init__(self, root_dir, task, plane, train=True, transform=None, weights=None):
        super().__init__()
        self.task = task
        self.plane = plane
        self.root_dir = root_dir
        self.train = train
        if self.train:
            self.folder_path = self.root_dir + 'train/{0}/'.format(plane)
            self.records = pd.read_csv(
                self.root_dir + 'train-{0}.csv'.format(task), header=None, names=['id', 'label'])
        else:
            transform = None
            self.folder_path = self.root_dir + 'valid/{0}/'.format(plane)
            self.records = pd.read_csv(
                self.root_dir + 'valid-{0}.csv'.format(task), header=None, names=['id', 'label'])

        self.records['id'] = self.records['id'].map(
            lambda i: '0' * (4 - len(str(i))) + str(i))
        self.paths = [self.folder_path + filename +
                      '.npy' for filename in self.records['id'].tolist()]
        self.labels = self.records['label'].tolist()

        self.transform = transform
        if weights is None:
            pos = np.sum(self.labels)
            neg = len(self.labels) - pos
            self.weights = torch.FloatTensor([1, neg / pos])
        else:
            self.weights = torch.FloatTensor(weights)

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, index):
        array = np.load(self.paths[index])
        label = self.labels[index]
        if label == 1:
            label = torch.FloatTensor([[0, 1]])
        elif label == 0:
            label = torch.FloatTensor([[1, 0]])

        if self.transform:
            array = self.transform(array)
        else:
            array = np.stack((array,)*3, axis=1)
            array = torch.FloatTensor(array)

        # if label.item() == 1:
        #     weight = np.array([self.weights[1]])
        #     weight = torch.FloatTensor(weight)
        # else:
        #     weight = np.array([self.weights[0]])
        #     weight = torch.FloatTensor(weight)

        return array, label, self.weights

I've only trained my models using 1 and 2 epochs for each plane of the MRI instead of 35 as in the tutorial, not sure if that has anything to do with it. Other than that I'm stranded as to what this could be? I also removed normalize=False in the options for train_dataset as it kept giving me an error and I read that it could be removed, but I'm not so sure?

Output: Only a tensor that contains a single value can be converted to a scalar with item(), try printing the contents of prediction, I imagine this is a vector of probabilities indicating which label is most likely. Using argmax on prediction will give you your actual predicted label (assuming your labels are 0-n).


Instruction: PyTorch AttributeError: 'UNet3D' object has no attribute 'size'

Input:
I am making an image segmentation transfer learning project using Pytorch. I am using the weights of this pre-trained model and class UNet3D.
https://github.com/MrGiovanni/ModelsGenesis
When I run the following codes I get this error at the line which MSELoss is called: "AttributeError: 'DataParallel' object has no attribute 'size' ".
When I delete the first line I get a similar error: "AttributeError: 'UNet3D' object has no attribute 'size'
"
How can I convert DataParallel or UNet3D class to an object which MSELoss can use? I do not need DataParallel for now. I need to run the UNet3D() class for transfer learning.
model = nn.DataParallel(model, device_ids = [i for i in range(torch.cuda.device_count())])
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), conf.lr, momentum=0.9, weight_decay=0.0, nesterov=False)
scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
initial_epoch=10
for epoch in range(initial_epoch, conf.nb_epoch):
    scheduler.step(epoch)
    model.train()
    for batch_ndx, (x,y) in enumerate(train_loader):
        x, y = x.float().to(device), y.float().to(device)
        pred = model
        loss = criterion(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-46-20d1943b3498> in <module>
     25         x, y = x.float().to(device), y.float().to(device)
     26         pred = model
---> 27         loss = criterion(pred, y)
     28         optimizer.zero_grad()
     29         loss.backward()

/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    430
    431     def forward(self, input, target):
--> 432         return F.mse_loss(input, target, reduction=self.reduction)
    433
    434

/opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2528                 mse_loss, tens_ops, input, target, size_average=size_average, reduce=reduce,
   2529                 reduction=reduction)
-> 2530     if not (target.size() == input.size()):
   2531         warnings.warn("Using a target size ({}) that is different to the input size ({}). "
   2532                       "This will likely lead to incorrect results due to broadcasting. "

/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595
    596     def __setattr__(self, name, value):

AttributeError: 'UNet3D' object has no attribute 'size'

Output:
You have a typo on this line:
pred = model

should be
pred = model(x)

model is nn.Module object which describes the network.  x, y, pred are (supposed to be) torch tensors.
Aside from this particular case, I think it would be good to think about how to solve this type of problems in general.
You saw an error (exception) on a certain line.  Is the problem there, or earlier?  Can you isolate the problem?
For example, if you print out the arguments you're passing to criterion(pred, y) just before the call, do they look right?  (they don't)
What happens if you create a couple of tensors of the right shape just before the call and pass them instead?  (works fine)
What is the error really saying?  "AttributeError: 'UNet3D' object has no attribute 'size'" - well, of course it's not supposed to have a size, but why is the code trying to access it's size?  Actually, why is the code even able to access that object on that line?  (since the model is not supposed to be passed to the criterion function - right?)
Maybe useful further reading: https://ericlippert.com/2014/03/05/how-to-debug-small-programs/


Instruction: Training works but prediction produces constant values (cnn with pytorch)

Input:
I have a model trying to predict the class of image: cat or dog. I receive 95% accuracy in training. However when I try to predict a single image, I am stuck with almost constant output every time I run the model. There are some non-constant values, but they mostly look like catastrophic failure.
I read similar topics from forums but that hasn't helped, as it appears there is no particular solution for this problem...
I have tried the following:

Changing epochs 5 to 15,20,30...
Changing lr = 0.001 to 0.01, 0.0001...
I implemented with both dropout regularization model and batch
normalization model...
I changed testing pictures...
Changing last activation layer from softmax to torch.sigmoid...
Reducing batch size from 100 to 30, 75...
Trying with a batch, which results with normal acc, loss and
predictions.
My dataset is scaled which is mentioned in forums as solution.
My optim is Adam which is mentioned in forums as solution.
Loading dataset with torch.data.DataLoader...
Sampling randomly...
I saved and load the model, in case there are problems with that.
However, I already checked that state_dict's are different...
I re-prepared data which resulted the constant value to change
otherwise (dog to cat), somehow? Idk if that's a coincidence though.

Infos:

Dataset :
https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip

Here is all my code with predictions in Jupyter Notebook, feel free to investigate. I am really tired of this solution. Any help is  highly appreciated!

https://github.com/yusuftengriverdi/neural_networks/blob/master/CNN_Last.ipynb

Similar topics around the web:

https://discuss.pytorch.org/t/rnn-predicting-a-constant-output/40397/5
https://discuss.pytorch.org/t/cnn-does-not-predict-properly-does-not-converge-as-expected/43567
https://discuss.pytorch.org/t/making-a-prediction-with-a-trained-model/2193
https://datascience.stackexchange.com/questions/46779/predict-gives-the-same-output-value-for-every-image-keras
https://github.com/keras-team/keras/issues/6447
PyTorch model prediction fail for single item
Having trouble with CNN prediction

Output:
If something works in training but fails during prediction, the most likely cause is you're not preprocessing the data the same way.
I had a look at the notebook (huge amount of code, in future please condense this to just the relevant parts here).  At a glance - this is your prediction code which doesn't work as expected:
img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))
plt.imshow(img, cmap='gray')
x = torch.Tensor([i for i in img]).view(-1, 50, 50)
y= torch.Tensor([0,1]).to(device)
test_x = x.view(-1, 1, 50, 50)
test_x = test_x.to(device)
net.eval()
#with torch.no_grad():
yhat.append(net(test_x))

But during training you're using a dataloader
testloader = DataLoader(v_dataset, batch_size = BATCH_SIZE, sampler= test_sampler)
...

test_dt = next(iter(testloader))
X, y = test_dt[0].view(-1, 1, 50, 50), test_dt[1]
val_acc, val_loss = fwd_pass(X.view(-1, 1, 50, 50).to(device), y.to(device))

which  works (since your test/validation accuracy goes up to a good level).
Figure out what the dataloader code path does which the other code path doesn't do, and you'll have the solution.  Eg, load the same image in both ways and compare - same dimensions?  data average / standard deviation the same?  etc
For a shortcut - just use a dataloader to make predictions as well.  P.S. Yes, it is okay to create a dataloader for just one image.


Instruction: error in BatchNorm2d in pytorch CNN model

Input:
my database has grayscale images of size 128 * 128* 1   each  with batch size =10
i am using cnn model but I got this error in BatchNorm2d
expected 4D input (got 2D input)
I posted the way which i used to transform my image (gray scale - tensor - normalize ) and divide it to batches
data_transforms = {
    'train': transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.Resize(128),
        transforms.CenterCrop(128),
        transforms.ToTensor(),
        transforms.Normalize([0.5], [0.5])
    ]),
    'val': transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.Resize(128),
        transforms.CenterCrop(128),
        transforms.ToTensor(),
        transforms.Normalize([0.5], [0.5])
    ]),
}

data_dir = '/content/drive/My Drive/Colab Notebooks/pytorch'
dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x])
         for x in ['train', 'val']}
dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=10,
                                               shuffle=True, num_workers=25)
                for x in ['train', 'val']}
dset_sizes = {x: len(dsets[x]) for x in ['train', 'val']}
dset_classes = dsets['train'].classes

I used this model
class HeartNet(nn.Module):
    def __init__(self, num_classes=7):

        super(HeartNet, self).__init__()

        self.features = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(64),
            nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(128),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(kernel_size=2, stride=2)
            )

        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(16*16*256, 2048),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(2048),
            nn.Linear(2048, num_classes)
            )

        nn.init.xavier_uniform_(self.classifier[1].weight)
        nn.init.xavier_uniform_(self.classifier[4].weight)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 16 * 16 * 256)
        x = self.classifier(x)
        return x

How can I solve this problem ?

Output:
You have a problem with the batch norm layer inside your self.classifier sub network: While your self.features sub network is fully convolutional and required BatchNorm2d, the self.classifier sub network is a fully-connected multi-layer perceptron (MLP) network and is 1D in nature. Note the how the forward function removes the spatial  dimensions from the feature map x before feeding it to the classifier.
Try replacing the BatchNorm2d in self.classifier with BatchNorm1d.


Instruction: How to specify the input dimension of pytorch nn.Linear?

Input:
For example, I have defined  a model shown below:
class Net(nn.module):
   def __init__():
      self.conv11 = nn.Conv2d(in_channel,out1_channel,3)
      self.conv12 = nn.Conv2d(...)
      self.conv13 = nn.Conv2d(...)
      self.conv14 = nn.Conv2d(...)
      ...
      #Here is the point
      flat = nn.Flatten()
      #I don't want to compute the size of data after flatten, but I need a linear layer.

      fc_out = nn.Linear(???,out_dim)

The problem is the Linear layer, I don't want to compute the size of the input to the Linear layer, but the defining model needs to specify it. How can I solve this problem?

Output:
The way I do it is to put in some arbitrary value and let the model throw an error. You will be able to see the number of input features in the error description.
There are other ways to do it as well. You can compute the size by hand and write a comment next to each nn.Conv2d layer depicting the layer output. Before you use the nn.Flatten(), you will have the output, simply multiply all the dimensions except the bacthsize. The resulting value is the number of input features for nn.Linear() layer.
If you don't want to do any of this, you can try torchlayers. A handy package that lets you define pytorch models like Keras.


Instruction: PyTorch gradients have different shape for CUDA and CPU

Input:
I’m dealing with a strange issue where the gradients after backward pass have different shapes depending on whether CUDA or CPU is used. The model used is relatively simple:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.relu1 = nn.ReLU()
        self.relu2 = nn.ReLU()
        self.relu3 = nn.ReLU()
        self.relu4 = nn.ReLU()

    def forward(self, x):
        x = self.pool1(self.relu1(self.conv1(x)))
        x = self.pool2(self.relu2(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = self.relu3(self.fc1(x))
        x = self.relu4(self.fc2(x))
        x = self.fc3(x)
        return x

The input tensor has shape (1, 3, 32, 32), and the relevant section of code is as follows, with the method generate_gradients being of particular importance:
class VanillaBackprop():
    """
        Produces gradients generated with vanilla back propagation from the image
    """
    def __init__(self, model):
        self.model = model
        self.gradients = None
        # Put model in evaluation mode
        self.model.eval()
        # Hook the first layer to get the gradient
        self.hook_layers()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model.to(self.device)

    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients = grad_in[0]

        # Register hook to the first layer
        try:
            first_layer = list(self.model.features._modules.items())[0][1]
        except:
            first_layer = list(self.model._modules.items())[0][1]
        first_layer.register_backward_hook(hook_function)

    def generate_gradients(self, input_image, target_class):
        # Forward
        model_output = self.model(input_image.to(self.device))
        # Zero grads
        self.model.zero_grad()
        # Target for backprop
        one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
        one_hot_output[0][target_class] = 1
        # Backward pass
        model_output.backward(gradient=one_hot_output.to(self.device))
        # Convert Pytorch variable to numpy array
        gradients_as_arr = self.gradients.data.cpu().numpy()[0]
        return gradients_as_arr

When on CPU, self.gradients has shape (1, 3, 32, 32), while on CUDA it has shape (1, 6, 28, 28). How is that possible, and how do I fix this? Any help is much appreciated.

Output:
It looks like the issue stems from the register_backward_hook() function. As pointed out in the PyTorch forums:

You might want to double check the register_backward_hook() doc. But
it is known to be kind of broken at the moment and can have this
behavior.
I would recommend you use autograd.grad() for this though. That will
make it simpler than backward+access to the .grad field.

I, however, opted to use register_hook() instead of register_backward_hook() (as opposed to autograd.grad() as suggested), which seems to work as well:
class VanillaBackprop():
    """
        Produces gradients generated with vanilla back propagation from the image
    """
    def __init__(self, model):
        self.model = model
        self.gradients = None
        # Put model in evaluation mode
        self.model.eval()
        # Hook the first layer to get the gradient

    def hook_input(self, input_tensor):
        def hook_function(grad_in):
            self.gradients = grad_in
        input_tensor.register_hook(hook_function)

    def generate_gradients(self, input_image, target_class):
        # Register input hook
        self.hook_input(input_image)
        # Forward
        model_output = self.model(input_image)
        # Zero grads
        self.model.zero_grad()
        # Target for backprop
        device = next(self.model.parameters()).device
        one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
        one_hot_output[0][target_class] = 1
        one_hot_output = one_hot_output.to(device)
        # Backward pass
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        model_output.backward(gradient=one_hot_output.to(device))
        # Convert Pytorch variable to numpy array
        # [0] to get rid of the first channel (1,3,224,224)
        gradients_as_arr = self.gradients.data.cpu().numpy()[0]
        return gradients_as_arr


Instruction: Split and extract values from a PyTorch tensor according to given dimensions

Input: I have a tensor Aof sizetorch.Size([32, 32, 3, 3]) and I want to split it and extract a tensor B of size torch.Size([16, 16, 3, 3]) from it. The tensor can be 1d or 4d and split has to be according to the given new tensor dimensions. I have been able to generate the target dimensions but I'm unable to split and extract the values from the source tensor. I ave tried torch.narrow but it takes only 3 arguments and I need 4 in many cases. torch.split takes dim as an int due to which tensor is split along one dimension only. But I want to split it along multiple dimensions.

Output:
You have multiple options:

use .split multiple times
use .narrow multiple times
use slicing

e.g.:
t = torch.rand(32, 32, 3, 3)

t0, t1 = t.split((16, 16), 0)
print(t0.shape, t1.shape)
>>> torch.Size([16, 32, 3, 3]) torch.Size([16, 32, 3, 3])

t00, t01 = t0.split((16, 16), 1)
print(t00.shape, t01.shape)
>>> torch.Size([16, 16, 3, 3]) torch.Size([16, 16, 3, 3])

t00_alt, t01_alt = t[:16, :16, :, :], t[16:, 16:, :, :]
print(t00_alt.shape, t01_alt.shape)
>>> torch.Size([16, 16, 3, 3]) torch.Size([16, 16, 3, 3])


Instruction: PyTorch - RuntimeError: Error(s) in loading state_dict for VGG:

Input:
I've trained a model using PyTorch and saved a state dict file. I have loaded the pre-trained model using the code below. I am getting an error message regarding RuntimeError: Error(s) in loading state_dict for VGG:
RuntimeError: Error(s) in loading state_dict for VGG:
    Missing key(s) in state_dict: "features.0.weight", "features.0.bias", "features.2.weight", "features.2.bias", "features.5.weight", "features.5.bias", "features.7.weight", "features.7.bias", "features.10.weight", "features.10.bias", "features.12.weight", "features.12.bias", "features.14.weight", "features.14.bias", "features.17.weight", "features.17.bias", "features.19.weight", "features.19.bias", "features.21.weight", "features.21.bias", "features.24.weight", "features.24.bias", "features.26.weight", "features.26.bias", "features.28.weight", "features.28.bias", "classifier.0.weight", "classifier.0.bias", "classifier.3.weight", "classifier.3.bias", "classifier.6.weight", "classifier.6.bias".
    Unexpected key(s) in state_dict: "state_dict", "optimizer_state_dict", "globalStep", "train_paths", "test_paths".

I am following instruction available at this site: https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-across-devices
Many Thanks
import argparse
import datetime
import glob
import os
import random
import shutil
import time
from os.path import join

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from torchvision.transforms import ToTensor
from tqdm import tqdm
import torch.optim as optim

from convnet3 import Convnet
from dataset2 import CellsDataset

from convnet3 import Convnet
from VGG import VGG
from dataset2 import CellsDataset
from torchvision import models
from Conv import Conv2d

parser = argparse.ArgumentParser('Predicting hits from pixels')
parser.add_argument('name',type=str,help='Name of experiment')
parser.add_argument('data_dir',type=str,help='Path to data directory containing images and gt.csv')
parser.add_argument('--weight_decay',type=float,default=0.0,help='Weight decay coefficient (something like 10^-5)')
parser.add_argument('--lr',type=float,default=0.0001,help='Learning rate')
args = parser.parse_args()

metadata = pd.read_csv(join(args.data_dir,'gt.csv'))
metadata.set_index('filename', inplace=True)

# create datasets:

dataset = CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
dataset = DataLoader(dataset,num_workers=4,pin_memory=True)
model_path = '/Users/nubstech/Documents/GitHub/CellCountingDirectCount/VGG_model_V1/checkpoints/checkpoint.pth'

class VGG(nn.Module):
    def __init__(self, pretrained=True):
        super(VGG, self).__init__()
        vgg = models.vgg16(pretrained=pretrained)
        # if pretrained:
        vgg.load_state_dict(torch.load(model_path))
        features = list(vgg.features.children())
        self.features4 = nn.Sequential(*features[0:23])

        self.de_pred = nn.Sequential(Conv2d(512, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))

    def forward(self, x):
        x = self.features4(x)
        x = self.de_pred(x)

        return x

model=VGG()
#model.load_state_dict(torch.load(model_path),strict=False)
model.eval()

#optimizer = torch.optim.Adam(model.parameters(),lr=args.lr,weight_decay=args.weight_decay)

for images, paths in tqdm(dataset):

    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float()

    # code to print training data to a csv file
    #filename=CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
    print(preds)
    paths_test = np.array([paths])
    names_preds = np.hstack(paths)
    print(names_preds)
    df=pd.DataFrame({'Image_Name':names_preds, 'Target':targets.detach(), 'Prediction':preds.detach()})
    print(df)
    # save image name, targets, and predictions
    df.to_csv(r'model.csv', index=False, mode='a')

Code for saving the state dict
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)

Output:
The problem is that what is being saved is not the same as what is expected to be loaded.  The code is trying to load only a state_dict; it is saving quite a bit more than that - looks like a state_dict inside another dict with additional info.  The load method doesn't have any logic to look inside the dict.
This should work:
import torch, torchvision.models
model = torchvision.models.vgg16()
path = 'test.pth'
torch.save(model.state_dict(), path) # nothing else here
model.load_state_dict(torch.load(path))


Instruction: Where are those numbers coming from in pytorch neural networks?

Input:
I'm newbie in Pytorch (python), i was just scrolling through their official tutorial and i found this simple neural network architecture. Everything is clear, but those numbers, last three fully connected layers, where are they coming from?
self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension and 16 from last conv layer
self.fc2 = nn.Linear(120, 84) # but here, 120? 84? why? is this just random or there is some logic behind it?
self.fc3 = nn.Linear(84, 10)

full code - https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py

Output:
self.fc1 = nn.Linear(16 * 6 * 6, 120)
self.fc2 = nn.Linear(120, 84) # you can use any number instead of 120 play with this number and see which gives you best result.
self.fc3 = nn.Linear(84, 10)

120 is number of units in first layer after conv layer , 84 in second layer and 10 in last which probably is dimension of your output layer ie. 10 possible type of classification.
You are correct the dimension of second and third layer is not fixed and you can try different value of num of units and choose one which gives you the best result. You can play with it but you can also look at some of the best performing models and follow the structure they use.


Instruction: Pytorch, INPUT (normal tensor) and WEIGHT (cuda tensor) mismatch

Input:
DISCLAIMER I know, this question has already asked multiple times, but i tried their solutions, none of them worked for me, so after all those effort, i can't find anything else and eventually i have to ask again.
I'm doing image classification with cnns (PYTORCH), i wan't to train it on GPU (nvidia gpu, compatible with cuda/cuda installed), i successfully managed to put net on it, but the problem is with data.
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    print("Running on the GPU")
    print("Available GPU", torch.cuda.device_count())

Running on the GPU
Available GPU 1

net = Net()
net.to(device)

for epoch in range(2):
    running_loss=0.0
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data[0].to(device), data[1].to(device) # putting data on gpu
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

dataiter = iter(testloader)
images, labels = dataiter.next()

print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

out = net(images)

ERROR
----------------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-81-76c52eabb174> in <module>
----> 1 out = net(images)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-57-c74f8361a10b> in forward(self, x)
     11
     12     def forward(self, x):
---> 13         x = self.pool(F.relu(self.conv1(x)))
     14         x = self.pool(F.relu(self.conv2(x)))
     15         x = x.view(-1, 16*5*5)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/conv.py in forward(self, input)
    351
    352     def forward(self, input):
--> 353         return self._conv_forward(input, self.weight)
    354
    355 class Conv3d(_ConvNd):

~/anaconda3/envs/home/lib/python3.8/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    347                             weight, self.bias, self.stride,
    348                             _pair(0), self.dilation, self.groups)
--> 349         return F.conv2d(input, weight, self.bias, self.stride,
    350                         self.padding, self.dilation, self.groups)
    351

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

inputs.is_cuda

True

and same for labels.
What I've tried:
https://stackoverflow.com/q/59013109/13287412
https://github.com/sksq96/pytorch-summary/issues/57
https://blog.csdn.net/qq_27261889/article/details/86575033
https://blog.csdn.net/public669/article/details/96510293

but nothing worked so far...

Output:
Your images tensor is located on the CPU while your net is located on the GPU. Even when evaluating you want to make sure that your input tensors and model are located on the same device otherwise you will get tensor data type errors.
out = net(images.to(device))


Instruction: Should I transpose features or weights in Neural network?

Input:
I am learning Neural network.
Here is the complete piece of code:
https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%201%20-%20Tensors%20in%20PyTorch%20(Exercises).ipynb
When I transpose features, I get the following output:
import torch
def activation(x):
    return 1/(1+torch.exp(-x))

### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))

product = features.t() * weights + bias
output = activation(product.sum())

tensor(0.9897)

However, if I transpose weights, I get a different output:
weights_prime = weights.view(5,1)
prod = torch.mm(features, weights_prime) + bias
y_hat = activation(prod.sum())

tensor(0.1595)

Why does this happen?

Update
I took a look at the solution:
https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%201%20-%20Tensors%20in%20PyTorch%20(Solution).ipynb
And I saw this:
y = activation((features * weights).sum() + bias)
why can a matrix features(1,5) multiply another matrix weights(1,5) without transposing weights first?
Update 2
After read several posts, I realized that
matrixA * matrixB is different from torch.mm(matrixA,matrixB) and torch.matmul(matrixA,matrixB).
Could someone confirm my three understandings between?

So the * means element-wise multiplication, whereas torch.mm() and torch.matmul() are matrix-wise multiplication.

differences between torch.mm() and torch.matmul(): mm() is used specifically for 2 dimensions matrix, whereas matmul() can be used for more complicated cases.

In Neutral Network for this Udacity coding exercise mentioned in my above link, it needs element-wise multiplication.

Update 3
Just to bring in the Video screenshot for someone who has the same confusion:

And here is the video link: https://www.youtube.com/watch?time_continue=98&v=6Z7WntXays8&feature=emb_logo

Output:
Looking at https://pytorch.org/docs/master/generated/torch.nn.Linear.html
The typical linear (fully connected) layer in torch uses input features of shape  (N,∗,in_features) and weights of shape (out_features,in_features) to produce an output of shape (N,*,out_features). Here N is the batch size, and * is any number of other dimensions (may be none).
The implementation is:
output = input.matmul(weight.t())

So, the answer is that neither of your formulas is correct according to convention; the standard formula is the one above.
You may use a non-standard shape since you're implementing things from scratch; as long as it's consistent it may work, but I don't recommend it for learning.  It's unclear what 1 and 5 is in your code, but presumably you want 5 input features and one output feature, with a batch size of 1 as well.  In which case the standard shapes should be input = torch.randn((1, 5)) for batch size=1 and in_features=5, and weights = torch.randn((5, 1)) for in_features=5 and out_features=1.
There is no reason why weights should ever be the same shape as features; thus weights = torch.randn_like(features) doesn't make sense.
Lastly, for your actual questions:
"Should I transpose features or weights in Neural network?" - in torch convention, you should transpose weights, but use matmul with the features first.  Other frameworks may have a different convention; as long as in_features dimension of the weights is multiplied by the num_features dimension of the input, it would work.
"Why does this happen?" - these are two completely different calculations; there is no reason to think they would produce the same result.
"So the * means element-wise multiplication, whereas torch.mm() and torch.matmul() are matrix-wise multiplication." - Yes; mm is matrix-matrix only, matmul is vector-matrix or matrix-matrix, including batched versions of same - check the docs for everything matmul can do (which is kinda a lot).
"differences between torch.mm() and torch.matmul(): mm() is used specifically for 2 dimensions matrix, whereas matmul() can be used for more complicated cases." - Yes; the big difference is that matmul can broadcast.  Use it when you specifically intend that; use mm to prevent unintentional broadcasting.
"In Neutral Network for this Udacity coding exercise mentioned in my above link, it needs element-wise multiplication." - I doubt it; it's probably an error in the Udacity code.  This bit of code weights = torch.randn_like(features) looks like an error in any case; the dimensions of weights have a meaning different from the dimensions of features.


Instruction: Load pickle saved GPU tensor with CPU?

Input:
I save the last hidden layer of Bert for my following process using pickle on GPU.
# output is the last hidden layer of bert, transformed on GPU
with open(filename, 'wb') as f:
    pk.dump(output, f)

Is it possible to load it on my person laptop without GPU? I tried following code, but all failed.
# 1st try
with open(filename, 'rb') as f:
    torch.load(f, map_location='cpu')

# 2nd
torch.load(filename, map_location=torch.device('cpu'))

All get the following error
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Is it possible to load the file on my laptop?

Output:
If you using pytorch, you can save yourself some headache by saving the state_dict of the model instead of the model itself. The state_dict is an ordered dictionary that stores the weights of your neural network.
The saving routine:
import torch
model = MyFabulousPytorchModel()
torch.save(model.state_dict(), "best_model.pt")

Loading it requires you to initialize the model first:
import torch
device = 'cuda' if torch.cuda.is_available() else 'gpu'
model = MyFabulousPytorchModel()
model.load_state_dict(torch.load(PATH_TO_MODEL))
model.device(device)

There are many advantages of saving the state_dict instead of the object directly. One of them has to do with your issue: porting your model to a different environment isn't as painless as you were hoping for. Another advantage is that it is a lot easier to save checkpoints that allow you to resume training as if training had never stopped. All you have to do is save the state of the optimizer and the loss:
Saving checkpoint:
# somewhere in your training loop:
opt.zero_grad()
pred = model(x)
loss = loss_func(pred, target)

torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "loss":loss}, "checkpoing.pt")

I highly recommend checking the documentation for further information on how to save and load models using pytorch. It is quite a smooth process if you understand its inner workings. https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference
Hope that helps =)
Edit:
More directly, to solve your problem I recomend the following
1- On the computer you used to train the model:
import torch
model = torch.load("PATH_TO_MODEL")
torch.save(model.state_dict(), "PATH.pt")

2- On the other computer:
import torch
from FILE_WHERE_THE_MODEL_CLASS_IS_DEFINED import Model

model = Model() # initialize one instance of the model)
model.load_state_dict(torch.load("PATH.pt")


Instruction: PyTorch RuntimeError: CUDA out of memory. Tried to allocate 14.12 GiB

Input:
I have faced Cuda out of memory error for a simple fully connected layer model. I have tried torch.cuda.empty_cache() and gc.collect(). Also, I delete unnecessary variables by del and tried by reducing batch size. But the error does not resolved. Also, the error appears only for SUN datasets where 1440 test images are used to evaluate. But the code runs well for AWA2 datasets where no. of test images is 7913. I am using google colab here. I have used RTX 2060 too.
Here is the code snippet, where it gets error:
def euclidean_dist(x, y):
    # x: N x D
    # y: M x D
    torch.cuda.empty_cache()
    n = x.size(0)
    m = y.size(0)
    d = x.size(1)
    assert d == y.size(1)
    x = x.unsqueeze(1).expand(n, m, d)
    y = y.unsqueeze(0).expand(n, m, d)
    del n,m,d
    return torch.pow(x - y, 2).sum(2)

def compute_accuracy(test_att, test_visual, test_id, test_label):
    global s2v
    s2v.eval()
    with torch.no_grad():
        test_att = Variable(torch.from_numpy(test_att).float().to(device))
        test_visual = Variable(torch.from_numpy(test_visual).float().to(device))
        outpre = s2v(test_att, test_visual)
        del test_att, test_visual
        outpre = torch.argmax(torch.softmax(outpre, dim=1), dim=1)

    outpre = test_id[outpre.cpu().data.numpy()]

    #compute averaged per class accuracy
    test_label = np.squeeze(np.asarray(test_label))
    test_label = test_label.astype("float32")
    unique_labels = np.unique(test_label)
    acc = 0
    for l in unique_labels:
        idx = np.nonzero(test_label == l)[0]
        acc += accuracy_score(test_label[idx], outpre[idx])
    acc = acc / unique_labels.shape[0]
    return acc

The error is:
Traceback (most recent call last):   File "GBU_new_v2.py", line 234, in <module>
    acc_seen_gzsl = compute_accuracy(attribute, x_test_seen, np.arange(len(attribute)), test_label_seen)   File "GBU_new_v2.py", line 111, in compute_accuracy
    outpre = s2v(test_att, test_visual)   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)   File "GBU_new_v2.py", line 80, in forward
    a1 = euclidean_dist(feat, a1)   File "GBU_new_v2.py", line 62, in euclidean_dist
    return torch.pow(x - y, 2).sum(2)#.sqrt() # return: N x M RuntimeError: CUDA out of memory. Tried to allocate 14.12 GiB (GPU 0;
15.90 GiB total capacity; 14.19 GiB already allocated; 669.88 MiB free; 14.55 GiB reserved in total by PyTorch)

Output:
It seems like you have batches defined only for training, while during test you attempt to process the entire test set simultaneously.
You should split your test set into smaller "batches" and evaluate one batch at a time to combine all the batch scores at the end for one score for the model.


Instruction: Can I change volatile = False to torch.set_grad_enabled(True)??(in Pytorch)

Input:
I have old python codes, so I need to change some parts.
next_q_values.volatile=False

I have this code, and next_q_values is 'torch.Tensor'
when I run this code:

error occured: "volatile was removed and now has no effect use with
torch no_grad instead"

After search, I know that volatile = True is same as torch.no_grad(),
but I want to use volatile = False, So I can't use torch.no_grad().
Can I change volatile = False to torch.set_grad_enabled(True)?

Output:
If you have:
next_q_values.volatile = False

You can change it to:
with torch.no_grad():
    next_q_values
    ...
    # You do something with next_q_values here

Every operation on next_q_values should be within the scope of context manager.


Instruction: Pytorch MNIST code is returning IndexError

Input:
I've followed the Pytorch documentation and have made an extremely simple classifier for the MNIST dataset. Below is my code:
import numpy as np

import torch
import torchvision
from torchvision import transforms, datasets

import torch.nn as nn
import torch.nn.functional as F

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])
    ])

train = datasets.MNIST('', train=True, download=True, transform=transform)
test = datasets.MNIST('', train=False, download=True, transform=transform)
trainset = torch.utils.data.DataLoader(train, batch_size=1, shuffle=True)
testset = torch.utils.data.DataLoader(test, batch_size=1, shuffle=False)

class Classifier(nn.Module):
    def __init__(self, D_in, H, D_out):
        super(Classifier, self).__init__()
        self.linear_1 = torch.nn.Linear(D_in, H)
        self.linear_2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        x = self.linear_1(x).clamp(min=0)
        x = self.linear_2(x)
        return F.log_softmax(x, dim=1)

net = Classifier(28*28, 128, 10)
optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)

for epoch in range(3):
    running_loss = 0.0
    for X, label in iter(trainset):
        X = X.view(28*28, -1)

        optimizer.zero_grad()

        output = net(torch.flatten(X))
        loss = nn.CrossEntropyLoss(output, label)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1}] loss: {running_loss / 2000}')
            running_loss = 0.0
print("Finished training.")

torch.save(net.state_dict(), './classifier.pth')

For some reason, I'm getting the output
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

at the line: output = net(torch.flatten(X)
Thanks in advance for your help!

Output:
When flatten() you remove all dimensions including batch dimension!
Try:
output = net(x.view(x.shape[0], -1))


Instruction: Pytorch based Resnet18 achieves low accuracy on CIFAR100

Input:
I'm training a resnet18 on CIFAR100 dataset. After about 50 iterations the validation accuracy converged at about 34%. While the training accuracy reached almost 100%.
I doubt it's kinda overfitting, so i applied data augmentation like RandomHorizontalFlip and RandomRotation, which made the validation converge at about 40%.
I also tried decaying learning rate [0.1, 0.03, 0.01, 0.003, 0.001], decaying after each 50 iterations. Decaying learning rate seems not enhancing the performance.
Heard that Resnet on CIFAR100 may get 70%~80% accuracy. What trick else could I apply? Or is there anything wrong in my implementation? The same code on CIFAR10 can achieve about 80% accuracy.
My whole training and evaluation code is here below:
import torch
from torch import nn
from torch import optim
from torch.utils.data import DataLoader
from torchvision.models import resnet18
from torchvision.transforms import Compose, ToTensor, RandomHorizontalFlip, RandomRotation, Normalize
from torchvision.datasets import CIFAR10, CIFAR100
import os
from datetime import datetime
import matplotlib.pyplot as plt

def draw_loss_curve(histories, legends, save_dir):
    os.makedirs(save_dir, exist_ok=True)
    for key in histories[0][0].keys():
        if key != "epoch":
            plt.figure()
            plt.title(key)
            for history in histories:
                x = [h["epoch"] for h in history]
                y = [h[key] for h in history]
                # plt.ylim(ymin=0, ymax=3.0)
                plt.plot(x, y)
            plt.legend(legends)
            plt.savefig(os.path.join(save_dir, key + ".png"))

def cal_acc(out, label):
    batch_size = label.shape[0]
    pred = torch.argmax(out, dim=1)
    num_true = torch.nonzero(pred == label).shape[0]
    acc = num_true / batch_size
    return torch.tensor(acc)

class LrManager(optim.lr_scheduler.LambdaLR):
    def __init__(self, optimizer, lrs):
        def f(epoch):
            rate = 1
            for k in sorted(lrs.keys()):
                if epoch >= k:
                    rate = lrs[k]
                else:
                    break
            return rate
        super(LrManager, self).__init__(optimizer, f)

def main(cifar=100, epochs=250, batches_show=100):
    if torch.cuda.is_available():
        device = "cuda"
    else:
        device = "cpu"
        print("warning: CUDA is not available, using CPU instead")
    dataset_cls = CIFAR10 if cifar == 10 else CIFAR100
    dataset_train = dataset_cls(root=f"data/{dataset_cls.__name__}/", download=True, train=True,
                                transform=Compose([RandomHorizontalFlip(), RandomRotation(15), ToTensor(), Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))]))
    dataset_val = dataset_cls(root=f"data/{dataset_cls.__name__}/", download=True, train=False,
                              transform=Compose([ToTensor(), Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))]))
    loader_train = DataLoader(dataset_train, batch_size=128, shuffle=True)
    loader_val = DataLoader(dataset_val, batch_size=128, shuffle=True)
    model = resnet18(pretrained=False, num_classes=cifar).to(device)
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9, weight_decay=1e-5)
    lr_scheduler = LrManager(optimizer, {0: 1.0, 50: 0.3, 100: 0.1, 150: 0.03, 200: 0.01})
    criterion = nn.CrossEntropyLoss()

    history = []
    model.train()
    for epoch in range(epochs):
        print("-------------------  TRAINING  -------------------")
        loss_train = 0.0
        running_loss = 0.0
        acc_train = 0.0
        running_acc = 0.0
        for batch, data in enumerate(loader_train, 1):
            img, label = data[0].to(device), data[1].to(device)
            optimizer.zero_grad()
            pred = model(img)
            loss = criterion(pred, label)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            loss_train += loss.item()
            acc = cal_acc(pred, label)
            running_acc += acc.item()
            acc_train += acc.item()

            if batch % batches_show == 0:
                print(f"epoch: {epoch}, batch: {batch}, loss: {running_loss/batches_show:.4f}, acc: {running_acc/batches_show:.4f}")
                running_loss = 0.0
                running_acc = 0.0
        loss_train = loss_train / batch
        acc_train = acc_train / batch
        lr_scheduler.step()

        print("------------------- EVALUATING -------------------")
        with torch.no_grad():
            running_acc = 0.0
            for batch, data in enumerate(loader_val, 1):
                img, label = data[0].to(device), data[1].to(device)
                pred = model(img)
                acc = cal_acc(pred, label)
                running_acc += acc.item()
            acc_val = running_acc / batch
            print(f"epoch: {epoch}, acc_val: {acc_val:.4f}")

        history.append({"epoch": epoch, "loss_train": loss_train, "acc_train": acc_train, "acc_val": acc_val})
    draw_loss_curve([history], legends=[f"resnet18-CIFAR{cifar}"], save_dir=f"history/resnet18-CIFAR{cifar}[{datetime.now()}]")

if __name__ == '__main__':
    main()

Output:
Resnet18 from torchvision.models it's an ImageNet implementation. Because ImageNet samples much bigger(224x224) than CIFAR10/100 (32x32), the first layers designed to aggressively downsample the input ('stem Network'). It's lead to missing much valuable information on small CIFAR10/100 images.
To achieve good accuracy on CIFAR10, authors use different network structure as described in original paper:
https://arxiv.org/pdf/1512.03385.pdf
and explained in this article:
https://towardsdatascience.com/resnets-for-cifar-10-e63e900524e0
You can download resnet fo CIFAR10 from this repo: https://github.com/akamaster/pytorch_resnet_cifar10


Instruction: Loss does not decrease after modifying prediction with torch.floor()

Input:
For the following code, the loss decreases.
loss_function=nn.MSELoss()
loss=loss_function(pred,label)

But, the loss remains completely unchanged if I change the pred by floor function. I checked the parameters after opt.step(), they are not changing.
loss_function=nn.MSELoss()
loss=loss_function(torch.floor(pred),label)

Why this might happen?
My guess: This torch.floor(pred) operation breaks the computation graph. Other "real" mathematical operation like pred*3 (for example) does not break the computation graph.

Output:
It's not breaking the computation graph, the gradients are zero so your steps have no effect.
Consider the plot of floor(x) shown below. Notice that the function is discontinuous so it's technically not differentiable at whole numbers. Also, for every point where it is differentiable it's a flat function. In other words the derivative is zero almost everywhere. PyTorch simply assigns the gradient of floor to be zero everywhere, since there's really no alternative other than to raise an exception. This implies that, regardless of the value of the loss, the gradients of your loss function w.r.t. your parameters will be zero (following the chain rule/backprop). Therefore, any gradient descent based optimizations will have no affect on the model parameters.

(Image source: Wikipedia)


Instruction: Getting indices of values in high dimension matrix

Input:
I have  tensorA of size 10x4x9x2, the other tensorB is of size 10x5x2 that contains values from tensorA. Now, how can i find the index of each element in tensorB in tensorA.
Example:
First 2 elements of  TensorA:
         [[[[ 4.,  1.],
          [ 1.,  2.],
          [ 2.,  5.],
          [ 5.,  3.],
          [ 3., 11.],
          [11., 10.],
          [10., -1.],
          [-1., -1.],
          [-1., -1.]],

         [[12., 13.],
          [13.,  9.],
          [ 9.,  7.],
          [ 7.,  5.],
          [ 5.,  3.],
          [ 3.,  4.],
          [ 4.,  1.],
          [ 1.,  0.],
          [ 0., -1.]],
          ...... so on

Fist 2 elements of TensorB:
   [[[ 2.,  5.],
     [ 5.,  7.],
     [ 7.,  9.],
     [ 9., 10.],
     [10., 12.]],

     [[ 0.,  1.],
     [ 1.,  2.],
     [ 2.,  5.],
     [ 5., -1.],
     [-1., -1.]],

Now in tensorB the first element is [2,5] included in the first 5x2 matrix (dimension 0).
so the element should be matched against dimension 0 in  tensorA. And the output should be index
0,0,2 since it is the 3rd element.

Output:
You can compare the rows that are equal, sum along the last axis, and check that sum against the size of the searched tensor. Then the nonzero function will get you the indices you're looking for.
Since for the example tensors you have given, TensorB[0, 0] is [2., 5.], that looks like:
((TensorA == TensorB[0, 0]).sum(dim=3) == 2).nonzero()

This will return a tensor of [[0, 0, 2]] if that is the only matching row. If you don't want to hard-code 2 (the size of the searched tensor), you can use:
((TensorA == TensorB[0, 0]).sum(dim=3) == TensorB[0, 0].size()[0]).nonzero()


Instruction: one of the variables needed for gradient computation has been modified by an inplace operation : can't find inplace operation

Input:
I have this code below and I can't find the inplace operation that prevents the gradient from computing.

for epoch in range(nepoch):
    model.train()
    scheduler.step()

    for batch1 in loader1:
        torch.ones(len(batch1[0]), dtype=torch.float)
        x, label = batch1
        x = x1.to('cuda', non_blocking=True)
        optimizer.zero_grad()
        pred = model(x)
        pred = pred.squeeze() if pred.ndimension() > 1 else pred
        label = (label.float()).cuda(cuda0)
        weights = torch.ones(len(label))
        loss_fun = torch.nn.BCEWithLogitsLoss(weight=weights.cuda(cuda0))
        score = loss_fun(pred, label)
        label = np.array(np.round(label.cpu().detach())).astype(bool)
        pred = np.array(pred.cpu().detach()>0).astype(bool)
        torch.autograd.set_detect_anomaly(True)

        score.backward()
        optimizer.step()

At the end I have this error that pops up:
Warning: Error detected in MulBackward0. Traceback of forward call        that caused the error:
File "train.py", line 98, in <module>
  pred = model(x)
File "/home/anatole2/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
  result = self.forward(*input, **kwargs)
File "/home/anatole2/miniconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
  input = module(input)
File "/home/anatole2/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
  result = self.forward(*input, **kwargs)
File "/home/anatole2/best/PCEN_pytorch.py", line 30, in forward
  filtered[i] = filtered[i] + (1-exp(self.log_s)) * filtered[i-1]
(print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
File "train.py", line 116, in <module>
  score.backward()
File "/home/anatole2/miniconda3/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
  torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/anatole2/miniconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
  allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16, 1, 80]], which is output 0 of SelectBackward, is at version 378; expected version 377 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

If you could help me that'd be great !

Output:
The in-place operation seems to be on this line:
File "/home/anatole2/best/PCEN_pytorch.py", line 30, in forward
  filtered[i] = filtered[i] + (1-exp(self.log_s)) * filtered[i-1]

Note that it is using the value from filtered[i] and then storing the result in filtered[i].  This is what in-place means; the new value overwrites the old one.
To fix it, you'd need to do something like this:
filtered_new = torch.zeros_like(filtered)
...
filtered_new[i] = filtered[i] + (1-exp(self.log_s)) * filtered[i-1]

The part that makes this a bit complicated is that this seems to be inside a loop (I assume i is the loop counter) and it probably uses the values from the previous pass through the loop.  The modified version is not in-place, but probably won't produce the same results as the original either.  So you may have to do something like this:
filtered_new[i] = filtered[i] + (1-exp(self.log_s)) * filtered_new[i-1]

It's impossible to solve this without seeing more code around this, but basically - look around, and replace any operation which changes existing tensors with an operation which creates new tensors to store the results of the calculation.


Instruction: How to prevent the initial pytorch variable from changing using a function?

Input:
I want to apply a function to the variable x and saved as y. But why the x is also changed? How to prevent it?
import torch
def minus_min(raw):
  for col_i in range(len(raw[0])):
    new=raw
    new[:,col_i] = (raw[:,col_i] - raw[:,col_i].min())
  return new

x=torch.tensor([[0,1,2,3,4],
                [2,3,4,0,8],
                [0,1,2,3,4]])
y=minus_min(x)
print(y)
print(x)

output:
tensor([[0, 0, 0, 3, 0],
        [2, 2, 2, 0, 4],
        [0, 0, 0, 3, 0]])
tensor([[0, 0, 0, 3, 0],
        [2, 2, 2, 0, 4],
        [0, 0, 0, 3, 0]])

Output:
Because this assignment:
new[:,col_i] = (raw[:,col_i] - raw[:,col_i].min())

is an in-place operation. Therefore, x and y will share the underlying .data.
The smallest change that would solve this issue would be to make a copy of x inside the function:
def minus_min(raw):
  new = raw.clone()  # <--- here
  for col_i in range(len(raw[0])):
    new[:,col_i] = raw[:,col_i] - raw[:,col_i].min()
  return new

If you want, you can simplify your function (and remove the for loop):
y = x - x.min(dim=0).values


Instruction: How do I use BertForMaskedLM or BertModel to calculate perplexity of a sentence?

Input:
I want to use BertForMaskedLM or BertModel to calculate perplexity of a sentence, so I write code like this:
import numpy as np
import torch
import torch.nn as nn
from transformers import BertTokenizer, BertForMaskedLM
# Load pre-trained model (weights)
with torch.no_grad():
    model = BertForMaskedLM.from_pretrained('hfl/chinese-bert-wwm-ext')
    model.eval()
    # Load pre-trained model tokenizer (vocabulary)
    tokenizer = BertTokenizer.from_pretrained('hfl/chinese-bert-wwm-ext')
    sentence = "我不会忘记和你一起奋斗的时光。"
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    sen_len = len(tokenize_input)
    sentence_loss = 0.

    for i, word in enumerate(tokenize_input):
        # add mask to i-th character of the sentence
        tokenize_input[i] = '[MASK]'
        mask_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])

        output = model(mask_input)

        prediction_scores = output[0]
        softmax = nn.Softmax(dim=0)
        ps = softmax(prediction_scores[0, i]).log()
        word_loss = ps[tensor_input[0, i]]
        sentence_loss += word_loss.item()

        tokenize_input[i] = word
    ppl = np.exp(-sentence_loss/sen_len)
    print(ppl)

I think this code is right, but I also notice BertForMaskedLM's paramaters masked_lm_labels, so could I use this paramaters to calculate PPL of a sentence easiler?
I know the input_ids argument is the masked input, the masked_lm_labels argument is the desired output. But I couldn't understand the actual meaning of its output loss, its code like this:
if masked_lm_labels is not None:
    loss_fct = CrossEntropyLoss()  # -100 index = padding token
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size),
    masked_lm_labels.view(-1))
    outputs = (masked_lm_loss,) + outputs

Output:
Yes, you can use the  parameter labels (or masked_lm_labels, I think the param name varies in versions of huggingface transformers, whatever) to specify the masked token position, and use -100 to ignore the tokens that you dont want to include in the loss computing.
For example,
sentence='我爱你'
from transformers import BertTokenizer, BertForMaskedLM
import torch
import numpy as np

tokenizer = BertTokenizer(vocab_file='vocab.txt')
model = BertForMaskedLM.from_pretrained('bert-base-chinese')

tensor_input = tokenizer(sentence, return_tensors='pt')
# tensor([[ 101, 2769, 4263,  872,  102]])

repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
# tensor([[ 101, 2769, 4263,  872,  102],
#         [ 101, 2769, 4263,  872,  102],
#         [ 101, 2769, 4263,  872,  102]])

mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
# tensor([[0., 1., 0., 0., 0.],
#         [0., 0., 1., 0., 0.],
#         [0., 0., 0., 1., 0.]])

masked_input = repeat_input.masked_fill(mask == 1, 103)
# tensor([[ 101,  103, 4263,  872,  102],
#         [ 101, 2769,  103,  872,  102],
#         [ 101, 2769, 4263,  103,  102]])

labels = repeat_input.masked_fill( masked_input != 103, -100)
# tensor([[-100, 2769, -100, -100, -100],
#         [-100, -100, 4263, -100, -100],
#         [-100, -100, -100,  872, -100]])

loss,_ = model(masked_input, masked_lm_labels=labels)

score = np.exp(loss.item())

The function:
def score(model, tokenizer, sentence,  mask_token_id=103):
  tensor_input = tokenizer.encode(sentence, return_tensors='pt')
  repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
  mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
  masked_input = repeat_input.masked_fill(mask == 1, 103)
  labels = repeat_input.masked_fill( masked_input != 103, -100)
  loss,_ = model(masked_input, masked_lm_labels=labels)
  result = np.exp(loss.item())
  return result

score(model, tokenizer, '我爱你') # returns 45.63794545581973


Instruction: Weights not updating on my neural net (Pytorch)

Input:
I'm completely new to neural nets, so I tried to roughly follow some tutorials to create a neural net that can just distinguish if a given binary picture contains a white circle or if it is all black. So, I generated 1000 arrays of size 10000 representing a 100x100 picture with half of them containing a white circle somewhere. The generation of my dataset looks like this:
for i in range(1000):
   image = [0] * (IMAGE_SIZE * IMAGE_SIZE)

   if random() < 0.5:
      dataset.append([image, [[0]]])

   else:
      #inserts circle in image
      #...

      dataset.append([image, [[1]]])

np.random.shuffle(dataset)
np.save("testdataset.npy", dataset)

The double list around the classifications is because the net seemed to give that format as an output, so I matched that.
Now since I don't really have any precise idea of how pytorch works, I don't really now which parts of the code are relevant for solving my problem and which aren't. Therefore, I gave the code for the net and the training down below and really hope that someone can explain to me where I went wrong. I'm sorry if it's too much code. The code runs without errors, but if I print the parameters before and after training they didn't change in any way and the net will always just return a 0 for every image/array.
IMAGE_SIZE = 100
EPOCHS = 3
BATCH_SIZE = 50
VAL_PCT = 0.1

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(IMAGE_SIZE * IMAGE_SIZE, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 64)
        self.fc4 = nn.Linear(64, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.log_softmax(x, dim = 1)

net = Net()
optimizer = optim.Adam(net.parameters(), lr = 0.01)
loss_function = nn.MSELoss()
dataset = np.load("testdataset.npy", allow_pickle = True)

X = torch.Tensor([i[0] for i in dataset]).view(-1, 10000)
y = torch.Tensor([i[1] for i in dataset])

val_size = int(len(X) * VAL_PCT)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

for epoch in range(EPOCHS):
    for i in range(0, len(train_X), BATCH_SIZE):
        batch_X = train_X[i:i + BATCH_SIZE].view(-1, 1, 10000)
        batch_y = train_y[i:i + BATCH_SIZE]

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()

Output:
Instead of net.zero_grad() I would recommend using optimizer.zero_grad() as it's more common and de facto standard. Your training loop should be:
for epoch in range(EPOCHS):
    for i in range(0, len(train_X), BATCH_SIZE):
        batch_X = train_X[i:i + BATCH_SIZE].view(-1, 1, 10000)
        batch_y = train_y[i:i + BATCH_SIZE]

        optimizer.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()

I would recommend you reading a bit about different loss functions. It seems you have a classification problem, for that you should use the logits (binary classification) or cross entropy (multi class) loss. I would make the following changes to the network and loss function:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(IMAGE_SIZE * IMAGE_SIZE, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 64)
        self.fc4 = nn.Linear(64, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x

loss_function = nn.BCEWithLogitsLoss()

Check the documentation before using it: https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss
Good luck!


Instruction: How do I install the d2l library on Windows without conda?

Input: The d2l.ai textbook includes its own Python library, d2l.  Their instructions recommend installing with conda, a Python package manager that also functions as a general library manager.  But I've been burned by conda a couple times, and prefer pure pip.  How can I install d2l on Windows with pure pip?  I only need PyTorch, not MxNet or Tensorflow.

Output:
Using pip
This is kind of an embarrassing answer to my own question, but I recommend you follow the CPU or GPU instructions provided, following just the PYTORCH tab:
"C:\Program Files\Python37\Scripts\pip" install torch==1.5.1 torchvision -f https://download.pytorch.org/whl/torch_stable.html

and
"C:\Program Files\Python37\Scripts\pip" install -U d2l

I include the full paths to Python37 here because Python38 is in my path, and torch, and therefore d2l.torch only work with 3.5 <= python < 3.8.
Earlier, I had trouble with mxnet being missing, but when I ran pip install -U d2l again (with the full path to my prefered Python distribution), from d2l import torch as d2l worked just fine for me.
As standalone script
Sometimes it is convenient just to use the direct files. For example, for installing with torch, you can just save this file, renaming it to d2l.py or d2l_torch.py to distinguish it from the main torch library. Place it besides your other python script, and then you can import it with import d2l_torch or whatever you called the script.
If you read the contents of this python file, you will see it is simply a concatenation of all the code samples in the book, with a few convenience aliases at the end of the file that make it easier for the authors to write their book in three languages at the same time.  You can often get by without importing d2l at all and just copy-paste the example code that you need into your own code (with citation of course), or learn what torch commands they are aliasing.

Earlier instructions
In order to import d2l, you will need torch installed.  Do NOT try to install these with a simple pip install torch.  You need specific versions of both libraries for d2l to work.
In this example, I use Python 3.7.  These instructions assume that you have another version of Python as your default version and show how to use full paths to pip to ensure the right installation.
To install torch, go to https://pytorch.org/, in the colorful grid, click on pip, copy the command, open a command prompt as an administrator (right-click and select "Run as Administrator") then paste the command, which should look something like:
pip install torch===1.5.1 torchvision===0.6.1 -f https://download.pytorch.org/whl/torch_stable.html
Then, edit the command to replace pip replaced with the full path to your version of pip, e.g.:
"C:\Program Files\Python37\Scripts\pip" install torch===1.5.1 torchvision===0.6.1 -f https://download.pytorch.org/whl/torch_stable.html
(You don't need to edit the command as long as Python 3.7 is in your path.)
I expanded my own answer to a PyTorch-specific question in writing this answer.