Instruction: Iterating through DataLoader (PyTorch): RuntimeError: Expected object of scalar type unsigned char but got scalar type float for sequence element 9

Input:
I am new to PyTorch and am running into an expected error. The overall context is trying to build a building segmentation model off of Spacenet imagery. I am forked off of this repo from someone at Microsoft AI who built a segmentation model, and I am just trying to re-run her training scripts.
I've been able to download the data, and do the pre-processing. My issue comes when trying to actually train the model, I am trying to iterate through my DataLoader, and I get the following error message:

RuntimeError: Expected object of scalar type unsigned char but got
scalar type float for sequence element 9.

Snippets of code that are useful:
I have a dataset.py that creates the SpaceNetDataset class and looks like:
import os
# Ignore warnings
import warnings

import numpy as np
from PIL import Image
import torch
from torch.utils.data import Dataset

warnings.filterwarnings('ignore')

class SpaceNetDataset(Dataset):
    """Class representing a SpaceNet dataset, such as a training set."""

    def __init__(self, root_dir, splits=['trainval', 'test'], transform=None):
        """
        Args:
            root_dir (string): Directory containing folder annotations and .txt files with the
            train/val/test splits
            splits: ['trainval', 'test'] - the SpaceNet utilities code would create these two
                splits while converting the labels from polygons to mask annotations. The two
                splits are created after chipping larger images into the required input size with
                some overlaps. Thus to have splits that do not have overlapping areas, we manually
                split the images (not chips) into train/val/test using utils/split_train_val_test.py,
                followed by using the SpaceNet utilities to annotate each folder, and combine the
                trainval and test splits it creates inside each folder.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.root_dir = root_dir
        self.transform = transform
        self.image_list = []
        self.xml_list = []

        data_files = []
        for split in splits:
            with open(os.path.join(root_dir, split + '.txt')) as f:
                data_files.extend(f.read().splitlines())

        for line in data_files:
            line = line.split(' ')

            image_name = line[0].split('/')[-1]
            xml_name = line[1].split('/')[-1]

            self.image_list.append(image_name)
            self.xml_list.append(xml_name)

    def __len__(self):
        return len(self.image_list)

    def __getitem__(self, idx):
        img_path = os.path.join(self.root_dir, 'RGB-PanSharpen', self.image_list[idx])
        target_path = os.path.join(self.root_dir, 'annotations', self.image_list[idx].replace('.tif', 'segcls.tif'))

        image = np.array(Image.open(img_path))
        target = np.array(Image.open(target_path))
        target[target == 100] = 1  # building interior
        target[target == 255] = 2  # border

        sample = {'image': image, 'target': target, 'image_name': self.image_list[idx]}

        if self.transform:
            sample = self.transform(sample)

        return sample

To create the DataLoader, I have something like:
dset_train = SpaceNetDataset(data_path_train, split_tags, transform=T.Compose([ToTensor()]))
loader_train = DataLoader(dset_train, batch_size=train_batch_size, shuffle=True,
                          num_workers=num_workers)

I then iterate over the data loader by doing something like:
    for batch in loader_train:
        image_tensors = batch['image']
        images = batch['image'].cpu().numpy()
        break  # take the first shuffled batch

but then I get the error:
Traceback (most recent call last):

File "training/train_aml.py", line 137, in <module> sample_images_train, sample_images_train_tensors = get_sample_images(which_set='train')
File "training/train_aml.py", line 123, in get_sample_images for i, batch in enumerate(loader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp> return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate return torch.stack(batch, 0, out=out)

RuntimeError: Expected object of scalar type unsigned char but got scalar type float for sequence element 9.

The error seems quite similar to this one, although I did try a similar solution by casting:
    dtype = torch.cuda.CharTensor if torch.cuda.is_available() else torch.CharTensor
    for batch in loader:
        batch['image'] = batch['image'].type(dtype)
        batch['target'] = batch['target'].type(dtype)

but I end up with the same error.
A couple of other things that are weird:

This seems to be non-deterministic. Most of the time I get this error, but some times the code keeps running (not sure why)
The "Sequence Element" number at the end of the error message keeps changing. In this case it was "sequence element 9" sometimes it's "sequence element 2", etc. Not sure why.

Output:
Ah nevermind.
Turns out unsigned char comes from C++ where it gives you 0 to 255, so it makes sense that's what it expects from image data.
So I actually fixed this by doing:
        image = np.array(Image.open(img_path)).astype(np.int)
        target = np.array(Image.open(target_path)).astype(np.int)

inside the SpaceNetDataset class and it seemed to work!


Instruction: cannot import torch audio ' No audio backend is available.'

Input:
import torchaudio

When I just try to import torch audio on Pycharm, I have this error
61: UserWarning: No audio backend is available.

warnings.warn('No audio backend is available.')

Output:
You need to install the audio file I/O backend. If Linux it's Sox, if Windows it's SoundFile
To check if you have one set run str(torchaudio.get_audio_backend()) and if 'None' is the result then install the backend.
SoundFile for Windows pip install soundfile
Sox for Linux pip install sox
Check out the PyTorch Audio Backend docs here


Instruction: Huggingface GPT2 and T5 model APIs for sentence classification?

Input:
I've successfully used the Huggingface Transformers BERT model to do sentence classification  using the BERTForSequenceClassification class and API. I've used it for both 1-sentence sentiment analysis and 2-sentence NLI.
I can see that other models have analogous classes, e.g. XLNetForSequenceClassification and RobertaForSequenceClassification. This type of sentence classification usually involves placing a classifier layer on top of a dense vector representing the entirety of the sentence.
Now I'm trying to use the GPT2 and T5 models. However, when I look at the available classes and API for each one, there is no equivalent "ForSequenceClassification" class. For example, for GPT2 there are GPT2Model, GPT2LMHeadModel, and GPT2DoubleHeadsModel classes. Perhaps I'm not familiar enough with the research for GPT2 and T5, but I'm certain that both models are capable of sentence classification.
So my questions are:

What Huggingface classes for GPT2 and T5 should I use for 1-sentence classification?

What classes should I use for 2-sentence (sentence pair) classification (like natural language inference)?

Thank you for any help.

Output:
Well, why not use the code for GPT2LMHeadModel itself as an inspiration :
class MyGPT2LMHeadModel(GPT2PreTrainedModel):
    def __init__(self, config, num_classes):
        super().__init__(config)
        self.transformer = GPT2Model.from_pretrained('gpt2')
        #self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)
        self.lm_head = nn.Linear(config.n_embd, num_classes, bias=False)

...

    def forward(...):
        hidden_states = self.transformer(...)[0]
        lm_logits = self.lm_head(hidden_states)
...


Instruction: TensorRT (C++ API) undefined reference to `createNvOnnxParser_INTERNAL'

Input:
I am trying to create a tensorrt engine from ONNX model using the TensorRT C++ API. I have written code to read, serialize and write a tensorrt engine to disk as per the documentation. I have installed tensorrt7 on colab using debian installation instructions.
This is my c++ code that I am compiling using g++ rnxt.cpp -o rnxt
#include <cuda_runtime_api.h>
#include <NvOnnxParser.h>
#include <NvInfer.h>

#include <cstdlib>
#include <fstream>
#include <iostream>
#include <sstream>
#include <iterator>
#include <algorithm>

class Logger : public nvinfer1::ILogger
 {
     void log(Severity severity, const char* msg) override
     {
         // suppress info-level messages
         if (severity != Severity::kINFO)
             std::cout << msg << std::endl;
     }
 } gLogger;

int main(){

    int maxBatchSize = 32;

    nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(gLogger);
    const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
    nvinfer1::INetworkDefinition* network = builder->createNetworkV2(explicitBatch);

    nvonnxparser::IParser* parser = nvonnxparser::createParser(*network, gLogger);

    parser->parseFromFile("saved_resnext.onnx", 1);
    for (int i = 0; i < parser->getNbErrors(); ++i)
    {
        std::cout << parser->getError(i)->desc() << std::endl;
    }

    builder->setMaxBatchSize(maxBatchSize);
    nvinfer1::IBuilderConfig* config = builder->createBuilderConfig();
    config->setMaxWorkspaceSize(1 << 20);
    nvinfer1::ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);

    parser->destroy();
    network->destroy();
    config->destroy();
    builder->destroy();

    nvinfer1::IHostMemory *serializedModel = engine->serialize();

    std::ofstream engine_file("saved_resnext.engine");

    engine_file.write((const char*)serializedModel->data(),serializedModel->size());

    serializedModel->destroy();
    return 0;
 }

While compiling, I get the following error:
/tmp/ccJaGxCX.o: In function `nvinfer1::(anonymous namespace)::createInferBuilder(nvinfer1::ILogger&)':
rnxt.cpp:(.text+0x19): undefined reference to `createInferBuilder_INTERNAL'
/tmp/ccJaGxCX.o: In function `nvonnxparser::(anonymous namespace)::createParser(nvinfer1::INetworkDefinition&, nvinfer1::ILogger&)':
rnxt.cpp:(.text+0x43): undefined reference to `createNvOnnxParser_INTERNAL'
collect2: error: ld returned 1 exit status

I also get error related to <cuda_runtime_api.h> so I have added (pasted) those files from cuda's include directory (/usr/local/cuda-11.0/targets/x86_64-linux/include) to the /usr/include directoryafter which I am getting the said error. I don't have much experience with C++ and any help would be appreciated.
Edit: I have also installed libnvinfer using
!apt-get install -y libnvinfer7=7.1.3-1+cuda11.0
!apt-get install -y libnvinfer-dev=7.1.3-1+cuda11.0

Output:
This problem is due to nvonnxparser.so  was not linked in Makefile. Just add
target_link_libraries(${TARGET_NAME} nvonnxparser)

in your CMake.


Instruction: What does array[...,list([something]) mean?

Input:
I am going through the following lines of code but I didn't understand image[...,list()]. What do the three dots mean?
self.probability = 0.5
self.indices = list(permutations(range(3), 3))
if random.random() < self.probability:
            image = np.asarray(image)
            image = Image.fromarray(image[...,list(self.indices[random.randint(0, len(self.indices) - 1)])])

What exactly is happening in the above lines?
I have understood that the list() part is taking random channels from image? Am I correct?

Output:
list(permutations(range(3), 3)) generates all permutations of the intergers 0,1,2.
from itertools import permutations
list(permutations(range(3), 3))
# [(0, 1, 2), (0, 2, 1), (1, 0, 2), (1, 2, 0), (2, 0, 1), (2, 1, 0)]

So the following chooses among these tuples of permutations:
list(self.indices[random.randint(0, len(self.indices) - 1)])]

In any case you'll have a permutation over the last axis of image which is usually the image channels RGB (note that with the ellipsis (...) here image[...,ixs] we are taking full slices over all axes except for the last. So this is performing a shuffling of the image channels.
An example run -
indices = list(permutations(range(3), 3))
indices[np.random.randint(0, len(indices) - 1)]
# (2, 0, 1)

Here's an example, note that this does not change the shape, we are using integer array indexing to index on the last axis only:
a = np.random.randint(0,5,(5,5,3))
a[...,(0,2,1)].shape
# (5, 5, 3)


Instruction: Build vocabulary only from training data or entire data?

Input:
Should I build the vocabulary only from train data or all data, wouldn't that effect test data in both ways? I mean :

If we only build the vocab from train data, The model wouldn't recognize a lot of the words in the validation and testing data, if the word is not available in the vocabulary.

Would considering a pre-trained word embedding help in this situation (i.e. the model learns the new word not from training data but from the pre-trained word embedding)?

If yes, Would a randomly Initialized word embedding have the same effect?

On the contrary, I've seen many examples where the coders build their vocab from the entire data,  testing and validation data are shared with training data. Wouldn't this be an obvious data leakage problem?

Output:
If you're talking about word embeddings, then you should have some special token for out-of-vocabulary words (you probably don't want to have all unique words, but rather top N). E.g. add a special token like [UNK], and replace every unknown word with it.

If you have pre-trained word embeddings and small training set, use them as initial point.

Also, there's no reason to initialize embeddings for the words that you won't optimize during training.

The only information that may leak is word frequency, which is not a serious issue.


Instruction: pytorch , changing learning rate during training

Input:
x=np.linspace(0,20,100)

g=1+0.2*np.exp(-0.1*(x-7)**2)
y=np.sin(g*x)

plt.plot(x,y)

plt.show()

x=torch.from_numpy(x)

y=torch.from_numpy(y)

x=x.reshape((100,1))
y=y.reshape((100,1))

MM=nn.Sequential()
MM.add_module('L1',nn.Linear(1,128))
MM.add_module('R1',nn.ReLU())
MM.add_module('L2',nn.Linear(128,128))
MM.add_module('R2',nn.ReLU())
MM.add_module('L3',nn.Linear(128,128))
MM.add_module('R3',nn.ReLU())
MM.add_module('L4',nn.Linear(128,128))
MM.add_module('R5',nn.ReLU())
MM.add_module('L5',nn.Linear(128,1))
MM.double()
L=nn.MSELoss()

lr=3e-05           ######
opt=torch.optim.Adam(MM.parameters(),lr)     #########
Epo=[]
COST=[]

for epoch in range(8000):

  opt.zero_grad()
  err=L(torch.sin(MM(x)),y)
  Epo.append(epoch)
  COST.append(err)
  err.backward()
  if epoch%100==0:
    print(err)
  opt.step()

Epo=np.array(Epo)/1000.
COST=np.array(COST)
pred=torch.sin(MM(x)).detach().numpy()
Trans=MM(x).detach().numpy()
x=x.reshape((100))
pred=pred.reshape((100))
Trans=Trans.reshape((100))

fig = plt.figure(figsize=(10,10))
#ax = fig.gca(projection='3d')
ax = fig.add_subplot(2,2,1)
surf = ax.plot(x,y,'r')

    #ax.plot_surface(x_dat,y_dat,z_pred)
    #ax.plot_wireframe(x_dat,y_dat,z_pred,linewidth=0.1)
fig.tight_layout()
    #plt.show()
ax = fig.add_subplot(2,2,2)
surf = ax.plot(x,pred,'g')
fig.tight_layout()

ax = fig.add_subplot(2,2,3)
surff=ax.plot(Epo,COST,'y+')
plt.ylim(0,1100)

ax = fig.add_subplot(2,2,4)
surf = ax.plot(x,Trans,'b')
fig.tight_layout()

plt.show()

This is the original code 1.
For changing learning rate during training, I tried to move the position of 'opt' as
Epo=[]
COST=[]

for epoch in range(8000):
  lr=3e-05           ######
  opt=torch.optim.Adam(MM.parameters(),lr)     #########
  opt.zero_grad()
  err=L(torch.sin(MM(x)),y)
  Epo.append(epoch)
  COST.append(err)
  err.backward()
  if epoch%100==0:
    print(err)
  opt.step()

This is code 2.
The code 2 also operate, but the result is quite different with code 1.
What is the difference and for changing learning rate during training(like lr=(1-epoch/10000 *0.99), what should I do?

Output:
You shouldn't move the optimizer definition into the training loop, because the optimizer keeps many other information related to training history, e.g in case of Adam there are running averages of gradients that are stored and updated dynamically in the optimizer's internal mechanism,...
So instanciating a new optimizer each iteration makes you lose this history track.
To update the learning rate dynamically there are lot of schedulers classes proposed in pytorch (exponential decay, cyclical decay, cosine annealing , ...). you can check them from the documentation for the full list of schedulers or you can implement your own if needed: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
Example from the documentation: to decay the learning rate by multiplying it by 0.5 each 10 epochs you can use the StepLR scheduler as follows:
opt = torch.optim.Adam(MM.parameters(), lr)
scheduler = torch.optim.lr_scheduler.StepLR(opt, step_size=10, gamma=0.5)

And in your original code 1 you can do :
for epoch in range(8000):
  opt.zero_grad()
  err=L(torch.sin(MM(x)),y)
  Epo.append(epoch)
  COST.append(err)
  err.backward()
  if epoch%100==0:
    print(err)
  opt.step()
  scheduler.step()

As I say you have many other type of lr schedulers so you can choose from the documentation or implement your own


Instruction: TensorboardX input problem about add_scalar()

Input:
When I use the tensorboardX to plot my data loss, it show me that:
AssertionError                            Traceback (most recent call last)
<ipython-input-76-73419a51fcc9> in <module>
----> 1 writer.add_scalar('resnet34_loss', loss)

F:\Program Files\Python\lib\site-packages\tensorboardX\writer.py in add_scalar(self, tag, scalar_value, global_step, walltime)
    403             scalar_value = workspace.FetchBlob(scalar_value)
    404         self._get_file_writer().add_summary(
--> 405             scalar(tag, scalar_value), global_step, walltime)
    406
    407     def add_scalars(self, main_tag, tag_scalar_dict, global_step=None, walltime=None):

F:\Program Files\Python\lib\site-packages\tensorboardX\summary.py in scalar(name, scalar, collections)
    145     name = _clean_tag(name)
    146     scalar = make_np(scalar)
--> 147     assert(scalar.squeeze().ndim == 0), 'scalar should be 0D'
    148     scalar = float(scalar)
    149     return Summary(value=[Summary.Value(tag=name, simple_value=scalar)])

AssertionError: scalar should be 0D

I have turn the loss from float into np.array, and I have read the doc of tensorboardX, it tell me that add_scalar() function must input the scalar data and I do it, but it shows me a bug. Thanks for your help!

Output:
I had the same issue, here is a minimal sample to reproduce your error,
writer = SummaryWriter(osp.join('runs', 'hello'))
loss = np.random.randn(10)
writer.add_scalar(tag='Checking range', scalar_value=loss)
writer.close()

This returns,
Traceback (most recent call last):

  File "untitled0.py", line 26, in <module>
    writer.add_scalar(tag='Checking range', scalar_value=loss)

  File "/home/melike/anaconda2/envs/pooling/lib/python3.6/site-packages/torch/utils/tensorboard/writer.py", line 346, in add_scalar
    scalar(tag, scalar_value), global_step, walltime)

  File "/home/melike/anaconda2/envs/pooling/lib/python3.6/site-packages/torch/utils/tensorboard/summary.py", line 248, in scalar
    assert(scalar.squeeze().ndim == 0), 'scalar should be 0D'

AssertionError: scalar should be 0D

As indicated by the assertion error, scalar.squeeze().ndim should have 0-dimension. Let's check our scalar_value which is loss,
print(loss.squeeze().ndim)

This outputs
1

So, we found the reason of error, add_scalar expects 0-d scalar after squeeze operation and we gave it a 1-d scalar. Tensorboard page of PyTorch docs has add_scalar examples. Let's convert our code to that version.
writer = SummaryWriter(osp.join('runs', 'hello'))
loss = np.random.randn(10)
for i, val in enumerate(loss):
    writer.add_scalar(tag='Checking range', scalar_value=val, global_step=i)
writer.close()

And this is the output,


Instruction: Pytorch error "RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows"

Input:
I have sentences that I vectorize using sentence_vector() method of BiobertEmbedding python module (https://pypi.org/project/biobert-embedding/). For some group of sentences I have no problem but for some others I have the following error message :

File
"/home/nobunaga/.local/lib/python3.6/site-packages/biobert_embedding/embedding.py",
line 133, in sentence_vector
encoded_layers = self.eval_fwdprop_biobert(tokenized_text)   File "/home/nobunaga/.local/lib/python3.6/site-packages/biobert_embedding/embedding.py",
line 82, in eval_fwdprop_biobert
encoded_layers, _ = self.model(tokens_tensor, segments_tensors)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py",
line 730, in forward
embedding_output = self.embeddings(input_ids, token_type_ids)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py",
line 268, in forward
position_embeddings = self.position_embeddings(position_ids)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py",
line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)   File "/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/functional.py",
line 1467, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: index out of range: Tried to
access index 512 out of table with 511 rows. at
/pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

I discovered that for some group of sentences, the problem was related to tags like <tb> for instance. But for others, even when tags are removed, the error message is still there.
(Unfortunately I can't share the code for confidentiality reasons)
Do you have any ideas of what could be the problem?
Thank you by advance
EDIT : you are right cronoik, it will be better with an example.
Example :
sentences = ["This is the first sentence.", "This is the second sentence.", "This is the third sentence."

biobert = BiobertEmbedding(model_path='./biobert_v1.1_pubmed_pytorch_model')

vectors = [biobert.sentence_vector(doc) for doc in sentences]

This last line of code is what caused the error message in my opinion.

Output:
The problem is that the biobert-embedding module isn't taking care of the of the maximum sequence length of 512 (tokens not words!). This is the relevant source code. Have a look at the example below to force the error you received:
from biobert_embedding.embedding import BiobertEmbedding
#sentence has 385 words
sentence = "The near-ubiquity of ASCII was a great help, but failed to address international and linguistic concerns. The dollar-sign was not so useful in England, and the accented characters used in Spanish, French, German, and many other languages were entirely unavailable in ASCII (not to mention characters used in Greek, Russian, and most Eastern languages). Many individuals, companies, and countries defined extra characters as needed—often reassigning control characters, or using value in the range from 128 to 255. Using values above 128 conflicts with using the 8th bit as a checksum, but the checksum usage gradually died out. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data The near-ubiquity of ASCII was a great help, but failed to address international and linguistic concerns. The dollar-sign was not so useful in England, and the accented characters used in Spanish, French, German, and many other languages were entirely unavailable in ASCII (not to mention characters used in Greek, Russian, and most Eastern languages). Many individuals, companies, and countries defined extra characters as needed—often reassigning control"
longersentence = sentence + ' some'

biobert = BiobertEmbedding()
print('sentence has {} tokens'.format(len(biobert.process_text(sentence))))
#works
biobert.sentence_vector(sentence)
print('longersentence has {} tokens'.format(len(biobert.process_text(longersentence))))
#didn't work
biobert.sentence_vector(longersentence)

Output:
sentence has 512 tokens
longersentence has 513 tokens
#your error message....

What you should do is to implement a sliding window approach to process these texts:
import torch
from biobert_embedding.embedding import BiobertEmbedding

maxtokens = 512
startOffset = 0
docStride = 200

sentence = "The near-ubiquity of ASCII was a great help, but failed to address international and linguistic concerns. The dollar-sign was not so useful in England, and the accented characters used in Spanish, French, German, and many other languages were entirely unavailable in ASCII (not to mention characters used in Greek, Russian, and most Eastern languages). Many individuals, companies, and countries defined extra characters as needed—often reassigning control characters, or using value in the range from 128 to 255. Using values above 128 conflicts with using the 8th bit as a checksum, but the checksum usage gradually died out. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data. Text is considered plain-text regardless of its encoding. To properly understand or process it the recipient must know (or be able to figure out) what encoding was used; however, they need not know anything about the computer architecture that was used, or about the binary structures defined by whatever program (if any) created the data The near-ubiquity of ASCII was a great help, but failed to address international and linguistic concerns. The dollar-sign was not so useful in England, and the accented characters used in Spanish, French, German, and many other languages were entirely unavailable in ASCII (not to mention characters used in Greek, Russian, and most Eastern languages). Many individuals, companies, and countries defined extra characters as needed—often reassigning control"
longersentence = sentence + ' some'

sentences = [sentence, longersentence, 'small test sentence']
vectors = []
biobert = BiobertEmbedding()

#https://github.com/Overfitter/biobert_embedding/blob/b114e3456de76085a6cf881ff2de48ce868e6f4b/biobert_embedding/embedding.py#L127
def sentence_vector(tokenized_text, biobert):
    encoded_layers = biobert.eval_fwdprop_biobert(tokenized_text)

    # `encoded_layers` has shape [12 x 1 x 22 x 768]
    # `token_vecs` is a tensor with shape [22 x 768]
    token_vecs = encoded_layers[11][0]

    # Calculate the average of all 22 token vectors.
    sentence_embedding = torch.mean(token_vecs, dim=0)
    return sentence_embedding

for doc in sentences:
    #tokenize your text
    docTokens = biobert.process_text(doc)

    while startOffset < len(docTokens):
        print(startOffset)
        length = min(len(docTokens) - startOffset, maxtokens)

        #now we calculate the sentence_vector for the document slice
        vectors.append(sentence_vector(
                        docTokens[startOffset:startOffset+length]
                        , biobert)
                      )
        #stop when the whole document is processed (document has less than 512
        #or the last document slice was processed)
        if startOffset + length == len(docTokens):
            break
        startOffset += min(length, docStride)
    startOffset = 0

P.S.: Your partial success with removing <tb> was possible because removing <tb> will remove 4 tokens ('<', 't', '##b', '>').


Instruction: Modifying a pretrained model in PyTorch

Input:
I am attempting to modify this particular section of code in the mobilenetv2 model
(17): InvertedResidual(
      (conv): Sequential(
        (0): ConvBNReLU(
          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )

After the conv2d section, I want to add a max-pooling layer but I am having trouble figuring out to do so. I suspect it would be similar to doing something like this:
MobileNet.features = nn.Sequential(nn.Linear(1280, 1000), nn.LeakyReLU(), nn.Dropout(0.5), nn.Linear(1000,3), nn.LogSoftmax(dim=1))

Where I would do something like:
MobileNet.features[17].conv[0] = nn.ConvBRELU(nn.Conv2d(),nn.maxpool,nn.BatchNorm(),nn.ReLU())

but when I tried that I got the error message module 'torch.nn' has no attribute 'ConvBNReLU'
How can I go about modifying the provided section of code?

Output:
ConvBNReLU is not a nn module -- you can find all the available nn modules here.
It is defined in torchvision. You would need to import it by
from torchvision.models.mobilenet import ConvBNReLU

While you cannot just insert a max-pool in ConvBNReLU, it is just inherited from nn.Sequential and helps to specify the parameters. I would sugget you to make a new class, copying the code from ConvBNReLU, and insert a max-pool there.
class ConvMaxPoolBNReLU(nn.Sequential):
    def __init__(self, in_planes, out_planes, kernel_size=3, stride=1, groups=1, norm_layer=None):
        padding = (kernel_size - 1) // 2
        if norm_layer is None:
            norm_layer = nn.BatchNorm2d
        super(ConvBNReLU, self).__init__(
            nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=False),
            nn.MaxPool2d(2),
            norm_layer(out_planes),
            nn.ReLU6(inplace=True)
        )


Instruction: Config change for a pre-trained transformer model

Input:
I am trying to implement a classification head for the reformer transformer. The classification head works fine, but when I try to change one of the config parameters- config.axial_pos_shape i.e sequence length parameter for the model it throws an error;

size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([512, 1, 64]) from checkpoint, the shape in current model is torch.Size([64, 1, 64]).
size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 1024, 192]) from checkpoint, the shape in current model is torch.Size([1, 128, 192]).

The config:
{
  "architectures": [
    "ReformerForSequenceClassification"
  ],
  "attention_head_size": 64,
  "attention_probs_dropout_prob": 0.1,
  "attn_layers": [
    "local",
    "lsh",
    "local",
    "lsh",
    "local",
    "lsh"
  ],
  "axial_norm_std": 1.0,
  "axial_pos_embds": true,
  "axial_pos_embds_dim": [
    64,
    192
  ],
  "axial_pos_shape": [
    64,
    256
  ],
  "chunk_size_feed_forward": 0,
  "chunk_size_lm_head": 0,
  "eos_token_id": 2,
  "feed_forward_size": 512,
  "hash_seed": null,
  "hidden_act": "relu",
  "hidden_dropout_prob": 0.05,
  "hidden_size": 256,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": true,
  "layer_norm_eps": 1e-12,
  "local_attention_probs_dropout_prob": 0.05,
  "local_attn_chunk_length": 64,
  "local_num_chunks_after": 0,
  "local_num_chunks_before": 1,
  "lsh_attention_probs_dropout_prob": 0.0,
  "lsh_attn_chunk_length": 64,
  "lsh_num_chunks_after": 0,
  "lsh_num_chunks_before": 1,
  "max_position_embeddings": 8192,
  "model_type": "reformer",
  "num_attention_heads": 2,
  "num_buckets": [
    64,
    128
  ],
  "num_chunks_after": 0,
  "num_chunks_before": 1,
  "num_hashes": 1,
  "num_hidden_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 100
    }
  },
  "vocab_size": 320
}

Python Code:
config = ReformerConfig()
config.max_position_embeddings = 8192
config.axial_pos_shape=[64, 128]

#config = ReformerConfig.from_pretrained('./cnp/config.json', output_attention=True)

model = ReformerForSequenceClassification(config)
model.load_state_dict(torch.load("./cnp/pytorch_model.bin"))

Output:
I run into the same issue, trying to halve the size of the 65536 (128*512) by default max sequence length used in Reformer pre-training.
As @cronoik mentioned, you must:

load pretrained Reformer
resize it to your need by dropping unnecessary weights
save this new model
load this new model to perform your desired tasks

Those unnecessary weights are the ones from the Position Embeddings layer. In Reformer model, the Axial Position Encodings strategy was used to learn the position embeddings (rather than having fixed ones like BERT). Axial Position Encodings stores position embeddings in a memory efficient manner, using two small tensors rather than a big one.
However, the idea of position embeddings remains exactly the same, which is obtaining different embeddings for each position.
That said, in theory (correct me if I am misunderstanding somewhere), removing the last position embeddings to match your custom max sequence length should not hurt the performance. You can refer to this post from HuggingFace to see a more detailed description of Axial Position Encodings and understand where to truncate your position embeddings tensor.
I have managed to resize and use Reformer with a custom max length of 32768 (128*256) with the following code:
# Load intial pretrained model
model = ReformerForSequenceClassification.from_pretrained('google/reformer-enwik8', num_labels=2)

# Reshape Axial Position Embeddings layer to match desired max seq length
model.reformer.embeddings.position_embeddings.weights[1] = torch.nn.Parameter(model.reformer.embeddings.position_embeddings.weights[1][0][:256])

# Update the config file to match custom max seq length
model.config.axial_pos_shape = 128, 256
model.config.max_position_embeddings = 128*256 # 32768

# Save model with custom max length
output_model_path = "path/to/model"
model.save_pretrained(output_model_path)


Instruction: Saving a PyTorch models state_dict into redis cache

Input:
I’m building a distributed parameter/server type architecture and wanting to communicate model updates through table solutions on Azure.
I’m having a hard time finding any useful information about saving a PyTorch models state_dict into a redis cache. I’ve given up on Azure Cosmos tables because of the size limit (64kb) per entity and looked toward redis since model state_dict params/weights are much larger, even for a small model.
Does anyone have any recommendations for me on how to pursue this? Or if this is even possible?

Output:
My solution (after @GuyKorland commented above) was RedisAI.
I implemented key-value mehanism for model data and communicated it that way between VMs.
for name, param in model.named_parameters():
    redisai_client.tensorset(f'{name}',param.data.numpy().cpu().detach())


Instruction: Trouble with PyTorch's 'ToPILImage'

Input:
Why does this not work?
import torchvision.transforms.functional as tf
from torchvision import transforms
pic = np.random.randint(0, 255+1, size=28*28).reshape(28, 28)
pic = pic.astype(int)
plt.imshow(pic)
t = transforms.ToPILImage()
t(pic.reshape(28, 28, 1))
# tf.to_pil_image(pic.reshape(28, 28, 1))

A beautiful random picture is plotted by matplotlib, but no matter what datatype I chose for my NumPy ndarray, neither to_pil_image or ToPILImage work as expected.
The docs have this to say:

Converts a tensor ... or a numpy ndarray of shape H x W x C to a PIL Image while preserving the value range.
...
If the input has 1 channel, the  mode  is determined by the data type (i.e  int ,  float ,  short ).

None of these datatypes work except for "short".
Everything else results in:
TypeError: Input type int64/float64 is not supported

thrown from torchvision/transforms/functional.py in to_pil_image().
Further, even though the short datatype will work for the stand alone code snippet I provided first, it breaks down when used inside a transform.Compose() called from a Dataset object's __getitem__:
choices = transforms.RandomChoice([transforms.RandomAffine(30),
                                   transforms.RandomPerspective()])

transform = transforms.Compose([transforms.ToPILImage(),
                                transforms.RandomApply([choices], 0.5),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5,), (0.5,))])

trainset = MNIST('data/train.csv', transform=transform)
trainloader = DataLoader(trainset, batch_size=32, shuffle=True, num_workers=4)

RuntimeError: DataLoader worker (pid 12917) is killed by signal: Floating point exception.
RuntimeError: DataLoader worker (pid(s) 12917) exited unexpectedly

Output:
The above answer worked for me only with the following change:
pic = pic.astype('uint8')

hope it works for you.


Instruction: Why return self.head(x.view(x.size(0), -1)) in the nn.Module for pyTorch reinforcement learning example

Input:
I understand that the balancing the pole example requires 2 outputs. Reinforcement Learning (DQN) Tutorial
Here is the output for self.head
  print ('x',self.head)
  x = Linear(in_features=512, out_features=2, bias=True)

When I run the epochs below is the outputs:
print (self.head(x.view(x.size(0), -1)))
return self.head(x.view(x.size(0), -1))

tensor([[-0.6945, -0.1930]])
tensor([[-0.0195, -0.1452]])
tensor([[-0.0906, -0.1816]])
tensor([[ 0.0631, -0.9051]])
tensor([[-0.0982, -0.5109]])
...

The size of x is:
x = torch.Size([121, 32, 2, 8])

So I am trying to understand what x.view(x.size(0), -1) is doing?
I understand from the comment in the code that it's returning:
Returns tensor([[left0exp,right0exp]...]).
But how does x which is torch.Size([121, 32, 2, 8]) being reduced to a tensor of size 2?
Is there an alternative way of writing that makes more sense?  What if I had 4 outputs.  How would I represent that?  Why x.size(0).  Why -1?
So appears to take self.head with 4 outputs to 2 outputs.  Is that correct?
At the bottom is that class I am referring:
class DQN(nn.Module):

    def __init__(self, h, w, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)

        # Number of Linear input connections depends on output of conv2d layers
        # and therefore the input image size, so compute it.
        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1
        convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(w)))
        convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(h)))
        linear_input_size = convw * convh * 32
        self.head = nn.Linear(linear_input_size, outputs)

    # Called with either one element to determine next action, or a batch
    # during optimization. Returns tensor([[left0exp,right0exp]...]).
    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))

Output:
x.view(x.size(0), -1) is flattening the tensor, this is because the Linear layer only accepts a vector (1d array). To break it down, x.view() reshapes the tensor of the specified shape (more info). x.shape(0) returns 1st dimension of the tensor (which is the batch size, this should remain the constant). The -1 in x.view() is a filler, in other words, its dimensions that we don't know, so PyTorch automatically calculates it. For example, if x = torch.tensor([1,2,3,4]), to reshape the tensor to a 2x2, you could do x.view(2,2) or x.view(2,-1) or x.view(-1,2).
The output shape is not a tensor shape of 2, but that of 121,2 (the 121 is the batch size, and the 2 comes from the Linear layers output). So to change the output size from 2, to 4, you would have to change the outputs argument in the __init__ function to 4.


Instruction: What are the differences between torch.jit.trace and torch.jit.script in torchscript?

Input:
Torchscript provides torch.jit.trace and torch.jit.script to convert pytorch code from eager mode to script model. From the documentation, I can understand torch.jit.trace cannot handle control flows and other data structures present in the python. Hence torch.jit.script was developed to overcome the problems in torch.jit.trace.
But it looks like torch.jit.script works for all the cases, then why do we need torch.jit.trace?
Please help me understand the difference between these two methods

Output: If torch.jit.script works for your code, then that's all you should need. Code that uses dynamic behavior such as polymorphism isn't supported by the compiler torch.jit.script uses, so for cases like that, you would need to use torch.jit.trace.


Instruction: how to modify resnet 50 with 4 channels as input using pre-trained weights in Pytorch?

Input:
I would like to change the resnet50 so that I can switch to 4 channel input,
use the same weights for the rgb channels and initialize the last channel with a normal with mean 0 and variance 0.01.
here is my code:
import torch.nn as nn
import torch
from torchvision import models

from misc.layer import Conv2d, FC

import torch.nn.functional as F
from misc.utils import *

import pdb

class Res50(nn.Module):
    def __init__(self,  pretrained=True):
        super(Res50, self).__init__()

        self.de_pred = nn.Sequential(Conv2d(1024, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))

        self._initialize_weights()

        res = models.resnet50(pretrained=pretrained)
        pretrained_weights = res.conv1.weight

        res.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3,bias=False)

        res.conv1.weight[:,:3,:,:] = pretrained_weights
        res.conv1.weight[:,3,:,:].data.normal_(0.0, std=0.01)

        self.frontend = nn.Sequential(
            res.conv1, res.bn1, res.relu, res.maxpool, res.layer1, res.layer2
        )

        self.own_reslayer_3 = make_res_layer(Bottleneck, 256, 6, stride=1)
        self.own_reslayer_3.load_state_dict(res.layer3.state_dict())

    def forward(self,x):
        x = self.frontend(x)
        x = self.own_reslayer_3(x)
        x = self.de_pred(x)
        x = F.upsample(x,scale_factor=8)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                m.weight.data.normal_(0.0, std=0.01)
                if m.bias is not None:
                    m.bias.data.fill_(0)
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.fill_(1)
                m.bias.data.fill_(0)

but it produces the following error, does anyone have any advice?
/usr/local/lib/python3.6/dist-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
Traceback (most recent call last):
  File "train.py", line 62, in <module>
    cc_trainer = Trainer(loading_data,cfg_data,pwd)
  File "/content/drive/My Drive/Folder/Code/trainer.py", line 28, in __init__
    self.optimizer = optim.Adam(self.net.CCN.parameters(), lr=cfg.LR, weight_decay=1e-4) #remenber was 1e-4
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py", line 44, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 206, in add_param_group
    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor

Output:
Ideally, ResNet accepts 3-channel input. To make it work for 4-channel input, you have to add one extra layer (2D conv), pass the 4-channel input through this layer to make the output of this layer suitable for ResNet architecture.
steps

Copy the model weight
weight = model.conv1.weight.clone()

Add the extra 2d conv for the 4-channel input
model.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False) #here 4 indicates 4-channel input

You can add Relu and BatchNorm on top of the extra con2d. In this example, I am not using.

Connect the extra cov2d with the ResNet model (the weight you copied before)
with torch.no_grad():
    model.conv1.weight[:, :3] = weight
    model.conv1.weight[:, 3] = model.conv1.weight[:, 0]

Done

Sorry, I didn't modify your code. You can adjust the changes in your code.


Instruction: Anaconda always want to replace my GPU Pytorch version to CPU Pytorch version when updating

Input:
I have a newly installed Anaconda3 (version 2020.02) environment, and I have installed Pytorch GPU version by the command conda install pytorch torchvision cudatoolkit=10.2 -c pytorch. I have verified that my Pytorch indeed runs fine on GPU.
However, whenever I update Anaconda by conda update --all, the following messages always shows:
The following packages will be SUPERSEDED by a higher-priority channel:

  pytorch            pytorch::pytorch-1.5.0-py3.7_cuda102_~ --> pkgs/main::pytorch-1.5.0-cpu_py37h9f948e0_0

In other words, it always want to replace my GPU version Pytorch to CPU version. I have tried that if continue the update, it will install the CPU version Pytorch and my previous Pytorch code on GPU could not run anymore. I have also tried the command conda update --all --no-channel-priority but the message still shows.
To my knowledge I have never modified Anaconda channels or add custom channels. How can I get rid of this message?

Output:
It's happening because, by default, conda prefers packages from a higher priority channel over any version from a lower priority channel.  -- conda docs
You can solve this problem by setting the priority of pytorch channel higher than the default channel by changing the order in .condarc -- more here
channels:
  - pytorch
  - defaults
  - conda-forge

channel_priority: true

or you can upgrade it by specifying as option:
conda update --all -c pytorch


Instruction: How can I fix the weights of 'torch.nn.Linear'?

Input:
I want to know why there are two tensors in parameter list of nn.linear??

I tried to set parameters, but it didn't work. How can I fix it?

XX=torch.from_numpy(X)
YY=torch.from_numpy(Y)
Ytt=torch.from_numpy(Yt)
XX=XX.view(100,1)
YY=YY.view(100,1)
Ytt=Ytt.view(100,1)
class model(torch.nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.linear1 = torch.nn.Linear(1,2).double()
    self.linear2 = torch.nn.Linear(2,2).double()
    self.linear3 = torch.nn.Linear(2,1).double()

  def forward(self,x):
    x=F.relu(self.linear1(x))
    x=F.relu(self.linear2(x))
    x=self.linear3(x)
    return x

M=model()
L=nn.MSELoss()
print(list(M.linear1.parameters()))
list(M.linear1.parameters())[0]=torch.Tensor([[-0.1],
        [ 0.2]])

print(list(M.linear1.parameters()))

Then
[Parameter containing:
tensor([[-0.2288],
        [ 0.2211]], dtype=torch.float64, requires_grad=True), Parameter containing:
tensor([-0.9185, -0.2458], dtype=torch.float64, requires_grad=True)]
[Parameter containing:
tensor([[-0.2288],
        [ 0.2211]], dtype=torch.float64, requires_grad=True), Parameter containing:
tensor([-0.9185, -0.2458], dtype=torch.float64, requires_grad=True)]

Output:
You have two parameter tensors in each nn.Linear: one for the weight matrix and the other for the bias. The function this layer implements is
y = Wx + b

You can set the values of a parameter tensor by accessing its data:
with torch.no_grad():
    M.linear1.weight.data[...] = torch.Tensor([[-0.1], [0.2]])


Instruction: How to write a RNN with RNNCell in pytorch?

Input:
I am trying to rewrite a code from this simple Vanilla RNN to RNNCell format in pytorch. This is the full code
import torch
import torch.nn as nn
from torch.autograd import Variable

torch.manual_seed(777)

class SimpleRNN(nn.Module):
    def __init__(self,inputs,hiddens,n_class):
        super().__init__()
        self.rnn = nn.RNNCell(inputs,hiddens)
        self.linear = nn.Linear(hiddens,n_class)
        self.hiddens = hiddens
    def forward(self,x):
        hx = torch.zeros((x.shape[1],hiddens))
        rnn_out = []
        for i in x:
            hx = self.rnn(i,hx)
            rnn_out.append(hx)
        linear_out =  self.linear(rnn_out.view(-1, hiddens))
        return linear_out

# hyperparameters
seq_len = 6      # |hihell| == 6, equivalent to time step
input_size = 5   # one-hot size
batch_size = 1   # one sentence per batch
num_layers = 1   # one-layer rnn
num_classes = 5  # predicting 5 distinct character
hidden_size = 4  # output from the RNN

# create an index to character mapping
idx2char = ['h', 'i', 'e', 'l', 'o']

# Teach hihell -> ihello
x_data = [[0, 1, 0, 2, 3, 3]]    # hihell
x_one_hot = [[[1, 0, 0, 0, 0],   # h 0
              [0, 1, 0, 0, 0],   # i 1
              [1, 0, 0, 0, 0],   # h 0
              [0, 0, 1, 0, 0],   # e 2
              [0, 0, 0, 1, 0],   # l 3
              [0, 0, 0, 1, 0]]]  # l 3

x_one_hot = torch.Tensor(x_one_hot)
y_data = torch.Tensor([1, 0, 2, 3, 3, 4])  # ihello

rnn = SimpleRNN(input_size,hidden_size,num_classes)

# train the model
num_epochs = 15
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnn.parameters(), lr = 0.1)
for epoch in range(1, num_epochs + 1):
    optimizer.zero_grad()
    outputs = rnn(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # check the current predicted string
    # max gives the maximum value and its
    # corresponding index, we will only
    # be needing the index
    _, idx = outputs.max(dim = 1)
    idx = idx.data.numpy()
    result_str = [idx2char[c] for c in idx]
    print('epoch: {}, loss: {:1.3f}'.format(epoch, loss.item()))
    print('Predicted string: ', ''.join(result_str))

I am trying to emulate the original RNN + fully connected class object from the tutorial and reusing many of its code. I computed the rnn_out and appended its value in a python list
        rnn_out = []
        for i in x:
            hx = rnn(i,hx)
            rnn_out.append(hx)

because its a python list I cant further execute the code and it will result in this error

AttributeError: 'list' object has no attribute 'view'

How should I write a RNN using RNNCell?

Output:
I am not sure the rest of your code is alright, but in order to fix this error, you can convert your rnn_out list to a torch tensor by adding the following line after the ending of your for loop:
rnn_out = torch.stack(rnn_out)


Instruction: Applying a 2D convolution kernel to each channel in Pytorch?

Input:
I have a single 2D kernel of size [3,3], and a Tensor of size [B, 64, H, W].
My question is, how can I apply the same 2D kernel to each input channel? Should I reshape/repeat the kernel? I tried to repeat my kernel as follows:
kernel = kernel.repeat((B, 64, 1, 1))

But when I apply it the tensor size changes to [1, 64, H, 1].

Output:
One way is to use grouped convolutions with one group per input channel.
Example using nn.functional.conv2d directly
# suppose kernel.shape == [3, 3] and x.shape == [B, 64, H, W]
weights = kernel[None, None, ...].repeat(64, 1, 1, 1)
y = nn.functional.conv2d(x, weights, groups=64)

or using nn.Conv2d
conv = nn.Conv2d(64, 64, 3, groups=64, bias=False)
conv.weight.data = kernel[None, None, ...].repeat(64, 1, 1, 1)
y = conv(x)

Of course you could also specify any padding, stride, or dilation that you want by including those arguments.


Instruction: Modified PyTorch loss function BCEWithLogitsLoss returns NaNs

Input:
I'm trying to solve a binary classification problem (target=0 and target=1) with an exception:
Some of my labels are classified as target=0.5 on purpose, and I wish to have zero loss for either classifying it as 0 or 1 (i.e both classes are "correct").
I tried to implement a custom loss from scratch, based on PyTorch's BCEWithLogitsLoss:
class myLoss(torch.nn.Module):

def __init__(self, pos_weight=1):
    super().__init__()
    self.pos_weight = pos_weight

def forward(self, input, target):
    epsilon = 10 ** -44
    my_bce_loss = -1 * (self.pos_weight * target * F.logsigmoid(input + epsilon)
                        + (1 - target) * log(1 - sigmoid(input) + epsilon))
    add_loss = (target - 0.5) ** 2 * 4
    mean_loss = (my_bce_loss * add_loss).mean()
    return mean_loss

epsilon was chosen so the log will be bounded to -100, as suggested in BCE loss.
However I'm still getting NaN errors, after several epochs:
Function 'LogBackward' returned nan values in its 0th output.

or
Function 'SigmoidBackward' returned nan values in its 0th output.

Any suggestions how can I correct my loss function? maybe by somehow inherit and modify forward function?

Update:
The way I call my custom loss function:
y = batch[:, -1, :].to(self.device, dtype=torch.float32)
y_pred_batch = self.model(x)

LossFun = myLoss(self.pos_weight)
batch_result.loss = LossFun.forward(y_pred_batch, y)

I use Temporal Convolutional Network model, implemented as follows:
out = self.conv1(x)
out = self.chomp1(out)
out = self.elu(out)
out = self.dropout1(out)
res = x if self.downsample is None else self.downsample(x)
return self.tanh(out + res)

Output:
Try it this way:
class myLoss(torch.nn.Module):

    def __init__(self, pos_weight=1):
      super().__init__()
      self.pos_weight = pos_weight

    def forward(self, input, target):
      epsilon = 10 ** -44
      input = input.sigmoid().clamp(epsilon, 1 - epsilon)

      my_bce_loss = -1 * (self.pos_weight * target * torch.log(input)
                          + (1 - target) * torch.log(1 - input))
      add_loss = (target - 0.5) ** 2 * 4
      mean_loss = (my_bce_loss * add_loss).mean()
      return mean_loss

To test I perform 1000 backwards:

target = torch.randint(high=2, size=(32,))
loss_fn = myLoss()
for i in range(1000):
  inp = torch.rand(1, 32, requires_grad=True)
  loss = loss_fn(inp, target)
  loss.backward()
  if torch.isnan(loss):
    print('Loss NaN')
  if torch.isnan(inp.grad).any():
    print('NaN')

All works nice.


Instruction: PyTorch - Save just the model structure without weights and then load and train it

Input:
I want to separate model structure authoring and training. The model author designs the model structure, saves the untrained model to a file and then sends it training service which loads the model structure and trains the model.
Keras has the ability to save the model config and then load it.
How can the same be accomplished with PyTorch?

Output:
You can write your own function to do that in PyTorch. Saving of weights is straight forward where you simply do a torch.save(model.state_dict(), 'weightsAndBiases.pth').

For saving the model structure, you can do this:
(Assume you have a model class named Network, and you instantiate yourModel = Network())
model_structure = {'input_size': 784,
              'output_size': 10,
              'hidden_layers': [each.out_features for each in yourModel.hidden_layers],
              'state_dict': yourModel.state_dict() #if you want to save the weights
}

torch.save(model_structure, 'model_structure.pth')

Similarly, we can write a function to load the structure.
def load_structure(filepath):
    structure = torch.load(filepath)
    model = Network(structure['input_size'],
                    structure['output_size'],
                    structure['hidden_layers'])
    # model.load_state_dict(structure['state_dict']) if you had saved weights as well

    return model

model = load_structure('model_structure.pth')
print(model)

Edit:
Okay, the above was the case when you had access to source code for your class, or if the class was relatively simple so you could define a generic class like this:
class Network(nn.Module):
    def __init__(self, input_size, output_size, hidden_layers, drop_p=0.5):
        ''' Builds a feedforward network with arbitrary hidden layers.

            Arguments
            ---------
            input_size: integer, size of the input layer
            output_size: integer, size of the output layer
            hidden_layers: list of integers, the sizes of the hidden layers

        '''
        super().__init__()
        # Input to a hidden layer
        self.hidden_layers = nn.ModuleList([nn.Linear(input_size, hidden_layers[0])])

        # Add a variable number of more hidden layers
        layer_sizes = zip(hidden_layers[:-1], hidden_layers[1:])
        self.hidden_layers.extend([nn.Linear(h1, h2) for h1, h2 in layer_sizes])

        self.output = nn.Linear(hidden_layers[-1], output_size)

        self.dropout = nn.Dropout(p=drop_p)

    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''

        for each in self.hidden_layers:
            x = F.relu(each(x))
            x = self.dropout(x)
        x = self.output(x)

        return F.log_softmax(x, dim=1)

However, that will only work for simple cases so I suppose that's not what you intended.
One option is, you can define the architecture of model in a separate .py file and import it along with other necessities(if the model architecture is complex) or you can altogether define the model then and there.
Another option is converting your pytorch model to onxx and saving it.
The other option is that, in Tensorflow you can create a .pb file that defines both the architecture and the weights of the model and in Pytorch you would do something like that this way:
torch.save(model, filepath)

This will save the model object itself, as torch.save() is just a pickle-based save at the end of the day.
model = torch.load(filepath)

This however has limitations, your model class definition might not for example be picklable(possible in some complicated models).
Because this is a such an iffy workaround, the answer that you'll usually get is - No, you have to declare the class definition before loading the trained model, ie you need to have access to the model class source code.
Side notes:
An official answer by one of the core PyTorch devs on limitations of loading a pytorch model without code:

We only save the source code of the class definition. We do not save beyond that (like the package sources that the class is referring to).

import foo

class MyModel(...):
    def forward(input):
        foo.bar(input)

Here the package foo is not saved in the model checkpoint.

There are limitations on robustly serializing python constructs. For example the default picklers cannot serialize lambdas. There are helper packages that can serialize more python constructs than the standard, but they still have limitations. Dill 25 is one such package.

Given these limitations, there is no robust way to have torch.load work without having the original source files.


Instruction: is it possible to remove the dtype from tensor in pytorch?

Input:
targets-> [{'boxes': tensor([[ 23.7296,  28.9209, 122.0997, 213.2374]], device='cuda:0',
       dtype=torch.float64), 'labels': tensor([1], device='cuda:0'), 'area': tensor([18131.2344], device='cuda:0'), 'iscrowd': tensor([0], device='cuda:0')}]

right now the boxes has the dtype=torch.float64 is it possible it just looks like
targets-> [{'boxes': tensor([[ 23.7296,  28.9209, 122.0997, 213.2374]], device='cuda:0',), 'labels': tensor([1], device='cuda:0'), 'area': tensor([18131.2344], device='cuda:0'), 'iscrowd': tensor([0], device='cuda:0')}]

Output:
All tensors have a dtype attribute, no exceptions. However, PyTorch has a default float dtype, usually torch.float32 (single precision 32bit floating point). When displaying tensors with this default dtype, it is omitted. However, your boxes tensor has a non-default dtype, torch.float64 and therefore it is being displayed.
You can use the .to() command to cast this tensor to the default torch.float32 dtype, and consequently make PyTorch not explicitly display the dtype:
targets[0]['boxes'] = targets[0]['boxes'].to(dtype=torch.float32)  #.to() in _not_ an in-place operation

This will result in

In [*]: targets
Out[*]:
[{'boxes': tensor([[ 23.7296,  28.9209, 122.0997, 213.2374]], device='cuda:0'),
  'labels': tensor([1], device='cuda:0'),
  'area': tensor([18131.2344], device='cuda:0'),
  'iscrowd': tensor([0], device='cuda:0')}]


Instruction: Runtime Error - element 0 of tensors does not require grad and does not have a grad_fn

Input:
I am using a Unet model for semantic segmentation - I have a custom dataset of images and their masks both in .png format. I have looked in the online forums and tried stuff, but not much works?
Any suggestions in how to resolve the error or improve the code would be helpful.
model.eval()
with torch.no_grad():
    for xb, yb in val_dl:
        yb_pred = model(xb.to(device))
        # yb_pred = yb_pred["out"].cpu()
        print(yb_pred.shape)
        yb_pred = torch.argmax(yb_pred,axis = 1)
        break

    print(yb_pred.shape)

criteron = nn.CrossEntropyLoss(reduction = 'sum')
opt = optim.Adam(model.parameters(), lr = 3e-4)

def loss_batch(loss_func, output, target, opt = None):
    loss = loss_func(output, target)

    if opt is not None:
        opt.zero_grad()
        loss.backward()
        opt.step()
    return loss.item(), None

lr_scheduler = ReduceLROnPlateau(opt, mode = 'min', factor = 0.5, patience= 20, verbose = 1)

def get_lr(opt):
    for param_group in opt.param_groups:
        return param_group['lr']

current_lr = get_lr(opt)
print('current_lr = {}'.format(current_lr))

def loss_epoch(model, loss_func, dataset_dl, sanity_check = False, opt = None):
    running_loss = 0.0
    len_data = len(dataset_dl.dataset)

    for xb, yb in dataset_dl:
        xb = xb.to(device)
        yb = yb.to(device)

        # xb = torch.tensor(xbh, requires_grad=True)

        output = model(xb)

        loss_b, metric_b = loss_batch(loss_func, output, yb, opt)
        running_loss += loss_b
        if sanity_check is True:
            break
    loss = running_loss/float(len_data)
    return loss, None

def train_val(model, params):
    num_epochs = params["num_epochs"]
    loss_func = params["loss_func"]
    opt = params["optimizer"]
    train_dl = params["train_dl"]
    val_dl = params["val_dl"]
    sanity_check = params["sanity_check"]
    lr_scheduler = params["lr_scheduler"]
    path2weights = params["path2weights"]

    loss_history = {"train": [],
                    "val": []}
    best_model_wts = copy.deepcopy(model.state_dict())
    best_loss = float('inf')

    for epoch in range(num_epochs):
        current_lr = get_lr(opt)
        print('Epoch {}/{}, current_lr = {}'.format(epoch, num_epochs - 1, current_lr))

        with torch.enable_grad():
            model.train()
            train_loss, _ = loss_epoch(model, loss_func, train_dl, sanity_check, opt)
        loss_history["train"].append(train_loss)
        model.eval()

        with torch.no_grad():
            val_loss, _ = loss_epoch(model, loss_func, val_dl, sanity_check, opt)
        loss_history["val"].append(val_loss)

        if val_loss < best_loss:
            best_loss = val_loss
            best_model_wts = copy.deepcopy(model.state_dict())
            torch.save(model.state_dict(), path2weights)
            print("copied best model weights!!")

        lr_scheduler.step(val_loss)
        if current_lr != get_lr(opt):
            print("Loading best model weights!!")
            model.load_state_dict(best_model_wts)
            print("train Loss: %.6f" %(train_loss))
            print("val_loss: %.6f" %(val_loss))
            print("-"*20)

        model.load_state_dict(best_model_wts)
        return model, loss_history, metric_history

path2models = "./models/"
if not os.path.exists(path2models):
    os.mkdir(path2models)

param_train = {
    "num_epochs": 10,
    "loss_func": criteron,
    "optimizer": opt,
    "train_dl": train_dl,
    "val_dl": val_dl,
    "sanity_check": False,
    "lr_scheduler": lr_scheduler,
    "path2weights": path2models + "weights.pt"
model, loss_hist, _ = train_val(model, param_train)

The error message looks like -
File "", line 10, in
model, loss_hist, _ = train_val(model, param_train)
File "", line 27, in train_val
val_loss, _ = loss_epoch(model, loss_func, val_dl, sanity_check, opt)
File "", line 13, in loss_epoch
loss_b, metric_b = loss_batch(loss_func, output, yb, opt)
File "", line 6, in loss_batch
loss.backward()
File "C:\Users\W540\anaconda3\lib\site-packages\torch\tensor.py", line 198, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "C:\Users\W540\anaconda3\lib\site-packages\torch\autograd_init_.py", line 100, in backward
allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
I am not sure which variable to set as require_grad = True or where I should enable grad...

Output:
You can try this before loss.backward():
loss = Variable(loss, requires_grad = True)

Or, because the Variable has been removed from PyTorch (still exists but deprecated), you can do the same thing simply by using following code:
loss.requires_grad = True


Instruction: Error trying to build docker for flask app

Input:
I'm trying to build a docker to run a flask app. I've never done this before. I have the flask app working locally. Here is my approach:
My directory structure for the project looks like this:
model.pkl  README.md          images/            static/
Dockerfile         flaskapp.py        requirements.txt   templates/

I can launch the flask app by running python flaskapp.py and it runs in my browser (locally).
I want to create a Docker so other machines can run this project without dealing with all the dependency stuff. To do so, I've done the following:

I created a Dockerfile with this inside:

FROM python:3
COPY requirements.txt /tmp
COPY flaskapp.py /tmp
COPY model.pkl /tmp
COPY images /tmp
COPY static /tmp
COPY templates /tmp
WORKDIR /tmp
ADD flaskapp.py /
RUN pip install -r requirements.txt
CMD [ "python", "flaskapp.py" ]

Ran the command docker build -t python-barcode .

That worked, so. I ran docker run python-barcode. The terminal printed out * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit), but it didn't work and I got this error on the browser:

This site can’t be reached0.0.0.0 refused to connect.
Try:

Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED

So I did some digging and I updated my Dockerfile to this (adding the last line):
FROM python:3
COPY requirements.txt /tmp
COPY flaskapp.py /tmp
COPY model.pkl /tmp
COPY images /tmp
COPY static /tmp
COPY templates /tmp
WORKDIR /tmp
ADD flaskapp.py /
RUN pip install -r requirements.txt
CMD [ "python", "flaskapp.py" ]
CMD ["flask", "run", "--host", "0.0.0.0" ]

Then running docker run python-barcode again, I get this error:

Usage: flask run [OPTIONS]

Error: Could not locate Flask application. You did not provide the FLASK_APP environment variable.

For more information see http://flask.pocoo.org/docs/latest/quickstart/

How should I proceed?
If its relevant, my flaskapp.py looks like this:
model = load_learner('', 'model.pkl')

app = Flask(__name__)
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0

def classify(document):
    X = document
    y = model.predict(X)

    return y

class ReviewForm(Form):
    pred = TextAreaField('',[validators.DataRequired(),validators.length(min=1)])

@app.route('/')
def index():
    form = ReviewForm(request.form)
    return render_template('reviewform.html', form=form)

@app.route('/results', methods=['POST'])
def results():
    form = ReviewForm(request.form)
    if request.method == 'POST' and form.validate():
        sequence = request.form['pred']
        y = classify(sequence)
        return render_template('results.html',
        y = y)
    return render_template('reviewform.html', form=form)

if __name__ == '__main__':
    app.run(host= '0.0.0.0')

EDIT 1
Now I am getting this error:
[2020-07-03 00:29:51,222] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flaskapp.py", line 94, in index
    return render_template('reviewform.html', form=form)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 133, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 930, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/local/lib/python3.8/site-packages/jinja2/loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 57, in get_source
    return self._get_source_fast(environment, template)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: reviewform.html

Output:
You can have only CMD per Dockerfile. so one will likely be ignored.
CMD [ "python", "flaskapp.py" ]
CMD ["flask", "run", "--host", "0.0.0.0" ]

Just remove the second one as you already listening to all interfaces in the code.
CMD [ "python", "flaskapp.py" ]

Now run the Docker container with this command.
docker run -p 5000:5000 -it python-barcode

And then you will able to hit endpoint
htpp://localhost:5000


Instruction: How to find the mean and Std of a 3 channel image

Input:
from numpy import asarray
from PIL import Image

image = Image.open('../input/chest-xray-pneumonia/chest_xray/train/NORMAL/IM-0115-0001.jpeg')
pixels = asarray(image)

pixels = pixels.astype('float32')
means = pixels.mean(axis=(0,1), dtype='float64')
stds = pixels.std(axis=(0,1), dtype='float64')
print('Means: %s, Stds: %s' % (means, stds))
pixels = (pixels - means) / stds
means = pixels.mean(axis=(0,1), dtype='float64')
stds = pixels.std(axis=(0,1), dtype='float64')
print('Means: %s, Stds: %s' % (means, stds))'''

output>> Means: 128.90747832983968, Stds: 62.30103035552067
         Means: 1.2235509834827096e-07, Stds: 1.0000000181304383

The problem is while putting 3 channel image ,only got two values each

Output:
Separate the image channels into r,g,b using OpenCV,
Then use numpy mean and std function to calculate the mean and standard deviations for each channel.
Example for separating the image into rgb channels.
import cv2
import numpy as np

img = cv2.imread("image.jpg")
b = img[:,:,0]
g = img[:,:,1]
r = img[:,:,2]


Instruction: Is there a way to obtain IUV map from image in tensorflow?

Input:
I have been using detectron2/densepose to generate IUV map which helped me generate UV texture from the input image. Now, for deployment, I need to have IUV map generation from input image in client-side using JavaScript. I am familiar with TensorFlow but the current dense pose model runs only on PyTorch and interconversion tools are giving errors too.
Any comment or solution to the problem will be very helpful.
IUV image needed:

Final UV map:

Output:
First, you need to use dump command to generate pkl file after that we can generate the needed IUV image.
Inside Densepose use apply_net.py to generate the pkl file. It has a lot of other options as well. You can check them out here.
python3 apply_net.py dump configs/densepose_rcnn_R_101_FPN_s1x.yaml /models/R_101_FPN_s1x.pkl /Images/frame.jpg --output output.pkl -v
After having the pkl file we will need to extract information from it and create your IUV needed image.
img = Image.open('/Images/frame.jpg')
img_w ,img_h = img.size
with open('output.pkl','rb') as f:
  data=pickle.load(f)
i = data[0]['pred_densepose'][0].labels.cpu().numpy()
uv = data[0]['pred_densepose'][0].uv.cpu().numpy()
iuv = np.stack((uv[1,:,:], uv[0,:,:], i * 0,))
iuv = np.transpose(iuv, (1,2,0))
iuv_img = Image.fromarray(np.uint8(iuv*255),"RGB")
iuv_img.show() #It shows only the croped person

box = data[0]["pred_boxes_XYXY"][0]
box[2]=box[2]-box[0]
box[3]=box[3]-box[1]
x,y,w,h=[int(v) for v in box]
bg=np.zeros((img_h,img_w,3))
bg[y:y+h,x:x+w,:]=iuv
bg_img = Image.fromarray(np.uint8(bg*255),"RGB")
bg_img.save('output.png')


Instruction: Why EfficientNet same model return different predictions

Input:
!pip install efficientnet_pytorch -q

import torch
import torch.nn as nn
import efficientnet_pytorch as efn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device

model = efn.EfficientNet.from_name('efficientnet-b0')
model = model.to(device)

img = torch.ones((2, 3, 680, 680))*0.5
img = img.to(device)
preds1 = model(img)
preds2 = model(img)
preds3 = model(img)

print(preds1[0][0])
print(preds2[0][0])
print(preds3[0][0])

del model, img, preds1, preds2, preds3

And preds1, preds2, and preds3 are different. I am confused about this point, model is same and input is same, why predictions are different?
tensor(0.2599, grad_fn=<SelectBackward>)
tensor(0.1364, grad_fn=<SelectBackward>)
tensor(0.1263, grad_fn=<SelectBackward>)

Output:
Wow, solved! I checked the Efficientnet-pytorch source code and suddendly found I should turn the model to eval() mode.
Now the predictions are same!!
model = efn.EfficientNet.from_name('efficientnet-b0')
model = model.to(device)
_ = model.eval()  ## hey, look here!!

img = torch.ones((2, 3, 680, 680))*0.5
img = img.to(device)
preds1 = model(img)
preds2 = model(img)
preds3 = model(img)

print(preds1[0][0])
print(preds2[0][0])
print(preds3[0][0])


Instruction: Error while inferencing a LSTM model with the help of onnx-runtime . Invalid Argument Error

Input:
I have exported a LSTM model from pytorch to onnx . The model takes sequences of length 200. It has hidden state size 256 , number of layers = 2.The forward function takes input size of (batches , sequencelength) as input along with a tuple consisting of hidden state and cell state. I am getting an error while inferencing the model with onnx runtime. hidden state and cell state dimensions are same.
ioio1 = np.random.rand(1,200)
ioio2 = np.zeros((2,1,256),dtype = np.float)
pred = runtime_session.run([output_name],{runtime_session.get_inputs()[0].name:ioio1,
                                          runtime_session.get_inputs()[1].name :ioio2,
                                          runtime_session.get_inputs()[2].name : ioio2})

InvalidArgument                           Traceback (most recent call last)
<ipython-input-204-3928823f661e> in <module>()
      1 pred = runtime_session.run([output_name],{runtime_session.get_inputs()[0].name:ioio1,
      2                                           runtime_session.get_inputs()[1].name :ioio2,
----> 3                                           runtime_session.get_inputs()[2].name : ioio2})

/usr/local/lib/python3.6/dist-packages/onnxruntime/capi/session.py in run(self, output_names, input_feed, run_options)
    109             output_names = [output.name for output in self._outputs_meta]
    110         try:
--> 111             return self._sess.run(output_names, input_feed, run_options)
    112         except C.EPFail as err:
    113             if self._enable_fallback:

InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (N11onnxruntime17PrimitiveDataTypeIdEE) , expected: (N11onnxruntime17PrimitiveDataTypeIlEE)

Output:
This issue is similar : https://github.com/microsoft/onnxruntime/issues/4423
Resolution: ioio1 = np.random.rand(1,200) is float64 (double) , which isn't the dtype your model is expecting.


Instruction: Does creating a data loader inside another data loader in pytorch slow things down (during meta-learning)?

Input:
I was trying to create a data loader for meta-learning but got that my code is extremely slow and I can't figure out why. I am doing this because a set of data sets (so I need data loaders for them) is what is used in meta-learning.
I am wondering if it's because I have a collate function generating data loaders.
Here is the collate function that generates data loaders (and receives ALL the data sets):
class GetMetaBatch_NK_WayClassTask:

    def __init__(self, meta_batch_size, n_classes, k_shot, k_eval, shuffle=True, pin_memory=True, original=False, flatten=True):
        self.meta_batch_size = meta_batch_size
        self.n_classes = n_classes
        self.k_shot = k_shot
        self.k_eval = k_eval
        self.shuffle = shuffle
        self.pin_memory = pin_memory
        self.original = original
        self.flatten = flatten

    def __call__(self, all_datasets, verbose=False):
        NUM_WORKERS = 0 # no need to change
        get_data_loader = lambda data_set: iter(data.DataLoader(data_set, batch_size=self.k_shot+self.k_eval, shuffle=self.shuffle, num_workers=NUM_WORKERS, pin_memory=self.pin_memory))
        #assert( len(meta_set) == self.meta_batch_size*self.n_classes )
        # generate M N,K-way classification tasks
        batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y = [], [], [], []
        for m in range(self.meta_batch_size):
            n_indices = random.sample(range(0,len(all_datasets)), self.n_classes)
            # create N-way, K-shot task instance
            spt_x, spt_y, qry_x, qry_y = [], [], [], []
            for i,n in enumerate(n_indices):
                data_set_n = all_datasets[n]
                dataset_loader_n = get_data_loader(data_set_n) # get data set for class n
                data_x_n, data_y_n = next(dataset_loader_n) # get all data from current class
                spt_x_n, qry_x_n = data_x_n[:self.k_shot], data_x_n[self.k_shot:] # [K, CHW], [K_eval, CHW]
                # get labels
                if self.original:
                    #spt_y_n = torch.tensor([n]).repeat(self.k_shot)
                    #qry_y_n = torch.tensor([n]).repeat(self.k_eval)
                    spt_y_n, qry_y_n = data_y_n[:self.k_shot], data_y_n[self.k_shot:]
                else:
                    spt_y_n = torch.tensor([i]).repeat(self.k_shot)
                    qry_y_n = torch.tensor([i]).repeat(self.k_eval)
                # form K-shot task for current label n
                spt_x.append(spt_x_n); spt_y.append(spt_y_n) # array length N with tensors size [K, CHW]
                qry_x.append(qry_x_n); qry_y.append(qry_y_n) # array length N with tensors size [K, CHW]
            # form N-way, K-shot task with tensor size [N,W, CHW]
            spt_x, spt_y, qry_x, qry_y = torch.stack(spt_x), torch.stack(spt_y), torch.stack(qry_x), torch.stack(qry_y)
            # form N-way, K-shot task with tensor size [N*W, CHW]
            if verbose:
                print(f'spt_x.size() = {spt_x.size()}')
                print(f'spt_y.size() = {spt_y.size()}')
                print(f'qry_x.size() = {qry_x.size()}')
                print(f'spt_y.size() = {qry_y.size()}')
                print()
            if self.flatten:
                CHW = qry_x.shape[-3:]
                spt_x, spt_y, qry_x, qry_y = spt_x.reshape(-1, *CHW), spt_y.reshape(-1), qry_x.reshape(-1, *CHW), qry_y.reshape(-1)
            ## append to N-way, K-shot task to meta-batch of tasks
            batch_spt_x.append(spt_x); batch_spt_y.append(spt_y)
            batch_qry_x.append(qry_x); batch_qry_y.append(qry_y)
        ## get a meta-set of M N-way, K-way classification tasks [M,K*N,C,H,W]
        batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y = torch.stack(batch_spt_x), torch.stack(batch_spt_y), torch.stack(batch_qry_x), torch.stack(batch_qry_y)
        return batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y

that is passed to another data loader here:
def get_meta_set_loader(meta_set, meta_batch_size, n_episodes, n_classes, k_shot, k_eval, pin_mem=True, n_workers=4):
    """[summary]

    Args:
        meta_set ([type]): the meta-set
        meta_batch_size ([type]): [description]
        n_classes ([type]): [description]
        pin_mem (bool, optional): [Since returning cuda tensors in dataloaders is not recommended due to cuda subties with multithreading, instead set pin=True for fast transfering of the data to cuda]. Defaults to True.
        n_workers (int, optional): [description]. Defaults to 4.

    Returns:
        [type]: [description]
    """
    if n_classes > len(meta_set):
        raise ValueError(f'You really want a N larger than the # classes in the meta-set? n_classes, len(meta_set = {n_classes, len(meta_set)}')
    collator_nk_way = GetMetaBatch_NK_WayClassTask(meta_batch_size, n_classes, k_shot, k_eval)
    episodic_sampler = EpisodicSampler(total_classes=len(meta_set), n_episodes=n_episodes)
    episodic_metaloader = data.DataLoader(
        meta_set,
        num_workers=n_workers,
        pin_memory=pin_mem, # to make moving to cuda more efficient
        collate_fn=collator_nk_way, # does the collecting to return M N,K-shot task
        batch_sampler=episodic_sampler # for keeping track of the episode
        )
    return episodic_metaloader

(will generate a smaller example)

related:

https://discuss.pytorch.org/t/what-does-runtimeerror-cuda-driver-error-initialization-error-mean/87505/6

Output:
Conceptually pytorch dataloaders should have no problem being fast even if one is inside the other. One way to debug your issue is to use the line_profiler package to get a better idea of where the slowdown happens.
If you cannot resolve the issue after using the line_profiler, please update your questions with the output of the profiler to help us understand what might be wrong. Allow the profiler to run for some time to gather enough statistics about the execution of your dataloader. The @profile decorator works for both functions and class functions too so it should work for your dataloader functions.


Instruction: Pytorch: RuntimeError: expected dtype Float but got dtype Long

Input:
I encounter this weird error when building a simple NN in Pytorch. I dont understand this error and why this consern Long and Float datatype in backward function. Anyone encounter this before? Thanks for any help.
Traceback (most recent call last):
  File "test.py", line 30, in <module>
    loss.backward()
  File "/home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: expected dtype Float but got dtype Long (validate_dtype at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/TensorIterator.cpp:143)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x7f5856661b5e in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::TensorIterator::compute_types() + 0xce3 (0x7f587e3dc793 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site
-packages/torch/lib/libtorch_cpu.so)
frame #2: at::TensorIterator::build() + 0x44 (0x7f587e3df174 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages
/torch/lib/libtorch_cpu.so)
frame #3: at::native::smooth_l1_loss_backward_out(at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long)
 + 0x193 (0x7f587e22cf73 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xe080b7 (0x7f58576960b7 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cuda.so)
frame #5: at::native::smooth_l1_loss_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, long) + 0x16e (0x7f587
e23569e in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0xed98af (0x7f587e71c8af in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xe22286 (0x7f587e665286 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cpu.so)

Here is the source code:
import torch
import torch.nn as nn
import numpy as np
import torchvision
from torchvision import models
from UTKLoss import MultiLoss
from ipdb import set_trace

# out features [13, 2, 5]
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 20)
model_ft.cuda()

criterion = MultiLoss()
optimizer = torch.optim.Adam(model_ft.parameters(), lr = 1e-3)

image = torch.randn((1, 3, 128, 128)).cuda()
age = torch.randint(110, (1,)).cuda()
gender = torch.randint(2, (1,)).cuda()
race = torch.randint(5, (1,)).cuda()
optimizer.zero_grad()
output = model_ft(image)
age_loss, gender_loss, race_loss = criterion(output, age, gender, race)
loss = age_loss + gender_loss + race_loss
loss.backward()
optimizer.step()

Here is what I define my loss function
import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, output, age, gender, race):
        age_pred = output[:, :13]
        age_pred = torch.sum(age_pred, 1)
        gender_pred = output[:, 13: 15]
        race_pred = output[:, 15:]
        age_loss = F.smooth_l1_loss(age_pred.view(-1, 1), age.cuda())
        gender_loss = F.cross_entropy(gender_pred, torch.flatten(gender).cuda(), reduction='sum')
        race_loss = F.cross_entropy(race_pred, torch.flatten(race).cuda(), reduction='sum')
        return age_loss, gender_loss, race_loss

Output:
Change the criterion call to:
age_loss, gender_loss, race_loss = criterion(output, age.float(), gender, race)

If you look at your error we can trace it to:
frame #3: at::native::smooth_l1_loss_backward_out

In the MultiLoss Class, the smooth_l1_loss works with age. So I changed it's type to float (as the expected dtype is Float) while passing it to the criterion. You can check that age is torch.int64 (i.e. torch.long) by printing age.dtype
I am not getting the error after doing this. Hope it helps.


Instruction: pytorch albumentations augmentation p value?

Input:
augmented_images(raw.image_id.unique()[1230], albumentations.HorizontalFlip(p=1))

for augmented_image what is the p=1 mean? is value difference make angle different?
if its not it how should I make various angle different horizontal augmentation?

Output:
As you can see in the docs of albumentations.HorizontalFlip:

Parameters:   p (float) – probability of applying the transform. Default: 0.5.

If you want to rotate, you should consider using albumentations.augmentations.transforms.Rotate:

Rotate(limit=90, interpolation=1, border_mode=4, value=None, mask_value=None, always_apply=False, p=0.5)
Rotate the input by an angle selected randomly from the uniform distribution.
Parameters:

limit ((int, int) or int) – range from which a random angle is picked. If limit is a single int an angle is picked from (-limit, limit). Default: (-90, 90)
[...]
p (float) – probability of applying the transform. Default: 0.5.


Instruction: RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS

Input:
I am running code that UI downloaded from github. It is supposed to be working (I saw that other people managed to activate it). When I try to run it I get the following error message:
RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS

The code uses pytorch 0.4.1. I have cuda installed.
When I run the command cat /usr/local/cuda/version.txt
I get the answer:
CUDA Version 10.0.130

When I run the command conda list -n <my env name>
I see:
cudatoolkit ver 9.0
cudnn       ver 7.6.5

And now, my question:
What should I do to avoid this error?
Do I need to use pip install for a more recent version of cudnn? If so, which one?

Output:
I also faced the same issue. In my case, the PyTorch version was 0.4.1, and the Cuda version was 9.0. I solved the issue by adding this piece of code:
torch.backends.cudnn.benchmark = True


Instruction: Hugging face: tokenizer for masked lm question

Input:
I am using transformer version 3.0.0 for my project and have some questions.
I want to use a bert model with masked lm pretraining for protein sequences.
To get a character level tokenizer I derived from the BertTokenizer
from transformers import BertTokenizer
class DerivedBertTok(BertTokenizer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
    def tokenize(self, text):
        if isinstance(text, np.ndarray):
            assert len(text) == 1
            text = text[0]
        return [x if x in self.vocab else self.unk_token for x in text]

my vocab looks like this
[PAD]
[CLS]
[SEP]
[UNK]
[MASK]
A
R
N
D
B
C
E
Q
Z
G
H
I
L
K
M
F
P
S
T
W
Y
V

The usage seems quite similar to what i have seen in the docs:
d_tokenizer = DerivedBertTok(
    vocab_file=vocab_path,
    do_lower_case=False,
    do_basic_tokenize=False,
    tokenize_chinese_chars=False
)
d_tokenizer.encode_plus(np.array(["AXEF"])[0],
                      max_length=20,
                      pad_to_max_length=True,
                      add_special_tokens=True,
                      truncation=True,
                      return_tensors='pt')

From this I was building a pytorch Dataset with a custom collate function.
all the collate function does is taking all input tensors and stacking them
from transformers import BatchEncoding
    def collate_fn(self, batch):
        # this function will not work for higher dimension inputs
        elem = batch[0]
        elem_type = type(elem)
        if isinstance(elem, BatchEncoding):
            new_shapes = {key: (len(batch), value.shape[1]) for key, value in elem.items()}
            outs = {key: value.new_empty(new_shapes[key]) for key, value in elem.items()}
            if torch.utils.data.get_worker_info() is not None:
                [v.share_memory_() for v in outs.values()]
            return {key: torch.stack(tuple((d[key].view(-1) for d in batch)), 0, out=outs[key]) for key in elem.keys()}
        else:
            raise ValueError(f"type: {elem_type} not understood")

Question 1:
So I was wondering if the BatchEncoding or another class is already capable of doing this (and doing it possibly better?). Or using a different Dataset/ DataLoader class altogether.
Question 2:
Additionally, I want to mask some of the Inputs as required for the masked LM, however I did not manage find any implementation in the transformer library. Are there any recommendations for doing this?

Output: After some more digging I found a DataCollator, which implements replacing token randomly with the mask token at: https://github.com/huggingface/transformers/blob/615be03f9d961c0c9722fe10e7830e011066772e/src/transformers/data/data_collator.py#L69. So I changed my DataSource to return raw text instead of the BatchEncoding in the __getitem__ method and then do the encoding and masking in the collate function.


Instruction: How to use pytorch in flask and run it on wsgi_mod for apache2

Input:
I'm trying to run a flask app with apache in wsgi_mod
As Describe in following link
https://pytorch.org/tutorials/recipes/deployment_with_flask.html
i configure my app for working with pytorch
when my app tries to
import torchvision
it hangs with no error log.
Is there any way to fix this issue?
the main python code is:
import torchvision.models as models
import torchvision.transforms as transforms
from flask import Flask
app = Flask(__name__)
@app.route('/')
def hello():
    return ("Hello World qwertyuiop!!")
if __name__ == "main":
    app.run()

and the apache2 config is:
User daemon
Group daemon

</IfModule>

LoadFile "G:/Python3764/python37.dll"
LoadModule wsgi_module "G:/Python3764/lib/site-packages/mod_wsgi/server/mod_wsgi.cp37-win_amd64.pyd"
WSGIPythonHome "G:/Python3764"

<VirtualHost *:80>
    ServerName localhost:80
    WSGIScriptAlias / "D:/WSGIAppDir/web.wsgi"
    DocumentRoot "D:/WSGIAppDir"
    <Directory "D:/WSGIAppDir">
        Require all granted
    </Directory>
</VirtualHost>

import sys
sys.path.insert(0, 'D:/WSGIAppDir')
from hello import app as application

Output:
Add following line to the file sites-available/your-site.conf
WSGIApplicationGroup %{GLOBAL}