Instruction: How to use TPUs with PyTorch?

Input:
I am trying to use TPU using pytorch_xla, but it shows import error in _XLAC.
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

import torch_xla
import torch_xla.core.xla_model as xm

ImportError                               Traceback (most recent call last)
<ipython-input-60-6a19e980152f> in <module>()
----> 1 import torch_xla
      2 import torch_xla.core.xla_model as xm

/usr/local/lib/python3.6/dist-packages/torch_xla/__init__.py in <module>()
     39 import torch
     40 from .version import __version__
---> 41 import _XLAC
     42
     43 _XLAC._initialize_aten_bindings()

ImportError: /usr/local/lib/python3.6/dist-packages/_XLAC.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at6native6einsumENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN3c108ArrayRefINS_6TensorEEE

Output:
Please try this:
!pip uninstall -y torch
!pip install torch==1.8.2+cpu -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
!pip install -q cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.8-cp37-cp37m-linux_x86_64.whl
import torch_xla

It worked for me.
Source: googlecolab/colabtools#2237


Instruction: Input dimension reshape when using PyTorch model with CoreML

Input:
I have a seq2seq model in PyTorch that I want to run with CoreML. When exporting the model to ONNX the input dimensions are fixed to the shape of the tensor used during export, and again with the conversion from ONNX to CoreML.

import torch
from onnx_coreml import convert

x = torch.ones((32, 1, 1000))  # N x C x W
model = Model()
torch.onnx.export(model, x, 'example.onnx')

mlmodel = convert(model='example.onnx', minimum_ios_deployment_target='13')
mlmodel.save('example.mlmodel')

For the ONNX export you can export dynamic dimension -

torch.onnx.export(
    model, x, 'example.onnx',
    input_names = ['input'],
    output_names = ['output'],
    dynamic_axes={
        'input' : {0 : 'batch', 2: 'width'},
        'output' : {0 : 'batch', 1: 'owidth'},
    }
)

But this leads to a RunTimeWarning when converting to CoreML -

  RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  Blob with zero size found:

For inference in CoreML I would like the batch (first) and width (last) dimension to either be dynamic or have the ability to statically change them.

Is that possible?

Output:
The dimensions of the input can be made dynamic in ONNX by specifying dynamic_axes for torch.onnx.export.

torch.onnx.export(
    model,
    x,
    'example.onnx',
    # Assigning names to the inputs to reference in dynamic_axes
    # Your model only has one input: x
    input_names=["input"],
    # Define which dimensions should be dynamic
    # Names of the dimensions are optional, but recommended.
    # Could just be: {"input": [0, 2]}
    dynamic_axes={"input": {0: "batch", 2: "width"}}
)

Now the exported model accepts inputs of size [batch, 1, width], where batch and width are dynamic.


Instruction: getting unicode decode error while trying to load pre-trained model using torch.load(PATH)

Input:
Trying to load a ResNet 18 pre-trained model using the torch.load(PATH) but getting Unicode decode error please help.

Traceback (most recent call last):
  File "main.py", line 312, in <module>
    main()
  File "main.py", line 138, in main
    checkpoint = torch.load(args.resume)
  File "F:\InsSoft\Anaconda\lib\site-packages\torch\serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "F:\InsSoft\Anaconda\lib\site-packages\torch\serialization.py", line 773, in _legacy_load
    result = unpickler.load()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte

Output:
This error hits whenever the model is pretrained on torch version < 0.4 but using torch version > 0.4 for testing / resuming.
so use checkpoint = torch.load(args.resume,encoding='latin1')


Instruction: How to know node/feature contributions?

Input:
I'm working on GCN (Graph Convolutional Network) in PyTorch, in my application: a patient is a graph, nodes represent its genes, for each gene I have 2 features (gene structure and expression value).

The task is I'm doing a regression model to predict the risk of each patient to get a disease.

My question is,

1- how to know which nodes (genes) contribute to the prediction?

2- and which feature of the 2 that I have (gene structure and expression value) contribute to the prediction?

Any suggestions/ideas would be helpful, thanks.

Output:
I am suggesting possibly the simplest solution. However, it can work well.

According to your description of the problem, you want to learn the graph (that represents a patient) representation which can be used to predict the risk of getting a disease. As we know, GCN (graph convolution network) can provide vector representations for each node in the graph.

All the node representations can be turned into a single vector representation which would represent the entire graph and this can be done in many ways. For example, you can use max-pooling or self-attentive pooling. In both ways, you can identify which nodes contributed most to the final prediction.

For example, in self-attentive pooling, every node gets a weight and the single vector representation is a weighted vector representation. So, the weights can indicate the nodes' contribution. If we use max-pooling, then we can count how many of the features from a node is pooled while applying the max-pooling. The count itself can indicate the contribution.

  Which feature of the 2 that I have (gene structure and expression value) contribute to the prediction?

The same above idea you can apply. For example, you can have learnable weights for the 2 features to combine them while computing the single vector representation of the graph.


Instruction: pytorch model not updating

Input:
I put my training code below. I am using  torch.optim.SGD as optimizer. I thought optimizer.step() would be doing the update but the model accuracy seems to stay the same. My friend said he didn't use the optimizer.step() and his works fine.
I tried taking it out, still the same result. What can I be doing wrong?
I don't think there's a problem with the accuracy calculation.

class FNet(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.fc1 = nn.Linear(128*256, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 2)

    def forward(self, X):
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = self.fc3(X)
        return F.softmax(X, dim=1)

def main():

    learning_rate = 0.01
    model = FNet()
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=5e-04) # you can play with momentum and weight_decay parameters as well
    accs = [0,]

    for epoch in range(max_epoch):

        train(epoch, model, optimizer, train_batch)

        acc = test(model, val_batch)
        accs.append(acc)

def train(epoch, model, optimizer, trainloader):
    model.train()
    optimizer.zero_grad()
    for batch_idx, (data, labels) in enumerate(trainloader):

        outputs = model(data)
        loss = F.nll_loss(outputs, labels)
        loss.backward()

        optimizer.step()

def test(model, testloader):
    correct = 0
    total = 0
    model.eval()

    for batch_idx, (data, labels) in enumerate(testloader):

            outputs = model(data.view(-1,128*256))

            for sample_idx,output in enumerate(outputs):

                if torch.argmax(output) == labels[sample_idx]:
                    correct = correct + 1
                total = total + 1

    accuracy = correct/total
    return accuracy

Output:
I think this line should be under your for loop
optimizer.zero_grad(). You need to clear the parameter gradients after each loop.

try this

def train(epoch, model, optimizer, trainloader):
    model.train()
    for batch_idx, (data, labels) in enumerate(trainloader):
        optimizer.zero_grad()
        outputs = net(data)
        loss = F.nll_loss(outputs, labels)
        loss.backward()

        optimizer.step()


Instruction: Is it mandatory in pytorch to add modules to ModuleList to access its parameters

Input:
I read some posts about ModuleList and all of them said that adding modules to ModuleList gives access to parameters of the Neural Network but in “Training a classifier” example of 60 mins blitz pytorch tutorial the modules are not added to any ModuleList and still the parameters could be accessed using

optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

This is confusing. Please clarify how the parameters are accessible even though the modules have not been added to any ModuleList

class mmodel(nn.Module):
    def __init__(self):
        super(mmodel,self).__init__()
        self.lst=[]
        self.lst.append(nn.Linear(1,1))
        self.lst.append(nn.Linear(1,1))
        self.mlist = nn.ModuleList([nn.Linear(2,1,bias=False),nn.Linear(3,1)])
    def forward(self,x):
        for m in self.lst:
            print(type(m))
            x = m(x)
        for m in self.mlist:
            print(type(m))

mm = mmodel()
mm(X)
print(list(mm.parameters()))

<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
[Parameter containing:
tensor([[0.2302, 0.3712]], requires_grad=True), Parameter containing:
tensor([[-0.3451, -0.0274,  0.3990]], requires_grad=True), Parameter containing:
tensor([0.3258], requires_grad=True)]

As evident from the above outputs the parameters of modules added in a python list are not visible in the forward method. Only the parameters of the modules added in the ModuleList are visible in the forward method

Output:
Calling module.parameters() lists all nn.Parameter of the module. Concretely, every attribute on the module that is an instance of nn.Parameter will be in that list. Additionally to listing all the parameters of that module, it will also list all parameters of the submodules (unless module.parameters(recurse=False) is used). That means it will also collect all parameters of every attribute that is an instance of nn.Module, which includes all subclasses.

However, if you assign an ordinary list of modules to your module, they won't be included since that is an instance of list, but not nn.Module. For convenience, nn.ModuleList can be used in place of a regular Python list. nn.ModuleList is an instance of nn.Module but acts similar to a list, albeit much more restricted.

Let's take a look at an example to understand what's considered a submodule:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        # This is an nn.Module, therefore it is considered a submodule
        self.single_linear = nn.Linear(10, 20)

        # This is a list, that happens to contain nn.Modules.
        # This is not a submodule.
        self.linears = [nn.Linear(20, 10), nn.Linear(20, 30), nn.Linear(20, 40)]

        # A list with ints, clearly not a submodule either.
        self.numbers = [0, 1, 2, 3]

        # This is a module, but acts like an ordinary list
        # So it's a submodule as well.
        self.linears_module_list = nn.ModuleList(
            [nn.Linear(20, 10), nn.Linear(20, 30), nn.Linear(20, 40)]
        )

In this model we have assigned 4 new attributes, single_linear, linears, numbers and linears_module_list. When they are assigned, PyTorch checks whether they are instances of nn.Module, and if they are, they will be registered as submodules.

We can verify that (leaving out numbers, because that is pretty clear):

model = Model()

isinstance(model.single_linear, nn.Module)  # => True

isinstance(model.linears, nn.Module)  # => False
isinstance(model.linears, list)       # => True

isinstance(model.linears_module_list, nn.Module)  # => True
isinstance(model.linears_module_list, list)       # => False

Only single_linear and linears_module_list are in fact submodules, as they are the only attributes that are instances of nn.Module. The submodules can be seen with model._modules, but that's an implementation detail and you shouldn't rely on it.

# The registered modules
model._modules
# => OrderedDict([
#       ('single_linear', Linear(in_features=10, out_features=20, bias=True)),
#       ('linears_module_list', ModuleList(
#           (0): Linear(in_features=20, out_features=10, bias=True)
#           (1): Linear(in_features=20, out_features=30, bias=True)
#           (2): Linear(in_features=20, out_features=40, bias=True)
#       )
#   )])


Instruction: BERT encoding layer produces same output for all inputs during evaluation (PyTorch)

Input:
I don't understand why my BERT model returns the same output during evaluation. The output of my model during training seems correct, as the values were different, but is totally the same during evaluation.

Here is my BERT model class

class BERTBaseUncased(nn.Module):
    def __init__(self):
        super(BERTBaseUncased, self).__init__()
        self.bert = BertModel.from_pretrained("bert-base-uncased")
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768, 4)

    def forward(self, ids, mask, token_type_ids):
        _, o2 = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids) # Use one of the outputs
        bo = self.bert_drop(o2)
        return self.out(bo)

My dataset class

class BERTDataset:
    def __init__(self, review, target, tokenizer, classes=4):
        self.review = review
        self.target = target
        self.tokenizer = tokenizer
        self.max_len = max_len
        self.classes = classes

    def __len__(self):
        return len(self.review)

    def __getitem__(self, item):
        review = str(self.review)
        review = " ".join(review.split())

        inputs = self.tokenizer.encode_plus(review, None, add_special_tokens=True, max_length= self.max_len,
                                            pad_to_max_length=True, return_token_type_ids=True,
                                            return_attention_masks=True)

        ids = inputs["input_ids"]
        mask = inputs["attention_mask"]
        token_type_ids = inputs["token_type_ids"]

        return {
            'ids': torch.tensor(ids, dtype=torch.long),
            'mask': torch.tensor(mask, dtype=torch.long),
            'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),
            'targets': torch.tensor(to_categorical(self.target[item], self.classes), dtype=torch.float)
        }

My evaluation function

def eval_fn(data_loader, model, device):
    model.eval()

    total_loss = 0.0

    with torch.no_grad():
        for bi, d in tqdm(enumerate(data_loader), total=len(data_loader)):
            ids = d['ids']
            token_type_ids = d['token_type_ids']
            mask = d['mask']
            targets = d['targets']

            ids = ids.to(device, dtype=torch.long)
            token_type_ids = token_type_ids.to(device, dtype=torch.long)
            mask = mask.to(device, dtype=torch.long)
            targets = targets.to(device, dtype=torch.float)

            outputs = model(
                ids=ids,
                mask=mask,
                token_type_ids=token_type_ids
            )

            loss = loss_fn(outputs, targets)
            total_loss += loss.item()

And my training function

def train_fn(data_loader, model, optimizer, device, scheduler):
    model.train()

    total_loss = 0.0

    for bi, d in tqdm(enumerate(data_loader), total=len(data_loader)):
        ids = d['ids']
        token_type_ids = d['token_type_ids']
        mask = d['mask']
        targets = d['targets']

        ids = ids.to(device, dtype=torch.long)
        token_type_ids = token_type_ids.to(device, dtype=torch.long)
        mask = mask.to(device, dtype=torch.long)
        targets = targets.to(device, dtype=torch.float)

        optimizer.zero_grad()

        outputs = model(
            ids=ids,
            mask=mask,
            token_type_ids=token_type_ids
        )

        loss = loss_fn(outputs, targets)
        total_loss += loss.item()
        loss.backward()

        optimizer.step()
        scheduler.step()

    return total_loss/len(data_loader)

Thanks!

Output:
In case anybody else has the problem, perhaps you forgot to use one of the recommended learning rates from the official paper:  5e-5, 3e-5, 2e-5
Gradients seem to polarize if the learning rate is too high, such as 0.01, causing repeatedly the same logits for the val set.


Instruction: Difficulty in Implementing a simple single-layer RNN using Pytorch's base class “nn.Linear” class

Input:
While working on making a simple RNN using Pytorch nn.linear function. So firstly I initialized my weights as

self.W_x = nn.Linear(self.input_dim, self.hidden_dim, bias=True)
self.W_h = nn.Linear(self.hidden_dim, self.hidden_dim, bias=True)

Now in the main step where I am getting the result of the current state by using the previous state and the values of the weights using this code statement

h_t = np.tanh((inp * self.W_x) + (prev_h * self.W_h))

So here I am getting the python error as shown below

TypeError: mul(): argument 'other' (position 1) must be Tensor, not Linear

Can anyone help me with his regards...

Output:
Your W_x and W_h are not weights, but linear layers, which use a weight and bias (since bias=True). They need to be called as a function.

Furthermore, you cannot use NumPy operations with PyTorch tensors, but if you convert your tensors to NumPy arrays you can't back propagate through them, since only PyTorch operations are tracked in the computational graph. There is no need for np.tanh anyway, as PyTorch has torch.tanh as well.

h_t = torch.tanh(self.W_x(inp) + self.W_h(prev_h))


Instruction: How to handle class imbalance in multi-label classification using pytorch

Input:
We are attempting to implement multi-label classification using CNN in pytorch. We have 8 labels and around 260 images using a 90/10 split for train/validation sets.

The classes are highly imbalanced with the most frequent class occurring in over 140 images. On the other hand, the least frequent class occurs in less than 5 images.

We attempted BCEWithLogitsLoss function initially that led to the model predicting the same label for all images.

We then implemented a focal loss approach to handle class imbalance as follows:

    import torch.nn as nn
    import torch

    class FocalLoss(nn.Module):
        def __init__(self, alpha=1, gamma=2):
            super(FocalLoss, self).__init__()
            self.alpha = alpha
            self.gamma = gamma

        def forward(self, outputs, targets):
            bce_criterion = nn.BCEWithLogitsLoss()
            bce_loss = bce_criterion(outputs, targets)
            pt = torch.exp(-bce_loss)
            focal_loss = self.alpha * (1 - pt) ** self.gamma * bce_loss
            return focal_loss

This resulted in the model predicting empty sets (no labels) for every image since it could not get a greater than 0.5 confidence for any classes.

Is there a approach in pytorch to help address this situation?

Output:
There's basically three ways of dealing with this.

Discard data from the more common class
Weight minority class loss values more heavily
Oversample the minority class

Option 1 is implemented by selecting the files you include in your Dataset.

Option 2 is implemented with the pos_weight parameter for BCEWithLogitsLoss

Option 3 is implemented with a custom Sampler passed to your Dataloader

For deep learning, oversampling typically works best.


Instruction: Weird behavior when calling cuda() on different tensors in pytorch

Input:
I am trying to train a pytorch neural network on a GPU device. In order to do so, I load my inputs and network onto the default cuda enabled GPU decive. However, when I load my inputs, the model's weights do not stay cuda tensors. Here is my train function

def train(network: nn.Module, name: str, learning_cycles: dict, num_epochs):
    # check we have a working gpu to train on
    assert(torch.cuda.is_available())
    # load model onto gpu
    network = network.cuda()
    # load train and test data with a transform
    transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    train_set = torchvision.datasets.CIFAR10(root='./data', train=True,
 download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=128,
shuffle=True, num_workers=2)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(network.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(num_epochs):
        for i, data in enumerate(train_loader, 0):
            inputs, labels = data
            # load inputs and labels onto gpu
            inputs, labels = inputs.cuda(), labels.cuda()
            optimizer.zero_grad()
            outputs = network(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

When calling train, I get the following error.

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Interestingly, when I delete the line inputs, labels = inputs.cuda(), labels.cuda() I get the error RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

I would very much like to train my network, and I have searched the internet to no avail. Any good ideas?

Output:
Given that a device mismatch crops up regardless of the device the inputs are on, it's likely that some of your model's parameters are not being moved over to the GPU when you call network = network.cuda(). You have model parameters on both the CPU and the GPU.

Post your model code. It's likely you have a Pytorch module in an incorrect container.

Lists of modules should be in a nn. ModuleList. Modules in a Python list will not transfer over. Compare

layers1 = [nn.Linear(256, 256), nn.Linear(256, 256), nn.Linear(256, 256)]
layers2 = nn.ModuleList([nn.Linear(256, 256), nn.Linear(256, 256), nn.Linear(256, 256)])

If you called model.cuda() on a model with the above two lines, the layers in layer1 would remain on the CPU, while the layers in layer2 would be moved to the GPU.

Similarly, a list of nn.Parameter objects should be contained in an nn.ParameterList object.

There's also nn. ModuleDict and nn.ParameterDict for dictionary containers.


Instruction: Training models interactively in Pytorch

Input: I need to train two models in parallel. Each model has a different activation function with trainable parameters. I want to train model one and model two in the way that the parameters of the activation function from model one (e.g., alpha1) is separated from the parameters in model two (e.g., alpha2) by a gap of 2; i.e., |alpha_1 - alpha_2| > 2. I wonder how I could include it into the loss function for training.

Output:
Example module definition

I will use torch.nn.PReLU as parametric activation you talk about.
get_weight created for convenience.

import torch

class Module(torch.nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.input = torch.nn.Linear(in_features, 2 * in_features)
        self.activation = torch.nn.PReLU()
        self.output = torch.nn.Linear(2 * in_features, out_features)

    def get_weight(self):
        return self.activation.weight

    def forward(self, inputs):
        return self.output(self.activation(self.inputs(inputs)))

Modules and setup

Here I'm using one optimizer to optimize parameters of both modules you talk about. criterion can be mean squared error, cross entropy or any other thing you need.

module1 = Module(20, 1)
module2 = Module(20, 1)

optimizer = torch.optim.Adam(
    itertools.chain(module1.parameters(), module2.parameters())
)
critertion = ...

Training

Here is a single step, you should pack it in a for-loop over your data as is usually done, hopefully it's enough for you to get the idea:

inputs = ...
targets = ...

output1 = module1(inputs)
output2 = module2(inputs)

loss1 = criterion(output1, targets)
loss2 = criterion(output2, targets)

total_loss = loss1 + loss2
total_loss += torch.nn.functional.relu(
    2 - torch.abs(module1.get_weight() - module2.get_weight()).sum()
)
total_loss.backward()

optimizer.step()

This line is what you are after in this case:

total_loss += torch.nn.functional.relu(
    2 - torch.abs(module1.get_weight() - module2.get_weight()).sum()
)

relu is used so the network won't reap infinite benefit solely from creating divergent weights. If there wasn't one, loss would become negative the greater the difference between weights would be. In this case the bigger the difference the better, but it makes no difference after the gap is greater or equal to 2.

You may have to increase 2 to 2.1 or something if you have to pass the threshold of 2 as the incentive to optimize the value when it's close to 2.0 would be small.

Edit

Without explicitly given threshold it might be hard, but maybe something like this would work:

total_loss = (
    (torch.abs(module1) + torch.abs(module2)).sum()
    + (1 / torch.abs(module1) + 1 / torch.abs(module2)).sum()
    - torch.abs(module1 - module2).sum()
)

It's kinda hackish for the network, but might be worth a try (if you apply additional L2 regularization).

In essence, this loss will have optimum at -inf, +inf pairs of weights in the corresponding positions and never will be smaller than zero.

For those weights

weights_a = torch.tensor([-1000.0, 1000, -1000, 1000, -1000])
weights_b = torch.tensor([1000.0, -1000, 1000, -1000, 1000])

Loss for each part will be:

(torch.abs(module1) + torch.abs(module2)).sum() # 10000
(1 / torch.abs(module1) + 1 / torch.abs(module2)).sum() # 0.0100
torch.abs(module1 - module2).sum() # 10000

In this case network can reap easy benefits just by making the weights greater with opposite signs in both modules and disregard what you want to optimize (large L2 on weights of both modules might help and I think optimum value would be 1/-1 in case L2's alpha is equal to 1) and I suspect the network might be highly unstable.

With this loss function if the network gets a sign of large weight wrong it will be heavily penalized.

In this case you would be left with L2 alpha parameter to tune to make it work, which is not that strict, but still requires a hyperparameter choice.


Instruction: trouble importing Pytorch in Jupyter notebook

Input:
Iam new to deep learning and Iam trying to import Pytorch on Jupyter Notebook.
I installed Pytorch with the following lines of code in Anaconda Prompt.

conda create -n pytorch_p37 python=3.7
conda activate pytorch_p37
conda install pytorch torchvision -c pytorch
conda install jupyter
conda list

it all executed well.
but on importing pytorch it shows errors.

import torch

this error below:-

OSError: [WinError 126] The specified module could not be found

error showing image

Output:
!pip install torch

It worked for me in a Anaconda's Jupyter notebook.


Instruction: How to solve UserWarning: Using a target size (torch.Size([])) that is different to the input size (torch.Size([1]))?

Input:
I am trying to run code from a book I purchased about reinforcement learning in Pytorch.
The code should work according to the book, but for me the model doesn't converge and the reward remains negative. It also get the following user warning:

/home/user/.local/lib/python3.6/site-packages/ipykernel_launcher.py:30: UserWarning: Using a target size (torch.Size([])) that is different to the input size (torch.Size([1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

I am a complete beginner in Pytorch, but I assume a size([]) is not a valid tensor size? I think something is going wrong in the code, but after trying to work trough it for a while, I have yet to find anything. I also messaged the book publisher some time ago, but I unfortunately did not hear back from them.

That's why I would like to ask here if anyone has ever seen this error and perhaps knows how to fix it?

The code is for implementing A2C reinforcement learning on a mountain car gym environment. I can also be found here: https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter08/chapter8/actor_critic_mountaincar.py

'''
Source codes for PyTorch 1.0 Reinforcement Learning (Packt Publishing)
Chapter 8: Implementing Policy Gradients and Policy Optimization
Author: Yuxi (Hayden) Liu
'''

import torch
import gym
import torch.nn as nn
import torch.nn.functional as F

env = gym.make('MountainCarContinuous-v0')

class ActorCriticModel(nn.Module):
    def __init__(self, n_input, n_output, n_hidden):
        super(ActorCriticModel, self).__init__()
        self.fc = nn.Linear(n_input, n_hidden)
        self.mu = nn.Linear(n_hidden, n_output)
        self.sigma = nn.Linear(n_hidden, n_output)
        self.value = nn.Linear(n_hidden, 1)
        self.distribution = torch.distributions.Normal

    def forward(self, x):
        x = F.relu(self.fc(x))
        mu = 2 * torch.tanh(self.mu(x))
        sigma = F.softplus(self.sigma(x)) + 1e-5
        dist = self.distribution(mu.view(1, ).data, sigma.view(1, ).data)
        value = self.value(x)
        return dist, value

class PolicyNetwork():
    def __init__(self, n_state, n_action, n_hidden, lr=0.001):
        self.model = ActorCriticModel(n_state, n_action, n_hidden)
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, returns, log_probs, state_values):
        """
        Update the weights of the Actor Critic network given the training samples
        @param returns: return (cumulative rewards) for each step in an episode
        @param log_probs: log probability for each step
        @param state_values: state-value for each step
        """
        loss = 0
        for log_prob, value, Gt in zip(log_probs, state_values, returns):
            advantage = Gt - value.item()
            policy_loss = - log_prob * advantage

            value_loss = F.smooth_l1_loss(value, Gt)

            loss += policy_loss + value_loss

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    def predict(self, s):
        """
        Compute the output using the continuous Actor Critic model
        @param s: input state
        @return: Gaussian distribution, state_value
        """
        self.model.training = False
        return self.model(torch.Tensor(s))

    def get_action(self, s):
        """
        Estimate the policy and sample an action, compute its log probability
        @param s: input state
        @return: the selected action, log probability, predicted state-value
        """
        dist, state_value = self.predict(s)
        action = dist.sample().numpy()
        log_prob = dist.log_prob(action[0])
        return action, log_prob, state_value

def actor_critic(env, estimator, n_episode, gamma=1.0):
    """
    continuous Actor Critic algorithm
    @param env: Gym environment
    @param estimator: policy network
    @param n_episode: number of episodes
    @param gamma: the discount factor
    """
    for episode in range(n_episode):
        log_probs = []
        rewards = []
        state_values = []
        state = env.reset()

        while True:
            state = scale_state(state)
            action, log_prob, state_value = estimator.get_action(state)
            action = action.clip(env.action_space.low[0],
                                 env.action_space.high[0])
            next_state, reward, is_done, _ = env.step(action)

            total_reward_episode[episode] += reward
            log_probs.append(log_prob)
            state_values.append(state_value)
            rewards.append(reward)

            if is_done:
                returns = []

                Gt = 0
                pw = 0

                for reward in rewards[::-1]:

                    Gt += gamma ** pw * reward
                    pw += 1
                    returns.append(Gt)

                returns = returns[::-1]
                returns = torch.tensor(returns)
                returns = (returns - returns.mean()) / (returns.std() + 1e-9)

                estimator.update(returns, log_probs, state_values)
                print('Episode: {}, total reward: {}'.format(episode, total_reward_episode[episode]))

                break

            state = next_state

import sklearn.preprocessing
import numpy as np

state_space_samples = np.array(
    [env.observation_space.sample() for x in range(10000)])
scaler = sklearn.preprocessing.StandardScaler()
scaler.fit(state_space_samples)

def scale_state(state):
    scaled = scaler.transform([state])
    return scaled[0]

n_state = env.observation_space.shape[0]
n_action = 1
n_hidden = 128
lr = 0.0003
policy_net = PolicyNetwork(n_state, n_action, n_hidden, lr)

n_episode = 200
gamma = 0.9
total_reward_episode = [0] * n_episode

actor_critic(env, policy_net, n_episode, gamma)

Output:
size([]) is valid, but it represents a single value, not an array, whereas size([1]) is a 1 dimensional array containing only one item item. It is like comparing 5 to [5]. One solution to this is
            returns = returns[::-1]
            returns_amount = len(returns)
            returns = torch.tensor(returns)
            returns = (returns - returns.mean()) / (returns.std() + 1e-9)
            returns.resize_(returns_amount, 1)

This converts returns into a 2-dimensional array, so that each Gt you get from it will be a 1-dimensional array, not a float.


Instruction: How to build a model to predict a graph (not a image) in time series?

Input: There is an adjacent matrix dataset that is based on time series. I would like to know if it is possible to build a neural network model to predict tn time point's matrix by using the previous time-series data. In my opinion, traditional models such as CNN may not fit for the sparse matrix graph.

Output:
Maybe you should give a look at Graph Neural Networks (specialy Spatial-Temporal Graph Networks). They use temporal information about graphs and its adjacency matrix to predict future nodes states, such values in the next-step.

You can read this survey paper as a start point and follow its cited works therefore.


Instruction: Autoencoder to encode features/categories of data

Input:
My question is regarding the use of autoencoders (in PyTorch). I have a tabular dataset with a categorical feature that has 10 different categories. Names of these categories are quite different - some names consist of one word, some of two or three words. But all in all I have 10 unique category names. What I'm trying to do is to create an autoencoder which will encode names of these categories - for example, if I have a category named 'Medium size class', I want to see if it is possible to train autoencoder to encode this name as something like 'mdmsc' or something like that. The use of it would be to found out which data points are hard to encode or not typical or something like that. I tried to adapt autoencoder architectures from various tutorials online however nothing seems to work for me or I simply do not know how to use them as they are all about images. Maybe someone has any idea how this type of autoencoder might be accomplished if it is at all possible?

Edit: here's the model I have so far (I just tried to adapt some architectures I found online):

class Autoencoder(nn.Module):

def __init__(self, input_shape, encoding_dim):
    super(Autoencoder, self).__init__()

    self.encode = nn.Sequential(
        nn.Linear(input_shape, 128),
        nn.ReLU(True),
        nn.Linear(128, 64),
        nn.ReLU(True),
        nn.Linear(64, encoding_dim),
    )

    self.decode = nn.Sequential(
        nn.Linear(encoding_dim, 64),
        nn.ReLU(True),
        nn.Linear(64, 128),
        nn.ReLU(True),
        nn.Linear(128, input_shape)
    )

def forward(self, x):
    x = self.encode(x)
    x = self.decode(x)
    return x

model = Autoencoder(input_shape=10, encoding_dim=5)

And also I use LabelEncoder() and then OneHotEncoder()to give these features/categories I mentioned numerical form. However, after training, output is the same as was input (no changes on the category name) but when I try to use only encoder part I'm unable to apply LabelEncoder() and then OneHotEncoder() because of dimension issues. I feel like maybe I can do something differently at the beginning, then I try to give those features numerical form, however I'm not sure what should I do.

Output:
First you will need to set up a train_loader depending on your data that will iterate over your data points.

Then you need to figure out what kind of loss you are going to use and optimizer:

# mean-squared error loss
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=.001)  #learning rate depend on your task

Once you have that ready, you can train your autoencoder with basic steps:

for epoch in range(epochs):
        for features in train_loader:

            optimizer.zero_grad()
            outputs = model(batch_features)
            train_loss = criterion(outputs, features)
            train_loss.backward()
            optimizer.step()

Once model is done training you can examine embeddings using:

embedding = model.encode(your_input)


Instruction: Pytorch runtime error: Cuda Out of memory. Works fine with jupyter notebook but doesn't as a script

Input:
I have a special kind of problem. I am able to run the code in jupyter notebook perfectly fine with no OOM error. However when i run the same code as a script in linux it gives me the OOM error. Has anyone have the same issue. I tried gc.collect() and torch.cuda.empty_cache() inside the code and nothing helps.

It always gives me this error.
RuntimeError: CUDA out of memory. Tried to allocate 1.30 GiB (GPU 0; 7.79 GiB total capacity; 4.80 GiB already allocated; 922.69 MiB free; 6.12 GiB reserved in total by PyTorch

The code:

def lemmatize(phrase):
    """Return lematized words"""
    spa = spacy.load("en_core_web_sm")
    return " ".join([word.lemma_ for word in spa(phrase)])

def reading_csv(path_to_csv):
    """Return text column in csv"""
    data = pd.read_csv(path_to_csv)
    ctx_paragraph = []
    for txt in data['text']:
        if not pd.isna(txt):
            ctx_paragraph.append(txt)
    return ctx_paragraph

def processing_question(ques, paragraphs, domain_lemma_cache, domain_pickle):
    """Return answer"""
    #Lemmatizing whole csv text column
    lemma_cache = domain_lemma_cache
    if not os.path.isfile(lemma_cache):
        lemmas = [lemmatize(par) for par in tqdm(paragraphs)]
        df = pd.DataFrame(data={'context': paragraphs, 'lemmas': lemmas})
        df.to_feather(lemma_cache)
    df = pd.read_feather(lemma_cache)
    paragraphs = df.context
    lemmas = df.lemmas
    #Vectorizor cache
    if not os.path.isfile(VEC_PICKLE_LOC):
        vectorizer = TfidfVectorizer(
            stop_words='english', min_df=5, max_df=.5, ngram_range=(1, 3))
        vectorizer.fit_transform(lemmas)
        pickle.dump(vectorizer, open(VEC_PICKLE_LOC, "wb"))
    #Vectorized lemmas cache cache
    if not os.path.isfile(domain_pickle):
        tfidf = vectorizer.fit_transform(lemmas)
        pickle.dump(tfidf, open(domain_pickle, "wb"))
    vectorizer = pickle.load(open(VEC_PICKLE_LOC, "rb"))
    tfidf = pickle.load(open(domain_pickle, "rb"))
    question = ques
    query = vectorizer.transform([lemmatize(question)])
    (query > 0).sum(), vectorizer.inverse_transform(query)
    scores = (tfidf * query.T).toarray()
    results = (np.flip(np.argsort(scores, axis=0)))
    qapipe = pipeline('question-answering',
                      model='distilbert-base-uncased-distilled-squad',
                      tokenizer='bert-base-uncased',
                      device=0)
    candidate_idxs = [(i, scores[i]) for i in results[0:10, 0]]
    contexts = [(paragraphs[i], s) for (i, s) in candidate_idxs if s > 0.01]
    question_df = pd.DataFrame.from_records([{
        'question': question,
        'context':  ctx
    } for (ctx, s) in contexts])
    preds = qapipe(question_df.to_dict(orient="records"))
    answer_df = pd.DataFrame.from_records(preds)
    answer_df["context"] = question_df["context"]
    answer_df = answer_df.sort_values(by="score", ascending=False)
    return answer_df

Output:
I had a similar thing happen to me recently.
I would run my model in a Jupyter notebook, on a AWS EC2 p2.xlarge instance, and the model would run correctly.  Then, I would ssh into the same instance, and re-run a .py script of the same model, and receive the OOM errors that you described.
All I had to do was reset the kernal of the Jupyter notebook, to get the .py script to work.


Instruction: Kernel size can't be greater than actual input size

Input:
I have a data with depth = 3 and I want to pass it through 3 convolution layers with 3x3x3 kernels each.
My current code is below. The first input is
[batch_size=10, in_channels=1, depth=3, height=128, width=256]
and I notice after the first conv3d layer the output is [10,8,1,126,254]. Obviously it has now depth 1 and doesn't accept it for another 3x3x3 layer. How can I achieve this?

class CNet(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.conv1 = nn.Conv3d(1, 8, kernel_size=3, stride=1, padding=0)
        self.conv2 = nn.Conv3d(8, 16, kernel_size=3, stride=1, padding=0)
        self.conv3 = nn.Conv3d(16, 32, kernel_size=3, stride=1, padding=0)
        self.fc1 = nn.Linear(value, 2)

    def forward(self, X):
        X = F.relu(self.conv1(X))
        X = F.relu(self.conv2(X))
        X = F.max_pool2d(X,2)
        X = self.conv3(X)
        X = F.max_pool2d(X,2)
        X = self.fc1(X)
        return F.softmax(X,dim =1)

Output:
You need to use padding. If you only want to pad the input for the convolutions after the first one and only in the depth dimensions to get the minimum dimension of 3, you would use padding=(1, 0, 0) (it's 1 because the same padding is applied to both sides, i.e. (padding, input, padding) along that dimension).

self.conv2 = nn.Conv3d(8, 16, kernel_size=3, stride=1, padding=(1, 0, 0))
self.conv3 = nn.Conv3d(16, 32, kernel_size=3, stride=1, padding=(1, 0, 0))

However, it is common to use padding=1 for all dimensions when using kernel_size=3, because that keeps the dimensions unchanged, which makes it much easier to build deeper network, as you don't need to worry about the sizes suddenly getting too small, as it happened already for your depth dimension. Also when no padding is used, the corners are only included in a single calculation, whereas all other elements contribute to multiple calculations. It is recommended to use kernel_size=3 and padding=1 for all your convolutions.

self.conv1 = nn.Conv3d(1, 8, kernel_size=3, stride=1, padding=1)
self.conv2 = nn.Conv3d(8, 16, kernel_size=3, stride=1, padding=1)
self.conv3 = nn.Conv3d(16, 32, kernel_size=3, stride=1, padding=1)


Instruction: Unable to import torch (ImportError: libcudart.so.10.0)

Input:
I'm currently working on a Nvidia Jetson Nano and I'm not very familiar with Linux. I am trying to run a python file which imports a package called torch. I have installed it alongside with  torchvision while following the instructions from NVIDIA here.

When I run pip list on my terminal, I am able to see torch listed as one of the packages installed. However, I am unable to run the python file due to the error seen below. When I try to run it on python shell, the same error pops up.
FYI: Previously it had issues as the system was using python 2 by default but I have already fixed the path by switching to python 3 by editing the .bashrc file.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

I have tried uninstalling and installing via pip but to no avail. When I try to install the pytorch package (following the instructions from a github repo here), an error occurs as seen below and it is due to the same issue. It is able to detect that the torch package is installed but there seems to be an internal issue.

Requirement already satisfied: torch==1.4.0 from file:///home/jiayi/jetson-inference/build/torch-1.4.0-cp36-cp36m-linux_aarch64.whl in /home/jiayi/.local/lib/python3.6/site-packages (1.4.0)
[jetson-inference]  cloning torchvision...
[sudo] password for jiayi:
Cloning into 'torchvision-36'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (56/56), done.
remote: Total 8219 (delta 37), reused 29 (delta 15), pack-reused 8148
Receiving objects: 100% (8219/8219), 10.22 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (5631/5631), done.
[jetson-inference]  building torchvision for Python 3.6...
Traceback (most recent call last):
  File "setup.py", line 14, in <module>
    import torch
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

[jetson-inference]  installation complete, exiting with status code 0
[jetson-inference]  to run this tool again, use the following commands:

    $ cd <jetson-inference>/build
    $ ./install-pytorch.sh

Output: I meet the exact same problem. The problem seems to be cuda 10.2. Downgrading to 10.0 does not help either. Probably the solution is to manually install everything from Jetpack and making sure that the cuda version to be installed is 10.0.


Instruction: How to fine tune BERT on unlabeled data?

Input:
I want to fine tune BERT on a specific domain. I have texts of that domain in text files. How can I use these to fine tune BERT?
I am looking here currently.

My main objective is to get sentence embeddings using BERT.

Output:
The important distinction to make here is whether you want to fine-tune your model, or whether you want to expose it to additional pretraining.
The former is simply a way to train BERT to adapt to a specific supervised task, for which you generally need in the order of 1000 or more samples including labels.
Pretraining, on the other hand, is basically trying to help BERT better "understand" data from a certain domain, by basically continuing its unsupervised training objective ([MASK]ing specific words and trying to predict what word should be there), for which you do not need labeled data.
If your ultimate objective is sentence embeddings, however, I would strongly suggest you to have a look at Sentence Transformers, which is based on a slightly outdated version of Huggingface's transformers library, but primarily tries to generate high-quality embeddings. Note that there are ways to train with surrogate losses, where you try to emulate some form ofloss that is relevant for embeddings.
Edit: The author of Sentence-Transformers recently joined Huggingface, so I expect support to greatly improve over the upcoming months!


Instruction: TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

Input:
The following is from a project that I'm doing in Udacity on Deep Learning. The project is on Generating TV scripts. The error that i encountered is the one below.
The following function is the one after model training.

def generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len=100):
    """
    Generate text using the neural network
    param decoder: The PyTorch Module that holds the trained neural network
    param prime_id: The word id to start the first prediction
    param int_to_vocab: Dict of word id keys to word values
    param token_dict: Dict of puncuation tokens keys to puncuation values
    param pad_value: The value used to pad a sequence
    param predict_len: The length of text to generate
    return: The generated text
    """
    rnn.eval()

    # create a sequence (batch_size=1) with the prime_id
    current_seq = np.full((1, sequence_length), pad_value)
    current_seq[-1][-1] = prime_id
    predicted = [int_to_vocab[prime_id]]

    for _ in range(predict_len):
        if train_on_gpu:
            current_seq = torch.LongTensor(current_seq).cuda()
        else:
            current_seq = torch.LongTensor(current_seq)

        # initialize the hidden state
        hidden = rnn.init_hidden(current_seq.size(0))

        # get the output of the rnn
        output, _ = rnn(current_seq, hidden)

        # get the next word probabilities
        p = F.softmax(output, dim=1).data
        if(train_on_gpu):
            p = p.cpu() # move to cpu

        # use top_k sampling to get the index of the next word
        top_k = 5
        p, top_i = p.topk(top_k)
        top_i = top_i.numpy().squeeze()

        # select the likely next word index with some element of randomness
        p = p.numpy().squeeze()
        word_i = np.random.choice(top_i, p=p/p.sum())

        # retrieve that word from the dictionary
        word = int_to_vocab[word_i]
        predicted.append(word)

        # the generated word becomes the next "current sequence" and the cycle can continue
        current_seq = np.roll(current_seq, -1, 1)
        current_seq[-1][-1] = word_i

    gen_sentences = ' '.join(predicted)

    # Replace punctuation tokens
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        gen_sentences = gen_sentences.replace(' ' + token.lower(), key)
    gen_sentences = gen_sentences.replace('\n ', '\n')
    gen_sentences = gen_sentences.replace('( ', '(')

    # return all the sentences
    return gen_sentences

after this the following code is run:

# run the cell multiple times to get different results!
gen_length = 400 # modify the length to your preference
prime_word = 'jerry' # name for starting the script

pad_word = helper.SPECIAL_WORDS['PADDING']
generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)
print(generated_script)

Upon running this code, I get the following error

TypeError                                 Traceback (most recent call last)
<ipython-input-40-68a17c4d1704> in <module>()
      7 """
      8 pad_word = helper.SPECIAL_WORDS['PADDING']
----> 9 generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)
     10 print(generated_script)

3 frames
<ipython-input-39-b86c7a305356> in generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len)
     53
     54         # the generated word becomes the next "current sequence" and the cycle can continue
---> 55         current_seq = np.roll(current_seq, -1, 1)
     56         current_seq[-1][-1] = word_i
     57

<__array_function__ internals> in roll(*args, **kwargs)

/usr/local/lib/python3.6/dist-packages/numpy/core/numeric.py in roll(a, shift, axis)
   1179
   1180     """
-> 1181     a = asanyarray(a)
   1182     if axis is None:
   1183         return roll(a.ravel(), shift, 0).reshape(a.shape)

/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asanyarray(a, dtype, order)
    136
    137     """
--> 138     return array(a, dtype, copy=False, order=order, subok=True)
    139
    140

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __array__(self, dtype)
    490     def __array__(self, dtype=None):
    491         if dtype is None:
--> 492             return self.numpy()
    493         else:
    494             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Can anyone please help me out?

Output:
np.roll(current_seq, -1, 1) requires the input to be a NumPy array, but current_seq is a tensor, so it tries to convert it to a NumPy array, which fails, because the tensor is on the GPU. In order to convert it to a NumPy array, you need to have the tensor on the CPU.

current_seq = np.roll(current_seq.cpu(), -1, 1)


Instruction: Convert np array of arrays to torch tensor when inner arrays are of different sizes

Input:
I have several videos, which I have loaded frame by frame into a numpy array of arrays. For example if I have 8 videos, they are converted into an 8 dimensional numpy array of arrays where each inner array has a different dimension depending on the number of frames of the individual video. When I print

array.shape

my output is (8,)

Now I would like to create a dataloader for this data, and for that I would like to convert this numpy array into a torch tensor. However when I try to convert it using the torch.from_numpy or even simply the torch.tensor functions I get the error

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.

which I assume is because my inner arrays are of different sizes. One possible solution is to artificially add a dimension to my videos to make them be of the same size and then use np.stack but that may lead to possible problems later on. Is there any better solution?

Edit: Actually adding a dimension won't work because np.stack requires all dimensions to be the same.

Edit: Sample Array would be something like:

[ [1,2,3], [1,2], [1,2,3,4] ]

This is stored as a (3,) shaped np array. The real arrays are actually 4-dimensional( Frames x Height x Width x Channels), so this is just an example.

Output:
You can use rnn util function pad_sequence to make them same size.

ary
array([list([1, 2, 3]), list([1, 2]), list([1, 2, 3, 4])], dtype=object)

from torch.nn.utils.rnn import pad_sequence
t = pad_sequence([torch.tensor(x) for x in ary], batch_first=True)

t
tensor([[1, 2, 3, 0],
        [1, 2, 0, 0],
        [1, 2, 3, 4]])
t.shape
torch.Size([3, 4])


Instruction: Can I use a PyTorch or Tensorflow project on a machine without GPU?

Input:
I'm a noob when it comes to Python and machine learning. I'm trying to run two different projects that have to do with something called Deep Image Matting:

https://github.com/Joker316701882/Deep-Image-Matting with Tensorflow
https://github.com/huochaitiantang/pytorch-deep-image-matting with Pytorch

I'm just trying to run the tests in these projects but I run into various problems. Can I run these on a machine without GPU? I thought that GPU is only for speeding up processing, but I'm only interested in seeing these run before getting a machine with GPU.
I apologize in advance, as I know I'm a total noob in this

When I try the Tensorflow project:

I get an error with this line gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction = args.gpu_fraction) probably because I was tf2 and this requires tf1
After I downgraded to tf1 when I try to run the test I get W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
 and InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'MaxPoolWithArgmax' with these attrs.  Registered devices: [CPU], Registered kernels:
<no registered kernels> and now I'm stuck because I have no clue what this means

When I try the Pytorch project:

First I get this error: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
So I added map_location=torch.device('cpu') when the model is loaded, but now I get RuntimeError: Error(s) in loading state_dict for VGG16:
    size mismatch for conv6_1.weight: copying a param with shape torch.Size([512, 512, 1, 1]) from checkpoint, the shape in current model is torch.Size([512, 512, 3, 3]). And I'm stuck again

Can someone help?

thank you in advance!

Output:
For the PyTorch one, there were two problems and it looks like you've solved the first one on your own with map_location. The second problem is that the weights in your checkpoint and the weights in your model don't have the same shape! A quick detour to the github repo; let's visit net.py in core. Take a look at lines 26 to 28:

# model released before 2019.09.09 should use kernel_size=1 & padding=0
# self.conv6_1 = nn.Conv2d(512, 512, kernel_size=1, padding=0,bias=True)
self.conv6_1 = nn.Conv2d(512, 512, kernel_size=3, padding=1,bias=True)

I'm guessing the checkpoint is loading weights where conv6_1 has a kernel size of 1 rather than 3, like the commented out line of code. So try uncommenting the line with kernel_size=1 and comment out the line with kernel_size=3.


Instruction: Install torch on python 3.8.1 windows 10

Input: I have been reading this post How to install pytorch in windows? but no one answer work for me on the versio 3.8.1 of python. Anything else I can do?

Output:
Maybe this can help you.

pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Please ensure that you have met the prerequisites, depending on your package manager. You can also use Anaconda as a package manager since it installs all dependencies.


Instruction: PyTorch C++ FrontEnd returning multiple Tensors in forward

Input:
I was wonder how can I return a std::vector<torch::Tensor> in my forward pass of a Module Class,
I read about the Macro of FORWARD_HAS_DEFAULT_ARGS in the docs, but didn’t really
understand how to use it, and also how to use it for making it possible to return a vector in return.
Thank you in advance.

Output:
FORWARD_HAS_DEFAULT_ARGS is a C++ macro and according to documentation:

  This macro enables a module with default arguments in its forward
  method to be used in a Sequential module.

So it's not what you are after.

I assume you are returning multiple torch::Tensor values contained in std::vector. You could just do that, but you should appropriately unpack it after returning like this:

# Interprets returned IValue as your desired return type
# You may have to use module.forward(inputs) depending how you loaded model
auto outputs = module->forward(inputs).toTensorVector();
# Print first tensor
std::cout << outputs[0] << std::endl;

If you want to return multiple values of different types from forward method you should just return std::tuple containing your desired types.

After this you can unpack it like this (for two torch::Tensor return values) (source here):

auto outputs = module->forward(inputs).toTuple();
torch::Tensor out1 = outputs->elements()[0].toTensor();
torch::Tensor out2 = outputs->elements()[1].toTensor();

You could also concatenate pytorch tensors (if that's all you are returning and they are of the same shape) and use view or a-like methods to unpack it. C++ frontend is pretty similar to Python's all in all, refer to docs if in doubt.


Instruction: PyTorch error - 'numpy.ndarray' object has no attribute 'relu'

Input:
I am testing my CNN model, but keep on getting error "AttributeError: 'numpy.ndarray' object has no attribute 'relu'".

my dataset is extracted by below code:

import torch
from torch.utils.data import Dataset, DataLoader
from torch.autograd import Variable
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np

class MyDataset(Dataset):
    def __init__(self, data, target, transform=None):
        self.data = torch.from_numpy(data).float()
        self.target = torch.from_numpy(target).long()
        self.transform = transform

    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]

        if self.transform:
            x = self.transform(x)

        return x, y

    def __len__(self):
        return len(self.data)

numpy_data = np.random.randn(100,3,224,224) # 10 samples, image size = 224 x 224 x 3
numpy_target = np.random.randint(0,5,size=(100))

dataset = MyDataset(numpy_data, numpy_target)

my model is very simple as below:

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 36, kernel_size = 100)

    def forward(self, t):

        t = self.conv1(t)
        print(t.shape)
        print(type(t))
        t = F.relu(t)
        print(t.shape)

        return t

I test model using below:

sample, target = next(iter(dataset))
network=Network()
pred = network(sample.unsqueeze(0))

I got below result and error:

torch.Size([1, 6, 125, 125])
<class 'torch.Tensor'>

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-40-37e58cfe971f> in <module>
----> 1 pred = network(sample.unsqueeze(0))

C:\Miniconda\envs\py37_default\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-30-0d0592ad705d> in forward(self, t)
     22         print(t.shape)
     23         print(type(t))
---> 24         t = F.relu(t)
     25         print(t.shape)
     26         #t = F.max_pool2d(t, kernel_size=2, stride=2)

AttributeError: 'numpy.ndarray' object has no attribute 'relu'

I could not figure out why, type(t) does output as , why error is showing it is numpy.ndarray?

Output:
Where is F defined? F seems to be the numpy array.

Did you maybe mean to do:

import torch.nn.functional as F? Otherwise, the relu function isn't defined anywhere.


Instruction: Pytorch : GPU Memory Leak

Input:
I speculated that I was facing a GPU memory leak in the training of Conv nets using PyTorch framework. Below image

To resolve it, I added -

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

which resolved the memory problem, as shown below -

but as I was using torch.nn.DataParallel, so I expect my code to utilise all the GPUs, but now it is utilising only the GPU:1.

Before using os.environ['CUDA_LAUNCH_BLOCKING'] = "1", the GPU utilisation was below (which is equally bad)-

On digging further, I come to know that, when we use torch.nn.DataParallel, we are supposed to not use CUDA_LAUNCH_BLOCKING', because it puts the network in some deadlock mechanism.
So, now I have come back again in GPU memory issue, because I think my code is not utilising that much memory which it is showing without setting CUDA_LAUNCH_BLOCKING=1.
My code to use torch.nn.DataParallel-

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))

model_transfer.to(device)

How to resolve the GPU memory issue?
Edit:
Minimal code -

image_dataset = datasets.ImageFolder(train_dir_path,transform = transform)
train_loader = torch.utils.data.DataLoader(image_dataset['train'], batch_size=batch_size,shuffle = True)
model_transfer = models.resnet18(pretrained=True)

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))
model_transfer.to(device)

## Training function
for epoch in range(1, n_epochs+1):
    for batch_idx, (data, target) in enumerate(train_loader):
                    if use_cuda:
            data, target = data.to('cuda',non_blocking = True), target.to('cuda',non_blocking = True)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output,target)
        loss.backward()
        optimizer.step()
        train_loss += ((1 / (batch_idx + 1)) * (loss.item() - train_loss))

   ## Validation loop same as training loop so not mentioning here

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch,
        train_loss,
        valid_loss
        ))
        if valid_loss <= valid_loss_min:
           valid_loss_min,valid_loss))
           torch.save(model.state_dict(), 'case_3_model.pt')
           valid_loss_min = valid_loss

Output:
So the way I resolved some of my CUDA out of memory issue is by making sure to delete useless tensors and trim tensors that may stay referenced for some hidden reason. The problem may arise from either requesting for more memory than you have the capacity for or an accumulation of garbage data that you don't need, but somehow is left behind on the memory.

One of the most important aspects of this memory management is how you are loading in the data. Instead of reading the entire dataset, it may be more memory efficient to read from disk (using memmap when reading npy) or doing batch loading, where you only read a batch of images or whatever data you have at a time. Although this may be computationally slower, it does give you flexibility for not going out and buying more GPUS to store your memory just to run your code.

We're not sure how your code is structured in terms of reading the data or training your CNN so this is as much advice I can give.


Instruction: pythorch-lightning train_dataloader runs out of data

Input:
I started to use pytorch-lightning and faced a problem of my custom data loaders:

Im using an own dataset and a common torch.utils.data.DataLoader. Basically the dataset takes a path and loads the data corresponding to an given index the dataloader loads its.

def train_dataloader(self):
    train_set = TextKeypointsDataset(parameters...)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size, num_workers)
    return train_loader

When I use the pytorch-lightning modules train_dataloader and training_step everything runs fine. When I add val_dataloader and validation_step Im facing this error:

Epoch 1:  45%|████▌     | 10/22 [00:02<00:03,  3.34it/s, loss=5.010, v_num=131199]
ValueError: Expected input batch_size (1500) to match target batch_size (5)

In this case my dataset is really small (to test functionality) of 84 samples, my batch size is 8. The dataset for training and validation has the same length (just for testing purposes again).

So in total its 84 * 2 = 168 and 168 / 8 (batchsize) = 21, which are roughly the total steps (22) shown above. This means that after running on the training dataset for 10 times (10 * 8 = 80) the loader expects a new full sample of 8, but since there are only 84 samples I get an error (at least this is my current understanding).

I faced a similar problem in my own implementation (not using pytorch-lighntning) and used this pattern to solve it. Basically I am resetting the iterator, when running out of data:

try:
    data = next(data_iterator)
    source_tensor = data[0]
    target_tensor = data[1]

except StopIteration:  # reinitialize data loader if num_iteration > amount of data
    data_iterator = iter(data_loader)

Right now it seems like Im facing sth similar? I dont know how to reset/reinitialize the data loader in pytorch-lightning when my training_dataloader is running out of data. I guess there must be another sophisticated way Im not familar with. Thank you

Output:
The solution was:

I used source_tensor = source_tensor.view(-1, self.batch_size, self.input_size) which lead to some errors later on, now Im using source_tensor = source_tensor.permute(1, 0, 2), which fixed the problem.


Instruction: Confusion regarding batch size while using DataLoader in pytorch

Input:
I am new to pytorch.
I am training an ANN for classification on the MNIST dataset.

train_loader = DataLoader(train_data,batch_size=200,shuffle=True)

I am confused. The dataset is of 60,000 images and I have set batch size of 6000 and my model has 30 epochs.
Will every epoch see only 6000 images or will every epoch see 10 batches of 6000 images?

Output: Every call to the dataset iterator will return batch of images of size batch_size. Hence you will have 10 batches until you exhaust all the 60000 images.


Instruction: Pytorch: How to know if GPU memory being utilised is actually needed or is there a memory leak

Input:
I have 3 Tesla V100(16 GB). I am doing transfer learning using efficeint net (63 Million parameters) on images of (512,512) with a batch size of 20.
My GPU memory utilisation is below -

As you can see, it has almost filled up all the 3 GPUs(almost 80%).
My question is is there any theoretical way of calculating that the utilisation of GPU memory being shown is what is required by the model at a certain image and batch size or is there a memory leak in my GPU?

Output:
I'm not sure that is what you asked for but have you tried doing something like:

memory_usage = number_of_variables * memory_usage_per_variable.

So if you use torch.float32 tensors, and you have 125 000 variables sent to the GPU with .cuda(). Then you are using 4Gbytes of memory on your GPU. You can compare with how much memory you have available on your memory.

Another Sanity check would be to check memory usage on the GPU's per iteration your model, if it doubles then you have a memory leak.

Hope this helps.


Instruction: Pytorch could not find module

Input:
I have installed pytorch with command:

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch -y

Python complains regarding line import torch with message:

Could not find module 'C:\ProgramData\Anaconda3\envs\edx\lib\site-packages\torch\lib\caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax

This directory contains library caffe2_nvrtc.dll. What might be wrong and how to fix this error?

Output: I faced the same problem. If your OS is Windows then I would recommend using Anaconda and installing pytorch in separate conda environment. Quick solution is to search for nvcuda.dll file on google and download this file. If you are running the code on Jupyter notebook the output will give you the complete path of the 'lib' folder of conda environment. By default it is 'C:\Users\YourUserName\anaconda3\envs\mera_beta\Lib\site-packages\torch\lib'. Go this directory and paste the file in this folder. Rerun your code. Hopefully it will run.


Instruction: pip install torch killed at 99% -- Excessive memory usage

Input:
This is while I was installing torch on my laptop. It was getting killed continuously so I thought I will check the memory usage. It hanged my laptop, I had to take a picture with my phone.

If you can't see the image below, it shows pip using 5.8 GiB memory out of 7.8 GiB available. That was a sudden spike at 99%.

System Monitor pip memory usage

Output: If you are running low on memory you could try with pip install package --no-cache-dir


Instruction: Is torch.as_tensor() the same as torch.from_numpy() for a numpy array on a CPU?

Input:
On a CPU, is torch.as_tensor(a) the same as torch.from_numpy(a) for a numpy array, a?  If not, then why not?

From the docs for torch.as_tensor

  if the data is an ndarray of the corresponding dtype and
      the device is the cpu, no copy will be performed.

From the docs for torch.from_numpy:

  The returned tensor and ndarray share the same memory. Modifications to
      the tensor will be reflected in the ndarray and vice versa.

In both cases, any changes the resulting tensor changes the original numpy array.

a = np.array([[1., 2], [3, 4]])
t1 = torch.as_tensor(a)
t2 = torch.from_numpy(a)
t1[0, 0] = 42.
print(a)
# prints [[42., 2.], [3., 4.]]
t2[1, 1] = 55.
print(a)
# prints [[42., 2.], [3., 55.]]

Also, in both cases, attempting to resize_ the tensor results in an error.

Output:
They are basically the same, except than as_tensor is more generic:

Contrary to from_numpy, it supports a wide range of datatype, including list, tuple, and native Python scalars.
as_tensor supports changing dtype and device directly, which is very convenient in practice since the default dtype of Torch tensor is float32, while for Numpy array it is float64.

as_tensor is sharing memory with the original data if and only if the original object is a Numpy array, and the requested dtype, if any, is the same than the original data. Those are the same conditions than from_numpy, but are always satisfied by design for the later.


Instruction: How embedding_bag exactly works in PyTorch

Input: in PyTorch, torch.nn.functional.embedding_bag seems to be the main function responsible for doing the real job of embedding lookup. On PyTorch's documentation, it has been mentioned that embedding_bag does its job > without instantiating the intermediate embeddings. What does that exactly mean? Does this mean for example when the mode is "sum" it does in-place summation? or it just means that no additional Tensors will be produced when calling embedding_bag but still from the system's point of view all the intermediate row-vectors are already fetched into the processor to be used for calculating the final Tensor?

Output:
In the simplest case, torch.nn.functional.embedding_bag is conceptually a two step process. The first step is to create an embedding and the second step is to reduce (sum/mean/max, according to the "mode" argument) the embedding output across dimension 0. So you can get the same result that embedding_bag gives by calling torch.nn.functional.embedding, followed by torch.sum/mean/max. In the following example, embedding_bag_res and embedding_mean_res are equal.
>>> weight = torch.randn(3, 4)
>>> weight
tensor([[ 0.3987,  1.6173,  0.4912,  1.5001],
        [ 0.2418,  1.5810, -1.3191,  0.0081],
        [ 0.0931,  0.4102,  0.3003,  0.2288]])
>>> indices = torch.tensor([2, 1])
>>> embedding_res = torch.nn.functional.embedding(indices, weight)
>>> embedding_res
tensor([[ 0.0931,  0.4102,  0.3003,  0.2288],
        [ 0.2418,  1.5810, -1.3191,  0.0081]])
>>> embedding_mean_res = embedding_res.mean(dim=0, keepdim=True)
>>> embedding_mean_res
tensor([[ 0.1674,  0.9956, -0.5094,  0.1185]])
>>> embedding_bag_res = torch.nn.functional.embedding_bag(indices, weight, torch.tensor([0]), mode='mean')
>>> embedding_bag_res
tensor([[ 0.1674,  0.9956, -0.5094,  0.1185]])

However, the conceptual two step process does not reflect how it's actually implemented. Since embedding_bag does not need to return the intermediate result, it doesn't actually generate a Tensor object for the embedding. It just goes straight to computing the reduction, pulling in the appropriate data from the weight argument according to the indices in the input argument. Avoiding the creation of the embedding Tensor allows for better performance.
So the answer to your question (if I understand it correctly)

it just means that no additional Tensors will be produced when calling embedding_bag but still from the system's point of view all the intermediate row-vectors are already fetched into the processor to be used for calculating the final Tensor?

is yes.


Instruction: PyTorch Model Training: RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

Input:
After training a PyTorch model on a GPU for several hours, the program fails with the error

  RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

Training Conditions

Neural Network: PyTorch 4-layer nn.LSTM with nn.Linear output
Deep Q Network Agent (Vanilla DQN with Replay Memory)
state passed into forward() has the shape (32, 20, 15), where 32 is the batch size
50 seconds per episode
Error occurs after about 583 episodes (8 hours) or 1,150,000 steps, where each step involves a forward pass through the LSTM model.

My code also has the following values set before the training began

torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)

How can we troubleshoot this problem? Since this occurred 8 hours into the training, some educated guess will be very helpful here!

Thanks!

Update:

Commenting out the 2 torch.backends.cudnn... lines did not work. CUDNN_STATUS_INTERNAL_ERROR still occurs, but much earlier at around Episode 300 (585,000 steps).

torch.manual_seed(0)
#torch.backends.cudnn.deterministic = True
#torch.backends.cudnn.benchmark = False
np.random.seed(0)

System

PyTorch 1.6.0.dev20200525
CUDA 10.2
cuDNN 7604
Python 3.8
Windows 10
nVidia 1080 GPU

Error Traceback

RuntimeError                              Traceback (most recent call last)
<ipython-input-18-f5bbb4fdfda5> in <module>
     57
     58     while not done:
---> 59         action = agent.choose_action(state)
     60         state_, reward, done, info = env.step(action)
     61         score += reward

<ipython-input-11-5ad4dd57b5ad> in choose_action(self, state)
     58         if np.random.random() > self.epsilon:
     59             state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
---> 60             actions = self.q_eval.forward(state)
     61             action = T.argmax(actions).item()
     62         else:

<ipython-input-10-94271a92f66e> in forward(self, state)
     20
     21     def forward(self, state):
---> 22         lstm, hidden = self.lstm(state)
     23         actions = self.fc1(lstm[:,-1:].squeeze(1))
     24         return actions

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    575             result = self._slow_forward(*input, **kwargs)
    576         else:
--> 577             result = self.forward(*input, **kwargs)
    578         for hook in self._forward_hooks.values():
    579             hook_result = hook(self, input, result)

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    571         self.check_forward_args(input, hx, batch_sizes)
    572         if batch_sizes is None:
--> 573             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
    574                               self.dropout, self.training, self.bidirectional, self.batch_first)
    575         else:

RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

Update: Tried try... except on my code where this error occurs at, and in addition to RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR, we also get a second traceback for the error RuntimeError: CUDA error: unspecified launch failure

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-4-e8f15cc8cf4f> in <module>
     61
     62     while not done:
---> 63         action = agent.choose_action(state)
     64         state_, reward, done, info = env.step(action)
     65         score += reward

<ipython-input-3-1aae79080e99> in choose_action(self, state)
     58         if np.random.random() > self.epsilon:
     59             state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
---> 60             actions = self.q_eval.forward(state)
     61             action = T.argmax(actions).item()
     62         else:

<ipython-input-2-6d22bb632c4c> in forward(self, state)
     25         except Exception as e:
     26             print('error in forward() with state:', state.shape, 'exception:', e)
---> 27             print('state:', state)
     28         actions = self.fc1(lstm[:,-1:].squeeze(1))
     29         return actions

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\tensor.py in __repr__(self)
    152     def __repr__(self):
    153         # All strings are unicode in Python 3.
--> 154         return torch._tensor_str._str(self)
    155
    156     def backward(self, gradient=None, retain_graph=None, create_graph=False):

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _str(self)
    331                 tensor_str = _tensor_str(self.to_dense(), indent)
    332             else:
--> 333                 tensor_str = _tensor_str(self, indent)
    334
    335     if self.layout != torch.strided:

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _tensor_str(self, indent)
    227     if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
    228         self = self.float()
--> 229     formatter = _Formatter(get_summarized_data(self) if summarize else self)
    230     return _tensor_str_with_formatter(self, indent, formatter, summarize)
    231

~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in __init__(self, tensor)
     99
    100         else:
--> 101             nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
    102
    103             if nonzero_finite_vals.numel() == 0:

RuntimeError: CUDA error: unspecified launch failure

Output:
The error RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR is notoriously difficult to debug, but surprisingly often it's an out of memory problem. Usually, you would get the out of memory error, but depending on where it occurs, PyTorch cannot intercept the error and therefore not provide a meaningful error message.

A memory issue seems to be likely in your case, because you are using a while loop until the agent is done, which might take long enough that you run out of memory, it's just a matter of time. That can also possibly occur rather late, once the model's parameters in combination with a certain input is unable to finish in time.

You can avoid that scenario by limiting the number of allowed actions instead of hoping that the actor will be done in a reasonable time.

What you also need to be careful about, is that you don't occupy unnecessary memory. A common mistake is to keep computing gradients of the past states in future iterations. The state from the last iteration should be considered constant, since the current action should not affect past actions, therefore no gradients are required. This is usually achieved by detaching the state from the computational graph for the next iteration, e.g. state = state_.detach(). Maybe you are already doing that, but without the code it's impossible to tell.

Similarly, if you keep a history of the states, you should detach them and even more importantly put them on the CPU, i.e. history.append(state.detach().cpu()).


Instruction: Pytorch: IndexError: index out of range in self. How to solve?

Input:
This training code is based on the run_glue.py script found here:
# Set the seed value all over the place to make this reproducible.
seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# Store the average loss after each epoch so we can plot them.
loss_values = []

# For each epoch...
for epoch_i in range(0, epochs):

    # ========================================
    #               Training
    # ========================================

    # Perform one full pass over the training set.

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    # Measure how long the training epoch takes.
    t0 = time.time()

    # Reset the total loss for this epoch.
    total_loss = 0

    # Put the model into training mode. Don't be mislead--the call to
    # `train` just changes the *mode*, it doesn't *perform* the training.
    # `dropout` and `batchnorm` layers behave differently during training
    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        # Progress update every 100 batches.
        if step % 100 == 0 and not step == 0:
            # Calculate elapsed time in minutes.
            elapsed = format_time(time.time() - t0)

            # Report progress.
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using the
        # `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Always clear any previously calculated gradients before performing a
        # backward pass. PyTorch doesn't do this automatically because
        # accumulating the gradients is "convenient while training RNNs".
        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
        model.zero_grad()

        # Perform a forward pass (evaluate the model on this training batch).
        # This will return the loss (rather than the model output) because we
        # have provided the `labels`.
        # The documentation for this `model` function is here:
        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
        outputs = model(b_input_ids,
                    token_type_ids=None,
                    attention_mask=b_input_mask,
                    labels=b_labels)

        # The call to `model` always returns a tuple, so we need to pull the
        # loss value out of the tuple.
        loss = outputs[0]

        # Accumulate the training loss over all of the batches so that we can
        # calculate the average loss at the end. `loss` is a Tensor containing a
        # single value; the `.item()` function just returns the Python value
        # from the tensor.
        total_loss += loss.item()

        # Perform a backward pass to calculate the gradients.
        loss.backward()

        # Clip the norm of the gradients to 1.0.
        # This is to help prevent the "exploding gradients" problem.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        # Update parameters and take a step using the computed gradient.
        # The optimizer dictates the "update rule"--how the parameters are
        # modified based on their gradients, the learning rate, etc.
        optimizer.step()

        # Update the learning rate.
        scheduler.step()

    # Calculate the average loss over the training data.
    avg_train_loss = total_loss / len(train_dataloader)

    # Store the loss value for plotting the learning curve.
    loss_values.append(avg_train_loss)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))

    # ========================================
    #               Validation
    # ========================================
    # After the completion of each training epoch, measure our performance on
    # our validation set.

    print("")
    print("Running Validation...")

    t0 = time.time()

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()

    # Tracking variables
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Add batch to GPU
        batch = tuple(t.to(device) for t in batch)

        # Unpack the inputs from our dataloader
        b_input_ids, b_input_mask, b_labels = batch

        # Telling the model not to compute or store gradients, saving memory and
        # speeding up validation
        with torch.no_grad():

            # Forward pass, calculate logit predictions.
            # This will return the logits rather than the loss because we have
            # not provided labels.
            # token_type_ids is the same as the "segment ids", which
            # differentiates sentence 1 and 2 in 2-sentence tasks.
            # The documentation for this `model` function is here:
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            outputs = model(b_input_ids,
                            token_type_ids=None,
                            attention_mask=b_input_mask)

        # Get the "logits" output by the model. The "logits" are the output
        # values prior to applying an activation function like the softmax.
        logits = outputs[0]

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        # Calculate the accuracy for this batch of test sentences.
        tmp_eval_accuracy = flat_accuracy(logits, label_ids)

        # Accumulate the total accuracy.
        eval_accuracy += tmp_eval_accuracy

        # Track the number of batches
        nb_eval_steps += 1

    # Report the final accuracy for this validation run.
    print("  Accuracy: {0:.2f}".format(eval_accuracy/nb_eval_steps))
    print("  Validation took: {:}".format(format_time(time.time() - t0)))

print("")
print("Training complete!")

The error is as follows, while running the training for text classification using bert models came across the follow.
    ~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722         # remove once script supports set_grad_enabled
   1723         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1724     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1725
   1726

IndexError: index out of range in self

How can I fix it?

Output:
I think you have messed up with input dimension declared torch.nn.Embedding and with your input. torch.nn.Embedding is a simple lookup table that stores embeddings of a fixed dictionary and size.
Any input less than zero or more than declared input dimension raise this error.
Compare your input and the dimension mentioned in torch.nn.Embedding.
Attached code snippet to simulate the issue.
from torch import nn
input_dim = 10
embedding_dim = 2
embedding = nn.Embedding(input_dim, embedding_dim)
err = True
if err:
    #Any input more than input_dim - 1, here input_dim = 10
    #Any input less than zero
    input_to_embed = torch.tensor([10])
else:
    input_to_embed = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
embed = embedding(input_to_embed)
print(embed)

Hope this will solve your issue.


Instruction: forward() takes 1 positional argument but 2 were given

Input:
I'm trying to build a Model using EfficientNet-B0.
The details of the Model are shown in the code below.
I got the following error when I tried to learn.

TypeError                                 Traceback (most recent call last)
'''
<ipython-input-17-fb3850894108> in forward(self, *x)
     24         #x: bs*N x 3 x 128 x 128
     25         print(x.shape)     #([384, 3, 224, 224])
---> 26         x = self.enc(x)
     27         #x: bs*N x C x 4 x 4
     28         shape = x.shape

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() takes 1 positional argument but 2 were given

I suspect that m.children() may have an effect.
If anyone knows the cause of this error, please let me know.
Thank you.

class Model(nn.Module):
    def __init__(self, arch='efficientnet-b0', n=6, pre=True):
        super().__init__()
        m = EfficientNet.from_pretrained('efficientnet-b0')
        #print(*list(m.children())
        nc = m._fc.in_features
        print(nc)
        self.enc = nn.Sequential(*list(m.children())[:-2])
        #nc = list(m.children())[-2].in_features
        self.head = nn.Sequential(*list(m.children())[-2:])
        self.head._fc = nn.Linear(nc, n)
        #self.enc = nn.Sequential(*list(m.children()))
        #print('fc_infeatures : {}'.format(nc))
        #self.head = nn.Sequential(Flatten(),nn.Linear(nc,512),
        #                    relu(),nn.BatchNorm1d(512), nn.Dropout(0.5),nn.Linear(512,n))

    def forward(self, *x):
        print(x[0].shape)  #([32, 3, 224, 224])
        shape = x[0].shape
        n = len(x)           # n = 12
        #torch.stack直後では32*12*3*224*224(bs x N x 3 x 224 x 224)
        x = torch.stack(x,1).view(-1,shape[-3],shape[-2],shape[-1])
        #x: bs*N x 3 x 128 x 128
        print(x.shape)     #([384, 3, 224, 224])
        x = self.enc(x)
        #x: bs*N x C x 4 x 4
        shape = x.shape
        #concatenate the output for tiles into a single map
        x = x.view(-1,n,shape[1],shape[2],shape[3]).permute(0,2,1,3,4).contiguous()\
          .view(-1,shape[1],shape[2]*n,shape[3])
        #x: bs x C x N*4 x 4
        x = self.head(x)
        return x

Output:
Now you can easily get the network without the last layers by using the include_top parameter:
m = EfficientNet.from_pretrained('efficientnet-b0', include_top=False)

What it does, as can be easily seen in the code, is not calling forward method for the last layers (AveragePool, Dropout, FC).
Other alternative approaches can be summarized as followings:

Define a new MyEfficientNet that override the forward method to avoid calling the last layers

Overwrite the layers you don't need with an identity layer. For example:
m._fc = nn.Identity()

The first approach should be preferred when using the efficientnet as a features extractor while the others should be used when more customisation is needed.


Instruction: Is this the right way to compute gradients of two losses from two different NN's in pytorch?

Input:
I have a NN defined in pytorch and I have created two instances of that net as self.actor_critic_r1 and self.actor_critic_r2. I calculate the losses of each net i.e. loss1 and loss2 and I sum it up and calculate the grads in the following way,

loss_r1 = value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef
loss_r2 = value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef
self.optimizer_r1.zero_grad()
self.optimizer_r2.zero_grad()
loss = loss_r1 + loss_r2
loss.backward()
self.optimizer_r1.step()
self.optimizer_r2.step()
clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)

Alternatively, should I update the loss individually like this,

self.optimizer_r1.zero_grad()
(value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef).backward()
self.optimizer_r1.step()
clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
self.optimizer_r2.zero_grad()
(value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef).backward()
self.optimizer_r2.step()
clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)

I am not sure if this the right approach to update a network with multiple loss please provide your suggestion.

Output:
It should be the sum approach. If there is no interplay then the gradient of the 'wrong' loss will be zero for the 'wrong' optimizer anyway, and if there is interplay you likely want to optimize for that interplay.

Only if you know that there is interplay but you do not want to optimize for it should you use approach #2.