Instruction: Pytorch squeeze and unsqueeze

Input:
I don't understand what squeeze and unsqueeze do to a tensor, even after looking at the docs and related questions.
I tried to understand it by exploring it myself in python. I first created a random tensor with
x = torch.rand(3,2,dtype=torch.float)
>>> x
tensor([[0.3703, 0.9588],
        [0.8064, 0.9716],
        [0.9585, 0.7860]])

But regardless of how I squeeze it, I end up with the same results:
torch.equal(x.squeeze(0), x.squeeze(1))
>>> True

If I now try to unsqueeze I get the following,
>>> x.unsqueeze(1)
tensor([[[0.3703, 0.9588]],
        [[0.8064, 0.9716]],
        [[0.9585, 0.7860]]])
>>> x.unsqueeze(0)
tensor([[[0.3703, 0.9588],
         [0.8064, 0.9716],
         [0.9585, 0.7860]]])
>>> x.unsqueeze(-1)
tensor([[[0.3703],
         [0.9588]],
        [[0.8064],
         [0.9716]],
        [[0.9585],
         [0.7860]]])

However if I now create a tensor  x = torch.tensor([1,2,3,4]), and I try to unsqueeze it then it appears that 1 and -1 makes it a column where as 0 remains the same.
x.unsqueeze(0)
tensor([[1, 2, 3, 4]])
>>> x.unsqueeze(1)
tensor([[1],
        [2],
        [3],
        [4]])
>>> x.unsqueeze(-1)
tensor([[1],
        [2],
        [3],
        [4]])

Can someone provide an explanation of what squeeze and unsqueeze are doing to a tensor? And what's the difference between providing the arguements 0, 1 and -1?

Output:
Here is a visual representation of what squeeze/unsqueeze do for an effectively 2d matrix:

When you are unsqueezing a tensor, it is ambiguous which dimension you wish to 'unsqueeze' it across (as a row or column etc). The dim argument dictates this - i.e. position of the new dimension to be added.
Hence the resulting unsqueezed tensors have the same information, but the indices used to access them are different.


Instruction: Is there any difference between tensor2tensor and pytorch in view of memory?

Input:
I'm trying to train seq2seq model(transformer) with pytorch and tensor2tensor.
When using tensor2tensor, the batch size can be like 1024, while pytorch model shows CUDA out of memory error with 8 batch size.

Is there any technique used in tensor2tensor to make best use of memory.

If anyone know this, please tell me.

Thanks in advance.

Output:
In Tensor2Tensor by default, the batch size is specified in the number of tokens (subwords) per single GPU. This allows to use a higher number of short sequences (sentences) in one batch or a smaller number of long sequences. Most other toolkits use a fixed batch size specified in the number of sequences. Either way, it may be a good idea to limit the maximum sentence length in training to a reasonable number to prevent Out-of-memory errors and excessive padding.
Some toolkits also prefer to specify the total batch size per all GPU cards.


Instruction: How to split a dataset into a custom training set and a custom validation set with pytorch?

Input:
I'm using a non-torchvision dataset and I have extracted it with the ImageFolder method. I'm trying to split the dataset into 20% validation set and 80% training set. I can only find this method (random_split) from PyTorch library which allows splitting dataset. However, this is random every time. I'm wondering is there a way to split the dataset with a specific amount in the PyTorch library?

This is my code for extracting the dataset and split it randomly.

transformations = transforms.Compose([
    transforms.Resize(255),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

TrafficSignSet = datasets.ImageFolder(root='./train/', transform=transformations)

####### split data
train_size = int(0.8 * len(TrafficSignSet))
test_size = len(TrafficSignSet) - train_size
train_dataset_split, test_dataset_split = torch.utils.data.random_split(TrafficSignSet, [train_size, test_size])

#######put into a Dataloader
train_dataset = torch.utils.data.DataLoader(train_dataset_split, batch_size=32, shuffle=True)
test_dataset = torch.utils.data.DataLoader(test_dataset_split, batch_size=32, shuffle=True)

Output:
If you look "under the hood" of random_split you'll see it uses torch.utils.data.Subset to do the actual splitting. You can do so yourself with fixed indices:

import random

indices = list(range(len(TrafficSignSet))
random.seed(310)  # fix the seed so the shuffle will be the same everytime
random.shuffle(indices)
train_dataset_split = torch.utils.data.Subset(TrafficSignSet, indices[:train_size])
val_dataset_split = torch.utils.data.Subset(TrafficSignSet, indices[train_size:])


Instruction: Pytorch equivalent features in tensorflow?

Input: I recently was reading a Pytorch code and came across loss.backward() and optimizer.step() functions, are there any equivalent of these using tensorflow/keras?

Output:
loss.backward() equivalent in tensorflow is tf.GradientTape(). TensorFlow provides the tf.GradientTape API for automatic differentiation - computing the gradient of computation with respect to its input variables. Tensorflow "records" all operations executed inside the context of a tf.GradientTape onto a "tape". Tensorflow then uses that tape and the gradients associated with each recorded operation to compute the gradients of a "recorded" computation using reverse mode differentiation.
optimizer.step() equivalent in tensorflow is minimize(). Minimizes the loss by updating the variable list. Calling minimize() takes care of both computing the gradients and applying them to the variables.
If you want to process the gradients before applying them you can instead use the optimizer in three steps:

Compute the gradients with tf.GradientTape.
Process the gradients as you wish.
Apply the processed gradients with apply_gradients().

Hope this answers your question. Happy Learning.


Instruction: PyTorch nn.Transformer learns to copy target

Input:
I’m trying to train a Transformer Seq2Seq model using nn.Transformer class. I believe I am implementing it wrong, since when I train it, it seems to fit too fast, and during inference it repeats itself often. This seems like a masking issue in the decoder, and when I remove the target mask, the training performance is the same. This leads me to believe I am doing the target masking wrong. Here is my model code:
class TransformerModel(nn.Module):
    def __init__(self,
        vocab_size, input_dim, heads, feedforward_dim, encoder_layers, decoder_layers,
        sos_token, eos_token, pad_token, max_len=200, dropout=0.5,
        device=(torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu"))):

        super(TransformerModel, self).__init__()
        self.target_mask = None
        self.embedding = nn.Embedding(vocab_size, input_dim, padding_idx=pad_token)
        self.pos_embedding = nn.Embedding(max_len, input_dim, padding_idx=pad_token)
        self.transformer = nn.Transformer(
            d_model=input_dim, nhead=heads, num_encoder_layers=encoder_layers,
            num_decoder_layers=decoder_layers, dim_feedforward=feedforward_dim,
            dropout=dropout)
        self.out = nn.Sequential(
            nn.Linear(input_dim, feedforward_dim),
            nn.ReLU(),
            nn.Linear(feedforward_dim, vocab_size))

        self.device = device
        self.max_len = max_len
        self.sos_token = sos_token
        self.eos_token = eos_token

    # Initialize all weights to be uniformly distributed between -initrange and initrange
    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    # Generate mask covering the top right triangle of a matrix
    def generate_square_subsequent_mask(self, size):
        mask = (torch.triu(torch.ones(size, size)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def forward(self, src, tgt):
        # src: (Max source seq len, batch size, 1)
        # tgt: (Max target seq len, batch size, 1)

        # Embed source and target with normal and positional embeddings
        embedded_src = (self.embedding(src) +
           self.pos_embedding(
           torch.arange(0, src.shape[1]).to(self.device).unsqueeze(0).repeat(src.shape[0], 1)))
        # Generate target mask
        target_mask = self.generate_square_subsequent_mask(size=tgt.shape[0]).to(self.device)
        embedded_tgt = (self.embedding(tgt) +
            self.pos_embedding(
            torch.arange(0, tgt.shape[1]).to(self.device).unsqueeze(0).repeat(tgt.shape[0], 1)))
        # Feed through model
        outputs = self.transformer(src=embedded_src, tgt=embedded_tgt, tgt_mask=target_mask)
        outputs = F.log_softmax(self.out(outputs), dim=-1)
        return outputs

Output:
For those having the same problem, my issue was that I wasn't properly adding the SOS token to the target I was feeding the model, and the EOS token to the target I was using in the loss function.
For reference:
The target fed to the model should be: [SOS] ....
And the target used for the loss should be: .... [EOS]


Instruction: Unable to install fastai on Jupyter Notebook

Input:
I'm currently trying to get fastai installed on a conda environment using the command conda install -c fastai fastai as shown in the installation guide. This is what appears when that command is ran:

(fastai) C:\>conda install -v -c fastai fastai
Collecting package metadata (current_repodata.json): ...working... Unable to retrieve repodata (response: 404) for https://conda.anaconda.org/fastai/win-64/current_repodata.json

done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working...
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed
Traceback (most recent call last):
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\exceptions.py", line 1079, in __call__
    return func(*args, **kwargs)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\main.py", line 84, in _main
    exit_code = do_call(args, p)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\conda_argparse.py", line 82, in do_call
    return getattr(module, func_name)(args, parser)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\install.py", line 308, in install
    raise e
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your CUDA driver:

  - feature:/win-64::__cuda==9.2=0

Your installed CUDA driver is: 9.2

I can't say I completely understand what the issue is here. I previously thought it was an issue with PyTorch but after installing PyTorch successfully, I am still greeted with this message. Any ideas on how I can this package installed on my environment? Any help is appreciated. Thank you!

Output: Try downgrading from python3.8 to python 3.7, it works for me.


Instruction: Logloss metric in Fastai

Input:
i'm doing a competition in zindi plateform which they are using The evaluation metric for this challenge as Log Loss.

so i'm working with fastai library and i want the metric log loss .. i didn't find LogLoss as metric in this library !
i tried some codes like the function provided by sklearn from sklearn.metrics import log_loss but i didn't work
the link of the competition : https://zindi.africa/competitions/basic-needs-basic-rights-kenya-tech4mentalhealth

Output:
if needed as a metric (typically mostly used as a loss) you should be able to use cross_entropy function from pytorch:

  import torch.nn.functional as F

  metrics=[F.cross_entropy,(plus other metrics if needed)]

  model= cnn_learner(data, model, metrics=metrics,...)


Instruction: How to prune weights less than a threshold in PyTorch?

Input:
How to prune weights of a CNN (convolution neural network) model which is less than a threshold value (let's consider prune all weights which are <= 1).

How we can achieve that for a weight file saved in .pth format in pytorch?

Output:
PyTorch since 1.4.0 provides model pruning out of the box, see official tutorial.

As there is no threshold method to prune in PyTorch currently, you have to implement it yourself, though it's kinda easy once you get the overall idea.

Threshold Pruning method

Below is a code performing pruning:

from torch.nn.utils import prune

class ThresholdPruning(prune.BasePruningMethod):
    PRUNING_TYPE = "unstructured"

    def __init__(self, threshold):
        self.threshold = threshold

    def compute_mask(self, tensor, default_mask):
        return torch.abs(tensor) > self.threshold

Explanation:

PRUNING_TYPE can be one of global, structured, unstructured. global acts across whole module (e.g. remove 20% of weight with smallest value), structured acts on whole channels/modules. We need unstructured as we would like to modify each connection in specific parameter tensor (say weight or bias)
__init__ - pass here whatever you want or need to make it work, normal stuff
compute_mask - mask to be used to prune specific tensor. In our case all parameters below threshold should be zero. I did it with absolute value as it makes more sense. default_mask is not needed here, but is left as named parameter as that's what API requires atm.

Moreover, inheriting from prune.BasePruningMethod defines methods to apply the mask to each parameter, make pruning permanent etc. See base class docs for more info.

Example module

Nothing too fancy, you can put anything you want here:

class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.first = torch.nn.Linear(50, 30)
        self.second = torch.nn.Linear(30, 10)

    def forward(self, inputs):
        return self.second(torch.relu(self.first(inputs)))

module = MyModule()

You can also load your module via module = torch.load('checkpoint.pth')
if you need, it doesn't matter.

Prune module's parameters

We should define which parameter of our module (and whether it's weight or bias) should be pruned, like this:

parameters_to_prune = ((module.first, "weight"), (module.second, "weight"))

Now, we can apply globally our unstructured pruning to all defined parameters (threshold is passed as kwarg to __init__ of ThresholdPruning):

prune.global_unstructured(
    parameters_to_prune, pruning_method=ThresholdPruning, threshold=0.1
)

Results

weight attribute

To see the effect, check weights of first submodule simply with:

print(module.first.weight)

It is a weight with our pruning technique applied, but please notice it's not a torch.nn.Parameter anymore! Now it is simply an attribute of our model, hence it won't take part in training or evaluation currently.

weight_mask

We can check created mask via module.first.weight_mask to see everything is done correctly (it will be binary in this case).

weight_orig

Applying pruning creates a new torch.nn.Parameter with original weights named name + _orig, in this case weight_orig, let's see:

print(module.first.weight_orig)

This parameter will be used during training and evaluation currently!. After applying pruning via methods described above there are forward_pre_hooks added which "switch" original weight to weight_orig.

Due to such approach you can define and apply your pruning at any part of training or inference without "destroying" original weights.

Applying pruning permanently

If you wish to apply pruning permanently simply issue:

prune.remove(module.first, "weight")

And now our module.first.weight is once again parameter with entries appropriately pruned, module.first.weight_mask is removed and so is module.first.weight_orig. It's what you are probably after.

You can iterate over children to make it permanent:

for child in module.children():
    prune.remove(child, "weight")

You could define parameters_to_prune using the same logic:

parameters_to_prune = [(child, "weight") for child in module.children()]

Or if you want only convolution layers to be pruned (or anything else really):

parameters_to_prune = [
    (child, "weight")
    for child in module.children()
    if isinstance(child, torch.nn.Conv2d)
]

Advantages

uses "PyTorch way of pruning" so it's easier to communicate your intent to other programmers
define pruning on a per-tensor basis, single responsibility instead of going through everything
confine to predefined ways
pruning is not permanent hence you can recover from it if needed. Module can be saved with pruning masks and original weights so it leaves you some space to revert eventual mistake (e.g. threshold was too high and now all your weights are zero rendering results meaningless)
works with original weights during forward calls unless you want to finally change to pruned version (simple call to remove)

Disadvantages

IMO pruning API could be clearer
You can do it shorter (as provided by Shai)
might be confusing for those who do not know such thing is "defined" by PyTorch (still there are tutorials and docs so I don't think it's a major problem)


Instruction: Implementation of the Dense Synthesizer

Input:
I’m trying to understand the Synthesizer paper (https://arxiv.org/pdf/2005.00743.pdf 1) and there’s a description of the dense synthesizer mechanism that should replace the traditional attention model as described in the Transformer architecture.

The Dense Synthesizer is described as such:

So I tried to implement the layer and it looks like this but I’m not sure whether I’m getting it right:

class DenseSynthesizer(nn.Module):
    def __init__(self, l, d):
        super(DenseSynthesizer, self).__init__()
        self.linear1 = nn.Linear(d, l)
        self.linear2 = nn.Linear(l, l)

    def forward(self, x, v):
        # Equation (1) and (2)
        # Shape: l x l
        b = self.linear2(F.relu(self.linear1(x)))
        # Equation (3)
        # [l x l] x [l x d] -> [l x d]
        return torch.matmul(F.softmax(b), v)

Usage:

l, d = 4, 5

x, v =  torch.rand(l, d), torch.rand(l, d)

synthesis = DenseSynthesizer(l, d)
synthesis(x, v)

Example:

x and v are tensors:

x = tensor([[0.0844, 0.2683, 0.4299, 0.1827, 0.1188],
         [0.2793, 0.0389, 0.3834, 0.9897, 0.4197],
         [0.1420, 0.8051, 0.1601, 0.3299, 0.3340],
         [0.8908, 0.1066, 0.1140, 0.7145, 0.3619]])

v = tensor([[0.3806, 0.1775, 0.5457, 0.6746, 0.4505],
         [0.6309, 0.2790, 0.7215, 0.4283, 0.5853],
         [0.7548, 0.6887, 0.0426, 0.1057, 0.7895],
         [0.1881, 0.5334, 0.6834, 0.4845, 0.1960]])

And passing through a forward pass through the dense synthesis, it returns:

>>> synthesis = DenseSynthesizer(l, d)
>>> synthesis(x, v)

tensor([[0.5371, 0.4528, 0.4560, 0.3735, 0.5492],
        [0.5426, 0.4434, 0.4625, 0.3770, 0.5536],
        [0.5362, 0.4477, 0.4658, 0.3769, 0.5468],
        [0.5430, 0.4461, 0.4559, 0.3755, 0.5551]], grad_fn=<MmBackward>)

Is the implementation and understanding of the dense synthesizer correct?

Theoretically, how is that different from a multi-layered perceptron that takes in two different inputs and makes uses of it at different point in the forward propagation?

Output:
Is the implementation and understanding of the dense synthesizer correct?
Not exactly, linear1 = nn.Linear(d,d) according to the paper and not (d,l).
Of course this does not work if X.shape = (l,d) according to matrix multiplication rules.
This is because :

So F is applied to each  Xi in X for i in [1,l]
The resulting matrix B is then passed to the softmax function and multiplied by G(x).
So you'd have to modify your code to sequentially process the input then use the returned matrix to compute Y.
how is that different from a multi-layered perceptron that takes in two different inputs and makes uses of it at different point in the forward propagation?
To understand, we need to put things into context, the idea of introducing attention mechanism was first described here in the context of Encoder - Decoder : https://arxiv.org/pdf/1409.0473.pdf
The core idea is to allow the model to have control over how the context vector from the encoder is retrieved using a neural network instead of relying solely on the last encoded state :

see this post for more detail.
The Transformers introduced the idea of using "Multi-Head Attention" (see graph below) to reduce the computational burden and focus solely on the attention mechanism itself. post
https://arxiv.org/pdf/1706.03762.pdf

So where does the Dense synthesizer fits into all of that ?
It simply replaces the Dot product (as illustrated in the first pictures in your post) by F(.). If you replace what's inside the softmax by F you get the equation for Y

Conclusion
This is an MLP but applied step wise to the input in the context of sequence processing.
Thank you


Instruction: No module named 'torch.autograd'

Input:
Working with torch package:

import torch

from torch.autograd import Variable

x_data = [1.0,2.0,3.0]

y_data = [2.0,4.0,6.0]

w = Variable(torch.Tensor([1.0]), requires_grad = True)

def forward(x):

return x*w

def loss(x,y):

y_pred = forward(x)

return (y_pred-y)*(y_pred-y)

print("my prediction before training",4,forward(4))

for epoch in range(10):

for x_val, y_val in zip(x_data,y_data):

l= loss(x_val, y_val)

l.backward()

print("\tgrad: ", x_val, y_val, w.grad.data[0])

w.data=w.data-0.01*w.grad.data

w.grad.data.zero_()

print("progress:", epoch, l.data[0] )

print("my new prediction after training ", forward(4))

Got error:

runfile('C:/gdrive/python/temp2.py', wdir='C:/gdrive/python')

Traceback (most recent call last):

File "C:\gdrive\python\temp2.py", line 11, in <module>

from torch.autograd import Variable

ModuleNotFoundError: No module named 'torch.autograd'

Command conda list pytorch brings:

# packages in environment at C:\Users\g\.conda\envs\test:

#

# Name Version Build Channel

(test) PS C:\gdrive\python>

How to fix this problem?

Output:
It seems to me that you have installed pytorch using conda.
Might be you have torch named folder in your current directory.
Try changing the directory, or try installing pytorch using pip.
This https://github.com/pytorch/pytorch/issues/1851 might help you to solve your problem.


Instruction: Pytorch - Distributed Data Parallel Confusion

Input:
I was just looking at the DDP Tutorial:

https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

According to this:

  It’s common to use torch.save and torch.load to checkpoint modules
  during training and recover from checkpoints. See SAVING AND LOADING
  MODELS for more details. When using DDP, one optimization is to save
  the model in only one process and then load it to all processes,
  reducing write overhead. This is correct because all processes start
  from the same parameters and gradients are synchronized in backward
  passes, and hence optimizers should keep setting parameters to the
  same values. If you use this optimization, make sure all processes do
  not start loading before the saving is finished. Besides, when loading
  the module, you need to provide an appropriate map_location argument
  to prevent a process to step into others’ devices. If map_location is
  missing, torch.load will first load the module to CPU and then copy
  each parameter to where it was saved, which would result in all
  processes on the same machine using the same set of devices. For more
  advanced failure recovery and elasticity support, please refer to
  TorchElastic.

I dont understand what this means. Shouldn't only one process/first GPU be saving the model? Is saving and loading how weights are shared across the processes/GPUs?

Output:
When you're using DistributedDataParallel you have the same model across multiple devices, which are being synchronised to have the exact same parameters.

  When using DDP, one optimization is to save the model in only one process and then load it to all processes, reducing write overhead.

Since they are identical, it is unnecessary to save the models from all processes, as it would just write the same parameters multiple times. For example when you have 4 processes/GPUs you would write the same file 4 times instead of once. That can be avoided by only saving it from the main process.

That is an optimisation for the saving of the model. If you load the model right after you saved it, you need to be more careful.

  If you use this optimization, make sure all processes do not start loading before the saving is finished.

If you save it in only one process, that process will take time to write the file. In the meantime all other processes continue and they might load the file before it was fully written to disk, which may lead to all sorts of unexpected behaviour or failure, whether that file does not exist yet, you are trying to read an incomplete file or you load an older version of the model (if you overwrite the same file).

  Besides, when loading the module, you need to provide an appropriate map_location argument to prevent a process to step into others’ devices. If map_location is missing, torch.load will first load the module to CPU and then copy each parameter to where it was saved, which would result in all processes on the same machine using the same set of devices.

When saving the parameters (or any tensor for that matter) PyTorch includes the device where it was stored. Let's say you save it from the process that used GPU 0 (device = "cuda:0"), that information is saved and when you load it, the parameters are automatically put onto that device. But if you load it in the process that uses GPU 1 (device = "cuda:1"), you will incorrectly load them into "cuda:0". Now instead of using multiple GPUs, you have the same model multiple times in a single GPU. Most likely, you will run out of memory, but even if you don't, you won't be utilising the other GPUs anymore.

To avoid that problem, you should set the appropriate device for map_location of torch.load.

torch.load(PATH, map_location="cuda:1")

# Or load it on the CPU and later use .to(device) on the model
torch.load(PATH, map_location="cpu")


Instruction: Running and building Pytorch on Google Colab

Input:
I am trying to run a python package that requires pytorch-gpu. I have change the runtime type of my Colab notebook to GPU. When I run the command, I am facing the following error. Not sure if I am able to build pytorch on colab myself?

Traceback (most recent call last):
  File "inference_unet.py", line 9, in <module>
    import torchvision.transforms as transforms
  File "/usr/local/lib/python3.6/dist-packages/torchvision/__init__.py", line 10, in <module>
    from .extension import _HAS_OPS
  File "/usr/local/lib/python3.6/dist-packages/torchvision/extension.py", line 58, in <module>
    _check_cuda_version()
  File "/usr/local/lib/python3.6/dist-packages/torchvision/extension.py", line 54, in _check_cuda_version
    .format(t_major, t_minor, tv_major, tv_minor))
RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=10.2 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.

Output:
Now you can directly use pytorch-gpu on google colab, no need of installation.
Just change your runtime to gpu, import torch and torchvision and you are done.
I have attached screenshot doing just the same.

Hope the answer will find helpful.

But in case you want to install different version of pytorch or any other package then you can install using pip, just add ! before your pip command and run the cell.

for example,


Instruction: What is vectorised way of doing this operation in pytorch instead of two FOR loops

Input:
Hello,
I have a tensor 'A' in Pytorch of dimesnsions Batch x Channel x Height x Width. I want to reshape it into 'B' such that dimesnions H and W are increased by 'r' and channels reduced by a factor of 'r^2'. For 'r'=2, the illustration is shown in figure attached.
In the figure if 'B' had 4 channels then first 4 channels, with 4th channel having border color violet, then first four channels in 'A' will be peack/skin colored pixels with border colors red, green, blue and violet and remaining channels adjusted accordingly.

I know the 'pack' and 'unpack' each can be done with 2 for loops. But that takes more time. There shoud be vectorised way in PyTorch to do switch between 'A' and 'B' with just reshape and pemutation commands. Can someone help me on that?

In this example, the batch size is set to 1. But if batch dimension is more I would like the operations shown in the figure to operate individually on each batch entry.

Can someone please help me with a generic code to switch between A and B in the vectorised way in PyTorch. Better if that also works when Batch size is more than 1.

Please note the two operations cannot be done with already implemented nn.PixelShuffle

Thanks a lot.

Output: Although this can be done with careful permutation and reshaping, pytorch has already implemented this with nn.PixelShuffle.


Instruction: How to read numerical data from CSV in PyTorch?

Input:
I'm new to PyTorch; trying to implement a model I developed in TF and compare the results. The model is an Autoencoder model. The input data is a csv file including n samples each with m features (a n*m numerical matrix in a csv file). The targets (the labels) are in another csv file with the same format as the input file. I've been looking online but couldn't find a good documentation for reading non-image data from csv file with multiple labels. Any idea how can I read my data and iterate over it during training?

Thank you

Output:
Might you be looking for something like TabularDataset?

  class
  torchtext.data.TabularDataset(path, format, fields, skip_header=False, csv_reader_params={}, **kwargs)

  Defines a Dataset of columns stored in CSV, TSV, or JSON format.

It will take a path to a CSV file and build a dataset from it. You also need to specify the names of the columns which will then become the data fields.

In general, all of implementations of torch.Dataset for specific types of data are located outside of pytorch in the torchvision, torchtext, and torchaudio libraries.


Instruction: How to move axis on simple numpy array

Input:
I'm having trouble moving the 3 axis to the 1 position. I would like to move the 3 to the first 69 position. This is for a machine learning dataset and PyTorch will only accept the data if it's in a 3x69x69 format. Thanks for any help!

 # To get the images and labels from file
with h5py.File(r"C:\Users\ajbur\Downloads\Galaxy10.h5", 'r') as F:
    images = np.array(F['images'])
    labels = np.array(F['ans'])
np.shape(images)
np.moveaxis(images,0,-1).shape
np.shape(images)

output is [20000, 69, 69, 3]
I want it to be [20000, 3, 69, 69]

Output:
The second and third arguments of moveaxis are source and destination. To move the last axis to the second position you could do:

a = np.empty([20000, 69, 69, 3])
np.moveaxis(a, -1, 1).shape
>>> (20000, 3, 69, 69)


Instruction: How to sort a tensor by first dimension

Input:
I have a 2D tensor and I would like to sort by the first dimension like this example:

a = torch.FloatTensor(
   [[5, 5],
    [5, 3],
    [3, 5],
    [6, 4],
    [3, 7]])

And I expected this result after sorting:

a = torch.FloatTensor(
   [[3, 5],
    [3, 7],
    [5, 3],
    [5, 5],
    [6, 4]])

Is it possible to do this in pytorch? I know that is possible to do it in numpy, but I want do it in GPU using torch.

Output:
Sort by first column and use the indices to then sort the whole array:

a[a[:, 0].sort()[1]]

Output:

tensor([[3., 5.],
        [3., 7.],
        [5., 5.],
        [5., 3.],
        [6., 4.]])

And if you really need it interleaved:

b = a[a[:, 1].sort()[1]]
b[b[:, 0].sort()[1]]

Output:

tensor([[3., 5.],
        [3., 7.],
        [5., 3.],
        [5., 5.],
        [6., 4.]])


Instruction: ImportError: cannot import name 'mobilenet_v2' from 'torchvision.models'

Input:
I want to run a fastai deep learning model on my pc. Not train, just run the pre-trained model on my PC. I have the .pth file. I tried to import the fastai module that I installed and I recieved the error :

ImportError: cannot import name 'mobilenet_v2' from 'torchvision.models' (C:\file_path\__init__.py)

The Code I tried to execute:

#From the fastai library
from fastai import *
from torchvision.models import *
from fastai.vision import *

I can't find any solutions as to why I am getting this error.

Im running this code in anaconda, to be specific, the spyder IDE connected to my anaconda environment. I will re-edit this if anyone needs more specifications. Thank you.

Output:
I just finished fixing this problem with my system. Uninstall any pytorch, torchvision by conda and pip. Uninstall fastai as well.

Go to https://pytorch.org/get-started/locally/ and run the conda command there base on your cuda version and etc. Then

conda install -c fastai fastai


Instruction: Reading csv.gz file in torchtext

Input:
Pandas’s read_csv works for csv.gz as well.

Is there a way to achieve similar with PyTorch?https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Dataset doesn’t seem to have such an option.

Output:
TLDR: No, this is not supported by TabularDataset

torchtext.data.TabularDataset uses csv.reader.

Using csvreader against a gzipped file in Python suggests if you open the file with gzip.open, csv.reader can read it.

However, TabularDataset asks for a file path, not a file pointer, so digging into the source code, it uses

io.open(os.path.expanduser(path), encoding="utf8")

To open the filepath. Since .gz is not utf8, this won't read the file correctly.


Instruction: Pytorch: Loading sample of images using DataLoader

Input:
I use standard DataLoader from torch.utils.data. I create dataset class and then build DataLoader this way:

train_dataset = LandmarksDataset(os.path.join(args.data, 'train'), train_transforms, split="train")
train_dataloader = data.DataLoader(train_dataset, batch_size=args.batch_size, num_workers=2,
                                   pin_memory=True, shuffle=True, drop_last=True)

It works perfect, but dataset is big enough - 300k of images. So it takes a lot of time for reading images on using DataLoader. So it is really wretchedly to build such big DataLoader on debug stage! I just want to test some my hypothesis and want to do it fast! I don't need to load whole dataset for this.

I'm trying to find the way How to load just a small fixed part of dataset without building dataLoader on whole dataset?
At current moment all my ideas are just create another folder, copy some part of images here and use pipeline on it. But I suppose, Pytorch is clever enough to have some builtin methods for loading just a part of images from big dataset. Can you give me advice how to?

Output:
As far as I am aware there's no mechanism that does this for you. Your problem is in the LandmarksDataset class at the point where you're reading the paths of your train data folder. I assume os.listdir(train_data_folder).

Instead you could use a more efficient way os.scandir(train_data_folder) this returns a generator and calling next() on it will give you paths to your images within the train data. This way you can call next() as many times without changing the structure of your train data folder and build a subset of it.


Instruction: Pytorch: Convert 2D-CNN model to tflite

Input:
I'd like to convert a model (eg Mobilenet V2) from pytorch to tflite in order to run it on a mobile device.

Has anyone managed to do so?

All I found, was a method that uses ONNX to convert the model into an inbetween state. However, this seems not to work properly, as Tensorflow expects a NHWC-channel order whereas onnx and pytorch work with NCHW channel order.

There is a discussion on github, however in my case the conversion worked without complaints until a "frozen tensorflow graph model", after trying to convert the model further to tflite, it complains about the channel order being wrong...

Here is my code so far:

import torch
import torch.onnx
import onnx
from onnx_tf.backend import prepare

# Create random input
input_data = torch.randn(1,3,224,224)

# Create network
model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True)
model.eval()

# Forward Pass
output = model(input_data)

# Export model to onnx
filename_onnx = "mobilenet_v2.onnx"
filename_tf = "mobilenet_v2.pb"

torch.onnx.export(model, input_data, filename_onnx)

# Export model to tensorflow
onnx_model = onnx.load(filename_onnx)
tf_rep = prepare(onnx_model)
tf_rep.export_graph(filename_tf)

All working without errors until here (ignoring many tf warnings). Then I look up the names of the input and output tensors using netron ("input.1" and "473").

Finally I apply my usual tf-graph to tf-lite conversion script from bash:

tflite_convert \
    --output_file=mobilenet_v2.tflite \
    --graph_def_file=mobilenet_v2.pb \
    --input_arrays=input.1 \
    --output_arrays=473

My configuration:

torch                1.6.0.dev20200508 (needs pytorch-nightly to work with mobilenet V2 from torch.hub)
tensorflow-gpu       1.14.0
onnx                 1.6.0
onnx-tf              1.5.0

Here is the exact error message I'm getting from tflite:

Unexpected value for attribute 'data_format'. Expected 'NHWC'
Fatal Python error: Aborted

UPDATE:
Updating my configuration:

torch                1.6.0.dev20200508
tensorflow-gpu       2.2.0
onnx                 1.7.0
onnx-tf              1.5.0

using

tflite_convert \
    --output_file=mobilenet_v2.tflite \
    --graph_def_file=mobilenet_v2.pb \
    --input_arrays=input.1 \
    --output_arrays=473 \
    --enable_v1_converter  # <-- needed for conversion of frozen graphs

leading to another error:

Exception: <unknown>:0: error: loc("convolution"): 'tf.Conv2D' op is neither a custom op nor a flex op

Update:
Here is an onnx model of mobilenet v2 loaded via netron:

Here  is a gdrive link to my converted onnx and pb file

Output:
@Ahwar posted a nice solution to this using a Google Colab notebook.

It uses

torch                    1.5.0+cu101
torchsummary             1.5.1
torchtext                0.3.1
torchvision              0.6.0+cu101

tensorflow               1.15.2
tensorflow-addons        0.8.3
tensorflow-estimator     1.15.1

onnx                     1.7.0
onnx-tf                  1.5.0

The conversion is working and the model can be tested on my computer. However when pushing the model to the mobile phone it only works in CPU mode and is much slower (almost 10 fold) than a corresponding model created in tensorflow directly. GPU mode is not working on my mobile phone (in contrast to the corresponding model created in tensorflow directly)

Update:
Apparantly after converting the mobilenet v2 model, the tensorflow frozen graph contains many more convolution operations than the original pytorch model ( ~38 000 vs ~180 ) as discussed in this github issue.


Instruction: How most efficiently compute the diagonal of a matrix product

Input:
I want to compute the following:

import numpy as np
n= 3
m = 2
x = np.random.randn(n,m)

#Method 1
y = np.zeros(m)
for i in range(m):
    y[i] = x[:,i] @ x[:,i]

#Method 2
y2 = np.diag(x.T @ x)

The first method has the problem that it uses a for loop, which can't be very effecient (I need to do this in pytorch on a GPU millions of times)

The second method computes the full matrix product, when I only need the diagonal entries, so that can't be very efficient either.

I'm wondering whether there exist any clever way of doing this?

Output:
Use a manually constructed sum-product. You want the sums of the squares of the individual columns:

y = (x * x).sum(axis=0)

As Divakar suggests, np.einsum will likely offer a less memory-intensive option, since it does not require the temporary array x * x:

y = np.einsum('ij,ij->j', x, x)


Instruction: Bert pre-trained model giving random output each time

Input:
I was trying to add an additional layer after huggingface bert transformer, so I used BertForSequenceClassification inside my nn.Module Network. But, I see the model is giving me random outputs when compared to loading the model directly.

Model 1:

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5) # as we have 5 classes

import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode(texts[0], add_special_tokens=True, max_length = 512)).unsqueeze(0)  # Batch size 1

print(model(input_ids))

Out:

(tensor([[ 0.3610, -0.0193, -0.1881, -0.1375, -0.3208]],
        grad_fn=<AddmmBackward>),)

Model 2:

import torch
from torch import nn

class BertClassifier(nn.Module):
    def __init__(self):
        super(BertClassifier, self).__init__()
        self.bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5)
        # as we have 5 classes

        # we want our output as probability so, in the evaluation mode, we'll pass the logits to a softmax layer
        self.softmax = torch.nn.Softmax(dim = 1) # last dimension
    def forward(self, x):
        print(x.shape)
        x = self.bert(x)

        if self.training == False: # in evaluation mode
            pass
            #x = self.softmax(x)

        return x

# create our model

bertclassifier = BertClassifier()

print(bertclassifier(input_ids))

torch.Size([1, 512])
torch.Size([1, 5])
(tensor([[-0.3729, -0.2192,  0.1183,  0.0778, -0.2820]],
        grad_fn=<AddmmBackward>),)

They should be the same model, right. I found a similar issue here but no reasonable explanation https://github.com/huggingface/transformers/issues/2770

Does Bert has some ranomized parameter if so how to get reproducible output?
Why the two models give me different outputs? Is there something I'm doing wrong?

Output:
The reason is due to the random initialization of the classifier layer of Bert. If you print your model, you'll see

    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=5, bias=True)
)

There is a classifier in the last layer, this layer is added after bert-base. Now, the expectation is you'll train this layer for your downstream task.

If you want to get more insight:

model, li = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5, output_loading_info=True) # as we have 5 classes
print(li)

{'missing_keys': ['classifier.weight', 'classifier.bias'], 'unexpected_keys': ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'], 'error_msgs': []}

You can see the classifier.weight and bias are missing, so these part will be randomly initialized each time you call BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5).


Instruction: Neural network in pytorch

Input: I wanna create a Neural Network in PyTorch, that will have 2 inputs and 3 outputs with 1 hidden layer. The two inputs will be float numbers that represents features of an image and 3 outputs will be real numbers between 0 and 1. For example output (1, 0, 0) would mean that it is square and (0,1,0) would mean it is rectangle. Any idea how to do it in pytorch?

Output:
The network can be defined like this:

import torch
import torch.nn as nn
import torch.optim as optim
import torch.autograd as autograd
import torch.nn.functional as F
from torch.autograd import Variable

class Net(nn.Module):
    def __init__(self, num_inputs=2, num_outputs=3,hidden_dim=5):
        # define your network here
        super(Net, self).__init__()
        self.layer1 = nn.Linear(num_inputs,hidden_dim)
        self.layer2 = nn.Linear(hidden_dim,num_outputs)

    def forward(self, x):
        # implement the forward pass
        x = F.relu(self.layer1(x))
        x = F.sigmoid(self.layer2(x))
        return x

Although I have defined the network here, you should maybe look at some examples on the official pytorch website for example on how to train your model.


Instruction: 'Net' object has no attribute 'parameters'

Input:
I am fairly new to machine learning. I learned to write this code from youtube tutorials but I keep getting this error

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/aniket/Desktop/DeepLearning/PythonLearningPyCharm/CatVsDogs.py", line 109, in <module>
    optimizer = optim.Adam(net.parameters(), lr=0.001) # tweaks the weights from what I understand
AttributeError: 'Net' object has no attribute 'parameters'

this is the Net class

class Net():
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,32,5)
        self.conv2 = nn.Conv2d(32,64,5)
        self.conv3 = nn.Conv2d(64,128,5)
        self.to_linear = None
        x = torch.randn(50,50).view(-1,1,50,50)
        self.Conv2d_Linear_Link(x)
        self.fc1 = nn.Linear(self.to_linear, 512)
        self.fc2 = nn.Linear(512, 2)

    def Conv2d_Linear_Link(self , x):
        x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)),(2,2))
        x = F.max_pool2d(F.relu(self.conv3(x)),(2,2))

        if self.to_linear is None :
            self.to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.Conv2d_Linear_Link(x)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

and this is the function train

def train():
    for epoch in range(epochs):
        for i in tqdm(range(0,len(X_train), batch)):
            batch_x = train_X[i:i + batch].view(-1, 1, 50, 50)
            batch_y = train_y[i:i + batch]
            net.zero_grad() # i don't understand why we do this but we do we don't want the probabilites adding up
            output = net(batch_x)
            loss = loss_function(output, batch_y)
            loss.backward()
            optimizer.step()
        print(loss)

and the optimizer and loss functions and data

optimizer = optim.Adam(net.parameters(), lr=0.001) # tweaks the weights from what I understand
loss_function = nn.MSELoss() # gives the loss

Output:
You're not subclassing nn.Module. It should look like this:

class Net(nn.Module):
    def __init__(self):
        super().__init__()

This allows your network to inherit all the properties of the nn.Module class, such as the parameters attribute.


Instruction: ImportError: cannot import name 'Optional' from 'torch.jit.annotations'

Input:
I have installed cpuonly pytorch and torchvision in anaconda. But when i try to import torchvision i get the following error.
ImportError: cannot import name 'Optional' from 'torch.jit.annotations'(C:\Users\MSI\Anaconda3\lib\site-packages\torch\jit\annotations.py)
How can i fix this?

Output:
Not sure if you are installing the correct versions of the libraries. This combination seems to work:

conda create --name test5 python=3.6
conda install -c pytorch pytorch torchvision cpuonly
python
>>> import torchvision


Instruction: how to keep pytorch model in redis cache to access model faster for video streaming?

Input:
I have this code belonging to feature_extractor.py which is a part of this folder in here:

import torch
import torchvision.transforms as transforms
import numpy as np
import cv2
from .model import Net

class Extractor(object):
    def __init__(self, model_path, use_cuda=True):
        self.net = Net(reid=True)
        self.device = "cuda" if torch.cuda.is_available() and use_cuda else "cpu"
        state_dict = torch.load(model_path, map_location=lambda storage, loc: storage)['net_dict']
        self.net.load_state_dict(state_dict)
        print("Loading weights from {}... Done!".format(model_path))
        self.net.to(self.device)
        self.size = (64, 128)
        self.norm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])

    def _preprocess(self, im_crops):
        def _resize(im, size):
            return cv2.resize(im.astype(np.float32) / 255., size)

        im_batch = torch.cat([self.norm(_resize(im, self.size)).unsqueeze(0) for im in im_crops], dim=0).float()
        return im_batch

    def __call__(self, im_crops):
        im_batch = self._preprocess(im_crops)
        with torch.no_grad():
            im_batch = im_batch.to(self.device)
            features = self.net(im_batch)
        return features.cpu().numpy()

if __name__ == '__main__':
    img = cv2.imread("demo.jpg")[:, :, (2, 1, 0)]
    extr = Extractor("checkpoint/ckpt.t7")
    feature = extr(img)
    print(feature.shape)

Now Imagine 200 requests are in row to proceed. The process of loading model for each request makes the code run slowly.

So I thought it might be a good idea to keep the pytorch model in cache. I modified it like this:

from redis import Redis
import msgpack as msg

r = Redis('111.222.333.444')

class Extractor(object):
    def __init__(self, model_path, use_cuda=True):
        try:
            self.net = msg.unpackb(r.get('REID_CKPT'))
        finally:
            self.net = Net(reid=True)
            self.device = "cuda" if torch.cuda.is_available() and use_cuda else "cpu"
            state_dict = torch.load(model_path, map_location=lambda storage, loc: storage)['net_dict']
            self.net.load_state_dict(state_dict)
            print("Loading weights from {}... Done!".format(model_path))
            self.net.to(self.device)
            packed_net = msg.packb(self.net)
            r.set('REID_CKPT', packed_net)

        self.size = (64, 128)
        self.norm = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
        ])

Unfortunately this error comes up:

 File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
 File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
 TypeError: can not serialize 'Net' object

The reason obviously is because that it cannot convert Net object (pytorch nn.Module class) to bytes.

How can I efficiently save pytorch model in cache (or somehow keep it in RAM) and call for it for each request?

Thanks everyone.

Output: If you only need to keep model state on RAM, Redis is not necessary. You could instead mount RAM as a virtual disk and store model state there. Check out tmpfs.


Instruction: Obtain torch.tensor from string of floats

Input:
We can convert 1 dimensional array of floats, stored as a space separated numbers in text file, in to a numpy array or a  torch tensor as follows.

line = "1 5 3 7 4"
np_array = np.fromstring(line, dtype='int', sep=" ")

np_array
>> array([1, 5, 3, 7, 4])

And to convert above numpy array to a torch tensor, we can do following :

torch_tensor = torch.tensor(np_array)
torch_tensor
>>tensor([1, 5, 3, 7, 4])

How can I convert a string of numbers separated by space in to a torch.Tensor directly without
converting them to a numpy array? We can also do this by fist splitting the string at a space, mapping them to int or float, and then feeding it to torch.tensor. But like numpy's fromstring, is there any such method in pytorch?

Output:
What about

x = torch.tensor(list(map(float, line.split(' '))), dtype=torch.float32)


Instruction: volatile was removed and now had no effect use with.torch.no_grad() instread

Input:
my torch program stopped at this point
I guess i can not use volatile=True
how should I change it and what is the reason to stop?
 and How should I change this code?

images = Variable(images.cuda())
targets = [Variable(ann.cuda(), volatile=True) for ann in targets]

  train.py:166: UserWarning: volatile was removed and now has no effect.
  Use with torch.no_grad(): instead.

Output:
Variable doesn't do anything and has been deprecated since pytorch 0.4.0. Its functionality was merged with the torch.Tensor class. Back then the volatile flag was used to disable the construction of the computation graph for any operation which the volatile variable was involved in. Newer pytorch has changed this behavior to instead use with torch.no_grad(): to disable construction of the computation graph for anything in the body of the with statement.
What you should change will depend on your reason for using volatile in the first place. No matter what though you probably want to use
images = images.cuda()
targets = [ann.cuda() for ann in targets]

During training you would use something like the following so that the computation graph is created (assuming standard variable names for model, criterion, and optimizer).
output = model(images)
loss = criterion(images, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()

Since you don't need to perform backpropagation during evaluation you would use with torch.no_grad(): to disable the creation of the computation graph which reduces the memory footprint and speeds up computation.
with torch.no_grad():
    output = model(images)
    loss = criterion(images, targets)


Instruction: computation graph of setting weights in pytorch

Input:
I need a clarification of code written for some function in FastAI2 library.

this is the code WeightDropout written in FastAI2 library.

 class WeightDropout(Module):
        "A module that warps another layer in which some weights will be replaced by 0 during training."

        def __init__(self, module, weight_p, layer_names='weight_hh_l0'):
            self.module,self.weight_p,self.layer_names = module,weight_p,L(layer_names)
            for layer in self.layer_names:
                #Makes a copy of the weights of the selected layers.
                w = getattr(self.module, layer)
                delattr(self.module, layer)
                self.register_parameter(f'{layer}_raw', nn.Parameter(w.data))
                setattr(self.module, layer, F.dropout(w.data, p=self.weight_p, training=False))
                if isinstance(self.module, (nn.RNNBase, nn.modules.rnn.RNNBase)):
                    self.module.flatten_parameters = self._do_nothing

        def _setweights(self):
            "Apply dropout to the raw weights."
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))

        def forward(self, *args):
            self._setweights()
            with warnings.catch_warnings():
                #To avoid the warning that comes because the weights aren't flattened.
                warnings.simplefilter("ignore")
                return self.module.forward(*args)

        def reset(self):
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer,
    F.dropout(raw_w.data, p=self.weight_p, training=False))
            if hasattr(self.module, 'reset'): self.module.reset()

        def _do_nothing(self): pass

where above code randomly drops weights in weight matrix of hidden layers.I am primarily interested in ,

 def _setweights(self):
                "Apply dropout to the raw weights."
                for layer in self.layer_names:
                    raw_w = getattr(self, f'{layer}_raw')
                    setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))

my question is that, does this operation of changing weights is recorded in gradient computation.

Output:
No, assigning a new weight is not tracked in the computational graph, because an assignment has no derivative, therefore it's impossible to get a gradient through it.

Then why does that code work? The model is not overwriting the actual parameters, but it's using a modified version for the calculations, while keeping the original weights unchanged. It's a little obscure, but the most important part is when the parameters are copied when the model is created:

#Makes a copy of the weights of the selected layers.
w = getattr(self.module, layer)
delattr(self.module, layer)
self.register_parameter(f'{layer}_raw', nn.Parameter(w.data))

What happens here, is that for every parameters you create a copy which ends in _raw. For example, if you have a linear layer on your model (e.g. self.linear1 = nn.Linear(2, 4), you have two parameters with the names linear1.weight and linear1.bias. Now they are copied to linear1.weight_raw and linear1.bias_raw. To be precise, they are not copied, but reassigned to the *_raw attributes and then the original ones are deleted, hence they are just moved from the original to the raw versions. The originals need to be deleted, since they are no longer parameters (which would be optimised/learned).

Afterwards, when the dropout is applied, the parameters that are optimised/learned (*_raw versions) are unchanged, but the weight used for the actual calculations is the one with some weights randomly dropped. In the example with the linear layer that would look as follows if you do the calculations manually:

# A dummy input
input = torch.randn(1, 2)

# The raw parameters of the linear layer, randomly initialised
weight_raw = nn.Parameter(torch.randn(4, 2))
bias_raw = nn.Parameter(torch.randn(4))

# Randomly dropping elements of the parameters with 50% probability
weight = F.dropout(weight_raw, p=0.5)
bias = F.dropout(bias_raw, p=0.5)

# Calculation of the linear layer (forward)
output = torch.matmul(input, weight.transpose(0, 1)) + bias

From this you can see that there is no actual reassignment, but just the regular computational flow that you are familiar with.

Now you might be wondering why these *_raw parameters are created instead of applying the dropout in the forward pass (like in the example above). The reason for that is to avoid having to reimplement the forward pass, otherwise every module would need to have their forward method modified, but since they differ widely across modules, that cannot be done in a generic manner. This approach essentially hijacks the parameters, so that the forward pass uses a modified version of them.

Continuing the example from above:

# Using the actual module for the same calculation
linear1 = nn.Linear(2, 4)

# Delete the parameters, so that regular tensors can be assigned to them
# Otherwise it throws an error that the tensor is not an nn.Parameter
del linear1.weight
del linear1.bias

# Assign the parameters with dropped elements
linear1.weight = weight
linear1.bias = bias

# Run the forward pass directly
output_linear1 = linear1(input)

torch.equal(output, output_linear1) # => True

The bottom line is that the parameters are extracted from the modules, and the forward pass uses a modified version (after dropout) for the calculations, they are no longer parameters but intermediate results.


Instruction: Pytorch TypeError - eq() received an invalid combination of arguments

Input:
I'm working on a text classification problem with BERT. When training on the local machine everything works just fine, but when switching to the server, I get the following error:

<ipython-input-28-508d35ac5f5f> in flat_accuracy(preds, labels)
      5     pred_flat = np.argmax(preds, axis=1).flatten()
      6     labels_flat = labels.flatten()
----> 7     return np.sum(pred_flat == labels_flat) / len(labels_flat)
      8
      9 # Function to calculate the f1_score of our predictions vs labels

TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (numpy.ndarray)
 * (Number other)
      didn't match because some of the arguments have invalid types: (numpy.ndarray)

Code:

def flat_accuracy(preds, labels):
    pred_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return np.sum(pred_flat == labels_flat) / len(labels_flat)

Torch version on local machine: 1.4.0

Torch version on the server: 1.3.1

Any help would be greatly appreciated!

Output:
It could be that the eq implementation of the torch version on your server no longer lets you do elementwise comparison between a torch.Tensor and a np.ndarray. You should coerce either pred_flat to be a torch.Tensor, or coerce labels_flat to be a numpy array. Since you're using np.sum in the return statement and you are just returning a scalar value, I'd just move everything to numpy, so

labels_flat = labels.numpy()

but if you're on the GPU you may need to call labels.cpu().numpy(), and if you're tracking gradients on labels you might need labels.detach().cpu().numpy().


Instruction: Concatenating two torch tensors of different shapes in pytorch

Input:
I have two torch tensors. One with shape [64, 4, 300], and one with shape [64, 300]. How can I concatenate these two tensors to obtain the resultant tensor of shape [64, 5, 300]. I'm aware about the tensor.cat function used for this, but in order to use that function, I need to reshape the second tensor in order to match the number of dimensions of the tensor. I've heard that reshaping of the tensors should not be done, as it might mess up the data in the tensor. How can I do this concatenation?

I've tried reshaping, but following part makes me more doubtful about such reshaping.

a = torch.rand(64,300)

a1 = a.reshape(64,1,300)

list(a1[0]) == list(a)
Out[32]: False

Output:
You have to use torch.cat along first dimension and do unsqueeze at the first one as well, like this:

import torch

first = torch.randn(64, 4, 300)
second = torch.randn(64, 300)

torch.cat((first, second.unsqueeze(dim=1)), dim=1)
# Shape: [64, 5, 300]

It won't mess up with your data, it's only adding superficial 1 dimension (reshape doesn't if done correctl anyway).


Instruction: No module named 'torch.nn.functional'

Input:
I have python file with lines:

import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

It generates errors:

  File "C:\gdrive\python\a.py", line 5, in <module>
import torch.nn.functional as F
ModuleNotFoundError: No module named 'torch.nn.functional'

How to fix that error?

I have installed pytorch by using command:

conda install pytorch-cpu torchvision-cpu -c pytorch

Output:
It looks like you have an outdated version of PyTorch. Conda - pytorch-cpu was last published over a year ago and its latest version of PyTorch is 1.1.0, whereas PyTorch is currently at version 1.5.0. That packages has been abandoned.

You should install PyTorch with the official instructions given on PyTorch - Get Started locally, by selecting the version you want. In your case that would be Conda with CUDA None (to get the CPU only version).

The resulting command is:

conda install pytorch torchvision cpuonly -c pytorch


Instruction: Gradient clipping in pytorch has no effect (Gradient exploding still happens)

Input:
I have an exploding gradient problem when train the minibatch for 150-200 epochs with batch size = 256 and there’s about 30-60 minibatch (This depends on my specific config). But I have an exploding gradient issues even if I add the code below.

As you can see this below images, notice that in step about 40k there’s the swing of gradients between ± 20k, 40k and 60k respectively. I don’t know why this happens because i use the clip_grad_value_ above. Also Using the learning rate decay from 0.01 to about 0.008 at step 40k.

Or do I need to update the weight parameters by myself something like this
image
But i think optimizer.step() should do the job and the clip_grad_value_ is an inplace operation so i don’t need to take the return value from function. Please correct if i did anything wrong. Thank you very much

Best regards,
Mint

Output:
Your code looks right, but try using a smaller value for the clip-value argument. Here's the documentation on the clip_grad_value_() function you're using, which shows that each individual term in the gradient is set such that its magnitude does not exceed the clip value.
You have clip value set to 100, so if you have 100 parameters then abs(gradient).sum() can be as large as 10,000 (100*100).


Instruction: How does the Transformer Model Compute Self Attention?

Input:
In the transformer model, https://arxiv.org/pdf/1706.03762.pdf there is self-attention which is computed using softmax on Query (Q) and Key (K) vectors:

I am trying to understand the matrix multiplications:

Q = batch_size x seq_length x embed_size

K = batch_size x seq_length x embed_size

QK^T = batch_size x seq_length x seq_length

Softmax QK^T = Softmax (batch_size x seq_length x seq_length)

How is the softmax computed since there are seq_length x seq_length values per batch element?

A reference to Pytorch computation will be very helpful.

Cheers!

Output:
  How is the softmax computed since there are seq_length x seq_length values per batch element?

The softmax is performed on w.r.t the last axis (torch.nn.Softmax(dim=-1)(tensor) where tensor is of shape batch_size x seq_length x seq_length) to get the probability of attending to every element for each element in the input sequence.

Let's assume, we have a text sequence "Thinking Machines", so we have a matrix of shape "2 x 2" (where seq_length = 2) after performing QK^T.

I am using the following illustration (reference) to explain self-attention computation. As you know, first scaled-dot-product is performed QK^T/square_root(d_k) and then softmax is computed for each sequence element.

Here, Softmax is performed for the first sequence element "Thinking". The raw score of 14 and 12 is turned into a probability of 0.88 and 0.12 by doing softmax. These probability indicates that the token "Thinking" would attend itself with 88% probability, and the token "Machines"  with 12% probability. Similarly, the attention probability is computed for the token "Machines" too.

Note. I strongly suggest reading this excellent article on Transformer. For implementation, you can take a look at OpenNMT.


Instruction: How to see the adapted learning rate for Adam in pytorch?

Input:
There are many different optimizers with adaptive learning rate methods. Is it possible to see the adapted value of the initial learning rate for Adam?

Here is a similar question about Adadelta and the answer was to search for ["acc_delta"] key, but Adam has no that key.

Output:
AFAIK there is no super easy way to do this. However, you can recalculate the current learning rate of a certain paramter using the implementation of Adam in PyTorch: https://pytorch.org/docs/stable/_modules/torch/optim/adam.html
I came up with this minimal working example:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

def get_current_lr(optimizer, group_idx, parameter_idx):
    # Adam has different learning rates for each paramter. So we need to pick the
    # group and paramter first.
    group = optimizer.param_groups[group_idx]
    p = group['params'][parameter_idx]

    beta1, _ = group['betas']
    state = optimizer.state[p]

    bias_correction1 = 1 - beta1 ** state['step']
    current_lr = group['lr'] / bias_correction1
    return current_lr

x = Variable(torch.randn(100, 1)) #Just create a random tensor as input
model = nn.Linear(1, 1)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
niter = 20
for _ in range(0, niter):
    out = model(x)

    optimizer.zero_grad()
    loss = criterion(out, x) #Here we learn the identity mapping
    loss.backward()
    optimizer.step()
    group_idx, param_idx = 0, 0
    current_lr = get_current_lr(optimizer, group_idx, param_idx)
    print('Current learning rate (g:%d, p:%d): %.4f | Loss: %.4f'%(group_idx, param_idx, current_lr, loss.item()))

which should output something like this:
Current learning rate (g:0, p:0): 0.0100 | Loss: 0.5181
Current learning rate (g:0, p:0): 0.0053 | Loss: 0.5161
Current learning rate (g:0, p:0): 0.0037 | Loss: 0.5141
Current learning rate (g:0, p:0): 0.0029 | Loss: 0.5121
Current learning rate (g:0, p:0): 0.0024 | Loss: 0.5102
Current learning rate (g:0, p:0): 0.0021 | Loss: 0.5082
Current learning rate (g:0, p:0): 0.0019 | Loss: 0.5062
Current learning rate (g:0, p:0): 0.0018 | Loss: 0.5042
Current learning rate (g:0, p:0): 0.0016 | Loss: 0.5023
Current learning rate (g:0, p:0): 0.0015 | Loss: 0.5003
Current learning rate (g:0, p:0): 0.0015 | Loss: 0.4984
Current learning rate (g:0, p:0): 0.0014 | Loss: 0.4964
Current learning rate (g:0, p:0): 0.0013 | Loss: 0.4945
Current learning rate (g:0, p:0): 0.0013 | Loss: 0.4925
Current learning rate (g:0, p:0): 0.0013 | Loss: 0.4906
Current learning rate (g:0, p:0): 0.0012 | Loss: 0.4887
Current learning rate (g:0, p:0): 0.0012 | Loss: 0.4868
Current learning rate (g:0, p:0): 0.0012 | Loss: 0.4848
Current learning rate (g:0, p:0): 0.0012 | Loss: 0.4829
Current learning rate (g:0, p:0): 0.0011 | Loss: 0.4810

Note that monitoring the learning rate of every individual paramter is probably not feasible nor helpful for larger models.


Instruction: How to get the probability of a particular token(word) in a sentence given the context

Input:
I'm trying to calculate the probability or any type of score for words in a sentence using NLP. I've tried this approach with GPT2 model using Huggingface Transformers library, but, I couldn't get satisfactory results due to the model's unidirectional nature which for me didn't seem to predict within context. So I was wondering whether there is a way, to calculate the above said using BERT since it's Bidirectional.

I've found this post relatable, which I randomly saw the other day but didn't see any answer which would be useful for me as well.

Hope I will be able to receive ideas or a solution for this. Any help is appreciated. Thank you.

Output:
BERT is trained as a masked language model, i.e., it is trained to predict tokens that were replaced by a [MASK] token.
from transformers import AutoTokenizer, BertForMaskedLM

tok = AutoTokenizer.from_pretrained("bert-base-cased")
bert = BertForMaskedLM.from_pretrained("bert-base-cased")

input_idx = tok.encode(f"The {tok.mask_token} were the best rock band ever.")
logits = bert(torch.tensor([input_idx]))[0]
prediction = logits[0].argmax(dim=1)
print(tok.convert_ids_to_tokens(prediction[2].numpy().tolist()))

It prints token no. 11581 which is:

Beatles

To get a normalized probability distribution over BERT's vocabulary, you can normalize the logits using the softmax function, i.e., F.softmax(logits, dim=1), (assuming standart import torch.nn.fucntional as F).
The tricky thing is that words might be split into multiple subwords. You can simulate that by adding multiple [MASK] tokens, but then you have a problem with how to compare the scores of prediction so different lengths reliably. I would probably average the probabilities, but maybe there is a better way.


Instruction: Python get pytorch tensor size

Input:
I wanna know how to get the shape of this tensor in Python ? I have tried this :

> len(x)

But this prints 1, why ? I want to print the number of tuples here which is 3. Using len(x) prints only 1.

What's the problem ?

Here's the tensor :

 (x=array([[[[ 0.07499999,  0.        ],
         [ 0.0703125 ,  0.        ],
         [ 0.0703125 ,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.1953125 ,  0.        ],
         [ 0.05312502,  0.        ],
         [ 0.2890625 ,  0.        ],
         [ 0.015625  ,  0.        ],
         [ 0.32656252,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.23906249,  0.        ],
         [ 0.09218752,  0.        ],
         [ 0.22812498,  0.        ],
         [ 0.06406248,  0.        ],
         [ 0.19062501,  0.        ],
         [ 0.02031249,  0.        ],
         [ 0.17343748,  0.        ]],

        [[ 0.06875002,  0.        ],
         [ 0.06875002,  0.        ],
         [ 0.06875002,  0.        ],
         [ 0.09062499,  0.        ],
         [ 0.19375002,  0.        ],
         [ 0.05781251,  0.        ],
         [ 0.2921875 ,  0.        ],
         [ 0.01406252,  0.        ],
         [ 0.325     ,  0.        ],
         [ 0.08437502,  0.        ],
         [ 0.23124999,  0.        ],
         [ 0.09531248,  0.        ],
         [ 0.22031248,  0.        ],
         [ 0.06406248,  0.        ],
         [ 0.18906248,  0.        ],
         [ 0.02031249,  0.        ],
         [ 0.171875  ,  0.        ]],

        [[ 0.06718749,  0.        ],
         [ 0.06093752,  0.        ],
         [ 0.07187498,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.18593752,  0.        ],
         [ 0.03437501,  0.        ],
         [ 0.2765625 ,  0.        ],
         [-0.00312501,  0.        ],
         [ 0.29843748,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.21718752,  0.        ],
         [ 0.078125  ,  0.        ],
         [ 0.21249998,  0.        ],
         [ 0.07187498,  0.        ],
         [ 0.19062501,  0.        ],
         [ 0.13749999,  0.        ],
         [ 0.1796875 ,  0.        ]]]], dtype=float32), 0)

Output: It looks like your 3 tuples are located within the first (and only) index of x. In this case, len(x[0]) yields 3.


Instruction: Pytorch RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Input:
This code is built up as follows: My robot takes a picture, some tf computer vision model calculates where in the picture the target object starts. This information (x1 and x2 coordinate) is passed to a pytorch model. It should learn to predict the correct motor activations, in order to get closer to the target. After the movement is executed, the robot takes a picture again and the tf cv model should calculate whether the motor activation brought the robot closer to the desired state (x1 at 10, x2 coordinate at at31)

However every time i run the code pytorch is not able to calculate the gradients.

I'm wondering if this is some data-type problem or if it is a more general one: Is it impossible to calculate the gradients if the loss is not calculated directly from the pytorch network's output?

Any help and suggestions will be greatly appreciated.

#define policy model (model to learn a policy for my robot)
import torch
import torch.nn as nn
import torch.nn.functional as F
class policy_gradient_model(nn.Module):
    def __init__(self):
        super(policy_gradient_model, self).__init__()
        self.fc0 = nn.Linear(2, 2)
        self.fc1 = nn.Linear(2, 32)
        self.fc2 = nn.Linear(32, 64)
        self.fc3 = nn.Linear(64,32)
        self.fc4 = nn.Linear(32,32)
        self.fc5 = nn.Linear(32, 2)
    def forward(self,x):
        x = self.fc0(x)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        return x

policy_model = policy_gradient_model().double()
print(policy_model)
optimizer = torch.optim.AdamW(policy_model.parameters(), lr=0.005, betas=(0.9,0.999), eps=1e-08, weight_decay=0.01, amsgrad=False)

#make robot move as predicted by pytorch network (not all code included)
def move(motor_controls):
#define curvature
 #   motor_controls[0] = sigmoid(motor_controls[0])
    activation_left = 1+(motor_controls[0])*99
    activation_right = 1+(1- motor_controls[0])*99

    print("activation left:", activation_left, ". activation right:",activation_right, ". time:", motor_controls[1]*100)

#start movement

#main
import cv2
import numpy as np
import time
from torch.autograd import Variable
print("start training")
losses=[]
losses_end_of_epoch=[]
number_of_steps_each_epoch=[]
loss_function = nn.MSELoss(reduction='mean')

#each epoch
for epoch in range(2):
    count=0
    target_reached=False
    while target_reached==False:
        print("epoch: ", epoch, ". step:", count)
###process and take picture
        indices = process_picture()
###binary_network(sliced)=indices as input for policy model
        optimizer.zero_grad()
###output: 1 for curvature, 1 for duration of movement
        motor_controls = policy_model(Variable(torch.from_numpy(indices))).detach().numpy()
        print("NO TANH output for motor: 1)activation left, 2)time ", motor_controls)
        motor_controls[0] = np.tanh(motor_controls[0])
        motor_controls[1] = np.tanh(motor_controls[1])
        print("TANH output for motor: 1)activation left, 2)time ", motor_controls)
###execute suggested action
        move(motor_controls)
###take and process picture2 (after movement)
        indices = (process_picture())
###loss=(binary_network(picture2) - desired
        print("calculate loss")
        print("idx", indices, type(torch.tensor(indices)))
     #   loss = 0
      #  loss = (indices[0]-10)**2+(indices[1]-31)**2
       # loss = loss/2
        print("shape of indices", indices.shape)
        array=np.zeros((1,2))
        array[0]=indices
        print(array.shape, type(array))
        array2 = torch.ones([1,2])
        loss = loss_function(torch.tensor(array).double(), torch.tensor([[10.0,31.0]]).double()).float()
        print("loss: ", loss, type(loss), loss.shape)
       # array2[0] = loss_function(torch.tensor(array).double(),
        torch.tensor([[10.0,31.0]]).double()).float()
        losses.append(loss)
#start line causing the error-message (still part of main)
###calculate gradients
        loss.backward()
#end line causing the error-message (still part of main)

###apply gradients
        optimizer.step()

#Output (so far as intented) (not all included)

#calculate loss
idx [14. 15.] <class 'torch.Tensor'>
shape of indices (2,)
(1, 2) <class 'numpy.ndarray'>
loss:  tensor(136.) <class 'torch.Tensor'> torch.Size([])

#Error Message:
Traceback (most recent call last):
  File "/home/pi/Desktop/GradientPolicyLearning/PolicyModel.py", line 259, in <module>
    array2.backward()
  File "/home/pi/.local/lib/python3.7/site-packages/torch/tensor.py", line 134, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pi/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in
 backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Output:
If you call .detach() on  the prediction, that will delete the gradients. Since you are first getting indices from the model and then trying to backprop the error, I would suggest

prediction = policy_model(torch.from_numpy(indices))
motor_controls = prediction.clone().detach().numpy()

This would keep the predictions as it is with the calculated gradients that can be backproped.
Now you can do

loss = loss_function(prediction, torch.tensor([[10.0,31.0]]).double()).float()

Note, you might wanna call double of the prediction if it throws an error.


Instruction: How to extract position input-output indeces from huggingface transformer text tokenizator?

Input:
I want to solve stress prediction task with pretrained russian bert.

Input data looks like this:

  граммов сверху|000100000001000

Zeros mean no stress. Ones represent stress position character.

I want to map it as word -> vowel number index

So it will be like
граммов -> 1
сверху -> 1

So, for each token, it should be a linear layer with softmax.

I understand this part, but it's hard for me to deal with text preprocessing because text tokenizator can split a word into subword tokens.

Tokenizator takes an input string and returns tokens like this

bert_tokenizer.encode('граммов сверху')
->
[101, 44505, 26656, 102]

So, how to get position mapping between input chars and words?

The desired output should be like [[0, 7], [8, 14]]

I tried to read docs
https://huggingface.co/transformers/main_classes/tokenizer.html

And found convert_ids_to_tokens function
It works like

encoded = bert_tokenizer.encode('граммов сверху')
bert_tokenizer.convert_ids_to_tokens(encoded)
->
['[CLS]', 'граммов', 'сверху', '[SEP]']

But I'm not sure how to use original string and stress indices to calculate stress position number for token.

Output: Its turned out, tokenizer have return_offsets_mapping param, this solve my problem.


Instruction: How to use map_location='cpu' due to "RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False"

Input:
I was trying to download the following model at https://pytorch.org/hub/nvidia_deeplearningexamples_tacotron2/

import torch
tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')

I received:

>>> import torch
>>> tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')
Using cache found in .cache\torch\hub\nvidia_DeepLearningExamples_torchhub
...
  File "Anaconda3\envs\env3_pytorch\lib\site-packages\torch\serialization.py", line 79, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

So I used the following with map_location='cpu', but still get the same error.

>>> tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2', map_location='cpu')

torch.version.cuda shows a version but torch.cuda.is_available() is false.

>>> import torch
>>> torch.version.cuda
'9.0'
>>> torch.cuda.is_available()
False

How to get around this error related to map_location as I used exactly the command error message asked to use?
How can I use cuda in loading the model?

Output:
torch.hub.load does not specifically support map_location, it only forwards the extra arguments to the loading of the model, so it's implementation dependent whether that would be support.

In this case it is not supported, the loading is implemented in NVIDIA/DeepLearningExamples:torchhub - hubconf.py and it does not pass any map_location to torch.load when the checkpoint is loaded.

That means you need to load the checkpoint manually and apply it to the model. Thankfully, the model can be loaded with the same configuration without loading the checkpoint, by setting pretrained=False and the checkpoint can be loaded separately with torch.hub.load_state_dict_from_url, which supports map_location. There is only a small change that is required to the state dict, because it was trained with DistributedDataParallel, which wraps the module, such that every layer in the model became module.layer. In the state dict that module. prefix needs to be stripped.

tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2', pretrained=False)

checkpoint = torch.hub.load_state_dict_from_url('https://api.ngc.nvidia.com/v2/models/nvidia/tacotron2pyt_fp32/versions/1/files/nvidia_tacotron2pyt_fp32_20190306.pth', map_location="cpu")

# Unwrap the DistributedDataParallel module
# module.layer -> layer
state_dict = {key.replace("module.", ""): value for key, value in checkpoint["state_dict"].items()}

# Apply the state dict to the model
tacotron2.load_state_dict(state_dict)


Instruction: No matching distribution found for torch==1.5.0+cpu on Heroku

Input:
I am trying to deploy my Django app which uses a machine learning model. And the machine learning model requires pytorch to execute.
When i am trying to deploy it is giving me this error

ERROR: Could not find a version that satisfies the requirement torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0)
       ERROR: No matching distribution found for torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23))
 !     Push rejected, failed to compile Python app.
 !     Push failed

My requirements.txt is

asgiref==3.2.7
certifi==2020.4.5.1
chardet==3.0.4
cycler==0.10.0
dj-database-url==0.5.0
Django==3.0.6
django-heroku==0.3.1
future==0.18.2
gunicorn==20.0.4
idna==2.9
imageio==2.8.0
kiwisolver==1.2.0
matplotlib==3.2.1
numpy==1.18.4
Pillow==7.1.2
psycopg2==2.8.5
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
six==1.14.0
sqlparse==0.3.1
torch==1.5.0+cpu
torchvision==0.6.0+cpu
urllib3==1.25.9
whitenoise==5.0.1

And runtime.txt is python-3.7.5
However installing it on my computer is not giving any type of error when i use command pip install torch==1.5.0+cpu I am using python 3.7.5 and pip 20.0.2.
Complete code is here.
How to solve this issue i really need to deploy my app. Thanks

Output:
PyTorch does not distribute the CPU only versions over PyPI. They are only available through their custom registry.

If you select the CPU only version on PyTorch - Get Started Locally you get the following instructions:

pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Since you're not manually executing the pip install, you cannot simply add the -f https://download.pytorch.org/whl/torch_stable.html.

As an alternative, you can put it into your requirements.txt as a standalone line. It shouldn't really matter where exactly you put it, but it is commonly put at the very top.

-f https://download.pytorch.org/whl/torch_stable.html
asgiref==3.2.7
certifi==2020.4.5.1
chardet==3.0.4
cycler==0.10.0
dj-database-url==0.5.0
Django==3.0.6
django-heroku==0.3.1
future==0.18.2
gunicorn==20.0.4
idna==2.9
imageio==2.8.0
kiwisolver==1.2.0
matplotlib==3.2.1
numpy==1.18.4
Pillow==7.1.2
psycopg2==2.8.5
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
six==1.14.0
sqlparse==0.3.1
torch==1.5.0+cpu
torchvision==0.6.0+cpu
urllib3==1.25.9
whitenoise==5.0.1


Instruction: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor

Input:
My Model:

class myNet(nn.Module):
    def __init__(self):
        super(myNet,self).__init__()
        self.act1=Dynamic_relu_b(64)
        self.conv1=nn.Conv2d(3,64,3)
        self.pool=nn.AdaptiveAvgPool2d(1)
        self.fc=nn.Linear(128,20)
    def forward(self,x):
        x=self.conv1(x)
        x=self.act1(x)
        x=self.pool(x)
        x=x.view(x.shape[0],-1)
        x=self.fc(x)

        return x

A code that replicates the experiment is provided：

def one_hot_smooth_label(x,num_class,smooth=0.1):
    num=x.shape[0]
    labels=torch.zeros((num,20))
    for i in range(num):
        labels[i][x[i]]=1
    labels=(1-(num_class-1)/num_class*smooth)*labels+smooth/num_class
    return labels
images=torch.rand((4,3,300,300))
images=images.cuda()
labels=torch.from_numpy(np.array([1,0,0,1]))
model=myNet()
model=model.cuda()

output=model(images)
labels=one_hot_smooth_label(labels,20)
labels = labels.cuda()
criterion=nn.BCEWithLogitsLoss()

loss=criterion(output,labels)
loss.backward()

The error:

    RuntimeError                              Traceback (most recent call last)
<ipython-input-42-1268777e87e6> in <module>()
     21
     22 loss=criterion(output,labels)
---> 23 loss.backward()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101
    102

RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false) but got TensorOptions(dtype=float, device=cuda:0, layout=Strided, requires_grad=false) (validate_outputs at /pytorch/torch/csrc/autograd/engine.cpp:484)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7fcf7711b536 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x2d84224 (0x7fcfb1bad224 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x548 (0x7fcfb1baed58 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7fcfb1bb0ce2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::autograd::Engine::thread_init(int) + 0x39 (0x7fcfb1ba9359 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7fcfbe2e8378 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0xbd6df (0x7fcfe23416df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x76db (0x7fcfe34236db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #8: clone + 0x3f (0x7fcfe375c88f in /lib/x86_64-linux-gnu/libc.so.6)

After many experiments, I found that act1 in the model was the problem. If you delete act1, the error will not appear！

But I don't know why act1 has this problem.

What seems to be the wrong part of the error is requiers_grad=False, and I don't know which part set this.

This is the code about act1(Dynamic_relu_b):

class Residual(nn.Module):
    def __init__(self, in_channel, R=8, k=2):
        super(Residual, self).__init__()
        self.avg = nn.AdaptiveAvgPool2d((1, 1))
        self.relu = nn.ReLU(inplace=True)
        self.R = R
        self.k = k
        out_channel = int(in_channel / R)
        self.fc1 = nn.Linear(in_channel, out_channel)
        fc_list = []
        for i in range(k):
            fc_list.append(nn.Linear(out_channel, 2 * in_channel))
        self.fc2 = nn.ModuleList(fc_list)

    def forward(self, x):
        x = self.avg(x)
        x = torch.squeeze(x)
        x = self.fc1(x)
        x = self.relu(x)
        result_list = []
        for i in range(self.k):
            result = self.fc2[i](x)
            result = 2 * torch.sigmoid(result) - 1
            result_list.append(result)
        return result_list

class Dynamic_relu_b(nn.Module):
    def __init__(self, inchannel, R=8, k=2):
        super(Dynamic_relu_b, self).__init__()
        self.lambda_alpha = 1
        self.lambda_beta = 0.5
        self.R = R
        self.k = k
        self.init_alpha = torch.zeros(self.k)
        self.init_beta = torch.zeros(self.k)
        self.init_alpha[0] = 1
        self.init_beta[0] = 1
        for i in range(1, k):
            self.init_alpha[i] = 0
            self.init_beta[i] = 0

        self.residual = Residual(inchannel)

    def forward(self, input):
        delta = self.residual(input)
        in_channel = input.shape[1]
        bs = input.shape[0]
        alpha = torch.zeros((self.k, bs, in_channel))
        beta = torch.zeros((self.k, bs, in_channel))
        for i in range(self.k):
            for j, c in enumerate(range(0, in_channel * 2, 2)):
                alpha[i, :, j] = delta[i][:, c]
                beta[i, :, j] = delta[i][:, c + 1]
        alpha1 = alpha[0]
        beta1 = beta[0]
        max_result = self.dynamic_function(alpha1, beta1, input, 0)
        for i in range(1, self.k):
            alphai = alpha[i]
            betai = beta[i]
            result = self.dynamic_function(alphai, betai, input, i)
            max_result = torch.max(max_result, result)
        return max_result

    def dynamic_function(self, alpha, beta, x, k):
        init_alpha = self.init_alpha[k]
        init_beta = self.init_beta[k]
        alpha = init_alpha + self.lambda_alpha * alpha
        beta = init_beta + self.lambda_beta * beta
        bs = x.shape[0]
        channel = x.shape[1]
        results = torch.zeros_like(x)
        for i in range(bs):
            for c in range(channel):
                results[i, c, :, :] = x[i, c] * alpha[i, c] + beta[i, c]
        return results

How should I solve this problem?

Output:
In PyTorch two tensors need to be on the same device to perform any mathematical operation between them. But in your case one is on the CPU and the other on the GPU. The error is not as clear as it normally is, because it happened in the backwards pass. You were (un)lucky that your forward pass did not fail. That's because there is an exception to the same device restriction, namely when using scalar values in the mathematical operation, e.g. tensor * 2, and it even occurs when the scalar is a tensor: cpu_tensor * tensor(2, device='cuda:0'). You are using a lot of loops and accessing individual scalars to calculate further results.

While the forward pass works like that, in the backward pass when the gradients are calculated, the gradients are multiplied with the previous ones (application of the chain rule). At that point, the two are on different devices.

You have identified that it's in the Dynamic_relu_b. In there you need to make sure that every tensor that you create, is on the same device as the input. The two tensors you create in the forward method are:

alpha = torch.zeros((self.k, bs, in_channel))
beta = torch.zeros((self.k, bs, in_channel))

These are created on the CPU, but your input is on the GPU, so you need to put them on the GPU as well. To be generic, it should be put onto the device where the input is located.

alpha = torch.zeros((self.k, bs, in_channel), device=input.device)
beta = torch.zeros((self.k, bs, in_channel), device=input.device)

The biggest problem in your code are the loops. Not only did they obfuscated a bug, they are very harmful for performance, since they can neither be parallelised nor vectorised, and those are the reasons why GPUs are so fast. I'm certain that these loops can be replaced with more efficient operations, but you'll have to get out of the mindset of creating an empty tensor and then filling it one by one.

I'll give you one example from dynamic_function:

results = torch.zeros_like(x)
for i in range(bs):
    for c in range(channel):
        results[i, c, :, :] = x[i, c] * alpha[i, c] + beta[i, c]

You're multiplying x (size: [bs, channel, height, width]) with alpha (size: [bs, channel]), where every plane (height, width) of x is multiplied by a different element of alpha (scalar). That would be same as doing an element-wise multiplication with a tensor of the same size as the plane [height, width], but where all elements are the same scalar.

Thankfully, you don't need to repeat them yourself, since singular dimensions (dimensions with size 1) are automatically expanded to match the size of the other tensor, see PyTorch - Broadcasting Semantics for details. That means you only need to reshape alpha to have size [bs, channel, 1, 1].

The loop can therefore be replaced with:

results = x * alpha.view(bs, channel, 1, 1) + beta.view(bs, channel, 1, 1)

By eliminating that loop, you gain a lot of performance, and your initial error just got much clearer, because the forward pass would fail with the following message:

  File "main.py", line 78, in dynamic_function
    results = x * alpha.view(bs, channel, 1, 1) + beta.view(bs, channel, 1, 1)
RuntimeError: expected device cuda:0 but got device cpu

Now you would know that one of these is on the CPU and the other on the GPU.


Instruction: Why some people chain the parameters of two different networks and train them with same optimizer?

Input:
I was looking at CycleGAN's official pytorch implementation and there, author chained the parameters of both networks and used a single optimizer for both network. How does this work? Is it better than using two different optimizers for two different networks ?

all_params = chain(module_a.parameters(), module_b.parameters())
optimizer = torch.optim.Adam(all_params)

Output:
From chain documentation: https://docs.python.org/3/library/itertools.html#itertools.chain

itertools.chain(*iterables)

    Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted.

As parameters() gives you an iterable, you can use the optimizer to simultaneously optimize parameters for both of the networks. So, same optimizer states will be used for both models (Modules), if you use two different optimizers, the parameters will be optimized separately.

If you have a composite network, it becomes necessary to optimize the parameters (of all) at the same time, hence using a single optimizer for all of them is the way to go.


Instruction: NumPyro vs Pyro: Why is former 100x faster and when should I use the latter?

Input:
From Pytorch-Pyro's website:

  We’re excited to announce the release of NumPyro, a NumPy-backed Pyro using JAX for automatic differentiation and JIT compilation, with over 100x speedup for HMC and NUTS!

My questions:

Where is the performance gain (which is sometimes 340x or 2X) of NumPyro (over Pyro) coming from exactly?
And more importantly, why (rather, where) would I continue to use Pyro?

Extra:

How should I view the performance and features of NumPyro compared to Tensorflow Probability, in deciding which to use where?

Output: That's a good question. I just asked the same question in Pyro's dedicated forum. Here's the answer of one of their core developers: "There are many cool stuffs in Pyro that do not appear in NumPyro, for example, see Contributed code section in Pyro docs. For me, while developing, it is much easier to debug PyTorch code than Jax code (though Jax team has put much effort to help debugging in recent releases). Hence to implement a new inference algorithm, it is easier for me to work in Pyro."