Instruction: TensorFlow vs PyTorch convolution confusion

Input:
I am confused on how to replicate Keras (TensorFlow) convolutions in PyTorch.
In Keras, I can do something like this. (the input size is (256, 237, 1, 21) and the output size is (256, 237, 1, 1024).
import tensorflow as tf
x = tf.random.normal((256,237,1,21))
y = tf.keras.layers.Conv1D(filters=1024, kernel_size=5,padding="same")(x)
print(y.shape)
(256, 237, 1, 1024)

However, in PyTorch, when I try to do the same thing I get a different output size:
import torch.nn as nn
x = torch.randn(256,237,1,21)
m = nn.Conv1d(in_channels=237, out_channels=1024, kernel_size=(1,5))
y = m(x)
print(y.shape)
torch.Size([256, 1024, 1, 17])

I want PyTorch to give me the same output size that Keras does:
This previous question seems to imply that Keras filters are PyTorch's out_channels but thats what I have. I tried to add the padding in PyTorch of padding=(0,503) but that gives me torch.Size([256, 1024, 1, 1023]) but that still not correct. This also takes so much longer than keras does so I feel that I have incorrectly assigned a parameter.
How can I replicate what Keras did with convolution in PyTorch?

Output:
In TensorFlow, tf.keras.layers.Conv1D takes in a tensor of shape (batch_shape + (steps, input_dim)). Which means that what is commonly known as channels appears on the last axis. For instance in 2D convolution you would have (batch, height, width, channels). This is different from PyTorch where the channel dimension is right after the batch axis: torch.nn.Conv1d takes in shapes of (batch, channel, length). So you will need to permute two axes.
For torch.nn.Conv1d:

in_channels is the number of channels in the input tensor
out_channels is the number of filters, i.e. the number of channels the output will have
stride the step size of the convolution
padding the zero-padding added to both sides

In PyTorch there is no option for padding='same', you will need to choose padding correctly. Here stride=1, so padding must equal to kernel_size//2 (i.e. padding=2) in order to maintain the length of the tensor.

In your example, since x has a shape of (256, 237, 1, 21), in TensorFlow's terminology it will be considered as an input with:

a batch shape of (256, 237),
steps=1, so the length of your 1D input is 1,
21 input channels.

Whereas in PyTorch, x of shape (256, 237, 1, 21) would be:

batch shape of (256, 237),
1 input channel
a length of 21.

Have kept the input in both examples below (TensorFlow vs. PyTorch) as x.shape=(256, 237, 21) assuming 256 is the batch size, 237 is the length of the input sequence, and 21 is the number of channels (i.e. the input dimension, what I see as the dimension on each timestep).
In TensorFlow:
>>> x = tf.random.normal((256, 237, 21))
>>> m = tf.keras.layers.Conv1D(filters=1024, kernel_size=5, padding="same")
>>> y = m(x)
>>> y.shape
TensorShape([256, 237, 1024])

In PyTorch:
>>> x = torch.randn(256, 237, 21)
>>> m = nn.Conv1d(in_channels=21, out_channels=1024, kernel_size=5, padding=2)
>>> y = m(x.permute(0, 2, 1))
>>> y.permute(0, 2, 1).shape
torch.Size([256, 237, 1024])

So in the latter, you would simply work with x = torch.randn(256, 21, 237)...


Instruction: What is a tensor argument to Normal supposed to mean in Distributions Package of Pytorch?

Input:
I understand torch.Normal(loc, scale) is a class corresponding to univariate normal distribution in pytorch.
I understand how it works when loc and scale are numbers.
The problem is when the inputs to torch.Normal are tensors as opposed to numbers. In that case I do not understand it well. What is the exact interpretation/usage of such tensor arguments?
See for example y_dist in code below. loc and scale are tensors for y_dist. What does this exactly mean? I do not think this converts the univariate distribution to multivariate, does it? Does it instead form a group of univariate distributions?
import torch as pt
ptd = pt.distributions
x_dist = ptd.Normal(loc = 2, scale = 3)
x_samples = x_dist.sample()

batch_size = 256
y_dist = ptd.Normal(loc = 0.25 * pt.ones(batch_size, dtype=pt.float32), scale = pt.ones(batch_size, dtype=pt.float32))

Output:
As you said, if loc (a.k.a. mu) and scale (a.k.a. sigma) are floats then it will sample from a normal distribution, with loc as the mean, and scale as the standard deviation.
Providing tensors instead of floats will just make it sample from more than one normal distribution independently (unlike torch.distributions.MultivariateNormal of course)
If you look at the source code you will see loc and scale are broadcasted to the same shape on __init__.

Here's an example to show this behavior:
>>> mu = torch.tensor([-10, 10], dtype=torch.float)
>>> sigma = torch.ones(2, 2)
>>> y_dist = Normal(loc=mu, scale=sigma)

Above mu is 1D, while sigma is 2D, yet:
>>> y_dist.loc
tensor([[-10.,  10.],
        [-10.,  10.]])

So it will get two samples from N(-10, 1) and two samples from N(10, 1)
>>> y_dist.sample()
tensor([[ -9.1686,  10.6062],
        [-10.0974,   8.5439]])

Similarly:
>>> mu = torch.zeros(2, 2)
>>> sigma = torch.tensor([0.001, 1000], dtype=torch.float)
>>> y_dist = Normal(loc=mu, scale=sigma)

Will broadcast scale to be:
>>> y_dist.scale
tensor([[1.0000e-03, 1.0000e+01],
        [1.0000e-03, 1.0000e+01]])

>>> y_dist.sample()
tensor([[-8.0329e-04,  1.4213e+01],
        [-1.4907e-03,  3.1190e+02]])


Instruction: ImportError: libc10.so: cannot open shared object file: No such file or directory

Input:
While running smdataparallel, I see following error
# python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import smdistributed.dataparallel.torch.distributed as dist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/smdistributed/dataparallel/__init__.py", line 16, in <module>
    import smddpcommon as hc
ImportError: libc10.so: cannot open shared object file: No such file or directory

Output:
libc10.so is made available by pytorch. Hence first
import torch

and then import packages that depend on pytorch.


Instruction: How to make tensordot operations after permutation

Input:
I have 2 tensors, A and B:
A = torch.randn([32,128,64,12],dtype=torch.float64)
B = torch.randn([64,12,64,12],dtype=torch.float64)
C = torch.tensordot(A,B,([2,3],[0,1]))
D = C.permute(0,2,1,3) # shape:[32,64,128,12]

tensor D comes from the operations "tensordot -> permute". How can I implement a new operation f() to make the tensordot operation after f() like:
A_2 = f(A)
B_2 = f(B)
D = torch.tensordot(A_2,B_2)

Output:
Have you considered using torch.einsum which is very flexible?
D = torch.einsum('ijab,abkl->ikjl', A, B)

The problem with tensordot is that it outputs all dimensions of A before those of B and what you are looking for (when permuting) is to "interleave" dimensions from A and B.


Instruction: Multi label classification with unbalanced labels

Input:
I am building multi label classification network.
My GTs are vectors of length 512 [0,0,0,1,0,1,0,...,0,0,0,1]
Most of the time they are zeroes, each vector has about 5 ones, and rest are zeros .
I am thinking to do:
Use sigmoid for activation for output layer.
Use binary_crossentropy for loss function.
But how I can solve the unbalance issue ?
Network can learn to predict always zeros and still have really low learning loss score.
How I can make it actually learn to predict ones...

Output:
You cannot easily upsample as this is a multilabel case (what I've missed from the post originally).
What you can do is give 1 way higher weights, something like this:
import torch

class BCEWithLogitsLossWeighted(torch.nn.Module):
    def __init__(self, weight, *args, **kwargs):
        super().__init__()
        # Notice none reduction
        self.bce = torch.nn.BCEWithLogitsLoss(*args, **kwargs, reduction="none")
        self.weight = weight

    def forward(self, logits, labels):
        loss = self.bce(logits, labels)
        binary_labels = labels.bool()
        loss[binary_labels] *= labels[binary_labels] * self.weight
        # Or any other reduction
        return torch.mean(loss)

loss = BCEWithLogitsLossWeighted(50)
logits = torch.randn(64, 512)
labels = torch.randint(0, 2, size=(64, 512)).float()

print(loss(logits, labels))

Also you can use FocalLoss to focus on positive examples (there should be some implementations available in some libraries).
EDIT:
Focal Loss can be coded something along those lines also (functional form cause that's what I have in repo, but you should be able to work from that):
def binary_focal_loss(
    outputs: torch.Tensor,
    targets: torch.Tensor,
    gamma: float,
    weight=None,
    pos_weight=None,
    reduction: typing.Callable[[torch.Tensor], torch.Tensor] = None,
) -> torch.Tensor:

    probabilities = (1 - torch.sigmoid(outputs)) ** gamma
    loss = probabilities * torch.nn.functional.binary_cross_entropy_with_logits(
        outputs,
        targets.float(),
        weight,
        reduction="none",
        pos_weight=pos_weight,
    )

    return reduction(loss)


Instruction: How can I swap values in PyTorch Tensor?

Input:
I have the following tensor:
vector = torch.tensor([[1,5,3], [2,3,4]])

How can I swap values in the second axis?
e.g.
tensor([[1, 5, 3], [2, 3, 4]])
becomes:
tensor([[1, 3, 3], [2, 5, 4]])

Output:
You can use numpy's style of indexing :
>>> vector = torch.tensor([[1,5,3], [2,3,4]])
tensor([[1, 3, 4],
        [2, 5, 3]])
>>> vector[[0,1],1] = vector[[1,0],1]
>>> vector
tensor([[1, 3, 3],
        [2, 5, 4]])

In that case, we switch the value between the 0th and 1st index in the first dimension, only at the index 1 on the second dimension.


Instruction: Using a subset of classes in ImageNet

Input:
I'm aware that subsets of ImageNet exist, however they don't fulfill my requirement. I want 50 classes at their native ImageNet resolutions.
To this end, I used torch.utils.data.dataset.Subset to select specific classes from ImageNet. However, it turns out, class labels/indices must be greater than 0 and less than num_classes.
Since ImageNet contains 1000 classes, the idx of my selected classes quickly goes over 50. How can I reassign the class indices and do so in a way that allows for evaluation later down the road as well?
Is there a way more elegant way to select a subset?

Output:
I am not sure I understood your conclusions about labels being greater than zero and less than num_classes. The torch.utils.data.Subset helper takes in a torch.utils.data.Dataset and a sequence of indices, they correspond to indices of data points from the Dataset you would like to keep in the subset. These indices have nothing to do with the classes they belong to.
Here's how I would approach this:

Load your dataset through torchvision.datasets (custom datasets would work the same way). Here I will demonstrate it with FashionMNIST since ImageNet's data is not made available directly through torchvision's API.
>>> ds = torchvision.datasets.FashionMNIST('.')
>>> len(ds)
60000

Define the classes you want to select for the subset dataset. And retrieve all indices from the main dataset which correspond to these classes:
>>> targets = [1, 3, 5, 9]
>>> indices = [i for i, label in enumerate(ds.targets) if label in targets]

You have your subset:
>>> ds_subset = Subset(ds, indices)
>>> len(ds_subset)
24000

At this point, you can use a dictionnary to remap your labels using targets:
>>> remap = {i:x for i, x in enumerate(targets)}
{0: 1, 1: 3, 2: 5, 3: 9}

For example:
>>> x, y = ds_subset[10]
>>> y, remap[y] # old_label, new_label
1, 3


Instruction: Why is data augmentation degrading performance for Mask-RCNN?

Input:
I trained a Mask-RCNN for instance segmentation with and without data augmentation. The augmentation was simply a rotation which makes sense for the data concerned. I was very surprised that the augmentation run (dark blue) was worse that the non-augmentation run (light blue).
Since the augmentation plots see to just be shifted down but have the same shape I was wondering if there is something else at play. I am using a batch size of 2 and the dataset have 40K images, could that affect things?

Output:
Not quite an answer.
I had similar effects with it and I think all the parameters and how you train it is important. For example, with more layers (resnet34 vs. resnet18 for the backbone) you need more information to train the bigger network. In this case, augmentations are useful.
Another example is network resolution. I trained it with the default one min_size=800 and max_size=1333 on some learning rate and with the higher resolution, you have a higher potential for the aggressive growth of the network AP on a higher LR. Yet another example related to this is how many "levels" you have in your FPN and what is the grid settings for AnchorGenerator. If your augmentations generate samples smaller than the anchors on a particular level of FPN then they probably will cause more issues than do any good. And if your augmentations generate samples such a small that the details of your object are not visible - again, not very useful, especially on small networks.
There are tons of similar small issues that matter. I had a situation, that rotations made the result worse because, with some rotation angle, the rotated sample started to look like a part of the background and the detector based on maskrcnn failed to work with it. Cubic interpolation fixed it a little bit but eventually, I came up with the idea to limit the angle of the rotation.
Just experiment and find hyperparameters that play well for your particular task.


Instruction: How to Increase the Scope of Images a Neural Network Can Recognize?

Input:
I am working on an image recognition neural network with Pytorch. My goal is to take pictures of handwritten math equations, process them, and use the neural network to recognize each element. I've reached the point where I am able to separate every variable, number, or symbol from the equation, and everything is ready to be sent through the neural network. I've trained my network to recognize numbers quite well (this part was quite easy), but now I want to expand the scope of the neural network to recognizing letters as well as numbers. I loaded handwritten letters along with the numbers into tensors, shuffled the elements, and put them into batches. No matter how I vary my learning rate, my architecture (hidden layers and the number of neurons per layer), or my batch size I cannot get the neural network to recognize letters.
Here is my network architecture and the feed-forward function (you can see I experimented with the number of hidden layers):
class NeuralNetwork(nn.Module):

def __init__(self):

    super().__init__()
    inputNeurons, hiddenNeurons, outputNeurons = 784, 700, 36

    # Create tensors for the weights
    self.layerOne = nn.Linear(inputNeurons, hiddenNeurons)
    self.layerTwo = nn.Linear(hiddenNeurons, hiddenNeurons)
    self.layerThree = nn.Linear(hiddenNeurons, outputNeurons)
    #self.layerFour = nn.Linear(hiddenNeurons, outputNeurons)
    #self.layerFive = nn.Linear(hiddenNeurons, outputNeurons)

# Create function for Forward propagation
def Forward(self, input):

    # Begin Forward propagation
    input = torch.sigmoid(self.layerOne(torch.sigmoid(input)))
    input = torch.sigmoid(self.layerTwo(input))
    input = torch.sigmoid(self.layerThree(input))
    #input = torch.sigmoid(self.layerFour(input))
    #input = torch.sigmoid(self.layerFive(input))

    return input

And this is the training code block (the data is shuffled in a dataloader, the ground truths are shuffled in the same order, batch size is 10, total number letter and number data points is 244800):
neuralNet = NeuralNetwork()
params = list(neuralNet.parameters())
criterion = nn.MSELoss()
print(neuralNet)

dataSet = next(iter(imageDataLoader))
groundTruth = next(iter(groundTruthsDataLoader))

for i in range(15):

    for k in range(24480):

        neuralNet.zero_grad()

        prediction = neuralNet.Forward(dataSet)
        loss = criterion(prediction, groundTruth)
        loss.backward()

        for layer in range(len(params)):

            # Updating the weights of the neural network
            params[layer].data.sub_(params[layer].grad.data * learningRate)

Thanks for the help in advance!

Output:
First thing i would recommend is writing a clean Pytorch code
For eg.
if i see your NeuralNetwork class it should have forward method (f in lower case),
so that you wont call it using prediction = neuralNet.Forward(dataSet).  Reason being your hooks from neural network does not get dispatched if you use prediction = neuralNet.Forward(dataSet).
For more details refer this link
Second thing is : Since your dataset is not balance.....try to use undersampling / oversampling methods, which will be very helpful in your case.


Instruction: Python Error: the following arguments are required: experimentname

Input:
I am a beginner in Python and PyTorch environment. I am executing the program on GitHub.
I am getting the following error:
usage: main.py [-h] [--self_host SELF_HOST] [--cpu] [--port PORT]
               [--server_port SERVER_PORT] [--config CONFIG]
               [--devices_per_trial DEVICES_PER_TRIAL] [--dataroot DATAROOT]
               [--s3 S3] [--logroot LOGROOT] [--seed SEED]
               [--num_eigenthings NUM_EIGENTHINGS] [--batch_size BATCH_SIZE]
               [--eval_batch_size EVAL_BATCH_SIZE] [--momentum MOMENTUM]
               [--num_steps NUM_STEPS] [--max_samples MAX_SAMPLES] [--cuda]
               [--full_dataset] [--fname FNAME] [--mode {power_iter,lanczos}]
               experimentname
main.py: error: the following arguments are required: experimentname
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2

Upon %tb, I get following details:
Traceback (most recent call last):

  File "/home/ajinkya/Hessian_Expts/main.py", line 77, in <module>
    skeletor.execute(main)

  File "/home/ajinkya/miniconda3/lib/python3.8/site-packages/skeletor/launcher.py", line 233, in execute
    args = _parser.val.parse_args()

  File "/home/ajinkya/miniconda3/lib/python3.8/argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)

  File "/home/ajinkya/miniconda3/lib/python3.8/argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)

  File "/home/ajinkya/miniconda3/lib/python3.8/argparse.py", line 2034, in _parse_known_args
    self.error(_('the following arguments are required: %s') %

  File "/home/ajinkya/miniconda3/lib/python3.8/argparse.py", line 2521, in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)

  File "/home/ajinkya/miniconda3/lib/python3.8/argparse.py", line 2508, in exit
    _sys.exit(status)

SystemExit: 2

Kindly help to get rid of this error. Thanks in advance.

Output:
Your CLI seems different to the one exposed from pytorch-hessian-eigenthings/main.py. Anyhow, all options prefixed with -- are options (like --self_host, --cpu, ...). The only (required) positional argument is experimentname, so you need to provide it when calling main.py!
An extremely minimal call would be:
python main.py my_experiment


Instruction: Running into IndexError when following PyTorch CrossEntropyLoss example for classification

Input:
I am trying to follow exemplary usage of PyTorch's CrossEntropyLoss function for a classification problem using the fashionMNIST dataset.
I think the issue has to do with the dimension of the data output from my model. I'm not sure though, which makes figuring this out more difficult. Only running on Google Colab right now so I don't have easy access to a debugger (as far as I know).
Here's my code, please let me know what I am doing wrong!
import torch.nn as nn
import torch.optim as optim

model = nn.Sequential(
    nn.Linear(784, 1024),
    nn.ReLU(),
    nn.Linear(1024, 10),
    nn.ReLU(),
)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.0)
loss_fn = nn.CrossEntropyLoss()

# Model Training
for epoch in range(1):
  for data, label in trainset:
    optimizer.zero_grad()
    output = model(torch.flatten(data))
    loss = loss_fn(output, label)
    loss.backward()
    optimizer.step()

And then the error:
IndexError                                Traceback (most recent call last)
<ipython-input-3-6d9a9ea2e921> in <module>()
     16     optimizer.zero_grad()
     17     output = model(torch.flatten(data))
---> 18     loss = loss_fn(output, label)
     19     loss.backward()
     20     optimizer.step()

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in log_softmax(input, dim, _stacklevel, dtype)
   1603         dim = _get_softmax_dim('log_softmax', input.dim(), _stacklevel)
   1604     if dtype is None:
-> 1605         ret = input.log_softmax(dim)
   1606     else:
   1607         ret = input.log_softmax(dim, dtype=dtype)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Output:
It looks like you haven't converted your dataset into DataLoader object, mostly that's the reason you're facing this issue.
However, coming back to your problem. the loss function expecting the shape of the output variable from the model to be a 2-D tensor, something like [1, 10] where 1 is the number of batches (which should be 1 as you've used only 1 data at a time and the 10 is the number of logits/classes, so 10 predictions). So, in short, to solve your issue, change the line of loss calculation with the following line-
loss = loss_fn(output.unsqueeze(0), torch.Tensor([label]).type(torch.LongTensor))

Here, .unsqueeze() increases the dimension of a tensor, and the label is an int, hence we've turned it into a 1-D tensor before sending it into the loss function.
Note: As you haven't mentioned much in your question about what you're trying to do exactly, what type of data you're working with, so it's not obvious but I don't know why you're using ReLU() as your output layer's activation (It's something we almost never do unless you've some special reason) however seeing that you're using CrossEntropyLoss() loss, which indicates you're maybe trying to do multi-class classification, in that case, I would suggest you remove the ReLU() after the last layer, then your model will able to learn different classes using softmax() activation (which is inbuilt within CrossEntropyLoss() loss) so you'll get what you want from the model.


Instruction: Don't include an operation for gradient computation in PyTorch

Input:
I have a custom layer. Let the layer be called 'Gaussian'
class Gaussian(nn.Module):
  def __init__():
    super(Gaussian, self).__init__()

 #@torch.no_grad
  def forward(self, x):
    _r = np.random.randint(0, x.shape[0], x.shape[0])
    _sample = x[_r]
    _d = (_sample - x)
    _number = int(self.k * x.shape[0])
    x[1: _number] = x[1: _number] + (self.n * _d[1: _number]).detach()

    return x

The above class will be used as below:
cnn_model = nn.Sequential(nn.Conv2d(1, 32, 5), Gaussian(), nn.ReLU(), nn.Conv2d(32, 32, 5))

If x is the input, I want the gradient of x to exclude operations that are present in the Gaussian module, but include the calculations in other layers of the neural network(nn.Conv2d etc).
In the end, my aim is to use the Gaussian module to perform calculations but that calculations should not be included in gradient computation.
I tried to do the following:

Used the @torch.no_grad above the forward method of the Gaussian

Using detach after every operation in the Gaussian module:
x[1: _number] = x[1: _number] + (self.n * _d[1: _number]).detach() and similarly for other operations

Use y = x.detach() in the forward method. Perform the operations on y and then x.data = y

Are the above methods correct?
P.S: Question edited

Output:
The gradient calculation has sense when there are parameters to optimise.
If your module do not have any parameters, then no gradient will be stored, because there are no parameters to associate it.


Instruction: Pytorch - Caught StopIteration in replica 1 on device 1 error while Training on GPU

Input:
I am trying to train a BertPunc model on the train2012 data used in the git link: https://github.com/nkrnrnk/BertPunc.
While running on the server, with 4 GPUs enabled, below is the error I get:
StopIteration: Caught StopIteration in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/notebooks/model_BertPunc.py", line 16, in forward
    x = self.bert(x)
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/anaconda3/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 861, in forward
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask,
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/anaconda3/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 727, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
StopIteration

From the link: https://github.com/huggingface/transformers/issues/8145, this appears to be happening when the data gets moved back and forth between multiple GPUs.
As per the git link: https://github.com/interpretml/interpret-text/issues/117, we need to downgrade PyTorch version to 1.4 from 1.7 which I use currently. For me downgrading the version isnt an option as I have other scripts that use Torch 1.7 version. What should I do to overcome this error?
I cant put the whole code here as there are too many lines, but here is the snippet that gives me the error:
bert_punc, optimizer, best_val_loss = train(bert_punc, optimizer, criterion, epochs_top,
        data_loader_train, data_loader_valid, save_path, punctuation_enc, iterations_top, best_val_loss=1e9)

Here is my DataParallel code:
   bert_punc = nn.DataParallel(BertPunc(segment_size, output_size, dropout)).cuda()

I tried changing the Dataparallel line to divert the training to only 1 GPU , out of 4 present. But that gave me a space issue, and hence had to revert the code back to default.
Here is the link to all scripts that I am using: https://github.com/nkrnrnk/BertPunc
Please advice.

Output:
change
extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
to
extended_attention_mask = extended_attention_mask.to(dtype=torch.float32) # fp16 compatibility
For more details, see https://github.com/vid-koci/bert-commonsense/issues/6


Instruction: Updating weights manually in Pytorch

Input:
import torch
import math

# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# For this example, the output y is a linear function of (x, x^2, x^3), so
# we can consider it as a linear layer neural network. Let's prepare the
# tensor (x, x^2, x^3).
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-6

Then I print the weights
parameters = list(model.parameters())
print(parameters)

Results:
[Parameter containing:
tensor([[ 0.0407,  0.2680, -0.1148]], requires_grad=True), Parameter containing:
tensor([-0.0132], requires_grad=True)]

y_pred = model(xx)
loss = loss_fn(y_pred, y)
model.zero_grad()
loss.backward()

Updating the weights:
with torch.no_grad():
    for param in model.parameters():
        param -= 1e-6 * param.grad

and then
list(model.parameters())
[Parameter containing:
 tensor([[ 0.0532,  0.2472, -0.0393]], requires_grad=True),
 Parameter containing:
 tensor([-0.0167], requires_grad=True)]

The weights were updated. I got confused. How is that possible? I thought just param variable in the for loop changed, not model.parameters().
But when change the code a bit
with torch.no_grad():
    for param in model.parameters():
        param -= 1e-6

The weights didn't change. So I guess it related to param.grad. Can you guys explain to me about that?

Output:
param variable inside the loop references each element of model.parameters().  Thus, updating param is the same as updating the elements of model.parameters().
As for your second example, I think decrementing by 1e-6 is just not enough for you to see the effect. Try param -= 1. and see if this has any effect on model.parameters().


Instruction: LSTM/RNN in pytorch The relation between forward method and training model

Input:
I'm still fairly new to neural networks, so sorry on beforehand for any ambiguities to the following.
In a "standard" LSTM implementation for language task, we have the following (sorry for the very rough sketches):
class LSTM(nn.Module):
    def __init__(*args):
    ...

    def forward(self, input, states):

        lstn_in = self.model['embed'](input)
        lstm_out, hidden = self.model['lstm'](lstm_in,states)

        return lstm_out, hidden

Later on, we call upon this model in the training step:
def train(*args):

    for epoch in range(epochs):
        ....
        *init_zero_states
        ...
        out, states = model(input, states)
        ...
    return model

Let's just say, that I have 3 sentences as input:
sents = [[The, sun, is, shiny],
 [The, beach, was, very, windy],
 [Computer, broke, down, today]]

model = train(LSTM, sents)

All words in all sentences gets converted to embeddings and loaded into the model.
Now the question:

Does the self.model['lstm'] iterate though all words from all articles and makes one output after every word? or every sentence?

How does the model make distinction between the 3 sentences, such as after getting "The", "sun", "is", "shiny", does something (such as the states) in the 'lstm' reset and begin anew?

The "out" in the train step after out, states = model(input, states) is the output after running all 3 sentences and hence the combined "information" from all 3 sentences?

Thanks!

Output:
when using LSTMs in Pytorch you usually use the nn.LSTM function. Here is a quick example and then an explanation what happens inside:
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.embedder = nn.Embedding(voab_size, embed_size)

        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x = self.embedder(x)

        # every time you pass a new sentence into the model you need to create
        # a new hidden-state (the LSTM requires, unlike RNNs, two hidden-states in a tuple)

        hidden = (torch.zeros(num_layers, batch_size, hidden_size), torch.zeros(num_layers, batch_size, hidden_size))
        x, hidden = self.lstm(x, hidden)

        # x contains the output states of every timestep,
        # for classifiction we mostly just want the last one
        x = x[:, -1]

        x = self.fc(x)
        x = self.softmax(x)
        return x

So, when taking a look at the nn.LSTM function, you see all N embedded words are passed into it at once and you get as output all N outputs (one from every timestep). That means inside of the lstm function, it iterates over all words in the sentence embeddings. We just dont see that in the code. It also returns the hiddenstate of every timestep but you dont have to use that further. In most cases you can just ignore that.
As pseudo code:
def lstm(x):
    hiddenstates = init_with_zeros()
    outputs, hiddenstates = [], []
    for e in x:
        output, hiddenstate = neuralnet(e, hiddenstate)

        outputs.append(output)
        hiddenstates.append(hiddenstate)

    return outputs, hiddenstates

sentence = ["the", "sun", "is", "shiny"]
sentence = embedding(sentence)

outputs, hiddenstates = lstm(sentence)


Instruction: Estimating mixture of Gaussian models in Pytorch

Input:
I actually want to estimate a normalizing flow with a mixture of gaussians as the base distribution, so I'm sort of stuck with torch.  However you can reproduce my error in my code by just estimating a mixture of Gaussian model in torch.  My code is below:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True).to(device)
means = torch.tensor(np.random.randn(8,2),requires_grad=True).to(device)#torch.randn(8,2,requires_grad=True).to(device)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True).to(device)
mix = D.Categorical(weights)
comp = D.Independent(D.Normal(means,stdevs), 1)
gmm = D.MixtureSameFamily(mix, comp)

num_iter = 10001#30001
num_iter2 = 200001
loss_max1 = 100
for i in range(num_iter):
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    optimizer1.zero_grad()
    loss2.backward()
    optimizer1.step()

The error I get is:
0
8.089411823514835
Traceback (most recent call last):

  File "/home/cameron/AnacondaProjects/gmm.py", line 183, in <module>
    loss2.backward()

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

After as you see the model runs for 1 iteration.

Output:
There is ordering problem in your code, since you create Gaussian mixture model outside of training loop, then when calculate the loss the Gaussian mixture model will try to use the initial value of the parameters that you set when you define the model, but the optimizer1.step() already modify that value so even you set loss2.backward(retain_graph=True) there will still be the error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Solution to this problem is simply create new Gaussian mixture model whenever you update the parameters, example code running as expected:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True)
means = torch.tensor(np.random.randn(8,2),requires_grad=True)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True)

parameters = [weights, means, stdevs]
optimizer1 = optim.SGD(parameters, lr=0.001, momentum=0.9)

num_iter = 10001
for i in range(num_iter):
    mix = D.Categorical(weights)
    comp = D.Independent(D.Normal(means,stdevs), 1)
    gmm = D.MixtureSameFamily(mix, comp)

    optimizer1.zero_grad()
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    loss2.backward()
    optimizer1.step()

    print(i, loss2)


Instruction: Managing several versions of Python

Input:
Ok, I am relatively new to Python (more of a MATLAB / R /Stata user). I previously installed Python on my computer from the Python website. Everything was running smoothly until I had to install Pytorch too. I tried installing it via pip to no avail, so I had to make a new installation of Python but this time with Anaconda.
However, now I have a mess and I can not load Scypi on Anaconda and I can not load Pytorch in the regular Python I have. Having to run them separately is driving me insane. Is there a way that I can merge the two versions together or should I uninstall and stick to only one?

Output:
Put this in environment.yml file
name: myenv
channels:
  - pytorch
  - anaconda
dependencies:
  - pytorch
  - torchvision
  - torchaudio
  - cpuonly
  - scipy

You can change the name so that it suits your taste.
You should modify dependencies according to the install command provided by https://pytorch.org.
And you can continue after running the following command.
conda activate [your environment name]

But as described in the conda documentation, if you want to use pip, you should install pip for your conda environment and then use it after using conda for as many packages as possible.
This way, you don't need to uninstall your original python. And in regard to merging the two environment, I haven't heard of it and it seems not a good practice if it's possible.


Instruction: Why do sometimes CNN models predict just one class out of all others?

Input:
I am relatively new to the deep learning landscape, so please don't be as mean as Reddit! It seems like a general question so I won't be giving my code here as it doesn't seem necessary (if it is, here's the link to colab)
A bit about the data: You can find the original data here. It is a downsized version of the original dataset of 82 GB.
Once I trained my CNN on this, it predicts 'No Diabetic Retinopathy' (No DR) every single time, leading to an accuracy of 73%. Is the reason for this is just the vast amount of No DR images or something else? I have no idea! The 5 classes I have for prediction are ["Mild", "Moderate", "No DR", "Proliferative DR", "Severe"].
It's probably just bad code, was hoping you guys could help

Output:
I was about to comment:

A more rigorous approach would be to start measuring your dataset balance: how many images of each class do you have? This will likely give an answer to your question.

But couldn't help myself look at the link you gave. Kaggle already gives you an overview of the dataset:

Quick calculation: 25,812 / 35,126 * 100 = 73%. That's interesting, you said you had an accuracy of 74%. Your model is learning on an inbalanced dataset, with the first class being over represented, 25k/35k is enormous. My hypothesis is that your model keeps predicting the first class which means that on average you'll end up with an accuracy of 74%.
What you should do is balance your dataset. For example by only allowing 35,126 - 25,810 = 9,316 examples from the first class to appear during an epoch. Even better, balance your dataset over all classes such that each class will only appear n times each, per epoch.


Instruction: apex: TypeError: Class advice impossible in Python3

Input:
i installed apex:
pip uninstall apex

My configurations are as follows:

torch version 1.7.1
python version 3.7
cuda 11.0

I came across this error after running import pytorch:
TypeError: Class advice impossible in Python3. Use the @implementer class decorator instead.

Output:
I uninstalled the old version of apex and reinstalled a new version. It worked. Thanks.
git clone https://www.github.com/nvidia/apex
cd apex
python setup.py install


Instruction: How do I link in all of PyTorch with a pybind11 .so?

Input:
I have a pybind11 c++ project which uses the pytorch c++ api:
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <math.h>
#include <torch/torch.h>

...

void f()
{
...
   torch::Tensor dynamic_parameters = torch::full({1}, /*value=*/0.5, torch::dtype(torch::kFloat64).requires_grad(true));
   torch::optim::SGD optimizer({dynamic_parameters}, /*lr=*/0.01);
...
}

PYBIND11_MODULE(reson8, m)
{
    m.def("my_function", &my_function, "");
}

I use distutils to compile this into a .so that can be imported in Python:
from distutils.core import setup, Extension

def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)
      config.add_extension('reson8',
                           ['reson8.cpp'],
                           extra_info=info,
                           include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"])

      return config

if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)

It compiles with no error, but on running "import reson8" in python I get this error:
importerror: undefined symbol: _ZTVN5torch5optim9OptimizerE

I'm unsure if it's whether pytorch hasn't been linked into my so (although the .so is 10mb which is rather large if it doesn't include pytorch, but maybe all pybind11 .so files are large.)
How can I solve this issue?

Output:
I eventually found I needed to use the Anaconda version of torchlib rather than my own, as well as using Torch's CppExtension.  Here is my working setup.py:
from distutils.core import setup, Extension
from torch.utils.cpp_extension import BuildExtension, CppExtension

def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)

      config.ext_modules.append(CppExtension(
                name='reson8',
                sources=['reson8.cpp'],
                extra_info=info,
                extra_compile_args=['-g', '-D_GLIBCXX_USE_CXX11_ABI=0'],
                extra_ldflags=['-ltorch_python'],
                include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib"
                                          ]
                                ))

      return config

if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)


Instruction: Does NVLink accelerate training with DistributedDataParallel?

Input:
Nvidia's NVLink accelerates data transfer between several GPUs on the same machine.
I train large models on such a machine using PyTorch.
I see why NVLink would make model-parallel training faster, since one pass through a model will involve several GPUs.
But would it accelerate a data-parallel training process using DistributedDataParallel?

Output:
How does data-parallel training on k GPUs works?
You split your mini batch into k parts, each part is forwarded on a different GPU, and gradients are estimated on each GPU. However, (and this is super crucial) updating the weights must be synchronized between all GPUs. This is where NVLink becomes important for data-parallel training as well.


Instruction: LSTM Autoencoder Output Layer

Input:
I am trying to get a LSTM autoencoder to recreate its inputs. So far I have:
class getSequence(nn.Module):
    def forward(self, x):
        out, _ = x
        return out

class getLast(nn.Module):
    def forward(self, x):
        out, states = x
        states = states[len(states) - 1]
        return states

class AEncoder(nn.Module):
    def __init__(self, input_size, first_layer, second_layer, n_layers):
        super(AEncoder, self).__init__()
        self.n_layers = n_layers
        self.encode = nn.Sequential(nn.LSTM(input_size, first_layer, batch_first=True),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, second_layer),
                                    getLast())
        self.decode = nn.Sequential(nn.LSTM(second_layer, first_layer),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, input_size),
                                    getSequence())

    def forward(self, x):
        x = x.float()
        x = self.encode(x)
        x = x.repeat(32, 1, 1) # repeating last hidden state of self.encode
        x = self.decode(x)
        return x

While researching I have been seeing some people adding a time-distributed dense layer at the end of the self.decode. I am confused if that final layer is specific to other tasks autoencoders are used for, if so, can I ignore that layer if I am only trying to recreate inputs?

Output:
The time-distributed dense layer as name suggested just an ordinary dense layer that apply to every temporal slice of an input, you can think it as special form of RNN cell, i.e without recurrent hidden state.
So you can using any layer that is time-distributed as your output layer for an Autoencoder that deal with time-distributed inputs, e.g RNN layer with LSTM Cell, GRU Cell, Simple RNN Cell or time-distributed dense layer; As in research paper that propose the LSTM-Autoencoder, it basic model for reconstruct sequence of  vectors (image patches or features) only using one LSTM layer in both encoder and decoder, model structure is:

Following is an example to using time-distributed dense layer in decoder:
class Decoder(nn.Module):

  def __init__(self, seq_len, input_dim=64, n_features=1):
    super(Decoder, self).__init__()

    self.seq_len, self.input_dim = seq_len, input_dim
    self.hidden_dim, self.n_features = 2 * input_dim, n_features

    self.rnn = nn.LSTM(
      input_size=input_dim,
      hidden_size=self.hidden_dim,
      num_layers=1,
      batch_first=True)

    self.output_layer = nn.Linear(self.hidden_dim, n_features)

  def forward(self, x):
    x = x.repeat(self.seq_len, self.n_features)
    x = x.reshape((self.n_features, self.seq_len, self.input_dim))
    x, (hidden_n, cell_n) = self.rnn(x)
    x = x.reshape((self.seq_len, self.hidden_dim))
    return self.output_layer(x)


Instruction: The reason behind rgb image normalization parameters in pytorch

Input: I've seen transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) both in lots of tutorials and pytorch docs, I know the first param is mean and the second one is std. I can't understand why the values for different channels differ.

Output:
That can just be the distribution of the colours in the original dataset of the code authors (such as COCO or PascalVOC). It would only be through chance that all colours are equally represented. However, if you used the same mean in your case, I doubt it would make much of a difference due to the similarity of the means and stds.
For example, in my custom dataset taken from a GoPro camera, the means and standard deviations are as such:
mean: [0.2841186 , 0.32399923, 0.27048702],
std: [0.21937862, 0.26193094, 0.23754872]

where the means are hardly equal. This does not mean, however, that they are treated differently in the ML model. All this transform does is make sure that each feature is standardised (through z-score standardisation).
Think about it this way: if one colour is represented with a generally-higher intensity in your dataset (eg. if you have lots of vibrant "blue" pictures of the beach, the sky, lagoons, etc), than you will have to subtract a larger number from that channel to ensure that your data is standardised.


Instruction: Why does nn.Conv1d work on 2d feature [b, c, h, w]?

Input:
I am wondering why conv1d works on 2d feature(batch, channel, height, width).
An nn.Conv1d(channel, channel, kernel_size=(1,1)) works when I put 2d feature, but gives different result from nn.Conv2d(channel, channel, kernel_size=1).
I want to know why conv1d works and what it mean by 2d kernel size in 1d convolution.

Output:
"I want to know why conv1d works and what it mean by 2d kernel size in 1d convolution"
It doesn't have any reason not to work. Under the hood all this "convolution" means is "Dot Product", now it could be between matrix and vector, matrix and matrix, vector and vector, etc. Simply put, the real distinction between 1D and 2D convolution is the freedom one has to move along the spatial dimension of input. This means If you look at 1D convolution, It can move along one direction only, that is, the temporal dimension of the input (Note the kernel could be a vector, matrix whatever that doesn't matter). On the other hand, 2D convolution has the freedom to move along 2 dimensions (height and width) of the input that is the spatial dimension. If it still seems confusing, have a look at the gifs below.
1D Convolution in action:
Note: It's a 1D convolution with kernel size 3x3, look how it only moves down the input which is the temporal dimension.

2D Connvolution in action:
Note: It's a 2D convolution with kernel size 3x3, look how it moves along both width and height of the input which is the spatial dimension.

I think It's clear now what is the actual difference between 1D and 2D conv and why they both would produce different results for the same input.


Instruction: Functionality of `detach()` in PyTorch for this specific case

Input: I know that detach() is used for detaching a variable from the computational graph. In that context, are the following expressions x = x - torch.mean(x, dim=0).detach() and x = x - torch.mean(x, dim=0) equivalent? I just want to subtract the mean out, don't want to pass gradients through the average calculation.

Output: If you do not detach the mean, then you have lateral dependencies between all elements in the batch (dim=0) when estimating the gradient.


Instruction: Computing loss along batch dimension in PyTorch

Input:
I have tensors of BxCxHxW where B is the batch. I am interested in how is the loss implemented in PyTorch when batch size is more than 1. The below is what we usually do in PyTorch,
l1_loss = torch.nn.L1Loss()
loss = l1_loss(pred,gt)

Is it that the loss is the average loss over the batch?
If this is indeed the case, then is the following codes equivalent to what is we usually do in PyTorch (the code above)?
l1_loss = torch.nn.L1Loss()

for idx in range(B):
  if idx==0:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:])
  else:
    loss = l1_loss(pred[idx,:,:,:],gt[idx,:,:,:]) + loss

loss = loss/B

Output:
The documentation describes the behavior of L1loss : it is indeed (by default) the mean over the whole batch. You can change it easily to the sum instead :
l1_loss = torch.nn.L1Loss(reduction='sum')

Yes your code is equivalent to what Pytorch does. A version without the call to L1loss would be :
# Assuming your output is a vector of shape (B, F)
loss = torch.abs(pred - gt).sum(dim=1).mean(dim=0)


Instruction: How to create tensor directly on GPU, or on device of another tensor?

Input:
I found this discussion about this, in which the code
if std.is_cuda:
    eps = torch.FloatTensor(std.size()).cuda().normal_()
else:
    eps = torch.FloatTensor(std.size()).normal_()

becomes the nice
eps = std.new().normal_()

but it is said there that

The new() method is deprecated.

How to create a new tensor directly on a specific device?
How to create a new tensor on the same device like another tensor without the ugly if?

Output:
The documentation is quite clear now about this I think. Here are described the 4 main ways to create a new tensor, and you just have to specify the device to make it on gpu :
t1 = torch.zeros((3,3), device=torch.device('cuda'))
t2 = torch.ones_like(t1, device=torch.device('cuda'))
t3 = torch.randn((3,5), device=torch.device('cuda'))

And this link adds further information about the torch.tensor() constructor. Again, the device is an argument to be specified.
If you want to use the device of another tensor, you can access it with tensor.device:
t4 = torch.empty((2,2), device=t3.device)


Instruction: pytorch gathering from a 1d tensor

Input:
I have a long list of values stored in a 1d tensor from which I like to gather values. Given a 2d tensor where the second dimension denotes a set of indices, I would like to compute a new 2d tensor where the second dimension are all the values gathered given by the indices. Basically, I have a batch of indices and I use those indices to obtain a batch of values from the 1d-tensor.
I am new to pytorch but I managed to compute exactly that using pytorch's gather function
list_values = torch.tensor([1, 2, 3, 4, 5, 6])
list_values = list_values.unsqueeze(0)
list_values = list_values.expand((2, 6))
indices = torch.tensor([[1, 2], [2, 3]])
result = torch.gather(list_values, 1, indices)

This works perfectly fince and gives the correct result. However, If I am not mistaken, the expand operation is quite expensive in terms of memory as the number of elements in "list_value" grows.
Is there a better solution ?

Output:
Nope, take a look at the documentation :
Expanding a tensor does not allocate new memory, but only creates a new view on the existing tensor where a dimension of size one is expanded to a larger size by setting the stride to 0. Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory.
You solution is completely fine


Instruction: GCP AI Platform: Error when creating a custom predictor model version ( trained model Pytorch model + torchvision.transform)

Input:
Am currently trying to deploy a custom model to AI platform by following https://cloud.google.com/ai-platform/prediction/docs/deploying-models#gcloud_1. which is based on a combination of the pre-trained model from 'Pytorch' and 'torchvision.transform'. Currently, I keep getting below error which happens to be related to 500MB constraint on custom prediction.
ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to experience errors, please contact support.
Setup.py
from setuptools import setup
from pathlib import Path

base = Path(__file__).parent
REQUIRED_PACKAGES = [line.strip() for line in open(base/"requirements.txt")]
print(f"\nPackages: {REQUIRED_PACKAGES}\n\n")

# [torch==1.3.0,torchvision==0.4.1, ImageHash==4.2.0
# Pillow==6.2.1,pyvis==0.1.8.2] installs 800mb worth of files

setup(description="Extract features of a image",
      author=,
      name='test',
      version='0.1',
      install_requires=REQUIRED_PACKAGES,
      project_urls={
                    'Documentation':'https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines#tensorflow',
                    'Deploy':'https://cloud.google.com/ai-platform/prediction/docs/deploying-models#gcloud_1',
                    'Ai_platform troubleshooting':'https://cloud.google.com/ai-platform/training/docs/troubleshooting',
                    'Say Thanks!': 'https://medium.com/searce/deploy-your-own-custom-model-on-gcps-ai-platform-
 7e42a5721b43',
                    'google Torch wheels':"http://storage.googleapis.com/cloud-ai-pytorch/readme.txt",
                    'Torch & torchvision wheels':"https://download.pytorch.org/whl/torch_stable.html "
                    },
    python_requires='~=3.7',
    scripts=['predictor.py', 'preproc.py'])

Steps taken:
Tried adding ‘torch’ and torchvision directly to ‘REQUIRED_PACKAGES’ list in setup.py file in order to provide PyTorch + torchvision as a dependency to be installed while deployment.  I am guessing, Internally Ai platform downloads PyPI package for PyTorch which is +500 MB, this results in the failure of our model deployment. If I just deploy the model with 'torch' only and it seems to be working (of course throws error for not able to find library 'torchvision')
File size

pytorch (torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl about 111MB)
torchvision (torchvision-0.4.1+cpu-cp37-cp37m-linux_x86_64.whl about 46MB) from https://download.pytorch.org/whl/torch_stable.html and stored it on GKS.
The zipped predictor model  file (.tar.gz format) which is the output of  setup.py (5kb )
A trained PyTorch model (size 44MB)

In total, the model dependencies should be less than 250MB but still, keep getting this error. Have also tried to use the torch and torchvision provided from Google mirrored packages  http://storage.googleapis.com/cloud-ai-pytorch/readme.txt, but same memory issue persists. AI platform is quite new for us and would like some input from professional’s.
MORE INFO:
GCP CLI Input:
My environment variable:
BUCKET_NAME= “something”
MODEL_DIR=<>
VERSION_NAME='v6'
MODEL_NAME="something_model"
STAGING_BUCKET=$MODEL_DIR<>
# TORCH_PACKAGE=$MODEL_DIR"package/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl"
# TORCHVISION_PACKAGE=$MODEL_DIR"package/torchvision-0.4.1+cpu-cp37-cp37m-linux_x86_64.whl"
TORCH_PACKAGE=<>
TORCHVISION_PACKAGE=<>
CUSTOM_CODE_PATH=$STAGING_BUCKET"imt_ai_predict-0.1.tar.gz"
PREDICTOR_CLASS="predictor.MyPredictor"
REGION=<>
MACHINE_TYPE='mls1-c4-m2'

gcloud beta ai-platform versions create $VERSION_NAME   \
--model=$MODEL_NAME   \
--origin=$MODEL_DIR  \
 --runtime-version=2.3  \
 --python-version=3.7   \
--machine-type=$MACHINE_TYPE  \
 --package-uris=$CUSTOM_CODE_PATH,$TORCH_PACKAGE,$TORCHVISION_PACKAGE   \
--prediction-class=$PREDICTOR_CLASS \

GCP CLI Output:
 **[1] global**
 [2] asia-east1
 [3] asia-northeast1
 [4] asia-southeast1
 [5] australia-southeast1
 [6] europe-west1
 [7] europe-west2
 [8] europe-west3
 [9] europe-west4
 [10] northamerica-northeast1
 [11] us-central1
 [12] us-east1
 [13] us-east4
 [14] us-west1
 [15] cancel
Please enter your numeric choice:  1

To make this the default region, run `gcloud config set ai_platform/region global`.

Using endpoint [https://ml.googleapis.com/]
Creating version (this might take a few minutes)......failed.
ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: **Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to experience errors, please contact support.**

My finding:
Have found articles of people struggling in same ways for PyTorch package and made it work by installing torch wheels on the GCS (https://medium.com/searce/deploy-your-own-custom-model-on-gcps-ai-platform-
7e42a5721b43).
Have tried the same approach with torch and torchvision but no luck till now and waiting response from "cloudml-feedback@google.com cloudml-feedback@google.com". Any help on getting custom  torch_torchvision based custom predictor working on AI platform that will be great.

Output:
Got this fixed by a combination of few things. I stuck to 4gb CPU MlS1 machine and custom predictor routine (<500MB).

Install the libraries using setup.py parameter but instead of parsing just the package name and it's version, add correct torch wheel (ideally <100 mb).

REQUIRED_PACKAGES = [line.strip() for line in open(base/"requirements.txt")] +\
['torchvision==0.5.0', 'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl']

I reduced the steps for preprocessing taken. Couldn't fit in all of them so jsonify your SEND response and GET one from both preproc.py and predictor.py

import json
json.dump(your data to send to predictor class)

Import those function from the class of a library that is needed.

from torch import zeros,load
    your code

[Important]

Haven't tested different types of serializing object for the trained model, could be a difference there as well to which one (torch.save, pickle, joblib etc) is memory saving.

Found this link for those whose organization is a partner with GCP might be able to request more quota (am guessing from 500MB to 2GB or so). Didn't had to go in this direction as my issue was resolved and other poped up lol.
https://cloud.google.com/ai-platform/training/docs/quotas


Instruction: Numpy/PyTorch funny tensor product

Input:
I've got a 4 dimensional torch tensor parameter defined like this :
nn.parameter.Parameter(data=torch.Tensor((13,13,13,13)), requires_grad=True)

and four tensors with dims (batch_size,13) (or one tensor with dims (batch_size,4,13)).
I'd like to get a tensor with dims (batch_size) equal to the formula at the end of this picture :
[EDIT: I made a mistake in the first pict, I've corrected it]

I've seen in the torch documentation the function tensordot, but I can't manage to make it work by myself.

Output:
whenever you have a funny tensor product torch.einsum (or numpy.einsum) is your friend:
batch_size = 5
A = torch.rand(13, 13, 13, 13)
a = torch.rand(batch_size, 13)
b = torch.rand(batch_size, 13)
c = torch.rand(batch_size, 13)
d = torch.rand(batch_size, 13)
B = torch.einsum('ijkl,bi,bj,bk,bl->b', A, a, b, c, d)


Instruction: Subsetting a two dimensional tensor with a one dimensional tensor

Input:
I want to extract from each row of a two-dimensional tensor the column that is stored in another one dimensional tensor.
import torch
test_tensor = tensor([1,-2,3], [-2,7,4]).float()
select_tensor = tensor([1,2])

So in this particular example I would like to get the element in position 1 for the first row (so -2) and the element in position 2 for the second row (so 4).
I tried:
test_tensor[:, select_tensor]

But this selects the elements at position 1 and 2 for each row. I suspect it might be something very simple that I am missing.

Output:
You can use torch.gather
import torch
test_tensor = torch.tensor([[1,-2,3], [-2,7,4]]).float()
select_tensor = torch.tensor([1,2], dtype=torch.int64).view(-1,1) # number of dimension should match with the test tensor.
final_tensor = torch.gather(test_tensor, 1, select_tensor)
final_tensor

output
tensor([[-2.],
        [ 4.]])

or, use torch.view to flatten the output tensor: final_tensor.view(-1) will give you tensor([-2.,  4.])


Instruction: PyTorch: "ValueError: can't optimize a non-leaf Tensor" after changing pretrained model from 3 RGB Channels to 4 Channels

Input:
I have been trying to change the pretrained PyTorch Densenet's first conv layer from 3 channels to 4 channels while maintaining its original RGB channel's pretrained weights. I have done the following codes, but the optimizer part throws me this error:  "ValueError: can't optimize a non-leaf Tensor" .
import torchvision.models as models
import torch.nn as nn
backbone = models.__dict__['densenet169'](pretrained=True)

weight1 = backbone.features.conv0.weight.data.clone()
new_first_layer  = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
with torch.no_grad():
    new_first_layer.weight[:,:3] = weight1

backbone.features.conv0 = new_first_layer
optimizer = torch.optim.SGD(backbone.parameters(), 0.001,
                                 weight_decay=0.1)  # Changing this optimizer from SGD to ADAM

I have also tried to remove the argument with torch.no_grad(): but this issue still remains:
  ValueError                                Traceback (most recent call last)
<ipython-input-343-5fc87352da04> in <module>()
     11 backbone.features.conv0 = new_first_layer
     12 optimizer = torch.optim.SGD(res.parameters(), 0.001,
---> 13                                  weight_decay=0.1)  # Changing this optimizer from SGD to ADAM

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/sgd.py in __init__(self, params, lr, momentum, dampening, weight_decay, nesterov)
     66         if nesterov and (momentum <= 0 or dampening != 0):
     67             raise ValueError("Nesterov momentum requires a momentum and zero dampening")
---> 68         super(SGD, self).__init__(params, defaults)
     69
     70     def __setstate__(self, state):

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
     50
     51         for param_group in param_groups:
---> 52             self.add_param_group(param_group)
     53
     54     def __getstate__(self):

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
    231                                 "but one of the params is " + torch.typename(param))
    232             if not param.is_leaf:
--> 233                 raise ValueError("can't optimize a non-leaf Tensor")
    234
    235         for name, default in self.defaults.items():

ValueError: can't optimize a non-leaf Tensor

My PyTorch version is: 1.7.0.
Could you guys please help? Thanks alot!
Regards.

Output:
I think I have resolved this problem!:
import torchvision.models as models
import torch.nn as nn
from torch.autograd import Variable
backbone = models.__dict__['densenet169'](pretrained=True)
weight1 = backbone.features.conv0.weight.clone()
new_first_layer  = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False).requires_grad_()
new_first_layer.weight[:,:3,:,:].data[...] =  Variable(weight1, requires_grad=True)
backbone.features.conv0 = new_first_layer
optimizer = torch.optim.SGD(res.parameters(), 0.001,
                                 weight_decay=0.1)


Instruction: output prediction of pytorch lightning model

Input:
This is potentially a very easy question. I just started with PyTorch lightning and can't figure out how to receive the output of my model after training.
I am interested in both predictions of y_train and y_test as an array of some sort (PyTorch tensor or NumPy array in a later step) to plot next to the labels using different scripts.
dataset = Dataset(train_tensor)
val_dataset = Dataset(val_tensor)
training_generator = torch.utils.data.DataLoader(dataset, **train_params)
val_generator = torch.utils.data.DataLoader(val_dataset, **val_params)
mynet = Net(feature_len)
trainer = pl.Trainer(gpus=0,max_epochs=max_epochs, logger=logger, progress_bar_refresh_rate=20, callbacks=[early_stop_callback], num_sanity_val_steps=0)
trainer.fit(mynet)

In my lightning module I have the functions:
def __init__(self, random_inputs):

def forward(self, x):

def train_dataloader(self):

def val_dataloader(self):

def training_step(self, batch, batch_nb):

def training_epoch_end(self, outputs):

def validation_step(self, batch, batch_nb):

def validation_epoch_end(self, outputs):

def configure_optimizers(self):

Do I need a specific predict function or is there any already implemented way I don't see?

Output:
You can use the predict method as well. Here is the example from the document. https://pytorch-lightning.readthedocs.io/en/latest/starter/introduction_guide.html
class LitMNISTDreamer(LightningModule):

    def forward(self, z):
        imgs = self.decoder(z)
        return imgs

    def predict_step(self, batch, batch_idx: int , dataloader_idx: int = None):
        return self(batch)

model = LitMNISTDreamer()
trainer.predict(model, datamodule)


Instruction: How to use k channels in CNN for k FC Layers

Input:
I have an encoder, which outputs a tensor with shape (bn, c * k, 32, 32). I now want produce k means with shape (bn, k, 1, 2). So the means are 2-dim coordinates. To do so, I want to use k FC Layers, while for each mean k_i I only want to use c channels.
So my idea is, that I reshape the encoder output out to a 5d tensor with shape (bn, k, c, 32, 32). Then I can use the flattened out[:, 0] ... out[:, k] as input for the k linear layers.
The trivial solution would be to define the linear layers manually:
self.fc0 = nn.Linear(c * 32 * 32, 2)
...
self.fck = nn.Linear(c * 32 * 32, 2)

Then I could define the forward pass for each mean as follows:
mean_0 = self.fc0(out[:, 0].reshape(bn, -1))
...
mean_k = self.fck(out[:, k].reshape(bn, -1))

Is there a more efficient way to do that?

Output:
I believe you are looking for a grouped convolution. You can let axis=1 have k*c tensors, so the input shape is (bn, k*c, 32, 32). Then use a nn.Conv2d convolution layer with 2*k filters and set to receive k groups so it's not a fully connected channel-wise (only k groups of c maps: convolves c at a time.
>>> bn = 1; k = 5; c = 3
>>> x = torch.rand(bn, k*c, 32, 32)
>>> m = nn.Conv2d(in_channels=c*k, out_channels=2*k, kernel_size=32, groups=k)

>>> m(x).shape
torch.Size([4, 10, 1, 1])

Which you can then reshape to your liking.

In terms of number of parameters. A typical nn.Conv2d usage would be:
>>> m = nn.Conv2d(in_channels=c*k, out_channels=2*k, kernel_size=32)
>>> sum(layer.numel() for layer in m.parameters())
153610

Which is exactly c*k*2*k*32*32 weights, plus 2*k biases.
In your case, you would have
>>> m = nn.Conv2d(in_channels=c*k, out_channels=2*k, kernel_size=32, groups=k)
>>> sum(layer.numel() for layer in m.parameters())
30730

Which is exactly c*2*k*32*32 weights, plus 2*k biases. i.e. k times less than the previous layer. A given filter's has only c layers (not k*c) which means it will have an input with c channels (i.e. one of the k groups containing c maps)


Instruction: What is the best way to create augmentation on image dataset while training instance segmentation?

Input:
Background : I am using YOLACT instance segmentation model to train set of images. The dataset size is very small (~20 images). The model doesn't converge properly (of-course given the dataset size). I wanted to increase the dataset size by adding some augmented images. I know we have various image augmentation techniques and packages like imgaug , albumentation, opencv etc. but I need image & annotation file ( COCO JSON ) format to train the model.
So my question is :
Is there a package that helps me to automatically generate the annotations of augmented images ?
or
Is there a better way to address my issue ?
Thank you in advance for your help!

Output:
Simple Copy Paste is a strong method for data augmentation for instance segmentation related tasks.
Check about the research paper here.
For unofficial github code, check here
Albumentation and TorMentor are also useful libraries for data augmentation.


Instruction: understanding how PyTorch Linear works

Input:
I am considering the sample code from the documentation:
import torch
from torch import nn
#
m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
print(output.size())

The output is :
torch.Size([128, 30])

The constructor of Linear is :
def __init__(self, in_features: int, out_features: int, bias: bool = True) -> None:

This is consistent with the way the instance is created, i.e.:
m = nn.Linear(20, 30)

However, when m is used, it receives a tensor
output = m(input)

as input. I do not understand why. Where is this tensor defined in the source code?

Output:
When you do m(input), the __call__ (what is __call__?) method is called, which internally calls forward method and does other stuff. This logic is written in the base class: nn.Module. For simplicity, assume for now, that doing m(input) is equivalent to m.forward(input).
And what's the input to forward? A tensor.
def forward(self, input: Tensor) -> Tensor


Instruction: Changing Dropout value during training

Input:
How can I change Dropout during training? For example
Dropout= [0.1, 0.2, 0.3]

I tried passing it as as a list but I couldn't make it work.

Output:
To change the dropout probability during training, you should use the functional version, i.e. torch.nn.functional.dropout.
The input arguments to the functional version of dropout are

the input tensor
the dropout probability (which you can alter)
a boolean to indicate if it is in training mode (you can use the self.training)
and a flag to indicate if you want the operation to be performed in place.

Thus, you can alter the probability of the dropout in your forward method, according to your needs.
For example, you can do in your forward method:

def forward(self, x):

    ...

    # apply some layers to the input
    h = self.my_layers(x)

    # set the value of p
    p = self.get_value_for_p()

    # apply dropout with new p
    h = torch.nn.functional.dropout(h, p, self.training)

    ...

More on the functional version of dropout, here: https://pytorch.org/docs/stable/nn.functional.html#dropout-functions


Instruction: Calling .backward() function for two different neural networks but getting retain_graph=True error

Input:
I have an Actor Critic neural network where the Actor is its own class and the Critic is its own class with its own neural network and .forward() function. I then am creating an object of each of these classes in a larger Model class. My setup is as follows:
self.actor = Actor().to(device)
self.actor_opt = optim.Adam(self.actor.parameters(), lr=lr)
self.critic = Critic().to(device)
self.critic_opt = optim.Adam(self.critic.parameters(), lr=lr)

I then calculate two different loss functions and want to update each neural network separately. For the critic:
loss_critic = F.smooth_l1_loss(value, expected)
self.critic_opt.zero_grad()
loss_critic.backward()
self.critic_opt.step()

and for the actor:
loss_actor = -self.critic(state, action)
self.actor_opt.zero_grad()
loss_actor.backward()
self.actor_opt.step()

However, when doing this, I get the following error:
RuntimeError: Trying to backward through the graph a second time, but the saved
intermediate results have already been freed. Specify retain_graph=True when
calling backward the first time.

When reading up on this, I understood that I only need to retain_graph=True when calling backward twice on the same network, and in most cases this is not good to set to True as I will run out of GPU. Moreover, when I comment out one of the .backward() functions, the error goes away, leading me to believe that for some reason the code is thinking that both backward() functions are being called on the same neural network, even though I think I am doing it separately. What could be the reason for this? Is there a way to specify for which neural network I am calling the backward function on?
Edit:
For reference, the optimize() function in this code here https://github.com/wudongming97/PyTorch-DDPG/blob/master/train.py uses backward() twice with no issue (I've cloned the repo and tested it). I'd like my code to operate similarly where I backprop through critic and actor separately.

Output:
Yes, you shouldn't do it like that. What you should do instead, is propagating through parts of the graph.
What the graph contains
Now, graph contains both actor and critic. If the computations pass through the same part of graph (say, twice through actor), it will raise this error.

And they will, as you clearly use actor and critic joined with loss value (this line: loss_actor = -self.critic(state, action))

Different optimizers do not change anything here, as it's backward problem (optimizers simply apply calculated gradients onto models)

Trying to fix it

This is how to fix it in GANs, but not in this case, see Actual fix paragraph below, read on if you are curious about the topic

If part of a neural network (critic in this case) does not take part in the current optimization step, it should be treated as a constant (and vice versa).
To do that, you could disable gradient using torch.no_grad context manager (documentation) and set critic to eval mode (documentation), something along those lines:
self.critic.eval()
with torch.no_grad():
    loss_actor = -self.critic(state, action)
...

But, here is a problem:

We are turning off gradient (tape recording) for action and breaking the graph!

hence this is not a viable solution.
Actual solution
It is much simpler than you think, one can see it in PyTorch's repository also:

Do not backpropagate after critic/actor loss
Calculate all losses (for both critic and actor)
sum them together
zero_grad for both optimizers
backpropagate with this summed value
critic_optimizer.step() and actor_optimizer.step() at this point

Something along those lines:
self.critic_opt.zero_grad()
self.actor_opt.zero_grad()

loss_critic = F.smooth_l1_loss(value, expected)
loss_actor = -self.critic(state, action)

total_loss = loss_actor + loss_critic
total_loss.backward()

self.critic_opt.step()
self.actor_opt.step()


Instruction: Having trouble installing PyTorch

Input:
I'm having trouble installing PyTorch.
C:\Users\myself>pip install torch
Collecting torch
  Using cached torch-1.7.1-cp38-cp38-win_amd64.whl (184.0 MB)
Requirement already satisfied: numpy in c:\users\myself\appdata\local\packages\pythonsoftwarefoundation.python.3.8_qbz5n2kfra8p0\localcache\local-packages\python38\site-packages (from torch) (1.19.1)
Requirement already satisfied: typing-extensions in c:\users\myself\appdata\local\packages\pythonsoftwarefoundation.python.3.8_qbz5n2kfra8p0\localcache\local-packages\python38\site-packages (from torch) (3.7.4.3)
Installing collected packages: torch
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\\Users\\myself\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python38\\site-packages\\caffe2\\python\\serialized_test\\data\\operator_test\\learning_rate_adaption_op_test.test_learning_rate_adaption_op_normalization.zip'

WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available.
You should consider upgrading via the 'C:\Users\myself\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip' command.

It says that I'm missing a file in caffe2, but I'm not sure how I can get that file back.

Output:
Torch wheel contains caffe2 directory.
1.Try --no-cache-dir option
pip install --no-cache-dir torch

2. Removing the MAX_PATH Limitation
See here 3. Using Python on Windows — Python 3.7.9 documentation. You can find some more detailed discussion here.


Instruction: Python calculate phash from an image located at a url

Input:
I want to calculate the phash from about 10.000.000 pictures, from which i only have the url where they are located at.
I know how to download a picture and then calculate the phash after that, but i always have to safe the picture first.
Is it possible to download the picture and calculate the phash without saving it or is it even possible to not download the picture at all and just calculate the phash just using the url?
This is my code to download the first ten pictures and calculate the phash:
folder, pic_savefolder = 'data', 'data/pictures'
file = 'external-asset-url-clean.csv'
path = os.path.join(folder,file)
df = pd.read_csv(path, header=None, names=["URL"])
counter = 0
hashes = set()
headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"}

for image_url in df['URL']:
    filename = image_url.split('/')[-1]
    try:
        r = requests.get(image_url, allow_redirects=False,verify=False, headers=headers)
        pathlong = os.path.join(pic_savefolder, filename)
        with open(pathlong,"wb") as f:
            f.write(r.content)
            hash = imagehash.phash(Image.open(pathlong))
            hashes.add((hash))
        counter += 1
        if counter > 10:
            break
except Exception as e:
    print(e)
    print("\n")

Output:
Instead of writing to a file, you can pass the contents directly if you use the .raw property instead of the .content one.
Here is how that looks in code:
image_data = Image.open(requests.get(image_url, stream=True).raw)
hash = imagehash.phash(image_data)


Instruction: Is there a function in PyTorch for matrix left division?

Input:
MATLAB has the backslash "\" operator. SciPy has "lsqr." Does PyTorch have an equivalent operator that solves systems of linear equations?
Specifically, I need to solve the matrix equation for A*X=B for A, and I need autograd to be able to backpropagate error through the operation.

Output:
There is no \ operator in Python. The closest you will get to is Scipy's implementation: scipy.sparse.linalg.lsqr.
You can either use

torch.solve to solve linear equations of shape AX=B

torch.lsrsq to

solve the least-squares problem min ||AX-B||_2 (if A.size(0) >= A.size(1)) or

solve the least-norm problem min ||X||_2 such that AX=B (if A.size(0) < A.size(1))

For solving XA=B you would use the transposed matrices:
def lsqrt(A, B):
    XT, _ = torch.solve(B.T, A.T)
    return XT.T


Instruction: Process output data from YOLOv5 TFlite

Input:
❔Question
Hi, I have successfully trained a custom model based on YOLOv5s and converted the model to TFlite. I feel silly asking, but how do you use the output data?
I get as output:

StatefulPartitionedCall: 0 = [1,25200,7]
from the converted YOLOv5 model
Netron YOLOv5s.tflite model

But I expect an output like:

StatefulPartitionedCall:3 = [1, 10, 4] # boxes
StatefulPartitionedCall:2 = [1, 10] # classes
StatefulPartitionedCall:1 = [1, 10] #scores
StatefulPartitionedCall:0 = [1] #count
(this one is from a tensorflow lite mobilenet model (trained to give 10 output data, default for tflite))
Netron mobilenet.tflite model

It may also be some other form of output, but I honestly have no idea how to get the boxes, classes, scores from a [1,25200,7] array.
(on 15-January-2021 I updated pytorch, tensorflow and yolov5 to the latest version)
The data contained in the [1, 25200, 7] array can be found in this file: outputdata.txt
0.011428807862102985, 0.006756599526852369, 0.04274776205420494, 0.034441519528627396, 0.00012877583503723145, 0.33658933639526367, 0.4722323715686798
0.023071227595210075, 0.006947836373001337, 0.046426184475421906, 0.023744791746139526, 0.0002465546131134033, 0.29862138628959656, 0.4498370885848999
0.03636947274208069, 0.006819264497607946, 0.04913407564163208, 0.025004519149661064, 0.00013208389282226562, 0.3155967593193054, 0.4081345796585083
0.04930267855525017, 0.007249316666275263, 0.04969717934727669, 0.023645592853426933, 0.0001222355494974181, 0.3123127520084381, 0.40113094449043274
...

Should I add a Non Max Suppression or something else, can someone help me please? (github YOLOv5 #1981)

Output:
Thanks to @Glenn Jocher I found the solution. The output is [xywh, conf, class0, class1, ...]
My current code is now:
def classFilter(classdata):
    classes = []  # create a list
    for i in range(classdata.shape[0]):         # loop through all predictions
        classes.append(classdata[i].argmax())   # get the best classification location
    return classes  # return classes (int)

def YOLOdetect(output_data):  # input = interpreter, output is boxes(xyxy), classes, scores
    output_data = output_data[0]                # x(1, 25200, 7) to x(25200, 7)
    boxes = np.squeeze(output_data[..., :4])    # boxes  [25200, 4]
    scores = np.squeeze( output_data[..., 4:5]) # confidences  [25200, 1]
    classes = classFilter(output_data[..., 5:]) # get classes
    # Convert nx4 boxes from [x, y, w, h] to [x1, y1, x2, y2] where xy1=top-left, xy2=bottom-right
    x, y, w, h = boxes[..., 0], boxes[..., 1], boxes[..., 2], boxes[..., 3] #xywh
    xyxy = [x - w / 2, y - h / 2, x + w / 2, y + h / 2]  # xywh to xyxy   [4, 25200]

    return xyxy, classes, scores  # output is boxes(x,y,x,y), classes(int), scores(float) [predictions length]

To get the output data:
"""Output data"""
output_data = interpreter.get_tensor(output_details[0]['index'])  # get tensor  x(1, 25200, 7)
xyxy, classes, scores = YOLOdetect(output_data) #boxes(x,y,x,y), classes(int), scores(float) [25200]

And for the boxes:
for i in range(len(scores)):
    if ((scores[i] > 0.1) and (scores[i] <= 1.0)):
        H = frame.shape[0]
        W = frame.shape[1]
        xmin = int(max(1,(xyxy[0][i] * W)))
        ymin = int(max(1,(xyxy[1][i] * H)))
        xmax = int(min(H,(xyxy[2][i] * W)))
        ymax = int(min(W,(xyxy[3][i] * H)))

        cv2.rectangle(frame, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)
        ...


Instruction: torch row wise cosinus similarity

Input:
Assuming I am having 2 tensors with dimension:

tensor A: [512, 100]
tensor B: [512, 200, 100]

and I would like to use torch.cosine_similarity to produce a tensor C of dimensions [512, 1, 200] or [512, 200].
Using torch.cosine_similarity(A, B) I get an error:
'{RuntimeError}The size of tensor a (512) must match the size of tensor b (200) at non-singleton dimension 1'
I guess it can be done by the following:
desired_result = torch.stack([torch.cosine_similarity(A_row, B_row, axis=-1) for B_row, A_row in zip(B, A)])

but there should be a more optimized way. Any help/hint?

Output:
No need for stacking, broadcasting will do the job for you:
# (512, 100, 1) after unsqueeze
A = torch.randn(512, 100).unsqueeze(dim=-1)
B = torch.randn(512, 200, 100)

# (512, 200)
torch.cosine_similarity(B, A.unsqueeze(1), axis=-1)


Instruction: Is there a `Split` equivalent to torch.nn.Sequential?

Input:
A sample code for a Sequential block is
self._encoder = nn.Sequential(
        # 1, 28, 28
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=3, padding=1),
        # 32, 10, 10 = 16, (1//3)(28 + 2 * 1 - 3) + 1, (1//3)(28 + 2*1 - 3) + 1
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=2),
        # 32, 5, 5
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=2, padding=1),
        # 64, 3, 3
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=1),
        # 64, 2, 2
)

Is there some construct like nn.Sequential that puts modules in it in parallel?
I would like to now define something like
self._mean_logvar_layers = nn.Parallel(
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
)

Whose output should be two pipes of data - one for each element in self._mean_logvar_layers which are then feedable to the rest of the network. Kind of like a multi-headed network.

My current implementation:
self._mean_layer = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0)
self._logvar_layer = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0)

and
def _encode(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
    for i, layer in enumerate(self._encoder):
        x = layer(x)

    mean_output = self._mean_layer(x)
    logvar_output = self._logvar_layer(x)

    return mean_output, logvar_output

I would like to treat the parallel construct as a layer.
Is that doable in PyTorch?

Output:
Sequential split
What you can do is create a Parallel module (though I would name it differently as it implies this code actually runs in parallel, probably Split would be a good name) like this:
class Parallel(torch.nn.Module):
    def __init__(self, *modules: torch.nn.Module):
        super().__init__()
        self.modules = modules

    def forward(self, inputs):
        return [module(inputs) for module in self.modules]

Now you can define it as you wanted:
self._mean_logvar_layers = Parallel(
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=1, padding=0),
)

And use it like this:
mean, logvar = self._mean_logvar_layers(x)

One layer and split
As suggested by @xdurch0 we could use a single layers and split across channels instead, using this module:
class Split(torch.nn.Module):
    def __init__(self, module, parts: int, dim=1):
        super().__init__()
        self.parts
        self.dim = dim
        self.module = module

    def forward(self, inputs):
        output = self.module(inputs)
        chunk_size = output.shape[self.dim] // self.parts
        return torch.split(output, chunk_size, dim=self.dim)

This inside your neural network (notice 128 channels, those will be split into 2 parts, each of size 64):
self._mean_logvar_layers = Split(
    nn.Conv2d(in_channels=64, out_channels=128, kernel_size=2, stride=1, padding=0),
    parts=2,
)

And use it like previously:
mean, logvar = self._mean_logvar_layers(x)

Why this approach?
Everything will be done in one swoop instead of sequentially, hence faster, but might be too wide if you don't have enough GPU memory.
Can it work with Sequential?
Yes, it is still a layer. But next layer has to work with tuple(torch.Tensor, torch.Tensor) as inputs.
Sequential is also a layer, quite simple one, let's see forward:
def forward(self, inp):
    for module in self:
        inp = module(inp)
    return inp

It just passes output from previous model to the next and that's it.


Instruction: How to load a trained model to inference the predicted data

Input:
I trained and saved CNN model for 1000 epochs and now want to retrieve validation data (predicted images). In code below test_pred and test_real outputs predicted and real images in validation set. Should I load and run the saved model for another 1 epoch to retrieve predicted images (this will end in CUDA out of memory since data are huge)? Or there are other ways? You can see part of my code below:
for epoch in range(epochs):
    mse_train_losses= []
    mae_train_losses = []
    N_train = []
    mse_val_losses = []
    mae_val_losses = []
    N_test = []

    if save_model:
        if epoch % 50 ==0:
            checkpoint = {'state_dict' : model.state_dict(),'optimizer' : optimizer.state_dict()}
            save_checkpoint(checkpoint)

    model.train()
    for data in train_loader:

        x_train_batch, y_train_batch = data[0].to(device,
            dtype=torch.float), data[1].to(device, dtype=torch.float)
        y_train_pred = model(x_train_batch)            # 1) Forward pass
        mse_train_loss = criterion(y_train_batch, y_train_pred, x_train_batch, mse)
        mae_train_loss = criterion(y_train_batch, y_train_pred, x_train_batch, l1loss)

        optimizer.zero_grad()
        mse_train_loss.backward()
        optimizer.step()

        mse_train_losses.append(mse_train_loss.item())
        mae_train_losses.append(mae_train_loss.item())
        N_train.append(len(x_train_batch))

    test_pred=[]
    test_real=[]
    model.eval()
    with torch.no_grad():
        for data in test_loader:
            x_test_batch, y_test_batch = data[0].to(device,
                dtype=torch.float), data[1].to(device, dtype=torch.float)

            y_test_pred = model(x_test_batch)
            mse_val_loss = criterion(y_test_batch, y_test_pred, x_test_batch, mse)
            mae_val_loss = criterion(y_test_batch, y_test_pred, x_test_batch, l1loss)

            mse_val_losses.append(mse_val_loss.item())
            mae_val_losses.append(mae_val_loss.item())
            N_test.append(len(x_test_batch))

            test_pred.append(y_test_pred)
            test_real.append(y_test_batch)

Output:
When you append it to the list try using .cpu() at the end like this:
test_pred.append(t_test_pred.cpu())


Instruction: How to make R2 score in nn.LSTM pytorch

Input:
I tried to make loss function with R2in nn.LSTM but i couldnt find any documentation about it . I already use RMSE and MAE loss from pytorch.
My data is a time series and im doing time series forecasting
This is the code where i use the loss function of RMSE in data training
model = LSTM_model(input_size=1, output_size=1, hidden_size=512, num_layers=2, dropout=0).to(device)
criterion = nn.MSELoss(reduction="sum")
optimizer = optim.Adam(model.parameters(), lr=0.001)
callback = Callback(model, early_stop_patience=10 ,outdir="model/lstm", plot_every=20,)

from tqdm.auto import tqdm

def loop_fn(mode, dataset, dataloader, model, criterion, optimizer,device):
    if mode =="train":
        model.train()
    elif mode =="test":
        model.eval()
    cost = 0
    for feature, target in tqdm(dataloader, desc=mode.title()):
        feature, target = feature.to(device), target.to(device)
        output , hidden = model(feature,None)
        loss = torch.sqrt(criterion(output,target))

        if mode =="train":
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        cost += loss.item() * feature.shape[0]
    cost = cost / len(dataset)
    return cost

And this is the code to start data training
while True :
    train_cost = loop_fn("train", train_set, trainloader, model, criterion, optimizer,device)
    with torch.no_grad():
        test_cost  = loop_fn("test", test_set, testloader, model, criterion, optimizer,device)

    callback.log(train_cost, test_cost)

    callback.save_checkpoint()

    callback.cost_runtime_plotting()

    if callback.early_stopping(model, monitor="test_cost"):
        callback.plot_cost()
        break

Can anyone help me with the R2 loss function ? Thank you in advance

Output:
Here is an implemention,
"""
From https://en.wikipedia.org/wiki/Coefficient_of_determination
"""
def r2_loss(output, target):
    target_mean = torch.mean(target)
    ss_tot = torch.sum((target - target_mean) ** 2)
    ss_res = torch.sum((target - output) ** 2)
    r2 = 1 - ss_res / ss_tot
    return r2

You can use it as below,
loss = r2_loss(output, target)
loss.backward()


Instruction: Final step of PyTorch Gradient Accumulation for small datasets

Input:
I am training a BERT model on a relatively small dataset and cannot afford to lose any labelled sample as they must all be used for training. Due to GPU memory constraints, I am using gradient accumulation to train on larger batches (e.g. 32). According to PyTorch documentation, gradient accumulation is implemented as follows:
scaler = GradScaler()

for epoch in epochs:
    for i, (input, target) in enumerate(data):
        with autocast():
            output = model(input)
            loss = loss_fn(output, target)
            loss = loss / iters_to_accumulate

        # Accumulates scaled gradients.
        scaler.scale(loss).backward()

        if (i + 1) % iters_to_accumulate == 0:
            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)

            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()

However, if you are using e.g. 110 training samples, with batch size 8 and accumulation step 4 (i.e. effective batch size 32), this method would only train the first 96 samples (i.e. 32 x 3), i.e. wasting 14 samples. In order to avoid this, I'd like to modify the code as follows (notice change to the final if statement):
scaler = GradScaler()

for epoch in epochs:
    for i, (input, target) in enumerate(data):
        with autocast():
            output = model(input)
            loss = loss_fn(output, target)
            loss = loss / iters_to_accumulate

        # Accumulates scaled gradients.
        scaler.scale(loss).backward()

        if (i + 1) % iters_to_accumulate == 0 or (i + 1) == len(data):
            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)

            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()

Is this correct and really that simple, or will this have any side effects? It seems very simple to me, but I've never seen it done before. Any help appreciated!

Output:
As Lucas Ramos already mentioned, when using DataLoader where the underlying dataset's size is not divisible by the batch size, the default behavior is to have a smaller last batch:

drop_last (bool, optional) – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default: False)

Your plan is basically implementing gradient accumulation combined with drop_last=False - that is having the last batch smaller than all others.
Therefore, in principle there's nothing wrong with training with varying batch sizes.
However, there is something you need to fix in your code:
The loss is averaged over the mini-batch. So, if you process mini batches in the usual way you do not need to worry about it. However, when accumulating gradients you do it explicitly by dividing the loss by iters_to_accumulate:
loss = loss / iters_to_accumulate

In the last mini batch (with smaller size) you need to change the value of iter_to_accumulate to reflect this smaller minibatch size!
I proposed this revised code, breaking the training loop into two: an outer loop on mini-batches, and an inner one that accumulates gradients per mini batch. Note how using an iter over the DataLoader helps breaking the training loop into two:
scaler = GradScaler()

for epoch in epochs:
    bi = 0  # index batches
    # outer loop over minibatches
    data_iter = iter(data)
    while bi < len(data):
        # determine the range for this batch
        nbi = min(len(data), bi + iters_to_accumulate)
        # inner loop over the items of the mini batch - accumulating gradients
        for i in range(bi, nbi):
            input, target = data_iter.next()
            with autocast():
                output = model(input)
                loss = loss_fn(output, target)
                loss = loss / (nbi - bi)  # divide by the true batch size

            # Accumulates scaled gradients.
            scaler.scale(loss).backward()
        # done mini batch loop - gradients were accumulated, we can make an optimizatino step.

        # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad()
        bi = nbi