Instruction: How to efficiently repeat tensor element variable of time in pytorch?

Input:
For example, if I have a tensor A = [[1,1,1], [2,2,2], [3,3,3]], and B = [1,2,3]. How do I get C = [[1,1,1], [2,2,2], [2,2,2], [3,3,3], [3,3,3], [3,3,3]], and doing this batch-wise?
My current element-wise solution btw (takes forever...):
        def get_char_context(valid_embeds, words_lens):
            chars_contexts = []
            for ve, wl in zip(valid_embeds, words_lens):
                for idx, (e, l) in enumerate(zip(ve, wl)):
                    if idx ==0:
                        chars_context = e.view(1,-1).repeat(l, 1)
                    else:
                        chars_context = torch.cat((chars_context, e.view(1,-1).repeat(l, 1)),0)
                chars_contexts.append(chars_context)
            return chars_contexts

I'm doing this to add bert word embedding to a char level seq2seq task...

Output:
Use this:
import torch
# A is your tensor
B = torch.tensor([1, 2, 3])
C = A.repeat_interleave(B, dim = 0)

EDIT:
The above works fine if A is a single 2D tensor. To repeat all (2D) tensors in a batch in the same manner, this is a simple workaround:
A = torch.tensor([[[1, 1, 1], [2, 2, 2], [3, 3, 3]],
    [[1, 2, 3], [4, 5, 6], [2,2,2]]]) # A has 2 tensors each of shape (3, 3)
B = torch.tensor([1, 2, 3]) # Rep. of each row of every tensor in the batch

A1 = A.reshape(1, -1, A.shape[2]).squeeze()
B1 = B.repeat(A.shape[0])
C = A1.repeat_interleave(B1, dim = 0).reshape(A.shape[0], -1, A.shape[2])

C is:
tensor([[[1, 1, 1],
         [2, 2, 2],
         [2, 2, 2],
         [3, 3, 3],
         [3, 3, 3],
         [3, 3, 3]],

        [[1, 2, 3],
         [4, 5, 6],
         [4, 5, 6],
         [2, 2, 2],
         [2, 2, 2],
         [2, 2, 2]]])

As you can see each inside tensor in the batch is repeated in the same manner.


Instruction: How does the BERT model select the label ordering?

Input: I'm training BertForSequenceClassification for a classification task. My dataset consists of 'contains adverse effect' (1) and 'does not contain adverse effect' (0). The dataset contains all of the 1s and then the 0s after (the data isn't shuffled). For training I've shuffled my data and get the logits. From what I've understood, the logits are the probability distributions before softmax. An example logit is [-4.673831, 4.7095485]. Does the first value correspond to the label 1 (contains AE) because it appears first in the dataset, or label 0. Any help would be appreciated thanks.

Output:
The first value corresponds to label 0 and the second value corresponds to label 1. What BertForSequenceClassification does is feeding the output of the pooler to a linear layer (after a dropout which I will ignore in this answer). Let's look at the following example:
from torch import nn
from transformers import BertModel, BertTokenizer
t = BertTokenizer.from_pretrained('bert-base-uncased')
m = BertModel.from_pretrained('bert-base-uncased')
i = t.encode_plus('This is an example.', return_tensors='pt')
o = m(**i)
print(o.pooler_output.shape)

Output:
torch.Size([1, 768])

The pooled_output is a tensor of shape [batch_size,hidden_size] and represents the contextualized (i.e. attention was applied) [CLS] token of your input sequences. This tensor is feed to a linear layer to calculate the logits of your sequence:
classificationLayer = nn.Linear(768,2)
logits = classificationLayer(o.pooler_output)

When we normalize these logits we can see that the linear layer predicts that our input should belong to label 1:
print(nn.functional.softmax(logits,dim=-1))

Output (will differ since the linear layer is initialed randomly):
tensor([[0.1679, 0.8321]], grad_fn=<SoftmaxBackward>)

The linear layer applies a linear transformation: y=xA^T+b and you can already see that the linear layer is not aware of your labels. It 'only' has a weights matrix of size [2,768] to produce logits of size [1,2] (i.e.: first row corresponds to the first value and second row to the second):
import torch:

logitsOwnCalculation = torch.matmul(o.pooler_output,  classificationLayer.weight.transpose(0,1))+classificationLayer.bias
print(nn.functional.softmax(logitsOwnCalculation,dim=-1))

Output:
tensor([[0.1679, 0.8321]], grad_fn=<SoftmaxBackward>)

The BertForSequenceClassification model learns by applying a CrossEntropyLoss. This loss function produces a small loss when the logits for a certain class (label in your case) deviate only slightly from the expectation. That means the CrossEntropyLoss is the one that lets your model learn that the first logit should be high when the input does not contain adverse effect or small when it contains adverse effect. You can check this for our example with the following:
loss_fct = nn.CrossEntropyLoss()
label0 = torch.tensor([0]) #does not contain adverse effect
label1 = torch.tensor([1]) #contains adverse effect
print(loss_fct(logits, label0))
print(loss_fct(logits, label1))

Output:
tensor(1.7845, grad_fn=<NllLossBackward>)
tensor(0.1838, grad_fn=<NllLossBackward>)


Instruction: Pytorch - Custom Dataset out of Range

Input:
im new to Pytorch and DL in genereal, so i hope this is the right place to ask questions like this.
I wanted to create my first Dataset, but my Dataset always runs out of bound. This problem should be easiest to show with the codes and outputs.
class DataframeDataset(torch.utils.data.Dataset):
    """Load Pytorch Dataset from Dataframe

    """

    def __init__(self, data_frame, input_key, target_key, transform=None, features=None):
        self.data_frame = data_frame
        self.input_key = input_key
        self.target_key = target_key
        self.inputs = self.data_frame[input_key]
        self.targets = self.data_frame[target_key]
        self.transform = transform
        self.features = [input_key, target_key] if features is None else features
        self.len = len(self.inputs)

    def __len__(self):
        return self.len

    def __str__(self):
        return str(self.info())

    def info(self):
        info = {
            'features': self.features,
            'num_rows': len(self)
        }
        return "Dataset("+ str(info) + ")"

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        data = {
            self.input_key: self.inputs[idx],
            self.target_key: self.targets[idx]
        }

        if self.transform:
            return self.transform(data)

        return data

    @staticmethod
    def collate_fn(input_key, output_key):
        def __call__(batch):
            speeches = [data[input_key] for data in batch]
            sentences = [data[output_key] for data in batch]
            return speeches, sentences

        return __call__

with some mook data:
    data = [("x1", "y2", "A3"), ("x1", "y2", "b3"), ("x1", "y2", "c3"), ("x1", "y2", "d3")]
    df = pd.DataFrame(data, columns=['input', 'target', 'random'])
    print(df.head())

  input target random
0    x1     y2     A3
1    x1     y2     b3
2    x1     y2     c3
3    x1     y2     d3

    ds = DataframeDataset(data_frame=df, input_key="input", target_key="target", transform=None)
    print("Len:", len(ds))
    print("Ds", ds)
    print(ds[0])

Len: 4
Ds Dataset({'features': ['input', 'target'], 'num_rows': 4})
{'input': 'x1', 'target': 'y2'}

So the basic functions seem to work. However, if I want to iterate over the data with a foreach loop, then unfortunately the loop does not know the boundaries. So I get a key-error, because the torch accesses indicies outside the boundary.
    for idx, data in enumerate(ds):
        print(idx,"->",data)

0 -> {'input': 'x1', 'target': 'y2'}
1 -> {'input': 'x1', 'target': 'y2'}
2 -> {'input': 'x1', 'target': 'y2'}
3 -> {'input': 'x1', 'target': 'y2'}

Traceback (most recent call last):
  File "/home/warmachine/.local/lib/python3.8/site-packages/pandas/core/indexes/range.py", line 351, in get_loc
    return self._range.index(new_key)
ValueError: 4 is not in range`

If i do something like
    for idx in range(0, len(ds)):
        data = ds[idx]
        print(idx, "->", data)

it works, but i need to be able to use the for-each Style, so that i can use this Dataset within the Trainer of Hugging Face.
Ty in advcanded

Output:
If you want to use Foreach loops, you must implement an Iterator function. Here is an example from PyTorch:
https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset
Slightly modified, works for me.
class DataframeDataset(torch.utils.data.Dataset):
...
    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:
            return map(self.__getitem__, range(self.__len__()))

        per_worker = int(math.ceil((self.__len__()) / float(worker_info.num_workers)))
        worker_id = worker_info.id
        iter_start = worker_id * per_worker
        iter_end = min(iter_start + per_worker, self.__len__())
        return map(self.__getitem__, range(iter_start, iter_end))


Instruction: Remove rows from tensor by matching

Input:
I'm trying to do some operation like if there is tensor in pytorch
a = torch.tensor([[1,0]
                  ,[0,1]
                  ,[2,0]
                  ,[3,2]])

b = torch.tensor([[0,1]
                  ,[2,0]])

I want to remove the rows [0,1], [2,0] which are the rows of b from a.
Is there any way to do this?
# result
a = torch.tensor([[1,0]
                  ,[3,2]])

Output:
You could do it if the tensor shapes were broadcastable.
For a tensor a of shape (?, d) and a tensor b of shape (d,), you could write something like:
cmp = a.eq(b).all(dim=1).logical_not(), i.e. compare each d-dimensional row of a with b and give me the indices where the comparison is False.
From these you can then easily your new tensor like so:
a = a[cmp]
I doubt you'll find an elegant way of doing this when b itself contains a batch dimension; your best bet would be to write a for loop.
Full example:
>>> xs = torch.tensor([[1,0], [0,1], [2,0], [3,2]])
>>> ys = torch.tensor([[0,1],[2,0]])
>>> for y in ys:
...     xs = xs[xs.eq(y).all(dim=1).logical_not()]
>>> xs
tensor([[1, 0],
        [3, 2]])


Instruction: How to get multiple lines exported to wandb

Input:
I am using the library weights and  biases. My model outputs a curve (a time series). I'd like to see how this curve changes throughout training. So, I'd need some kind of slider where I can select epoch and it shows me the curve for that epoch. It could be something very similar to what it's done with histograms (it shows an image of the histograms across epochs and when you hover it display the histogram corresponding to that epoch). Is there a way to do this or something similar using wandb?
Currently my code looks like this:
for epoch in range(epochs):
   output = model(input)
   #output is shape (37,40) (lenght 40 and I have 37 samples)
   #it's enough to plot the first sample
   xs = torch.arange(40).unsqueeze(dim=1)
   ys = output[0,:].unsqueeze(dim=1)
   wandb.log({"line": wandb.plot.line_series(xs=xs, ys=ys,title="Out")}, step=epoch)

I'd appreciate any help! Thanks!

Output:
You can use wandb.log() with matplotlib. Create your plot using matplotlib:
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0, 50)
for i in range(1, 4):
    fig, ax = plt.subplots()
    y = x ** i
    ax.plot(x, y)
    wandb.log({'chart': ax})

Then when you look on your wandb dashboard for the run, you will see the plot rendered as plotly plot. Click the gear in the upper left hand corner to see a slider that lets you slide over training steps and see the plot at each step.


Instruction: How to change PyTorch sigmoid function to be steeper

Input:
My model works when I use torch.sigmoid. I tried to make the sigmoid steeper by creating a new sigmoid function:
def sigmoid(x):
    return 1 / (1 + torch.exp(-1e5*x))

But for some reason the gradient doesn't flow through it (I get NaN). Is there a problem in my function, or is there a way to simply change the PyTorch implementation to be steeper (as my function)?
Code example:
def sigmoid(x):
  return 1 / (1 + torch.exp(-1e5*x))

a = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.58, requires_grad=True)

c = sigmoid(a-b)
c.backward()
a.grad
>>> tensor(nan)

Output:
The issue seems to be that when the input to your sigmoid implementation is negative, the argument to torch.exp becomes very large, causing an overflow. Using torch.autograd.set_detect_anomaly(True) as suggested here, you can see the error:
RuntimeError: Function 'ExpBackward' returned nan values in its 0th output.

If you really need to use the function you have defined, a  possible workaround could be to put a conditional check on the argument (but I am not sure if it would be stable, so I cannot comment on its usefulness):
def sigmoid(x):
    if x >= 0:
        return 1./(1+torch.exp(-1e5*x))
    else:
        return torch.exp(1e5*x)/(1+torch.exp(1e5*x))

Here, the expression in the else branch is equivalent to the original function, by multiplying the numerator and denominator by torch.exp(1e5*x). This ensures that the argument to torch.exp is always negative or close to zero.
As noted by trialNerror, the exponent value is so high that except for values extremely close to zero, your gradient will evaluate to zero everywhere else since the actual slope will be extremely small and cannot be resolved by the data type. So if you plan to use it in a network you will likely find it very difficult to learn anything since gradients will almost always be zero. It might be better to select a smaller exponent, depending on your use case.


Instruction: PyTorch loads old data when using tensorboard

Input: In using tensorboard I have cleared my data directory and trained a new model but I am seeing images from an old model. Why is tensorboard loading old data, where is it being stored, and how do I remove it?

Output:
Tensorboard was built to have caches in case long training fails you have "bak"-like files that your board will generate visualizations from. Unfortunately, there is not a good practice to manually remove hidden temp files as they are not seen from displaying files including ones with the . (dot) prefix using bash. This memory is self-managed. For best practices, (1) have your tensorboard name be dynamic for results of each run: this can be done using datetime library in combination with an f-string in python so that the name of each run is separated by a time stamp. (This command be done right from python, say a jupyter notebook, if you import the subprocess package and run your bash command straight from the script.) (2) Additionally, you are strongly advised to save your logdir (log directory) separately from where you are running the code. These two practices together should solve all the problems related to tmp files erroneously populating new results.
How to "reset" tensorboard data after killing tensorflow instance


Instruction: Deep learning model test accuracy unstable

Input:
I am trying to train and test a pytorch GCN model that is supposed to identify person. But the test accuracy is quite jumpy like it gives 49% at 23 epoch then goes below near 45% at 41 epoch. So it's not increasing all the time though loss seems to decrease at every epoch.
My question is not about implementation errors rather I want to know why this happens. I don't think there is something wrong in my coding as I saw SOTA architecture has this type of behavior as well. The author just picked the best result and published saying that their models gives that result.
Is it normal for the accuracy to be jumpy (up-down) and am I just to take the best ever weights that produce that?

Output:
Accuracy is naturally more "jumpy", as you put it. In terms of accuracy, you have a discrete outcome for each sample - you either get it right or wrong. This makes it so that the result fluctuate, especially if you have a relatively low number of samples (as you have a higher sampling variance).
On the other hand, the loss function should vary more smoothly. It is based on the probabilities for each class calculated at your softmax layer, which means that they vary continuously. With a small enough learning rate, the loss function should vary monotonically. Any bumps you see are due to the optimization algorithm taking discrete steps, with the assumption that the loss function is roughly linear in the vicinity of the current point.


Instruction: How to load a pre-trained PyTorch model?

Input:
I'm following this guide on saving and loading checkpoints. However, something is not right. My model would train and the parameters would correctly update during the training phase. However, there seem to be a problem when I load the checkpoints. That is, the parameters are not being updated anymore.
My model:
import torch
import torch.nn as nn
import torch.optim as optim

PATH = 'test.pt'

class model(nn.Module):
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            #print(self.a, self.b, self.c)

        def load(self):
          try:
            checkpoint = torch.load(PATH)
            print('\nloading pre-trained model...')
            self.a = checkpoint['a']
            self.b = checkpoint['b']
            self.c = checkpoint['c']
            optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
            print(self.a, self.b, self.c)
          except: #file doesn't exist yet
            pass

        @property
        def b_opt(self):
            return torch.tanh(self.b)*2

        def train(self):
          print('training...')
          for epoch in range(3):
            print(self.a, self.b, self.c)
            for r in range(5):
              optimizer.zero_grad()
              loss = torch.square(5 * (r > 2) * (3) - model_net.a * torch.sigmoid((r - model_net.b)) * (model_net.c))
              loss.backward(retain_graph=True) #accumulate gradients

            #checkpoint save
            torch.save({
                'model': model_net.state_dict(),
                'a': model_net.a,
                'b': model_net.b,
                'c': model_net.c,
                'optimizer_state_dict': optimizer.state_dict(),
                }, PATH)

            optimizer.step()

model_net = model()
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

print(model_net.a)
print(model_net.b)
print(model_net.c)

This prints
Parameter containing:
tensor([0.4214], requires_grad=True)
Parameter containing:
tensor([0.3862], requires_grad=True)
Parameter containing:
tensor([0.8812], requires_grad=True)

I then run model_net.train() to see that the parameters are being updated and this outputs:
training...
Parameter containing:
tensor([0.9990], requires_grad=True) Parameter containing:
tensor([0.1580], requires_grad=True) Parameter containing:
tensor([0.1517], requires_grad=True)
Parameter containing:
tensor([1.0990], requires_grad=True) Parameter containing:
tensor([0.0580], requires_grad=True) Parameter containing:
tensor([0.2517], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

Running model_net.load() outputs:
loading pre-trained model...
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

And lastly, running model_net.train() again outputs:
training...
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

Update 1.
Following @jhso suggestion I changed my load to:
def load(self):
  try:
    checkpoint = torch.load(PATH)
    print('\nloading pre-trained model...')
    self.load_state_dict(checkpoint['model'])
    self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    print(self.a, self.b, self.c)
  except: #file doesn't exist yet
    pass

This almost seems to work (the network is training now), but I don't think the optimizer is loading correctly. That is because it doesn't go pass the line self.optimizer.load_state_dict(checkpoint['optimizer_state_dict']).
You can see that since it doesn't print(self.a, self.b, self.c) when I run
model_net.load()

Output:
The way you are loading your data is not the recommended way to load your parameters because you're overwriting the graph connections (or something along those lines...). You even save the model state_dict, so why not use it!
I changed the load function to:
def load(self):
      try:
        checkpoint = torch.load(PATH)
        print('\nloading pre-trained model...')
        self.load_state_dict(checkpoint['model'])
        self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        print(self.a, self.b, self.c)
        self.train()
      except: #file doesn't exist yet
        pass

But note to do this, you have to add your optimizer to your model:
model_net = model()
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)
model_net.optimizer = optimizer

Which then gave the output (running train, load, train):
Parameter containing:
tensor([0.2316], requires_grad=True) Parameter containing:
tensor([0.4561], requires_grad=True) Parameter containing:
tensor([0.8626], requires_grad=True)
Parameter containing:
tensor([0.3316], requires_grad=True) Parameter containing:
tensor([0.3561], requires_grad=True) Parameter containing:
tensor([0.9626], requires_grad=True)
Parameter containing:
tensor([0.4317], requires_grad=True) Parameter containing:
tensor([0.2568], requires_grad=True) Parameter containing:
tensor([1.0620], requires_grad=True)

loading pre-trained model...
Parameter containing:
tensor([0.4317], requires_grad=True) Parameter containing:
tensor([0.2568], requires_grad=True) Parameter containing:
tensor([1.0620], requires_grad=True)
training...
Parameter containing:
tensor([0.4317], requires_grad=True) Parameter containing:
tensor([0.2568], requires_grad=True) Parameter containing:
tensor([1.0620], requires_grad=True)
Parameter containing:
tensor([0.5321], requires_grad=True) Parameter containing:
tensor([0.1577], requires_grad=True) Parameter containing:
tensor([1.1612], requires_grad=True)
Parameter containing:
tensor([0.6328], requires_grad=True) Parameter containing:
tensor([0.0583], requires_grad=True) Parameter containing:
tensor([1.2606], requires_grad=True)


Instruction: Semantic Segmentation metric - making confusion matrix

Input:
I want to get meanIoU and pixel Accuracy, so I customize get metric function from Deeplabv3+ github Evaluator class: https://github.com/jfzhang95/pytorch-deeplab-xception.
Currently I am having difficulty writing the confusion matrix of the two images.
Here is my first code block:
for i, row in df.iterrows():    # dataframe have filenames of ground-truth and predicted image
    print(" --- Iterrows start --- ")
    print("Target Image: \n{}\n{}\n".format(row['Yt'], row['Yp']))
    # gt image : aaa.png | target(predicted image) : segmented_aaa.jpg

    # Read gt segmap imagefile
    gt_image = Image.open(row['Yt']).convert('L')
    gt_image = np.array(gt_image)
    gt_image = cv2.resize(gt_image, (513, 513))

    # Read target(predicted) segmap imagefile
    target_image = Image.open(row['Yp']).convert('L')
    target_image = np.array(target_image)
    target_image = cv2.resize(target_image, (513, 513))

    print(gt_image.shape)        # (513, 513)
    print(target_image.shape)    # (513, 513)
    print("gt_image:", "\n", gt_image)
    print("target_image:", "\n", target_image)
    print("gt_image type:", "\n", type(gt_image))
    print("target_image type:", "\n", type(target_image))
    print("gt_image unique:", "\n", np.unique(gt_image))
    print("target_image unique:", "\n", np.unique(target_image))
    print(gt_image.shape, target_image.shape)

    # mIoU = get_iou(target_image, gt_image, num_classes)
    # print(mIoU)

    evaluator.add_batch(gt_image, target_image)
    print(" --- Iterrows end --- ")

Result of first code block in console log:
(513, 513)
(513, 513)
gt_image:
 [[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
target_image:
 [[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
gt_image type:
 <class 'numpy.ndarray'>
target_image type:
 <class 'numpy.ndarray'>
gt_image unique:
 [  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17
  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35
  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53
  54  55  56  57  59  60  61  62  63  64  65  66  67  68  69  70  71  72
  73  74  75  77  78  79  80  81  83  84  85  86  87  88  89  90  91  92
  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107 108 109 110
 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146
 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164
 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218
 219 220]
target_image unique:
 [  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17
  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35
  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53
  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71
  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89
  90  91  92  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107
 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 160]

and Evaluator(second) code block is:
    def _generate_matrix(self, gt_image, pre_image):
        print(np.unique(gt_image))
        mask = (gt_image >= 0) & (gt_image < self.num_class)
        print("\nMASK INFO")
        print(mask, mask.shape)
        print(np.unique(mask))
        print()

        print("\nGT_IMAGE INFO")
        print(gt_image[mask])
        print(np.unique(gt_image[mask]))
        print(gt_image[mask].astype('int'))
        print(set(gt_image[mask].astype('int').tolist()))
        print(sorted(set(self.num_class * gt_image[mask].astype('int'))))
        print(pre_image[mask], set(pre_image[mask].tolist()))
        print()

        label = self.num_class * gt_image[mask].astype('int') + pre_image[mask]
        # label[10] = 449
        print("\nLABEL INFO")
        print(label, label.shape, len(label.tolist()))
        print(set(label.tolist()))
        print(len(set(label.tolist())))
        print()

        count = np.bincount(label, minlength=self.num_class**2)
        print("\nCOUNT INFO")
        print(count.shape)
        print()

        # !!!!!!!!!! ERROR POINT !!!!!!!!!!
        confusion_matrix = count.reshape(self.num_class, self.num_class)
        return confusion_matrix

    def add_batch(self, gt_image, pre_image):
        if gt_image.shape != pre_image.shape:
            print("GT_Image's shape is different with PRE_IMAGE's shape!")
            exit(0)
        # assert gt_image.shape == pre_image.shape
        self.confusion_matrix += self._generate_matrix(gt_image, pre_image)

Result of second code block in console log:
(513, 513) (513, 513)
[  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17
  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35
  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53
  54  55  56  57  59  60  61  62  63  64  65  66  67  68  69  70  71  72
  73  74  75  77  78  79  80  81  83  84  85  86  87  88  89  90  91  92
  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107 108 109 110
 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146
 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164
 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218
 219 220]

MASK INFO
[[ True  True  True ...  True  True  True]
 [ True  True  True ...  True  True  True]
 [ True  True  True ...  True  True  True]
 ...
 [ True  True  True ...  True  True  True]
 [ True  True  True ...  True  True  True]
 [ True  True  True ...  True  True  True]] (513, 513)

GT_IMAGE INFO
[0 0 0 ... 0 0 0]
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}
[0, 21, 42, 63, 84, 105, 126, 147, 168, 189, 210, 231, 252, 273, 294, 315, 336, 357, 378, 399, 420]
[0 0 0 ... 0 0 0] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 142, 145, 146}

LABEL INFO
[0 0 0 ... 0 0 0] (175389,) 175389
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 142, 145, 146, 147, 148, 153, 157, 162, 168, 173, 189, 210, 218, 231, 240, 252, 253, 254, 273, 277, 294, 295, 315, 317, 336, 337, 357, 361, 364, 377, 378, 384, 395, 399, 407, 410, 420, 421, 424, 425, 432, 496}
183

COUNT INFO
(497,)

cannot reshape array of size 497 into shape (21,21)

In my opinion, if there is a value like 480 that is greater than 21*21(=441), it will cause an error.
21 is the number of classes in the Pascal VOC dataset.
The ground-truth and predicted image used as input is this image:
 [ground-truth image]
 [predicted image]

Output:
Important is this: Save the inference image in PNG format.
In my case, metric's code block was fine, but it failed because I saved it with jpeg format. I saved images in png format and my code worked fine.


Instruction: Add 2-D tensor as column of dataframe

Input:
My dataframe looks like
   INCIDENT_NUMBER
0  INC000030884498
1  INC000029956111
2  INC000029555353
3  INC000029555338

I also have a 2-D tensor for the above four incidents like
  sample_concatenated_embedding=
  tensor(
  [[ 0.6993, -0.1427, -0.1532,  ...,  0.8386,  0.5151,  0.8906],
  [ 0.7382, -0.8497,  0.1363,  ...,  0.8054,  0.5432,  0.9082],
  [ 0.0835, -0.2431, -0.0815,  ...,  0.8025,  0.5217,  0.9041],
  [-0.0346, -0.2396, -0.5831,  ...,  0.7591,  0.6138,  0.9649]],
  grad_fn=<ViewBackward>)

The embeddings are of size [4, 161280]
I want to insert the tensor in the four consecutive lines of my Dataframe
The end Dataframe should look like
   INCIDENT_NUMBER      embedding
0  INC000030884498      [ 0.6993, -0.1427, -0.1532,  ...,  0.8386,  0.5151,  0.8906]
1  INC000029956111      [ 0.7382, -0.8497,  0.1363,  ...,  0.8054,  0.5432,  0.9082]
2  INC000029555353      [ 0.0835, -0.2431, -0.0815,  ...,  0.8025,  0.5217,  0.9041]
3  INC000029555338      [-0.0346, -0.2396, -0.5831,  ...,  0.7591,  0.6138,  0.9649]

If the tensors would have been series, i could have simply used the below command
 my_dataframe['embedding'] = sample_concatenated_embedding

I could use a for loop and easily insert into dataframe like
 empty_dataframe = pd.DataFrame(columns=['incident','embedding'])

 for item in range(0,4):
     INCIDENT_NUMBER = my_dataframe['INCIDENT_NUMBER'].iloc[item]
     temp_df = pd.DataFrame([[INCIDENT_NUMBER, sample_concatenated_embedding[item]], columns=['incident','embedding'])
     frames = [empty_dataframe, temp_df]
     empty_dataframe = pd.concat(frames)

But for loop would be inefficent. Is there any shorter way to achieve the end goal

Output:
If the value index of INCIDENT_NUMBER and value index of sample_concatenated_embedding matches. You can just convert sample_concatenated_embedding to list then assign it to the new column like
import pandas as pd

df = pd.DataFrame({'INCIDENT_NUMBER': ['INC000030884498', 'INC000029956111', 'INC000029555353', 'INC000029555338']})

data = [[ 0.6993, -0.1427, -0.1532, 0.8386,  0.5151,  0.8906],
        [ 0.7382, -0.8497,  0.1363, 0.8054,  0.5432,  0.9082],
        [ 0.0835, -0.2431, -0.0815, 0.8025,  0.5217,  0.9041],
        [-0.0346, -0.2396, -0.5831, 0.7591,  0.6138,  0.9649]]

df['embedding'] = data
df.rename(columns={'INCIDENT_NUMBER': 'incident'}, inplace=True)

print(df)

          incident                                            embedding
0  INC000030884498   [0.6993, -0.1427, -0.1532, 0.8386, 0.5151, 0.8906]
1  INC000029956111    [0.7382, -0.8497, 0.1363, 0.8054, 0.5432, 0.9082]
2  INC000029555353   [0.0835, -0.2431, -0.0815, 0.8025, 0.5217, 0.9041]
3  INC000029555338  [-0.0346, -0.2396, -0.5831, 0.7591, 0.6138, 0.9649]


Instruction: How can I reshape the (1006,19) result of keras regressor predictions into a (1006,1) numpy array?

Input:
I'm trying to create a stock prediction model in botch PyTorch and Keras. I have already followed some tutorials online and modified to fit my data and it works fine.
Now I'm translating that code into a compatible Keras model. I've already created the model and did the predictions but the problem is that the regressor.predict() function from Keras returns a (1006,19) numpy array whereas when I do predictions = model(x_test) it returns a (1006,1) which is what I need for my following work so I can plot the results.
Here's my Keras code so far:
from keras.models import Sequential
from keras.layers import LSTM, Dense, Dropout

lookback = 20
x_train_keras, y_train_keras, x_test_keras, y_test_keras = split_data(price, lookback)
print('x_train.shape = ',x_train_keras.shape)  # x_train.shape =  (1006, 19, 1)
print('y_train.shape = ',y_train_keras.shape)  # y_train.shape =  (1006, 1)
print('x_test.shape = ',x_test_keras.shape)  # x_test.shape =  (252, 19, 1)
print('y_test.shape = ',y_test_keras.shape)  # y_test.shape =  (252, 1)

regression = Sequential()
regression.add(LSTM(units=50, return_sequences=True, kernel_initializer='glorot_uniform', input_shape=(x_train_keras.shape[1],1)))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',return_sequences=True))
regression.add(Dropout(0.2))

regression.add(Dense(units=1))

regression.compile(optimizer='adam', loss='mean_squared_error')

from keras.callbacks import History

history = History()
history = regression.fit(x_train_keras, y_train_keras, batch_size=30, epochs=100, callbacks=[history])

train_predict_keras = regression.predict(x_train_keras)
train_predict_keras = train_predict_keras.reshape((train_predict_keras.shape[0], train_predict_keras.shape[1]))

predict = pd.DataFrame(scaler.inverse_transform(train_predict_keras))
original = pd.DataFrame(scaler.inverse_transform(y_train_keras))

fig = plt.figure()
fig.subplots_adjust(hspace=0.2, wspace=0.2)

plt.subplot(1,2,1)
ax = sns.lineplot(x=original.index, y=original[0], label='Data', color='royalblue')
ax = sns.lineplot(x=predict.index, y=predict[0], label='Training Prediction', color='tomato')
ax.set_title('Stock Price', size=14, fontweight='bold')
ax.set_xlabel("Days", size = 14)
ax.set_ylabel("Cost (USD)", size = 14)
ax.set_xticklabels('', size=10)

plt.subplot(1,2,2)
ax = sns.lineplot(data=history.history.get('loss'), color='royalblue')
ax.set_xlabel("Epoch", size = 14)
ax.set_ylabel("Loss", size = 14)
ax.set_title("Training Loss", size = 14, fontweight='bold')
fig.set_figheight(6)
fig.set_figwidth(16)

# Make predictions
test_predict_keras = regression.predict(x_test_keras)

# Invert predictions
train_predict_keras = scaler.inverse_transform(train_predict_keras)
y_train_keras = scaler.inverse_transform(y_train_keras)
test_predict_keras = scaler.inverse_transform(test_predict_keras.reshape((test_predict_keras.shape[0], test_predict_keras.shape[1])))
y_test = scaler.inverse_transform(y_test_keras)

# Calculate root MSE
trainScore = math.sqrt(mean_squared_error(y_train[:,0], y_train_pred[:,0]))
print(f'Train score {trainScore:.2f} RMSE')
testScore = math.sqrt(mean_squared_error(y_test[:,0], y_test_pred[:,0]))
print(f'Test score {testScore:.2f} RMSE')

# shift train predictions for plotting
trainPredictPlot_keras = np.empty_like(price)
trainPredictPlot_keras[:, :] = np.nan
trainPredictPlot_keras[lookback:len(train_predict_keras)+lookback, :] = train_predict_keras

# shift test predictions for plotting
testPredictPlot_keras = np.empty_like(price)
testPredictPlot_keras[:, :] = np.nan
testPredictPlot_keras[len(train_predict_keras)+lookback-1:len(price)-1, :] = test_predict_keras

original = scaler.inverse_transform(price['Close'].values.reshape(-1,1))

predictions_keras = np.append(trainPredictPlot_keras, testPredictPlot_keras, axis=1)
predictions_keras = np.append(predictions_keras, original, axis=1)
result_keras = pd.DataFrame(predictions_keras)

The error occurs in the trainPredictPlot_keras[lookback:len(train_predict_keras)+lookback, :] = train_predict_keras line saying could not broadcast input array from shape (1006,19) into shape (1006,1)

Output:
Set the return_sequences to False for the last LSTM layer. You need to do as follows:
....
....
regression.add(LSTM(units=50,kernel_initializer='glorot_uniform',
                     return_sequences=False))
regression.add(Dropout(0.2))
regression.add(Dense(units=1))
regression.compile(optimizer='adam', loss='mean_squared_error')

Check doc:

return_sequences: Boolean. Whether to return the last output. in the output sequence, or the full sequence. Default: False.


Instruction: Is .contiguous().flatten() the same as .view(-1) in PyTorch?

Input:
Are these exactly the same?
myTensor.contiguous().flatten()
myTensor.view(-1)

Will they return the same auto grad function etc?

Output:
No, they are not exactly the same.

myTensor.contiguous().flatten():

Here, contiguous() either returns a copy of myTensor stored in contiguous memory, or returns myTensor itself if it is already contiguous. Then, flatten() reshapes the tensor to a single dimension. However, the returned tensor could be the same object as myTensor, a view, or a copy, so the contiguity of the output is not guaranteed.
Relevant documentation:

It’s also worth mentioning a few ops with special behaviors:

reshape(), reshape_as() and flatten() can return either a view or new tensor, user code shouldn’t rely on whether it’s view or not.

contiguous() returns itself if input tensor is already contiguous, otherwise it returns a new contiguous tensor by copying data.

myTensor.view(-1):

Here, view() returns a tensor with the same data as myTensor, and will only work if myTensor is already contiguous. The result may not be contiguous depending on the shape of myTensor.


Instruction: How to accelerate batch-size data from memory when using dataloader

Input:
I am trying to use dataloader for training. The dataset is 150G, which are all .npz files. Due to the limitation of memory size, only one sample is read at a time from the disk. The following is part of the code.
class VimeoDataset(Dataset):
def __init__(self, mode, batch_size=32, num_workers = 8, num_gpus = 4):
    self.batch_size = batch_size
    self.num_workers = num_workers
    self.num_gpus = num_gpus
    self.mode = mode
    self.load_data()
    self.h = 256
    self.w = 448
    xx = np.arange(0, self.w).reshape(1,-1).repeat(self.h,0)
    yy = np.arange(0, self.h).reshape(-1,1).repeat(self.w,1)
    self.grid = np.stack((xx,yy),2).copy()
    self.npzs=[]

    count = self.batch_size * self.num_workers * self.num_gpus
    if self.mode == 'train':
        filelist = glob('/data/vimeoFlow2/dataset/train/*.npz')
        self.npzs = [filelist[i:i + count] for i in range(0, len(filelist), count)]
    else:
        filelist = glob('/data/vimeoFlow2/dataset/val/*.npz')
        self.npzs = [filelist[i:i + count] for i in range(0, len(filelist), count)]

def __len__(self):
    return len(self.npzs)

def load_data(self, index):
    self.data = []
    self.flow_data = []

    for i in range(len(self.npzs[index])):
        f = np.load(self.npzs[index][i])
        self.data.append(f['i0i1gt'])
        if self.mode == 'train':
            self.flow_data.append(f['ft0ft1'])
        else:
            self.flow_data.append(np.zeros((256, 448, 4)))

def getimg(self, index):
    data = self.meta_data[index]
    img0 = data[0:3].transpose(1, 2, 0)
    img1 = data[3:6].transpose(1, 2, 0)
    gt = data[6:9].transpose(1, 2, 0)
    flow_gt = (self.flow_data[index]).transpose(1, 2, 0)
    return img0, gt, img1, flow_gt

def __getitem__(self, index):
    img0, gt, img1, flow_gt = self.getimg(index)

dataset = VimeoDataset(mode = 'train',  batch_size=32, num_workers = 8, num_gpus = 4)
sampler = DistributedSampler(dataset)
train_data = DataLoader(dataset, batch_size=args.batch_size, pin_memory=True, num_workers=args.num_workers, drop_last=True, sampler=sampler)
dataset_val = VimeoDataset(mode = 'val',  batch_size=32, num_workers = 8, num_gpus = 4)
val_data = DataLoader(dataset_val, batch_size=args.batch_size, pin_memory=True, num_workers=args.num_workers)

However, reading data from the disk one by one causes the dataloader to be very time-consuming. So I want to improve this program, first load the amount of data of num_gpus×num_workers×batch_size into the memory, then read the data from the memory with __getitem__, and finally replace the data in the memory after each iteration. But I still don’t know how to achieve it. I have tried my idea as in the code above. I don't konw how to allocate the load_data function parameters.

Output:
It look like you are trying to use torch Dataset in the wrong way. Your Dataset subclass should neither batch the data itself nor use the number of workers.
Batching the data and loading it in parallel is the role of the DataLoader class. You Dataset subclass __getitem__ method should only returns 1 sample (and additionally one ground truth annotation) from the dataset, it should be data like Tensor or Array which can be concatenated in order to create a batch.
Take a look at the Dataset and DataLoader documentation that is pretty clear on this.

The purpose of DataLoader is to load (i.e. read from disk to memory) and pre-process your data in parallel. If you specified 8 workers, it roughly means that 8 parallel threads are calling the __getitem__ method to create a batch of items. Note that DataLoader already "caches" the data and load them in advance to be ready in time (take a look at the prefetch_factor parameter).
This should be a sufficient compromise between loading speed and memory consumption, you should try this before writing any custom caching, loading and parallel processing of your data.


Instruction: TypeError: img should be PIL Image. Got  - PyTorch

Input:
I'm trying to prepare some image data for my neural to classify. As part of the image preprocessing step, I'm applying the HOG filter in my dataset class as such:
class GetHogData(Dataset):

  def __init__(self, df, root, transform = None):
    self.df = df
    self.root = root
    self.transform = transform

  def __len__(self):
    return len(self.df)

  def __getitem__(self, idx):
    if torch.is_tensor(idx):
      idx = idx.tolist()

    img_path = os.path.join(self.root, self.df.iloc[idx, 0])
    # image = Image.open(img_path)
    image = cv2.imread(img_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    label = self.df.iloc[idx, 1]

    if self.transform:
      image = self.transform(image)

    hog_, hog_image = hog(
        image,
        orientations = 9,
        pixels_per_cell = (14,14),
        cells_per_block = (2,2),
        block_norm = "L1")

    image = np.transpose(image, (2, 0, 1))

    img_hog_lbl = {
        "image" : torch.tensor(image, dtype = torch.float32),
        "label" : torch.tensor(label, dtype = torch.long),
        "hog": torch.tensor(hog_, dtype = torch.float32)
    }
    return img_hog_lbl

After this, I define my train and validation transformation as such:
# Image mean and standard dev

img_mean = [0.485, 0.456, 0.406]
img_std = [0.229, 0.224, 0.225]

train_trans = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(img_mean, img_std)
    ])

test_trans = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(img_mean, img_std)
    ])

and finally, I create the loaders as such:
train_img = GetHogData(df = train_lab, root = "/content/train", transform = train_trans)
test_img = GetHogData(df = test_lab ,root = "/content/test", transform = test_trans)

However, when I attempt to preview the training image with test_img[1]  I get the error:
TypeError                                 Traceback (most recent call last)
<ipython-input-132-b9a9394eb1e0> in <module>()
----> 1 test_img[1]

5 frames
/usr/local/lib/python3.7/dist-packages/torchvision/transforms/functional_pil.py in resize(img, size, interpolation)
    207 def resize(img, size, interpolation=Image.BILINEAR):
    208     if not _is_pil_image(img):
--> 209         raise TypeError('img should be PIL Image. Got {}'.format(type(img)))
    210     if not (isinstance(size, int) or (isinstance(size, Sequence) and len(size) in (1, 2))):
    211         raise TypeError('Got inappropriate size arg: {}'.format(size))

TypeError: img should be PIL Image. Got <class 'numpy.ndarray'>

I've tried to add transforms.ToPILImage() to my transforms by doing:
# Image mean and standard dev

img_mean = [0.485, 0.456, 0.406]
img_std = [0.229, 0.224, 0.225]

train_trans = transforms.Compose([
        transforms.ToPILImage(),
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(img_mean, img_std)
    ])

test_trans = transforms.Compose([
        transforms.ToPILImage(),
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(img_mean, img_std)
    ])

but I got the error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-135-b9a9394eb1e0> in <module>()
----> 1 test_img[1]

1 frames
<ipython-input-129-8551c2e76038> in __getitem__(self, idx)
     27         pixels_per_cell = (14,14),
     28         cells_per_block = (2,2),
---> 29         block_norm = "L1")
     30
     31     image = np.transpose(image, (2, 0, 1))

/usr/local/lib/python3.7/dist-packages/skimage/feature/_hog.py in hog(image, orientations, pixels_per_cell, cells_per_block, block_norm, visualize, transform_sqrt, feature_vector, multichannel)
    273     n_blocks_col = (n_cells_col - b_col) + 1
    274     normalized_blocks = np.zeros((n_blocks_row, n_blocks_col,
--> 275                                   b_row, b_col, orientations))
    276
    277     for r in range(n_blocks_row):

ValueError: negative dimensions are not allowed

Does anybody have any ideas? Thanks in advance!
Edit - New Error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-154-b9a9394eb1e0> in <module>()
----> 1 test_img[1]

<ipython-input-151-8551c2e76038> in __getitem__(self, idx)
     27         pixels_per_cell = (14,14),
     28         cells_per_block = (2,2),
---> 29         block_norm = "L1")
     30
     31     image = np.transpose(image, (2, 0, 1))

ValueError: too many values to unpack (expected 2)

Output:
The problem is as I wrote in the comment, skimage requires the data to be ndarray but you give it a torch tensor thus that error.
Try this
    train_trans = transforms.Compose([
        transforms.ToPILImage(),
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(img_mean, img_std),
        lambda x: np.rollaxis(x.numpy(), 0, 3)
    ])

Edit
This is basically transform the output to ndarray and change channel axis.
But as you can see it's not the best way to fix things since you have to transform PIL image to tensor then transform tensor to ndarray and then transform ndarray back to tensor again.
The better way to do this is transform PIL image directly to ndarray and normalize that, for example.
in getitem
    if self.transform:
      image = self.transform(image)

      # add these
      image = np.array(image)

      mean = [0.485, 0.456, 0.406]
      std = [0.229, 0.224, 0.225]

      x[..., 0] -= mean[0]
      x[..., 1] -= mean[1]
      x[..., 2] -= mean[2]
      x[..., 0] /= std[0]
      x[..., 1] /= std[1]
      x[..., 2] /= std[2]

    # these are your code
    hog_, hog_image = hog(

And in transform just use
    train_trans = transforms.Compose([
        transforms.ToPILImage(),
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
    ])

Edit2
Refer to this line. You need to either add visualize=True in hog() or remove , hog_image. If you don't need hog_image then the latter is preferred.
    hog_, hog_image = hog(
        image, visualize=True,

hog_ = hog(


Instruction: Concatenating two multi-dimensional numpy arrays at specific index

Input:
I have two numpy arrays of sizes: [20,3,100,100] and [20,5,100,100].
They are 20 (100,100) images with 3 channels, and 5 channels, respectively. I want to concatenate the channels so that I have 20 (100,100) images with 8 channels.
I would like to concat them along (dim = 1) them together, without having to create a new numpy.zeros array of size [20,8,100,100]. Is this possible?

Output:
Concatenate merges arrays along an existing axis:
import numpy as np

a = np.zeros((20,3,100,100))
b = np.ones((20,5,100,100))
output = np.concatenate((a,b), axis=1)

output.shape
# (20, 8, 100, 100)


Instruction: The loss value does not decrease

Input:
I am implementing a simple feedforward neural network with Pytorch and the loss function does not seem to decrease. Because of some other tests I have done, the problem seems to be in the computations I do to compute pred, since if I slightly change the network so that it spits out a 2-dimensional vector for each entry and save it as pred, everything works perfectly.
Do you see the problem in defining pred here? Thanks
import torch
import numpy as np
from torch import nn

dt = 0.1

class Neural_Network(nn.Module):
    def __init__(self, ):
        super(Neural_Network, self).__init__()

        self.l1 = nn.Linear(2,300)
        self.nl = nn.Tanh()
        self.l2 = nn.Linear(300,1)

    def forward(self, X):
        z = self.l1(X)
        z = self.nl(z)
        o = self.l2(z)
        return o

N = 1000
X = torch.rand(N,2,requires_grad=True)
y = torch.rand(N,1)
NN = Neural_Network()

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(NN.parameters(), lr=1e-5)

epochs = 200

for i in range(epochs):  # trains the NN 1,000 times

    HH = torch.mean(NN(X))
    gradH =  torch.autograd.grad(HH, X)[0]
    XH= torch.cat((gradH[:,1].unsqueeze(0),-gradH[:,0].unsqueeze(0)),dim=0).t()
    pred = X + dt*XH

    #Optimize and improve the weights
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    print (" Loss: ", loss.detach().numpy())  # mean sum squared loss

P.S. With these X and y the loss is not expected to go to zero, I have added them here like them just for simplicity. I will apply this architecture to data points which are expected to satisfy this model. However I am just interested in seeing the loss decreasing.

My aim is to approximate with a neural network the Hamiltonian of a vector field where only some trajectory is known. For example only the updates x(t)\rightarrow x(t+\Delta t) for some choice of points. So the vector X contains the points x(t), while y contains the $x(t+\Delta t)$. My network above approximates in a simple way the Hamiltonian function H(x) and in order to optimize it I need to find the trajectories associated to this Hamiltonian.
In particular XH aims to be the Hamiltonian vector field associated to the approximated Hamiltonian. The time update pred = X + dt*XH is simply one step of forward Euler.
However, my main issue here can be abstracted in: how can I involve the gradient of a network with respect to its inputs in the loss function?

Output:
Probably because the gradient flow graph for NN is destroyed with the gradH step. (check HH.grad_fn vs gradH.grad_fn )
So your pred tensor (and subsequent loss) does not contain the necessary gradient flow through the NN network.
The loss contains gradient flow for the input X, but not for the NN.parameters().  Because the optimizer only take a step() over thoseNN.parameters(), the network NN is not being updated, and since X is neither being updated, loss does not change.
You can check how the loss is sending it's gradients backward by checking loss.grad_fn after loss.backward()
and here's a neat function (found on Stackoverflow) to check it:
def getBack(var_grad_fn):
    print(var_grad_fn)
    for n in var_grad_fn.next_functions:
        if n[0]:
            try:
                tensor = getattr(n[0], 'variable')
                print(n[0])
                print('Tensor with grad found:', tensor)
                print(' - gradient:', tensor.grad)
                print()
            except AttributeError as e:
                getBack(n[0])

with getBack(loss.grad_fn) after loss.backward() to check it for yourself (maybe reduce size of batch N before though)
Edit: It works by changing gradH = torch.autograd.grad(HH, X, create_graph=True)[0]


Instruction: I want to confirm which of these methods to calculate Dice Loss is correct

Input:
so I have 4 methods to calculate dice loss and 3 of them are returning the same results, so I can conclude that 1 of them is calculating it wrong, but I would to confirm it with you guys:
import torch
torch.manual_seed(0)

inputs = torch.rand((3,1,224,224))
target = torch.rand((3,1,224,224))

Method 1: flatten tensors
def method1(inputs, target):

    inputs = inputs.reshape( -1)

    target = target.reshape( -1)

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()

    print("method1", dice)

Method 2: flatten tensors except for batch size, sum all dims
def method2(inputs, target):
    num = target.shape[0]
    inputs = inputs.reshape(num, -1)

    target = target.reshape(num, -1)

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()/num

    print("method2", dice)

Method 3: flatten tensors except for batch size, sum dim 1
def method3(inputs, target):
    num = target.shape[0]
    inputs = inputs.reshape(num, -1)

    target = target.reshape(num, -1)

    intersection = (inputs * target).sum(1)
    union = inputs.sum(1) + target.sum(1)
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()/num

    print("method3", dice)

Method 4: don't flatten tensors
def method4(inputs, target):

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)

    print("method4", dice)

method1(inputs, target)
method2(inputs, target)
method3(inputs, target)
method4(inputs, target)

method 1,3 and 4 print: 0.5006
method 2 print: 0.1669
and it makes sense, since I am flattening the inputs and targets on 3 dimensions leaving out batch size, and then I am summing all 2 dimensions that result from the flattening instead of just dim 1
Method 4 seems to be the most optimized one

Output:
First, you need to decide what dice score you report: the dice score of all samples in the batch (methods 1,2 and 4) or the averaged dice score of each sample in the batch (method 3).
If I'm not mistaken, you want to use method 3 - you want to optimize the dice score of each of the samples in the batch and not a "global" dice score: Suppose you have one "difficult" sample in an "easy" batch. The misclassified pixels of the "difficult" sample will be negligible w.r.t all other pixels. But if you look at the dice score of each sample separately  then the dice score of the "difficult" sample will not be negligible.


Instruction: Indexing whole Tensor along specific dimension and specific channels

Input:
Let say we have a Tensor A with the dimension dim(A)=[i, j, k=6, u, v]. Now we are interested to get the whole tensor at dimension k with channels=[0:3]. I know we can get it this way:
B = A[:, :, 0:3, :, :]

Now I would like to know if there is any better "pythonic" way to achieve the same result without doing this suboptimal indexing. I mean something like.
B = subset(A, dim=2, index=[0, 1, 2])

No matter in which framework, i.e. pytorch, tensorflow, numpy, etc.
Thanks a lot

Output:
In numpy, you can use the take method:
B = A.take([0,1,2], axis=2)

In TensorFlow, there is not really a more concise way than using the traditionnal approach. Using tf.slice would be really verbose:
B = tf.slice(A,[0,0,0,0,0],[-1,-1,3,-1,-1])

You can potentially use the experimental version of take (since TF 2.4):
B = tf.experimental.numpy.take(A, [0,1,2], axis=2)

in PyTorch, you can use index_select:
torch.index_select(A, dim=2, index=torch.tensor([0,1,2]))

Note that you can skip listing explicitly the first dimensions (or the last) by using an ellipsis:
# Both are equivalent in that case
B = A[..., 0:3, :, :]
B = A[:, :, 0:3, ...]


Instruction: Can I find the number of specific numeric data in this MNIST training, test data?

Input:
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import torch.nn.init
import torch.nn.functional as F

device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)

learning_rate = 0.001
training_epochs = 15
batch_size = 100

mnist_train = dsets.MNIST(root='MNIST_data/', # Specify download path
                          train=True, # Specify True to download as training data
                          transform=transforms.ToTensor(), # Convert to tensor
                          download=True)

mnist_test = dsets.MNIST(root='MNIST_data/', # Specify download path
                         train=False, # If false is specified, download as test data
                         transform=transforms.ToTensor(), # Convert to tensor
                         download=True)

This is the part that loads the data of MNIST data classification code using CNN
In the book I refer to, it is said that you can see how many specific numeric data are in the training set and the test set by mentioning only that part.
For example, can you tell how many '5' data are in that training or test set?
Just knowing that you can access data tensors with mnist_train.train_data or mnist_train.train_labels, etc., I don't know how much I can think of how to know the number of specific numeric data. help

Output:
You can access the data and labels of the dataset, for either split, using the data and targets attributes respectively. So, for example, here you can access the training data and labels using mnist_train.data and mnist_train.labels respectively.
Since the targets attribute is a torch.Tensor for this dataset, you can count the number of instances of each target by using torch.bincount. Since there are 10 classes in total, the output will be a tensor of length 10, where the ith index specifies the number of data points of class i.
Example:
>>> mnist_train = dsets.MNIST(root='MNIST_data/', train=True, transforms.ToTensor(), download=True)
>>> mnist_train.targets
tensor([5, 0, 4,  ..., 5, 6, 8])
>>> torch.bincount(mnist_train.targets, minlength=10)
tensor([5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851, 5949])

You can see that class 5 has 5,421 data points in the training split.


Instruction: Why is the clip_grad_norm_ function used here?

Input:
I am learning LSTM with PyTorch from someone's code. Here he uses the clip_grad_norm_ function in the training process of a two layer LSTM. I want to know why he uses the clip_grad_norm_ function here, so I can understand the whole code properly (he used it in second last line).
for x, y in get_batches(data, batch_size, seq_length):
    counter += 1

    x = one_hot_encode(x, n_chars)
    inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

    if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()

    h = tuple([each.data for each in h])
    net.zero_grad()

    output, h = net(inputs, h)

    loss = criterion(output, targets.view(batch_size*seq_length).long())
    loss.backward()

    nn.utils.clip_grad_norm_(net.parameters(), clip)
    opt.step()

If you need more information about question then please let me know.

Output:
torch.nn.utils.clip_grad_norm_ performs gradient clipping. It is used to mitigate the problem of exploding gradients, which is of particular concern for recurrent networks (which LSTMs are a type of).
Further details can be found in the original paper.


Instruction: Modifying a Numpy float32 bitwise and returning the modified np.float32 value

Input:
I'm currently working on a bit chopping algorithm for simulation of memory footprint reduction when training neural networks. I'm using PyTorch to achieve this.
However, what I'm basically trying to do is set to 0 the less significant bits of the mantissa of the float32 value to see for now if the neural network will train and how much precision it will lose depending on the number of bits that are set to 0.
My problem is that every value on the tensors is of type Numpy float32, and I would like to get the literal bit representation of the value (how the actual float32 value is represented in memory) or as an integer, apply the bitwise modification and convert it back to np.float32.
However, I've tried the following (Note, x is of type numpy.ndarray):
    print(x)
    value_as_int = self.x.view(np.int32)
    print(value_as_int)
    value_as_int = value_as_int&0xFFFFFFFE
    print(value_as_int)
    new_float = value_as_int.view(np.float32)
    print(new_float)

Here's an example output of the part that works:
0.13498048
1040857171
1040857170

This does convert the value to its literal bit integer representation and allows me to set to 0 the last bit, although when trying to convert it back to np.float32 I get the following error:

ValueError: Changing the dtype of a 0d array is only supported if the itemsize is unchanged

Is there a proper way to do this? Am I missing something in my code or is it the wrong approach?
Thank you in advance

Output:
The problem here is that 0xFFFFFFFE is a python int (rather than a numpy int32). Numpy is implicitly upcasting to int64 when the bit-wise AND operator is appled. To avoid this you can make your bit-mask a np.uint32.
It seems that in Windows (but not Linux), you need to use np.uint32 instead of np.int32 to avoid getting a "Python int too large to convert to C long" error when your bit-mask is larger than 0x7FFFFFFF.
value_as_int = self.x.view(np.uint32)
value_as_int = value_as_int & np.uint32(0xFFFFFFFE)
new_float = value_as_int.view(np.float32)


Instruction: Accessing PyTorch modules - ResNet18

Input:
I am using a ResNet-18 coded as follows:
class ResidualBlock(nn.Module):
    '''
    Residual Block within a ResNet CNN model
    '''
    def __init__(self, input_channels, num_channels,
                 use_1x1_conv = False, strides = 1):
        # super(ResidualBlock, self).__init__()
        super().__init__()

        self.conv1 = nn.Conv2d(
            in_channels = input_channels, out_channels = num_channels,
            kernel_size = 3, padding = 1, stride = strides,
            bias = False
            )
        self.bn1 = nn.BatchNorm2d(num_features = num_channels)

        self.conv2 = nn.Conv2d(
            in_channels = num_channels, out_channels = num_channels,
            kernel_size = 3, padding = 1, stride = 1,
            bias = False
            )
        self.bn2 = nn.BatchNorm2d(num_features = num_channels)

        if use_1x1_conv:
            self.conv3 = nn.Conv2d(
                in_channels = input_channels, out_channels = num_channels,
                kernel_size = 1, stride = strides
                )
            self.bn3 = nn.BatchNorm2d(num_features = num_channels)
        else:
            self.conv3 = None

        self.relu = nn.ReLU(inplace = True)

        self.initialize_weights()

    def forward(self, X):
        Y = F.relu(self.bn1(self.conv1(X)))
        Y = self.bn2(self.conv2(Y))

        if self.conv3:
            X = self.bn3(self.conv3(X))
            # print(f"X.shape due to 1x1: {X.shape} & Y.shape = {Y.shape}")
        else:
            # print(f"X.shape without 1x1: {X.shape} & Y.shape = {Y.shape}")
            pass

        Y += X
        return F.relu(Y)

    def shape_computation(self, X):
        Y = self.conv1(X)
        print(f"self.conv1(X).shape: {Y.shape}")
        Y = self.conv2(Y)
        print(f"self.conv2(X).shape: {Y.shape}")

        if self.conv3:
            h = self.conv3(X)
            print(f"self.conv3(X).shape: {h.shape}")

    def initialize_weights(self):
        for m in self.modules():
            # print(m)
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_uniform_(m.weight)

                '''
                # Do not initialize bias (due to batchnorm)-
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
                '''

            elif isinstance(m, nn.BatchNorm2d):
                # Standard initialization for batch normalization-
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

            elif isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight)
                nn.init.constant_(m.bias, 0)

b0 = nn.Sequential(
    nn.Conv2d(in_channels = 3, out_channels = 64, kernel_size = 3, stride = 1, padding = 1),
    nn.BatchNorm2d(num_features = 64),
    nn.ReLU())

def create_resnet_block(input_filters, output_filters, num_residuals, first_block = False):
    # Python list to hold the created ResNet blocks-
    resnet_blk = []

    for i in range(num_residuals):
        if i == 0 and first_block:
            resnet_blk.append(ResidualBlock(input_channels = input_filters, num_channels = output_filters, use_1x1_conv = True, strides = 2))
        else:
            resnet_blk.append(ResidualBlock(input_channels = output_filters, num_channels = output_filters, use_1x1_conv = False, strides = 1))

    return resnet_blk

b1 = nn.Sequential(*create_resnet_block(input_filters = 64, output_filters = 64, num_residuals = 2, first_block = True))

b2 = nn.Sequential(*create_resnet_block(input_filters = 64, output_filters = 128, num_residuals = 2, first_block = True))

b3 = nn.Sequential(*create_resnet_block(input_filters = 128, output_filters = 256, num_residuals = 2, first_block = True))

b4 = nn.Sequential(*create_resnet_block(input_filters = 256, output_filters = 512, num_residuals = 2, first_block = True))

# Initialize a ResNet-18 CNN model-
model = nn.Sequential(
    b0, b1, b2, b3, b4,
    nn.AdaptiveAvgPool2d(output_size = (1, 1)),
    nn.Flatten(),
    nn.Linear(in_features = 512, out_features = 10))

The layer names are now as follows:
for layer_name, param in trained_model.named_parameters():
    print(f"layer name: {layer_name} has {param.shape}")

Result:
> layer name: 0.0.weight has torch.Size([64, 3, 3, 3])

> layer name: 0.0.bias has torch.Size([64])

> layer name: 0.1.weight has torch.Size([64])

> layer name: 0.1.bias has torch.Size([64])

> layer name: 1.0.conv1.weight has torch.Size([64, 64, 3, 3])

> layer name: 1.0.bn1.weight has torch.Size([64])

> layer name: 1.0.bn1.bias has torch.Size([64])

> layer name: 1.0.conv2.weight has torch.Size([64, 64, 3, 3])

> layer name: 1.0.bn2.weight has torch.Size([64])

> layer name: 1.0.bn2.bias has torch.Size([64])

> layer name: 1.0.conv3.weight has torch.Size([64, 64, 1, 1])

> layer name: 1.0.conv3.bias has torch.Size([64])

> layer name: 1.0.bn3.weight has torch.Size([64])

> layer name: 1.0.bn3.bias has torch.Size([64])

> layer name: 1.1.conv1.weight has torch.Size([64, 64, 3, 3])

> layer name: 1.1.bn1.weight has torch.Size([64])

> layer name: 1.1.bn1.bias has torch.Size([64])

> layer name: 1.1.conv2.weight has torch.Size([64, 64, 3, 3])

> layer name: 1.1.bn2.weight has torch.Size([64])

> layer name: 1.1.bn2.bias has torch.Size([64])

> layer name: 2.0.conv1.weight has torch.Size([128, 64, 3, 3])

> layer name: 2.0.bn1.weight has torch.Size([128])

> layer name: 2.0.bn1.bias has torch.Size([128])

> layer name: 2.0.conv2.weight has torch.Size([128, 128, 3, 3])

> layer name: 2.0.bn2.weight has torch.Size([128])

> layer name: 2.0.bn2.bias has torch.Size([128])

> layer name: 2.0.conv3.weight has torch.Size([128, 64, 1, 1])

> layer name: 2.0.conv3.bias has torch.Size([128])

> layer name: 2.0.bn3.weight has torch.Size([128])

> layer name: 2.0.bn3.bias has torch.Size([128])

> layer name: 2.1.conv1.weight has torch.Size([128, 128, 3, 3])

> layer name: 2.1.bn1.weight has torch.Size([128])

> layer name: 2.1.bn1.bias has torch.Size([128])

> layer name: 2.1.conv2.weight has torch.Size([128, 128, 3, 3])

> layer name: 2.1.bn2.weight has torch.Size([128])

> layer name: 2.1.bn2.bias has torch.Size([128])

> layer name: 3.0.conv1.weight has torch.Size([256, 128, 3, 3])

> layer name: 3.0.bn1.weight has torch.Size([256])

> layer name: 3.0.bn1.bias has torch.Size([256])

> layer name: 3.0.conv2.weight has torch.Size([256, 256, 3, 3])

> layer name: 3.0.bn2.weight has torch.Size([256])

> layer name: 3.0.bn2.bias has torch.Size([256])

> layer name: 3.0.conv3.weight has torch.Size([256, 128, 1, 1])

> layer name: 3.0.conv3.bias has torch.Size([256])

> layer name: 3.0.bn3.weight has torch.Size([256])

> layer name: 3.0.bn3.bias has torch.Size([256])

> layer name: 3.1.conv1.weight has torch.Size([256, 256, 3, 3])

> layer name: 3.1.bn1.weight has torch.Size([256])

> layer name: 3.1.bn1.bias has torch.Size([256])

> layer name: 3.1.conv2.weight has torch.Size([256, 256, 3, 3])

> layer name: 3.1.bn2.weight has torch.Size([256])

> layer name: 3.1.bn2.bias has torch.Size([256])

> layer name: 4.0.conv1.weight has torch.Size([512, 256, 3, 3])

> layer name: 4.0.bn1.weight has torch.Size([512])

> layer name: 4.0.bn1.bias has torch.Size([512])

> layer name: 4.0.conv2.weight has torch.Size([512, 512, 3, 3])

> layer name: 4.0.bn2.weight has torch.Size([512])

> layer name: 4.0.bn2.bias has torch.Size([512])

> layer name: 4.0.conv3.weight has torch.Size([512, 256, 1, 1])

> layer name: 4.0.conv3.bias has torch.Size([512])

> layer name: 4.0.bn3.weight has torch.Size([512])

> layer name: 4.0.bn3.bias has torch.Size([512])

> layer name: 4.1.conv1.weight has torch.Size([512, 512, 3, 3])

> layer name: 4.1.bn1.weight has torch.Size([512])

> layer name: 4.1.bn1.bias has torch.Size([512])

> layer name: 4.1.conv2.weight has torch.Size([512, 512, 3, 3])

> layer name: 4.1.bn2.weight has torch.Size([512])

> layer name: 4.1.bn2.bias has torch.Size([512])

> layer name: 7.weight has torch.Size([10, 512])

> layer name: 7.bias has torch.Size([10])

In order to prune this model, I am referring to PyTorch pruning tutorial. It's mentioned here that to prune a module/layer, use the following code:
parameters_to_prune = (
    (model.conv1, 'weight'),
    (model.conv2, 'weight'),
    (model.fc1, 'weight'),
    (model.fc2, 'weight'),
    (model.fc3, 'weight'),
)

But for the code above, the modules/layers no longer have this naming convention. For example, to prune the first conv layer of this model:
> layer name: 0.0.weight has torch.Size([64, 3, 3, 3])

on trying the following code:
prune.random_unstructured(model.0.0, name = 'weight', amount = 0.3)

It gives me the error:
prune.random_unstructured(trained_model.0.0, name = 'weight', amount = 0.3)
                                           ^ SyntaxError: invalid syntax

How do I handle this?

Output:
this will work
import torch.nn.utils.prune as prune
prune.random_unstructured(list(model.children())[0][0] , name = 'weight', amount = 0.3) # first conv layer


Instruction: How to get class and bounding box coordinates from YOLOv5 predictions?

Input:
I am trying to perform inference on my custom YOLOv5 model. The official documentation uses the default detect.py script for inference. I have written my own python script but I cannot access the predicted class and the bounding box coordinates from the output of the model. Here is my code:
import torch

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='best.pt')
predictions = model("my_image.png")

print(predictions)

Output:
results = model(input_images)
labels, cord_thres = results.xyxyn[0][:, -1].numpy(), results.xyxyn[0][:, :-1].numpy()

This will give you labels, coordinates, and thresholds for each object detected, you can use it to plot bounding boxes.
You can check out this repo for more detailed code.
https://github.com/akash-agni/Real-Time-Object-Detection


Instruction: Neural network learning to sum two numbers

Input:
I am learning Pytorch, and I am trying to implement a really simple network which takes an input which is of length 2, i.e. a point in the plane, and aims to learn the sum of its components.
In principle the network should just learn a linear layer with weight matrix W = [1.,1.] and zero bias, so I expect to have very low training error. However, I don't see why I do not get this as expected.
The code I am writing is this:
import torch
from torch import nn, optim
import numpy as np

device = torch.device("cuda:0" if
torch.cuda.is_available() else "cpu")

N = 1000  # number of samples
D = 2  # input dimension
C = 1  # output dimension

def model(z):
  q = z[:,0]
  p = z[:,1]
  return q+p

X = torch.rand(N, D,requires_grad=True).to(device)
y = model(X)

lr = 1e-2 #Learning rate

Rete = nn.Sequential(nn.Linear(D, C))
Rete.to(device) #Convert to CUDA

criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(Rete.parameters(), lr=lr)

for t in range(5000):

    y_pred = Rete(X)
    loss = criterion(y_pred, y)
    print("[EPOCH]: %i, [LOSS]: %.6f" % (t, loss.item()))
    optimizer.zero_grad()
    optimizer.step()

Output:
There are 2 problems.
The first problem is that you forgot to backpropogate the loss:
optimizer.zero_grad()
loss.backward() # you forgot this step
optimizer.step()

It is important that optimizer.zero_grad() IS NOT put in between loss.backwards() and optimizer.step(), otherwise you'll be resetting the gradients before performing the gradient descent step. In general, it is advisable to either put optimizer.zero_grad() at the very beginning of your loop, or right after you call optimizer.step():
loss.backward()
optimizer.step()
optimizer.zero_grad() # this should go right after .step(), or at the very beginning of your training loop

Even after this change, you'll notice that your model still doesn't converge:
[EPOCH]: 0, [LOSS]: 0.232405
[EPOCH]: 1, [LOSS]: 0.225010
[EPOCH]: 2, [LOSS]: 0.218473
...
[EPOCH]: 4997, [LOSS]: 0.178762
[EPOCH]: 4998, [LOSS]: 0.178762
[EPOCH]: 4999, [LOSS]: 0.178762

This leads us to the second problem, the shapes of the output (y_pred) and labels (y) do not match. y_pred has shape (N, C) but y has shape (N,). To fix this, just reshape y to match y_pred:
y = y.reshape(-1, C)

Then our model will converge:
[EPOCH]: 0, [LOSS]: 1.732189
[EPOCH]: 1, [LOSS]: 1.680017
[EPOCH]: 2, [LOSS]: 1.628712
...
[EPOCH]: 4997, [LOSS]: 0.000000
[EPOCH]: 4998, [LOSS]: 0.000000
[EPOCH]: 4999, [LOSS]: 0.000000

Both of these bugs fail silently, which makes debugging them difficult. Unfortunately, these kinds of bugs are very easy to come across when doing machine learning. I highly recommend reading this blog post on best practices when training neural networks to minimize the risk of silent bugs.

Full code:
import torch
import numpy as np

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

N = 1000  # number of samples
D = 2  # input dimension
C = 1  # output dimension

X = torch.rand(N, D).to(device)  # (N, D)
y = torch.sum(X, axis=-1).reshape(-1, C)  # (N, C)

lr = 1e-2  # Learning rate

model = torch.nn.Sequential(torch.nn.Linear(D, C))  # model
model.to(device)

criterion = torch.nn.MSELoss()  # loss function
optimizer = torch.optim.Adam(model.parameters(), lr=lr)  # optimizer

for epoch in range(1000):
    y_pred = model(X)  # forward step
    loss = criterion(y_pred, y)  # compute loss
    loss.backward()  # backprop (compute gradients)
    optimizer.step()  # update weights (gradient descent step)
    optimizer.zero_grad()  # reset gradients
    if epoch % 50 == 0:
        print(f"[EPOCH]: {epoch}, [LOSS]: {loss.item():.6f}")


Instruction: Related to SubsetRandomSampler

Input:
I am using SubsetRandomSampler for splitting a classification dataset into test and validation. Can we split the dataset for each class.
import numpy as np
import torch
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

train_transforms = transforms.Compose([transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])
dataset = datasets.ImageFolder( '/data/images/train', transform=train_transforms )

validation_split = .2
shuffle_dataset = True
random_seed= 42
batch_size = 20

dataset_size = len(dataset) #4996
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))

if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)
validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=valid_sampler)

Output:
Did you mean train and validation not test and validation?
If so, the SubsetRandomSampler uses randomly select samples from indices. Therefore you can just randomly split the indices of each class before put them in train_indices and val_indices.
Like
indexs = [[] for _ in range(len(dataset.classes))]  # you can't use `[[]] * len(dataset.classes)`. Although there might be better ways but I don't know
for idx, (_, class_idx) in enumerate(dataset):
    indexs[class_idx].append(idx)
train_indices, val_indices = [], []
for cl_idx in indexs:
    size = len(cl_idx)
    split = int(np.floor(validation_split * size))
    np.random.shuffle(cl_idx)
    train_indices.extend(cl_idx[split:])
    val_indices.extend(cl_idx[:split])
train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)


Instruction: Stuck at this error "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu"

Input:
I am running the following code. If I try to run it on CPU only it runs fine but it takes too much time to train. So I thought to change the runtime to GPU and made appropriate changes. Now it is stuck.
import torch
from models.bert_attention_model import AttentionModel
from models.bert_cnn_model import BERTCNNSentiment

import sys

if sys.argv[1].lower() =="hinglish":
  data_path = "../data/hinglish/"
elif sys.argv[1].lower() == "spanglish":
  data_path = "../data/spanglish/"
else:
  print("Format: %s %s" %(argv[0], argv[1]))

train_name = "train.txt"
test_name = "test.txt"
model_save_names = ["../checkpoint/cnn_model.txt", "../checkpoint/attention_model.txt"]

import random
import numpy as np

SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

from transformers import BertTokenizer, AutoTokenizer, XLMRobertaTokenizer

tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")
print('XLM Roberta Tokenizer Loaded...')

init_token_idx = tokenizer.cls_token_id
eos_token_idx = tokenizer.sep_token_id
pad_token_idx = tokenizer.pad_token_id
unk_token_idx = tokenizer.unk_token_id

max_input_length = 150
print("Max input length: %d" %(max_input_length))

def tokenize_and_cut(sentence):
    tokens = tokenizer.tokenize(sentence)
    tokens = tokens[:max_input_length-2]
    return tokens

from torchtext import data

UID = data.Field(sequential=False, use_vocab=False, pad_token=None)
TEXT = data.Field(batch_first = True,
                  use_vocab = False,
                  tokenize = tokenize_and_cut,
                  preprocessing = tokenizer.convert_tokens_to_ids,
                  init_token = init_token_idx,
                  eos_token = eos_token_idx,
                  pad_token = pad_token_idx,
                  unk_token = unk_token_idx)

LABEL = data.LabelField()

from torchtext import datasets

fields = [('uid',UID),('text', TEXT),('label', LABEL)]
train_data, test_data = data.TabularDataset.splits(
                                        path = data_path,
                                        train = train_name,
                                        test = test_name,
                                        format = 'tsv',
                                        fields = fields,
                                        skip_header = True)
train_data, valid_data = train_data.split(random_state = random.seed(SEED))
print('Data loading complete')
print(f"Number of training examples: {len(train_data)}")
print(f"Number of validation examples: {len(valid_data)}")
print(f"Number of test examples: {len(test_data)}")

tokens = tokenizer.convert_ids_to_tokens(vars(train_data.examples[0])['text'])

LABEL.build_vocab(train_data, valid_data)

print(LABEL.vocab.stoi)

BATCH_SIZE = 128

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print("Device in use:",device)
train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    sort_key=lambda x: len(x.text),
    batch_size = BATCH_SIZE,
    device = device)

print('Iterators created')

print('Downloading XLM Roberta model...')

from transformers import XLMRobertaModel
bert = XLMRobertaModel.from_pretrained('xlm-roberta-base')

import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
print('XLM Roberta model downloaded')

OUTPUT_DIM = 3
DROPOUT = 0.3
N_FILTERS = 100
FILTER_SIZES = [2,3,4]
HIDDEN_DIM = 100

model_names = ["CNN_Model", "Attention_Model"]
models = [  BERTCNNSentiment(bert, OUTPUT_DIM, DROPOUT, N_FILTERS, FILTER_SIZES),
            AttentionModel(bert, BATCH_SIZE, OUTPUT_DIM, HIDDEN_DIM, 50000, 768)  ]

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

for i in range(2):
    print(f'The {models[i]} has {count_parameters(models[i]):,} trainable parameters')

for i in range(2):
    print("Parameters for " + f'{model_names[i]}')
    for name, param in models[i].named_parameters():
        if param.requires_grad:
            print(name)

import torch.optim as optim
from sklearn.metrics import confusion_matrix
def clip_gradient(model, clip_value):
    params = list(filter(lambda p: p.grad is not None, model.parameters()))
    for p in params:
        p.grad.data.clamp_(-clip_value, clip_value)

optimizers = [optim.Adam(models[0].parameters()), optim.Adam(models[1].parameters())]

criterion = nn.CrossEntropyLoss()
nll_loss = nn.NLLLoss()
log_softmax = nn.LogSoftmax()

for i in range(2):
    models[i] = models[i].to(device)
criterion = criterion.to(device)
nll_loss = nll_loss.to(device)
log_softmax = log_softmax.to(device)

from sklearn.metrics import f1_score

def categorical_accuracy(preds, y):
    count0,count1,count2 = torch.zeros(1),torch.zeros(1),torch.zeros(1)
    count0 = torch.zeros(1).to(device)
    count1 = torch.zeros(1).to(device)
    count2 = torch.zeros(1).to(device)
    total0,total1,total2 = torch.FloatTensor(1),torch.FloatTensor(1),torch.FloatTensor(1)
    max_preds = preds.argmax(dim = 1, keepdim = True) # get the index of the max probability
    correct = max_preds.squeeze(1).eq(y)
    predictions = max_preds.squeeze(1)
    true_correct = [0,0,0]
    for j,i in enumerate(y.cpu().numpy()):
      true_correct[y.cpu().numpy()[j]]+=1
      if i==0:
        count0+=correct[j]
        total0+=1
      elif i==1:
        count1+=correct[j]
        total1+=1
      elif i==2:
        count2+=correct[j]
      else:
        total2+=1
    metric=torch.FloatTensor([count0/true_correct[0],count1/true_correct[1],count2/true_correct[2],f1_score(y.cpu().numpy(),predictions.cpu().numpy(),average='macro')])
    return correct.sum() / torch.FloatTensor([y.shape[0]]),metric,confusion_matrix(y.cpu().numpy(),max_preds.cpu().numpy())

def train(model, iterator, optimizer, criterion, i):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:
        optimizer.zero_grad()

        if (i == 0):
          predictions =  model(batch.text).squeeze(1)
        else:
          predictions =  model(batch.text, batch_size = len(batch)).squeeze(1)

        loss = criterion(predictions, batch.label)

        acc,_,_ = categorical_accuracy(predictions, batch.label)

        loss.backward()
        clip_gradient(model, 1e-1)
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion, i):

    epoch_loss = 0
    epoch_acc = 0
    epoch_all_acc = torch.FloatTensor([0,0,0,0])
    confusion_mat = torch.zeros((3,3))
    confusion_mat_temp = torch.zeros((3,3))

    model.eval()

    with torch.no_grad():

        for batch in iterator:
            if (i == 0):
              predictions = model(batch.text).squeeze(1)
            else:
              predictions = model(batch.text,batch_size=len(batch)).squeeze(1)

            loss = criterion(predictions, batch.label)

            acc,all_acc,confusion_mat_temp = categorical_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
            epoch_all_acc += all_acc
            confusion_mat+=confusion_mat_temp
    return epoch_loss / len(iterator), epoch_acc / len(iterator),epoch_all_acc/len(iterator),confusion_mat

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

N_EPOCHS = 40

best_f1 = [-1, -1]
for epoch in range(N_EPOCHS):
    for i in range(2):
        start_time = time.time()

        train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
        valid_loss, valid_acc,tot,conf = evaluate(models[i], valid_iterator, criterion, i)
        f1 = tot[3]
        end_time = time.time()

        epoch_mins, epoch_secs = epoch_time(start_time, end_time)

        if f1 > best_f1[i]:
            best_f1[i] = f1

            path = model_save_names[i]
            print(path)
            torch.save(models[i].state_dict(), path)

        print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
        print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
        print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
        print(tot)
        print(conf)

for i in range(2):
    path = model_save_names[i]
    models[i].load_state_dict(torch.load(path))

def ensemble_evaluate(models, iterator, criterion):

    epoch_loss = 0
    epoch_acc = 0
    epoch_all_acc = torch.FloatTensor([0,0,0,0])
    models[0].eval()
    models[1].eval()
    confusion_mat = torch.zeros((3,3))
    confusion_mat_temp = torch.zeros((3,3))

    with torch.no_grad():

        for batch in iterator:

          predictions0 = models[0](batch.text).squeeze(1)
          predictions1 = models[1](batch.text, batch_size=len(batch)).squeeze(1)

          predictions = F.softmax(predictions0, dim=1) * F.softmax(predictions1, dim=1)
          loss = criterion(predictions, batch.label)

          acc,all_acc,confusion_mat_temp = categorical_accuracy(predictions, batch.label)
          epoch_loss += loss.item()
          epoch_acc += acc.item()
          epoch_all_acc += all_acc
          confusion_mat += confusion_mat_temp
    print(confusion_mat)
    return epoch_loss / len(iterator), epoch_acc / len(iterator),epoch_all_acc/len(iterator)

def ensemble_write_to_file(models, test_iterator):
    label_dict = {'0':'negative', '1':'neutral', '2':'positive'}
    file = open("answer.txt", "w")
    file.write('Uid,Sentiment\n')
    count = 0
    for batch in test_iterator:
      predictions0 = models[0](batch.text).squeeze(1)
      predictions1 = models[1](batch.text, batch_size=len(batch)).squeeze(1)
      predictions = F.softmax(predictions0, dim=1) * F.softmax(predictions1, dim=1)
      max_preds = predictions.argmax(dim = 1, keepdim = True).detach().cpu().numpy()
      for i,row in enumerate(batch.uid.cpu().numpy()):
        count += 1
        label_number = max_preds[i][0]
        label_number_str = list(LABEL.vocab.stoi.keys())[list(LABEL.vocab.stoi.values()).index(label_number)]
        predicted_label_name = label_dict[label_number_str]
        if count != len(test_data):
          file.write('%s,%s\n'%(row,predicted_label_name))
        else:
          file.write('%s,%s'%(row,predicted_label_name))
    file.close()

valid_loss, valid_acc, tot = ensemble_evaluate(models, test_iterator, criterion)
print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
print(tot)

Here is the output which is I am getting. I have shown only the requird output necessary for debugging
Traceback (most recent call last):
  File "main.py", line 268, in <module>
    train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
  File "main.py", line 212, in train
    acc,_,_ = categorical_accuracy(predictions, batch.label)
  File "main.py", line 184, in categorical_accuracy
    count1+=correct[j]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Can someone tell me where to make the appropriate changes so that the code may run on the GPU in Google Colab.
Edit_1:
Thank you so much for the help you provided it actually solved my problem, but now a similar new problem arised. Note that I have updated the code to include the changes you mentioned
Traceback (most recent call last):
  File "main.py", line 271, in <module>
    train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
  File "main.py", line 215, in train
    acc,_,_ = categorical_accuracy(predictions, batch.label)
  File "main.py", line 194, in categorical_accuracy
    return correct.sum() / torch.FloatTensor([y.shape[0]]),metric,confusion_matrix(y.cpu().numpy(),max_preds.cpu().numpy())
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Output:
The problem is exactly as the error says, Pytorch expects all operations to be done in the same device but the two tensors you are adding are in different places.
You need to add .to(device) to these variables
    count0,count1,count2 = torch.zeros(1),torch.zeros(1),torch.zeros(1)

Like
count0 = torch.zeros(1).to(device)
count1 = torch.zeros(1).to(device)
count2 = torch.zeros(1).to(device)

But just in case if there is NameError: name 'device' is not defined you can just use the y's or pred's device like
device = y.device

count0 = torch.zeros(1).to(device)
count1 = torch.zeros(1).to(device)
count2 = torch.zeros(1).to(device)


Instruction: pytorch modifying the input data to forward to make it suitable to my model

Input:
Here is what I want to do.
I have an individual data of shape (20,20,20) where 20 tensors of shape (1,20,20) will be used as an input for 20 separate CNN. Here's the code I have so far.
class MyModel(torch.nn.Module):
   def __init__(self, ...):
       ...
        self.features = nn.ModuleList([nn.Sequential(
            nn.Conv2d(1,10, kernel_size = 3, padding = 1),
            nn.ReLU(),
            nn.Conv2d(10, 14, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(14, 18, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(28*28*18, 256)
        ) for _ in range(20)])
        self.fc_module = nn.Sequential(
            nn.Linear(256*n_selected, cnn_output_dim),
            nn.Softmax(dim=n_classes)
        )

    def forward(self, input_list):
        concat_fusion = cat([cnn(x) for x,cnn in zip(input_list,self.features)], dim = 0)
        output = self.fc_module(concat_fusion)
        return output

The shape of the input_list in forward function is torch.Size([100, 20, 20, 20]), where 100 is the batch size.
However, there's an issue with
concat_fusion = cat([cnn(x) for x,cnn in zip(input_list,self.features)], dim = 0)

as it results in this error.
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [10, 1, 3, 3], but got 3-dimensional input of size [20, 20, 20] instead

First off, I wonder why it expects me to give 4-dimensional weight [10,1,3,3]. I've seen
"RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead"?
but I'm not sure where those specific numbers are coming from.

I have an input_list which is a batch of 100 data. I'm not sure how I can deal with individual data of shape (20,20,20) so that I can actually separate this into 20 pieces to use it as an independent input to 20 CNN.

Output:
why it expects me to give 4-dimensional weight [10,1,3,3].
Note the following log means the nn.Conv2d with kernel (10, 1, 3, 3) requiring a 4 dimensional input.
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [10, 1, 3, 3]

How to separate input into 20 pieces along channels.
Iteration over input_list(100, 20, 20, 20) produces 100 tensors of shape (20, 20, 20).
If you want to split input along channel, try to slice input_list along second dimension.
concat_fusion = torch.cat([cnn(input_list[:, i:i+1]) for i, cnn in enumerate(self.features)], dim = 1)


Instruction: OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

Input:
I am trying to run a model on TPU as given in colab notebook. The model was working fine, but today I could not run the model.
I used the following code to install pytorch-xla.
VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

I try to install required libraries as below:
!pip install -U nlp
!pip install sentencepiece
!pip install numpy --upgrade

However, when I try the following
import nlp

It gives the following error:
OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

I searched the error and I tried the followings, but still does not work. Any ideas how to fix it? Note: It was working a few days ago, however, today it is not.
!pip install mkl
#!export PATH="$PATH:/opt/intel/bin"
#!export LD_LIBRARY_PATH="$PATH:opt/intel/mkl/lib/intel64_lin/"
!export LID_LIBRAEY_PATH="$LID_LIBRARY_PATH:/opt/intel/mkl/lib/intel64_lin/"

Output:
import os
os.environ['LD_LIBRARY_PATH']='/usr/local/lib'

!echo $LD_LIBRARY_PATH
!sudo ln -s /usr/local/lib/libmkl_intel_lp64.so /usr/local/lib/libmkl_intel_lp64.so.1
!sudo ln -s /usr/local/lib/libmkl_intel_thread.so /usr/local/lib/libmkl_intel_thread.so.1
!sudo ln -s /usr/local/lib/libmkl_core.so /usr/local/lib/libmkl_core.so.1

!ldconfig
!ldd /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch.so

worked for me. We will also try to fix the problem internally.