Instruction: I'm having a problem trying to load a Pytoch model: "Can't find Identity in module"

Input:
When trying to load a pytorch model it gives the following attribute error
model = torch.load('../input/melanoma-model/melanoma_model_0.pth')
model = model.to(device)
model.eval()

AttributeError                            Traceback (most recent call
last)  in
1 arch = EfficientNet.from_pretrained('efficientnet-b2')
2 model = Net(arch=arch)
----> 3 torch.load('../input/melanoma-model/melanoma_model_0.pth')
4 model = model.to(device)
5 model.eval()
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in
load(f, map_location, pickle_module, **pickle_load_args)
591                     return torch.jit.load(f)
592                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 593         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
594
595
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in
_legacy_load(f, map_location, pickle_module, **pickle_load_args)
771     unpickler = pickle_module.Unpickler(f, **pickle_load_args)
772     unpickler.persistent_load = persistent_load
--> 773     result = unpickler.load()
774
775     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)
AttributeError: Can't get attribute 'Identity' on <module
'efficientnet_pytorch.utils' from
'/opt/conda/lib/python3.7/site-packages/efficientnet_pytorch/utils.py'>

Output:
First you need a model class to load the parameters from the .pth into. And you are missing one step:
model = Model()   # the model class (yours has probably another name)
model.load_state_dict(torch.load('../input/melanoma-model/melanoma_model_0.pth'))
model = model.to(device)
model.eval()

There you go, I hope that solved your problem!


Instruction: syntax pytorch.nn functions neural networks, class or functions?

Input:
I am puzzled by the syntax of the functions used in neural networks in pytorch.
Here is an example of how one can define a linear transformation layer: (cf. https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)
m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
print(output.size())
torch.Size([128, 30])

Can someone explain me where the expression nn.Linear(20,30)(input) comes from ? It disturbs me a bit.
Indeed, one can define a class neural network with such cosntructor : (for example)
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=0):
        super(NeuralNet, self).__init__()
        self.fc1            = nn.Linear(input_size, hidden_size, bias=True)
        self.fc2            = nn.Linear(hidden_size, hidden_size, bias=True)
        self.fc3            = nn.Linear(hidden_size, hidden_size, bias=True)
        self.fc4            = nn.Linear(hidden_size, num_classes, bias=False)
        self.dropout        = nn.Dropout(p=p)

and I was trying to write an attribute using numpy function, like:
  self.enter_reshape  = np.reshape(-1, input_size * input_size)
  self.exit_reshape   = np.reshape(input_size, num_classes / input_size)

or, using the view function from pytorch:
self.reshape = view(-1, self.num_flat_features())

The closest thing I know about is the partial function and closures, where one could write f(z)(x)(y). I looked into the definition of Linear, and I saw that linear is an object, but I don't see where they redefined __call__ magic function, which I thought would be used here when one calls the object.
So basically, can one explain what is up with such writting, and also, would it be possible to give to the neural network the numpy or view functions as attributes?

Output:
torch.nn.Linear inherits from torch.nn.Module (see source code), which in turn defines __call__ method.
You can see source code for torch.nn.Module here. This class allows users to make their own Modules by inheritance (as you did in your example) and is a base for all PyTorch defined modules like nn.Linear (see available methods, documentation here).
Its __call__ essentially calls forward but running registered hooks (and registering), checking torchscript etc. (see source code here, with relevant line here.

would it be possible to give to the neural network the numpy or view
functions as attributes?

From the example you've given, what you are trying to do is probably partial function (or lambda as in the example below) saved as attribute (though that is pretty uncommon and never seen it tbh), like this:
import torch

class MyModule(torch.nn.Module):
    def __init__(self, shape: int = -1):
        super().__init__()  # required
        self.reshape = lambda tensor: torch.reshape(tensor, (shape,))

    def forward(self, tensor):
        return self.reshape(tensor)

module = MyModule()
module(torch.randn(4, 5, 6)).shape # [120] shape

You shouldn't use numpy with pytorch unless you really need it and/or there is no sensible pytorch counterpart (although you can if you transform torch.Tensor to numpy). Also you shouldn't do anything like the code above as it's really confusing, just save attributes (anything like input_shape, hidden_dim, output_size) and use it in forward:
class MyModule(torch.nn.Module):
    def __init__(self, shape: int = -1):
        super().__init__()  # required
        self.shape = shape

    def forward(self, tensor):
        return torch.reshape(tensor, (self.shape,))


Instruction: Perform Delta Function between elements in PyTorch tensor

Input:
I have a 1 dimensional pyTorch tensor (dtype: int32) and was wondering if there was a way to perform a Dirac Delta function on the elements in this tensor, i.e:
f = tensor[1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1]
f_after_dirac_delta = tensor[0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1]

Thanks for any help in advance!
EDIT: as @GirishDattatrayHegde mentioned, the term Dirac-Delta was misleading. The correct term should have been a Kronecker-Delta. My apologies.

Output:
If I understand correctly, you want to compare successive elemeents of your tensor. This should work :
import torch
f = torch.tensor([1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1])
f_dirac = (f[1:] == f[:-1]).to(torch.long)


Instruction: How to rotate a Torch Tensor by a random number of degrees

Input:
as part of training a CNN, I am working with an array inputs that contain <class 'torch.Tensor'> objects. I want to rotate an individual <class 'torch.Tensor'> object by some random number of degrees x, as shown here:
def rotate(inputs, x):
    # Rotate inputs[0] by x degrees, x can take on any value from 0 - 180 degrees

How can I do this? For existing implementations, I can only find that torch has a rot90 function, but that limits me to multiples of 90 degrees which does not help my scenario.
Thanks, Vinny

Output:
To transform an torch.tensor you can use scipy.ndimage.rotate function (read here),that rotates a torch.tensor but also it converts it to numpy.ndarray, so you have to convert it back to torch.tensor. See this toy example.
Function:
def rotate(inputs, x):
    return torch.from_numpy(ndimage.rotate(inputs, x, reshape=False))

Detailed explanation:
import torch
from scipy import ndimage
alpha = torch.rand(3,3)
print(alpha.dtype)#torch.float32

angle_in_degrees = 45
output = ndimage.rotate(alpha, angle_in_degrees, reshape=False)

print(output.dtype) #numpy_array

output = torch.from_numpy(output) #convert it back to torch tensor

print(output.dtype)  #torch.float32

Also, You can directly transform PIL image before converting it to tensor if that's a possibility. To transform PIL image you can use PyTorch builtin torchvision.transforms.functional.rotate (read here).


Instruction: Python PyTorch Pyro - Multivariate Distributions

Input:
How does one sample a multivariate distribution in Pyro? I just want a (M, N) Beta distribution, but the following doesn't work:
impor torch
import pyro
with pyro.plate("theta_plate", M):
    theta = pyro.sample("theta",
                        pyro.distributions.Beta(concentration0=torch.ones(N),
                                                concentration1=torch.ones(N)))

Output:
Use to_event(n) to declare depdent samples.
import torch
import pyro
import pyro.distributions as dist

def model(N, M):
    with pyro.plate("theta_plate", M):
        theta = pyro.sample("theta", dist.Beta(torch.ones(N),1.).to_event(1))
    return theta

if __name__ == '__main__':
    print(model(10,12).shape) # (10,12)


Instruction: Non-MNIST Digit Recognition Pytorch

Input: I'm looking for examples of pytorch being used to classify non-MNIST digits. After hours of searching, it appears the algorithms are against me. Does anyone have a good example? Thanks.

Output:
I am posting this as answer since i do not have the rep to comment,
Please view the google street view dataset (SVHN). It is like MNIST but there is much more noise present in the data. Another option for you could be to use GANs and make more images which practically wouldn't have existed before. You could also try your hand at non - english mnist data-sets (though it moves away from your original goal).
Link to SVHN with pytorch: https://github.com/potterhsu/SVHNClassifier-PyTorch
Link to original SVHN: https://pytorch.org/docs/stable/torchvision/datasets.html#svhn
P.S. You could also try making a dataset on your own! This is quite fun to do.


Instruction: Reset parameters of a neural network in pytorch

Input:
I have a neural network with the following structure:
class myNetwork(nn.Module):
    def __init__(self):
        super(myNetwork, self).__init__()
        self.bigru = nn.GRU(input_size=2, hidden_size=100, batch_first=True, bidirectional=True)
        self.fc1 = nn.Linear(200, 32)
        torch.nn.init.xavier_uniform_(self.fc1.weight)
        self.fc2 = nn.Linear(32, 2)
        torch.nn.init.xavier_uniform_(self.fc2.weight)

I need to reinstate the model to an unlearned state by resetting the parameters of the neural network. I can do so for nn.Linear layers by using the method below:
def reset_weights(self):
    torch.nn.init.xavier_uniform_(self.fc1.weight)
    torch.nn.init.xavier_uniform_(self.fc2.weight)

But, to reset the weight of the nn.GRU layer, I could not find any such snippet.
My question is how does one reset the nn.GRU layer? Any other way of resetting the network is also fine. Any help is appreciated.

Output:
You can use reset_parameters method on the layer. As given here
for layer in model.children():
   if hasattr(layer, 'reset_parameters'):
       layer.reset_parameters()

Or Another way would be saving the model first and then reload the module state. Using torch.save and torch.load see docs for more Or  Saving and Loading Models


Instruction: How can I import/use two different versions of a library (pytorch) in one program in Python?

Input:
I need to use two different versions of pytorch in different parts of the same python webserver. Unfortunately, I can't install them both on the same conda environment that I'm using. I've tried importing one of them from the path itself:
MODULE_PATH = "/home/abc/anaconda3/envs/env/lib/python3.7/site-packages/torch/__init__.py"
MODULE_NAME = "torch"
import importlib
import sys
spec = importlib.util.spec_from_file_location(MODULE_NAME, MODULE_PATH)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)

Which works fine for importing a different version than the one in the active environment, but then I run into an error when trying to import the second one (I've tried simply 'import torch' and also the same as above):
File "/home/abc/anaconda3/envs/env2/lib/python3.7/site-packages/torch/__init__.py", line 82, in <module>
    __all__ += [name for name in dir(_C)
NameError: name '_C' is not defined

Any ideas on how I can use both versions? Thanks!

Output:
In principle, importing two libraries with the same name is not possible. Sure, it might be the case that you could do some import-sorcery and manage to do it. But keep in mind that pytorch is not a straightforward Python package.
Now, even if you manage to solve this, it seems extremely strange to me that you need for your own service two different versions. Having that situation will just be a headache for you in the long run. My advice would be to reconsider how you're doing it.
Without knowing your situation, I'd recommend splitting the web service into two. This will allow you to have two environments and the two versions of pytorch you need.


Instruction: Why is my loss function always returning nan?

Input:
I'm trying to learn about ML and I tried to make a simple linear model, but when I run it, loss comes out as null:

So I tried to find whats the problem about. If I print first 10 y_pred, only about 17 of them have numbers, the rest of them are null. Maybe im doing something wrong, please help.
import torch
from torch.utils.data import TensorDataset, DataLoader
import torch.nn as nn
import numpy as np

#Input (temp, rainfall, humidity)
inputs = np.array([[73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70], [73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70], [73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70]], dtype='float32')

#Target (apples, oranges)
targets = np.array([[56, 70], [81, 101], [119, 133], [22, 37], [103, 119], [56, 70], [81, 101], [119, 133], [22, 37], [103, 119], [56, 70], [81, 101], [119, 133], [22, 37], [103, 119]], dtype='float32')

inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)

#Define datasets
train_ds = TensorDataset(inputs, targets)
train_ds[0:3]

#Hyperparameters
batch_size = 5
num_epochs = 100
learning_rate = 0.01

train_dl = DataLoader(dataset=train_ds, batch_size=batch_size, shuffle=True)

model = nn.Linear(3,2) #inputs(temp, rainfall, humidity) , targets(apples, oranges)

loss_f = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for xb, yb in train_dl:
        y_pred = model(xb)
        loss = loss_f(y_pred, yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    if(epoch+1) % 10 == 0:
        print(f'epoch = {epoch+1}/{num_epochs}, loss = {loss.item():.4f}')

print(f'Final loss = {loss.item():.4f}')

edit:
y_pred.shape = torch.Size([5, 2]), yb.shape = torch.Size([5, 2])

Output:
It is not about the Loss function. Your model is predicting Nan and Infy numbers. Possible solution

Reduce the learning rate (Ex.:learning_rate = 0.001) or
Reduce batch size (Ex.: batch_size = 2) or
Add more layers into the model with activation functions or
Normalize inputs


Instruction: Using PyTorch with Celery

Input:
I'm trying to run a PyTorch model in a Django app. As it is not recommended to execute the models (or any long-running task) in the views, I decided to run it in a Celery task. My model is quite big and it takes about 12 seconds to load and about 3 seconds to infer. That's why I decided that I couldn't afford to load it at every request. So I tried to load it at settings and save it there for the app to use it. So my final scheme is:

When the Django app starts, in the settings the PyTorch model is loaded and it's accessible from the app.
When views.py receives a request, it delays a celery task
The celery task uses the settings.model to infer the result

The problem here is that the celery task throws the following error when trying to use the model
[2020-08-29 09:03:04,015: ERROR/ForkPoolWorker-1] Task app.tasks.task[458934d4-ea03-4bc9-8dcd-77e4c3a9caec] raised unexpected: RuntimeError("Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method")
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  /*...*/
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/torch/cuda/__init__.py", line 191, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Here's the code in my settings.py loading the model:
if sys.argv and sys.argv[0].endswith('celery') and 'worker' in sys.argv: #In order to load only for the celery worker
    import torch
    torch.cuda.init()
    torch.backends.cudnn.benchmark = True
    load_model_file()

And the task code
@task
def getResult(name):
    print("Executing on GPU:", torch.cuda.is_available())
    if os.path.isfile(name):
        try:
            outpath = model_inference(name)
            os.remove(name)
            return outpath
        except OSError as e:
            print("Error", name, "doesn't exist")
    return ""

The print in the task shows "Executing on GPU: true"
I've tried setting torch.multiprocessing.set_start_method('spawn') in the settings.py before and after the torch.cuda.init() but it gives the same error.

Output:
Setting this method works as long as you're also using Process from the same library.
from torch.multiprocessing import Pool, Process

Celery uses "regular" multiprocessing library, thus this error.
If I were you I'd try either:

run it single threaded to see if that helps
run it with eventlet to see if that helps
read this


Instruction: About pytorch the output is not what I expect

Input:
import torch
a = torch.randn(2, 2)
a = ((a * 3) / (a - 1))
print(a)

The output is:
tensor([[  -0.7242,    2.0021],
        [-280.8320,    0.6750]])

But I think it should be:
tensor([[  6,    6],
        [  6,    6]])

Why I'm wrong?

Output:
torch.randn will preduce a random 2X2 tensor, as stated in the docs

Returns a tensor filled with random numbers from a normal distribution
with mean 0 and variance 1 (also called the standard normal
distribution).

In your case you will get the desired output if you will generate a tensor where all the values are 2.
a = torch.ones([2, 2], dtype=torch.float64) * 2.

will give you the desired one.


Instruction: About pytorch, I can't understand why that's the output

Input:
The code is as below about pytorch, it's about derivative, I think the output is 18 but it's 4.5, I don'k know why:
import torch
x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(x.grad)

The output:
tensor([[4.5000, 4.5000],
[4.5000, 4.5000]])
I think the derivative is 2*3*(1+2), so it should be:
tensor([[18, 18],
        [18, 18]])

Why the output is 4.5?
Some people think it's the mean method that make the derivative /4, but when I do the code "print(out)", the output is "tensor(27., grad_fn=)" rather than (4.5., grad_fn=), I'm a new learning of pytorch, so I don't know what it does with "tensor.mean()", but since the output of "print(out)" is 27, so I don't think there is a "/4" process in "tensor.mean()", so I don't think it should include "/4" process in the derivative computation, is that correct?(Please help me~)

Output:
Here's how I think it goes:
y = x + 2 and z = y * y * 3,
so z is 3 * (x+2)^2
Next, out = z.mean(), or sigma z / n which is sigma z/4 since we had a total of 4 numbers in z.
Thus, you find the derivative of sigma (3 * (x + 2)^2)/4 at x = 1.
That gives (3/4) * 2(x + 2) at x = 1, which is 4.5
So I think you had it all figured out, except that in the last step you missed dividing by 4, which you need to do since there's a mean() function in there.
Edit:
Since you're confused with how the mean() is affecting the output, let's do a tensor of values [1,2,3,4] instead of torch.ones() to see the effect.
x = torch.tensor([1.0,2.0,3.0,4.0], requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(x.grad)

this will output
tensor([4.5000, 6.0000, 7.5000, 9.0000])

How? Remember we derived the equation for our derivative to be:
(3/4) * 2(x + 2)
 Now you substitute x to be 1, and get 4.500.
 Then for x = 2, you get 6.000, for x = 3 you get 7.500 and so on.
In the earlier example, you had four instances of x = 1 which is why you had x.grad to be [[4.5, 4.5], [4.5, 4.5]]


Instruction: Custom CNN gives wrong output shape

Input:
I need some help. I am trying to make a custom CNN, which should accept one channel images and do binary classification. This is the model:
class custom_small_CNN(nn.Module):

    def __init__(self, input_channels=1, output_features=1):
        super(custom_small_CNN, self).__init__()

        self.input_channels = input_channels
        self.output_features = output_features

        self.conv1 = nn.Conv2d(self.input_channels, 8, kernel_size=(7, 7), stride=(2, 2), padding=(6, 6), dilation=(2, 2))
        self.conv2 = nn.Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1))
        self.pool = nn.MaxPool2d(kernel_size=(2, 2))
        self.fc1 = nn.Linear(in_features=1024, out_features=self.output_features, bias=True)
        self.dropout = nn.Dropout(p=0.5)
        self.softmax = nn.Softmax(dim=1)
        self.net_name = 'Custom_Small_CNN'

        self.net = nn.Sequential(self.conv1, self.pool, self.conv2, self.pool, self.fc1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.pool(x)
        #x = self.dropout(x)
        x = self.conv2(x)
        x = self.pool(x)
        x = x.view(-1, 1024)
        x = self.dropout(x)
        x = self.fc1(x)
        if not self.output_features == 1:
            x = self.softmax(x)
        return x

However, when I put an example batch with 4 images (all zeros) in the model like this:
x = torch.from_numpy(np.zeros((4, 1, 256, 256))).float()
net = custom_small_CNN(output_features=2, input_channels=1).float()
output = net(x)

the output has shape torch.Size([16, 2]) instead of torch.Size([4, 2]), which is what I want and what e.g. a ResNet delivers as an output. What am I missing?
Thanks!

Output:
When you apply pooling layer, it returns (batch_size, 2, 2, num_filters), so when you reshape it x = x.view(-1, 1024), it results in (batch_size * 4, num_filters) as shape.
Instead of reshaping like that you should either flatten or average the output of pooling layer. Flattening is most commonly used here.
So, replacing following line
x = x.view(-1, 1024)

with
x = nn.Flatten()(x)

would result in correct final output shape


Instruction: How to Find Confusion Matrix and Plot it for Image Classifier in PyTorch

Input:
Basically this is the VGG-16 Model, I have performed Transfer Learning and Fine Tuned the model, I have trained this model 2 weeks ago and found both the test and train accuracy but now I need Class wise accuracy of the model too, I am trying to find out the Confusion matrix and wanna plot the matrix too. Training Code:
# Training the model again from the last CNN Block to The End of the Network
dataset = 'C:\\Users\\Sara Latif Khan\\OneDrive\\Desktop\\FYP_\\Scene15\\15-Scene'
model = model.to(device)
optimizer = Adam(filter(lambda p: p.requires_grad, model.parameters()))

#Training Fixed Feature Extractor for 15 epochs
num_epochs = 5
batch_loss = 0
cum_epoch_loss = 0 #cumulative loss for each batch

for e in range(num_epochs):
    cum_epoch_loss = 0

    for batch, (images, labels) in enumerate(trainloader,1):
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()
        logps = model(images)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()

        batch_loss += loss.item()
        print(f'Epoch({e}/{num_epochs} : Batch number({batch}/{len(trainloader)}) : Batch loss : {loss.item()}')
        torch.save(model, dataset+'_model_'+str(e)+'.pt')

print(f'Training loss : {batch_loss/len(trainloader)}')

This is the code I am using to check the accuracy of my model based on data from the test loader.
model. to('cpu')

model.eval()
with torch.no_grad():
    num_correct = 0
    total = 0

    #set_trace ()
    for batch, (images,labels) in enumerate(testloader,1):

        logps = model(images)
        output = torch.exp(logps)

        pred = torch.argmax(output,1)
        total += labels.size(0)

        num_correct += (pred==labels).sum().item()
        print(f'Batch ({batch} / {len(testloader)})')

        # to check the accuracy of model on 5 batches
        # if batch == 5:
            # break

    print(f'Accuracy of the model on {total} test images: {num_correct * 100 / total }% ')

Next, I need to find the class-wise accuracy of the model. I am working on the Jupyter Notebook. Should I reload a saved model and find the cm or what will the appropriate way of doing it.

Output:
You have to save all the predictions and targets of the test set.
predictions, targets = [], []
for images, labels in testloader:
    logps = model(images)
    output = torch.exp(logps)
    pred = torch.argmax(output, 1)

    # convert to numpy arrays
    pred = pred.detach().cpu().numpy()
    labels = labels.detach().cpu().numpy()

    for i in range(len(pred)):
        predictions.append(pred[i])
        targets.append(labels[i])

Now you have all the predictions and actual targets of the test-set stored.
Next step is to create the confusion matrix. I think I can just give you my function I always use:
def create_confusion_matrix(y_true, y_pred, classes):
    """ creates and plots a confusion matrix given two list (targets and predictions)
    :param list y_true: list of all targets (in this case integers bc. they are indices)
    :param list y_pred: list of all predictions (in this case one-hot encoded)
    :param dict classes: a dictionary of the countries with they index representation
    """

    amount_classes = len(classes)

    confusion_matrix = np.zeros((amount_classes, amount_classes))
    for idx in range(len(y_true)):
        target = y_true[idx][0]

        output = y_pred[idx]
        output = list(output).index(max(output))

        confusion_matrix[target][output] += 1

    fig, ax = plt.subplots(1)

    ax.matshow(confusion_matrix)
    ax.set_xticks(np.arange(len(list(classes.keys()))))
    ax.set_yticks(np.arange(len(list(classes.keys()))))

    ax.set_xticklabels(list(classes.keys()))
    ax.set_yticklabels(list(classes.keys()))

    plt.setp(ax.get_xticklabels(), rotation=45, ha="left", rotation_mode="anchor")
    plt.setp(ax.get_yticklabels(), rotation=45, ha="right", rotation_mode="anchor")

    plt.show()

So y_true are all the targets, y_pred all the predictions and classes is a dictionary which maps the labels to the actual class-names, for example:
classes = {"dog": [1, 0], "cat": [0, 1]}

Then simply call:
create_confusion_matrix(targets, predictions, classes)

Probably you will have to adapt it to your code a little but I hope this works for you. :)


Instruction: Pytorch crossentropy loss with 3d input

Input:
I have a network which outputs a 3D tensor of size (batch_size, max_len, num_classes). My groud truth is in the shape (batch_size, max_len). If I do perform one-hot encoding on the labels, it'll be of shape (batch_size, max_len, num_classes) i.e the values in max_len are integers in the range [0, num_classes]. Since the original code is too long, I have written a simpler version that reproduces the original error.
criterion = nn.CrossEntropyLoss()
batch_size = 32
max_len = 350
num_classes = 1000
pred = torch.randn([batch_size, max_len, num_classes])
label = torch.randint(0, num_classes,[batch_size, max_len])
pred = nn.Softmax(dim = 2)(pred)
criterion(pred, label)

the shape of pred and label are respectively,torch.Size([32, 350, 1000]) and torch.Size([32, 350])
The error encountered is
ValueError: Expected target size (32, 1000), got torch.Size([32, 350, 1000])

If I one-hot encode labels for computing the loss
x = nn.functional.one_hot(label)
criterion(pred, x)

it'll throw the following error
ValueError: Expected target size (32, 1000), got torch.Size([32, 350, 1000])

Output: From the Pytorch documentation, CrossEntropyLoss expects the shape of its input to be (N, C, ...), so the second dimension is always the number of classes. Your code should work if you reshape preds to be of size (batch_size, num_classes, max_len).


Instruction: how to reduce the second dimension in pytorch python tensor operation

Input:
I have the below tensor with the following shape
    seq = dataset['features'][...]
    print(f'shape of seq before unsequeeze {seq.shape}')
    shape of seq before unsequeeze (461, 1024)

I am trying to convert the shape in (461, 512) How should I achieve this in  pytorch tensor operation.
examples feature x as below,
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
x = torch.tensor([[0.1,0.8,0.8,0.4,0.9,0.4,0.4,0.5,0.4,0.2,0.3,0.8,0.2,0.7,0.6,0.1,0.2,0.1,0.1,0.4,0.2,0.1,0.0,0.5,0.2,0.4,0.3,0.3,0.7,0.1,0.4,0.6,0.5,1.0,0.1,0.8,0.9,0.0,0.2,0.9,0.8,0.0,0.9,0.7,0.2,0.2,0.9,0.6,0.1,0.2,0.6,0.0,0.1,0.1,0.3,0.5,0.8,0.8,0.4,0.4,0.7,0.7,0.4,0.2,0.1,1.0,0.3,0.8,0.1,0.7,0.7,0.9,0.6,0.3,0.8,0.2,0.9,0.6,0.7,0.8,0.2,0.1,1.0,0.6,0.5,0.5,0.5,0.8,0.8,0.3,0.1,0.2,0.5,0.9,0.6,0.8,0.0,0.6,0.2,0.1,0.8,0.4,0.8,0.5,0.8,0.4,0.7,0.6,0.8,0.1,0.4,0.8,1.0,0.9,0.4,0.4,0.4,0.1,0.7,0.3,0.8,0.6,0.4,0.5,0.9,0.1,0.9,0.7,0.4,0.7,0.1,0.8,0.2,0.2,0.7,0.2,0.9,0.6,0.2,0.9,0.1,0.9,0.2,1.0,0.9,0.6,0.3,0.6,0.9,0.6,0.0,0.3,0.4,0.6,0.7,0.9,0.2,0.6,0.2,0.5,0.3,0.3,0.4,0.4,0.1,0.2,0.6,0.0,0.7,0.5,0.5,0.2,0.5,0.6,0.5,0.5,0.7,0.8,0.4,0.5,0.8,0.8,0.1,0.5,0.7,0.8,0.1,0.1,0.8,0.6,0.6,0.4,1.0,0.4,0.6,0.9,0.1,0.6,0.3,1.0,0.7,0.2,0.5,0.5,1.0,0.5,0.4,0.3,0.7,0.1,1.0,0.9,0.4,0.6,0.6,0.6,0.2,0.0,0.9,0.9,0.2,0.1,0.5,0.5,0.8,0.7,0.8,0.0,0.0,0.1,0.5,0.5,0.5,0.8,0.1,0.5,1.0,0.3,0.2,0.8,0.9,0.4,0.4,0.9,0.2,0.4,0.9,0.9,0.3,0.7,0.4,0.9,0.5,0.7,0.8,0.5,0.5,0.5,0.8,0.7,0.9,0.2,0.8,1.0,0.1,0.9,0.6,0.5,0.0,0.2,0.8,0.2,0.8,0.5,0.9,0.9,0.5,0.6,0.1,0.8,1.0,0.3,0.1,0.5,0.9,0.1,0.0,0.5,0.3,0.1,0.5,0.8,0.3,0.4,0.4,0.3,0.2,0.8,0.7,0.6,0.3,0.5,0.1,0.7,0.4,0.2,0.1,0.1,0.4,0.2,0.8,0.8,0.4,0.1,0.0,0.3,0.2,0.0,1.0,0.2,0.6,0.5,0.7,0.7,0.7,0.1,0.2,0.1,0.1,0.9,0.6,0.5,1.0,0.4,0.4,0.8,0.7,0.5,0.6,0.9,0.0,0.8,0.3,0.1,0.5,0.9,0.9,0.9,0.7,0.7,1.0,0.6,0.6,1.0,0.8,1.0,0.4,0.3,0.2,1.0,0.9,0.2,0.7,0.1,0.3,0.1,0.1,0.7,0.6,0.8,0.8,0.7,0.7,0.4,0.8,0.4,0.1,0.0,1.0,0.2,0.6,0.8,0.3,0.9,0.3,0.6,0.6,0.4,0.7,0.0,0.2,0.9,0.2,0.1,0.4,0.9,0.5,0.2,0.4,1.0,0.1,0.3,0.8,0.8,0.2,0.2,0.6,0.8,0.1,0.0,0.5,1.0,0.5,0.7,0.3,0.5,0.0,0.2,0.6,0.7,0.6,0.4,0.2,0.0,0.4,0.4,0.0,0.3,0.3,0.8,0.5,0.7,0.4,0.1,0.8,0.4,0.1,0.3,1.0,0.3,0.6,0.5,0.6,0.2,0.9,0.4,0.4,0.8,0.0,0.3,0.8,0.3,0.1,0.0,0.5,0.5,0.8,0.6,1.0,0.7,0.8,0.7,0.7,0.6,0.0,0.6,0.6,0.3,0.7,0.2,1.0,0.6,0.4,0.8,0.4,0.7,0.3,0.8,0.8,0.1,0.1,0.2,0.2,0.7,0.1,0.8,0.4,1.0,0.6,1.0,0.3,0.9,0.9,0.9,0.9,1.0,0.2,0.3,0.9,0.5,0.5,0.4,0.1,0.4,0.0,0.7,0.2,0.6,0.8,0.2,0.8,0.2,0.6,0.9,0.1,0.3,0.4,0.2,0.9,0.3,0.9,0.1,0.1,0.7,1.0,0.4,0.2,0.9,0.2,0.5,0.1,0.3,0.6,0.5,0.6,0.5,0.3,0.4,0.3,0.9,0.7,0.1,0.2,0.8,1.0,0.5,0.0,0.8,0.2,0.2,0.0,1.0,0.2,1.0,0.5,1.0,0.9,0.5,0.2,0.5,0.8,0.4,0.9,0.9,0.2,0.5,0.5,0.2,0.6,0.3,0.3,0.8,0.3,0.5,0.4,0.2,0.7,0.8,0.9,0.2,0.9,0.6,0.0,0.3,0.8,0.5,0.3,0.9,0.9,0.7,0.4,0.9,0.3,0.7,0.4,0.3,0.5,0.8,0.9,0.7,0.6,0.5,0.1,0.9,0.6,0.5,0.2,0.7,0.3,0.3,0.1,0.0,0.2,0.5,0.9,0.7,0.3,0.3,1.0,0.3,0.6,0.9,0.1,0.9,0.3,0.7,0.1,0.7,0.6,0.6,0.5,0.1,0.1,0.3,0.5,0.7,0.1,0.7,0.4,0.8,0.4,0.6,0.8,0.7,0.6,0.0,0.1,0.3,0.8,0.2,0.5,0.7,0.0,0.4,1.0,0.2,0.2,0.4,0.3,0.9,0.2,0.4,0.3,0.4,0.2,0.5,0.6,0.6,0.8,0.7,0.3,0.1,0.7,0.5,0.1,0.4,1.0,0.2,0.8,0.5,0.7,0.3,0.7,0.6,0.7,0.5,1.0,0.2,0.8,0.0,0.1,0.2,0.6,0.0,0.2,0.1,0.2,0.4,0.6,0.2,1.0,0.3,0.1,0.1,0.7,0.0,0.7,0.0,0.7,0.9,0.1,0.2,0.8,0.7,0.5,0.3,0.8,0.3,0.0,0.1,0.1,0.8,0.9,0.2,0.5,0.5,0.4,0.4,0.8,0.9,0.4,1.0,0.8,0.4,0.2,0.1,0.3,0.1,0.7,0.9,0.2,0.9,0.8,0.7,0.2,0.7,0.4,0.0,1.0,0.7,0.3,0.6,0.9,0.1,0.5,0.2,0.5,0.7,0.3,0.9,0.7,0.2,1.0,0.6,0.4,0.3,0.1,0.1,0.0,0.3,0.9,0.7,0.5,0.9,0.8,0.6,0.8,0.1,0.4,0.5,0.8,0.7,0.4,0.8,0.4,0.1,0.6,0.8,0.0,0.9,0.7,0.7,0.7,0.7,0.3,0.4,0.4,0.2,0.6,0.3,0.4,1.0,0.2,0.3,0.0,0.5,1.0,0.8,0.7,0.3,0.2,0.7,0.1,0.5,0.2,0.3,0.4,0.8,0.4,0.2,0.3,0.9,0.5,0.1,0.7,0.0,0.3,0.3,0.1,0.1,0.8,0.2,0.6,0.2,0.0,0.3,0.6,0.4,0.7,0.6,0.2,0.8,0.4,0.3,0.7,0.3,0.7,0.9,0.4,0.8,0.9,0.4,0.5,0.4,0.6,0.7,0.5,0.6,0.6,0.4,0.4,0.8,0.3,0.9,0.8,0.9,0.6,0.1,0.9,1.0,1.0,0.8,0.8,0.2,0.1,0.1,0.4,0.9,0.9,0.9,0.6,0.4,0.8,0.6,0.6,0.4,0.6,0.6,0.8,1.0,0.2,0.3,0.4,0.9,0.3,0.7,0.9,0.6,1.0,0.5,0.3,0.5,0.9,0.1,0.9,0.6,0.4,0.9,0.9,0.7,0.9,0.0,0.3,0.7,0.2,0.1,0.2,0.6,0.1,0.6,0.3,0.5,0.1,0.5,0.7,0.1,0.9,0.4,0.1,0.4,1.0,0.1,0.7,0.5,0.6,0.1,0.4,1.0,0.3,0.8,0.3,0.9,0.8,0.9,0.4,0.2,0.2,0.7,0.0,0.8,0.7,0.3,0.2,0.2,0.3,0.9,0.8,0.2,0.3,0.4,0.2,0.9,0.4,0.6,0.2,0.5,0.6,0.0,0.3,0.2,0.9,0.7,0.5,0.7,0.8,0.8,0.2,0.7,0.7,0.5,0.1,0.0,0.3,0.6,0.4,1.0,1.0,0.1,0.2,0.4,0.5,0.0,0.2,0.6,0.8,0.7,0.5,0.2,0.3,0.7,0.4,0.7,0.8,0.2,0.7,0.8,0.9,0.7,0.2,0.5,0.7,0.9,0.7,0.5,0.1,1.0,0.5,0.6,0.9,0.5,0.7,0.3,0.9,0.8],
[0.5,0.6,0.0,0.9,0.9,0.4,0.4,0.9,0.1,0.7,0.8,0.7,1.0,0.5,0.6,0.5,0.9,0.7,0.2,0.4,0.6,0.7,0.4,0.2,0.3,0.3,0.9,1.0,0.0,0.5,0.5,0.6,0.1,0.6,0.1,1.0,0.8,0.4,0.2,0.6,0.9,0.2,0.1,0.5,0.0,0.5,0.3,0.9,0.5,0.0,0.9,0.4,0.4,0.5,0.7,0.9,0.1,0.9,0.0,0.2,0.6,0.8,0.7,0.1,0.6,0.2,0.2,0.8,0.7,0.2,0.1,0.2,0.6,0.8,0.6,0.4,0.8,0.8,0.9,0.7,0.8,0.4,0.5,0.1,0.7,0.9,0.2,0.3,0.0,0.7,0.0,0.1,0.7,0.8,0.9,0.7,0.6,0.3,0.7,0.7,0.2,0.1,0.3,0.7,0.3,0.8,0.2,0.1,0.8,0.9,0.2,0.4,0.5,0.5,0.9,0.9,0.3,0.7,0.1,0.6,0.7,0.2,0.6,0.9,0.8,0.7,0.0,0.4,0.1,0.6,0.5,0.1,0.8,0.7,0.9,0.7,0.5,0.7,0.8,0.8,0.2,0.5,0.3,0.4,0.8,0.4,0.1,0.3,0.4,0.3,0.4,0.7,0.4,0.7,0.9,0.2,0.8,0.3,0.8,0.3,0.8,0.7,0.3,0.4,0.4,0.6,0.1,0.3,0.6,0.5,0.9,0.7,0.3,0.6,0.5,0.3,0.4,0.2,0.8,0.3,0.1,0.9,0.9,0.6,0.1,0.4,0.2,0.4,0.8,0.9,0.1,0.4,0.8,0.5,0.4,0.8,0.9,1.0,0.1,0.8,0.8,0.8,0.8,0.8,0.3,0.1,1.0,0.2,0.9,0.2,0.9,0.7,0.9,1.0,0.4,0.2,0.5,0.4,0.3,0.2,0.1,0.1,0.8,0.7,0.0,0.3,1.0,1.0,0.0,0.5,0.0,0.5,0.6,0.8,0.2,0.4,0.0,0.8,0.5,0.8,0.6,0.3,0.4,0.7,0.9,0.0,0.8,0.7,0.9,0.9,0.2,0.3,0.3,0.9,0.3,0.3,0.3,0.6,0.8,0.5,0.5,0.0,0.5,0.8,1.0,0.4,1.0,0.3,0.5,0.5,0.6,0.6,0.7,0.1,0.3,0.6,0.4,0.2,0.8,1.0,0.6,0.9,0.7,0.5,0.1,0.7,0.6,1.0,0.4,0.9,0.3,0.6,0.1,1.0,0.8,0.7,0.7,0.5,0.0,0.6,0.5,1.0,0.6,0.9,0.8,0.9,0.7,1.0,0.9,1.0,0.3,0.2,0.5,0.3,0.8,0.1,0.9,0.6,0.9,0.9,0.3,0.4,0.1,0.6,0.0,0.0,0.2,0.2,0.9,0.9,0.6,1.0,0.2,0.7,1.0,0.8,1.0,0.2,0.3,0.3,0.9,0.5,0.1,0.2,0.5,0.9,0.1,0.5,0.2,1.0,0.7,0.4,0.2,0.1,0.4,0.4,0.7,0.8,0.3,0.6,0.0,1.0,0.8,1.0,0.1,0.2,0.9,0.4,0.8,0.0,0.0,1.0,0.1,0.3,0.0,0.7,0.6,0.9,0.4,0.4,0.9,0.4,0.8,0.7,0.7,0.5,0.3,0.6,0.5,0.5,0.5,0.9,0.8,0.4,0.8,0.6,0.4,0.2,0.9,1.0,0.8,0.2,0.2,0.8,0.9,0.7,0.1,0.8,0.7,0.3,0.1,0.2,0.3,0.6,0.6,0.6,0.7,0.4,0.1,0.9,0.5,0.5,0.5,0.4,0.6,0.2,0.7,0.6,0.3,0.3,0.2,0.4,0.2,0.9,0.9,0.9,0.7,0.8,0.3,0.0,0.4,0.1,0.9,0.6,0.3,0.0,0.7,0.1,0.8,0.6,0.3,0.6,0.8,0.2,0.1,0.4,0.8,0.9,1.0,0.7,0.8,0.1,0.4,0.1,0.4,0.9,0.4,0.6,0.7,0.2,0.5,0.6,0.8,0.6,0.6,0.9,0.7,0.4,0.3,0.5,0.1,0.8,0.9,0.4,0.0,0.4,0.0,0.3,0.6,0.8,0.1,0.4,0.1,0.6,0.7,0.1,0.0,0.0,0.0,0.8,0.7,0.6,0.8,0.6,0.9,0.1,0.4,0.0,0.4,0.0,0.4,0.7,0.5,0.1,0.9,0.3,0.3,0.1,0.3,0.6,0.6,0.8,0.8,0.9,0.2,0.0,0.6,0.3,1.0,0.6,0.7,1.0,1.0,0.9,0.4,0.1,0.6,0.9,0.1,0.1,0.1,0.2,0.5,0.0,0.8,0.5,0.0,0.8,0.4,0.1,0.2,0.2,0.8,0.9,0.6,0.3,0.2,0.5,0.0,0.1,0.1,0.8,0.9,1.0,0.8,0.2,0.8,0.3,0.8,0.2,0.0,0.1,1.0,0.7,0.1,0.8,0.2,0.5,0.3,0.6,0.1,0.7,0.7,0.5,0.2,0.3,0.5,0.5,1.0,0.2,0.3,0.4,0.1,0.1,0.7,1.0,0.7,0.6,0.9,1.0,0.4,0.8,0.1,0.4,0.1,0.9,0.7,0.4,0.0,0.0,0.3,0.3,0.5,0.6,0.3,0.8,0.5,0.3,0.1,0.9,0.5,0.1,0.3,0.9,0.4,0.3,0.4,0.2,0.9,0.5,0.4,0.9,0.8,0.9,0.9,0.9,0.6,0.6,0.3,0.4,0.3,0.3,0.4,0.4,0.2,0.3,0.7,0.1,0.4,0.1,0.7,0.2,0.7,0.7,0.1,0.3,1.0,0.4,0.4,0.0,0.1,0.4,0.6,0.9,0.5,0.1,0.6,0.9,0.1,0.2,0.4,0.5,0.5,0.1,0.7,0.0,0.1,1.0,0.6,0.1,0.5,0.7,0.2,0.7,0.1,0.1,0.5,0.5,0.2,0.7,0.0,0.9,0.3,0.2,0.9,0.2,0.2,0.5,0.5,0.6,0.3,0.4,0.9,0.4,0.5,0.8,0.1,0.4,0.5,0.9,0.5,0.4,0.3,1.0,0.7,0.5,0.1,0.0,0.3,0.0,0.5,0.5,0.9,0.6,0.3,0.7,0.1,0.9,0.1,0.9,0.1,0.8,0.0,0.9,0.0,0.0,0.7,0.6,1.0,0.5,0.9,0.7,0.4,0.5,0.6,0.3,0.6,0.9,0.4,0.3,0.3,1.0,0.2,1.0,0.3,0.7,0.9,0.8,0.8,0.7,0.6,0.6,0.8,0.5,0.3,0.4,0.5,0.1,0.3,0.4,0.0,0.2,0.8,0.3,1.0,0.5,0.0,0.7,0.9,0.3,0.3,0.9,0.9,0.5,0.0,0.0,0.6,0.7,0.6,0.5,0.1,0.8,0.3,0.3,0.1,0.7,0.0,0.6,0.0,0.1,0.9,0.1,0.4,0.1,0.5,1.0,0.3,0.2,0.8,0.6,0.3,0.5,0.3,0.1,0.9,0.1,0.9,0.9,0.1,0.8,0.7,0.8,0.3,0.5,1.0,0.1,0.7,0.4,0.7,0.7,0.9,0.9,1.0,0.3,0.8,0.3,0.3,0.5,0.2,0.6,0.4,0.5,0.7,0.8,0.9,0.8,0.9,0.2,0.0,0.5,0.2,1.0,0.7,0.4,0.1,0.6,0.6,0.0,0.4,0.6,0.6,0.4,0.1,0.7,1.0,0.1,0.4,0.3,0.9,0.1,0.0,0.1,0.6,0.1,1.0,0.1,0.3,0.3,0.4,0.3,0.8,0.2,0.5,0.1,0.3,0.8,0.7,0.0,0.4,0.5,0.2,0.0,0.5,0.8,0.2,0.6,0.9,0.8,0.9,0.5,0.7,0.5,0.9,0.9,0.3,0.5,0.3,1.0,0.8,0.7,0.9,0.6,0.6,0.5,0.8,0.2,0.7,0.6,0.3,0.1,0.9,0.2,0.4,0.9,0.3,0.2,0.5,0.5,0.9,0.2,1.0,0.9,0.8,0.2,0.2,1.0,0.4,0.4,0.6,0.8,0.3,0.2,0.6,0.0,0.5,0.9,0.6,0.3,0.4,0.8,0.5,0.6,0.7,0.6,0.0,0.1,0.3,0.7,0.4,0.1,0.2,0.7,0.2,0.3,0.8,0.2,0.4,0.2,1.0,1.0,0.7,0.8,0.2,0.5,0.3,0.5,0.4,0.6,0.5,0.3,0.6,0.5,1.0,0.7,0.8,0.9,0.0,0.6,0.3,0.9,0.3,0.9,0.5,0.7,0.5,0.1,0.1,0.3,0.7,0.8,0.1,0.0,0.7,0.5,1.0,0.3,0.8,0.7,0.7,0.2,0.9,0.5,0.6,0.1,0.5,0.5,0.0,0.2,0.7,0.9,0.1,0.9,0.3,0.2]]).to(
    device, dtype=torch.int64
)

x.shape
torch.Size([2, 1024])

I need to reduce  the feature size to 512 keeping the first dim batch size in tact,
x.shape
torch.Size([2, 512])

Thanks

Output:
z = torch.narrow(x,1,0,512)
z.shape
torch.Size([2, 512])

if you use x.unsqueeze(0), then the dim 2 will require the change,
z = torch.narrow(x,2,0,512)


Instruction: pytorch DataLoader extremely slow first epoch

Input:
When I create a PyTorch DataLoader and start iterating -- I get an extremely slow first epoch (x10--x30 slower then all next epochs). Moreover, this problem occurs only with the train dataset from the Google landmark recognition 2020 from Kaggle. I can't reproduce this on synthetic images, also, I tried to create a folder with 500k images from GLR2020, and everything worked well. Found few similar problems in the PyTorch forum without any solutions.
import argparse
import pandas as pd
import numpy as np
import os, sys
import multiprocessing, ray
import time
import cv2
import logging
import albumentations as albu
from torch.utils.data import Dataset, DataLoader

samples = 50000 # count of samples to speed up test
bs = 64 # batch size
dir = '/hdd0/datasets/ggl_landmark_recognition_2020/train' # directory with train data
all_files = pd.read_csv('/hdd0/datasets/ggl_landmark_recognition_2020/train.csv')
files = np.random.choice(all_files.id.values, 50000)
files = [os.path.join(_[0], _[1], _[2], _+'.jpg') for _ in files]

# augmentations
aug =  albu.Compose([albu.Resize(400, 400),
        albu.Rotate(limit=15),
        albu.ChannelDropout(p=0.1),
        albu.Normalize(),])

class ImgDataset:
    def __init__(self, path, files, augmentation = None):
        self.path = path
        self.files = {k:v for k, v in enumerate(files)}
        self.augmentation = augmentation

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        img_name = self.files[idx]
        img = np.array(cv2.imread(os.path.join(self.path, img_name)))
        if self.augmentation is not None:
            return self.augmentation(image=img)['image']

dtset = ImgDataset(dir,files, aug)
torchloader = DataLoader(dataset= dtset, batch_size=64, num_worker=16, shuffle=True)
for _ in range(3):
   t1 = time.time()
   for idx, val in enumerate(torchloader):
       pass
   t2 = time.time()
   print(str(t2-t1) +' sec')

Here are some examples of execution speed with different num_workers in DataLoader
#num_workers=0
273.1584792137146 sec
83.15653467178345 sec
83.67923021316528 sec

# num_workers = 8
165.62366938591003 sec
10.405716896057129 sec
10.495309114456177 sec

# num_workers = 16
156.60744667053223 sec
8.051618099212646 sec
7.922858238220215 sec

Looks like the problem is not with DataLoader, but with dataset. When I delete and reinitialise DataLoader object after first "long" iteration, everything still works fine. When I reinitialise dataset -- long first iteration appears again.
Moreover, I tracked my cpu utilisation via htop during this epochs with num_workers setted to 32, and during the first epoch, utilisation is really low; only 1-2 of 32 cores are working, during other epochs ~all cores are working.

Output:
Slavka,
TLDR: This is a caching effect.
I did not download the whole GLR2020 dataset but I was able to observe this effect on the image dataset that I had locally (80000 jpg images of approx 400x400 size).
To find the reasons for the difference in performance I tried the following:

reducing the augmentation to just resizing
testing just ImgDataset.__getitem__() function
ImgDataset.__getitem__() without augmentation
just loading the raw jpg image and passing it from the dataset without even numpy conversion.

It turns out that the difference comes from the image loading timing. Python (or OS itself) implements some kind of caching which is observed when loading image multiple times in the following test.
for i in range(5):
    t0 = time.time()
    data = cv2.imread(filename)
    print (time.time() - t0)

0.03395271301269531
0.0010004043579101562
0.0010004043579101562
0.0010008811950683594
0.001001119613647461

same is observed when just reading from file to variable
for i in range(5):
    t0 = time.time()
    with open(filename, mode='rb') as file:
        data = file.read()
    print (time.time() - t0)

0.036234378814697266
0.0028831958770751953
0.0020024776458740234
0.0031833648681640625
0.0028734207153320312

One way to reduce the loading speed is to keep the data on very fast local SSD. If size allows, try loading part of the dataset into RAM and writing custom dataloader to feed from there...
BTW Based on my findings this effect should be reproducible with any dataset - see if you used different drives or some caching.


Instruction: Can't understand about pytorch tensor broadcast

Input:
I have the following code:
import torch
d = 2
n = 50
X = torch.randn(n,d)
z = torch.tensor([[-1.0], [2.0]])
y = X @ z
X.size()
z.size()
y.size()

The output is:
torch.Size([50, 2])
torch.Size([2, 1])
torch.Size([50, 1])

My question is, why after broadcasting, the size of the result y is [50,1] rather than [50,2], I think it should be [50,2], am I correct?

Output:
The @ is not broadcasting but multiplication.

In python 3.5, the @ operator was introduced for matrix
multiplication, following PEP465. This is implemented e.g. in numpy
as the matmul operator.

So the size of y is fine.
multiply a matrix of size [50,2] with a vector of size [2,1] will output a vector of size [50,1].
An example showing it more clearly is:
import torch
xx = torch.ones(3, 2)
zz = torch.tensor([[-1.0], [2.0]])
yy = xx @ zz

print(xx)
print(zz)
print(yy)

# tensor([[1., 1.],
#         [1., 1.],
#         [1., 1.]])
# tensor([[-1.],
#         [ 2.]])
# tensor([[1.],
#         [1.],
#         [1.]])

As you can see the third output is indeed just the multiplication of the 2 tensors.
If you wish to do broadcasting I recommend that you will refer to https://medium.com/ai%C2%B3-theory-practice-business/understanding-broadcasting-in-pytorch-ca9e9533f05f


Instruction: How can I save my training progress in PyTorch for a certain batch no.?

Input:
I'm simply trying to train a ResNet18 model using PyTorch library. The training dataset consists of 25,000 images. Therefore, it is taking a lot of time for even the first epoch to complete. Therefore, I want to save the progress after a certain no. of batch iteration is completed. But I can't figure out how to modify my code and how to use the torch.save() and torch.load() functions in my code to save the periodic progress.
My code is given below:

                # BUILD THE NETWORK
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torch.nn.functional as F
import torchvision
import torchvision.models as models
from torchvision import transforms
from PIL import Image
import matplotlib.pyplot as plt

                # DOWNLOAD PRETRAINED MODELS ON ImageNet

model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained = True)
model_resnet34 = torch.hub.load('pytorch/vision', 'resnet34', pretrained = True)

for name, param in model_resnet18.named_parameters():
    if('bn' not in name):
        param.requires_grad = False

for name, param in model_resnet34.named_parameters():
    if('bn' not in name):
        param.requires_grad = False

num_classes = 2

model_resnet18.fc = nn.Sequential(nn.Linear(model_resnet18.fc.in_features, 512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

model_resnet34.fc = nn.Sequential(nn.Linear(model_resnet34.fc.in_features, 512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

                    # FUNCTIONS FOR TRAINING AND LOADING DATA

def train(model, optimizer, loss_fn, train_loader, val_loader, epochs = 5, device = "cuda"):
    print("Inside Train Function\n")
    for epoch in range(epochs):
        print("Epoch : {} running".format(epoch))
        training_loss = 0.0
        valid_loss = 0.0
        model.train()
        k = 0
        for batch in train_loader:
            optimizer.zero_grad()
            inputs, targets = batch
            inputs = inputs.to(device)
            output = model(inputs)
            loss = loss_fn(output, targets)
            loss.backward()
            optimizer.step()
            training_loss += loss.data.item() * inputs.size(0)
            print("End of batch loop iteration {} \n".format(k))
            k = k + 1
        training_loss /= len(train_loader.dataset)

        model.eval()
        num_correct = 0
        num_examples = 0
        for batch in val_loader:
            inputs, targets = batch
            inputs.to(device)
            output = model(inputs)
            targets = targets.to(device)
            loss = loss_fn(output, targets)
            valid_loss += loss.data.item() * inputs.size(0)

            correct = torch.eq(torch.max(F.softmax(output, dim = 1), dim = 1)[1], targets).view(-1)
            num_correct += torch.sum(correct).item()
            num_examples += correct.shape[0]
        valid_loss /= len(val_loader.dataset)

        print('Epoch: {}, Training Loss: {:.4f}, Validation Loss: {:.4f}, accuracy = {:.4f}'.format(epoch, training_loss, valid_loss, num_correct / num_examples))

batch_size = 32
img_dimensions = 224

img_transforms = transforms.Compose([ transforms.Resize((img_dimensions, img_dimensions)),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])
])

img_test_transforms = transforms.Compose([ transforms.Resize((img_dimensions, img_dimensions)),
                                           transforms.ToTensor(),
                                           transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])
])

def check_image(path):
    try:
        im = Image.open(path)
        return True
    except:
        return False

train_data_path = "E:\Image Recognition\dogsandcats\\train\\"
train_data = torchvision.datasets.ImageFolder(root=train_data_path,transform=img_transforms, is_valid_file=check_image)

validation_data_path = "E:\\Image Recognition\\dogsandcats\\validation\\"
validation_data = torchvision.datasets.ImageFolder(root=validation_data_path,transform=img_test_transforms, is_valid_file=check_image)

test_data_path = "E:\\Image Recognition\\dogsandcats\\test\\"
test_data = torchvision.datasets.ImageFolder(root=test_data_path,transform=img_test_transforms, is_valid_file=check_image)

num_workers = 6
train_data_loader      = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=num_workers)
validation_data_loader = torch.utils.data.DataLoader(validation_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)
test_data_loader       = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)

print(torch.cuda.is_available(), "\n")

if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

print(f'Num training images: {len(train_data_loader.dataset)}')
print(f'Num validation images: {len(validation_data_loader.dataset)}')
print(f'Num test images: {len(test_data_loader.dataset)}')

def test_model(model):
    print("Inside Test Model Function\n")
    correct = 0
    total = 0
    with torch.no_grad():
        for data in test_data_loader:
            images, labels = data[0].to(device), data[1].to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    print('correct: {:d}  total: {:d}'.format(correct, total))
    print('accuracy = {:f}'.format(correct / total))

model_resnet18.to(device)
optimizer = optim.Adam(model_resnet18.parameters(), lr=0.001)
if __name__ == "__main__":
    train(model_resnet18, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)
    test_model(model_resnet18)

model_resnet34.to(device)
optimizer = optim.Adam(model_resnet34.parameters(), lr=0.001)
if __name__ == "__main__":
    train(model_resnet34, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)
    test_model(model_resnet34)

import os
def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    return classes, class_to_idx

def make_prediction(model, filename):
    labels, _ = find_classes('E:\\Image Recognition\\dogsandcats\\test\\test')
    img = Image.open(filename)
    img = img_test_transforms(img)
    img = img.unsqueeze(0)
    prediction = model(img.to(device))
    prediction = prediction.argmax()
    print(labels[prediction])

make_prediction(model_resnet34, 'E:\\Image Recognition\\dogsandcats\\test\\test\\3.jpg') #dog
make_prediction(model_resnet34, 'E:\\Image Recognition\\dogsandcats\\test\\test\\5.jpg') #cat

torch.save(model_resnet18.state_dict(), "./model_resnet18.pth")
torch.save(model_resnet34.state_dict(), "./model_resnet34.pth")

# Remember that you must call model.eval() to set dropout and batch normalization layers to
# evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

resnet18 = torch.hub.load('pytorch/vision', 'resnet18')
resnet18.fc = nn.Sequential(nn.Linear(resnet18.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet18.load_state_dict(torch.load('./model_resnet18.pth'))
resnet18.eval()

resnet34 = torch.hub.load('pytorch/vision', 'resnet34')
resnet34.fc = nn.Sequential(nn.Linear(resnet34.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet34.load_state_dict(torch.load('./model_resnet34.pth'))
resnet34.eval()

# Test against the average of each prediction from the two models
models_ensemble = [resnet18.to(device), resnet34.to(device)]

correct = 0
total = 0

if __name__ == '__main__':
    with torch.no_grad():
        for data in test_data_loader:
            images, labels = data[0].to(device), data[1].to(device)
            predictions = [i(images).data for i in models_ensemble]
            avg_predictions = torch.mean(torch.stack(predictions), dim=0)
            _, predicted = torch.max(avg_predictions, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

if total != 0:
    print('accuracy = {:f}'.format(correct / total))
print('correct: {:d}  total: {:d}'.format(correct, total))

To be very precise, I want to save my progress at the end of  for batch in train_loader: loop, for say k = 1500.
If anyone can guide me about modifying my code so that I can save my progress and resume it later, then it will be a great and highly appreciated.

Output:
Whenever you want to save your training progress, you need to save two things:

Your model's state dict
Your optimizer's state dict

This can be done in the following way:
def save_checkpoint(model, optimizer, save_path, epoch):
    torch.save({
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'epoch': epoch
    }, save_path)

To resume training, you can restore your model and optimizer's state dict.
def load_checkpoint(model, optimizer, load_path):
    checkpoint = torch.load(load_path)
    model.load_state_dict(checkpoint['model_state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    epoch = checkpoint['epoch']

    return model, optimizer, epoch

You can save your model at any point in training, wherever you need to. However, it should be ideal to save after finishing an epoch.


Instruction: Cupy works well with TITAN V, but not with TITAN RTX

Input:
I am using cupy to run a cuda code with pytorch.
My env is
ubuntu 20, anaconda-python 3.7.6, nvidia-driver 440, cuda 10.2, cupy-cuda102, torch 1.4.0
First, I wrote a simple main code
import data_load_test
from tqdm import tqdm
import torch
from torch.utils.data import DataLoader

def main():

    dataset = data_load_test.DataLoadTest()
    training_loader = DataLoader(dataset, batch_size=1)
    with torch.cuda.device(0):
        pbar = tqdm(training_loader)
        for epoch in range(3):
            for i, img in enumerate(pbar):
                print("see the message")

if __name__ == "__main__":
    main()

and data loader like this.
from torch.utils.data import Dataset
import cv2
import cupy as cp

def read_cuda_file(cuda_path):
    f = open(cuda_path, 'r')
    source_line = ""
    while True:
        line = f.readline()
        if not line: break
        source_line = source_line + line
    f.close()
    return source_line

class DataLoadTest(Dataset):
    def __init__(self):
        source = read_cuda_file("cuda/cuda_code.cu")
        cuda_source = '''{}'''.format(source)
        module = cp.RawModule(code=cuda_source)
        self.myfunc = module.get_function('myfunc')

        self.input = cp.asarray(cv2.imread("hi.png",-1), cp.uint8)
        h, w, c = self.input.shape
        self.h = h
        self.w = w
        self.output = cp.zeros((w, h, 3), dtype=cp.uint8)

        self.block_size = (32, 32)
        self.grid_size = (h // self.block_size[1], w // self.block_size[0])

    def __len__(self):
        return 1

    def __getitem__(self, idx):
        self.myfunc(self.grid_size, self.block_size, (self.input, self.output, self.h, self.w))
        return cp.asnumpy(self.output)

And my cuda code is,
#define PI 3.14159265358979323846f
extern "C"{
__global__ void myfunc(const unsigned char* refImg, unsigned char* warpImg, const long long cols, const long long rows)
{

    long long x = blockDim.x * blockIdx.x + threadIdx.x;
    long long y = blockDim.y * blockIdx.y + threadIdx.y;

    long long indexImg = y * cols + x;

    warpImg[indexImg * 3] = 0;
    warpImg[indexImg * 3 + 1] = 1;
    warpImg[indexImg * 3 + 2] = 2;
}
}

I have two GPUs TITAN V (device 0) and TITAN RTX (device 1)
When I run this code with TITAN V,(main function 3rd line)
with torch.cuda.device(0):

it works fine, but
with TITAN RTX,
with torch.cuda.device(1):

It gives an error message like this.
  File "cupy/core/raw.pyx", line 66, in cupy.core.raw.RawKernel.__call__
  File "cupy/cuda/function.pyx", line 162, in cupy.cuda.function.Function.__call__
  File "cupy/cuda/function.pyx", line 144, in cupy.cuda.function._launch
  File "cupy/cuda/driver.pyx", line 293, in cupy.cuda.driver.launchKernel
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_CONTEXT_IS_DESTROYED: context is destroyed

Please help.

Output:
In main() when dataLoadTest() class is instantiated, it is happening on the default device 0, so cuPy is compiling myFunc() there.
The next line “with torch.cuda.device(0):“ is where you switch to device 1 in the version that fails?
What happens if you call
cuPy.cuda.Device(1).use()

as the first line in main(), to make sure myFunc() gets instantiated on device 1?


Instruction: Why W_q matrix in torch.nn.MultiheadAttention is quadratic

Input:
I am trying to implement nn.MultiheadAttention in my network. According to the docs,

embed_dim  – total dimension of the model.

However, according to the source file,

embed_dim must be divisible by num_heads

and

self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))

If I understand properly, this means each head takes only a part of features of each query, as the matrix is quadratic. Is it a bug of realization or is my understanding wrong?

Output:
Each head uses a different part of the projected query vector. You can imagine it as if the query gets split into num_heads vectors that are independently used to compute the scaled dot-product attention. So, each head operates on a different linear combination of the features in queries (and keys and values, too). This linear projection is done using the self.q_proj_weight matrix and the projected queries are passed to F.multi_head_attention_forward function.
In F.multi_head_attention_forward, it is implemented by reshaping and transposing the query vector, so that the independent attentions for individual heads can be computed efficiently by matrix multiplication.
The attention head sizes are a design decision of PyTorch. In theory, you could have a different head size, so the projection matrix would have a shape of embedding_dim × num_heads * head_dims. Some implementations of transformers (such as C++-based Marian for machine translation, or Huggingface's Transformers) allow that.


Instruction: Unable to install pillow 6.2.1 in conda

Input:
After upgrading pytorch / torch-vision the following error occurs:
python -c "import torch ; import torchvision as tv; print(torch.__version__, tv.__version__)
> "
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/__init__.py", line 4, in <module>
    from torchvision import datasets
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/datasets/__init__.py", line 9, in <module>
    from .fakedata import FakeData
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/datasets/fakedata.py", line 3, in <module>
    from .. import transforms
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/__init__.py", line 1, in <module>
    from .transforms import *
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 17, in <module>
    from . import functional as F
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 5, in <module>
    from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION
ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/home/pointr/anaconda3/lib/python3.7/site-packages/PIL/__init__.py)

This has been noted as due to an incompatibility between conda and pytorch 7.0.0  https://github.com/pytorch/vision/issues/1712 . So  I need to downgrade to pillow 6.2.1:
The command posted to do this is:  conda install pillow=6.2.1 -y
However that is failing:
 (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |

That is hanging.  So now what ?   The workaround needs itself a workaround ..  Is conda broken for torchvision now?   The primary reason I am using conda in the first place is torch / torchvision ..
Update Conda tried to resolve conflicts. After 20 minutes of this it was 13% done. Ridiculous. This is a 2020 core i7 mini-tower. No sane program takes more than low double digits seconds to resolve dependencies.  I finally killed it.  I am going to try the suggestion to do directly from pip:
pip install Pillow==6.2.1

OK - that is hanging ..  I am going to uninstall pillow and reinstall it with that version.
Another update  @erip has recommended conda install -c conda-forge pillow=6.2.1 - so here we go:
conda install -c conda-forge pillow=6.2.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \

I will let the dust settle on that one but looking quite unlikely.

Output:
The only thing that worked for me is to uninstall /  reinstall conda
https://docs.anaconda.com/anaconda/install/uninstall/
https://docs.conda.io/en/latest/miniconda.html
And here are the versions conda elected to install:
(base) pointr@alienware:~/anaconda3$ python -c "import cv2; import PIL;print('cv2: ' + cv2.__version__); print('PIL: ' + PIL.__version__)"

cv2: 4.1.0
PIL: 7.1.2

(base) pointr@alienware:~/anaconda3$ python -c "import torch ; import torchvision as tv; print('torch:' + torch.__version__); print('torchvision: ' + tv.__version__)"

torch:1.3.1
torchvision: 0.4.2


Instruction: Normal distribution sampling in pytorch-lightning

Input:
In Pytorch-Lightning you usually never have to specify cuda or gpu. But when I want to create a gaussian sampled Tensor using torch.normal I get
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

so, how do I have to change torch.normal such that pytorch-lightning works properly? Since I use the code on different machines on cpu and on gpu
centers = data["centers"] #already on GPU... sometimes...

lights = torch.normal(0, 1, size=[100, 3])
lights += centers

Output:
The recommended way is to do lights = torch.normal(0, 1, size=[100, 3], device=self.device) if this is inside lightning class.
You could also do: lights = torch.normal(0, 1, size=[100, 3]).type_as(tensor), where tensor is some tensor which is on cuda.


Instruction: Latent vector variance much larger in CoreML than PyTorch

Input:
I have a PyTorch model that I've converted to CoreML. In PyTorch, the inferred latent vector's values range to a limit of around ±1.6, but the converted mlmodel varies as much as ±55.0. What might be causing this huge discrepancy? The conversion is pretty straightforward:
encoder = Encoder_CNN(latent_dim, n_c)
enc_fname = os.path.join(models_dir, encoder.name + '_153.pth.tar')
encoder.load_state_dict(torch.load(enc_fname, map_location={'cuda:0': 'cpu'}))
encoder.cpu()
encoder.eval()

img_in = torch.rand(1, 1, 28, 28).cpu()
traced_encoder_model = torch.jit.trace(encoder, img_in)
traced_encoder_model.save("{}_encoder_mobile_{}_{}.pt".format(model_type, latent_dim, n_c))

coreml_encoder = ct.convert(traced_encoder_model, inputs=[ct.ImageType(name=name, shape=img_in.shape)])
coreml_encoder.save("{}_encoder_{}_{}.mlmodel".format(model_type, latent_dim, n_c))

Output: You probably are using different input normalization for PyTorch and Core ML. Your img_in consists of values between 0 and 1. I don't see the inference code for Core ML, but your input pixels are probably between 0 and 255 there. You can fix this by specifying image preprocessing settings when you convert the PyTorch model to Core ML.


Instruction: Weight & Biases Detectron2 Google Colab - wandb: ERROR Unable to log event [Errno 95] Operation not supported

Input:
I am training a Faster-RCNN model by Detectron2 in Google Colab. I would like to track my experiments with Weights and Biases (WandB).
My dataset is uploaded to Google Drive and mounted to the session via:
from google.colab import drive
drive.mount('/content/gdrive')

Following the suggestion from https://github.com/facebookresearch/detectron2/issues/774
I am trying to link WandB via Tensorboard with:
import wandb
wandb.init(sync_tensorboard=True)

Once the training starts I get the following error repeatedly:
wandb: ERROR Unable to log event [Errno 95] Operation not supported: '/content/gdrive/My Drive/Data/output/events.out.tfevents.1598810231.3dc4616192b5.103.0' -> '/content/gdrive/My Drive/Data/wandb/run-20200830_175618-3fp3tyhs/events.out.tfevents.1598810231.3dc4616192b5.103.0'

In this case, in my WandB account, I can see that there is an active experiment running but there are no logs of losses, learning rate, etc., only hardware info like the specs of the GPU show up.
Interestingly, when I add the linking between Tensorboard and WandB in the Demo Colab Notebook of Detectron2 (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5), it works perfectly: the logging of losses, learning rate, etc shows up in my WandB account.
Can I get some tips regarding what is going wrong in my case?

Output:
After one week the problem disappeared. I assume that someone must have fixed the bug that caused this issue. I can now use:
import wandb
wandb.init(sync_tensorboard=True)

and all the training metrics are synchronized to WandB without any problems.


Instruction: GCP AI Platform Notebook driver too old?

Input:
I am trying to run the following Hugging Face Transformers tutorial on GCP's AI Platform Notebook with 32 vCPUs, 208 GB RAM, and 2 NVIDIA Tesla T4s.
However, when I try to run the part
model = DistillBERTClass()
model.to(device)
I get the following Assertion Error:
AssertionError: The NVIDIA driver on your system is too old (found version 10010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.

However, when I run
!nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   38C    P0    22W /  70W |     10MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   39C    P8    10W /  70W |     10MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |

The version on the NVIDIA driver is compatible with the latest PyTorch version, which I am using.
Has anyone else ran into this error, and is there a way around it?

Output:
You can try a newer NVIDIA driver version,  we support latest CUDA 11 driver version, and then install Pytorch on top of it:
gcloud beta notebooks instances create cuda11 \
--vm-image-project=deeplearning-platform-release \
--vm-image-family=common-cu110-notebooks-debian-9 \
--machine-type=n1-standard-1 \
--location=us-west1-a \
--format=json

Image family:

common-cu110-notebooks-debian-9
common-cu110-notebooks-debian-10


Instruction: How do I retain grads and also change device type in pytorch?

Input: When I change my input variable from CPU to cuda, it loses all its grad and also loses its is_leaf status. How do I circumvent this? I want to keep the gradients and also change it to another device.

Output:
A leaf tensor is one which has its requires_grad attribute set to True. When you do any out-of-place operation on a tensor the resulting tensor is no longer a leaf tensor. This includes creating a copy of the tensor on a different device using .to(device), .cuda(), or .cpu(). The recommended way to set the requires_grad attribute of an existing tensor is to use the in-place Tensor.requires_grad_ method. If you want the tensor on the GPU to be the leaf node then you will need to set requires_grad after copying to the desired device.
For example
input = input.to('cuda')
input.requires_grad_(True)   # need to set requires_grad after copying to GPU

or a little more concisely
input = input.to('cuda').requires_grad_(True)


Instruction: How to replace every row that are filled with zero with a certain value from a Pytorch tensor?

Input:
I have a Pytorch tensor of the size bsize x 50 x 50 where some of the rows are completely filled with zeros:
         [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],
         ...,
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0]],

        [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],

I want to  replace the rows filled with zeros with a negative value -100 throughout in the tensor.
Expected tensor -
         [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],
         ...,
         [-100, -100, -100,  ..., -100,-100, -100],
         [-100, -100, -100,  ..., -100,-100, -100],
         [-100, -100, -100,  ..., -100,-100, -100]],

        [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],

Whats the best way to do this avoiding a loop over the row shape ?

Output:
Assuming x is your tensor with BxRxC (batch, rows, and columns), you can do something like this:
x[(x == 0).all(dim=-1)] = -100

Basically:

x == 0 returns a boolean tensor (shape BxRxC) with True where it is equal to zero;
then, .all(dim=-1) returns another boolean tensor, now with shape BxR because we chose to do all in the last dimension (-1), with True where all the columns are True;
finally, we use this boolean tensor to index the original tensor and the -100 is assigned to the True positions.


Instruction: Hugging face - RuntimeError: Caught RuntimeError in replica 0 on device 0 on Azure Databricks

Input:
How do I run the run_language_modeling.py script from hugging face using the pretrained roberta case model to fine-tune  using my own data on the Azure databricks with a GPU cluster.
Using Transformer version 2.9.1 and 3.0 .
Python 3.6
Torch `1.5.0
torchvision 0.6
This is the script I ran below on Azure databricks
%run '/dbfs/FileStore/tables/dev/run_language_modeling.py' \
  --output_dir='/dbfs/FileStore/tables/final_train/models/roberta_base_reduce_n' \
  --model_type=roberta \
  --model_name_or_path=roberta-base \
  --do_train \
  --num_train_epochs 5 \
  --train_data_file='/dbfs/FileStore/tables/final_train/train_data/all_data_desc_list_full.txt' \
  --mlm

This is the error I get after running the above command.
/dbfs/FileStore/tables/dev/run_language_modeling.py in <module>
   279
   280 if __name__ == "__main__":
--> 281     main()

/dbfs/FileStore/tables/dev/run_language_modeling.py in main()
   243             else None
   244         )
--> 245         trainer.train(model_path=model_path)
   246         trainer.save_model()
   247         # For convenience, we also re-save the tokenizer to the same directory,

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path)
   497                     continue
   498
--> 499                 tr_loss += self._training_step(model, inputs, optimizer)
   500
   501                 if (step + 1) % self.args.gradient_accumulation_steps == 0 or (

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in _training_step(self, model, inputs, optimizer)
   620             inputs["mems"] = self._past
   621
--> 622         outputs = model(**inputs)
   623         loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
   624

/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
   548             result = self._slow_forward(*input, **kwargs)
   549         else:
--> 550             result = self.forward(*input, **kwargs)
   551         for hook in self._forward_hooks.values():
   552             hook_result = hook(self, input, result)

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
   153             return self.module(*inputs[0], **kwargs[0])
   154         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 155         outputs = self.parallel_apply(replicas, inputs, kwargs)
   156         return self.gather(outputs, self.output_device)
   157

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
   163
   164     def parallel_apply(self, replicas, inputs, kwargs):
--> 165         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
   166
   167     def gather(self, outputs, output_device):

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
    83         output = results[i]
    84         if isinstance(output, ExceptionWrapper):
---> 85             output.reraise()
    86         outputs.append(output)
    87     return outputs

/databricks/python/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
   393             # (https://bugs.python.org/issue2651), so we work around it.
   394             msg = KeyErrorMessage(msg)
--> 395         raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
   output = module(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 239, in forward
   output_hidden_states=output_hidden_states,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 762, in forward
   output_hidden_states=output_hidden_states,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 439, in forward
   output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 371, in forward
   hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 315, in forward
   hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
 File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
   result = self.forward(*input, **kwargs)
 File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 240, in forward
   attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA out of memory. Tried to allocate 96.00 MiB (GPU 0; 11.17 GiB total capacity; 10.68 GiB already allocated; 95.31 MiB free; 10.77 GiB reserved in total by PyTorch)```

Please how do I resolve this

Output:
The out of memory error is likely caused by not cleaning up the session and or freeing up the GPU.
From the similar Github issue.

It is because of mini-batch of data does not fit on to GPU memory. Just decrease the batch size. When I set batch size = 256 for cifar10 dataset I got the same error; Then I set the batch size = 128, it is solved.


Instruction: How to get stable output for torch.nn.Transformer

Input:
Looks like Transformer layers of pytorch give not reproducible outputs. It happens both for cpu and gpu. I know that it sometimes happens because of parallel computations on gpu.
emb = nn.Embedding(10, 12).to(device)
inp1 = torch.LongTensor([1, 2, 3, 4]).to(device)
inp1 = emb(inp1).reshape(inp1.shape[0], 1, 12) #S N E

encoder_layer = nn.TransformerEncoderLayer(d_model=12, nhead=4)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=4)

out1 = transformer_encoder(inp1)
out2 = transformer_encoder(inp1)

out1 and out2 are different. It can be multiprocessing on cpu, but results looks too shaky. How to fix this?

Output:
nn.TransformerEncoderLayer has a default dropout rate of 0.1. The indices to be dropped will be randomized in every iteration when the model is in training mode.
If you want to train the model with dropout, just ignore this behavior in training and call model.eval() in testing.
If you want to disable such random behavior in training, set dropout=0 like so
nn.TransformerEncoderLayer(d_model=12, nhead=4, dropout=0)

Full testing script:
import torch
import torch.nn as nn

device = 'cpu'

emb = nn.Embedding(10, 12).to(device)
inp1 = torch.LongTensor([1, 2, 3, 4]).to(device)
inp1 = emb(inp1).reshape(inp1.shape[0], 1, 12) #S N E

encoder_layer = nn.TransformerEncoderLayer(d_model=12, nhead=4, dropout=0).to(device)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=4).to(device)

out1 = transformer_encoder(inp1)
out2 = transformer_encoder(inp1)

print((out1-out2).norm())


Instruction: Why can't PyCharm show PyTorch Module object attributes in debug mode

Input:
I define a sub-class of PyTorch's Module in PyCharm and create an instance a:
from torch.nn import Module

class AModule(Module):
    def __init__(self):
        self.something = 10

    def __repr__(self):
        return "AModule"

a = AModule()

If I run the debugger and examine a, I can't see its attributes:

I checked and Module is written in Python (as opposed to being implemented in C), so why is that?

Output:
This is caused by not having properly initialized Module with a super call in the first like of __init__:
super(AModule, self).__init__()

However, PyCharm could have shown more useful information, so I created this issue.


Instruction: How can I have submodules of a PyTorch Module that are not attributes of the module

Input:
I would like to have a PyTorch sub-class of Module that keeps sub-modules in a list (because there may be a variable number of sub-modules depending on the constructor's arguments). I set this list in the following way:
self.hidden_layers = [torch.nn.Linear(i, o) for i, o in pairwise(self.layer_sizes)]

According to this and this question, a submodule is only registered by __setattr__, when a Module object is assigned to an attribute of self. Because hidden_layers is not assigned an object of type Module, the submodules in the list are not registered as submodules, and as a result self.parameters() does not iterate over the submodules' parameters.
I suppose I could explicitly call __subattr__ for each element of the list but that would be quite ugly. Is there a more correct way to register a submodule that is not a direct attribute of Module?

Output:
As answered nn.ModuleList is what you want.
What you can also use is nn.Sequential. You can create a list of layers and then combine them via nn.Sequential, which will just act as a wrapper and combines all layers to essential one layer/module. This has the advantage that you only need one call to forward it through all the layers, which is nice if you have a dynamic count of modules, so you don't have to write the loops on your own.
One example would be in the pytorch ResNet code: https://github.com/pytorch/vision/blob/497744b9d510ff2df756f479ee5a19fce0d579b6/torchvision/models/resnet.py#L177


Instruction: How to prune a Detectron2 model?

Input:
I'm a teacher who is studying computer vision for months. I was very excited when I was able to train my first object detection model using Detectron2's Faster R-CNN model. And it works like a charm! Super cool!
But the problem is that, in order to increase the accuracy, I used the largest model in the model zoo.
Now I want to deploy this as something people can use to ease their job. But, the model is so large that it takes ~10 seconds to infer a single image on my CPU which is Intel i7-8750h.
Therefore, it's really difficult to deploy this model even on a regular cloud server. I need to use either GPU servers or latest model CPU servers which are really expensive and I'm not sure if I can even compensate for server expenses for months.
I need to make it smaller and faster for deployment.
So, yesterday I found that there's something like pruning the model!! I was very excited (since I'm not a computer or data scientists, don't blame me (((: )
I read official pruning documentation of PyTorch, but it's really difficult for me to understand.
I found global pruning is of the easiest one to do.
But the problem is, I have no idea what parameters should I write to prune.
Like I said, I used Faster R-CNN X-101 model. I have it as "model_final.pth". And it uses Base RCNN FPN.yaml and its meta architecture is "GeneralizedRCNN".
It seems like an easy configuration to do. But like I said, since it's not my field it's very hard for a person like me.
I'd be more than happy if you could help me on this step by step.
I'm leaving my cfg.yaml which I used training the model and I saved it using "dump" method in Detectron2 config class just in case. Here's the Drive link.
Thank you very much in advance.

Output:
So I guess, you are trying to optimize inference time and achieving satisfactory accuracy. Without knowing details about your object types, training size, image size, it will be hard to provide suggestions. However, as you know, ML project development is an iterative process, you can have a look at the following page and check inference and accuracy.
https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md#coco-object-detection-baselines
I would suggest, you try R50-FPN backbone and see how your accuracy comes. Then, you will get a better understanding of what to do next.


Instruction: Extracting execution DAG from PyTorch Module

Input:
Let us consider a simple feed-forward deep net or even a resnet. In both case, you can think of the execution flow as a DAG (directed acyclic graph) where nodes are layers (it's a chain if there's no skip connexion).
For some reason, I need this DAG for pyTorch models. Up to now I have been using custom object subclassing nn.Module (that stores the information that I need) but I would like to be able ultimately to use an arbitrary model in input of my pipeline. Is there a way to extract this DAG automatically ? (it does not matter if it's not working in all cases, I am mainly looking for general ideas / principles to achieve my goal).

Output:
That is indeed possible. Consider following evaluation of a simple formula as a stand in for a more complicated network:
import torch
a = torch.randn(3, requires_grad=True)
b = 3*a + 1
c = torch.relu(b)
d = c.sum()

Then d has a .grad_fn attribute, from which you can recurse through the evaluation graph like so
d.grad_fn
d.grad_fn.next_functions
d.grad_fn.next_functions
d.grad_fn.next_functions[0][0].next_functions
d.grad_fn.next_functions[0][0].next_functions[0][0].next_functions

Basically next_functions gives you a list of the arguments to the current module/operation, and each entries of this list is a tuple with the actual object, as well as an integer that indicates the position of the argument. This is documented in more detail here.
If you don't want to do it yourself, you can also use the visualization tool for the computation graph that is built into tensorboard, that is SummaryWriter.add_graph() as documented here.


Instruction: PyTorch Tensorboard not as described in documentation

Input:
I'm using PyTorch's ùtils.tensorboard.writer to log training of an RNN. For the àdd_hparams() function the Docs say:

Params:

hparam_dict (dict) – Each key-value pair in the dictionary is the name of the hyper parameter and it’s corresponding value. The type of the value can be one of bool, string, float, int, or None.
metric_dict (dict) – Each key-value pair in the dictionary is the name of the metric and it’s corresponding value. Note that the key used here should be unique in the tensorboard record. Otherwise the value you added by add_scalar will be displayed in hparam plugin. In most cases, this is unwanted.
hparam_domain_discrete – (Optional[Dict[str, List[Any]]]) A dictionary that contains names of the hyperparameters and all discrete values they can hold
run_name (str) – Name of the run, to be included as part of the logdir. If unspecified, will use current timestamp.

Source: https://pytorch.org/docs/master/tensorboard.html

But when I try to use the run_name parameter, I get the error

TypeError: add_hparams() got an unexpected keyword argument 'run_name'

So I looked up the writer.py file I imported and found the cause to be the add_hparams() function itself:
def add_hparams(self, hparam_dict, metric_dict):

I checked my installation of PyTorch, but it's up to date. Is this some sort of nightly feature, if so, how can I download the nightly version of torch?

Output:
The docs you linked to are the ones from the current master branch of PyTorch. So yes, it is a nightly feature. The docs of the stable version (1.6) do not mention add_hparams.
You can get the command to download PyTorch nightly here by selecting Preview (Nightly) instead of Stable.


Instruction: masked kernel in convolution in pytorch

Input:
Let's say:
x = torch.arange(16, dtype=torch.float).reshape(1, 1, 4, 4)

and a 2d convolution layer is:
layer = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=2, stride=2)
layer.weight.data[:] = 1.
layer.bias.data[:] = 0.

Normally, passing x to layer will give:
>>layer(x)
tensor([[[[10., 18.],
              [42., 50.]]]], grad_fn=<MkldnnConvolutionBackward>)

Considering having 4 mask filters, how it is done to mask kernel in each step?
for example the following picture indicates 4 filters(white: True, black: False)

The output should be:
tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=<MkldnnConvolutionBackward>)

P.S: all masks are obtained by missing pixels in 2d input array. So 4 masks above are actually a matrix with the same shape as input.

Output:
I found the solution.
Element-wise multiplication between input and the mask before feeding it to a Conv2d method would be enough.(masking input is much easier than masking kernel itself !!):
mask = torch.tensor([[[1, 1, 1, 0]], [[1, 0, 1, 1]], [[1, 1, 0, 1]], [[0, 1, 1, 1]]], dtype=torch.float, requires_grad=False).reshape(1, 1, 4, 4)

>>layer(torch.mul(x, mask))
tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=<MkldnnConvolutionBackward>)

P.S Thanks to @Shai I got the idea from partial convolution represented in this paper. However it does some extra manipulation on output. it defines a mask ratio and I guess does some weighting the final output based on it.