Instruction: Unexpected input data type. Actual: (tensor(double)) , expected: (tensor(float))

Input:
I am learning this new ONNX framework that allows us to deploy the deep learning (and others) model into production.
However, there is one thing I am missing. I thought that the main reason for having such a framework is so that for inference purposes e.g. when we have a trained model and want to use it in a different venv (where for example we cannot have PyTorch) the model still can be used.
I have preped a "from scratch" example here:
# Modules
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset
import torchvision
import onnx
import onnxruntime
import matplotlib.pyplot as plt
import numpy as np

# %config Completer.use_jedi = False

# MNIST Example dataset
train_loader = torch.utils.data.DataLoader(
      torchvision.datasets.MNIST(
          'data', train=True, download=True,
          transform=torchvision.transforms.Compose([
              torchvision.transforms.ToTensor(),
          ])),
      batch_size=800)

# Take data and labels "by hand"
inputs_batch, labels_batch = next(iter(train_loader))

# Simple Model

class CNN(nn.Module):
    def __init__(self, in_channels, num_classes):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=in_channels,
                               out_channels = 10, kernel_size = (3, 3), stride = (1, 1), padding=(1, 1))
        self.pool = nn.MaxPool2d(kernel_size=(2, 2), stride = (2, 2))
        self.conv2 = nn.Conv2d(in_channels = 10, out_channels=16, kernel_size = (3, 3), stride = (1, 1), padding=(1, 1))
        self.fc1 = nn.Linear(16*7*7, num_classes)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = x.reshape(x.shape[0], -1)
        x = self.fc1(x)

        return x

# Training setting
device = 'cpu'
batch_size = 64
learning_rate = 0.001
n_epochs = 10

# Dataset prep
dataset = TensorDataset(inputs_batch, labels_batch)
TRAIN_DF = DataLoader(dataset = dataset, batch_size = batch_size, shuffle = True)

# Model Init
model = CNN(in_channels=1, num_classes=10)
optimizer = optim.Adam(model.parameters(), lr = learning_rate)

# Training Loop
for epoch in range(n_epochs):
    for data, labels in TRAIN_DF:
        model.train()
        # Send Data to GPU
        data = data.to(device)
        # Send Data to GPU
        labels = labels.to(device)

#       data = data.reshape(data.shape[0], -1)

        # Forward
        pred = model(data)
        loss = F.cross_entropy(pred, labels)

        # Backward
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# Check Accuracy
def check_accuracy(loader, model):
    num_correct = 0
    num_total = 0

    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device)
            y = y.to(device)

    #             x = x.reshape(x.shape[0], -1)

            scores = model(x)
            _, pred = scores.max(1)

            num_correct += (pred == y).sum()
            num_total += pred.size(0)

    print(F"Got {num_correct} / {num_total} with accuracy {float(num_correct)/float(num_total)*100: .2f}")

check_accuracy(TRAIN_DF, model)

# Inference with ONNX

# Create Artifical data of the same size
img_size = 28
dummy_data = torch.randn(1, img_size, img_size)
dummy_input = torch.autograd.Variable(dummy_data).unsqueeze(0)

input_name = "input"
output_name = "output"

model_eval = model.eval()

torch.onnx.export(
    model_eval,
    dummy_input,
    "model_CNN.onnx",
    input_names=["input"],
    output_names=["output"],
)

# Take Random Image from Training Data
X_pred = data[4].unsqueeze(0)

# Convert the Tensor image to PURE numpy and pretend we are working in venv where we only have numpy - NO PYTORCH
X_pred_np = X_pred.numpy()

X_pred_np = np.array(X_pred_np)

IMG_Rando = np.random.rand(1, 1, 28, 28)

np.shape(X_pred_np) == np.shape(IMG_Rando)

ort_session = onnxruntime.InferenceSession(
    "model_CNN.onnx"
)

def to_numpy(tensor):
    return (
        tensor.detach().gpu().numpy()
        if tensor.requires_grad
        else tensor.cpu().numpy()
    )

# compute ONNX Runtime output prediction

# WORKS
# ort_inputs = {ort_session.get_inputs()[0].name: X_pred_np}

# DOES NOT WORK
ort_inputs = {ort_session.get_inputs()[0].name: IMG_Rando}

# WORKS
# ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(X_pred)}
ort_outs = ort_session.run(None, ort_inputs)

ort_outs

Firstly, we create a simple model and train it on the MNIST dataset.
Then we export the trained model using the ONNX framework.
Now, when I want to classify an image using the X_pred_np It works even though it is a "pure" NumPy, which is what I want.
However, I suspect that this particular case works only because it has been derived from the PyTorch tensor object, and thus "under the hood" it still has PyTorch attributes.
While when I try to inference on the random "pure" NumPy object IMG_Rando, there seems to be a problem:
Unexpected input data type. Actual: (tensor(double)) , expected: (tensor(float)).
Referring that PyTorch form is needed.
Is there a way how to be able to use only numpy Images for the ONNX predictions?. So the inference can be performed in separated venv where no pytorch is installed?
Secondly, is there a way that ONNX would remember the actual classes?
In this particular case, the index corresponds to the label of the image. However, in animal classification, ONNX would not provide us with the "DOG" and "CAT" and other labels but would only provide us the index of the predicted label. Which we would need to run throw our own "prediction dictionary" so we know that the fifth label is associated with "cat" and sixth label is associated with "dog" etc.

Output:
As an improvement to the accepted answer, the idiomatic way to generate random numbers in Numpy is now by using a Generator. This offers the benefit of being able to create the array in the right type directly, rather than using the expensive astype operation, which copies the array (as in the accepted answer). Thus, the improved solution would look like:
rng = np.random.default_rng() # set seed if desired
IMG_Rando = rng.random((1, 1, 28, 28), dtype=np.float32)


Instruction: Index tensor must have the same number of dimensions as self tensor

Input:
I have a dataset which looks like
ID  Target  Weight    Score   Scale_Cat   Scale_num
0   A   D   65.1       87        Up  1
1   A   X   35.8       87        Up  1
2   B   C   34.7       37.5    Down    -2
3   B   P   33.4       37.5    Down    -2
4   C   B   33.1       37.5    Down    -2
5   S   X   21.4       12.5    NA  9

This dataset consists of nodes (ID) and targets (neighbors) and it has been used as sample for testing label propagation. Classes/Labels are within the column Scale_num and can take values from -2 to 2 at step by one. The label 9 means unlabelled and it is the label that I would like to predict using label propagation algorithm.
Looking for some example on Google about label propagation, I have found this code useful (difference is in label assignment, since in my df I have already information on data which have labelled - from -2 to 2 at step by 1, and unlabelled, i.e. 9): https://mybinder.org/v2/gh/thibaudmartinez/label-propagation/master?filepath=notebook.ipynb
However, trying to use my classes instead of (-1,0,1) as in the original code, I have got some errors. A user has provided some help here:  RunTimeError during one hot encoding, for fixing a RunTimeError, unfortunately still without success.
In the answer provided on that link, 40 obs and labels are randomly generated.
import random
labels = list()
for i in range(0,40):
    labels.append(list([(lambda x: x+2 if x !=9 else 5)(random.sample(classes,1)[0])]))

index_aka_labels = torch.tensor(labels)
torch.zeros(40, 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)

The error I am getting, still a RunTimeError, seems to be still due to a wrong encoding. What I tried is the following:
import random
labels = list(df['Scale_num'])

index_aka_labels = torch.tensor(labels)
torch.zeros(len(df), 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)

getting the error
---> 7 torch.zeros(len(df), 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)

RuntimeError: Index tensor must have the same number of dimensions as self tensor

For sure, I am missing something (e.g., the way to use classes and labels as well as src, which has never been defined in the answer provided in that link).
The two functions in the original code which are causing the error are as follows:
def _one_hot_encode(self, labels):
    # Get the number of classes
    classes = torch.unique(labels) # probably this should be replaced
    classes = classes[classes != -1] # unlabelled. In my df the unlabelled class is identified by 9
    self.n_classes = classes.size(0)

    # One-hot encode labeled data instances and zero rows corresponding to unlabeled instances
    unlabeled_mask = (labels == -1) # In my df the unlabelled class is identified by 9
    labels = labels.clone()  # defensive copying
    labels[unlabeled_mask] = 0
    self.one_hot_labels = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)
    self.one_hot_labels = self.one_hot_labels.scatter(1, labels.unsqueeze(1), 1)
    self.one_hot_labels[unlabeled_mask, 0] = 0

    self.labeled_mask = ~unlabeled_mask

def fit(self, labels, max_iter, tol):

    self._one_hot_encode(labels)

    self.predictions = self.one_hot_labels.clone()
    prev_predictions = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)

    for i in range(max_iter):
        # Stop iterations if the system is considered at a steady state
        variation = torch.abs(self.predictions - prev_predictions).sum().item()

        prev_predictions = self.predictions
        self._propagate()

I would like to understand how to use in the right way my classes/labels definition and info from my df in order to run the label propagation algorithm with no errors.

Output:
I suspect it's complaining about index_aka_labels lacking the singleton dimension. Note that in your example which works:
import random
labels = list()
for i in range(0,40):
    labels.append(list([(lambda x: x+2 if x !=9 else 5)(random.sample(classes,1)[0])]))

index_aka_labels = torch.tensor(labels)
torch.zeros(40, 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)

If you run index_aka_labels.shape, it returns (40,1). When you just turn your pandas series into a tensor, however, it will return a tensor of shape (M) (where M is the length of the series). If you simply run:
import random
labels = list(df['Scale_num'])
index_aka_labels = torch.tensor(labels)[:,None] #create another dimension
torch.zeros(len(df), 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)

the error should disappear.
One more thing, you are not converting your labels into indices as you did in the top example. To do that, you can run:
import random
labels = list(df['Scale_num'])
index_aka_labels = torch.tensor(labels)[:,None] #create another dimension
index_aka_labels = index_aka_labels + 2 # labels are [-2,-1,0,1,2] and convert them to [0,1,2,3,4]
index_aka_labels[index_aka_labels==11] = 5 #convert label 9 to index 5
torch.zeros(len(df), 6, dtype=src.dtype).scatter_(1, index_aka_labels, 1)


Instruction: Pytorch broadcasting command not found

Input:
I have the following segment of nested for loop in my code. The nested loop is slowing down my complete execution.
for a torch tensor extended_output with shape [batchSize,nClass*repeat] and another torch tensor with dimension [batchSize,nClass], I want the aggregation to happen as follows:
for q in range(nClass):
    for u in range(repeat):
        output[:,q]=output[:,q]+extended_output[:,(q+u*nClass)]

Here, nClass,repeat all are integer variables with value 1400 and 8 repectively.
Can this nested for loop be avoided using pytorch broadcasting? Any help will be highly useful.
A sample working cpode might be like this
import torch
nClass=1400
repeat=8
batchSize=64
output=torch.zeros([batchSize,nClass])
extended_output=torch.rand([batchSize,nClass*repeat])

for q in range(nClass):
    for u in range(repeat):
        output[:,q]=output[:,q]+extended_output[:,(q+u*nClass)]

Output:
Sorry for the short and probably over-simplified example. I fear a bigger one would be much more difficult to visualize. But I hope this suits your purpose. Here's what I would do:
import torch
nClass    = 3
repeat    = 2
batchSize = 4

torch.manual_seed(0)

output          = torch.zeros([batchSize,nClass])
extended_output = torch.rand([batchSize,nClass*repeat])

for q in range(nClass):
    for u in range(repeat):
        output[:,q]=output[:,q]+extended_output[:,(q+u*nClass)]

idxs = (torch.arange(repeat)*nClass).unsqueeze(0)
idxs = idxs + torch.arange(nClass).unsqueeze(1)
output_vectorized = extended_output[:, idxs].sum(2)

output:
extended_output =
tensor([[0.4963, 0.7682, 0.0885, 0.1320, 0.3074, 0.6341],
        [0.4901, 0.8964, 0.4556, 0.6323, 0.3489, 0.4017],
        [0.0223, 0.1689, 0.2939, 0.5185, 0.6977, 0.8000],
        [0.1610, 0.2823, 0.6816, 0.9152, 0.3971, 0.8742]])
output =
tensor([[0.6283, 1.0756, 0.7226],
        [1.1224, 1.2453, 0.8573],
        [0.5408, 0.8665, 1.0939],
        [1.0762, 0.6794, 1.5558]])
output_vectorized =
tensor([[0.6283, 1.0756, 0.7226],
        [1.1224, 1.2453, 0.8573],
        [0.5408, 0.8665, 1.0939],
        [1.0762, 0.6794, 1.5558]])


Instruction: Can torch.where() used in a equivalent broadcsating form?

Input:
I have the following segment of for loop in my code. The nested loop is slowing down my complete execution.
for q in range(batchSize):
    temp=torch.where((composition_matrix == pred[q]).all(dim=1))[0]
    if len(temp)==0:
        output[q]=0
    else:
        output[q]=int(temp[0])

Here, composition_matrix is [14000,2] dimensional pytorch tensor with only positive integers as cell values. pred and output both are a [batchSize,2] dimensional torch tensor.
As this for loop is slowing my code a lot and I am unable to get the equivalent broadcasting solution to this code segment.
Does a broadcasting solution exists to eleminate this for loop?
I shall be grateful for any help.
A minimum reproducible example is
import torch
composition_matrix=torch.randint(3, 10, (14000,2))
batchSize=64
pred=torch.randint(3, 10, (batchSize,2))
output=torch.zeros([batchSize])

for q in range(batchSize):
    temp=torch.where((composition_matrix == pred[q]).all(dim=1))[0]
    if len(temp)==0:
        output[q]=0
    else:
        output[q]=int(temp[0])

Output:
To make it simple, you first need to understand what the operation is essentially doing. You've got two tensors. Tensor A is of shape (14000, 2) and tensor B is of shape (64, 2). The operation you want to do is:

For each row B[i] in B, compare that B[i] (of shape (2,) with A (of
shape (14000, 2)). If B[i] occurs within A, set output[i] = index of
first occurrence.

This can actually be done in two lines of code (maybe even one line):
comp = (composition_matrix[:, None, :] == pred).all(dim=-1)
output = torch.argmax(comp.float(), axis=0)

The first line creates comp, the broadcasted comparison of composition_matrix and pred, a boolean tensor of shape (14000, 64).

The second line needs to find the "index of the first match". This can be done quite simply with argmax: it will return the index of the first "1" (or if all the values are "0", will return the first index, ie, 0).

(Note that torch does not support argmax for "bool" tensors, and so comp needed to be cast to another data type.)


Instruction: Numerical inconsistency between loop and builtin function

Input:
I'm trying to compute the sum of an array of random numbers. But there seems to be an inconcistancy between the results when I do it one element at a time and when I use the built-in function. Furthermore, the error seems to increase when I decrease the data precision.
import torch
columns = 43*22
rows    = 44
torch.manual_seed(0)
array = torch.rand([rows,columns], dtype = torch.float64)
array_sum = 0
for i in range(rows):
    for j in range(columns):
        array_sum += array[i, j]

torch.abs(array_sum - array.sum())

results in:
tensor(3.6380e-10, dtype=torch.float64)

using dtype = torch.float32 results in:
tensor(0.1426)

using dtype = torch.float16 results in (a whooping!):
tensor(18784., dtype=torch.float16)

I find it hard to believe no one has ever asked about it. Yet, I haven't found a similar question in SO.
Can anyone please help me find some explanation or the source of this error?

Output:
The first mistake is this:
you should change the summation line to
array_sum += float(array[i, j])

For float64 this causes no problems, for the other values it is a problem, the explenation will follow.
To start with: when doing floating point arithmetic, you should always keep in mind that there are small mistakes due to rounding errors. The most simple way to see this is in a python shell:
>>> .1+.1+.1-.3
5.551115123125783e-17

But how do you take these errors into account?
When summing n positive integers to a total tot, the analysis is fairly simple and it the rule is:
error(tot) < tot * n * machine_epsilon
Where the factor n is usually a gross over-estimation and the machine_epsilon is dependant on the type (representation size) of floating point-number.
And is approximatly:
float64: 2*10^-16
float32: 1*10^-7
float16: 1*10^-3

And one would generally expect as an error approximately within a reasonable factor of tot*machine_epsilon.
And for my tests with float16 we get (with always +-40000 variables summing to a total of +- 20000):
error(float64) = 3*10^-10 ≈ 80* 20000 * 2*10^-16
error(float32) = 1*10^-1  ≈ 50* 20000 * 1*10^-7

which is acceptable.
Then there is another problem with the float 16. There is the machine epsilon = 1e-4 and you can see the problem with
>>> ar = torch.ones([1], dtype=float16)
>>> ar
tensor([2048.], dtype=torch.float16)
>>> ar[0] += .5
>>> ar
tensor([2048.], dtype=torch.float16)

Here the problem is that when the value 2048 is reached, the value is not precise enough to be able to add a value 1 or less. More specifically: with a float16 you can 'represent' the value 2048, and you can represent the value 2050, but nothing in between because it has too little bits for that precision. By keeping the sum in a float64 variable, you overcome this problem. Fixing this we get for float16:
error(float16) = 16  ≈ 8* 20000 * 1*10^-4

Which is large, but acceptable as a value relative to 20000 represented in float16.
If you ask yourself, which of the two methods is 'right' then the answer is none of the two, they are both approximations with the same precision, but a different error.
But as you probably guessed using the sum() method is faster, better and more reliable.


Instruction: How to use Pytorch LinAlg Solver?

Input:
I'm trying to replicate the first example on https://pytorch.org/docs/stable/generated/torch.linalg.solve.html
import torch
import time

Acuda = torch.randn(2,3,3,device='cuda')
bcuda = torch.randn(2,3,4,device='cuda')

t1 = time.time()
torch.linalg.torch.solve(Acuda,bcuda)

print('torch took: ',time.time()-t1)

As result I'm getting
Traceback (most recent call last):
 File "linalg_solver_test.py", line 10, in <module>
     torch.linalg.torch.solve(Acuda,bcuda)
     RuntimeError: A must be batches of square matrices, but they are 4 by 3 matrices

My Pytorch Version is 1.7.1.
In contray to the example on the documentation page, I'm using torch.linalg.torch.solve
as torch.linalg.solve does not exist.

Output:
You should use the latest PyTorch 1.9 for LinAlg, because it explicitly mentions "Major improvements to support scientific computing, including torch.linalg" (https://github.com/pytorch/pytorch/releases/tag/v1.9.0)
PyTorch 1.7.1 is rather old. Looks like this version's LinAlg solver doesn't support non-square matrices.


Instruction: Manualy convert pytorch weights to tf.keras weights for convolutional layer

Input:
I'm trying to convert pytorch model to tf.keras model including weights conversion and came across an output missmatch between libraries' outputs.
Here I define two convolutional layers, which should be identical
torch_layer = torch.nn.Conv2d(
    in_channels=3,
    out_channels=64,
    kernel_size=(7, 7),
    stride=(2, 2),
    padding=(3, 3),
    dilation=1,
    groups=1,
    bias=False,
    padding_mode='zeros'
)

tf_layer = tf.keras.layers.Conv2D(
    filters=64,
    kernel_size=(7, 7),
    strides=(2, 2),
    padding='same',
    dilation_rate=(1, 1),
    groups=1,
    activation=None,
    use_bias=False
)
# define model to specify input channel size
tf_model = tf.keras.Sequential([tf.keras.layers.Input((256, 256, 3), batch_size=1), tf_layer])

now I have torch weights and I convert them to tf.keras format
# output_channels, input_channels, x, y
torch_weights = np.random.rand(64, 3, 7, 7)
# x, y, input_channels, output_channels
tf_weights = np.transpose(torch_weights, (2, 3, 1, 0))

# assign weights
torch_layer.weight = torch.nn.Parameter(torch.Tensor(torch_weights))
tf_model.layers[0].set_weights([tf_weights])

now I define input and the outputs are different (shape is the same, values are different), what am I doing wrong?
torch_inputs = np.random.rand(1, 3, 256, 256)
tf_inputs = np.transpose(torch_inputs, (0, 2, 3, 1))

torch_output = torch_layer(torch.Tensor(torch_inputs))
tf_output = tf_model.layers[0](tf_inputs)

Output:
In tensorflow, set_weights is basically used for outputs from get_weights, so it is better to use assign to avoid making mistakes.
Besides, 'same' padding in tensorflow is a little bit complicated. For details, see my SO answer. It depends on input_shape, kernel_size and strides. In your example here, it is translated to torch.nn.ZeroPad2d((2,3,2,3)) in pytorch.
Example codes: from tensorflow to pytorch
np.random.seed(88883)

#initialize the layers respectively
torch_layer = torch.nn.Conv2d(
    in_channels=3,
    out_channels=64,
    kernel_size=(7, 7),
    stride=(2, 2),
    bias=False
)
torch_model = torch.nn.Sequential(
              torch.nn.ZeroPad2d((2,3,2,3)),
              torch_layer
              )

tf_layer = tf.keras.layers.Conv2D(
    filters=64,
    kernel_size=(7, 7),
    strides=(2, 2),
    padding='same',
    use_bias=False
)

#setting weights in torch layer and tf layer respectively
torch_weights = np.random.rand(64, 3, 7, 7)
tf_weights = np.transpose(torch_weights, (2, 3, 1, 0))

with torch.no_grad():
  torch_layer.weight = torch.nn.Parameter(torch.Tensor(torch_weights))

tf_layer(np.zeros((1,256,256,3)))
tf_layer.kernel.assign(tf_weights)

#prepare inputs and do inference
torch_inputs = torch.Tensor(np.random.rand(1, 3, 256, 256))
tf_inputs = np.transpose(torch_inputs.numpy(), (0, 2, 3, 1))

with torch.no_grad():
  torch_output = torch_model(torch_inputs)
tf_output = tf_layer(tf_inputs)

np.allclose(tf_output.numpy() ,np.transpose(torch_output.numpy(),(0, 2, 3, 1))) #True

Edit: from pytorch to tensorflow
torch_layer = torch.nn.Conv2d(
    in_channels=3,
    out_channels=64,
    kernel_size=(7, 7),
    stride=(2, 2),
    padding=(3, 3),
    bias=False
)

tf_layer=tf.keras.layers.Conv2D(
    filters=64,
    kernel_size=(7, 7),
    strides=(2, 2),
    padding='valid',
    use_bias=False
    )

tf_model = tf.keras.Sequential([
           tf.keras.layers.ZeroPadding2D((3, 3)),
           tf_layer
           ])


Instruction: CUDA error: device-side assert triggered on Colab

Input:
I am trying to initialize a tensor on Google Colab with GPU enabled.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

t = torch.tensor([1,2], device=device)

But I am getting this strange error.

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Even by setting that environment variable to 1 seems not showing any further details.
Anyone ever had this issue?

Output:
While I tried your code, and it did not give me an error, I can say that usually the best practice to debug CUDA Runtime Errors: device-side assert like yours is to turn collab to CPU and recreate the error. It will give you a more useful traceback error.
Most of the time CUDA Runtime Errors can be the cause of some index mismatching so like you tried to train a network with 10 output nodes on a dataset with 15 labels. And the thing with this CUDA error is once you get this error once, you will recieve it for every operation you do with torch.tensors. This forces you to restart your notebook.
I suggest you restart your notebook, get a more accuracate traceback by moving to CPU, and check the rest of your code especially if you train a model on set of targets somewhere.


Instruction: Complicated vector multiplication without iterating through the vector

Input:
I'm trying to calculate a loss value in a variation of multiclass classification.
I have my y tensor (the values correspond to the classes):
y = torch.tensor([ 1,  0,  2])

My y_pred is a 3x3 matrix of probability distributions:
y_pred = torch.tensor([[0.4937, 0.2657, 0.2986],
        [0.2553, 0.3845, 0.4384],
        [0.2510, 0.3498, 0.2630]])

The complication is that I also have a distance matrix (each class has some distance to other classes):
d_mtx = torch.tensor([[0, 0.7256, 0.7433],
        [0.6281, 0, 0.1171],
        [0.7580, 0.2513, 0]])

The loss that I'm trying to calculate is:
loss = 0
for class_value in range(len(y)):
    dis = torch.dot(d_mtx[y[class_value]], y_pred[class_value])
    loss += dis

Is there a way to calculate it efficiently without the iteration?
Update 1:
Tried @Yahia Zakaria approach and it works if my y_pred has the same size as my d_mtx, but otherwise I get an error:
RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 0

For example:
y = torch.tensor([ 1,  0,  2, 1])
y_pred = torch.tensor([[0.4937, 0.2657, 0.2986],
        [0.2553, 0.3845, 0.4384],
        [0.2510, 0.3498, 0.2630],
        [0.2510, 0.3498, 0.2630]])

d_mtx = torch.tensor([[0, 0.7256, 0.7433],
        [0.6281, 0, 0.1171],
        [0.7580, 0.2513, 0]])

Output:
You could do it like that:
loss = (d_mtx[y] * y_pred).sum()

This solution assumes the y is of type torch.int64 which is valid for the example you have shown.


Instruction: Why we need to save pytorch models with .net extension?

Input:
I'm a new learner for Pytorch and I am working on a Character_Level_LSTM_Exercise.
Why they save the model with .net extension in the model name?
I'm searching for the explanation but I didn't get any good explanation.
# change the name, for saving multiple files
model_name = 'rnn_x_epoch.net'

checkpoint = {'n_hidden': net.n_hidden,
              'n_layers': net.n_layers,
              'state_dict': net.state_dict(),
              'tokens': net.chars}

with open(model_name, 'wb') as f:
    torch.save(checkpoint, f)

Output:
You can use whatever extension you like! Just make sure to be consistent.
The docu recommends to use .pt extension.
https://pytorch.org/docs/stable/generated/torch.save.html
For more explanation and more extensions options see Soumith Chintala's
 comment.


Instruction: Computing gradient in Tensorflow vs PyTorch

Input:
I am trying to compute the gradient for a loss of a simple linear model. However, I face the problem that while using TensorFlow the gradient is computed as 'none'. Why is this happening and how to compute the gradient using TensorFlow?
import numpy as np
import tensorflow as tf

inputs = np.array([[73, 67, 43],
                   [91, 88, 64],
                   [87, 134, 58],
                   [102, 43, 37],
                   [69, 96, 70]], dtype='float32')

targets = np.array([[56, 70],
                    [81, 101],
                    [119, 133],
                    [22, 37],
                    [103, 119]], dtype='float32')

inputs = tf.convert_to_tensor(inputs)
targets = tf.convert_to_tensor(targets)

w = tf.random.normal(shape=(2, 3))
b = tf.random.normal(shape=(2,))
print(w, b)

def model(x):
  return tf.matmul(x, w, transpose_b = True) + b

def mse(t1, t2):
  diff = t1-t2
  return tf.reduce_sum(diff * diff) / tf.cast(tf.size(diff), 'float32')

with tf.GradientTape() as tape:
  pred = model(inputs)
  loss = mse(pred, targets)

print(tape.gradient(loss, [w, b]))

Here is the working code using PyTorch. The gradients are computed as expected.
import torch

inputs = np.array([[73, 67, 43],
                   [91, 88, 64],
                   [87, 134, 58],
                   [102, 43, 37],
                   [69, 96, 70]], dtype='float32')

targets = np.array([[56, 70],
                    [81, 101],
                    [119, 133],
                    [22, 37],
                    [103, 119]], dtype='float32')

inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)

w = torch.randn(2, 3, requires_grad = True)
b = torch.randn(2, requires_grad = True)

def model(x):
  return x @ w.t() + b

def mse(t1, t2):
  diff = t1 - t2
  return torch.sum(diff * diff) / diff.numel()

pred = model(inputs)
loss = mse(pred, targets)
loss.backward()

print(w.grad)
print(b.grad)

Output:
Your code doesn't work because in tensorflow, gradients are only computed for tf.Variables. When you create a layer, TF automatically marks its weights and biases as a variable (unless you specify trainable=False).
So, in order to make your code work, all you need to do is wrap your w and b with tf.Variable
w = tf.Variable(tf.random.normal(shape=(2, 3)), name='w')
b = tf.Variable(tf.random.normal(shape=(2,)), name='b')

Use these lines to define your weights and biases, and you will get actual values in your final print.


Instruction: Getting a specific vector from a matrix using elements of other vectors

Input:
I have 2 vectors:
x = torch.tensor([ 0,  1,  0,  0,  2,  2,  3,  3,  0,  0,  2,  2,  4,  4,  0,  5,  0,  4,
         6,  7,  6,  8,  6,  9,  6, 10,  5, 11,  4, 11,  8, 12, 10, 12,  9, 12,
        13, 14, 15, 16, 17, 14, 17, 18, 19, 16, 19, 18])

y = torch.tensor([ 1,  0,  2,  3,  0,  3,  2,  0,  2,  4,  0,  4,  2,  0,  5,  0,  4,  0,
         7,  6,  8,  6,  9,  6, 10,  6, 11,  5, 11,  4, 12,  8, 12, 10, 12,  9,
        14, 13, 16, 15, 14, 17, 18, 17, 16, 19, 18, 19])

I also have a matrix:
mtx = torch.rand(20,20)

Is there a way to get the corresponding [i,i] vector from the matrix using the 2 vectors?
That is, x[0] = 0, y[0] = 1, so the first element of the vector will be mtx[0,1].
x[1] = 1, y[1] = 0, so the second element of the vector will be mtx[1,0], and so on.
I'm looking for an answer without iterating through the vectors (e.g for element in x...?

Output:
You can use fancy indexing:

output_vector = mtx[[x,y]]

where output_vector is the vector of mtx values at the indices in the lists 'x' and 'y' as requested.


Instruction: Can someone please explain about the functionality of "torch.bernoulli"?

Input:
I have a question about torch.bernoulli. According to the documentation:

Draws binary random numbers (0 or 1) from a Bernoulli distribution. The input tensor should be a tensor containing probabilities to be used for drawing the binary random number. Hence, all values in input have to be in the range:
0 ≤ input i ≤ 1. The i-th element of the output tensor will draw a value 1 according to i-th probability value given in input.

I don't quite get it. For example, if we have torch.bernoulli([0.599, 0.0846, 0.0179, 0.0742, 0.0742]), once it returns tensor([1., 0., 0., 0., 0.]). Another time it returns tensor([0., 0., 0., 1., 0.]). I don't understand why it behaves like that. The documentation states "...for drawing the binary random number...". But the whole proccess is vague to me. Can someone explain please?

Output:
Let's simplify your example to the input tensor [0.60, 0.07, 0.12].  Now, on any given draw, there is a ...
60% chance that element 1 will be 1
 7% chance that element 2 will be 1
12% chance that element 3 will be 1
... and any elements that aren't 1 are 0.

These events are independent.  For instance, there is a chance of 0.60 * 0.07 * 0.12 that all three elements will be 1.  There is a 0.40 * 0.93 * 0.88 chance that all three will be 0.
Does that help clear it up?


Instruction: Problem with installing pytorch with pip3: -f option requires 1 argument

Input:
I am trying to install torch in linux with cuda version 11.1
I checked this: Start Locally | PyTorch
It says that the code is
pip3 install --user torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f

However, this line gives this error
-f option requires 1 argument

Can someone help?

Output:
you must have missed https://download.pytorch.org/whl/torch_stable.html
use the below cmd
pip3 install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Loss of accuracy when inferring with a pytorch rnn model

Input:
I am training a model using a pytorch RNN model and have multiple csv files to train and infer. If I train file #1 and infer on file #1 I get ~100% accurate predictions. If I train on file #1 and infer on, say, file #4 or file #2 then accuracy drops to ~80%. Here’s what I am doing:
1. Read the file and separate the features (X) and labels (y) into two dataframes.
2. The range of my values, both features and labels, is high. So I apply scaling transformation.
3. Then I split data as train and test.
4. Instantiate model.train() and run train data through the rnn model.
5. Instantiate model.eval() and get the predictions from the model with the test data.
6. Reverse scale the predictions.
7. Calculate mean-square error.

So far this is all good. My MSE is very, very low which is good.
After training, I need to infer a randomly selected file. Here’s what I am doing for inference:
1. Read the single file and separate the features (X) and labels (y) into two dataframes.
2. Apply scaling transformation.
3. Instantiate model.eval().
4. Get the predictions.
5. Reverse scale the predictions

If the inference file is the same as the trained file accuracy is close to 100%. If I use a different file for inference why does the accuracy drop? Am I doing something wrong? Unfortunately, I cannot share the code due to confidentiality.

Output:
With the additional information provided in the comment, I would say it is most likely a problem with over-fitting, rather than any mistake in implementation.
Your model is learning the class distribution of file #1, which is then useful to predict the test set of file #1, but which does not translate to the other test sets.
To solve this, my suggestion would be to sample a training set from all the available files, such that it more closely resembles the distribution found in the collection of test sets, rather than a single test set.
Delving into other RNN over-fitting solutions might be worthwhile too.


Instruction: How to batch matrix-vector multiplication (one matrix, many vectors) in pytorch without duplicating the matrix in memory

Input:
I have n vectors of size d and a single d x d matrix J. I'd like to compute the n matrix-vector multiplications of J with each of the n vectors.
For this, I'm using pytorch's expand() to get a broadcast of J, but it seems that when computing the matrix vector product, pytorch instantiates a full n x d x d tensor in the memory. e.g. the following code
device = torch.device("cuda:0")
n = 100_000_000
d = 10

x = torch.randn(n, d, dtype=torch.float32, device=device)
J = torch.randn(d, d, dtype=torch.float32, device=device).expand(n, d, d)
y = torch.sign(torch.matmul(J, x[..., None])[..., 0])

raises
RuntimeError: CUDA out of memory. Tried to allocate 37.25 GiB (GPU 0; 11.00 GiB total capacity; 3.73 GiB already allocated; 5.69 GiB free; 3.73 GiB reserved in total by PyTorch)

which means that pytorch, unnecessarily, tries to allocate space for n copies of the matrix J
How can I perform this task in a vectorized way (the matrices are small, so I don't want to loop over each matrix-vector multiplication) without exhausting my GPU's memory?

Output:
I think this will solve it:
import torch
x = torch.randn(n, d)
J = torch.randn(d, d) # no need to expand

y = torch.matmul(J, x.T).T

Verifying using your expression:
Jex = J.expand(n, d, d)
y1 = torch.matmul(Jex, x[..., None])[..., 0]
y = torch.matmul(J, x.T).T

torch.allclose(y1, y) # using allclose for float values
# tensor(True)


Instruction: An easy way to create a torch tensor from multiple elements of tuple through concatenate

Input:
Inputs
I have list as follow
r1 = [([[[1, 2, 3], [1, 2, 3]],
        [[4, 5, 6], [4, 5, 6]]],
       [[[7, 8], [7, 8]],
        [[9, 10], [9, 10]]]),

      ([[[11, 12, 13], [11, 12, 13]],
        [[14, 15, 16], [14, 15, 16]]],
       [[[17, 18], [17, 18]],
        [[19, 20], [19, 20]]])]

I'm going to make 2 torch tensors from the input above.
my desired output is as follow
output
output =
[tensor([[[ 1,  2,  3],
          [ 1,  2,  3]],

         [[ 4,  5,  6],
          [ 4,  5,  6]],

         [[11, 12, 13],
          [11, 12, 13]],

         [[14, 15, 16],
          [14, 15, 16]]]),

 tensor([[[ 7,  8],
          [ 7,  8]],

         [[ 9, 10],
          [ 9, 10]],

         [[17, 18],
          [17, 18]],

         [[19, 20],
          [19, 20]]])]

My code is as follows.
output = []
for i in range(len(r1[0])):
    templates = []
    for j in range(len(r1)):
        templates.append(torch.tensor(r1[j][i]))
        template = torch.cat(templates)
    output.append(template)

Is there a simpler or easier way to get the result I want?

Output:
This will do:
output = [torch.Tensor([*a, *b]) for a, b in zip(*r1)]

It concatenates the corresponding items of the two list first then create the Tensor


Instruction: Pytorch Dataloader for reading a large parquet/csv file

Input:
I am trying to get Pytorch to train records of a single parquet file, without having to read the entire file in memory at once since it won't fit in memory. Since the file is stored remotely, I would rather keep it as a single file, as training using IO for many files is extremely expensive. How can I use Pytorch's IterableDataset or Dataset to read smaller chunks of the file during training when I want to specify the number of batches in the DataLoader? I know that the map-style Dataset won't work in this case since I need everything in one file rather than reading the index of each file.
I managed to implement this in Tensorflow using tfio.IODataset and tf.data.Dataset, but I can't find the equivalent way to implement it in Pytorch.

Output:
I found a workaround using torch.utils.data.Dataset, but the data must be manipulated using dask beforehand such that each partition is a user, stored as its own parquet file, but can be read only once later. In the following code, the labels and the data are stored separately for the multivariate timeseries classification problem (but can be easily adapted to other tasks as well):
import dask.dataframe as dd
import pandas as pd
import numpy as np
import torch
from torch.utils.data import TensorDataset, DataLoader, IterableDataset, Dataset

# Breakdown file
raw_ddf = dd.read_parquet("data.parquet") # Read huge file using dask
raw_ddf = raw_ddf.set_index("userid") # set the userid as index
userids = raw_ddf.index.unique().compute().values.tolist() # get a list of indices
new_ddf = raw_ddf.repartition(divisions = userids) # repartition by userids
new_ddf.to_parquet("my_folder") # this will save each user as its own parquet file within "my_folder"

# Dask to read the partitions
train_ddf = dd.read_parquet("my_folder/*.parquet") # read all files

# Read labels file
labels_df = pd.read("label.csv")
y_labels = np.array(labels_df["class"])

# Define the Dataset class
class UsersDataset(Dataset):
    def __init__(self, dask_df, labels):
        self.dask_df = dask_df
        self.labels = labels

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        X_df = self.dask_df.get_partition(idx).compute()
        X = np.row_stack([X_df])
        X_tensor = torch.tensor(X, dtype=torch.float32)
        y = self.labels[idx]
        y_tensor = torch.tensor(y, dtype=torch.long)
        sample = (X_tensor, y_tensor)
        return sample

# Create a Dataset object
user_dataset = UsersDataset(dask_df=ddf_train, labels = y_train)

# Create a DataLoader object
dataloader = DataLoader(user_dataset, batch_size=4, shuffle=True, num_workers=0)

# Print output of the first batch to ensure it works
for i_batch, sample_batched in enumerate(dataloader):
    print("Batch number ", i_batch)
    print(sample_batched[0]) # print X
    print(sample_batched[1]) # print y

    # stop after first batch.
    if i_batch == 0:
        break

I would like to know how can I adapt my approach when using >= 2 workers to read the data, without duplicate entries. Any insights on this are greatly appreciated.


Instruction: Pytorch - building from source - CMAKE_BUILD_WITH_INSTALL_RPATH

Input:
I am building PyTorch from source as my GPU card is not supported by the packages in pip or conda.
I am in Ubuntu 20.04. I followed the instructions at :
I get this error. I searched the internet but there are no useful pointers.
There are numerous errors like below:
CMake Error at modules/observers/CMakeLists.txt:12 (add_library):
  The install of the caffe2_observers target requires changing an RPATH from
  the build tree, but this is not supported with the Ninja generator unless
  on an ELF-based platform.  The CMAKE_BUILD_WITH_INSTALL_RPATH variable may
  be set to avoid this relinking step.

Any idea how this could be resolved. I already tried setting up an environment variable as below but it did not help.
$ export CMAKE_BUILD_WITH_INSTALL_RPATH=On
$ echo ${CMAKE_BUILD_WITH_INSTALL_RPATH}
On

Output:
I found that installing cmake solves the problem.
sudo apt install cmake

I could not debug why it worked. I was in ubuntu version 20.04. May be it had a cmake that was too old and running the above command fetches the latest version.
I thought this is still a useful information for someone facing the same issues - so, noting down here.


Instruction: Pytorch CNN Model: Dimension out of range Error

Input:
I keep getting this error "IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)" when I use cross entropy loss my CNN model.
My layers look like this:
    x= self.conv1(x)
    x= F.relu(x)
    x= self.pool1(x)
    x= F.relu(x)
    x= self.conv3(x)
    x= F.relu(x)
    x= self.conv4(x)
    x= F.relu(x)

    x= self.pool2(x)

    x = torch.flatten(x, 1)

    x= self.lin1(x)
    x= F.relu(x)
    x= self.lin2(x)
    x= F.relu(x)
    x= self.lin3(x)
    x= F.relu(x)
    x= self.sftmax(x)

My code for training is pretty much the one on the [Pytorch website].
where x_train is of shape (300, 1000) and y_train has 300 labels. I want to feed in one array in x_train at a time, then optimize based on its corresponding label in y_train.
When I am training, the outputs= net(Input) works without error. For the first epoch, I get an output like tensor([[[0.1072, 0.2725, 0.2963, 0.2395, 0.3821]]], dtype=torch.float64, grad_fn=<UnsqueezeBackward0>). Then I pass in the Label, which is tensor([5.], dtype=torch.float64).
After this first epoch though, I get an error "IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)". Is this related to the shape of the input and outputs? I'm not sure what is wrong.
Any advice would be appreciated!

Output:
I believe the issue might be due to your usage of the squeeze(0) right before you pass in outputs to the loss function.
Note in the docs below it says that the output tensor and input sensor will share the same memory space.
https://pytorch.org/docs/stable/generated/torch.squeeze.html
Edit
After testing with the collab attached to the example. It seems that is not the case. Although I noticed that your labels tensor is of size one. Whereas the example has label arrays that match the size of the output space.
I believe you might be extracting a single value out of the training set as opposed to the expected tensor size? Essentially the output should match the training set output in size so it can calculate the loss between the two.


Instruction: What is the fastest way to use Pytorch Conv2d to apply multiple filters to every layer in multiple CT scans?

Input:
Assume an input data set contains CT scans of 100 patients, each scan containing 16 layers and each layer containing 512 x 512 pixels.  I want to apply eight 3x3 convolution filters to each layer in every CT scan.  So, the input array has shape [100, 16, 512, 512] and the kernels array has shape [8, 3, 3].  After the convolutions are applied, the goal is an output array with a shape [100, 16, 8, 512, 512].  The following code uses Pytorch Conv2d function to achieve this; however, I want to know if the groups parameter (and/or other means) can somehow eliminate the need for the loop.
 for layer_index in range(0, number_of_layers):
    # Getting current ct scan layer for all patients
    # ct_scans dimensions are:  [patient, scan layer, pixel row, pixel column]
    # ct_scans shape: [100, 16, 512, 512]
    image_stack = ct_scans[:, layer_index, :, :]
    # Converting from numpy to tensor format
    image_stack_t = torch.from_numpy(image[:, None, :, :])
    # Applying convolution to create 8 filtered versions of current scan layer across all patients
    # shape of kernels is: [8, 3, 3]
    filtered_image_stack_t = conv2d(image_stack_t, kernels, padding=1, groups=1)
    # Converting from tensor format back to numpy format
    filtered_image_stack = filtered_image_stack_t.numpy()
    # Amassing filtered ct scans for all patients back into one array
    # filtered_ct_scans dimensions are: [patient, ct scan layer, filter number, pixel row, pixel column]
    # filtered_ct_scans shape is: [100, 16, 8, 512, 512]
    filtered_ct_scans[:, layer_index, :, :, :] = filtered_image_stack

So far, my attempts to use anything other than groups=1 leads to errors.  I also found the following similar posts; however, they don't address my specific question.
How to use groups parameter in PyTorch conv2d function with batch?
How to use groups parameter in PyTorch conv2d function

Output:
You do not need to use grouped convolutions. Resizing you input appropriately is all that is needed.
import torch
import torch.nn.functional as F

ct_scans = torch.randn((100,16,512,512))
kernels = torch.randn((8,1,3,3))

B,L,H,W = ct_scans.shape #(batch,layers,height,width)
ct_scans = ct_scans.view(-1,H,W)
ct_scans.unsqueeze_(1)
out = F.conv2d(ct_scans, kernels)
out = out.view(B,L,*out.shape[1:])
print(out)


Instruction: What information does Pytorch nn.functional.interpolate use?

Input:
I have a tensor img in PyTorch of size bx2xhxw and want to upsample it using torch.nn.functional.interpolate. But while interpolation I do not wish channel 1 to use information from channel 2. To do this should I do,
img2 = torch.rand(b,2,2*h,2*w) # create a random torch tensor.
img2[:,0,:,:] = nn.functional.interpolate(img[:,0,:,:], [2*h,2*w], mode='bilinear', align_corners=True)
img2[:,1,:,:] = nn.functional.interpolate(img[:,1,:,:], [2*h,2*w], mode='bilinear', align_corners=True)
img=img2

or simply using
img = nn.functional.interpolate(img, [2*h,2*w], mode='bilinear', align_corners=True)

will solve my purpose.

Output:
You should use (2). There is no communication in the first and second dimensions (batch and channel respectively) for all types of interpolation (1D, 2D, 3D), as they should be.
Simple example:
import torch
import torch.nn.functional as F

b = 2
c = 4
h = w = 8

a = torch.randn((b, c, h, w))
a_upsample = F.interpolate(a, [h*2, w*2], mode='bilinear', align_corners=True)

a_mod = a.clone()
a_mod[:, 0] *= 1000
a_mod_upsample = F.interpolate(a_mod, [h*2, w*2], mode='bilinear', align_corners=True)

print(torch.isclose(a_upsample[:,0], a_mod_upsample[:,0]).all())
print(torch.isclose(a_upsample[:,1], a_mod_upsample[:,1]).all())
print(torch.isclose(a_upsample[:,2], a_mod_upsample[:,2]).all())
print(torch.isclose(a_upsample[:,3], a_mod_upsample[:,3]).all())

Output:
tensor(False)
tensor(True)
tensor(True)
tensor(True)

One can tell that a large change in the first channel has no effect in other channels.


Instruction: AttributeError: Caught AttributeError in DataLoader worker process 0

Input:
I have done some googling, and this error seems to commonly appear but I am unsure about how to solve it/ I am currently doing the PyTorch torchvision tutorial (https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) for segmentation. However, I am getting thrown the titular error:
Traceback (most recent call last):
  File "main.py", line 139, in <module>
    main()
  File "main.py", line 131, in main
    print_freq=10)
  File "/engine.py", line 26, in train_one_epoch
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
  File "/utils.py", line 180, in log_every
    for obj in iterable:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 311, in __getitem__
    return self.dataset[self.indices[idx]]
  File "main.py", line 64, in __getitem__
    img, target = self.transforms(img.target)
  File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 546, in __getattr__
    raise AttributeError(name)
AttributeError: target

I believe I have followed the tutorial completely, copying their code - I'm a bit unsure, therefore, why this error is arising. For a beginner who's trying to learn how to use ML in computer vision it's especially difficult to figure out what's going on here.
I don't know if this is important, but I'm running Python3 on Ubuntu.
Many thanks in advance for your help! :)

Output:
I think you mistyped the , with . in __getitem__ function of PennFudanDataset
Your version:
        if self.transforms is not None:
            img, target = self.transforms(img.target)

Tutorial:
        if self.transforms is not None:
            img, target = self.transforms(img, target)


Instruction: Extracting hidden representations from an autoencoder using Pytorch

Input: After having trained an AutoEncoder with PyTorch, how can I extract the low-dimensional embeddings of input features at some hidden-level?

Output:
You can just define your model such that it optionally returns the intermediate pytorch variable calculated during the forward pass. Simple example:
class Autoencoder(nn.Module):
    def __init__(self, input_size, hidden_size):
    super().__init__()
    self.encoder = nn.Sequential(
    nn.Linear(input_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, 3)) #reduce the size

    self.decoder = nn.Sequential(
    nn.Linear(3, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, input_size),
    nn.ReLU()) #reduce the size

def forward(self, x, return_encoding = False):
    encoded = self.encoder(x)
    decoded = self.decoder(encoded)

    if return_encoding:
        return decoded,encoded
    return decoded


Instruction: TypeError: __array__() takes 1 positional argument but 2 were given

Input:
I've been doing the pytorch tutorial (https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) and have been getting this error that I don't know how to fix. The full error is below:
Traceback (most recent call last):
  File "main.py", line 146, in <module>
    main()
  File "main.py", line 138, in main
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
  File "/engine.py", line 26, in train_one_epoch
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
  File "/utils.py", line 180, in log_every
    for obj in iterable:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 311, in __getitem__
    return self.dataset[self.indices[idx]]
  File "main.py", line 64, in __getitem__
    img, target = self.transforms(img, target)
  File "/transforms.py", line 26, in __call__
    image, target = t(image, target)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/transforms.py", line 50, in forward
    image = F.to_tensor(image)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py", line 129, in to_tensor
    np.array(pic, mode_to_nptype.get(pic.mode, np.uint8), copy=True)
TypeError: __array__() takes 1 positional argument but 2 were given

I believe it means somewhere I'm using an array with 2 arguments which isn't allowed, but I don't really know where abouts that is happening - perhaps in one of their pre written libraries?
I can share the code in full if desired, but thought its a bit unwieldy. Does anyone know what might be causing this error?

Output:
I had the same error when using:
torch==1.9.0
torchvision==0.10.0

In my requirements.txt file I downgraded the torch library, which forced me to downgrade torchvision, and that fixed the error for me.  The library versions I ended up using that did not raise the error were:
torch==1.8.1
torchvision==0.9.1


Instruction: What does the group_ids parameter of the TimeSeriesDataSet class specifically do in PyTorch Forecasting?

Input:
I am currently working with PyTorch Forecasting and I want to create dataset with TimeSeriesDataSet. My original data lies in a pandas Dataframe and looks like this:
date         amount        location
2014-01-01     5               A
2014-01-01     7               B
    ...       ...             ...
2017-12-30     4               H
2017-12-31     8               I

So in total I got nine different unique values in "location" and an amount for each location per date. Now I am wondering what the group_ids parameter for the TimeSeriesDataSet class does and what it exact behaviour is? I am not really getting the idea based on the documentation.
Thanks a lot in advance!

Output:
A time-series dataset usually contains multiple time-series for different entities/individuals.
group_ids is a list of columns which uniquely determine entities with associated time series. In your example it would be location:

group_ids (List[str]) – list of column names identifying a time series. This means that the group_ids identify a sample together with the time_idx. If you have only one timeseries, set this to the name of column that is constant.


Instruction: torchaudio "RuntimeError: Error loading audio file: failed to open file" for wav file

Input:
I was following this tutorial
However, when I ran the code under "Training", it gave me the following error.
RuntimeError: Error loading audio file: failed to open file /Users/leonardchoo/Desktop/dev_m1/audio_cnn/UrbanSound8K/fold10/30344-3-0-1.wav

The problem is, each time I run this, the error occurs at a different file.
error at...
# 1st run
fold2/106015-5-0-7.wav
# 2nd run
fold2/76086-4-0-22.wav
# 3rd run
fold9/14111-4-0-6.wav

I couldn't find anything like this on the web. I am genuinely confused by this.
My entire code can be found in this COLAB Notebook
The dataset is from here
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-2d83571e984b> in <module>
     59
     60 num_epochs = 2   # Just for demo, adjust this higher.
---> 61 training(myModel, train_dl, num_epochs)

<ipython-input-14-2d83571e984b> in training(model, train_dl, num_epochs)
     19
     20         # Repeat for each batch in the training set
---> 21         for i, data in enumerate(train_dl):
     22             # Get the input features and target labels, and put them on the GPU
     23             inputs, labels = data[0].to(device), data[1].to(device)

/usr/local/lib/python3.9/site-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.9/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.9/site-packages/torch/utils/data/dataset.py in __getitem__(self, idx)
    309
    310     def __getitem__(self, idx):
--> 311         return self.dataset[self.indices[idx]]
    312
    313     def __len__(self):

<ipython-input-8-8743d21efeae> in __getitem__(self, idx)
     32         class_id = self.df.loc[idx, 'classID']
     33
---> 34         aud = AudioUtil.open(audio_file)
     35         # Some sounds have a higher sample rate, or fewer channels compared to the
     36         # majority. So make all sounds have the same number of channels and same

<ipython-input-7-f64618dd0374> in open(audio_file)
     13     @staticmethod
     14     def open(audio_file):
---> 15         sig, sr = torchaudio.load(audio_file)
     16         return (sig, sr)
     17

/usr/local/lib/python3.9/site-packages/torchaudio/backend/sox_io_backend.py in load(filepath, frame_offset, num_frames, normalize, channels_first, format)
    150                 filepath, frame_offset, num_frames, normalize, channels_first, format)
    151         filepath = os.fspath(filepath)
--> 152     return torch.ops.torchaudio.sox_io_load_audio_file(
    153         filepath, frame_offset, num_frames, normalize, channels_first, format)
    154

RuntimeError: Error loading audio file: failed to open file /Users/leonardchoo/Desktop/dev_m1/audio_cnn/UrbanSound8K/fold10/30344-3-0-1.wav

Output:
It was a silly trouble... i was missing /audio in the path...
I did not realize this was the case because the file it gave me error at always changed.


Instruction: What is causing the allocation of 12 GB of memory and causing CUDA out of memory error? Model, Data or something else?

Input:
I am trying to build a 3D CNN based video classifier using Pytorch. When i try to run a single datapoint i run into this error:
CUDA out of memory. Tried to allocate 1.20 GiB (GPU 0; 14.76 GiB total capacity; 12.60 GiB already allocated; 1.09 GiB free; 12.61 GiB reserved in total by PyTorch)

My data of 1000 videos has a size of around 90MB on disk. I think i am only loading model and data onto GPU. I am not able to understand what is causing the already allocated 12.6 GiB of memory? is my model too big or it has something to do with data loaded or something else? Here is a snippet of my model. It is based on C3D by Tran et al, 2015.
    self.conv1 = nn.Conv3d(3, 64, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool1 = nn.MaxPool3d(kernel_size=(1, 2, 2), stride=(1, 2, 2))

    self.conv2 = nn.Conv3d(64, 128, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool2 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv3a = nn.Conv3d(128, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv3b = nn.Conv3d(256, 256, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool3 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv4a = nn.Conv3d(256, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv4b = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool4 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2))

    self.conv5a = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.conv5b = nn.Conv3d(512, 512, kernel_size=(3, 3, 3), padding=(1, 1, 1))
    self.pool5 = nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2), padding=(0, 1, 1))

    self.fc6 = nn.Linear(373248, 4096)
    self.fc7 = nn.Linear(4096, 128)
    self.fc8 = nn.Linear(128, 2)

    self.dropout = nn.Dropout(p=0.5)

    self.relu = nn.ReLU()
    self.softmax = nn.Softmax()

Also, i am using Dataset, DataLoader and DeviceDataLoader to load the data. Even if they load all the data at once, what is causing to inflate 90MB of data to 12.6GiB.

Output:
It's important to note that the real reason you have out of memory issues most of the time is not necessarily the inherent size of the model itself (though it is directly related to this). Even 100,000,000 parameters, with single floating point precision, only takes about 1 GB to store. (Now, in your case, your model actually is so large that just your parameters may consume on the order of 100GB of memory, but in most models this wouldn't be the  case).
Generally, you'll have memory issues when the forward pass through the model computes and stores the gradient of each parameter, with respect to the previous layer's parameters. This can temporarily double (or more) the number of floating point values being stored in memory, so the combined total of your model, your data, and the gradient/computation graph exceed the available GPU memory.
As suggested above, your solutions are

reduce your batch size
downsample your data (if possible)
reduce the complexity of your model

(As an aside, I'm not totally sure how you are intending to train this model but you cannot possibly hope to feed a whole video to your model as input. Aside from the memory issues, it would be extremely difficult for the model to learn the temporal relationships necessary to make sense of the data. Conventional video processing techniques would probably pass a single frame or a few frames at a time to a model and use a transformer, lstm, or other sequence-to-output model to learn the temporal context.)


Instruction: What is the output of pytorch RNN?

Input:
I have a simple rnn code below.
rnn = nn.RNN(1, 1, 1, bias = False, batch_first = True)
t = torch.ones(size = (1, 2, 1))
output, hidden = rnn(t)
print(rnn.weight_ih_l0)
print(rnn.weight_hh_l0)
print(output)
print(hidden)

# Outputs
Parameter containing:
tensor([[0.7199]], requires_grad=True)

Parameter containing:
tensor([[0.4698]], requires_grad=True)

tensor([[[0.6168],
     [0.7656]]], grad_fn=<TransposeBackward1>)
tensor([[[0.7656]]], grad_fn=<StackBackward>)

tensor([[[0.7656]]], grad_fn=<StackBackward>)

My understanding from the PyTorch documentation is that the output from above is the hidden state.
So, I tried to manually calculate the output using the below
hidden_state1 = torch.tanh(t[0][0] * rnn.weight_ih_l0)
print(hidden_state1)
hidden_state2 = torch.tanh(t[0][1] * rnn.weight_ih_l0 + hidden_state1 * rnn.weight_hh_l0)
print(hidden_state2)

tensor([[0.6168]], grad_fn=<TanhBackward>)
tensor([[0.7656]], grad_fn=<TanhBackward>)

The result was correct. hidden_state1 and hidden_state2 match the output.
Shouldn’t the hidden_states get multiplied with output weights to get the output?
I checked for weights connecting from hidden state to output. But there are no weights at all.
If the objective of rnn is to calculate only hidden states, Could anyone tell me how to get the output?

Output:
Shouldn’t the hidden_states get multiplied with output weights to get
the output

Yes and No. It depends on your problem formulation. Suppose you are dealing with a case where output from last timestep only matters. In that case it really doesn't make sense to multiply hidden state to output weight in each unit.
That's why pytorch only gives you hidden output as an abstract value, after that you can really go wild and do whatever you want with hidden states according to your problem.
In your particular case suppose you want to apply another linear layer to the output at each timestep. You can do so simply by defining a linear layer and propagating the output of hidden unit.
#Linear Layer
##hidden_feature_size = 1 in your case
lin_layer = nn.Linear(hidden_feature_size, output_feature_size)
#output from first timestep
linear_layer(output[0])
#output from second timestep
linear_layer(output[1])


Instruction: To calculate euclidean distance between vectors in a torch tensor with multiple dimensions

Input:
There is a random initialized torch tensor of the shape as below.
Inputs
tensor1 = torch.rand((4,2,3,100))
tensor2 = torch.rand((4,2,3,100))

tensor1 and tensor2 are torch tensors with 24 100-dimensional vectors, respectively.
I want to get a tensor with a shape of torch.size([4,2,3]) by obtaining the Euclidean distance between vectors with the same index of two tensors.
I used dist = torch.nn.functional.pairwise_distance(tensor1, tensor2) to get the results I wanted.
However, the pairwise_distance function calculates the euclidean distance for the second dimension of the tensor. So dist shape is torch.size([4,3,100]).
I have performed transpose several times to solve these problems. My code is as follows.
tensor1 = tensor1.transpose(1,3)
tensor2 = tensor2.transpose(1,3)
dist = torch.nn.functional.pairwise_distance(tensor1, tensor2)
dist = dist.transpose(1,2)

Is there a simpler or easier way to get the result I want?

Output:
Here ya go
dist = (tensor1 - tensor2).pow(2).sum(3).sqrt()

Basically that's what Euclidean distance is.
Subtract -> power by 2 -> sum along the unfortunate axis you want to eliminate-> square root


Instruction: Using model.eval() on neural network results in same output everytime for very different inputs

Input:
I have a simple network implemented in pytorch say,
class network:

    def __init__(self):

        self.device = device
        #these are the 3 convolutional synapses; Same convolution;
        self.layer     = sequential(
                                conv2d(3, 3, (23), padding=11),
                                batch_norm_2d(3),
                                Swish(),

                                conv2d(3, 3, (11), padding=5),
                                batch_norm_2d(3),
                                Swish(),

                                conv2d(3, 3, (5), padding=2),
                                batch_norm_2d(3),
                                Swish(),

                                conv2d(3, 4, (3), padding=15, stride=2),
                                batch_norm_2d(4),
                                Swish(),

                                conv2d(4, 8, (3), padding=15, stride=2),
                                batch_norm_2d(8),
                                Swish(),

                                conv2d(8, 4, (1)),
                                batch_norm_2d(4),
                                Swish(),

                                conv2d(4, 8, (3), padding=15, stride=2),
                                batch_norm_2d(8),
                                Swish(),

                                conv2d(8, 16, (3), padding=15, stride=2),
                                batch_norm_2d(16),
                                Swish(),

                                conv2d(16, 8, (1)),
                                batch_norm_2d(8),
                                Swish(),

                                conv2d(8, 16, (3), padding=15, stride=2),
                                batch_norm_2d(16),
                                Swish(),

                                conv2d(16, 32, (3), padding=15, stride=2),
                                batch_norm_2d(32),
                                Swish(),

                                conv2d(32, 16, (1)),
                                batch_norm_2d(16),
                                Swish(),

                                conv2d(16, 32, (3), padding=15, stride=2),
                                batch_norm_2d(32),
                                Swish(),

                                conv2d(32, 64, (3), padding=15, stride=2),
                                batch_norm_2d(64),
                                Swish(),

                                conv2d(64, 32, (1)),
                                batch_norm_2d(32),
                                Swish(),

                                conv2d(32, 64, (3), padding=15, stride=2),
                                batch_norm_2d(64),
                                Swish(),

                                conv2d(64, 128, (3), padding=15, stride=2),
                                batch_norm_2d(128),
                                Swish(),

                                conv2d(128, 64, (1)),
                                batch_norm_2d(64),
                                Swish(),

                                conv2d(64, 128, (3), padding=15, stride=2),
                                batch_norm_2d(128),
                                Swish(),

                                conv2d(128, 256, (3), padding=15, stride=2),
                                batch_norm_2d(256),
                                Swish(),

                                conv2d(256, 128, (1)),
                                batch_norm_2d(128),
                                Swish(),

                                flatten(1, -1),

                                linear(128*29*29, 8*8*2*5),
                                batch_norm_1d(8*8*2*5),
                                Swish()
            )

        #loss and optimizer functions for ethirun
        self.Loss_1 = IoU_Loss() #the loss function for bounding box.
        self.Loss_2 = tor.nn.SmoothL1Loss(reduction='mean')

        #the optimizer
        self.Optimizer =     tor.optim.AdamW(self.parameters())#tor.optim.SGD(self.parameters(), lr=1e-2, momentum=0.9, weight_decay=1e-5, nesterov=True)
        self.Scheduler = tor.optim.lr_scheduler.StepLR(self.Optimizer, 288, gamma=0.5)
        self.sizes = tor.tensor(range(0, 5), dtype=tor.int64, device=self.device)

    def forward(self, input):
         return self.layer(input)

    def backprop(self, preds, lbls, val_or_trn):
    #takes predictions and labels and calculates error and backpropagates
         mask = tor.index_select(lbls, -1, self.sizes[0])
         preds.register_hook(lambda grad: grad * mask.float())
         error = self.Loss_2(preds, lbls)

         if val_or_trn == 1:
             #backpropagation
             error.backward()
             self.Optimizer.step()
             self.Scheduler.step()

             #zeroing the gradients.
             self.Optimizer.zero_grad()

        return error.detach()

model = network()

Where the inputs, outputs and channels are arbitrary. Then say I create some random input tensor like this,
input_data = torch.randn(1, 3, 256, 256)
Then I predict some result in this data like this,
model(input_data)
And say I also change the input_data variable by initiating the torch.randn command a bunch of different times while keeping the model same. That is not re-initiating the model=network() command.
I get this error,
Expected more than 1 value per channel when training, got input size torch.Size([1, some_value])
So, I tried running it in evaluation mode by using the model.eval() function like this,
model.eval()

with tor.no_grad()
   pred = model(input_data)

model.train()

This works without errors. However no matter how I change the input_data variable I always get the same value in pred. If I however re-initiate the model's parameters I get a new pred Which once again does not change with different inputs. Unless I once again re-initiate the model using model=network(). What am I doing wrong?
Edit: To give more info on my problem I'm trying to create a yolo like network from scratch. And this is the dataset I'm using https://www.kaggle.com/devdgohil/the-oxfordiiit-pet-dataset

Output:
Basically that's what the Batchnorm doing. You use Batchnorm to make training less prone to overfit but don't use batchnorm in eval so that you can get the correct result Same go for Dropout.
Every CNN model with batch normalization and/or dropout does the same. The output of the same input will be different during train and eval
Which is exactly why Pytorch has the model.eval(). To turn these layers off during inference to get the correct output.
Edit
The problem is the activation and Batch Normalization at the output.
Only use something that will make the result similar to the ground truth. Like use sigmoid when you want output to be in range of 0-1 or tanh for -1 to 1 or softmax for probability across the axis.
Imagine relu function (which is basically the simpler version of swish and softplus). It will turn everything below 0 to 0. And chances are you need some output to be below 0 so your model won't converge at all.


Instruction: Indexing elements from a batch tensor in PyTorch

Input:
Say, I have a batch of images in PyTorch. For each image, I also have a pixel location say (x, y). The pixel value can be read using img[x, y] for one image. I am trying to read pixel value for each image in the batch. Please see below the code snippet:
import torch

# create tensors to represent random images in torch format
img_1 = torch.rand(1, 200, 300)
img_2 = torch.rand(1, 200, 300)
img_3 = torch.rand(1, 200, 300)
img_4 = torch.rand(1, 200, 300)

# for each image, x-y value are know, so creating a tuple
img1_xy = (0, 10, 70)
img2_xy = (0, 40, 20)
img3_xy = (0, 30, 50)
img4_xy = (0, 80, 60)

# this is what I am doing right now
imgs = [img_1, img_2, img_3, img_4]
imgs_xy = [img1_xy, img2_xy, img3_xy, img4_xy]
x = [img[xy] for img, xy in zip(imgs, imgs_xy)]
x = torch.as_tensor(x)

My Concerns and Questions

In each image, the pixel location i.e., (x, y) is know. However, I have to create a tuple with one more element, i.e., 0 to make sure that tuple matches the shape of image. Any elegant way?
Instead of using tuple, can't we use tensor and then get the pixel values?
All images can be concatenated to make a batch as img_batch = torch.cat((img_1, img_2, img_3, img_4)). But what about tuple?

Output:
You can concatanate the images to form a  (4, 200, 300) shaped stacked tensor. Then, we can index into this with the known (x, y) pairs for each image as follows: we need [0, x1, y1] for first image, [1, x2, y2] for second image, [2, x3, y3] for third image and so on. These can be achieved with "fancy indexing":
# stacking as you did
>>> stacked_imgs = torch.cat(imgs)
>>> stacked_imgs.shape
(4, 200, 300)

# no need for 0s in front
>>> imgs_xy = [(10, 70), (40, 20), (30, 50), (80, 60)]

# need xs together and ys together: take transpose of `imgs_xy`
>>> inds_x, inds_y = torch.tensor(imgs_xy).T

>>> inds_x
tensor([10, 40, 30, 80])

>>> inds_y
tensor([70, 20, 50, 60])

# now we index into the batch
>>> num_imgs = len(imgs)
>>> result = stacked_imgs[range(num_imgs), inds_x, inds_y]
>>> result
tensor([0.5359, 0.4863, 0.6942, 0.6071])

We can check the result:
>>> torch.tensor([img[0, x, y] for img, (x, y) in zip(imgs, imgs_xy)])

tensor([0.5359, 0.4863, 0.6942, 0.6071])

To answer your questions:
1: Since we stacked the images, that issue is mitigated and we use range(4) to index into each individual image instead.
2: Yes, we indeed turn x, y positions into tensors.
3: We directly index with them after they are separated into tensors.


Instruction: Is there a resnet50_v2 pretrained on imageNet for Pytorch?

Input:
I'm new to ML, and as the title states, I'm wondering if there's a pretrained ResNet50_v2 with ImageNet on PyTorch?
I looked at https://pytorch.org/vision/stable/models.html
and found torchvision.models.resnet50(..) but I don't think that's the same as ResNet50_v2?
Thank you in advance!

Output:
The official position is that it will not be added, as you can see here:

fmassa: "We've added ResNeXt to torchvision, but ResNetv2 didn't really caught up with the community so we won't be adding it"


Instruction: How to change parts of parameters' device type of a module in pytorch?

Input:
I define a net and two parameters are on CPU, I try to move those two parameters to GPU. However, when I print the device, I find that those two are not moved to GPU.
How to change the model parameter device?
for p in net.parameters():
    if p.device == torch.device('cpu'):
        p = p.to('cuda')

for p in net.parameters():
    if p.device == torch.device('cpu'):
        print(p.device)

Output:
cpu
cpu

Output:
You're dealing with parameters. Unlike a Module, you have to attribute them back to the original variable if you want to replace them. Additionally, you'll want to change the .data of a given parameter, otherwise it won't work because the .to(...) actually generates a copy.
for p in net.parameters():
    if p.device == torch.device('cpu'):
        p.data = p.to('cuda')

Note that if any of these parameters have .grad, they will not be moved to the GPU. Take a look here at how parameters are usually moved. As you'll see, you'll have to do the same for the gradients:
for p in net.parameters():
    if p.device == torch.device('cpu'):
        p.data = p.to('cuda')
        if p.grad is not None:
            p.grad.data = p.grad.to('cuda')


Instruction: pytorch torchvision.datasets.ImageFolder FileNotFoundError: Found no valid file for the classes .ipynb_checkpoints

Input:
Tried to load training data with pytorch torch.datasets.ImageFolder in Colab.
transform = transforms.Compose([transforms.Resize(400),
                                transforms.ToTensor()])
dataset_path = 'ss/'
dataset = datasets.ImageFolder(root=dataset_path, transform=transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=20)

I encountered the following error :
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-27-7abcc1f434b1> in <module>()
      2                                 transforms.ToTensor()])
      3 dataset_path = 'ss/'
----> 4 dataset = datasets.ImageFolder(root=dataset_path, transform=transform)
      5 dataloader = torch.utils.data.DataLoader(dataset, batch_size=20)

3 frames
/usr/local/lib/python3.7/dist-packages/torchvision/datasets/folder.py in make_dataset(directory, class_to_idx, extensions, is_valid_file)
    100         if extensions is not None:
    101             msg += f"Supported extensions are: {', '.join(extensions)}"
--> 102         raise FileNotFoundError(msg)
    103
    104     return instances

FileNotFoundError: Found no valid file for the classes .ipynb_checkpoints. Supported extensions are: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, .webp

My Dataset folder contains a subfolder with many training images in png format, still the ImageFolder can't access them.

Output:
I encountered the same problem when I was using IPython notebook-like tools.
First please check if there is any hidden files under your dataset_path. Use ls -a if you are under a Linux environment.
The case happen to me is I found a hidden file called .ipynb_checkpoints which is located parallelly to image class subfolders. I think that file causes confusion to PyTorch dataset. I made sure it is not useful so I simply deleted it. Then the dataset works fine.
Or if you would like to simply ignore that file, you may also try this.


Instruction: Cant install tensorflow for huggingface transformers library

Input:
Im trying to use huggingface transformers library in my python project. I am a first time python programmer, and I am stuck on this error message, even though tensorflow has been installed on my machine:
>>> from transformers import pipeline

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.

I have discovered that tensorflow does not exist, even though I have installed it via pip. I have tried uninstalling it and reinstalling it, and but when I try to import the package, it just comes back as a ModuleNotFoundError
>>> import tensorflow

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    import tensorflow
  File "C:\Users\######\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tensorflow\__init__.py", line 41, in <module>
    from tensorflow.python.tools import module_util as _module_util
ModuleNotFoundError: No module named 'tensorflow.python

I have tried uninstalling and re-installing using pip and conda. I even tried installing pytorch using the same methods. It always says that the package was succesfully installed, and yet the error persists.
I am using Python 3.9 and my OS is Windows 10. I dont know what I am doing wrong. But I know that a solution will definitely not be to uninstall and reinstall a package.
Pip version (pip -V):
pip 21.1.3 from c:\users\######\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

Python version (python -V):
Python 3.9.5

Python path list
I tried comparing the output of sys.path with the output of pip -V.
The closest path I saw for the pip -V path is down at the bottom, however I did not find the exact directory.
>>> import sys
>>> sys.path

['', 'C:\\windows\\system32', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0\\python39.zip', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0\\DLLs', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0\\lib', 'C:\\Users\\######\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0', 'C:\\Users\\######\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0', 'C:\\Program Files\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0\\lib\\site-packages']

Closest path:
C:\Users\######\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages

Output:
From comments

You have multiple python interpreters installed, that is why
installing stuff does not show in your python interpreter, use pip -V and compare it to the python version that appears in the interpreter. Remove one and use only one then your issue will be
resolved (paraphrased from Dr.Snoopy)


Instruction: RuntimeError: Given groups=1, weight of size [64, 32, 3, 3], expected input[128, 64, 32, 32] to have 32 channels, but got 64 channels instead

Input:
I am trying to experiment with why we have a Vanishing & exploding gradient, and why Resnet is so helpful in avoiding the two problems above. So I decided to train a plain Convolution network with many layers just to know why the model LOSS increases as I train with many layers e.g 20 layers. but I am getting this error at some point, I can figure out what might be the issue, but I know it is from my model Architecture.
images.shape: torch.Size([128, 3, 32, 32])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-80-0ad7109b33c1> in <module>
      1 for images, labels in train_dl:
      2     print('images.shape:', images.shape)
----> 3     out = model(images)
      4     print('out.shape:', out.shape)
      5     print('out[0]:', out[0])

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-78-81b21c16ed79> in forward(self, xb)
     31
     32     def forward(self, xb):
---> 33         return self.network(xb)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    421
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424
    425 class Conv3d(_ConvNd):

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [64, 32, 3, 3], expected input[128, 64, 32, 32] to have 32 channels, but got 64 channels instead

My model Architecture is
class Cifar10CnnModel(ImageClassificationBase):
    def __init__(self):
        super().__init__()
        self.network = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 64 x 16 x 16

            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 128 x 8 x 8

            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 256 x 4 x 4

            nn.Flatten(),
            nn.Linear(256*4*4, 1024),
            nn.ReLU(),
            nn.Linear(1024, 512),
            nn.ReLU(),
            nn.Linear(512, 10))

    def forward(self, xb):
        return self.network(xb)

for images, labels in train_dl:
    print('images.shape:', images.shape)
    out = model(images)
    print('out.shape:', out.shape)
    print('out[0]:', out[0])
    break

Output:
I can see by the model, it looks like you made a typo on the 4th conv block in your sequential.
You have
nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),

However, you already convert the image to 64 channels and you then pass it in to the next conv block as an image with 32 channels which results in the error you have above.
Fix this to:
self.network = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),

            # Change this from 32 to now 64 like I did here.
            nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2), # output: 64 x 16 x 16

Sarthak Jain


Instruction: Undoing tensor dstack and restack column-wise

Input:
I have two, tensors, a and b:
import torch

a = torch.tensor(([1,2],
                  [3,4],
                  [5,6],
                  [7,8]))

b = torch.tensor(([0,0],
                  [1,1],
                  [2,2],
                  [3,3]))

Which I can stack both horizontally or depth-wise.
d = torch.dstack([a, b])
h = torch.hstack([a, b])

Now, is there any PyTorch function, preferably in one line, that I can apply to d to get h? It sounds like I want to undo the depth-wise stacking, and re-stack them column-wise. I've tried reshaping, and flattening, but neither work, as they both disrupt the ordering of the values.

Output:
in your case use torch.unbind
import torch

a = torch.tensor(([1,2],
                  [3,4],
                  [5,6],
                  [7,8]))

b = torch.tensor(([0,0],
                  [1,1],
                  [2,2],
                  [3,3]))

d = torch.dstack([a, b])
h = torch.hstack(torch.unbind(d,2)) # get h from d


Instruction: Error calculating area "IndexError: too many indices for tensor of dimension 1"

Input:
I'm repurposing some code from here to perform object detection:
# Create boxes list
boxes = [
    [annotation['xmin'], annotation['ymin'], annotation['xmax'], annotation['ymax']]
    for annotation in image_annotations
]

...

area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

During training, I was hitting this error:
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\src\screenshot_dataset.py", line 94, in __getitem__
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
IndexError: too many indices for tensor of dimension 1

Output:
Following a suggestion here, I discovered some of my images did not contain any bounding boxes and was causing this error:
# Create boxes list
boxes = [
    [annotation['xmin'], annotation['ymin'], annotation['xmax'], annotation['ymax']]
    for annotation in image_annotations
]

if len(boxes) == 0:
    print ("Help!")


Instruction: How can I concatenate the 4 corners of the image quickly when loading image in deep learning?

Input:
What is the most effective way to concatenate 4 corner, shown in this photo ?
(conducting in getitem())
left_img = Image.open('image.jpg')
...
output = right_img

Output:
This is how I would do it.
Firstly I would convert the image to a Tensor Image temporarily
from torchvision import transforms
tensor_image = transforms.ToTensor()(image)

Now assuming you have a 3 channel image (although similiar principles apply to any matrices of any number of channels including 1 channel gray scale images).
You can find the Red channel with tensor_image[0] the Green channel with tensor_image[1] and the the Blue channel with tensor_image[2]
You can make a for loop iterating through each channel like
for i in tensor_image.size(0):
  curr_channel = tensor_image[i]

Now inside that for loop with each channel you can extract the

First corner pixel with float(curr_channel[0][0])
Last top corner pixel with float(curr_channel[0][-1])
Bottom first pixel with float(curr_channel[-1][0])
Bottom and last pixel with float(curr_channel[-1][-1])

Make sure to convert all the pixel values to float or double values before this next appending step
Now you have four values that correspond to the corner pixels of each channel
Then you can make a list called new_image = []
You can then append the above mentioned pixel values using
new_image.append([[curr_channel[0][0], curr_channel[0][-1]], [curr_channel[-1][0], curr_channel[-1][-1]]])
Now after iterating through every channel you should have a big list that contains three (or tensor_image.size(0)) number of lists of lists.
Next step is to convert this list of lists of lists to a torch.tensor by running
new_image = torch.tensor(new_image)

To make sure everything is right new_image.size() should return torch.Size([3, 2, 2])
If that is the case you now have your wanted image but it is tensor format.
The way to convert it back to PIL is to run
final_pil_image = transforms.ToPILImage()(new_image)

If everything went good, you should have a pil image that fulfills your task. The only code it uses is clever indexing and one for loop.
There is a possibility however if you look more than I can, then you can avoid using a for loop and perform operations on all the channels without the loop.
Sarthak Jain


Instruction: Input dimension of Pytorch CNN model

Input:
I have input data for my 2D CNN model, say; X_train with shape (torch.Size([716, 50, 50])
my model is:
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=4,stride=1,padding = 1)
        self.mp1 = nn.MaxPool2d(kernel_size=4,stride=2)
        self.conv2 = nn.Conv2d(32,64, kernel_size=4,stride =1)
        self.mp2 = nn.MaxPool2d(kernel_size=4,stride=2)
        self.fc1= nn.Linear(2304,256)
        self.dp1 = nn.Dropout(p=0.2)
        self.fc2 = nn.Linear(256,10)

    def forward(self, x):
        in_size = x.size(0)
        x = F.relu(self.mp1(self.conv1(x)))
        x = F.relu(self.mp2(self.conv2(x)))
        x = x.view(in_size,-1)
        x = F.relu(self.fc1(x))
        x = self.dp1(x)
        x = self.fc2(x)

        return F.log_softmax(x, dim=1)

but when I run the model, I always get this error:
---> x = F.relu(self.mp1(self.conv1(x)))
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 1, 4, 4], but got 3-dimensional input of size [64, 50, 50] instead

I understand my input for the model is of size 64 (batch size), 50*50 (size of each input, in this case is signal picture).
But I don't understand why it still requires 4-dimensional input where I had set my in_channels for nn.Conv2d to be 1.
How to solve this input dimension problem or to change the dimension requirement of model input?

Output:
Whether in_channels is 1 or 42 does not matter: it is still an added dimension. It is useful to read the documentation in this respect.
In- and output are of the form N, C, H, W

N: batch size
C: channels
H: height in pixels
W: width in pixels

So you need to add the dimension in your case:
# Add a dimension at index 1
x = x.unsqueeze(1)


Instruction: PEGASUS From pytorch to tensorflow

Input:
I have fine-tuned PEGASUS model for abstractive  summarization using this script which uses huggingface.
The output model is in pytorch.
Is there a way to transorm it into tensorflow model so I can use it in a javascript backend?

Output:
There are several ways in which you can potentially achieve a conversion, some of which might not even need Tensorflow at all.
Firstly, the way that does what you intend to do: PEGASUS seems to be completely based on the BartForConditionalGeneration model, according to the transformer implementation notes. This is important, because there exists a script to convert PyTorch checkpoints to TF2 checkpoints. While this script does not explicitly allow you to convert a PEGASUS model, it does have options available for BART. Running it with the respective parameters should give you the desired output.
Alternatively, you can potentially achieve the same by exporting the model into the ONNX format, which also has JS deployment options. Specific details for how to convert a Huggingface model to ONNX can be found here.