Instruction: Expected input batch_size (18) to match target batch_size (6)

Input:
Is RNN for image classification available only for gray image?
The following program works for gray image classification.
If RGB images are used, I have this error:

Expected input batch_size (18) to match target batch_size (6)

at this line loss = criterion(outputs, labels).
My data loading for train, valid and test are as follows.
input_size  = 300
inputH = 300
inputW = 300

#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_resize_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])

train_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.RandomHorizontalFlip(),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
valid_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
test_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                        tt.ToTensor(),
                        tt.Normalize(*stats)])

#Create dataset
train_ds = ImageFolder('./data/train', train_tfms)
valid_ds = ImageFolder('./data/valid', valid_tfms)
test_ds = ImageFolder('./data/test', test_tfms)

from torch.utils.data.dataloader import DataLoader
batch_size = 6

#Training data loader
train_dl = DataLoader(train_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Validation data loader
valid_dl = DataLoader(valid_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Test data loader
test_dl = DataLoader(test_ds, 1, shuffle = False, num_workers = 1, pin_memory=True)

My model is as follows.
num_steps = 300
hidden_size = 256 #size of hidden layers
num_classes = 5
num_epochs = 20
learning_rate = 0.001
# Fully connected neural network with one hidden layer
num_layers = 2 # 2 RNN layers are stacked
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True, dropout=0.2)#batch must have first dimension
        #our inpyt needs to have shape
        #x -> (batch_size, seq, input_size)
        self.fc = nn.Linear(hidden_size, num_classes)#this fc is after RNN. So needs the last hidden size of RNN

    def forward(self, x):
        #according to ducumentation of RNN in pytorch
        #rnn needs input, h_0 for inputs at RNN (h_0 is initial hidden state)

        #the following one is initial hidden layer
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)#first one is number of layers and second one is batch size
        #output has two outputs. The first tensor contains the output features of the hidden last layer for all time steps
        #the second one is hidden state f
        out, _ = self.rnn(x, h0)
        #output has batch_size, num_steps, hidden size
        #we need to decode hidden state only the last time step
        #out (N, 30, 128)
        #Since we need only the last time step
        #Out (N, 128)
        out = out[:, -1, :] #-1 for last time step, take all for N and 128
        out = self.fc(out)
        return out

stacked_rnn_model = RNN(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()#cross entropy has softmax at output
#optimizer = torch.optim.Adam(stacked_rnn_model.parameters(), lr=learning_rate) #optimizer used gradient optimization using Adam
optimizer = torch.optim.SGD(stacked_rnn_model.parameters(), lr=learning_rate)
# Train the model
n_total_steps = len(train_dl)
    for epoch in range(num_epochs):
        t_losses=[]
        for i, (images, labels) in enumerate(train_dl):
            # origin shape: [6, 3, 300, 300]
            # resized: [6, 300, 300]
            images = images.reshape(-1, num_steps, input_size).to(device)
            print('images shape')
            print(images.shape)
            labels = labels.to(device)

            # Forward pass
            outputs = stacked_rnn_model(images)
            print('outputs shape')
            print(outputs.shape)
            loss = criterion(outputs, labels)
            t_losses.append(loss)
            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

Printing images and outputs shapes are
images shape
torch.Size([18, 300, 300])
outputs shape
torch.Size([18, 5])

Where is the mistake?

Output:
Tl;dr: You are flattening the first two axes, namely batch and channels.

I am not sure you are taking the right approach but I will write about that layer.
In any case, let's look at the issue you are facing. You have a data loader that produces (6, 3, 300, 300), i.e. batches of 6 three-channel 300x300 images. By the look of it you are looking to reshape each batch element (3, 300, 300) into (step_size=300, -1).
However instead of that you are affecting the first axis - which you shouldn't - with images.reshape(-1, num_steps, input_size). This will have the desired effect when working with a single-channel images since dim=1 wouldn't be the "channel axis". In your case your have 3 channels, therefore, the resulting shape is: (6*3*300*300//300//300, 300, 300) which is (18, 300, 300) since num_steps=300 and input_size=300. As a result you are left with 18 batch elements instead of 6.
Instead what you want is to reshape with (batch_size, num_steps, -1). Leaving the last axis (a.k.a. seq_length) of variable size. This will result in a shape (6, 300, 900).

Here is a corrected and reduced snippet:
batch_size = 6
channels = 3
inputH, inputW = 300, 300
train_ds = TensorDataset(torch.rand(100, 3, inputH, inputW), torch.rand(100, 5))
train_dl = DataLoader(train_ds, batch_size)

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        # (batch_size, seq, input_size)
        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)
        # (batch_size, hidden_size)
        self.fc = nn.Linear(hidden_size, num_classes)
        # (batch_size, num_classes)

    def forward(self, x):
        out, _ = self.rnn(x)
        out = out[:, -1, :]
        out = self.fc(out)
        return out

num_steps = 300
input_size = inputH*inputW*channels//num_steps
hidden_size = 256
num_classes = 5
num_layers = 2

rnn = RNN(input_size, hidden_size, num_layers, num_classes)
for x, y in train_dl:
    print(x.shape, y.shape)
    images = images.reshape(batch_size, num_steps, -1)
    print(images.shape)
    outputs = rnn(images)
    print(outputs.shape)
    break

As I said in the beginning I am a bit wary about this approach because you are essentially feeding your RNN a RGB 300x300 image in the form of a sequence of 300 flattened vectors... I can't say if that makes sense and terms of training and if the model will be able to learn from that. I could be wrong!


Instruction: How can I run both TensorFlow and Torch on Mac M1 MacBook Pro?

Input:
I encounter some problems on my MacBook Pro M1. I thought it would be easier to start coding on it, apparently it's not a ML beast yet...
I need to use both PyTorch and TensorFlow on Python. I have installed TensorFlow 2.0 for Mac OS.
The problem is: TensorFlow won't work when you use a x86_64 terminal. (So it doesn't work with PyCharm). However, I can import TensorFlow 2.0 from an arm terminal.
Paradoxically, PyTorch won't install on a arm terminal, only on a x86_64 terminal. So, on the same Python terminal, I'm not able to import both torch and TensorFlow 2.0.
Since HuggingFace transformers is crucial for me, and transformers needs both TensorFlow 2.0 and PyTorch, I need to go back on my old computer to code. I'm very disappointed!
Anyone successfully imported both PyTorch and TensorFlow on a Mac M1 device?
And does anyone know if there is a way to force PyCharm to use an arm terminal, so I can use TensorFlow 2.0 on PyCharm on my M1 MPB?
Thank you!

Output:
After some research, I found this answer: https://github.com/pytorch/pytorch/issues/48145 .
So if someone tries to run both Tensorflow and Torch on Mac M1 with PyCharm Apple Silicon Version, here's how to proceed:

Create a new virtual env with Tensorflow Mac OS
From terminal (not from PyCharm, I got an error), with sudo rights, install the whl file for torch, from the GitHub issue: https://github.com/wizyoung/AppleSiliconSelfBuilds/blob/main/builds/torch-1.8.0a0-cp39-cp39-macosx_11_0_arm64.whl

Now you can open a PyCharm project with your freshly created virtual environment, and you'll be able to import both Tensorflow and Torch. However, a lot of librairies will be tricky to install like PyTorch...


Instruction: What does does if self.transforms mean?

Input:
What does
if self.transforms:
            data = self.transforms(data)

do? I don't understand the logic behind this line - what is the condition the line is using?
I'm reading an article on creating a custom dataset with pytorch based on the below implementation:
#custom dataset
class MNISTDataset(Dataset):
    def __init__(self, images, labels=None, transforms=None):
        self.X = images
        self.y = labels
        self.transforms = transforms

    def __len__(self):
        return (len(self.X))

    def __getitem__(self, i):
        data = self.X.iloc[i, :]
        data = np.asarray(data).astype(np.uint8).reshape(28, 28, 1)

        if self.transforms:
            data = self.transforms(data)

        if self.y is not None:
            return (data, self.y[i])
        else:
            return data
train_data = MNISTDataset(train_images, train_labels, transform)
test_data = MNISTDataset(test_images, test_labels, transform)
# dataloaders
trainloader = DataLoader(train_data, batch_size=128, shuffle=True)
testloader = DataLoader(test_data, batch_size=128, shuffle=True)

thank you! i'm basically trying to understand why it works & how it applies transforms to the data.

Output:
The dataset MNISTDataset can optionnaly be initialized with a transform function. If such transform function is given it be saved in self.transforms else it will keep its default values None. When calling a new item with __getitem__, it first checks if the transform is a truthy value, in this case it checks if self.transforms can be coerced to True which is the case for a callable object. Otherwise it means self.transforms hasn't been provided in the first place and no transform function is applied on data.

Here's a general example, out of a torch/torchvision context:
def do(x, callback=None):
    if callback: # will be True if callback is a function/lambda
        return callback(x)
    return x

do(2) # returns 2
do(2, callback=lambda x: 2*x) # returns 4


Instruction: How to use an embedding layer as a linear layer in PyTorch?

Input:
I'm currently working on a personal reimplementation of the Transformer paper and had a question.
On page 5 in section "3.4 Embeddings and Softmax," it states:

In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation.

I've currently implemented my model to use just one embedding layer for both source and target tensors, but I'm wondering if there would be a way that I could use the weights of the embedding layer as a linear layer. What I've currently done is something like:
output = previous_layer(previous_input)
final_output = torch.matmul(output, embedding_layer.embedding.weight.transpose(1, 0))

I've transposed the weight matrix before matrix multiplication because it's of shape (vocab_size, embedding_dim) and the shape of output is (batch_size, seq_len, embedding_dim). Is this the proper way to use an embedding layer as a linear layer? If not, I'd like some tips on what I should be doing.
Thanks.
Edit
Specific line for weight sharing from link in answer.

Output:
You could define a nn.Linear layer and replace its weights by copying the weights from the nn.Embedding:
trg_emb = nn.Embedding(trg_enc_dim, embedding_dim)
src_emb = nn.Embedding(src_enc_dim, embedding_dim)

trg_projection = nn.Linear(embedding_dim, trg_enc_dim, bias=False)
trg_projection.weight = trg_emb.weight # copy to nn.Linear

You can also copy to the source embedding so your two embedding layers share the same weights as well.
src_emb.weight = trg_emb.weight

This would mean the source embedding would end up having the same size as the target embedding i.e. trg_enc_dim x embedding_dim.
Here's a possible PyTorch implementation for inspiration.


Instruction: PyTorch equivalent to tf.nn.softmax_cross_entropy_with_logits and tf.nn.sigmoid_cross_entropy_with_logits

Input:
I found the post here. Here, we try to find an equivalence of tf.nn.softmax_cross_entropy_with_logits in PyTorch. The answer is still confusing to me.
Here is the Tensorflow 2 code
import tensorflow as tf
import numpy as np

# here we assume 2 batch size with 5 classes

preds = np.array([[.4, 0, 0, 0.6, 0], [.8, 0, 0, 0.2, 0]])
labels = np.array([[0, 0, 0, 1.0, 0], [1.0, 0, 0, 0, 0]])

tf_preds = tf.convert_to_tensor(preds, dtype=tf.float32)
tf_labels = tf.convert_to_tensor(labels, dtype=tf.float32)

loss = tf.nn.softmax_cross_entropy_with_logits(logits=tf_preds, labels=tf_labels)

It give me the loss as
<tf.Tensor: shape=(2,), dtype=float32, numpy=array([1.2427604, 1.0636061], dtype=float32)>

Here is the PyTorch code
import torch
import numpy as np

preds = np.array([[.4, 0, 0, 0.6, 0], [.8, 0, 0, 0.2, 0]])
labels = np.array([[0, 0, 0, 1.0, 0], [1.0, 0, 0, 0, 0]])

torch_preds = torch.tensor(preds).float()
torch_labels = torch.tensor(labels).float()

loss = torch.nn.functional.cross_entropy(torch_preds, torch_labels)

However, it raises:

RuntimeError: 1D target tensor expected, multi-target not supported

It seems that the problem is still unsolved. How to implement tf.nn.softmax_cross_entropy_with_logits in PyTorch?
What about tf.nn.sigmoid_cross_entropy_with_logits?

Output:
- tf.nn.softmax_cross_entropy_with_logits
Edit: This is actually not equivalent to F.cross_entropy. The latter can only handle the single-class classification setting. Not the more general case of multi-class classification, whereby the label can be comprised of multiple classes. Indeed, F.cross_entropy takes a unique class id as target (per instance), not a probability distribution over classes as tf.nn.softmax_cross_entropy_with_logits can expect to receive.
>>> logits = torch.tensor([[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]])
>>> labels = torch.tensor([[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]])

In order to get the desired result apply a log-softmax to your logits then take the negative log-likelihood:
>>> -torch.sum(F.log_softmax(logits, dim=1) * labels, dim=1)
tensor([0.1698, 0.8247])

- tf.nn.sigmoid_cross_entropy_with_logits
For this one you can apply F.binary_cross_entropy_with_logits.
>>> F.binary_cross_entropy_with_logits(logits, labels, reduction='none')
tensor([[0.0181, 2.1269, 1.3133],
        [0.6931, 1.0067, 1.1133]])

It is equivalent to applying a sigmoid then the negative log-likelihood, considering each class as a binary classification task:
>>> labels*-torch.log(torch.sigmoid(logits)) + (1-labels)*-torch.log(1-torch.sigmoid(logits))
tensor([[0.0181, 2.1269, 1.3133],
        [0.6931, 1.0067, 1.1133]])

having imported torch.nn.functional as F.


Instruction: Changing batch,height,width,alpha to batch,alpha,height,width for pytorch

Input:
I have batch of images like
torch.Size([10, 512, 512, 3])
I can loop to the images and can see 10 images. But to feed this batch to pytorch i have to convert it to
torch.Size([10, 3, 512, 512])
I tried lot of ways but unable to get the solution for this
How can we do that ?

Output:
Use permute:
import torch
x = torch.rand(10, 512, 512, 3)

y = x.permute(0, 3, 1, 2)

x.shape: torch.Size([10, 512, 512, 3])
y.shape: torch.Size([10, 3, 512, 512])


Instruction: Tensor manipulation - creating a positional tensor from a given tensor

Input:
I have a input tensor which has zero padding at the start and then a sequence of values. So something like:
x = torch.tensor([[0, 2, 8, 12],
                  [0, 0, 6, 3]])

What I need is another tensor having same shape and retaining 0's for the padding and an increasing sequence for the rest of the numbers. So my output tensor should be:
y = ([[0, 1, 2, 3],
      [0, 0, 1, 2]])

I tried something like:
MAX_SEQ=4
seq_start = np.nonzero(x)
start = seq_start[0][0]
pos_id = torch.cat((torch.from_numpy(np.zeros(start, dtype=int)).to(device), torch.arange(1, MAX_SEQ-start+1).to(device)), 0)
print(pos_id)

This works if the tensor is 1 dimensional but needs additional logic to handle it for 2-D shape. This can be done as np.nonzeros returns a tuple and we could probably loop thru' those tuples updating a counter or something. However I am sure there must be a simple tensor operation which should do this in 1-2 lines of code and also perhaps more effectively.
Help appreciated

Output:
A possible solution in three small steps:

Find the index of the first non zero element for each row. This can be done with a trick explained here (adapted here for non-binary tensors).
> idx = torch.arange(x.shape[1], 0, -1)
tensor([4, 3, 2, 1])

> xbin = torch.where(x == 0, 0, 1)
tensor([[0, 1, 1, 1],
        [0, 0, 1, 1]])

> xbin*idx
tensor([[0, 3, 2, 1],
        [0, 0, 2, 1]])

> indices = torch.argmax(xbin*idx, dim=1, keepdim=True)
tensor([[1],
        [2]])

Create an arangement for the resulting tensor (without padding). This can be done by applying torch.repeat and torch.view on a torch.arange call:
> rows, cols = x.shape
> seq = torch.arange(1, cols+1).repeat(1, rows).view(-1, cols)
tensor([[1, 2, 3, 4],
        [1, 2, 3, 4]])

Lastly - here's the trick! - we substract the index of the first non-zero element with the arangement, for each row. Then we mask the padding values and replace them with zeros:
> pos_id = seq - indices
tensor([[ 0,  1,  2,  3],
        [-1,  0,  1,  2]])

> mask = indices > seq - 1
tensor([[ True, False, False, False],
        [ True,  True, False, False]])

> pos_id[mask] = 0
tensor([[0, 1, 2, 3],
        [0, 0, 1, 2]])


Instruction: Convert 5D tensor to 4D tensor in PyTorch

Input:
In PyTorch I have a 5D tensor X of dimensions B x 9 x C x H x W. I want to convert it into a 4D tensor Y with dimensions B x 9C x H x W such that concatenation happens channel wise.
To illustrate let,
a = X[1,0,:,:,:]
b = X[1,1,:,:,:]
c = X[1,2,:,:,:]
...
i = X[1,8,:,:,:]

Then in the tensor Y, a to i should be channel wise concatenated.

Output:
Try:
import torch
x = torch.rand(3, 4, 3, 2, 6)
print(x.shape)
y=x.flatten(start_dim=1, end_dim=2)
print(y.shape)

torch.Size([3, 4, 3, 2, 6])
torch.Size([3, 12, 2, 6])


Instruction: transpose on PyTorch : IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

Input:
I want to transpose the data I have using transpose, but I am encountering such an error. My data and related process uploaded to github.
https://github.com/nurkbts/error/blob/main/error.ipynb

Output:
When using torch.bmm (batch matrix multiplication), both tensors must have three dimensions (first one being the batch). Please read the documentation for the details.
Since you were trying to use bmm, you should just use the @ operator (equivalent to applying torch.matmul). Also, don't forget to transpose. This will give you a shape (64, 64).
_scores = queries@keys.T / np.sqrt(64)


Instruction: ValueError: Expected target size (128, 44), got torch.Size([128, 100]), LSTM Pytorch

Input:
I want to build a model, that predicts next character based on the previous characters.
I have spliced text into sequences of integers with length = 100(using dataset and dataloader).
Dimensions of my input and target variables are:
inputs dimension: (batch_size,sequence length). In my case (128,100)
targets dimension: (batch_size,sequence length). In my case (128,100)

After forward pass I get dimension of my predictions: (batch_size, sequence_length, vocabulary_size) which is in my case (128,100,44)
but when I calculate my loss using nn.CrossEntropyLoss() function:
batch_size = 128
sequence_length   = 100
number_of_classes = 44
# creates random tensor of your output shape
output = torch.rand(batch_size,sequence_length, number_of_classes)
# creates tensor with random targets
target = torch.randint(number_of_classes, (batch_size,sequence_length)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

I get an error:
ValueError: Expected target size (128, 44), got torch.Size([128, 100])

Question is: how should I handle calculation of the loss function for many-to-many LSTM prediction? Especially sequence dimension? According to nn.CrossEntropyLoss Dimension must be(N,C,d1,d2...dN), where N is batch_size,C - number of classes. But what is D? Is it related to sequence length?

Output:
As a general comment, let me just say that you have asked many different questions, which makes it difficult for someone to answer. I suggest asking just one question per StackOverflow post, even if that means making several posts. I will answer just the main question that I think you are asking: "why is my code crashing and how to fix it?" and hopefully that will clear up your other questions.
Per your code, the output of your model has dimensions (128, 100, 44) = (N, D, C). Here N is the minibatch size, C is the number of classes, and D is the dimensionality of your input. The cross entropy loss you are using expects the output to have dimension (N, C, D) and the target to have dimension (N, D). To clear up the documentation that says (N, C, D1, D2, ..., Dk), remember that your input can be an arbitrary tensor of any dimensionality. In your case inputs have length 100, but nothing is to stop someone from making a model with, say, a 100x100 image as input. (In that case the loss would expect output to have dimension (N, C, 100, 100).) But in your case, your input is one dimensional, so you have just a single D=100 for the length of your input.
Now we see the error, outputs should be (N, C, D), but yours is (N, D, C). Your targets have the correct dimensions of (N, D). You have two paths the fix the issue. First is to change the structure of your network so that its output is (N, C, D), this may or may not be easy or what you want in the context of your model. The second option is to transpose your axes at the time of loss computation using torch.transpose https://pytorch.org/docs/stable/generated/torch.transpose.html
batch_size = 128
sequence_length   = 100
number_of_classes = 44
# creates random tensor of your output shape (N, D, C)
output = torch.rand(batch_size,sequence_length, number_of_classes)
# transposes dimensionality to (N, C, D)
tansposed_output = torch.transpose(output, 1, 2)
# creates tensor with random targets
target = torch.randint(number_of_classes, (batch_size,sequence_length)).long()

# define loss function and calculate loss
criterion = nn.CrossEntropyLoss()
loss = criterion(transposed_output, target)
print(loss)


Instruction: How to add a new dimension to a PyTorch tensor?

Input:
In NumPy, I would do
a = np.zeros((4, 5, 6))
a = a[:, :, np.newaxis, :]
assert a.shape == (4, 5, 1, 6)

How to do the same in PyTorch?

Output:
a = torch.zeros(4, 5, 6)
a = a[:, :, None, :]
assert a.shape == (4, 5, 1, 6)


Instruction: When I import torchvision, I get an error that the cuda version of pytorch and torchvision are different

Input:
I am getting the following error when importing torchvision.
Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=11.0 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.

How can I change the cuda version of pytorch to 10.1?
'conda install pytorch torchvision cudatoolkit=10.1-c pytorch' from anaconda prompt
I get the same error even if I do it.
I am using windows10, and I am using python version 3.7 in the virtual environment of jupyter notebook.

Output:
Uninstalling and installing it back might work
!conda uninstall pytorch torchvision

With:
!conda install pytorch torchvision cudatoolkit=10.1 -c pytorch


Instruction: LSTM for predicting characters: cell state and hidden state in the training loop

Input:
My goal is to build a model that predicts next character.
I have built a model and here is my training loop:
model = Model(input_size = 30,hidden_size = 256,output_size = len(dataset.vocab))

EPOCH = 10
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
init_states = None

for epoch in range(EPOCH):
    loss_overall = 0.0

    for i, (inputs,targets) in enumerate(dataloader):
        optimizer.zero_grad()

        pred = model.forward(inputs)
        loss = criterion(pred, targets)
        loss.backward()
        optimizer.step()

As you can see I return only predictions of the model, but not cell_state and hidden_state.
So alternative is : pred,cell_state,hidden_state = model.forward(inputs)
My question is: should I do it for the prediction of characters task? Why/why not?
And in general: when should I return my hidden and cell state?

Output:
To understand hidden states, here's a excellent diagram by @nnnmmm from this other StackOverflow post.

The hidden states are (h_n, c_n) i.e. the hidden states at the last timestep. Notice how you can't access the previous states for timesteps < t and all hidden layers. Retrieving those final hidden states would be useful if you need to access hidden states for a bigger RNN comprised of multiple hidden layers. However, usually you would just use a single nn.LSTM module and set its num_layers to the desired value.
You don't need to use hidden states. If you want to read more about this thread from the PyTorch forum.

Back to your other question, let's take this model as an example
rnn = nn.LSTM(input_size=10, hidden_size=256, num_layers=2, batch_first=True)

This means an input sequence has seq_length elements of size input_size. Considering the batch on the first dimension, its shape turns out to be (batch, seq_len, input_size).
out, (h, c) = rnn(x)

If you are looking to build a character prediction model I see two options.

You could evaluate a loss at every timestep. Consider an input sequence x and it's target y and the RNN output out. This means for every timestep t you will compute loss(out[t], y[t])`. And the total loss on this input sequence would be averaged over all timesteps.

Else just consider the prediction on the last timestep and compute the loss: loss(out[-1], y) where y is the target which only contains the seq_length+1-th character of the sequence.

If you're using nn.CrossEntropyLoss, both approaches will only require a single function call, as explained in your last thread.


Instruction: How to conda install CUDA enabled PyTorch in a Docker container?

Input:
I am trying to build a Docker container on a server within which a conda environment is built. All the other requirements are satisfied except for CUDA enabled PyTorch (I can get PyTorch working without CUDA however, no problem). How do I make sure PyTorch is using CUDA?
This is the Dockerfile :
# Use nvidia/cuda image
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

# set bash as current shell
RUN chsh -s /bin/bash

# install anaconda
RUN apt-get update
RUN apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
        apt-get clean
RUN wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh -O ~/anaconda.sh && \
        /bin/bash ~/anaconda.sh -b -p /opt/conda && \
        rm ~/anaconda.sh && \
        ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
        echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
        find /opt/conda/ -follow -type f -name '*.a' -delete && \
        find /opt/conda/ -follow -type f -name '*.js.map' -delete && \
        /opt/conda/bin/conda clean -afy

# set path to conda
ENV PATH /opt/conda/bin:$PATH

# setup conda virtual environment
COPY ./requirements.yaml /tmp/requirements.yaml
RUN conda update conda \
    && conda env create --name camera-seg -f /tmp/requirements.yaml \
    && conda install -y -c conda-forge -n camera-seg flake8

# From the pythonspeed tutorial; Make RUN commands use the new environment
SHELL ["conda", "run", "-n", "camera-seg", "/bin/bash", "-c"]

# PyTorch with CUDA 10.2
RUN conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

RUN echo "conda activate camera-seg" > ~/.bashrc
ENV PATH /opt/conda/envs/camera-seg/bin:$PATH

This gives me the following error when I try to build this container ( docker build -t camera-seg . ):
.....

Step 10/12 : RUN conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
 ---> Running in e0dd3e648f7b
ERROR conda.cli.main_run:execute(34): Subprocess for 'conda run ['/bin/bash', '-c', 'conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch']' command failed.  (See above for error)

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

The command 'conda run -n camera-seg /bin/bash -c conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch' returned a non-zero code: 1

This is the requirements.yaml:
name: camera-seg
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.6
  - numpy
  - pillow
  - yaml
  - pyyaml
  - matplotlib
  - jupyter
  - notebook
  - tensorboardx
  - tensorboard
  - protobuf
  - tqdm

When I put pytorch, torchvision and cudatoolkit=10.2 within the requirements.yaml, then PyTorch is successfully installed but it cannot recognize CUDA ( torch.cuda.is_available() returns False ).
I have tried various solutions, for example, this, this and this and some different combinations of them but all to no avail.
Any help is much appreciated. Thanks.

Output:
I got it working after many, many tries. Posting the answer here in case it helps anyone.
Basically, I installed pytorch and torchvision through pip (from within the conda environment) and rest of the dependencies through conda as usual.
This is how the final Dockerfile looks:
# Use nvidia/cuda image
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

# set bash as current shell
RUN chsh -s /bin/bash
SHELL ["/bin/bash", "-c"]

# install anaconda
RUN apt-get update
RUN apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
        apt-get clean
RUN wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh -O ~/anaconda.sh && \
        /bin/bash ~/anaconda.sh -b -p /opt/conda && \
        rm ~/anaconda.sh && \
        ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
        echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
        find /opt/conda/ -follow -type f -name '*.a' -delete && \
        find /opt/conda/ -follow -type f -name '*.js.map' -delete && \
        /opt/conda/bin/conda clean -afy

# set path to conda
ENV PATH /opt/conda/bin:$PATH

# setup conda virtual environment
COPY ./requirements.yaml /tmp/requirements.yaml
RUN conda update conda \
    && conda env create --name camera-seg -f /tmp/requirements.yaml

RUN echo "conda activate camera-seg" >> ~/.bashrc
ENV PATH /opt/conda/envs/camera-seg/bin:$PATH
ENV CONDA_DEFAULT_ENV $camera-seg

And this is how the requirements.yaml looks like:
name: camera-seg
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.6
  - pip
  - numpy
  - pillow
  - yaml
  - pyyaml
  - matplotlib
  - jupyter
  - notebook
  - tensorboardx
  - tensorboard
  - protobuf
  - tqdm
  - pip:
    - torch
    - torchvision

Then I build the container using the command docker build -t camera-seg . and PyTorch is now being able to recognize CUDA.


Instruction: Pytorch batch row-wise application of function

Input:
I would like to figure out a way to apply a function which calculates pairwise distances, let's call it dists(A, B), row-wise for every input element in a batch, meaning:
(100, 16, 3) -- input, 100 is the batch size so 100 instances, 16 is let's say image size, and 3 filters (asking for Conv2D)
(5, 3) -- tensor for which I want to calculate the row-wise distance (assume it's A in dists(A, B) and is fixed)
Now, for every instance I am supposed to get back a matrix of shape (5, 16). Naturally, I could use a for to span the batch and get my final (100,5,16) result. However, I would love to know if there is an easier way to apply my function row-wise, in parallel, using GPU.
Thank you very much for your time.

Output:
Suppose we are using the L1 distance:
import torch
# data and target
a = torch.randn(100, 16, 3)
b = torch.randn(5, 3)

# Reshape the tensors
a = a.unsqueeze(1)
b = b.unsqueeze(0).unsqueeze(2)

print(a.shape, b.shape)

# Compute distance
dist = (a-b).abs().sum(3)
print(dist.shape)


Instruction: Pytorch tutorial code error: "NameError: name 'net' is not defined"

Input:
The code comes from a tutorial for PyTorch. I'm using Google Collabs notebook to run the code.
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

  def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    self.fc1 = nn.Linear(16 * 6 * 6, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

  def forward(self, x):
    x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

  def num_flat_features(self, x):
    size = x.size()[1:]
    num_features = 1
    for s in size:
      num_features *= s
    return num_features

  net = Net()
  print(net)

# The code works up until here. It's the following chunk that returns an error.
params = list(net.parameters())
print(len(params))
print(params[0].size())

The error is:
NameError                                 Traceback (most recent call last)

<ipython-input-17-ad79a1eff4f3> in <module>()
     32   print(net)
     33
---> 34 params = list(net.parameters())
     35 print(len(params))
     36 print(params[0].size())

NameError: name 'net' is not defined

The tutorial says the output should be this instead:
10
torch.Size([6, 1, 3, 3])

It looks to me like net is defined, so I'm unclear why this error is happening. I'm not expert with Python to begin with, so maybe there is something obvious I'm missing.

Output:
Your indentation implies that these lines:
  net = Net()
  print(net)

are part of the Net class because they are in the same scope as the class definition.
Move them outside of that class definition (ie, remove the whitespace indentation for those lines) and it should work.
I'd also suggest moving to indentations with four spaces, not two, to make Python's whitespace easier to scan.


Instruction: Cannot export PyTorch model to ONNX

Input:
I am trying to convert a pre-trained torch model to ONNX, but recive the following error:
RuntimeError: step!=1 is currently not supported

I'm trying this on a pre-trained colorization model: https://github.com/richzhang/colorization
Here is the code I ran in Google Colab:
!git clone https://github.com/richzhang/colorization.git
cd colorization/
import colorizers
model = colorizer_siggraph17 = colorizers.siggraph17(pretrained=True).eval()
input_names = [ "input" ]
output_names = [ "output" ]
dummy_input = torch.randn(1, 1, 256, 256, device='cpu')
torch.onnx.export(model, dummy_input, "test_converted_model.onnx", verbose=True,
                  input_names=input_names, output_names=output_names)

I appreciate any help :)
UPDATE 1: @Proko suggestion solved the ONNX export issue. Now I have a new possibly related problem when I try to convert the ONNX to TensorRT. I get the following error:
[TensorRT] ERROR: Network must have at least one output

Here is the code I used:
import torch
import pycuda.driver as cuda
import pycuda.autoinit
import tensorrt as trt
import onnx

TRT_LOGGER = trt.Logger()

def build_engine(onnx_file_path):
    # initialize TensorRT engine and parse ONNX model
    builder = trt.Builder(TRT_LOGGER)
    builder.max_workspace_size = 1 << 25
    builder.max_batch_size = 1
    if builder.platform_has_fast_fp16:
        builder.fp16_mode = True

    network = builder.create_network()
    parser = trt.OnnxParser(network, TRT_LOGGER)

    # parse ONNX
    with open(onnx_file_path, 'rb') as model:
        print('Beginning ONNX file parsing')
        parser.parse(model.read())
    print('Completed parsing of ONNX file')

    # generate TensorRT engine optimized for the target platform
    print('Building an engine...')
    engine = builder.build_cuda_engine(network)
    context = engine.create_execution_context()
    print("Completed creating Engine")

    return engine, context

ONNX_FILE_PATH = 'siggraph17.onnx' # Exported using the code above
engine,_ = build_engine(ONNX_FILE_PATH)

I tried to force the build_engine function to use the output of the network by:
network.mark_output(network.get_layer(network.num_layers-1).get_output(0))

but it did not work.
I appropriate any help!

Output:
Like I have mentioned in a comment, this is because slicing in  torch.onnx supports only step = 1 but there are 2-step slicing in the model:
self.model2(conv1_2[:,:,::2,::2])

Your only option as for now is to rewrite slicing to be some other ops. You can do it by using range and reshape to obtain proper indices. Consider the following function "step-less-arange" (I hope it is generic enough for anyone with similar problem):
def sla(x, step):
    diff = x % step
    x += (diff > 0)*(step - diff) # add length to be able to reshape properly
    return torch.arange(x).reshape((-1, step))[:, 0]

usage:
>> sla(11, 3)
tensor([0, 3, 6, 9])

Now you can replace every slice like this:
conv2_2 = self.model2(conv1_2[:,:,self.sla(conv1_2.shape[2], 2),:][:,:,:, self.sla(conv1_2.shape[3], 2)])

NOTE:  you should optimize it. Indices are calculated for every call so it might be wise to pre-compute it.
I have tested it with my fork of the repo and I was able to save the model:
https://github.com/prokotg/colorization


Instruction: How to dump confusion matrix using TensorBoard logger in pytorch-lightning?

Input:
The official doc only states
>>> from pytorch_lightning.metrics import ConfusionMatrix
>>> target = torch.tensor([1, 1, 0, 0])
>>> preds = torch.tensor([0, 1, 0, 0])
>>> confmat = ConfusionMatrix(num_classes=2)
>>> confmat(preds, target)

This doesn't show how to use the metric with the framework.
My attempt (methods are not complete and only show relevant parts):
def __init__(...):
    self.val_confusion = pl.metrics.classification.ConfusionMatrix(num_classes=self._config.n_clusters)

def validation_step(self, batch, batch_index):
    ...
    log_probs = self.forward(orig_batch)
    loss = self._criterion(log_probs, label_batch)

    self.val_confusion.update(log_probs, label_batch)
    self.log('validation_confusion_step', self.val_confusion, on_step=True, on_epoch=False)

def validation_step_end(self, outputs):
    return outputs

def validation_epoch_end(self, outs):
    self.log('validation_confusion_epoch', self.val_confusion.compute())

After the 0th epoch, this gives
    Traceback (most recent call last):
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 521, in train
        self.train_loop.run_training_epoch()
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\training_loop.py", line 588, in run_training_epoch
        self.trainer.run_evaluation(test_mode=False)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 613, in run_evaluation
        self.evaluation_loop.log_evaluation_step_metrics(output, batch_idx)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\evaluation_loop.py", line 346, in log_evaluation_step_metrics
        self.__log_result_step_metrics(step_log_metrics, step_pbar_metrics, batch_idx)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\evaluation_loop.py", line 350, in __log_result_step_metrics
        cached_batch_pbar_metrics, cached_batch_log_metrics = cached_results.update_logger_connector()
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 378, in update_logger_connector
        batch_log_metrics = self.get_latest_batch_log_metrics()
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 418, in get_latest_batch_log_metrics
        batch_log_metrics = self.run_batch_from_func_name("get_batch_log_metrics")
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 414, in run_batch_from_func_name
        results = [func(include_forked_originals=False) for func in results]
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 414, in <listcomp>
        results = [func(include_forked_originals=False) for func in results]
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 122, in get_batch_log_metrics
        return self.run_latest_batch_metrics_with_func_name("get_batch_log_metrics",
*args, **kwargs)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 115, in run_latest_batch_metrics_with_func_name
        for dl_idx in range(self.num_dataloaders)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 115, in <listcomp>
        for dl_idx in range(self.num_dataloaders)
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\trainer\connectors\logger_connector\epoch_result_store.py", line 100, in get_latest_from_func_name
        results.update(func(*args, add_dataloader_idx=add_dataloader_idx, **kwargs))
      File "C:\code\EPMD\Kodex\Templates\Testing\venv\lib\site-packages\pytorch_lightning\core\step_result.py", line 298, in get_batch_log_metrics
        result[dl_key] = self[k]._forward_cache.detach()
    AttributeError: 'NoneType' object has no attribute 'detach'

It does pass the sanity validation check before training.
The failure happens on the return in validation_step_end. Makes little sense to me.
The exact same method of using mertics works fine with accuracy.
How to get a correct confusion matrix?

Output:
Updated answer, August 2022

class IntHandler:
    def legend_artist(self, legend, orig_handle, fontsize, handlebox):
        x0, y0 = handlebox.xdescent, handlebox.ydescent
        text = plt.matplotlib.text.Text(x0, y0, str(orig_handle))
        handlebox.add_artist(text)
        return text

class LightningClassifier(LightningModule):
    ...

    def _common_step(self, batch, batch_nb, stage: str):
        assert stage in ("train", "val", "test")
        logger = self._logger
        augmented_image, labels = batch

        outputs, aux_outputs = self(augmented_image)
        loss = self._criterion(outputs, labels)

        return outputs, labels, loss

    def validation_step(self, batch, batch_nb):
        stage = "val"
        outputs, labels, loss = self._common_step(batch, batch_nb, stage=stage)
        self._common_log(loss, stage=stage)

        return {"loss": loss, "outputs": outputs, "labels": labels}

    def validation_epoch_end(self, outs):
        # see https://github.com/Lightning-AI/metrics/blob/ff61c482e5157b43e647565fa0020a4ead6e9d61/docs/source/pages/lightning.rst
        # each forward pass, thus leading to wrong accumulation. In practice do the following:
        tb = self.logger.experiment  # noqa

        outputs = torch.cat([tmp['outputs'] for tmp in outs])
        labels = torch.cat([tmp['labels'] for tmp in outs])

        confusion = torchmetrics.ConfusionMatrix(num_classes=self.n_labels).to(outputs.get_device())
        confusion(outputs, labels)
        computed_confusion = confusion.compute().detach().cpu().numpy().astype(int)

        # confusion matrix
        df_cm = pd.DataFrame(
            computed_confusion,
            index=self._label_ind_by_names.values(),
            columns=self._label_ind_by_names.values(),
        )

        fig, ax = plt.subplots(figsize=(10, 5))
        fig.subplots_adjust(left=0.05, right=.65)
        sn.set(font_scale=1.2)
        sn.heatmap(df_cm, annot=True, annot_kws={"size": 16}, fmt='d', ax=ax)
        ax.legend(
            self._label_ind_by_names.values(),
            self._label_ind_by_names.keys(),
            handler_map={int: IntHandler()},
            loc='upper left',
            bbox_to_anchor=(1.2, 1)
        )
        buf = io.BytesIO()

        plt.savefig(buf, format='jpeg', bbox_inches='tight')
        buf.seek(0)
        im = Image.open(buf)
        im = torchvision.transforms.ToTensor()(im)
        tb.add_image("val_confusion_matrix", im, global_step=self.current_epoch)

output:

Also based on this


Instruction: Error in getting accuracy using test set with PyTorch

Input:
I am trying to find the accuracy of my model that I created with PyTorch, but I get an error. Originally I had a different error, which is fixed, but now I get this error.
I use this to get my test set:
testset = torchvision.datasets.FashionMNIST(MNIST_DIR, train=False,
                                       download=True, transform=torchvision.transforms.Compose([
                                            torchvision.transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1),
                                            torchvision.transforms.ToTensor(), # image to Tensor
                                            torchvision.transforms.Normalize((0.1307,), (0.3081,)) # image, label
                                            ]))
testloader = torch.utils.data.DataLoader(testset, batch_size=100,
                                         shuffle=False)

When I try to access the test set I created, it tries to retrain the model for some reason, then proceeds to error out.
This is the code that gets the accuracy and calls the test set
correct = 0
total = 0
with torch.no_grad():
    print("entered here")
    for (x, y_gt) in testloader:
        x = x.to(device)
        y_gt = y_gt.to(device)
        outputs = teacher_model(x)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))

This is the error I am getting:
Traceback (most recent call last):
  File "[path]/train_teacher_1.py", line 134, in <module>
    outputs = teacher_model(x)
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "[path]\models.py", line 17, in forward
    x = F.relu(self.layer1(x))
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "[path]\anaconda3\lib\site-packages\torch\nn\functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0

Please let me know if you would like the rest of the code for training the model. I left it out because the post got too long.
I am new to PyTorch and any help is appreciated. Thanks in advance.

Output:
I was able to figure out the issue and I needed to check the size of x.
I added this to the for loop to fix it:
x = torch.flatten(x, start_dim=1, end_dim=-1)


Instruction: Confused Regarding PyTorch GRU Docs

Input:
This may be too basic of a question, but what do the docs mean by the input to the GRU needs to be 3 dimensional? The GRU docs for PyTorch state:

input of shape (seq_len, batch, input_size): tensor containing the
features of the input sequence. The input can also be a packed
variable length sequence. See
torch.nn.utils.rnn.pack_padded_sequence() for details.

https://pytorch.org/docs/stable/generated/torch.nn.GRU.html
Let us say I am trying to predict the next # in a sequence and have the following dataset:
n, label
1, 2
2, 3
3, 6
4, 9
...

If I window the data using the prior 2 inputs for consideration when guessing the next, the dataset becomes:
t-2, t-1, t, label
na, na, 1, 2
na, 1, 2, 3
1, 2, 3, 6
2, 3, 4, 10
...

where t-x just represents using an input value from a prior time step.
So, when creating a sequential loader, it should create the following tensor for the line 1,2,3,6:
inputs: tensor([[1,2,3]]) #shape(1,3)
labels: tensor([[6]])     #shape(1,1)

I currently understand the input shape as (# batches, # features per batch)
and the output shape as (# batches, # output features per batch)
My question is, should that input tensor actually look like:
tensor([[[1],[2],[3]]])

Which represents (# batches, #prior inputs to consider, #features per input)
I guess I am better trying to understand why the input to a GRU has 3 dimensions in PyTorch. What does that 3rd dimension fundamentally represent? And if I have a transformed dataset like above, how to properly pass it to the model.
Edit:
So the pattern present is:
1 + 1 = 2
2 + 1 = 3
3 + 2 + 1 = 6
4+ 3 + 2 + 1 = 10

I want it where t-2, t-1, and t represent the features at each time step used to help guess. For example, at every point in time there could be 2 features. The dimensions would be (1 batch size, 3 timesteps, 2 features).
My question is wether the GRU takes a flattened input:
(1 batch size, 3 time steps * 2 features per time step)

or the unflattened input:
(1 batch size, 3 time steps, 2 features per timestep)

I am currently under the impression that it is the 2nd input, but would like to check my understanding.

Output: I figured it out. Essentially, the sequence length of 3 means that the input to the system needs to be: [[[1],[2],[3]], [[2], [3], [4]]] for a batch size of 2, sequence length of 3, and feature input per time step of 1. Essentially each sequence is an input at some time t to consider.


Instruction: Pytorch, standard layer to convert sequential output to binary?

Input:
I am working on a new Pytorch model which takes sequential data as input and I need to output just a single value, which I will then use a binary cross-entropy function to evaluate as a probability of 1 or 0.
To be more concrete, lets say my sequence is 1000 time steps and only 2 dimensions, like a 2-dimensional sine wave, so the data shape would be 1000 x 2.
I have done something like this before using an RNN, which there is a lot of content online. Because of the recurrent structure of the RNN, in order to do this we just look at final output of the RNN after processing the sequence. In this way the the final step output would be 2 dimensions, then we can apply a linear layer to convert 2 -> 1 dimension, et voila, its done.
MY PROBLEM:
What I am attempting to do now is not using a recurrent network, but instead an encoder with attention (Transformer). So the output of the encoder is now still 1000 steps long and whatever my embedded dimension is, likes say 8. So the output of the sequential encoder is shape 1000 x 8. So my issue is that I need to convert this output to a single value, to which I can apply the binary cross-entropy function. I am not finding an obvious way to do this.
IDEAS:
Traditionally with this kind of sequential model, the encoder feeds into a decoder and the decoder can then output a variable length sequence (this is used to language translation problems). My problem is different in that I don't want to output another sequence but just a single value. Maybe I need to convert the decoder in such a way where this works? The decoder usually takes a target value as well as the output from the encoder as input. The output from the decoder then has the same shape as this target value. An idea would be to use the traditional decoder and give a 1 length target, I would then get a 1 length output and I could use a traditional linear layer to convert this to my desired output. However this doesn't seem entirely logical because I really am not interested in outputting a sequence but just 1 value.
Anyways just looking for some more ideas from the community, if you have any. Thanks!

Output:
I think this paper does what you wanted :) (Probably not the first paper that does this but it is the one that I recently read)

Prepend an extra token to your sequence. The token can have a learnable embedding.
After the transformer, discard (or not compute) the output at other positions. We only take the output from the first position, and transform it to the target that you needed.

Image taken from the paper:


Instruction: Pytorch normalize 2D tensor

Input:
For more robustnes of my model I want to normalize my feature tensor.
I tried doing it the way that is to the best of my knowledge standard for pictures:
class Dataset(torch.utils.data.Dataset):
'Characterizes a dataset for PyTorch'
def __init__(self, input_tensor, transform = transforms.Normalize(mean= 0.5, std=0.5)):
    self.labels = input_tensor[:,:,-1]
    self.features = input_tensor[:,:,:-1]
    self.transform = transform

def __len__(self):
    return self.labels_planned.shape[0]

def __getitem__(self, index):

    # Load data and get label
    X = self.features[index]
    y = self.labelslabels[index]
    if self.transform:
        X = self.transform(X)
    return X, y

But receive this error message:
ValueError: Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = torch.Size([8, 25]).

Everywhere I looked people suggest that one should use .view to generate the third dimension in order to comply with the standard shape of pictures, but this seems very odd to me. Is there maybe a cleaner way to do this.
Also where should I best place the normalization? Just for the batch or for the entire train dataset?

Output:
You are asking two different questions, I will try to answer both.

Indeed, you should first reshape to (c, h, w) where c is the channel dimension In most cases, you will need that extra dimension because most 'image' layers are built to receive 3d dimensional tensors - not counting the batch dimension - such as nn.Conv2d, BatchNorm2d, etc... I don't believe there's anyways around it, and doing so would restrict yourself to one-layer image datasets.
You can broadcast to the desired shape with torch.reshape or Tensor.view:
X = X.reshape(1, *X.shape)

Or by adding an additional dimension using torch.unsqueeeze:
X.unsqueeze(0)

About normalization. Batch-normalization and dataset-normalization are two different approaches.
The former is a technique that can achieve improved performance in convolution networks. This kind of operation can be implemented using a nn.BatchNorm2d layer and is done using learnable parameters: a scale factor (~ std) and a bias (~ mean). This type of normalization is applied when the model is called and is applied per-batch.
The latter is a pre-processing technique which allows making different features have the same scale. This normalization can be applied inside the dataset per-element. It requires you measure the mean and standard deviation of your training set.


Instruction: Pytorch Resnet CNN only works when test data contains all classes

Input:
Hoping to get a hand with a weird CNN training issue.
I am training a Resnet classifier to predict 4 classes of image from a ~10k image dataset. The code is pretty simple. Here's the Resnet/CNN setup part:
####################################
########### LOAD RESNET ############
####################################
device = torch.device("cuda" if torch.cuda.is_available()
                                  else "cpu")
model = models.resnet50(pretrained=True)

#
for param in model.parameters():
    param.requires_grad = False

#
model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 #nn.Dropout(0.2),
                                 nn.Linear(512, 10),
                                 nn.LogSoftmax(dim=1))

#
criterion = nn.NLLLoss()

#
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)

# move model to gpu
model.to(device)

And here is the training stage (it batches the data in 500 images and shuffles the test datasets) and some accuracy results after some epochs:
   trainloader, testloader, n_batches = make_trainloader(all_data,
                                                          vals,
                                                          batch_size=500,
                                                          randomize=True)
    ...
    for inputs, labels in trainloader:
        ...
        inputs, labels = inputs.to(device), labels.to(device)

        ...
        # PREDICT;
        outputs = model(inputs)

...

epoch #:  12
Loss: 0.1689 Acc: 0.9400
labels:       tensor([0, 0, 1, 0, 3, 0, 0, 2, 1, 2], device='cuda:0')
predictions:  tensor([0, 0, 1, 0, 3, 0, 0, 2, 1, 2], device='cuda:0')

So the weird thing is that I can't seem to predict well on single images but only on large batches of data with mixed classes. For example, if I provide 500 images from class 1, the prediction is random, but if I provide 500 images mixed from the 4 classes (much like during training), the prediction is great (just like during training).
It seems that I'm confused about how to use the ResNet classifier on single images even though it does seem to learn to predict the individual labels of the input data (see labels and prediction output above). Or that my classifier isn't learning single images, but groups of images, not sure.
Any help or direction is appreciated (I can provide more code, but didn't want to make too long of a message).  Here's the prediction code:
# Predict
randomize = False

# load data from above
inputs = test_data[:2000]
vals_inputs = test_vals[:2000]

print ("test data size: ", vals_inputs.shape)

trainloader, testloader, n_batches = make_trainloader(inputs,
                                                      vals_inputs,
                                                      batch_size=500,
                                                      randomize=randomize)
for inputs, labels in trainloader:
    # load to device
    inputs, labels = inputs.to(device), labels.to(device)

    # PREDICT;
    outputs = model(inputs)
    _, preds = torch.max(outputs, 1)

    print ("prediction: ", preds[:10])
    print ("labels: ", labels[:10])
    ...

test data size:  torch.Size([2000])
prediction:  tensor([1, 1, 2, 1, 2, 3, 2, 3, 2, 3], device='cuda:0')
labels:      tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')
0 Loss: 3.2936 Acc: 0.1420

prediction:  tensor([1, 3, 3, 3, 3, 1, 2, 1, 1, 2], device='cuda:0')
labels:      tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], device='cuda:0')
0 Loss: 2.1462 Acc: 0.2780

prediction:  tensor([3, 3, 1, 2, 0, 1, 2, 1, 3, 2], device='cuda:0')
labels:      tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2], device='cuda:0')
0 Loss: 2.1975 Acc: 0.2560

Versus when I simply shuffle the data the accuracy is very high:
# Predict
randomize = True
...
test data size:  torch.Size([2000])
prediction:  tensor([0, 0, 3, 2, 0, 2, 0, 3, 0, 2], device='cuda:0')
labels:      tensor([0, 0, 3, 2, 0, 2, 0, 3, 0, 2], device='cuda:0')
0 Loss: 0.1500 Acc: 0.9580

prediction:  tensor([0, 3, 3, 3, 0, 0, 3, 2, 3, 3], device='cuda:0')
labels:      tensor([0, 2, 3, 0, 0, 0, 3, 2, 0, 3], device='cuda:0')
0 Loss: 0.1714 Acc: 0.9340

prediction:  tensor([3, 3, 2, 2, 3, 1, 3, 0, 2, 2], device='cuda:0')
labels:      tensor([3, 3, 2, 2, 3, 1, 3, 0, 2, 2], device='cuda:0')
0 Loss: 0.1655 Acc: 0.9400

Output:
You need to call model.eval() before testing. (And change it back by calling model.train() before training)
In training mode, BatchNorm will normalize your features by mean and variance. And you can expect that a batch with all classes being 1 will have very different statistic than a batch with a mixture of classes.


Instruction: How to find intersection of two sets of 2D tensors (points on a 2D plane) in Pytorch

Input:
I have two lists of Pytorch 2D tensors, which are points on a plane:
ListA = tensor([ [1.0,2.0], [1.0,3.0], [4.0,8.0] ], device='cuda:0')
ListB = tensor([ [5.0,7.0], [1.0,2.0], [4.0,8.0] ], device='cuda:0')

How to compute ?
Desired output = tensor([ [1.0,2.0] ,  [4.0,8.0] ], device='cuda:0')

I would like to find the Intersection between two lists ListA and ListB.
Note : Computation should be carried out only on CUDA.

Output:
There is no direct way in PyTorch to accomplish this (i.e., through a function). However, a workaround can be.
Flattening both tensors:
combined = torch.cat((ListA.view(-1), ListB.view(-1)))
combined
Out[52]: tensor([1., 2., 1., 3., 4., 8., 5., 7., 1., 2., 4., 8.], device='cuda:0')

Finding unique elements:
unique, counts = combined.unique(return_counts=True)
intersection = unique[counts > 1].reshape(-1, ListA.shape[1])
intersection
Out[55]:
tensor([[1., 2.],
        [4., 8.]], device='cuda:0')

Benchmarks:
def find_intersection_two_tensors(A: tensor, B:tensor):
       combined = torch.cat((A.view(-1), B.view(-1)))
       unique, counts = combined.unique(return_counts=True)
       return unique[counts > 1].reshape(-1, A.shape[1])

Timing it
%timeit find_intersection_two_tensors(ListA, ListB)
207 µs ± 2.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

If you are ok with moving to CPU, numpy could be a better solution in regards to performance:
def find_intersection_two_ndarray(AGPU: tensor, BGPU: tensor):
      A = AGPU.view(-1).cpu().numpy()
      B = BGPU.view(-1).cpu().numpy()
      C = np.intersect1d(A, B)
      return torch.from_numpy(C).cuda('cuda:0')

Timing it
%timeit find_intersection_two_ndarray(ListA, ListB)
85.4 µs ± 1.57 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)


Instruction: Can you use a different image size during transfer learning?

Input:
I have made a switch from TensorFlow to PyTorch recently. I use a famous Github repo for training on EfficientNets. I wrote the model initiation class as follows:
    class CustomEfficientNet(nn.Module):
        def __init__(self, config: type, pretrained: bool=True):
            super().__init__()
            self.config = config
            self.model = geffnet.create_model(
                model_name='EfficientNetB5',
                pretrained=pretrained)
            n_features = self.model.classifier.in_features
            self.model.classifier = nn.Linear(n_features, num_classes=5)

        def forward(self, input_neurons):
            output_predictions = self.model(input_neurons)
            return output_predictions

In addition, in my transforms, I tend to useResize(img_size = 512, img_size=512) for my training on certain image classification tasks (Mostly Kaggle Competitions). So the question here is, the official input size for EfficientNetB5 is 456x456, but I used 512x512 or even 256x256 and get very decent results. Is this normal? Or did I miss out the source code where the author will resize into the native resolution for you?
PS: This seems to be the norm in all the PyTorch Tutorials I saw on Kaggle. My full code can be seen here in this notebook here; I like to not leave logic gaps and therefore this question popped up.

Output:
Yes you can use different input sizes when it comes to transfer learning, after all the model that you load is just the set of weights of the fixed sequence of layers and fixed convolution kernel sizes. But I believe that there is some sort of minimum size that the model needs to work efficiently. You would still need to re-train the model but it will still converge quite quickly.
You would have to check the official implementation on the minimum size of the model like the one in VGG16 where they specify that the width and height need to be at least 32.


Instruction: Unable to make predictions due to incompatible matrix shape/size

Input:
I am trying to make a model to predict insurance cost based on the individual. And this is the code for it.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset

import numpy as np
import pandas as pd

from LSR import ListSearchReplace as LSR

csv = pd.read_csv("main.csv")
partialInputs = csv[["age", "bmi", "children"]]
smoker, sex = list(csv["smoker"]), list(csv["sex"])

L1 = LSR(smoker)
L1.replace("yes", 1, True)
L1.replace("no", 0, True)

L2 = LSR(sex)
L2.replace("female", 1, True)
L2.replace("male", 0, True)

pdReadySmoker = pd.DataFrame({"smoker": smoker})
pdReadySex = pd.DataFrame({"sex": sex})

SmokerAndSex = pd.merge(pdReadySmoker, pdReadySex, how="outer", left_index=True, right_index=True)

INPUTS = pd.merge(partialInputs, SmokerAndSex, how="outer", left_index=True, right_index=True)
TARGETS = csv["charges"]

INPUTS = torch.from_numpy(np.array(INPUTS, dtype='float32'))
TARGETS = torch.from_numpy(np.array(TARGETS, dtype='float32'))

print(INPUTS.shape, TARGETS.shape)

loss_fn = F.mse_loss
model = nn.Linear(5, 3) # <-- changing this, changes the error message.
opt = torch.optim.SGD(model.parameters(), lr=1e-5)

trainDataset = TensorDataset(INPUTS, TARGETS)
BATCH_SIZE = 5
trainDataloader = DataLoader(trainDataset, BATCH_SIZE, shuffle=True)

def fit(numEpochs, model, loss_fn, opt, trainDataloader):
    for epochs in range(numEpochs):
        for inputBatch, targetBatch in trainDataloader:
            preds = model(inputBatch)
            loss = loss_fn(preds, targetBatch)
            loss.backward()

            opt.step()

            opt.zero_grad()

        e = epoch + 1
        if e % 10 == 0:
            print(f"Epoch: {e/numEpochs}, loss: {loss.item():.4f}")

fit(100, model, loss_fn, opt, trainDataloader) <-- error

Error produced:
<ipython-input-7-b7028a3d94fd>:5: UserWarning: Using a target size (torch.Size([5])) that is different to the input size (torch.Size([5, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  loss = loss_fn(preds, targetBatch)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-d8f5bcdc847d> in <module>
----> 1 fit(100, model, loss_fn, opt, trainDataloader)

<ipython-input-7-b7028a3d94fd> in fit(numEpochs, model, loss_fn, opt, trainDataloader)
      3         for inputBatch, targetBatch in trainDataloader:
      4             preds = model(inputBatch)
----> 5             loss = loss_fn(preds, targetBatch)
      6             loss.backward()
      7

D:\coding\machine-learning\env-ml\lib\site-packages\torch\nn\functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2657         reduction = _Reduction.legacy_get_string(size_average, reduce)
   2658
-> 2659     expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   2660     return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   2661

D:\coding\machine-learning\env-ml\lib\site-packages\torch\functional.py in broadcast_tensors(*tensors)
     69         if any(type(t) is not Tensor for t in tensors) and has_torch_function(tensors):
     70             return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 71     return _VF.broadcast_tensors(tensors)  # type: ignore
     72
     73

RuntimeError: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 1

I've tried changing the dimensions of the of model, and these are a few of the changes made and the associated errors:
model = nn.Linear(5, 1338)

Error:
RuntimeError: The size of tensor a (1338) must match the size of tensor b (5) at non-singleton dimension 1

model = nn.Linear(1338, 1338)

Error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (5x5 and 1338x1338)

Sometimes this error, will make me change the matrix to the correct shape, but that results in the previous error regarding non-singleton dimension

Output:
This should be quite straight-forward, you only have a single layer. This is a matter of sorting the shapes right.
You are feeding a nn.Linear layer an input with shape input_shape. This type of layer takes two arguments: in_features the number of features in the input vector, and out_features the number of features in the resulting vector. Since you are using the F.mse_loss, your target vector needs to have the same shape as your prediction.
Bear in mind the first dimension is the batch dimension. In summary, your input tensor has shape (batch, input_size), your dense layer is defined as nn.Linear(input_size, out_size) and your target tensor has shape (batch, output_size).

Coming back to your case, your TARGETS tensor is of shape (1338) so you either mean to:

have a single prediction with 1338 components which would match a nn.Linear(?, 1338) and it would actually correspond to (1, 1338) (a single element in the batch). This can be fixed with TARGETS = TARGETS.unsqueeeze(0).

or, there are actually 1338 predictions one element, which would match a nn.Linear(?, 1) and the appropriate target shape would be (1338, 1). This can be fixed with TARGETS = TARGETS.unsqueeeze(-1) (adds an additional axis after the last dimension).


Instruction: Create a knn adjacent matrix in Pytorch

Input:
In Pytorch, say I have a top-k indexing matrix P(B,N,k), a weight matrix W(B,N,N) and a target matrix A(B,N,N), I want to get a adjacent matrix that operates as the following loops:
for i in range(B):
     for ii in range(N):
         for j in range(k):
             if weighted:
                 A[i][ii][P[i][ii][j]] = W[i][ii][P[i][ii][j]]
             else:
                 A[i][ii][P[i][ii][j]] = 1

How to implement it more efficiently and concisely in Pytorch?

Output:
I think you are looking for torch.scatter_:
A.scatter_(dim=2, index=P, src=W)  # for the weighted version
A.scatter_(dim=2, index=P, src=torch.ones_like(W))  # for the un-weighted version


Instruction: Pytorch Loss Function for making embeddings similar

Input:
I am working on an embedding model, where there is a BERT model, which takes in text inputs and output a multidimensional vector. The goal of the model is to find similar embeddings (high cosine similarity) for texts which are similar and different embeddings (low cosine similarity) for texts that are dissimilar.
When training in mini-batch mode, the BERT model gives a N*D dimensional output where N is the batch size and D is the output dimension of the BERT model.
Also, I have a target matrix of dimension N*N, which contains 1 in the [i, j] th position if the sentence[i] and sentence[j] are similar in sense and -1 if not.
What I want to do is find the loss/error for the entire batch by finding the cosine similarity of all embeddings in the BERT output and comparing it to the target matrix.
What I did was simply multiply the tensor with its transpose and then take elementwise sigmoid.
scores = torch.matmul(document_embedding, torch.transpose(document_embedding, 0, 1))
scores = torch.sigmoid(scores)

loss = self.bceloss(scores, targets)

But this does not seem to work.
Is there any other way to do this?
P.S. What I want to do is similar to the method described in this paper.

Output:
To calculate the cosine similarity between two vectors you would have used nn.CosineSimilarity. However, I don't think this allows you to get the pair-similarity from a set of n vectors. Fortunately enough, you can implement it yourself with some tensor manipulation.
Let us call x your document_embedding of shape (n, d) where d is the embedding size. We'll take n=3 and d=5. So x is made up of [x1, x2, x3].T.
>>> x = torch.rand(n, d)
tensor([[0.8620, 0.9322, 0.4220, 0.0280, 0.3789],
        [0.2747, 0.4047, 0.6418, 0.7147, 0.3409],
        [0.6573, 0.3432, 0.5663, 0.2512, 0.0582]])

The cosine similarity is a normalized dot product. The x@x.T matrix multiplication will give you the pairwise dot product: which contains: ||x1||², <x1/x2>, <x1/x3>, <x2/x1>, ||x2||², etc...
>>> sim = x@x.T
tensor([[1.9343, 1.0340, 1.1545],
        [1.0340, 1.2782, 0.8822],
        [1.1545, 0.8822, 0.9370]])

To normalize take the vector of all norms: ||x1||, ||x2||, and ||x3||:
>>> norm = x.norm(dim=1)
tensor([1.3908, 1.1306, 0.9680])

Construct the matrix containing the normalization factors: ||x1||², ||x1||.||x2||, ||x1||.||x3||, ||x2||.||x1||, ||x2||², etc...
>>> factor = norm*norm.unsqueeze(1)
tensor([[1.9343, 1.5724, 1.3462],
        [1.5724, 1.2782, 1.0944],
        [1.3462, 1.0944, 0.9370]])

Then normalize:
>>> sim /= factor
tensor([[1.0000, 0.6576, 0.8576],
        [0.6576, 1.0000, 0.8062],
        [0.8576, 0.8062, 1.0000]])

Alternatively, a quicker way which avoids having to create the norm matrix, is to normalize before multiplying:
>>> x /= x.norm(dim=1, keepdim=True)
>>> sim = x@x.T
tensor([[1.0000, 0.6576, 0.8576],
        [0.6576, 1.0000, 0.8062],
        [0.8576, 0.8062, 1.0000]])

For the loss function I would apply nn.CrossEntropyLoss straight away between the predicted similarity matrix and the target matrix, instead of applying sigmoid + BCE. Note: nn.CrossEntropyLoss includes nn.LogSoftmax.


Instruction: Why am I getting a low error before I did any optimization?

Input:
I am using a model training program I have built for a toy example and trying to use it on another example.
The only difference is this model was used for regression, hence I was using MSE as the error criterion, and now it is used for binary classification, hence I am using BCEWithLogitsLoss.
The model is very simple:
class Model(nn.Module):
    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc1 = nn.Sequential(
            nn.Linear(input_size, 8*input_size),
            nn.PReLU() #parametric relu - same as leaky relu except the slope is learned
        )
        self.fc2 = nn.Sequential(
            nn.Linear(8*input_size, 80*input_size),
            nn.PReLU()
        )
        self.fc3 = nn.Sequential(
            nn.Linear(80*input_size, 32*input_size),
            nn.PReLU()
        )
        self.fc4 = nn.Sequential(
            nn.Linear(32*input_size, 4*input_size),
            nn.PReLU()
        )
        self.fc = nn.Sequential(
            nn.Linear(4*input_size, output_size),
            nn.PReLU()
        )

    def forward(self, x, dropout=dropout, batchnorm=batchnorm):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.fc4(x)
        x = self.fc(x)

        return x

And this is where I run it:
model = Model(input_size, output_size)

if (loss == 'MSE'):
    criterion = nn.MSELoss()
if (loss == 'BCELoss'):
    criterion = nn.BCEWithLogitsLoss()

optimizer = torch.optim.SGD(model.parameters(), lr = lr)

model.train()
for epoch in range(num_epochs):
    # Forward pass and loss
    train_predictions = model(train_features)
    print(train_predictions)
    print(train_targets)

    loss = criterion(train_predictions, train_targets)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    train_size = len(train_features)
    train_loss = criterion(train_predictions, train_targets).item()
    pred = train_predictions.max(1, keepdim=True)[1]
    correct = pred.eq(train_targets.view_as(pred)).sum().item()
    #train_loss /= train_size
    accuracy = correct / train_size
    print('\nTrain set: Loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        train_loss, correct, train_size,
        100. * accuracy))

However, when I print the loss, for some reason the loss already starts very low (around 0.6) before I have done any backwards pass! It remains this low all subsequent epochs.
The prediction vector, however, looks like random garbage...
tensor([[-0.0447],
        [-0.0640],
        [-0.0564],
        ...,
        [-0.0924],
        [-0.0113],
        [-0.0774]], grad_fn=<PreluBackward>)
tensor([[0.],
        [0.],
        [0.],
        ...,
        [0.],
        [0.],
        [1.]])
epoch: 1, loss = 0.6842

I have no clue why is it doing that, and would appriciate any help.
Thanks!
EDIT:
I added the params if they can help anyone figuring this out:
if (dataset == 'adult_train.csv'):
    input_size=9
    print_every = 1
    output_size = 1
    lr = 0.001
    num_epochs = 10
    loss='BCELoss'

EDIT2: Added accuracy calculation in the middle block

Output:
BCELoss is not error.
The entropy of a Bernoulli distribution with p=0.5 is -ln(0.5) = 0.693. This is the loss you would expect if

Your data is evenly distributed
Your network is guessing randomly

or

Your network always predicts a uniform distribution

Your model is in the second case. The network is currently guessing slightly negative logits for every prediction. Those will be interpreted as 0 class predictions. Since it seems your data is imbalanced towards 0 labels your accuracy will be the same as a model that always predicts 0. This is just an artifact of random weight initialization. If you keep reinitializing your model you'll find that sometimes it will always predict 1 too.


Instruction: How do I load the CelebA dataset on Google Colab, using torch vision, without running out of memory?

Input:
I am following a tutorial on DCGAN. Whenever I try to load the CelebA dataset, torchvision uses up all my run-time's memory(12GB) and the runtime crashes. Am looking for ways on how I can load and apply transformations to the dataset without hogging my run-time's resources.
To Reproduce
Here is the part of the code that is causing issues.
# Root directory for the dataset
data_root = 'data/celeba'
# Spatial size of training images, images are resized to this size.
image_size = 64

celeba_data = datasets.CelebA(data_root,
                              download=True,
                              transform=transforms.Compose([
                                  transforms.Resize(image_size),
                                  transforms.CenterCrop(image_size),
                                  transforms.ToTensor(),
                                  transforms.Normalize(mean=[0.5, 0.5, 0.5],
                                                       std=[0.5, 0.5, 0.5])
                              ]))

The full notebook can be found here
Environment

PyTorch version: 1.7.1+cu101

Is debug build: False

CUDA used to build PyTorch: 10.1

ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)

GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)

CMake version: version 3.12.0

Python version: 3.6 (64-bit runtime)

Is CUDA available: True

CUDA runtime version: 10.1.243

GPU models and configuration: GPU 0: Tesla T4

Nvidia driver version: 418.67

cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

HIP runtime version: N/A

MIOpen runtime version: N/A

Versions of relevant libraries:

[pip3] numpy==1.19.4
[pip3] torch==1.7.1+cu101
[pip3] torchaudio==0.7.2
pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.8.2+cu101
[conda] Could not collect

Additional Context
Some of the things I have tried are:

Downloading and loading the dataset on seperate lines. e.g:

# Download the dataset only
datasets.CelebA(data_root, download=True)
# Load the dataset here
celeba_data = datasets.CelebA(data_root, download=False, transforms=...)

Using the ImageFolder dataset class instead of the CelebA class. e.g:

# Download the dataset only
datasets.CelebA(data_root, download=True)
# Load the dataset using the ImageFolder class
celeba_data = datasets.ImageFolder(data_root, transforms=...)

The memory problem is still persistent in either of the cases.

Output:
I did not manage to find a solution to the memory problem. However, I came up with a workaround, custom dataset. Here is my implementation:
import os
import zipfile
import gdown
import torch
from natsort import natsorted
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms

## Setup
# Number of gpus available
ngpu = 1
device = torch.device('cuda:0' if (
    torch.cuda.is_available() and ngpu > 0) else 'cpu')

## Fetch data from Google Drive
# Root directory for the dataset
data_root = 'data/celeba'
# Path to folder with the dataset
dataset_folder = f'{data_root}/img_align_celeba'
# URL for the CelebA dataset
url = 'https://drive.google.com/uc?id=1cNIac61PSA_LqDFYFUeyaQYekYPc75NH'
# Path to download the dataset to
download_path = f'{data_root}/img_align_celeba.zip'

# Create required directories
if not os.path.exists(data_root):
  os.makedirs(data_root)
  os.makedirs(dataset_folder)

# Download the dataset from google drive
gdown.download(url, download_path, quiet=False)

# Unzip the downloaded file
with zipfile.ZipFile(download_path, 'r') as ziphandler:
  ziphandler.extractall(dataset_folder)

## Create a custom Dataset class
class CelebADataset(Dataset):
  def __init__(self, root_dir, transform=None):
    """
    Args:
      root_dir (string): Directory with all the images
      transform (callable, optional): transform to be applied to each image sample
    """
    # Read names of images in the root directory
    image_names = os.listdir(root_dir)

    self.root_dir = root_dir
    self.transform = transform
    self.image_names = natsorted(image_names)

  def __len__(self):
    return len(self.image_names)

  def __getitem__(self, idx):
    # Get the path to the image
    img_path = os.path.join(self.root_dir, self.image_names[idx])
    # Load image and convert it to RGB
    img = Image.open(img_path).convert('RGB')
    # Apply transformations to the image
    if self.transform:
      img = self.transform(img)

    return img

## Load the dataset
# Path to directory with all the images
img_folder = f'{dataset_folder}/img_align_celeba'
# Spatial size of training images, images are resized to this size.
image_size = 64
# Transformations to be applied to each individual image sample
transform=transforms.Compose([
    transforms.Resize(image_size),
    transforms.CenterCrop(image_size),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5],
                          std=[0.5, 0.5, 0.5])
])
# Load the dataset from file and apply transformations
celeba_dataset = CelebADataset(img_folder, transform)

## Create a dataloader
# Batch size during training
batch_size = 128
# Number of workers for the dataloader
num_workers = 0 if device.type == 'cuda' else 2
# Whether to put fetched data tensors to pinned memory
pin_memory = True if device.type == 'cuda' else False

celeba_dataloader = torch.utils.data.DataLoader(celeba_dataset,
                                                batch_size=batch_size,
                                                num_workers=num_workers,
                                                pin_memory=pin_memory,
                                                shuffle=True)

This implementation is memory efficient and works for my use case, even during training the memory used averages around(4GB). I would however, appreciate further intuition as to what might be causing the memory problems.


Instruction: Display number of images per class using Pytorch

Input:
I am using Pytorch with FashionMNIST dataset I would like to display 8 image sample from each of the 10 classes. However, I did not figure how to split the training test into train_labels since I need to loop on the labels(class) and print 8 of each class.
any idea how I can achieve this?
classes = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot')

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              #  transforms.Lambda(lambda x: x.repeat(3,1,1)),
                                transforms.Normalize((0.5, ), (0.5,))])
# Download and load the training data
trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True)
# Download and load the test data
testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=True)

print('Training set size:', len(trainset))
print('Test set size:',len(testset))

Output:
If I understand you correctly you want to group your dataset by labels then display them.
You can start by constructing a dictionnary to store examples by label:
examples = {i: [] for i in range(len(classes))}

Then iterate over the trainset and append to the list using the label's index:
for x, i in trainset:
    examples[i].append(x)

However, this will go over the whole set. If you'd like to early stop and avoid gathering more than 8 per-class you can do so by adding conditions:
n_examples = 8
for x, i in trainset:
    if all([len(ex) == n_examples for ex in examples.values()])
        break
    if len(examples[i]) < n_examples:
        examples[i].append(x)

Only thing left is to display with torchvision.transforms.ToPILImage:
transforms.ToPILImage()(examples[3][0])

If you want to show more than one, you could use two consecutive torch.cat, one on dim=1 (by rows) then on dim=2 (by columns) to create a grid.
grid = torch.cat([torch.cat(examples[i], dim=1) for i in range(len(classes))], dim=2)
transforms.ToPILImage()(grid)

Possible result:


Instruction: Huggingface Transformer - GPT2 resume training from saved checkpoint

Input:
Resuming the GPT2 finetuning, implemented from run_clm.py
Does GPT2 huggingface  has a parameter to resume the training from the saved checkpoint, instead training again from the beginning? Suppose the python notebook crashes while training, the checkpoints will be saved, but when I train the model again still it starts the training from the beginning.
Source: here
finetuning code:
!python3 run_clm.py \
    --train_file source.txt \
    --do_train \
    --output_dir gpt-finetuned \
    --overwrite_output_dir \
    --per_device_train_batch_size 2 \
    --model_name_or_path=gpt2 \
    --save_steps 100 \
    --num_train_epochs=1 \
    --block_size=200 \
    --tokenizer_name=gpt2

From the above code, run_clm.py is a script provided by huggingface to finetune gpt2 to train with the customized dataset

Output:
To resume training from checkpoint you use the --model_name_or_path parameter. So instead of giving the default gpt2 you direct this to your latest checkpoint folder.
So your command becomes:
!python3 run_clm.py \
    --train_file source.txt \
    --do_train \
    --output_dir gpt-finetuned \
    --overwrite_output_dir \
    --per_device_train_batch_size 2 \
    --model_name_or_path=/content/models/checkpoint-5000 \
    --save_steps 100 \
    --num_train_epochs=1 \
    --block_size=200 \
    --tokenizer_name=gpt2


Instruction: Expected hidden[0] size (2, 8, 256), got [8, 256]

Input:
I have correct shape of hidden layer for printing as below.
print(h0.shape)
print(x.shape)

torch.Size([2, 8, 256])
torch.Size([8, 300, 300])

But I still have error as  Expected hidden[0] size (2, 8, 256), got [8, 256]
What could be wrong?
The whole code is as follows.
import torch
import torch.nn as nn
import torchvision
import matplotlib.pyplot as plt
import torchvision.transforms as tt
from torchvision.datasets import ImageFolder
from PIL import Image
import numpy as np
from torch.autograd import Variable

seq_len = input_size
hidden_size = 256 #size of hidden layers
num_classes = 5
num_epochs = 20
batch_size = 8
learning_rate = 0.001
# Fully connected neural network with one hidden layer
num_layers = 2 # 2 RNN layers are stacked
class LSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(LSTM, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)#batch must have first dimension
        #our inpyt needs to have shape
        #x -> (batch_size, seq, input_size)
        self.fc = nn.Linear(hidden_size, num_classes)#this fc is after RNN. So needs the last hidden size of RNN

    def forward(self, x):
        #according to ducumentation of RNN in pytorch
        #rnn needs input, h_0 for inputs at RNN (h_0 is initial hidden state)

        #the following one is initial hidden layer
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)#first one is number of layers and second one is batch size
        #output has two outputs. The first tensor contains the output features of the hidden last layer for all time steps
        #the second one is hidden state f
        print(h0.shape)
        print(x.shape)
        out, _ = self.lstm(x, h0)
        print(out.shape)
        #output has batch_size, seq_len, hidden size
        #we need to decode hidden state only the last time step
        #out (N, 30, 128)
        #Since we need only the last time step
        #Out (N, 128)
        out = out[:, -1, :] #-1 for last time step, take all for N and 128
        out = self.fc(out)
        return out

stacked_lstm_model = LSTM(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()#cross entropy has softmax at output
optimizer = torch.optim.Adam(stacked_lstm_model.parameters(), lr=learning_rate) #optimizer used gradient optimization using Adam

# Train the model
n_total_steps = len(train_dl)
for epoch in range(num_epochs):
    t_losses=[]
    for i, (images, labels) in enumerate(train_dl):
        # origin shape: [8, 1, 300, 300]
        # resized: [8, 300, 300]
        images = images.reshape(-1, seq_len, input_size).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = stacked_lstm_model(images)
        loss = criterion(outputs, labels)
        t_losses.append(loss)
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}')
    avgd_trainloss = sum(t_losses)/len(t_losses)
    acc=0
    v_losses=[]
    with torch.no_grad():
      n_correct = 0
      n_samples = 0
      for v_images, v_labels in valid_dl:
          v_images = v_images.reshape(-1, seq_len, input_size).to(device)
          v_labels = v_labels.to(device)
          v_outputs = stacked_lstm_model(v_images)
          v_loss = criterion(v_outputs, v_labels)
          v_losses.append(v_loss)
          # max returns (value ,index)
          _, v_predicted = torch.max(v_outputs.data, 1)
          n_samples += v_labels.size(0)
          n_correct += (v_predicted == v_labels).sum().item()

      acc = 100.0 * n_correct / n_samples
    avgd_validloss = sum(v_losses)/len(v_losses)
    print (f'Epoch [{epoch+1}/{num_epochs}], Train loss: {avgd_trainloss.item():.4f}, Valid loss: {avgd_validloss.item():.4f}, Valid accu: {acc.item():.2f}')

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    n_correct = 0
    n_samples = 0
    for images, labels in test_dl:
        images = images.reshape(-1, seq_len, input_size).to(device)
        labels = labels.to(device)
        outputs = stacked_lstm_model(images)
        # max returns (value ,index)
        _, predicted = torch.max(outputs.data, 1)
        n_samples += labels.size(0)
        n_correct += (predicted == labels).sum().item()

    acc = 100.0 * n_correct / n_samples
    print(f'Accuracy of the network on test images: {acc} %')

Output:
The LSTM requires two hidden states, not one. So instead of
h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)

use
h0 = (torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device), torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device))

So you need two hidden states in a tuple.


Instruction: RNN - RuntimeError: input must have 3 dimensions, got 2

Input:
I’m getting the following error:

RuntimeError: input must have 3 dimensions, got 2

I have a single feature column that I am trying to feed into a GRU neural net.
Below are my data loader and neural net. I have also included the output of my data loader when I retrieve a batch of data.
What am I doing wrong???
def batch_data(feature1, sequence_length, batch_size):
“”"
Batch the neural network data using DataLoader
:param feature1: the single feature column
:param sequence_length: The sequence length of each batch
:param batch_size: The size of each batch; the number of sequences in a batch
:return: DataLoader with batched data
“”"
    # total number of batches we can make
    n_batches = len(feature1)//batch_size

    # Keep only enough characters to make full batches
    feature1= feature1[:n_batches * batch_size]

    y_len = len(feature1) - sequence_length

    x, y = [], []
    for idx in range(0, y_len):
        idx_end = sequence_length + idx
        x_batch = feature1[idx:idx_end]
        x.append(x_batch)
        # only making predictions after the last item in the batch
        batch_y = feature1[idx_end]
        y.append(batch_y)

    # create tensor datasets
    data = TensorDataset(torch.from_numpy(np.asarray(x)), torch.from_numpy(np.asarray(y)))

    data_loader = DataLoader(data, shuffle=False, batch_size=batch_size)

    # return a dataloader
    return data_loader

# test dataloader on subset of actual data

test_text = data_subset_b
t_loader = batch_data(test_text, sequence_length=5, batch_size=10)

data_iter = iter(t_loader)
sample_x, sample_y = data_iter.next()

print(sample_x.shape)
print(sample_x)
print()
print(sample_y.shape)
print(sample_y)

When I pass in data, the following batch is generated…
torch.Size([10, 5])
tensor([[ 0.0045, 0.0040, -0.0008, 0.0005, -0.0012],
[ 0.0040, -0.0008, 0.0005, -0.0012, 0.0000],
[-0.0008, 0.0005, -0.0012, 0.0000, -0.0015],
[ 0.0005, -0.0012, 0.0000, -0.0015, 0.0008],
[-0.0012, 0.0000, -0.0015, 0.0008, 0.0000],
[ 0.0000, -0.0015, 0.0008, 0.0000, 0.0000],
[-0.0015, 0.0008, 0.0000, 0.0000, -0.0008],
[ 0.0008, 0.0000, 0.0000, -0.0008, -0.0039],
[ 0.0000, 0.0000, -0.0008, -0.0039, -0.0026],
[ 0.0000, -0.0008, -0.0039, -0.0026, -0.0082]], dtype=torch.float64)

torch.Size([10])
tensor([ 0.0000, -0.0015, 0.0008, 0.0000, 0.0000, -0.0008, -0.0039, -0.0026,
-0.0082, 0.0078], dtype=torch.float64)

Output:
As suggested by the error you got, the input tensor shape expected by the GRU is three dimensional with shape (batch_size, seq_len, input_size)1
But you are feeding a tensor of shape (10, 5). You said your input has one feature value, so you should add a dimension for input_size of size 1. This can be done like this
sample_x.unsqueeze(-1)


Instruction: Adding noise when using embedding layer in pytorch

Input:
I'm building a generator g,  that receives a latent-code (vector of shape 100) and outputs an image. Specifically, I have 1000 MNIST images, and I want the network to learn a latent code z_i for each image x_i, such that g(z_i)=x_i (this approach is known as Generative Latent Optimization). So I've used nn.Embedding(1000,embedding_dim=100) and standard generator architecture, that receives the code from embedding and outputs an image. As for loss, I combine reconstruction loss with regularization on the embedding-vector weights.
My probelm is: I'd like to add noise to the latent-code vector before it is inserted to the generator (in order to make the latent-code compact). However I'm a beginner, and I don't know whether I should call detach() when adding the noise or not. I'm not sure of my approach entirely. I don't want to learn the scale of the noise or anything.
Here's my attempt:
class net(nn.Module):
  def __init__():
    self.embed = nn.Embedding(1000,embedding_dim=100)
    self.generator = nn.sequential( nn.Linear(100, 84), .... )
  def forward(batch_indices):
    batch_codes = self.embed(batch_indices)
    noise = torch.randn_like(batch_codes) * sigma
    noisy_batch_codes = batch_codes + noise # SHOULD THIS BE batch_codes.detach() + noise ??
    return self.generator(noisy_batch_codes)

g = net()
optim = SGD(g.parameters(), lr=0.01)
for epoch in range(num_epochs):
  for orig_images, orig_images_idx in trainloader:
    optim.zero_grad()
    output = g(orig_images_idx)
    reconstruction_loss = nn.MSELoss()(output, orig_images)
    embed_vector_weights = g.embed.weight[orig_images_idx]
    reg_loss = torch.norm(embed_vector_weights) * reg_coeff
    loss = reconstruction_loss + reg_loss
    loss.backward()
    optim.step()

Output: If you detach before adding noise the gradients won't propagate to your encoder (the emedding layer in this case) so your encoder weights will never be updated. Therefore you should probably not detach if you want the encoder to learn.