Instruction: Why do we need to inherit from nn.Module in PyTorch?

Input:
I am going through Udacity's Intro To Deep Learning with Pytorch course In the Neural Network part the instructor says that "in the init method, we need to call super, we need to do that because then PyTorch will know to register all the different layers and operation, if you don't do this part it wont be able to track the things that you are adding to your network and it wont work". Could you kindly elaborate and explain what exactly is the role of super keyword here and what does nn.Module inherits which helps in "keeping track" of changes.
Further the Jupyter notebooks says the following about use of super,
class Network(nn.Module):

Here we're inheriting from nn.Module. Combined with
super().__init__() this creates a class that tracks the architecture
and provides a lot of useful methods and attributes. It is mandatory
to inherit from nn.Module when you're creating a class for your
network. The name of the class itself can be anything.

Output:
class Network(nn.Module) means that you defined a Network class that inherits all the methods and properties from nn.Module (Network is child class and nn.Module is parent class)
By using super().__init__() , Network's __init__ will be same as its parent __init__.
class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()

For example when you make a model:
model = Network()
by calling this model, actually nn.Module will handle this call and track changes.
visit this link if you want to see nn.Module source codes.


Instruction: Pytorch custom criterion depending on target

Input:
Im doing a research project where I want to create a custom loss function depending on the targets. I.e. I want to penalize with BCEWithLogitsLoss plus adding a hyperparameter lambda. I only want to add this hyperparameter if the model is not correctly detecting a class.
With more detail, I have a pretrained model that I want to retrain freezing some of the layers. This model detects faces in images with some probability. I want to penalize certain kind of images if they are incorrectly classified with a factor lambda (suppose that the images that need that penalization have a special character in the name or so)
From the source code of pytorch:
import torch.nn.modules.loss as l

class CustomBCEWithLogitsLoss(l._Loss):
    def __init__(self, weight: Optional[Tensor] = None, size_average=None, reduce=None, reduction: str = 'mean',
                 pos_weight: Optional[Tensor] = None) -> None:
        super(BCEWithLogitsLoss, self).__init__(size_average, reduce, reduction)
        self.register_buffer('weight', weight)
        self.register_buffer('pos_weight', pos_weight)
        self.weight: Optional[Tensor]
        self.pos_weight: Optional[Tensor]

    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        return F.binary_cross_entropy_with_logits(input, target,
                                                  self.weight,
                                                  pos_weight=self.pos_weight,
                                                  reduction=self.reduction)

Here, forward has two tensors as inputs, so I dont know how to add here the class of the images that I want to penalize with lambda. Adding lambda to the constructor is ok, but how to do the forward pass if it only allows tensors?
Edit:
To clarify the question, Suppose that I have a training/testing folder with the images. The files with the character @ in the filename are the ones that I want to classify correctly way more than the files without the character, with a factor lambda.
How can I tell in the regular fashion of training a model in pytorch, that those files have to use a lambda penalization (let's say that the loss function is lambda * BCEWithLogitLoss) but the other ones not? I'm using DataLoader.

Output:
You can create a custom class for your dataset or instead build on top of an existing built-in dataset. For instance, you can use datasets.ImageFolder as a base class. The logic added on top is to identify if the filename contains the special token, for example @ and provide this information in the element returned by __getitem__. Looking at the parent __getitem__ function from datasets.DatasetFolder, a minimal working implementation could be:
class Dataset(datasets.ImageFolder):
    def __init__(self, token, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.token = token

    def __getitem__(self, index):
        path, _ = self.samples[index]      # retrieve path of instance
        match = self.token in path         # determine if there is a match
        x, y = super().__getitem__(index)  # call parent to get input and label
        return x, int(match), y

Each dataset element consists of the input image tensor and a label 0, or 1 designating whether that input has the token in its filename.
>>> ds = Dataset(token='@', root='root_to_dataset')

Which you would use like any other dataset: with a DataLoader wrapper
>>> dl = DataLoader(ds, batch_size=2)

Now when iterating over this daloader, you will have:
>>> for x, m, y in dl:
...    # x is the batch of images (b, c, h, w)
...    # m is the batch of {0,1} whether inputs have the pattern in their path (b,)
...    # y is the batch of labels (b,)

Now that we have this, we need to apply the lambda factor on the loss terms. However, we can't assume here that all elements in a given minibatch will follow the criteria (i.e. have the pattern in their filename), therefore we need to handle this element-wise and not in reduced form.
If you take a look at the source file for built-in loss functions nn.modules.loss you will notice all loss functions are based on a class named _Loss which expects a reduction parameter. This will be useful for us.
First, consider switching off reduction on your loss function:
>>> bce = nn.BCEWithLogitLoss(reduction='none') # provide additional args if necessary

Considering we have the mask of "matches" m which contains a 1 when the input has the token in their filename, and 0 otherwise. And given a lambda factor lamb with which we want to weigh the elements where m=1, we can provide the following coefficient to our loss term to perform the desired operation:
>>> coeff = lamb*m + 1-m
# if m=0 => coeff=1;
# if m=1 => coeff=lamb;

To apply the loss strategy properly we simple point-wise multiply coeff with the unreduced loss term (which is shaped (b,)).
>>> weighted = coeff*bce_loss

All in all, this would look like this:
>>> for x, m, y in dl:
...    y_pred = model(x)
...    bce_loss = bce(y_pred, y)
...    coeff = lamb*m + 1-m
...    bce_weighted = torch.mean(coeff*bce_loss)
...    bce_weighted.backward()


Instruction: Pytorch: assigns values to a tensor by index

Input:
How to assign values to a Tensor by index like Numpy in python?
In numpy, we can fill values to an array by index:
array = np.zeros((10, 8, 3), dtype=np.float32)
for n in range(10):
    for k in range(4):
        array[n, k, :] = x, y, -2  # x,y are diffrent values in every loop
        array[n, 4 + k, :] = x, y, 0.4

If there is a zeros tensor using torch.zeros, how to fill values to it in Pytorch by the indexes?

Output:
Group the values to a tensor and then assign:
import torch

array = torch.zeros((10, 8, 3), dtype=torch.float32)
for n in range(10):
    for k in range(4):
        x, y = 1, -1
        array[n, k, :] = torch.tensor([x, y, -2])  # x,y are diffent values in every loop
        array[n, 4 + k, :] = torch.tensor([x, y, 0.4])


Instruction: how to improve rotation in a spatial transformation network

Input:
I am applying a spatial transformation network to a dataset I created.
The dataset consists of boots and shoes that are slightly rotated (random rotation between 10° and 30°) as shown in the dataset images figure. I trained my model on the Fashionmnist dataset and also tested it. I expected to get images that are aligned, but I got something like this:

this how how my CNN and stn look likes:
class STN_CNN(nn.Module):
def __init__(self):
    super(STN_CNN, self).__init__()
    self.cnn = nn.Sequential(
        nn.Conv2d(1, 10, kernel_size=3, stride=1, padding=0),
        nn.MaxPool2d(2, stride=2),
        nn.ReLU(),
        nn.Conv2d(10, 16, kernel_size=3, stride=1, padding=0),
        nn.MaxPool2d(2, stride=2),
        nn.ReLU()
    )
    self.classifier = nn.Sequential(
        nn.Linear(16*2*2, 32),
        nn.ReLU(),
        nn.Linear(32, 10)
    )
    self.localization = nn.Sequential(
        nn.Conv2d(1, 20, kernel_size=5, stride=1, padding=0),
        nn.MaxPool2d(2, stride=2),
        nn.ReLU(),
        nn.Conv2d(20, 20, kernel_size=5, stride=1, padding=0),
        nn.ReLU()
    )
    self.fc_loc = nn.Sequential(
        nn.Linear(20*8*8, 20),
        nn.ReLU(),
        nn.Linear(20, 6)
    )
    self.AvgPool = nn.AvgPool2d(2, stride=2)

    self.fc_loc[2].weight.data.zero_()
    self.fc_loc[2].bias.data.copy_(torch.tensor([1, 0, 0, 0, 1, 0], dtype=torch.float))

def stn(self, x):
    x_loc = self.localization(x)
    x_loc = x_loc.view(-1, 20*8*8)
    theta = self.fc_loc(x_loc)
    theta = theta.view(-1, 2, 3)

    grid = F.affine_grid(theta, x.size())
    x = F.grid_sample(x, grid)
    x = self.AvgPool(x)

    return x

def forward(self, x):
    x = self.stn(x)
    x = self.cnn(x)
    x = x.view(-1, 16*2*2)
    x = self.classifier(x)

    return x

i even trained my dataset for 100 epochs but i'm not getting any improvement
could someone just tell me how to improve the rotation part in my stn. Or Maybe if i'm doing something wrong just let me know. I'll be really happy if someone could help.
Thank you in advance.

Output: The problem was that I trained the model with a different pixel format. For example, imagine that the pixels in each image are normalized between 0 and 1 and the test image is not normalized example between 0-255. The matplotlib method allows plotting both form this has acted badly on the training.


Instruction: How to inspect values in binarized FairSeq datasets?

Input:
Running the fairseq-preprocess script produces binary files with integer indices corresponding to token ids in a dictionary.
When I no longer have the original tokenized texts, what is the simplest way to explore the binarized dataset? The documentation does not say much about how a dataset can be loaded for debugging purposes.

Output:
I worked around this by loading the trained model and using it to decode the binarized sentences back to strings:
from fairseq.models.transformer import TransformerModel

model_dir = ???
data_dir = ???

model = TransformerModel.from_pretrained(
    model_dir,
    checkpoint_file='checkpoint_best.pt',
    data_name_or_path=data_dir,
    bpe='sentencepiece',
    sentencepiece_model=model_dir + '/sentencepiece.joint.bpe.model'
)
model.task.load_dataset('train')
data_bin = model.task.datasets['train']
train_pairs = [
    (model.decode(item['source']), model.decode(item['target']))
    for item in data_bin
]


Instruction: Need clear concept of the dimensions of output and hidden from LSTM layers

Input:
I know that the output carries all hiddens from the last layer of all the time steps and the hidden is the last time step hiddens of all the layers.
This context has each document with 850 tokens. Each token is embedded into 100 dimension. I took a 2-layer LSTM with 100 dim hidden.
I thought it would take a token at a time step and produce 100 dim hidden. For 850 tokens in a document it will produce output = [1, 850, 100], hidden [1, 2, 100] and cell [1, 2, 100]. But the hidden and cell are [2, 850, 100].
input_dim = len(tok2indx)   # size of the vocabulary
emb_dim = 100               # Embedding of each word
hid_dim = 100               # The dimention of each hiddenstate comming out from a time step
n_layers = 2                # LSTM layers

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()

        self.hid_dim = hid_dim
        self.n_layers = n_layers

        self.embedding = nn.Embedding(input_dim, emb_dim)
        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout, device=device)
        self.dropout = nn.Dropout(dropout)

    def forward(self, X):

        embedded = self.embedding(X).to(device)
        outputs, (hidden, cell) = self.rnn(embedded)

        return outputs, hidden, cell

If the encoder is passed a single document
enc = Encoder()
encd = enc.forward(train_x[:1])
print(encd[0].shape, encd[1].shape, encd[2].shape)

Output:
torch.Size([1, 850, 100]) torch.Size([2, 850, 100]) torch.Size([2, 850, 100])

With ten documents
encd = enc.forward(train_x[:10])
print(encd[0].shape, encd[1].shape, encd[2].shape)

Output:
torch.Size([10, 850, 100]) torch.Size([2, 850, 100]) torch.Size([2, 850, 100])

Output:
What's tripping you up is the input format to LSTM. The default input shape to a LSTM layer is Sequence (L), batch (N), features (H). While in you code you are sending input as NLH (batch, sequence, features). To use this correctly set the parameter batch_first=True (to the LSTM layer), then the input and output will be as you expect.
But there is a catch here too. Only the output (1st of the outputs) will be NLH while both hidden and cell (2nd and 3rd of the outputs) will still be LNH format.
The second thing to note here is the hidden cell will have dimensionality equal to the number of layers ie 2 in your example (each layer will require fill of its own hidden weights), hence the output [2, 850, 100] instead of [1, 850, 100].


Instruction: genereate unique row index in a 2D tensor as an output 1D tensor with PyTorch

Input:
When I implement target in in-batch multi-class classification on PyTorch (version 1.6), I have the following problem.
I got a variable D <class 'torch.Tensor'> （related to label description) of size as torch.Size([16, 128]), i.e. [data_size,token_id_size].
The original idea was to generate a target tensor of torch.Size([16]), each value is unique, corresponding to the rows in D, from 0 to 16 as [0,1,2,...,15], for in-batch multi-class classification.
This can be done using target = torch.LongTensor(torch.arange(16))
But there maybe repeated, non-unique rows in D, so I would like that the same, unique row in D has the its unique index in target. For example D has row0, row1, row8 the same token_ids or vector and the other rows are all different from each other, then target should be [0,0,2,3,4,5,6,0,8,9,10,11,12,13,14,15] or [0,0,1,2,3,4,5,0,6,7,8,9,10,11,12,13], wher the former has still indexes 0-15 (but no 1 and 7) and the latter has indexes of all in 0-13.
How can I implement this?

Output:
See answers of the simplified question (i) generate 1D tensor as unique index of rows of an 2D tensor and (ii) generate 1D tensor as unique index of rows of an 2D tensor (keeping the order and the original index), which address the problem of this question.
But these seem not useful to improve the contrastive multi-class classification.


Instruction: What replacement variables are available to filename in the ModelCheckpoint callback in Pytorch Lightning

Input:
What replacement variables {replace_me} are available to populate the filename attribute on the ModelCallback checkpoint in pytorch?
I want to pass the object_id hparam and the checkpoint version (the same as used to create the parent folder) ver like so:
pl.callbacks.ModelCheckpoint(filename='weights_{object_id:03}_{ver}_{epoch}-{step}')

# Saves to file:
# weights_object_id=000_ver=0_epoch=3-step=27

Output:
Arbitrary metrics which have been logged and {epoch} and {step} can be used, as shown in ModelCheckpoint.
Metrics can be logged during training (in your model class which extends LightningModule) like:
def training_step(self, batch, batch_idx):
    loss, logs = self.step(batch, batch_idx)
    self.log_dict(
        {f"train_{k}": v for k, v in logs.items()},
        on_step=True,
        on_epoch=True
    )
    return loss

If you want additional variables, which aren't logged, extend ModelCheckpoint and override the format_checkpoint_name() method.


Instruction: AttributeError: module 'torch.utils' has no attribute 'make_grid'

Input: I tried following along with the PyTorch transfer learning tutorial and found it pulled an error code when I brought up the make grid.

Output:
It's because it's out of date. As of 1.11.0 it has been moved to torchvision.utils.make_grid instead of torch.utils.make_grid
https://pytorch.org/vision/main/generated/torchvision.utils.make_grid.html?highlight=make_grid#torchvision.utils.make_grid


Instruction: Dropout in custom LSTM in pytorch

Input:
I have built a custom peephole lstm, and I want to imitate the dropout part in the already built in nn.lstm. So, how to add the dropout like what this intialization of this lstm, nn.LSTM(input_size, hidden_size, dropout=0.3), do? I have an idea of how to do it, which is by just applying a normal dropout just before returning the output, like this:
# init method
self.dropout = nn.Dropout(0.3)

# forward method

hidden_seq = self.dropout(hidden_seq)

return hidden_seq, (h_t, c_t)

I just want to make sure that this is the right way. If not what to do?

Output:
nn.LSTM(... dropout=0.3) applies a Dropout layer on the outputs of each LSTM layer except the last layer. You can have multiple stacked layers by passing parameter num_layers > 1. If you want to add a dropout to the final layer (or if LSTM has only one layer), you have to add it as you are doing now.
If you want to replicate what LSTM dropout does (which is only in case of multiple layers), you can stack LSTM layers manually and add a dropout layer in between.


Instruction: Tensorboard: How to view pytorch model summary?

Input:
I have the following network.
import torch
import torch.nn as nn
from torch.utils.tensorboard import SummaryWriter

class Net(nn.Module):
    def __init__(self,input_shape, num_classes):
        super(Net, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(4,4)),

            nn.Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(4,4)),
        )

        x = self.conv(torch.rand(input_shape))
        in_features = np.prod(x.shape)

        self.classifier = nn.Sequential(
            nn.Linear(in_features=in_features, out_features=num_classes),
        )

    def forward(self, x):
        x = self.feature_extractor(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

net = Net(input_shape=(1,64,1292), num_classes=4)
print(net)

This prints the following:-
Net(
  (conv): Sequential(
    (0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=(4, 4), stride=(4, 4), padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=(4, 4), stride=(4, 4), padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=320, out_features=4, bias=True)
  )
)

However, I am trying various experiments and I want to keep track of network architecture on Tensorboard. I know there is a function writer.add_graph(model, input_to_model) but it requires input, or at least its shape should be known.
So, I tried writer.add_text("model", str(model)), but formatting is screwed up in tensorboard.

My question is, is there a way to at least visualize the way I can see by using print function in the tensorboard?

Output:
I can see everything is going right but there is just a formatting issue. Tensorboard understands markdown so you can actually replace \n with
 and   with &nbsp;.
Here is a detailed walkthrough. Suppose you have the following model:-
import torch
import torch.nn as nn
from torch.utils.tensorboard import SummaryWriter

class Net(nn.Module):
    def __init__(self,input_shape, num_classes):
        super(Net, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(4,4)),

            nn.Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=(4,4)),
        )

        x = self.conv(torch.rand(input_shape))
        in_features = np.prod(x.shape)

        self.classifier = nn.Sequential(
            nn.Linear(in_features=in_features, out_features=num_classes),
        )

    def forward(self, x):
        x = self.feature_extractor(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

net = Net(input_shape=(1,64,1292), num_classes=4)
print(net)

This prints the following and if can actually show it in the Tensorboard.
Net(
  (conv): Sequential(
    (0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=(4, 4), stride=(4, 4), padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=(4, 4), stride=(4, 4), padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=320, out_features=4, bias=True)
  )
)

There is function in add_graph(model, input) in SummaryWriter but you must create dummy input and in some cases it is difficult of to always know them. Instead do following:-
writer = SummaryWriter()

model_summary = str(model).replace( '\n', '
').replace(' ', '&nbsp;')
writer.add_text("model", model_summary)

writer.close()

Above produces following text in tensorboard:-


Instruction: How padding=zeros works in pytorch in functional.conv1d

Input:
This following code below giving a output of shape (1,1,3) for the shape of xodd is (1,1,2). The given kernel shape is(112, 1, 1).
from torch.nn import functional as F
output = F.conv1d(xodd, kernel, padding=zeros)

How the padding=zeros works?
And also, How can I write an equivalent code in tensorflow so that the output is as same as the above output?

Output:
What is padding=zeros?
If we set paddin=zeros, we don't need to add numbers at the right and the left of the tensor.
Padding=0:
from torch.nn import functional as F
import torch
inputs = torch.randn(33, 16, 6) # (minibatch,in_channels,features)
filters = torch.randn(20, 16, 5) # (out_channels, in_channels, kernel_size)
out_tns = F.conv1d(inputs, filters, stride=1, padding=0)
print(out_tns.shape)
# torch.Size([33, 20, 2]) # (minibatch,out_channels,(features-kernel_size+1))

Padding=2:(We want to add two numbers at the right and the left of the tensor)
inputs = torch.randn(33, 16, 6) # (minibatch,in_channels,features)
filters = torch.randn(20, 16, 5) # (out_channels, in_channels, kernel_size)
out_tns = F.conv1d(inputs, filters, stride=1, padding=2)
print(out_tns.shape)
# torch.Size([33, 20, 6]) # (minibatch,out_channels,(features-kernel_size+1+2+2))

How can I write an equivalent code in tensorflow:
import tensorflow as tf
input_shape = (33, 6, 16)
x = tf.random.normal(input_shape)
out_tf = tf.keras.layers.Conv1D(filters = 20,
                                kernel_size = 5,
                                strides = 1,
                                input_shape=input_shape[1:])(x)
print(out_tf.shape)
# TensorShape([33, 2, 20])

# If you want that tensor have shape exactly like pytorch you can transpose
tf.transpose(out_tf, [0, 2, 1]).shape
# TensorShape([33, 20, 2])


Instruction: generate 1D tensor as unique index of rows of an 2D tensor

Input:
Let's say we transform a 2D tensor to a 1D tensor by giving each, different row a different index, from 0 to the number of rows - 1.
[[1,2],[1,3],[1,4]] -> [0,1,2]

But if there are same rows, then we repeate the index, like this below.
[[1,2],[1,2],[1,4]] -> [0,0,2]

[[1,2],[1,3],[1,2]] -> [0,1,0]

How to implement this on PyTorch?

Output:
You can do so using torch.Tensor.unique and returning the inverse which provides the indices out of the box:
>>> _, i = x.unique(dim=0, return_inverse=True)

>>> i # first example
tensor([0, 1, 2])


Instruction: Is AI Gym's action and state data normalized?

Input:
I am trying to implement a DDPG agent to control the Gym's Pendulum.
Since I am new to gym, I was wondering if the state data collected via env.step(action) is already normalized or I should do that manually. Also, should action be normalized or in the [-2, 2] range?
Thanks

Output: env.step(action) returns tuple (observation, reward, done, info). If you're referring to data in observation, then answer is no, it's not normalized (all with accordance to observation space section: three coordinates with values in [-1; 1] for the first two and [-8; 8] for the last one). action should be normalized to [-2; 2] range, though it'll be addinionally clipped to this range.


Instruction: Using matrix inversion in Torch versus Numpy in Gaussian Process Regression (previous question solved)

Input:
I am building a Torch-based Gaussian Process model which allows me to use custom kernels and take advantage of auto derivatives. However, I find that even in the simplest case, the Numpy-based implementation gave a very result than that from Torch-based one.
Allow me to attached the code
import torch
import matplotlib.pyplot as plt
import numpy as np

length_scale, sigma_s, sigma_n = (0.8, 1.0, 0.1)

def secret_function(x, noise=0.0):
    return torch.sin(x) + noise * torch.randn(x.shape)

X = 10 * torch.rand(40, 1) - 4
X = X.reshape(-1,1)
Y = secret_function(X, noise=1e-1)

x = torch.linspace(-8, 8, 100).reshape(-1, 1)
y = secret_function(x)

x_all = torch.cat([X, x],0)
# The following cdist compute 2-norm distance, not squared one
K = ((-0.5)*torch.cdist(x_all, x_all, p =2)/length_scale**2).exp()
K = K * sigma_s**2

L1 = X.shape[0]

K_X = K[:L1,:L1]
K_x = K[L1:,L1:]
K_xX = K[L1:,:L1]
K_Xx = K[:L1,L1:]

K_inv = torch.linalg.inv(K_X + sigma_n**2*torch.eye(L1))
tmp = torch.matmul(K_xX, K_inv)

mu = torch.matmul(tmp, Y)
covar = K_x - torch.matmul(tmp, K_Xx)

var = torch.diagonal(covar, 0)
std = torch.sqrt(var).reshape(-1,1)

plt.figure(figsize=(12, 6))

plt.plot(x.numpy(), mu.numpy(),'b-.')
plt.plot(X.numpy(), Y.numpy(), 'r.')

plt.fill_between(x.numpy().flat, (mu - 2 * std).numpy().flat, (mu + 2 * std).numpy().flat, color="#dddddd")

And here is what I get for the regression result

The exp kernel is used, which causes the predictive mean not smooth. Below I try with same regression but using Numpy library.
X = X.numpy()
Y = Y.numpy()
x = x.numpy()
y = y.numpy()

x_all = np.vstack([X,x])
dist2 = (x_all**2).sum(1)[:,None] + (x_all**2).sum(1) - 2*x_all.dot(x_all.T)
K_np = sigma_s**2 * np.exp((-0.5)*dist2/length_scale**2)

K_Xnp = K_np[:L1,:L1]
K_xnp = K_np[L1:,L1:]
K_xXnp = K_np[L1:,:L1]
K_Xxnp = K_np[:L1,L1:]

K_invnp = np.linalg.inv(K_Xnp + sigma_n**2*np.eye(L1))
tmp_np = np.matmul(K_xXnp, K_invnp)

mu_np = np.matmul(tmp_np, Y)
covar_np = K_xnp - np.matmul(tmp_np, K_Xxnp)

var_np = np.diagonal(covar_np, 0)
std_np = np.sqrt(var_np).reshape(-1,1)

plt.figure(figsize=(12, 6))

plt.plot(x, mu_np,'b-.')
#plt.plot(x, y,'k-')
plt.plot(X, Y, 'r.')

plt.fill_between(x.flat, (mu_np - 2 * std_np).flat, (mu_np + 2 * std_np).flat, color="#dddddd")

Now I got a much smoother prediction from the numpy-based regression.

What can I do in the torch version to get the same smooth result as
np's one? (lesson learned. cdist computes p-norm distance, not squared distance)

Output:
It looks like one time you are computing pairwise distances
torch.cdist(x_all, x_all, p =2)

while the other time you're using squared distances
dist2 = (x_all**2).sum(1)[:,None] + (x_all**2).sum(1) - 2*x_all.dot(x_all.T)

But note that the latter is about the most expensive and unsable way of computing it, I'd recommend using scipy.spatial.distance.cdist or at least doing something like
dist2 = ((x_all[None, :, :] - x_all[:, None, :])**2).sum(axis=-1)


Instruction: Pythonic squeeze and unsqueeze data dimensions

Input:
For giving the data in good dimensions to a PyTorch Model, I use squeeze en unsqueeze function like this:
   inps = torch.FloatTensor(data[0])
   tgts = torch.FloatTensor(data[1])

   tgts = torch.unsqueeze(tgts, -1)
   tgts = torch.unsqueeze(tgts, -1)
   tgts = torch.unsqueeze(tgts, -1)
   inps = torch.unsqueeze(inps, -1)
   inps = torch.unsqueeze(inps, -1)
   inps = torch.unsqueeze(inps, -1)

and this:
    inps = torch.FloatTensor(data[0])
    tgts = torch.FloatTensor(data[1])

    tgts = torch.unsqueeze(tgts, 1)
    tgts = torch.unsqueeze(tgts, 1)
    tgts = torch.unsqueeze(tgts, 1)
    inps = torch.unsqueeze(inps, 1)
    inps = torch.unsqueeze(inps, 1)
    inps = torch.unsqueeze(inps, 1)

But of course, I'm kinda embarrassed to have this repetitive part in my code. Is there another way, more pythonic and clean, to write this code, please?

Output:
You can use torch.Tensor.view like below:
how_many_unsqueeze = 3
extra_dims = (1,) * how_many_unsqueeze
# extra_dims -> (1,1,1)

inps.view(-1, *extra_dims) # -> (-1,1,1,1)
tgts.view(-1, *extra_dims) # -> (-1,1,1,1)

You can use torch.reshape like below:
But after using like in your question you need back to original shape
Instead of unsqueeze
inps = torch.reshape(inps, (len(data[0]),1,1,1))
tgts = torch.reshape(tgts, (len(data[1]),1,1,1))

Instead of squeeze
inps = torch.reshape(inps, (len(data[0]),))
tgts = torch.reshape(tgts, (len(data[1]),))


Instruction: Pytorch ;Optimizer got an empty parameter list

Input:
Am new in the deep learning concept using pytorch and am try to build a binary classifier model. I have tried some of the solution here on stack overflow but I can't seem to solve it. Maybe it is due to the nature of my code. Can someone figure out what could be the cause of this error in my code
Here is my code
import torch
import torch.nn as nn
import numpy as np
from sklearn.datasets import make_blobs
import matplotlib.pyplot as pyp

# creating a dummy dataset from the make_blobs dataset
number_of_samples=5000
#divide the dataset into training(80%) and testing(20%)
training_number=int(number_of_samples*0.8)
#creating the dummy datasest
x,y=make_blobs(n_samples=number_of_samples,centers=2,n_features=64,cluster_std=10,random_state=2020)
y=y.reshape(-1,1)
#converting the numpy arrays into torch tensors
x,y=torch.from_numpy(x),torch.from_numpy(y)
x,y=x.float(),y.float()
#splitting the datasets into training and testing
x_train,x_test=x[:training_number],x[training_number:]
y_train,y_test=y[:training_number],y[training_number:]

#printing the shapes of each dataset
print("x_train shape:",x_train.shape)
print("x_test shape:",x_test.shape)
print("y_train shape:",y_train.shape)
print("y_test shape:",y_test.shape)

#a class to define the neural network us torch nn module
#neural network will have 3 hidden layers and 1 output layer
#hidden layers will have 64,256 and 1024 neurons
#output layer will have a single neuron
class neuralnetwork(nn.Module):
    def _init_(self):
        super().__init__()
        torch.manual_seed(2020)
        self.fc1 = nn.Linear(64, 256)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(256, 1024)
        self.relu2 = nn.ReLU()
        self.out = nn.Linear(1024, 1)
        self.final = nn.Sigmoid()

    def forward(self, x):
        op = self.fc1(x)
        op = self.relu1(op)
        op = self.fc2(op)
        op = self.relu2(op)
        op = self.out(op)
        y = self.final(op)
        return y

#defining the loss,optimizer and training function for the neural network
def train_network(model,optimizer,loss_function,num_epochs,batch_size,x_train,y_train):
    #start model training
    model.train()
    loss_for_every_epoch=nn.ModuleList()
    for epoch in range(num_epochs):
        train_loss=0.0
        for i in range(0,x_train.shape[0],batch_size):
            #extract train batch from x and y
            input_data=x_train[i:min(x_train.shape[0]),i+batch_size]
            labels=y_train[i:min(y_train.shape[0]),i+batch_size]
            #set gradients to zero before beginning optimization
            optimizer.zero_grad()
            #forwad pass
            output_data=model(input_data)
            #calculate loss
            loss=loss_function(output_data,labels)
            #backpropagate
            loss.backward()
            #update weights
            optimizer.step()
            train_loss+=loss.item()*batch_size
        print("Epoch: {} - Loss:{:.4f}".format(epoch+1,train_loss ))
        loss_for_every_epoch.extend([train_loss])
    #predict
    y_test_prediction=model(x_test)
    a=np.where(y_test_prediction>0.5,1,0)
    return loss_for_every_epoch

#create an object of the class
model=neuralnetwork()
#define the loss function
loss_function = nn.BCELoss()#binary cross entropy loss function
#define optimizer
adam_optimizer=torch.optim.Adam(params=model.parameters(),lr=0.001)
#define epochs and batch size
number_of_epochs=100
batch_size=16
#Calling the function for training and pass model, optimizer, loss and related paramters
adam_loss=train_network(model,adam_optimizer,loss_function,number_of_epochs,batch_size,x_train,y_train)

I get the error
Value error:optimizer got an empty parameter list

The error is mainly generated from this section of code
 #create an object of the class
    model=neuralnetwork()
    #define the loss function
    loss_function = nn.BCELoss()#binary cross entropy loss function
    #define optimizer
    adam_optimizer=torch.optim.Adam(params=model.parameters(),lr=0.001)
    #define epochs and batch size
    number_of_epochs=100
    batch_size=16
    #Calling the function for training and pass model, optimizer, loss and related paramters
    adam_loss=train_network(model,adam_optimizer,loss_function,number_of_epochs,batch_size,x_train,y_train)

What could be the cause in my code.
Here is the full stack trade
    Traceback (most recent call last)
g:\My Drive\CODE\pythondatascience\simpleneuralnetwork.ipynb Cell 7' in <cell line: 6>()
      4 loss_function = nn.BCELoss()#binary cross entropy loss function
      5 #define optimizer
----> 6 adam_optimizer=torch.optim.Adam(params=model.parameters(),lr=0.001)
      7 #define epochs and batch size
      8 number_of_epochs=100

File c:\Users\DAVE\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\optim\adam.py:81, in Adam.__init__(self, params, lr, betas, eps, weight_decay, amsgrad, maximize)
     78     raise ValueError("Invalid weight_decay value: {}".format(weight_decay))
     79 defaults = dict(lr=lr, betas=betas, eps=eps,
     80                 weight_decay=weight_decay, amsgrad=amsgrad, maximize=maximize)
---> 81 super(Adam, self).__init__(params, defaults)

File c:\Users\DAVE\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\optim\optimizer.py:49, in Optimizer.__init__(self, params, defaults)
     47 param_groups = list(params)
     48 if len(param_groups) == 0:
---> 49     raise ValueError("optimizer got an empty parameter list")
     50 if not isinstance(param_groups[0], dict):
     51     param_groups = [{'params': param_groups}]

ValueError: optimizer got an empty parameter list

Output: It should be def __init__, not def _init_ in neuralnetwork class. You are not initializing your model object at all. Thus, it does not have any parameters.


Instruction: Initializing weights of each module in a Sequential Module in LibTorch

Input:
I am trying to port some code from PyTorch to LibTorch.
Supposing in a struct inheriting from torch::nn::Module I have a registered sequential module like
branch1 = register_module("branch1", torch::nn::Sequential(torch::nn::Conv2d(torch::nn::Conv2dOptions(in_channels, branch_channels, kernel_size).padding(0)),
                torch::nn::BatchNorm2d(torch::nn::BatchNorm2dOptions(branch_channels)),
                torch::nn::ReLU());

I am interested in applying a weight initialization function to each component separately (ideally with a different initialization algorithm per module type), say a function that takes in a torch::nn::Module or a pointer to a torch::nn::Module, what is the simplest way to achieve this?
Edit: My current attempt.
#include <torch/torch.h>

using namespace std;

void init_conv(torch::nn::Conv2d& conv) {
    torch::NoGradGuard noGrad;
    torch::nn::init::kaiming_normal_(conv->weight, 0.0, torch::kFanOut, torch::kReLU);
    torch::nn::init::constant_(conv->bias, 0);
}

void init_bn_2d(torch::nn::BatchNorm2d& bn_2d) {
    torch::NoGradGuard noGrad;
    torch::nn::init::constant_(bn_2d->weight, 1);
    torch::nn::init::constant_(bn_2d->bias, 0);
}

void initialize_sequential(torch::nn::Sequential& seq) {
torch::NoGradGuard noGrad;
vector<shared_ptr<torch::nn::Module>> mods = seq->modules();
for (auto mod = std::begin(mods); mod != end(mods); ++mod) {
    shared_ptr<torch::nn::Module> m = *mod;
    torch::nn::Module* m_ = m.get();
    if (typeid(*m_) == typeid(torch::nn::Conv2dImpl*)) {
        torch::nn::Conv2d* c = dynamic_cast<torch::nn::Conv2d*>(m_);
        init_conv(*c);
    }
    if (typeid(*m_) == typeid(torch::nn::BatchNorm2dImpl*)) {
        torch::nn::BatchNorm2d* bn = dynamic_cast<torch::nn::BatchNorm2d*>(m_);
        init_bn_2d(*bn);
    }
  }
}

Output:
I can use the apply() function on the sequential object like this:
    #include <torch/torch.h>
    void sequential_init_weights(torch::nn::Module& m){
           if ((typeid(m) == typeid(torch::nn::Conv2dImpl))) {
                auto p = m.named_parameters(false);
                auto w = p.find("weight");
                auto b = p.find("bias");

                if (w != nullptr) torch::nn::init::kaiming_normal_(*w, 0.0,
                                  torch::kFanOut, torch::kReLU);
                if (b != nullptr) torch::nn::init::constant_(*b, 0.0);
            }
            if ((typeid(m) == typeid(torch::nn::BatchNorm2dImpl))) {
                auto p = m.named_parameters(false);
                auto w = p.find("weight");
                auto b = p.find("bias");

                if (w != nullptr) torch::nn::init::constant_(*w, 1.0);
                if (b != nullptr) torch::nn::init::constant_(*b, 0.0);
            }
        }

    struct example_mod : torch::nn::Module {
        example_mod(int64_t in_channels, int64_t out_channels) {
            m = register_module("m", torch::nn::Sequential(torch::nn::Conv2d(torch::nn::Conv2dOptions(in_channels, out_channels, 1)),
                        torch::nn::BatchNorm2d(torch::nn::BatchNorm2dOptions(out_channels),
             torch::nn::ReLU()));
            m->apply(sequential_init_weights);
        }
        torch::nn::Sequential m = nullptr;
    };

Basically just write a function that parses the modules by typeid then used the named parameters to get what you need and pass those to an init function, seems to work pretty well.


Instruction: Int object is not iterable when looping through a dataset

Input:
I am trying to extract data in batches from a dataset to train a model
Here is part of the code
#defining the loss,optimizer and training function for the neural network
def train_network(model,optimizer,loss_function,num_epochs,batch_size,x_train,y_train):
    #start model training
    model.train()
    loss_for_every_epoch=nn.ModuleList()
    for epoch in range(num_epochs):
        train_loss=0.0
        for i in range(0,x_train.shape[0],batch_size):
            #extract train batch from x and y
            input_data=x_train[i:min(x_train.shape[0]),i+batch_size]
            labels=y_train[i:min(y_train.shape[0]),i+batch_size]
            #set gradients to zero before beginning optimization
            optimizer.zero_grad()
            #forwad pass
            output_data=model(input_data)
            #calculate loss
            loss=loss_function(output_data,labels)
            #backpropagate
            loss.backward()
            #update weights
            optimizer.step()
            train_loss+=loss.item()*batch_size
        print("Epoch: {} - Loss:{:.4f}".format(epoch+1,train_loss ))
        loss_for_every_epoch.extend([train_loss])
    #predict
    y_test_prediction=model(x_test)
    a=np.where(y_test_prediction>0.5,1,0)
    return loss_for_every_epoch

#create an object of the class
model=neuralnetwork()
#define the loss function
loss_function = nn.BCELoss()#binary cross entropy loss function
#define optimizer
adam_optimizer=torch.optim.Adam(params=model.parameters(),lr=0.001)
#define epochs and batch size
number_of_epochs=100
batch_size=16
#Calling the function for training and pass model, optimizer, loss and related paramters
adam_loss=train_network(model,adam_optimizer,loss_function,number_of_epochs,batch_size,x_train,y_train)

But i get the following error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
g:\My Drive\CODE\pythondatascience\simpleneuralnetwork.ipynb Cell 7' in <cell line: 11>()
      9 batch_size=16
     10 #Calling the function for training and pass model, optimizer, loss and related paramters
---> 11 adam_loss=train_network(model,adam_optimizer,loss_function,number_of_epochs,batch_size,x_train,y_train)

g:\My Drive\CODE\pythondatascience\simpleneuralnetwork.ipynb Cell 5' in train_network(model, optimizer, loss_function, num_epochs, batch_size, x_train, y_train)
      7 train_loss=0.0
      8 for i in range(0,4000,batch_size):
      9     #extract train batch from x and y
---> 10     input_data=x_train[i:min(x_train.shape[0]),i+batch_size]
     11     labels=y_train[i:min(y_train.shape[0]),i+batch_size]
     12     #set gradients to zero before beginning optimization

TypeError: 'int' object is not iterable

What could be the cause of the error cause thse source that am using to write the programmed did it in the exact same way
In addition to that can someone explain to me specifically what this  line means
input_data=x_train[i:min(x_train.shape[0]),i+batch_size]

x_train is a dataset

Output:
input_data=... is taking a slice of your data to use it as an input to your training algorithm.
Although the comparison statement is invalid (x_train.shape should return a tuple of ints, you shouldn't be able to do min(x_train.shape[0]) without comparing it to something else. My guess is that you should have input_data=x_train[i:min(x_train.shape[0],i+batch_size)]. You have the same issue for y_train as well.


Instruction: PyTorch and Neural Networks: How many parameters in a layer?

Input:
Ive seen many sources talk about the number of parameters in a neural network and mention that it is calculated as:
num parameters =  ((shape of width of the filter * shape of height of the filter * number of filters in the previous layer+1)*number of filters)
but I've been having trouble understanding how that applies to networks created using nn from torch
for example how many parameters would this network have?
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10)
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

Output:
The object nn.Linear represents a matrix with dimention [m, n].
For example, nn.Linear(28*28, 512) has (28*28)*512 parameters(weights).
Check here for more information about it.
The object nn.Flatten() and nn.ReLU() do not contain parameters.


Instruction: How to save/load a model checkpoint with several losses in Pytorch?

Input:
Using Ubuntu 20.04, Pytorch 1.10.1.
I am trying to solve a music generation task with a transformer architecture and multi-embeddings, for processing tokens with several characteristics.
In each training iteration, I have to calculate the loss of each token characteristic and store it in a vector, then I suppose that I should store in a checkpoint a vector containing all of them (or something similar), instead of what I'm doing now which is saving the total loss. I would like to know how to store all losses in the checkpoint (be able to keep training when loading it), or if it isn't needed at all.
The epochs loop:
for epoch in range(0, epochs):

    print('Epoch: ', epoch)

    loss = trfrmr.train(epoch+1, model, train_loader, train_loss_func, opt, lr_scheduler, num_iters=-1)
    loss_train.append(loss)

    torch.save({
        'epoch': epoch,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': opt.state_dict(),
        'loss': loss,
        }, "model_pop909_checkpoint.pth")

The training loop:
for batch_num, batch in enumerate(dataloader):
    time_before = time.time()

    opt.zero_grad()

    x = batch[0].to(get_device())
    tgt = batch[1].to(get_device())

    # x is the input sequence (N,T,Z), that should be input into the transformer forward function as (T,N,Z)
    y = model(x.permute(1, 0, 2))

    # tgt is the real output sequence, of shape (N,T,Z), T is sequence length, N batch size, Z the different token types
    # y are the output logits, is a list of Z tensors of shape (T,N,C*) where C is the vocabulary size, and will vary depending on the token type (pitch, velocity etc...)
    losses = []
    for j in range(LEN_VOCAB):
        aux_loss = loss.forward(y[j].permute(1, 2, 0),
                                        tgt[..., j])  # shapes (N,C,T) and (N,T), see Pytorch cross-entropy for details
        losses.append(aux_loss)

    losses_sum = sum(losses)  # here we sum, but we could also have mean for instance

    losses_sum.backward()
    opt.step()

    if lr_scheduler is not None:
         lr_scheduler.step()

    lr = opt.param_groups[0]['lr']

    loss_hist.append(losses_sum)
    if batch_num == num_iters:
       break

Thanks in advance.
HOURS LATER EDIT: SOLUTION TO MY SPECIFIC PROBLEM
The problem was that when loading again the model I wasn't doing it properly (not loading optimizer parameters, but only model ones). Now in my code, at the beginning of the loop I do:
if loaded:
    print('Loading model and optimizer...')
    model.load_state_dict(checkpoint['model_state_dict'], strict=False)
    opt.load_state_dict(checkpoint['optimizer_state_dict'])
    print('Loaded succesfully!')

And I also load the epoch:
epoch = 0
if loaded:
    print('Loading epoch value...')
    epoch = checkpoint['epoch']
    print('Loaded succesfully!')

Output: As far as I can tell from your code, your loss function has no custom learnable parameters; it's just recalculated every time your model iterates. Thus there is no need to save its value other than keeping a history of it; it is not required to continue training from a checkpoint.


Instruction: How do I set the dimensions of Conv1d correctly?

Input:
This is a toy example as I'm learning PyTorch and using it on one-dimensional time series, in this case a sine wave.
I'm trying to use Conv1d, but I get the following error:
RuntimeError: Given groups=1, weight of size [5, 1, 2], expected input[1, 994, 5] to have 1 channels, but got 994 channels instead

My 'lookback' is 5 time steps, and the shape of my data batch is [994, 5].
What am I doing wrong?
import torch;from torch.utils.data import Dataset, DataLoader
import torch.nn.functional as F;import pytorch_lightning as pl
from torch import nn, tensor

class TsDs(torch.utils.data.Dataset):
  def __init__(self, s, l=5): super().__init__();self.l,self.s=l,s
  def __len__(self): return self.s.shape[0] - 1 - self.l
  def __getitem__(self, i): return self.s[i:i+self.l], torch.log(self.s[i+self.l+1]/self.s[i+self.l])
  def plt(self): plt.plot(self.s)

class TsDm(pl.LightningDataModule):
  def __init__(self, length=5000, batch_size=1000): super().__init__();self.batch_size=batch_size;self.s = torch.sin(torch.arange(length)*0.2) + 5
  def train_dataloader(self): return DataLoader(TsDs(self.s[:3999]), batch_size=self.batch_size, shuffle=False)
  def val_dataloader(self): return DataLoader(TsDs(self.s[4000:]), batch_size=self.batch_size)

dm = TsDm()

class MyModel(pl.LightningModule):
    def __init__(self, learning_rate=0.01):
        super().__init__();self.learning_rate = learning_rate
        super().__init__();self.learning_rate = learning_rate
        self.network = nn.Sequential(nn.Conv1d(1,5,2),nn.ReLU(),nn.Linear(5,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
        # self.network = nn.Sequential(nn.Linear(5,5),nn.ReLU(),nn.Linear(5,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
    def forward(self, x): return self.network(x)
    def step(self, batch, batch_idx, stage):
        x, y = batch
        loss = -torch.mean(self(x)*y)
        print(loss)
        return loss
    def training_step(self, batch, batch_idx): return self.step(batch, batch_idx, "train")
    def validation_step(self, batch, batch_idx): return self.step(batch, batch_idx, "val")
    def configure_optimizers(self): return torch.optim.SGD(self.parameters(), lr=self.learning_rate)

mm = MyModel(0.01);trainer = pl.Trainer(max_epochs=10)
trainer.fit(mm, datamodule=dm)

Output:
There are two issues in your code:

Looking at the documentation of nn.Conv1d, your input shape should be (B, C, L). In your default case, you have L=5, the sequence length, but you need to create that extra dimension representing the feature size of a sequence element, here C=1. You can do so by changing TsDs's  __getitem__ function to:
def __getitem__(self, i):
    x = self.s[i:i+self.l] # minibatch x shaped (1, self.l)
    y = torch.log(self.s[i+self.l+1]/self.s[i+self.l]) # minibatch y shaped (1,)
    return x, y

Your convolutional layer has a stride of 1 and a size of 2, this means its output will be shaped (B, 5, L-1=4). The following layer is a fully connected layer instantiated as nn.Linear(5, 3), which means it expects (*, H_in=5) and will output (*, H_out). You can either

You can flatten the conv1d output with nn.Flatten and feed it to a bigger fully connected layer (for instance nn.Linear(20, 3).

You can use a convolutional layer with a wider kernel, if you use a kernel of 5 (your sequence length you will end up with a tensor of (B, 5, 1) which you feed to a nn.Linear(5, 3). Although this approach doesn't really scale when L is changed.

You could apply a nn.AvgPool1d to get an average representation of the sequence after the convolutional layers have been applied.

Those are just a few directions...


Instruction: Where can one get the coco utils modules? (unable to find them in hf datasets)

Input:
I have been following this tutorial which is a walk-through of fine-tuning a pre-trained Detr model
While trying to evaluate the model, I should be using from datasets import get_coco_api_from_dataset.  However, I am getting an ImportError while doing that:
ImportError: cannot import name 'get_coco_api_from_dataset'
I don't seem to find the coco_eval module too. So, this line fails: from datasets.coco_eval import CocoEvaluator.  Were the coco-related modules shifted somewhere, or do I need to import them from a different package?
Can someone please help me understand how can I import this module?

Output: So it seems it gets cloned in the step before in that notebook. See https://github.com/facebookresearch/detr/tree/main/datasets.


Instruction: Python - AttributeError: 'numpy.ndarray' object has no attribute 'to'

Input:
I now have the updated code as follows:
    # Hyperparameters
random_seed = 123
learning_rate = 0.01
num_epochs = 10
batch_size = 128

device = torch.device("cuda:1" if torch.cuda.is_available() else "cpu")
for epoch in range(num_epochs):
model = resnet34.train()
for batch_idx, (features, targets) in enumerate(train_generator):
    features = features.to(device)
    targets = targets.to(device)

    ### FORWARD AND BACK PROP
    logits = model(features)
    cost = torch.nn.functional.cross_entropy(logits, targets)
    optimizer.zero_grad()

    cost.backward()

    ### UPDATE MODEL PARAMETERS
    optimizer.step()

    ### LOGGING
    if not batch_idx % 50:
        print ('Epoch: %03d/%03d | Batch %03d/%03d | Cost: %.4f'
               %(epoch+1, num_epochs, batch_idx,
                 len(datagen)//batch_size, cost))

model = model.eval() # eval mode to prevent upd. batchnorm params during inference
with torch.set_grad_enabled(False): # save memory during inference
    print('Epoch: %03d/%03d training accuracy: %.2f%%' % (
          epoch+1, num_epochs,
          compute_accuracy(model, train_generator)))

When having only one image, the code runs fine. But, when I add another image or more, I get the following:
features = features.to(device)
targets = targets.to(device)
AttributeError: 'numpy.ndarray' object has no attribute 'to'

Output:
It would be nice to see your train_generator code for clarity, but it does not seem to be a torch DataLoader. In this case, you should probably convert your arrays to tensors manually. There are several ways to do so:

torch.from_numpy(numpy_array) - for numpy arrays;
torch.as_tensor(list) - for common lists and tuples;
torch.tensor(array) should also work but the above ways will avoid copying the data when possible.


Instruction: How to explicitly include decision variables in pytorch

Input:
I’m implementing a neural network from a paper in PyTorch. Here is the screenshot of the paper:
Here N_Psi is a neural network, and K is a decision matrix. The way I came up with is to include an extra linear layer for K, but wondering if there's any chance to explicitly define K as decision variables in a more direct way?
Any hint would be very helpful. Thanks in advance!

Output:
You can define your decision matrix as a fully connected layer with no bias, using nn.Linear. Then you have to add this additional layer to your optimizer parameter list. Given N your neural network, K you linear layer, and optim your torch.optim.Optimizer class, you can:
optimizer = optim(list(N.parameters()) + list(K.parameters()))

Then in the inference stage, given x_n+1 and x_n, do something like:
mse = F.mse_loss(N(x_n+1), K(N(x_n)))
reg_1 = K.weight.pow(2).sum()
reg_2 = p2v(N.parameters()).sum()
loss = mse + lamb_1*reg_1 + lamb_2*reg_2

Where we imported:

torch.nn.functional as F
torch.nn.utils.parameters_to_vector as p2v


Instruction: How to chang the output of the concrete function in Tensorflow 2.x?

Input:
I have trained a TensorFlow model and saved it to a local disk. when I loaded it and do inference, how can I get the output of the intermediate layer?
I use the example in the tutorial as a demo.
The model is:

class MyModel(Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.conv1 = Conv2D(32, 3, activation='relu')
    self.flatten = Flatten()
    self.d1 = Dense(128, activation='relu')
    self.d2 = Dense(10)

  def call(self, x):
    x = self.conv1(x)
    x = self.flatten(x)
    x = self.d1(x)
    return self.d2(x)

# Create an instance of the model
model = MyModel()

I save the model to local and load it in another place.
# save model to local
tf.saved_model.save(model, export_dir="./saved_model")

# load model from local
loaded = tf.saved_model.load("./saved_model")
concrete_fun = loaded.signatures["serving_default"]
# do reference
out = concrete_fun(tf.zeros((2, 28, 28, 1)))
out["output_1"].shape

As I know, the concrete function is unique to the input and output.
How can I get the output, weights and bias of intermediate layers, for example self.conv1?

Output:
You can try running:
print([var for var in concrete_fun.trainable_variables])

to get your each layer's weights and biases. To access the graph of your model, you can run concrete_fun.graph. See here for more details.
To access the output of intermediate layers, it would be easiest to save the model like this:
model.save('your_model', save_format='tf')

and then load it:
model.save('your_model', save_format='tf')
model = tf.keras.models.load_model('your_model')
conv_layer = model.get_layer(index=0)
print(conv_layer(tf.random.normal((1, 28, 28, 1))).shape)


Instruction: What should I think about when writing a custom loss function?

Input:
I'm trying to get my toy network to learn a sine wave.
I output (via tanh) a number between -1 and 1, and I want the network to minimise the following loss, where self(x) are the predictions.
loss = -torch.mean(self(x)*y)

This should be equivalent to trading a stock with a sinusoidal price, where self(x) is our desired position, and y are the returns of the next time step.
The issue I'm having is that the network doesn't learn anything. It does work if I change the loss function to be torch.mean((self(x)-y)**2) (MSE), but this isn't what I want. I'm trying to focus the network on 'making a profit', not making a prediction.
I think the issue may be related to the convexity of the loss function, but I'm not sure, and I'm not certain how to proceed. I've experimented with differing learning rates, but alas nothing works.
What should I be thinking about?
Actual code:
%load_ext tensorboard
import matplotlib.pyplot as plt; plt.rcParams["figure.figsize"] = (30,8)
import torch;from torch.utils.data import Dataset, DataLoader
import torch.nn.functional as F;import pytorch_lightning as pl
from torch import nn, tensor
def piecewise(x): return 2*(x>0)-1

class TsDs(torch.utils.data.Dataset):
  def __init__(self, s, l=5): super().__init__();self.l,self.s=l,s
  def __len__(self): return self.s.shape[0] - 1 - self.l
  def __getitem__(self, i): return self.s[i:i+self.l], torch.log(self.s[i+self.l+1]/self.s[i+self.l])
  def plt(self): plt.plot(self.s)

class TsDm(pl.LightningDataModule):
  def __init__(self, length=5000, batch_size=1000): super().__init__();self.batch_size=batch_size;self.s = torch.sin(torch.arange(length)*0.2) + 5 + 0*torch.rand(length)
  def train_dataloader(self): return DataLoader(TsDs(self.s[:3999]), batch_size=self.batch_size, shuffle=True)
  def val_dataloader(self): return DataLoader(TsDs(self.s[4000:]), batch_size=self.batch_size)

dm = TsDm()

class MyModel(pl.LightningModule):
    def __init__(self, learning_rate=0.01):
        super().__init__();self.learning_rate = learning_rate
        super().__init__();self.learning_rate = learning_rate
        self.conv1 = nn.Conv1d(1,5,2)
        self.lin1 = nn.Linear(20,3);self.lin2 = nn.Linear(3,1)
        # self.network = nn.Sequential(nn.Conv1d(1,5,2),nn.ReLU(),nn.Linear(20,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
        # self.network = nn.Sequential(nn.Linear(5,5),nn.ReLU(),nn.Linear(5,3),nn.ReLU(),nn.Linear(3,1), nn.Tanh())
    def forward(self, x):
        out = x.unsqueeze(1)
        out = self.conv1(out)
        out = out.reshape(-1,20)
        out = nn.ReLU()(out)
        out = self.lin1(out)
        out = nn.ReLU()(out)
        out = self.lin2(out)
        return nn.Tanh()(out)

    def step(self, batch, batch_idx, stage):
        x, y = batch
        loss = -torch.mean(self(x)*y)
        # loss = torch.mean((self(x)-y)**2)
        print(loss)
        self.log("loss", loss, prog_bar=True)
        return loss
    def training_step(self, batch, batch_idx): return self.step(batch, batch_idx, "train")
    def validation_step(self, batch, batch_idx): return self.step(batch, batch_idx, "val")
    def configure_optimizers(self): return torch.optim.SGD(self.parameters(), lr=self.learning_rate)

#logger = pl.loggers.TensorBoardLogger(save_dir="/content/")
mm = MyModel(0.1);trainer = pl.Trainer(max_epochs=10)
# trainer.tune(mm, dm)
trainer.fit(mm, datamodule=dm)
#

Output:
If I understand you correctly, I think that you were trying to maximize the unnormalized correlation between the network's prediction, self(x), and the target value y.
As you mention, the problem is the convexity of the loss wrt the model weights. One way to see the problem is to consider that the model is a simple linear predictor w'*x, where w is the model weights, w' it's transpose, and x the input feature vector (assume a scalar prediction for now). Then, if you look at the derivative of the loss wrt the weight vector (i.e., the gradient), you'll find that it no longer depends on w!
One way to fix this is change the loss to,
loss = -torch.mean(torch.square(self(x)*y))

or
loss = -torch.mean(torch.abs(self(x)*y))

You will have another big problem, however: these loss functions encourage unbound growth of the model weights. In the linear case, one solves this by a Lagrangian relaxation of a hard constraint on, for example, the norm of the model weight vector. I'm not sure how this would be done with neural networks as each layer would need it's own Lagrangian parameter...


Instruction: How to free GPU from CUDA (using Pytorch)?

Input:
I'm using spark/face-alignment to generate faces that are almost the same.
 fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, flip_input=False) # try to use GPU with Pytorch depenencies.
 imageVector.append( convertImagefa(image, fa))
 del fa
 gc.collect()
 torch.cuda.empty_cache() # trying to clean up cuda.
 return imageVector

I'm on a 1 machine with 4 threads that all try to access the GPU.  As such I have worked out a strategy that every 4th request it uses the GPU.  This seems to fit in memory.
My issue is that when I clean up after cuda it never actually fully cleans.  I'll see the load move around the threads and some space free up but CUDA never lets go of the last 624MiB.  Is there a way to clean it all the way up?
nvidia-smi
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     17132  C   .../face-the-same/bin/python      624MiB |
|    0   N/A  N/A     17260  C   .../face-the-same/bin/python     1028MiB |
|    0   N/A  N/A     17263  C   .../face-the-same/bin/python      624MiB |
|    0   N/A  N/A     17264  C   .../face-the-same/bin/python      624MiB |

FYI: I ended up using a distributed lock to pin the GPU computation to one executor/process id.  THis was the outcome derived from the comment made from @Jan.

Output: According to https://discuss.pytorch.org/t/pytorch-do-not-clear-gpu-memory-when-return-to-another-function/125944/3 this is due to the CUDA context remaining in place unless you end the script. They recommend calling torch.cuda.empty_cache() to clear the cache, however, there will always be a remainder. To get rid of that you could switch to processes instead of threads so that the process can actually be killed without killing your programm (but that'll be quite some effort I suppose).


Instruction: How to create RNN layer on top of BERT multilingual in pytorch

Input:
I am working on a classification problem. I want to pass the BERT embedding to RNN layer and then FCN layer at the end for classification. But I am facing some issues, is there anyone who have worked on the same problem.
I created this class as below
class BERTClass(torch.nn.Module):
    def __init__(self):
        super(BERTClass, self).__init__()
        self.l1 = BertModel.from_pretrained('bert-base-multilingual-cased', return_dict=False)
        # for param in self.l1.parameters():
        #   param.requires_grad = False
        self.l2 = torch.nn.Dropout(0.4)
        self.l3 = torch.nn.RNN(768, 1028)
        self.activation = torch.nn.ReLU()
        self.l4 = torch.nn.Dropout(0.2)
        self.l5 = torch.nn.Linear(1028, 128)
        self.activation2 = torch.nn.ReLU()
        self.l6 = torch.nn.Linear(128, 10)

    def forward(self, ids, mask, token_type_ids):
        _, output_1= self.l1(ids, attention_mask = mask, token_type_ids = token_type_ids)
        output_2 = self.l2(output_1)
        output3 = self.l3(output_2)
        act = self.activation(output3)
        output4 = self.l4(act)
        output5 = self.l5(output4)
        act2 = self.activation2(output5)
        output6 = self.l6(act2)
        return output6

model = BERTClass()

but I am getting an error
<ipython-input-23-bbe09bd88901> in forward(self, ids, mask, token_type_ids)
     22         output_2 = self.l2(output_1)
     23         output3 = self.l3(output_2)
---> 24         act = self.activation(output3)
     25         output4 = self.l4(act)
     26         output5 = self.l5(output4)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/activation.py in forward(self, input)
     96
     97     def forward(self, input: Tensor) -> Tensor:
---> 98         return F.relu(input, inplace=self.inplace)
     99
    100     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in relu(input, inplace)
   1440         result = torch.relu_(input)
   1441     else:
-> 1442         result = torch.relu(input)
   1443     return result
   1444

TypeError: relu(): argument 'input' (position 1) must be Tensor, not tuple

Output:
The output of torch.nn.RNN is a tuple of shape (output, h_n) (for more info about this layer, visit this link)
So the input of activation layer in your code, should be just first element of RNN output (output3[0]).
Final code would be:
def forward(self, ids, mask, token_type_ids):
    _, output_1= self.l1(ids, attention_mask = mask, token_type_ids = token_type_ids)
    output_2 = self.l2(output_1)
    output3 = self.l3(output_2)
    act = self.activation(output3[0])
    output4 = self.l4(act)
    output5 = self.l5(output4)
    act2 = self.activation2(output5)
    output6 = self.l6(act2)
    return output6


Instruction: RuntimeError: Given groups=1, weight of size [64, 64, 1, 1], expected input[4, 1, 1080, 1920] to have 64 channels, but got 1 channels instead

Input:
I want to train a U-net segmentation model on the German Asphalt Pavement Distress (GAPs) dataset using U-Net. I'm trying to modify the model at https://github.com/khanhha/crack_segmentation to train on that dataset.
Here is the folder containing all the related files and folders:
https://drive.google.com/drive/folders/14NQdtMXokIixBJ5XizexVECn23Jh9aTM?usp=sharing
I modified the training file, and renamed it as "train_unet_GAPs.py". When I try to train on Colab using the following command:
!python /content/drive/Othercomputers/My\ Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py -data_dir "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/GAPs/" -model_dir /content/drive/Othercomputers/My\ Laptop/crack_segmentation_khanhha/crack_segmentation-master/model/ -model_type resnet101

I get the following error:
total images = 2410
create resnet101 model
Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to /root/.cache/torch/hub/checkpoints/resnet101-63fe2227.pth
100% 171M/171M [00:00<00:00, 212MB/s]
Started training model from epoch 0
Epoch 0:   0% 0/2048 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 259, in <module>
    train(train_loader, model, criterion, optimizer, validate, args)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/train_unet_GAPs.py", line 118, in train
    masks_pred = model(input_var)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/drive/Othercomputers/My Laptop/crack_segmentation_khanhha/crack_segmentation-master/unet/unet_transfer.py", line 224, in forward
    conv2 = self.conv2(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torchvision/models/resnet.py", line 144, in forward
    out = self.conv1(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 447, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 444, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 64, 1, 1], expected input[4, 1, 1080, 1920] to have 64 channels, but got 1 channels instead
Epoch 0:   0% 0/2048 [00:08<?, ?it/s]

I think that this is because the images of GAPs dataset are grayscale images (with one channel), while Resnet expects to receive RGB images with 3 channels.
How can I solve this issue? How can I modify the model to receive grayscale images instead of RGB images? I need help with that. I have no experience with torch, and I think this implementation uses built-in Resnet model.

Output:
I figured out few things with your code.
According to the trace back, you are using a resnet based Unet model.
Your current model forward method is defined as :
def forward(self, x):
    #conv1 = self.conv1(x)
    #conv2 = self.conv2(conv1)
    conv2 = self.conv2(x)
    conv3 = self.conv3(conv2)
    conv4 = self.conv4(conv3)
    conv5 = self.conv5(conv4)
    ...

Your error comes from self.conv2(x), because, conv2 takes a matrix with a number of channels of 64. It means, something is missing, or.. commented :)
By changing
    #conv1 = self.conv1(x)
    #conv2 = self.conv2(conv1)
    conv2 = self.conv2(x)

into
    conv1 = self.conv1(x)
    conv2 = self.conv2(conv1)

Will fix the problem the problem of 64 channels as input. But, there is another problem :
Using an input of (B,1,H,W), no matters what B, H and W are, won't be possible with your current architecture. Why ? Because of this :
resnet34 = torchvision.models.resnet34(pretrained=False)
resnet101 = torchvision.models.resnet101(pretrained=False)
resnet152 = torchvision.models.resnet152(pretrained=False)

print(resnet34.conv1)
-> Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

print(resnet101.conv1)
-> Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

print(resnet152.conv1)
-> Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)

In any case, the layer conv1 of resnet, takes a 3 channels input.
Once you have made those modifications, you should also try your network with a dummy example like :
model = UNetResNet(34,num_classes=2)
out = model(torch.rand(4,3,1920,1920))
print(out.shape)
-> (4,2,1920,1920) | (batch_size, num_classes, H, W)

Why your width and height are the same here ? Because your current architecture only supports squared images.
For example :
-> (1080,1920) = dim mismatching during concatenation part
-> (1920,1920) = success
-> (108,192) = dim mismatching during concatenation part
-> (192,192) = success

Conclusion :

Modify your network to accept grayscale images if your dataset is made of grayscale images.
Preprocess your images to make Width=Height.

Edit (device mismatch) :
class UNetResNet(nn.Module):

    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d

        if encoder_depth == 34:
            self.encoder = torchvision.models.resnet34(pretrained=pretrained)
            bottom_channel_nr = 512
        elif encoder_depth == 101:
            self.encoder = torchvision.models.resnet101(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 152:
            self.encoder = torchvision.models.resnet152(pretrained=pretrained)
            bottom_channel_nr = 2048
        else:
            raise NotImplementedError('only 34, 101, 152 version of Resnet are implemented')

        self.pool = nn.MaxPool2d(2, 2)

        self.relu = nn.ReLU(inplace=True)

        #self.conv1 = nn.Sequential(self.encoder.conv1,
        #                           self.encoder.bn1,
        #                           self.encoder.relu,
        #                           self.pool)

        self.conv1 = nn.Sequential(nn.Conv2d(1,64,kernel_size=(7,7),stride=(2,2),padding=(3,3),bias=False), # 1 Here is for grayscale images, replace by 3 if you need RGB/BGR
                                   nn.BatchNorm2d(64),
                                   nn.ReLU(),
                                   self.pool
                                )

        self.conv2 = self.encoder.layer1

        self.conv3 = self.encoder.layer2

        self.conv4 = self.encoder.layer3

        self.conv5 = self.encoder.layer4

        self.center = DecoderBlockV2(bottom_channel_nr, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec5 = DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec4 = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8,
                                   is_deconv)
        self.dec3 = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2,
                                   is_deconv)
        self.dec2 = DecoderBlockV2(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                                   is_deconv)
        self.dec1 = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
        self.dec0 = ConvRelu(num_filters, num_filters)
        self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

    def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        pool = self.pool(conv5)
        center = self.center(pool)

        dec5 = self.dec5(torch.cat([center, conv5], 1))

        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(F.dropout2d(dec0, p=self.dropout_2d))


Instruction: How do you design an LSTM to recognize images after extracting features with a CNN?

Input:
I am creating a captcha image recognition system. It first extracts the features of the images with ResNet and then uses LSTM to recognize the words and letter in the image. An fc layer is supposed to connect the two. I have not designed a LSTM model before and am very new to machine learning, so I am pretty confused and overwhelmed by this.
I am confused enough that I am not even totally sure what questions I should ask. But here are a couple things that stand out to me:

What is the purpose of embedding the captions if the captcha images are all randomized?
Is the linear fc layer in the first part of the for loop the correct way to connect the CNN feature vectors to the LSTM?
Is this a correct use of the LSTM cell in the LSTM?

And, in general, if there are any suggestions of general directions to look into, that would be really appreciated.
So far, I have:
class LSTM(nn.Module):
    def __init__(self, cnn_dim, hidden_size, vocab_size, num_layers=1):
        super(LSTM, self).__init__()

        self.cnn_dim = cnn_dim      #i think this is the input size
        self.hidden_size = hidden_size
        self.vocab_size = vocab_size      #i think this should be the output size

        # Building your LSTM cell
        self.lstm_cell = nn.LSTMCell(input_size=self.vocab_size, hidden_size=hidden_size)

        '''Connect CNN model to LSTM model'''
        # output fully connected layer
        # CNN does not necessarily need the FCC layers, in this example it is just extracting the features, that gets set to the LSTM which does the actual processing of the features
        self.fc_in = nn.Linear(cnn_dim, vocab_size)   #this takes the input from the CNN takes the features from the cnn              #cnn_dim = 512, hidden_size = 128
        self.fc_out = nn.Linear(hidden_size, vocab_size)     # this is the looper in the LSTM           #I think this is correct?

        # embedding layer
        self.embed = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)

        # activations
        self.softmax = nn.Softmax(dim=1)

    def forward(self, features, captions):

        #features: extracted features from ResNet
        #captions: label of images

        batch_size = features.size(0)
        cnn_dim = features.size(1)

        hidden_state = torch.zeros((batch_size, self.hidden_size)).cuda()  # Initialize hidden state with zeros
        cell_state = torch.zeros((batch_size, self.hidden_size)).cuda()   # Initialize cell state with zeros

        outputs = torch.empty((batch_size, captions.size(1), self.vocab_size)).cuda()
        captions_embed = self.embed(captions)

        '''Design LSTM model for captcha image recognition'''
        # Pass the caption word by word for each time step
        # It receives an input(x), makes an output(y), and receives this output as an input again recurrently
        '''Defined hidden state, cell state, outputs, embedded captions'''

        # can be designed to be word by word or character by character

        for t in range(captions).size(1):
            # for the first time step the input is the feature vector
            if t == 0:
                # probably have to get the output from the ResNet layer
                # use the LSTM cells in here i presume

                x = self.fc_in(features)
                hidden_state, cell_state = self.lstm_cell(x[t], (hidden_state, cell_state))
                x = self.fc_out(hidden_state)
                outputs.append(hidden_state)

            # for the 2nd+ time steps
            else:
                hidden_state, cell_state = self.lstm_cell(x[t], (hidden_state, cell_state))
                x = self.fc_out(hidden_state)
                outputs.append(hidden_state)

            # build the output tensor
            outputs = torch.stack(outputs,dim=0)

        return outputs

Output:
nn.Embedding() is usually used to transfer a sparse one-hot vector to a dense vector (e.g. transfer 'a' to [0.1,0.2,...]) for computation practically. I do not understand why you try to embed captions, which looks like ground-truth. If you want to compute loss with that, try nn.CTCLoss().
If you are going to send a string to LSTM, it is recommended to embed characters in the string with nn.Embedding() firstly, which makes them dense and computational-practical. But if the inputs of LSTM is something extracted from CNN (or other modules), it is already dense and computational-practical and not necessary to project them with fc_in from my view.
I often use nn.LSTM() instead of nn.LSTMCell(), for the latter is troublesome.

There are some bugs in your code and I fixed them:
import torch
from torch import nn

class LSTM(nn.Module):
    def __init__(self, cnn_dim, hidden_size, vocab_size, num_layers=1):
        super(LSTM, self).__init__()

        self.cnn_dim = cnn_dim  # i think this is the input size
        self.hidden_size = hidden_size
        self.vocab_size = vocab_size  # i think this should be the output size

        # Building your LSTM cell
        self.lstm_cell = nn.LSTMCell(input_size=self.vocab_size, hidden_size=hidden_size)

        '''Connect CNN model to LSTM model'''
        # output fully connected layer
        # CNN does not necessarily need the FCC layers, in this example it is just extracting the features, that gets set to the LSTM which does the actual processing of the features
        self.fc_in = nn.Linear(cnn_dim,
                               vocab_size)  # this takes the input from the CNN takes the features from the cnn              #cnn_dim = 512, hidden_size = 128
        self.fc_out = nn.Linear(hidden_size,
                                vocab_size)  # this is the looper in the LSTM           #I think this is correct?

        # embedding layer
        self.embed = nn.Embedding(num_embeddings=self.vocab_size, embedding_dim=self.vocab_size)

        # activations
        self.softmax = nn.Softmax(dim=1)

    def forward(self, features, captions):

        # features: extracted features from ResNet
        # captions: label of images

        batch_size = features.size(0)
        cnn_dim = features.size(1)

        hidden_state = torch.zeros((batch_size, self.hidden_size)).cuda()  # Initialize hidden state with zeros
        cell_state = torch.zeros((batch_size, self.hidden_size)).cuda()  # Initialize cell state with zeros

        # outputs = torch.empty((batch_size, captions.size(1), self.vocab_size)).cuda()
        outputs = torch.Tensor([]).cuda()
        captions_embed = self.embed(captions)

        '''Design LSTM model for captcha image recognition'''
        # Pass the caption word by word for each time step
        # It receives an input(x), makes an output(y), and receives this output as an input again recurrently
        '''Defined hidden state, cell state, outputs, embedded captions'''

        # can be designed to be word by word or character by character

        # for t in range(captions).size(1):
        for t in range(captions.size(1)):
            # for the first time step the input is the feature vector
            if t == 0:
                # probably have to get the output from the ResNet layer
                # use the LSTM cells in here i presume

                x = self.fc_in(features)
                # hidden_state, cell_state = self.lstm_cell(x[t], (hidden_state, cell_state))
                hidden_state, cell_state = self.lstm_cell(x, (hidden_state, cell_state))
                x = self.fc_out(hidden_state)
                # outputs.append(hidden_state)
                outputs = torch.cat([outputs, hidden_state])

            # for the 2nd+ time steps
            else:
                # hidden_state, cell_state = self.lstm_cell(x[t], (hidden_state, cell_state))
                hidden_state, cell_state = self.lstm_cell(x, (hidden_state, cell_state))
                x = self.fc_out(hidden_state)
                # outputs.append(hidden_state)
                outputs = torch.cat([outputs, hidden_state])

            # build the output tensor
            # outputs = torch.stack(outputs, dim=0)

        return outputs

m = LSTM(16, 32, 10)
m = m.cuda()
features = torch.randn((2, 16))
features = features.cuda()
captions = torch.randn((2, 10))
captions = torch.clip(captions, 0, 9)
captions = captions.long()
captions = captions.cuda()
m(features, captions)

This paper may help you somewhat: https://arxiv.org/abs/1904.01906


Instruction: Pytorch: How to get 2D data into a DataLoader?

Input:
I have a data set like this:
edge_origins = np.array([[0,1,2,3,4],[6,7,8]])
edge_destinations = np.array([[1,2,3,4,5],[7,8,9]])
target = np.array([0,1])
x = [[np.array([0.1,0.5,0.2]),np.array([0.5,0.6,0.23]),
     np.array([0.1,0.5,0.5]),np.array([0.1,0.6,0.23]),
     np.array([0.1,0.4,0.4]),np.array([0.52,0.6,0.23])],
     [np.array([0.1,0.3,0.3]),np.array([0.3,0.6,0.23]),
     np.array([0.1,0.1,0.2]),np.array([0.4,0.6,0.23])]]

This is a list of two networks. The first network has 6 nodes with 5 edges and a class 0, and then 4 nodes with 3 edges and class 1 networks.
I want to develop a model in Pytorch that will classify each network into it's class, and then i'll give it a new set of networks to classify.
So ultimately, I want to be able to shuffle these lists (simultaneously, i.e. maintaining the order between the data and the classes), split into train and test, and then read the train and test data into two data loaders, and feed these into a PyTorch network.
I wrote this:
edge_origins = np.array([[0,1,2,3,4],[6,7,8]])
edge_destinations = np.array([[1,2,3,4,5],[7,8,9]])
target = np.array([0,1])
x = [[np.array([0.1,0.5,0.2]),np.array([0.5,0.6,0.23]),
     np.array([0.1,0.5,0.5]),np.array([0.1,0.6,0.23]),
     np.array([0.1,0.4,0.4]),np.array([0.52,0.6,0.23])],
     [np.array([0.1,0.3,0.3]),np.array([0.3,0.6,0.23]),
     np.array([0.1,0.1,0.2]),np.array([0.4,0.6,0.23])]]

edge_index = torch.tensor([edge_origins, edge_destinations], dtype=torch.long)
dataset = Data(x=x, edge_index=edge_index, y=y, num_classes = len(set(target)))
print(dataset)

And the error is:
    edge_index = torch.tensor([edge_origins, edge_destinations], dtype=torch.long)
ValueError: expected sequence of length 5 at dim 2 (got 3)

But then once that is fixed I think the next step is:
torch.manual_seed(12345)
dataset = dataset.shuffle()

train_dataset = dataset[:1] #for toy example
test_dataset = dataset[1:]

print(f'Number of training graphs: {len(train_dataset)}')
print(f'Number of test graphs: {len(test_dataset)}')

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)

class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = GCNConv(dataset.num_node_features, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, hidden_channels)
        self.conv3 = GCNConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, dataset.num_classes)

    def forward(self, x, edge_index, batch):
        # 1. Obtain node embeddings
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = self.conv2(x, edge_index)
        x = x.relu()
        x = self.conv3(x, edge_index)

        # 2. Readout layer
        x = global_mean_pool(x, batch)  # [batch_size, hidden_channels]

        # 3. Apply a final classifier
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin(x)

        return x

model = GCN(hidden_channels=64)
print(model)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

def train():
    model.train()

    for data in train_loader:  # Iterate in batches over the training dataset.
         out = model(data.x, data.edge_index, data.batch)  # Perform a single forward pass.
         loss = criterion(out, data.y)  # Compute the loss.
         loss.backward()  # Derive gradients.
         optimizer.step()  # Update parameters based on gradients.
         optimizer.zero_grad()  # Clear gradients.

def test(loader):
     model.eval()

     correct = 0
     for data in loader:  # Iterate in batches over the training/test dataset.
         out = model(data.x, data.edge_index, data.batch)
         pred = out.argmax(dim=1)  # Use the class with highest probability.
         correct += int((pred == data.y).sum())  # Check against ground-truth labels.
     return correct / len(loader.dataset)  # Derive ratio of correct predictions.

for epoch in range(1, 171):
    train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Acc: {train_acc:.4f}, Test Acc: {test_acc:.4f}')

Could someone demonstrate to me how to get my data running into the Pytorch network above?

Output:
In Pytorch Geometric the Data object is used to contain only one graph. So you could iterate through all your arrays like so:
data_list = []
for i in range(2):
    edge_index_curr = torch.tensor([edge_origins[i],
                                    edge_destinations[i],
                                   dtype=torch.long)
    data = Data(x=torch.tensor(x[i]), edge_index=edge_index_curr, y=torch.tensor(target[i]))
    datas.append(data)

You can then use this list of Data to create your own Dataloader:
loader = DataLoader(data_list, batch_size=32)

If you need to split into train/val/test (I would advise having more than 2 samples for this case) you can do it manually or using sklearn.model_selection.
For data augmentation if you really do have very little data, pytorch-geometric comes with transforms.


Instruction: How does custom loss function in pyTorch work?

Input:
I see pytorch provides support to write custom loss functions. Consider following hinge loss.
class MarginRankingLossExp(nn.Module):
    def __init__(self) -> None:
        super(MarginRankingLossExp, self).__init__( )

    def forward(self,input1,input2,target):
        # loss_without_reduction = max(0, −target * (input1 − input2) + margin)
        neg_target = -target
        input_diff = input2-input1
        mul_target_input = neg_target*input_diff
        add_margin = mul_target_input
        zeros=torch.zeros_like(add_margin)
        loss = torch.max(add_margin, zeros)
        return loss.mean()

This has only forward and constructor function defined. How does pytorch calculate gradient for custom functions? Does it differentiate it somehow?
Also,  This function is non differentiable at y=margin but it didn't throw any error.

Output:
Your function will be differentiable by PyTorch's autograd as long as all the operators used in your function's logic are differentiable. That is, as long as you use torch.Tensor and built-in torch operators that implement a backward function, your custom function will be differentiable out of the box.
In a few words, on inference, a computational graph will be constructed on the fly. That is, for every operation you make, the tensors necessary to compute the gradients will be matched for a later backward pass. Assuming that you use only differentiable operators (i.e. most operators are mathematically differentiable and as such PyTorch provides the backward functionality for them). You will be able to perform backpropagation on the graph: from the end of it from the loss term, up to its leaves on parameters and inputs.
A very easy way to tell if your function is differentiable by Autograd is to infer its output with inputs which require gradient computation. Then check for a grad_fn callback on the output:
>>> x1 = torch.rand(1,10,2,2, requires_grad=True)
>>> x2 = torch.rand(1,10,2,2, requires_grad=True)
>>> y = torch.rand(1,10,2,2)

Here we can check with:
>>> MarginRankingLossExp()(x1, x2, y)
tensor(0.1045, grad_fn=<MeanBackward0>)

Where you notice MeanBackward0 which refers to torch.Tensor.mean, being the very last operator applied by MarginRankingLossExp.forward.


Instruction: How to convert a matrix to a list using List Comprehension

Input:
dataset[i][j] is a 100x30 matrix.
And I want to convert to a list using list comprehension.
I tried
train_dataset = [dataset[i][j] for i,j in [range(30), range(100)]]
but there was an error : ValueError: too many values to unpack (expected 2)
How could I assign the conversion, so len(train_dataset)=3000 ?

Output:
If I got this right, dataset is a matrix, with size 100x30, and you are trying to get a list from it?
If this is the case, you can do:
dataset = [[x for x in range(30)] for j in range(100)]
train_dataset = [dataset[i][j] for i in range(100) for j in range(30)]

print(train_dataset)
print(len(train_dataset))

dataset will be:
[0, ..., 29]
[0, ..., 29]
    x100
[0, ..., 29]

and your output will be:
[0, ..., 29, 0, ..., 29... x100]

resulting an array of size 3000.


Instruction: Why ab of L*a*b is divided by 110 to normalize it between -1 and 1?

Input:
This article tries to explain image colorizing strategy of pix2pix research.
This question references this article only.
In the dataset 'Making Dataset and DataLoaders' section, inside __getitem__ method, after converting the image to L*a*b, they divide L part of the tensor by 50 and subtract 1. This will surely bring the L values between -1 and 1 as L ranges from 0 to 100.
But ab values are divided by 110, which is strange, as ab values range from -128 to 128. As of my understanding, ab should be divided by 128 to bring its values between -1 and 1.
If anyone understands the logic behind this, please take the time to tell me.

Output:
Theoriticaly a and b are not bounded but often clamped for practical reason.
from wikipedia

The a* and b* axes are unbounded, and depending on the reference white
they can easily exceed ±150 to cover the human gamut. Nevertheless,
software implementations often clamp these values for practical
reasons. For instance, if integer math is being used it is common to
clamp a* and b* in the range of −128 to 127.

I think that the source of the 110 is this Matlab implementation
However, I assume that this DOESN'T hold for the skimage.color implemneation so it may be a mistake.
2: http://ai.stanford.edu/~ruzon/software/rgblab.html scikit.


Instruction: Reducing the dimensions of a 4D feature tensor from ResNet to fit into a 2D LSTM model

Input: I am designing a machine learning model that takes a feature tensor from ResNet and uses an LSTM to identify the sequences of letters in the image. The feature tensor that's from ResNet is 4-D , however, LSTM_cell wants inputs that are 2-D. I know about other methods such as .view() and .squeeze() that are able to reduce dimensions. However, it seems as if I do this, it changes the size of the dimensions of the feature vectors. At first the vector is [128, 2, 5, 512] but it needs to be [128, 512]. However, calling .view(-1,512) multiplies the dimensions to get [1280, 512]. How would you change dimensions without multiplying?

Output:
Outputs of CNN should be a 3-D Tensor (e.g. [128, x, 512]) so that it can be treated as a sequence. Then you can feed them into nn.LSTMCell() with an x-iteration for-loop.
However, 4-D Tensor remains some spatial features and it is not appropriate to be fed into LSTM. A typical practice is to redesign your CNN architecture to make sure that it produces a 3-D Tensor. For example, you can add an nn.Conv2d() or something else at the end of CNN network to make the outputs as shape [128, x, 512].


Instruction: Pytorch uses cuda despite disabling it. How to force it to use CPU-only

Input:
I tried disabling cuda for pytorch following this stackoverflow question and a few others.
At OS level, before initializing python -> set CUDA_VISIBLE_DEVICES ''
But when I enter the python prompt, I still see Cuda is available
>>> import torch
>>> torch.cuda.is_available()
True

A subsequent operation confirms it.
>>> a = torch.ones(5)
>>> b = a
>>> a.add_(1)
tensor([2., 2., 2., 2., 2.])
>>> print(b)
tensor([2., 2., 2., 2., 2.])

I can see that b also got updated. How do I force pytorch not to use GPU ?

Output:
You could use a.to("cpu") to move a tensor to a particular device, in this case CPU. Then a.is_cuda() should return False, confirming that it is not on GPU.
The same can be done with entire models, etc., e.g. model.to("cpu")


Instruction: How to detect objects with a custom YOLOv5 model?

Input:
I trained a YOLOv5 model from a custom dataset with the provided training routine on github (from inside tutorial.ipynb).
Using this model for detecting objects in unseen images gets me decent results when executing:
!python detect.py --weights custom_weights.pt --img 224 --conf 0.5 --source data/images

Now I want to use my model in a small project. Using the following approach does not lead to good results. It either detects complete nonsense or nothing at all on the same images used above.
model = torch.hub.load('ultralytics/yolov5', 'custom', path='custom_weights.pt', force_reload=True)
img = cv2.imread('test.jpg')
model.eval()
pred = model(img)
bboxes = pred.xyxy

Am I forced to use detect.py and hence cloning the whole YOLO repository into my project?
Or is there something I am missing when calling the model like I do right now?

Output:
Seems like calling the model with a numpy array is not working, at least not straightforward. When the image is transformed to a torch.tensor, you can apply non maximum supression (from yolov5/utils/general.py) to the output and receive a valid result:
model = torch.hub.load('ultralytics/yolov5', 'custom', path='custom_weights.pt', force_reload=True)
img = cv2.imread('image.jpg')
img = torch.from_numpy(img)/255.0
img = img.unsqueeze(0)
img = torch.permute(img, (0, 3, 1, 2))
pred = model(img)
pred = non_max_suppression(pred)