Instruction: How to calculate f1 score during evaluation on test set?

Input:
I am trying to calculate the f1 score during evaluation of my own test set but i'm not able to solve as I am very unexperienced. I've tried to use both f1 score from Scikit-Learn and from torchmetrics but they give me everytime different errors.
This is my code:
# Function to test the model
from sklearn.metrics import f1_score

since = time.time()
total=0
correct=0
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
y_pred=[]
y_true=[]

# Iterate over data.
with torch.no_grad():
    for inputs, labels in dataloadersTest_dict['Test']:
        inputs = inputs.to(device)
        labels = labels.to(device)
        #outputs = model(inputs)
        predicted_outputs = model(inputs)
        _, predicted = torch.max(predicted_outputs, 1)
        total += labels.size(0)
        print(total)
        correct += (predicted == labels).sum().item()
        print(correct)
        #f1 score
        temp_true=labels.numpy()
        temp_pred=predicted.numpy()
        y_true.append(temp_true.tolist())
        y_pred.append(temp_pred.tolist())

time_elapsed = time.time() - since
test_acc=100 * correct / total

print('Evaluation completed in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
print('Accuracy: %d %%' % (test_acc))
print('F1 Score:')
f1=f1_score(y_true,y_pred, average = 'macro')
print(f1)

Output:
The error trace should be available in order to spot the problem but I guess the problem is due to passing a nested list to f1_score instead of a single list. It must be fixed by changing the collecting strategy of the final lists.
# Iterate over data.
y_true, y_pred = [], []
with torch.no_grad():
    for inputs, labels in dataloadersTest_dict['Test']:
        inputs = inputs.to(device)
        labels = labels.to(device)
        #outputs = model(inputs)
        predicted_outputs = model(inputs)
        _, predicted = torch.max(predicted_outputs, 1)
        total += labels.size(0)
        print(total)
        correct += (predicted == labels).sum().item()
        print(correct)
        #f1 score
        temp_true=labels.numpy()
        temp_pred=predicted.numpy()
        y_true+=temp_true.tolist()
        y_pred+=temp_pred.tolist()


Instruction: Save a model on gpu and load it on CPU

Input:
I have a model which I train based on the torch in GPU.
Now, i want to upload it on cpu.
I am using this code to save an load the model.
Here is my model and the training phase:
model              = VAE(input_size ,  lead,     hidden_dim,hidden_dim1,hidden_dimd,  latent_dim, device,  num_hidden= lead).to(device)

optimizer          = torch.optim.Adam(model.parameters(), lr =learning_rate )

losses, kl_loss, l_loss, validate_loss = trainv(model, device, epochs, train_iterator, optimizer, validate_iterator)

model = VAE()
torch.save(model.state_dict(), "model.pt")

#load
device = torch.device('cpu')
model = VAE()
model.load_state_dict(torch.load(PATH, map_location=device))

Here is the error:
TypeError: __init__() missing 8 required positional arguments:

Output:
Did you define your model class before torch.save() ? This code works without errors in Google Collab:
import torch
import torch.nn as nn

class TheModelClass(nn.Module):
    def __init__(self):
        super(TheModelClass, self).__init__()

        self.linear = nn.Linear(125, 1)

    def forward(self, src):

        output = self.linear(src)

        return

model = TheModelClass()

torch.save(model.state_dict(), "model.pt") # you need to define your model before
device = torch.device('cpu')
model = TheModelClass() # you even don't need to redefine your model
model.load_state_dict(torch.load('/content/model.pt', map_location=device))


Instruction: How to extract a specific digit from the MNIST dataset with dataloader?

Input:
I am feeding the MNIST dataset to train my neural network in the following manner
indices = torch.arange(60000)
dataset = datasets.MNIST(root="dataset/", transform=transforms, download=True)
datasetsmall = data_utils.Subset(dataset, indices)
loader = DataLoader(datasetsmall, batch_size=batch_size, shuffle=True)

However, since the training is taking huge time to complete I have decided to train the model with only a specific digit from the MNIST dataset, for example the digit 4. How can I just extract the digit 4 and feed it to my neural network in the same way. The loop to train the neural network is like
for batch_idx, (real, _) in enumerate(loader):

Now I want only the digit 4 in the loader. How should I proceed in that case?

Output:
Does this code solve your problem?
import torch
from torchvision import datasets
from torch.utils.data import TensorDataset, DataLoader
from torchvision.transforms import ToTensor

cls = 4 # needed class
batch_size = 32

dataset = datasets.MNIST(root="dataset/", download=True, transform=ToTensor())
dataset = list(filter(lambda i: i[1] == cls, dataset))
loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

s = 0
for i in loader:
  s += 1

print(f'We\'ve got {s} batches with batch_size {batch_size} only for class {cls}')

# print(i) # uncomment this line if you want to examine last batch by yourself

Result:
We've got 183 batches with batch_size 32 only for class 4


Instruction: Question regarding latent space in Autoencoder

Input:
I am just start learning AE few days ago. From what I know about AE, a latent space will be created after the encoder and then the decoder will regenerate images based on the latent spaces. In my other project, it requires me to embed some new feature into the AE latent space. Below are the AE code that I have try.
AE module
# build autoencoder
import torch
import matplotlib.pyplot as plt

# Creating a PyTorch class
# 28*28 ==> 9 ==> 28*28
class AE(torch.nn.Module):
    def __init__(self):
        super().__init__()

        # Building an linear encoder with Linear
        # layer followed by Relu activation function
        # 784 ==> 9
        self.encoder = torch.nn.Sequential(
            torch.nn.Linear(64 * 64, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 32),
            torch.nn.ReLU(),
            torch.nn.Linear(32, 3)
            )

        # Building an linear decoder with Linear
        # layer followed by Relu activation function
        # The Sigmoid activation function
        # outputs the value between 0 and 1
        # 9 ==> 784
        self.decoder = torch.nn.Sequential(
            torch.nn.Linear(3,32),
            torch.nn.ReLU(),
            torch.nn.Linear(32, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 64*64),
            torch.nn.Sigmoid()
            )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

Init
# Model Initialization
model = AE().to(device)

# Validation using MSE Loss function
loss_function = torch.nn.MSELoss()

# Using an Adam Optimizer with lr = 0.1
optimizer = torch.optim.Adam(model.parameters(),
                             lr = 1e-1,
                             weight_decay = 1e-8)

training
num_epochs = 100
output =[]
for epoch in range(num_epochs):
    for data in loader:
        img, _ = data
        img = img.reshape(-1,64*64)
        img    = img.to(device)
        recon = model(img)
        loss   = loss_function(recon, img.data)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'epoch [{epoch + 1}/{num_epochs}], loss:{loss. Item(): .4f}')
    output. Append((epoch,img,recon))

My question is may I know how can I obtain the latent space? From what I know about the code there is only encoder and decoder. How can I retrieve the latent space so that I can embed new feature to it? Thank you.

Output:
You can simply modify your forward(self, x) function to also return the laten space embedding generated by the encoder:
def forward(self, x):
    encoded = self.encoder(x)
    decoded = self.decoder(encoded)
    return encoded, decoded

In your train loop you can just add:
embedding, recon = model(img)


Instruction: tf.bitcast equivalent in pytorch?

Input:
This question is different from tf.cast equivalent in pytorch?.
bitcast do bitwise reinterpretation(like reinterpret_cast in C++) instead of "safe" type conversion.
This operation is useful when you want to store bfloat16 tensor with numpy.
x = torch.ones(224, 224, 3, dtype=torch.bfloat16
x_np = bitcast(x, torch.uint8).numpy()

Currently numpy doesn't natively support bfloat16, so x.numpy() will raise TypeError: Got unsupported ScalarType BFloat16

Output:
Use the 2nd overload torch.Tensor.view.
Its semantic is closely similar to numpy.ndarray.view.


Instruction: Are torch.nn.ReLU and torch.nn.Sigmoid trainable?

Input:
I build a simple GRU model with PyTorch. It includes 4 sub-modules.
I noted that some dictionaries return by the state_dict() of them are empty after training, while ones of the other sub-modules certainly have some weights and bias.
The code:
class GruModel(nn.Module):
    def __init__(self, inputs, nodes, layers=2):
        super(GruModel, self).__init__()
        self.gru_m = nn.GRU(input_size=inputs, num_layers=layers, hidden_size=nodes,
                            batch_first=True, dropout=0.5)
        self.activt_f = nn.ReLU()
        self.output_f = nn.Linear(nodes, 1)
        self.probab_f = nn.Sigmoid()

    def forward(self, x, h):
        o, h = self.gru_m(x, h)
        o = self.activt_f(o[:, -1])
        out = self.output_f(o)
        return self.probab_f(out)

    def trainWith(self, ...):
        ''' training body '''
        criterion = nn.BCEWithLogitsLoss()
        optimizer = torch.optim.Adadelta(self.parameters(), lr=learn_rat)
        lr_schdlr = torch.optim.lr_scheduler.ExponentialLR(
            optimizer, gamma=0.99)
        t_loader = torch_data.DataLoader(...)
        for e in range(epochs):
            for x, p_label, n_label in t_loader:
                optimizer.zero_grad()
                out = self(x, self.init_hidden(batch_size))
                loss = criterion(out, p_label)
                loss.backward()
                optimizer.step()
            lr_schdlr.step()

    def save(self, full_path: str):
        print(self.gru_m.state_dict())
        print(self.activt_f.state_dict())
        print(self.output_f.state_dict())
        print(self.probab_f.state_dict())

In real running, the state_dict of the sub-module self.gru_m and self.output_f
have values as expected, but the ones of the sub-module self.activt_f (nn.ReLU) and self.probab_f (nn.Sigmoid) have nothing.
Don't care about my training process, I feed it with tons of data and run through hundreds of epochs, and the model can do classification as I expected.
I'm interested in whether the later two modules are trainable, or they  do NOT need any weights and bias with them to do their work?
If so, can we say that the torch.nn.Sigmoid is same as torch.nn.functional.sigmoid? Because they are all dummy functions, not stateful objects.

Output:
The two layer modules you are mentioning are activation functions which are not parametrized. This means they are not "trainable" since they don't hold any parameters.
However, nn modules are classes (they can be stateful) while nn.functional utilities are functions (they are not stateful).


Instruction: How to evaluate a pyTorch/DGL tensor

Input:
From a DGL graph I want to see the adjacency matrix with
adjM = g.adjacency_matrix()
adjM

and I get the following which is fine:
tensor(indices=tensor([[0, 0, 0, 1],
                       [1, 2, 3, 3]]),
       values=tensor([1., 1., 1., 1.]),
       size=(4, 4), nnz=4, layout=torch.sparse_coo)

Now I want to have the adjacency matrix and the node values each by itself. I imagine something of this kind:
adjMatrix = adjM.indices   # or
adjMatrix = adjM[0]
nodeValues = adjM.values   # or
nodeValues = adjM[1]

But this form is not estimated by pyTorch/DGL.
My beginner's question:

how to do this correctly and sucsessfully? and
is there a tutorial for a nuby? ( I  have  searched a lot just for this detail...!)

Output:
Click here!
You will find the usage of dgl.adj().  As the doc said, the return is an adjacency matrix, and the return type is the SparseTensor.
I noticed that the output that you post is a SparseTensor.
You can try it as follows then you can get the entire adj_matrix
I create a dgl graph g, get the adjacency matrix as adj
g = dgl.graph(([0, 1, 2], [1, 2, 3]))
adj = g.adj()
adj

output is:
tensor(indices=tensor([[0, 1, 2],
                       [1, 2, 3]]),
       values=tensor([1., 1., 1.]),
       size=(4, 4), nnz=3, layout=torch.sparse_coo)

We can find that adj is the presence of sparse, and the sparse type is coo, we can use the following code to verify if adj is a SparseTensor
adj.is_sparse

output :
True

so we can use to_dense() get the original adj matrix
adj.to_dense()

the result is:
tensor([[0., 1., 0., 0.],
        [0., 0., 1., 0.],
        [0., 0., 0., 1.],
        [0., 0., 0., 0.]])

When you have a problem with DGL you can check the Deep Graph Library Tutorials and Documentation.


Instruction: Polynomial features using numpy or torch

Input:
Having tensor
[a, b]
I want to create a tensor of the form
[a, b, ab, a^2, b^2]

or even of higher order
[a, b, ab, a^2, b^2, (a^2)b, a(b^2), (a^2)(b^2), a^3, b^3]

I want to solve the issue in a short time. I can solve it with loops, but that's not the way I really would like to do that. However dynamic programming works for me, so using 2nd order to compute 3rd order is fine.
The final solution will be implemented in PyTorch, but NumPy implementation would be useful, I can port it to PyTorch on my own.
Edit:
As you have asked, I'm posting my attempt, which I'm not very proud of:
def polynomial(t: torch.Tensor) -> torch.Tensor:
    r = t.clone()
    r_c = torch.empty((t.shape[0], math.comb(t.shape[1], 2) + t.shape[1]))
    i = 0
    for idx in range(t.shape[1]):
        for jdx in range(idx, t.shape[1]):
            r_c[:, i] = (r[:, idx].unsqueeze(-1) * r[:, jdx].unsqueeze(-1)).squeeze(-1)
            i += 1
    r = torch.hstack([r, r_c])
    return r

For
t = torch.tensor([
        [1, 2, 3],
        [3, 4, 5],
        [5, 6, 7]
    ])
polynomial(t)

results in
tensor([[ 1.,  2.,  3.,  1.,  2.,  3.,  4.,  6.,  9.],
        [ 3.,  4.,  5.,  9., 12., 15., 16., 20., 25.],
        [ 5.,  6.,  7., 25., 30., 35., 36., 42., 49.]])

Output:
For anyone who will meet the problem:
def polynomial(t: torch.Tensor, degree: int = 2, interaction_only: bool = False) -> torch.Tensor:
    cols = t.hsplit(t.shape[1])
    if interaction_only:
        degree = 2
        combs = combinations(cols, degree)
    else:
        combs = combinations_with_replacement(cols, degree)
    prods = [torch.prod(torch.hstack(comb), -1).unsqueeze(-1) for comb in combs]
    r = torch.hstack(prods)
    return torch.hstack((t, r)) if degree == 2 else torch.hstack((polynomial(t, degree - 1), r))


Instruction: nn.LSTM doesn't seem to learn anything or not updating properly

Input:
I was trying out a simple LSTM use case form pytorch, with the following model.
class SimpleLSTM(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim):
        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
        self.lstm = nn.LSTM(batch_first=True, input_size=embedding_dim, num_layers=1, hidden_size=hidden_dim, bidirectional=True)
        self.linear = nn.Linear(hidden_dim*2, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):   # NxD, padded to same length with 0s in N-sized batch
        x = self.embedding(x)
        output, (final_hidden_state, final_cell_state) = self.lstm(x)
        x = self.linear(output[:,-1,:])
        x=self.sigmoid(x)
        return x

It is a binary classification, with BCELoss (combined with the Sigmoid output layer). Unfortunately, loss is stuck at 0.6969 (i.e. it is not learning anything).
I've tried using final_hidden_state, output[:,0,:] feeding into the linear layer, but so far no dice.
Everything else (optimizer, loss criterion, train loop, val loop) already works because I tried the exact same setup with a basic NN using nn.Embedding, nn.Linear, and nn.Sigmoid only, and could get to good loss decrease and high accuracy. In the SimpleLSTM, the only thing I added is the nn.LSTM.

Output:
Typically final_hidden_state is passed to linear, not output. Use it.
add 1-2 more linear layers after the LSTM.
try lower LR, especially when embeddings are not pre-trained.
Better yet, try loading pre-trained embeddings.


Instruction: Construct a 3D tensor from a 2D matrix

Input:
Given an n-by-n matrix A, where each row of A is a permutation of [n], e.g.,
import torch
n = 100
AA = torch.rand(n, n)
A = torch.argsort(AA, dim=1)

Also given another n-by-n matrix P, we want to construct a 3D tensor Q s.t.
Q[i, j, k] = P[A[i, j], k]

Is there any efficient way in pytorch?
I am aware of torch.gather but it seems hard to be directly applied here.

Output:
You can directly use:
Q = P[A]


Instruction: Pytorch submodules output shape

Input:
How does the output shape of submodules in pytorch is determined? why is the output shape of a certain sub-module is modified in the code below?
When I separate the head of a classical classifier from its backbone in the following way:
import torch, torchvision
from torchsummary import summary

effnet = torchvision.models.efficientnet_b0(num_classes = 2)

backbone = torch.nn.Sequential(*(list(effnet.children())[0]))
adaptive_pool = list(effnet.children())[1]
head = list(effnet.children())[2]

model = torch.nn.Sequential(*[backbone, adaptive_pool, head])
summary(model, (3,256,256), device = 'cpu') # <== Error

I get the following error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2560x1 and 1280x2)
This error is due to modified output shape of the sub-module adaptive_pool. To workaround this problem, flatten can be used as follows:
class flatten(torch.nn.Module):
    def forward(self, input):
        return input.view(input.size(0), -1)

model = torch.nn.Sequential(*[backbone, adaptive_pool,  flatten(), head])
summary(model, (3,256,256), device = 'cpu')

Why is the output shape of the sub-module adaptive_pool is modified?

Output:
The output of an nn.AdaptiveAvgPool2d is 4D even if the average is computed globally i.e output_size=1. In other words, the output shape of your global pooling layer is (N, C, 1, 1). This means you indeed need to flatten it for the layer which is fully connected.
In the referenced original efficient net classification network, the implementation of the flattening operation is done directly in the forward logic without the use of a dedicated layer. See this line.
Instead of implementing your own flattening layer, you can use the built-in nn.Flatten. More details about this module can be found here.
>>> model = nn.Sequential(backbone, adaptive_pool, nn.Flatten(1), head)


Instruction: Map a list of list to TPU tensor

Input:
I have this code to deconstruct a list of list to different tensors.
token_a_index, token_b_index, isNext, input_ids, segment_ids, masked_tokens, masked_pos = map(torch.LongTensor, zip(*batch))

If I want to create these tensors on gpu, I can use below code:
token_a_index, token_b_index, isNext, input_ids, segment_ids, masked_tokens, masked_pos = map(torch.cuda.LongTensor, zip(*batch))

But now I want to create all these on TPU, what should I do? Is there anything like below?
token_a_index, token_b_index, isNext, input_ids, segment_ids, masked_tokens, masked_pos = map(torch.xla.LongTensor, zip(*batch))

Output:
You can use xla device following the guide here.
You can select the device and pass it to your function like this:
import torch_xla.core.xla_model as xm
device = xm.xla_device()
token_a_index, token_b_index, isNext, input_ids, segment_ids, masked_tokens, masked_pos = map(lambda x: torch.Tensor(x).to(device).long(), zip(*batch))

You can even parametrize the device variable, torch.device("cuda" if torch.cuda.is_available() else "cpu") can be used to select between cuda and cpu.


Instruction: How to concatenate feature maps?

Input:
I was trying to merge feature maps that I got from two different encoders which have different dimensions in Pytorch
simclr_features: torch.Size([543, 512])
imagenet_features: torch.Size([543, 1024])
so I wanted: torch.Size([543, 1536]). what is a possible solution to do that?

Output:
You can use torch.cat:
torch.cat((simclr_features, imagenet_features), dim=1)


Instruction: Distributed Data Parallel (DDP) Batch size

Input:
Suppose, I use 2 gpus in a DDP setting.
So, if I intend to use 16 as a batch size if I run the experiment on a single gpu,
should I give 8 as a batch size, or 16 as a batch size in case of using 2 gpus with DDP setting??
Does 16 is divided into 8 and 8 automatically?
Thank you -!

Output:
As explained here:

the application of the given module by splitting the input across the specified devices
The batch size should be larger than the number of GPUs used locally
each replica handles a portion of the input

If you use 16 as batch-size, it will be divided automatically between the two gpus.


Instruction: How can I get the maximum values of a tensor along a dimension?

Input:
I have a 3D tensor and would like to take the max values along the 0th dimension in Libtorch.
I know how to do this in Python (PyTorch) but I'm having trouble doing this in LibTorch.
In LibTorch my code is
auto target_q_T = torch::rand({5, 10, 1});
auto max_q = torch::max({target_q_T}, 0);
std::cout << max_q;

It returns this long, repeating error.
note: candidate: ‘template<class _Traits> std::basic_ostream<char, _Traits>& std::operator<<(std::basic_ostream<char, _Traits>&, const char*)’
  611 |     operator<<(basic_ostream<char, _Traits>& __out, const char* __s)
      |     ^~~~~~~~
/usr/include/c++/11/ostream:611:5: note:   template argument deduction/substitution failed:
/home/iii/tor/m_gym/multiv_normal.cpp:432:18: note:   cannot convert ‘max_q’ (type ‘std::tuple<at::Tensor, at::Tensor>’) to type ‘const char*’
  432 |     std::cout << max_q;
      |                  ^~~~~
In file included from /usr/include/c++/11/istream:39,
                 from /usr/include/c++/11/sstream:38,
                 from /home/iii/tor/m_gym/libtorch/include/c10/macros/Macros.h:246,
                 from /home/iii/tor/m_gym/libtorch/include/c10/core/DeviceType.h:8,
                 from /home/iii/tor/m_gym/libtorch/include/c10/core/Device.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/ATen/core/TensorBody.h:11,
                 from /home/iii/tor/m_gym/libtorch/include/ATen/core/Tensor.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/ATen/Tensor.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/function_hook.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/cpp_hook.h:2,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/variable.h:6,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/autograd/autograd.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/api/include/torch/autograd.h:3,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/iii/tor/m_gym/libtorch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /home/iii/tor/m_gym/multiv_normal.cpp:2:
/usr/include/c++/11/ostream:624:5: note: candidate: ‘template<class _Traits> std::basic_ostream<char, _Traits>& std::operator<<(std::basic_ostream<char, _Traits>&, const signed char*)’
  624 |     operator<<(basic_ostream<char, _Traits>& __out, const signed char* __s)
      |     ^~~~~~~~

This is how it works in Python.
target_q_np = torch.rand(5, 10, 1)
max_q = torch.max(target_q_np, 0)
max_q

torch.return_types.max(
values=tensor([[0.8517],
        [0.7526],
        [0.6546],
        [0.9913],
        [0.8521],
        [0.9757],
        [0.9080],
        [0.9376],
        [0.9901],
        [0.7445]]),
indices=tensor([[4],
        [2],
        [3],
        [4],
        [1],
        [0],
        [2],
        [4],
        [4],
        [4]]))

Output:
If you read the compiler error, it basically tells you that you are trying to print a tuple of two tensors. That's because the C++ code works exactly like the python code and returns the max values and their respective indices (your python code prints exactly that).
You need std get to extract the tensors from the tuple :
auto target_q_T = torch::rand({5, 10, 1});
auto max_q = torch::max({target_q_T}, 0);
std::cout << "max: " << std::get<0>(max_q)
          << "indices: " << std::get<1>(max_q)
          << std::endl;

In C++17 you should also be able to write
auto [max_t, idx_t] = torch::max({target_q_T}, 0);
std::cout << ... ;


Instruction: Changing 3D Convolutional Encoder layers to 3D Deconvolutional layers

Input:
I have a 3D convolutional network that is composed of an encoder and a decoder. In the decoder part, I want to use the type of convolutional layers that are used in the decoder part but making them to do deconvolution. In other words, I want that the mentioned convolutional layers increase the spatial size as like as the converse rate that is used in the encoder while decrease the number of channels.
The convolutional layer that I use inside my encoder contains sequential layers:
Conv_layer = nn.Sequential(
            BasicConv3d(64, 64, kernel_size=1, stride=1),
            SepConv3d(64, 192, kernel_size=3, stride=1, padding=1),
            nn.MaxPool3d(kernel_size=(1,3,3), stride=(1,2,2), padding=(0,1,1)),
        )

The type of convolutional layers that are used:
    class BasicConv3d(nn.Module):
        def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):
        super(BasicConv3d, self).__init__()
        self.conv = nn.Conv3d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)
        self.bn = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x

 class SepConv3d(nn.Module):
        def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):
        super(SepConv3d, self).__init__()
        self.conv_s = nn.Conv3d(in_planes, out_planes, kernel_size=(1,kernel_size,kernel_size), stride=(1,stride,stride), padding=(0,padding,padding), bias=False)
        self.bn_s = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu_s = nn.ReLU()

        self.conv_t = nn.Conv3d(out_planes, out_planes, kernel_size=(kernel_size,1,1), stride=(stride,1,1), padding=(padding,0,0), bias=False)
        self.bn_t = nn.BatchNorm3d(out_planes, eps=1e-3, momentum=0.001, affine=True)
        self.relu_t = nn.ReLU()

    def forward(self, x):
        x = self.conv_s(x)
        x = self.bn_s(x)
        x = self.relu_s(x)

        x = self.conv_t(x)
        x = self.bn_t(x)
        x = self.relu_t(x)
        return x

My question is that how I should chnge the kernel_size, stride and padding that change the mentioned layers to deconvolution that incease the spatial size of feature maps as the same with the converse rate of convolutional layers.

Output:
Basically, you want a nn.ConvTranspose3d with kernel size (3, 3, 3) and stride (1, 2, 2). You can see the formula relating input size to output size here:
D_out=(D_in−1)×stride − 2×padding + dilation×(kernel_size−1) + output_padding + 1

In your case, kernel_size=3, dilation=1 and stride is 1 for the temporal dimension and 2 for the spatial dimensions.
Thus the desired layers would be something like:
out_planes = 192
in_planes = 64
deconv_layer = nn.Sequential(
  nn.ConvTranspose3d(out_planes, out_planes, kernel_size=(3, 1, 1), stride=1, padding=0, output_padding=0)
  nn.BatchNorm3d(out_planes),
  nn.ReLU(inplace=True),
  nn.ConvTranspose3d(out_planes, in_planes, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), output_padding=(0, 1, 1))
  nn.BatchNorm3d(in_planes),
  nn.ReLU(inplace=True),
  BasicConv3d(in_planes, in_planes, kernel_size=1, stride=1),  # to be consistent with the Conv_layers structure
)


Instruction: Why don't the images align when concatenating two data sets in pytorch using torch.utils.data.ConcatDataset?

Input:
I wanted to concatenate multiple data sets where the labels are disjoint (so don't share labels). I did:
class ConcatDataset(Dataset):
    """

    ref: https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12
    """

    def __init__(self, datasets: list[Dataset]):
        """
        """
        # I think concat is better than passing data to a self.data = x obj since concat likely using the getitem method of the passed dataset and thus if the passed dataset doesnt put all the data in memory concat won't either
        self.concat_datasets = torch.utils.data.ConcatDataset(datasets)
        # maps a class label to a list of sample indices with that label.
        self.labels_to_indices = defaultdict(list)
        # maps a sample index to its corresponding class label.
        self.indices_to_labels = defaultdict(None)
        # - do the relabeling
        offset: int = 0
        new_idx: int = 0
        for dataset_idx, dataset in enumerate(datasets):
            assert len(dataset) == len(self.concat_datasets.datasets[dataset_idx])
            assert dataset == self.concat_datasets.datasets[dataset_idx]
            for x, y in dataset:
                y = int(y)
                _x, _y = self.concat_datasets[new_idx]
                _y = int(_y)
                # assert y == _y
                assert torch.equal(x, _x)
                new_label = y + offset
                self.indices_to_labels[new_idx] = new_label
                self.labels_to_indices[new_label] = new_idx
            num_labels_for_current_dataset: int = max([y for _, y in dataset])
            offset += num_labels_for_current_dataset
            new_idx += 1
        assert len(self.indices_to_labels.keys()) == len(self.concat_datasets)
        # contains the list of labels from 0 - total num labels after concat
        self.labels = range(offset)
        self.target_transform = lambda data: torch.tensor(data, dtype=torch.int)

    def __len__(self):
        return len(self.concat_datasets)

    def __getitem__(self, idx: int) -> tuple[Tensor, Tensor]:
        x = self.concat_datasets[idx]
        y = self.indices_to_labels[idx]
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y

but it doesn't even work to align the x images (so never mind if my relabling works!). Why?
def check_xs_align_cifar100():
    from pathlib import Path

    root = Path("~/data/").expanduser()
    # root = Path(".").expanduser()
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
    test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)

    concat = ConcatDataset([train, test])
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')

error
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 405, in <module>
    check_xs_align()
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 391, in check_xs_align
    concat = ConcatDataset([train, test])
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 71, in __init__
    assert torch.equal(x, _x)
TypeError: equal(): argument 'input' (position 1) must be Tensor, not Image
python-BaseException

Bonus: let me know if relabeling is correct please.
related discussion: https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12

Edit 1: PIL comparison fails
I did a PIL image comparison according to Compare images Python PIL but it failed:
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 419, in <module>
    check_xs_align_cifar100()
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 405, in check_xs_align_cifar100
    concat = ConcatDataset([train, test])
  File "/Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py", line 78, in __init__
    assert diff.getbbox(), f'comparison of imgs failed: {diff.getbbox()=}'
AssertionError: comparison of imgs failed: diff.getbbox()=None
python-BaseException
diff
PyDev console: starting.
<PIL.Image.Image image mode=RGB size=32x32 at 0x7FBE897A21C0>

code comparison:
                diff = ImageChops.difference(x, _x)  # https://stackoverflow.com/questions/35176639/compare-images-python-pil
                assert diff.getbbox(), f'comparison of imgs failed: {diff.getbbox()=}'

this also failed:
    assert list(x.getdata()) == list(_x.getdata()), f'\n{list(x.getdata())=}, \n{list(_x.getdata())=}'
AssertionError: ...long msg...

assert statement was:
                assert list(x.getdata()) == list(_x.getdata()), f'\n{list(x.getdata())=}, \n{list(_x.getdata())=}'

Edit 2: Tensor comparison Fails
I tried to convert images to tensors but it still fails:
AssertionError: Error for some reason, got: data_idx=1, x.norm()=tensor(45.9401), _x.norm()=tensor(33.9407), x=tensor([[[1.0000, 0.9922, 0.9922,  ..., 0.9922, 0.9922, 1.0000],

code:
class ConcatDataset(Dataset):
    """
    ref:
        - https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12
        - https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor
    """

    def __init__(self, datasets: list[Dataset]):
        """
        """
        # I think concat is better than passing data to a self.data = x obj since concat likely using the getitem method of the passed dataset and thus if the passed dataset doesnt put all the data in memory concat won't either
        self.concat_datasets = torch.utils.data.ConcatDataset(datasets)
        # maps a class label to a list of sample indices with that label.
        self.labels_to_indices = defaultdict(list)
        # maps a sample index to its corresponding class label.
        self.indices_to_labels = defaultdict(None)
        # - do the relabeling
        img2tensor: Callable = torchvision.transforms.ToTensor()
        offset: int = 0
        new_idx: int = 0
        for dataset_idx, dataset in enumerate(datasets):
            assert len(dataset) == len(self.concat_datasets.datasets[dataset_idx])
            assert dataset == self.concat_datasets.datasets[dataset_idx]
            for data_idx, (x, y) in enumerate(dataset):
                y = int(y)
                # - get data point from concataned data set (to compare with the data point from the data set list)
                _x, _y = self.concat_datasets[new_idx]
                _y = int(_y)
                # - sanity check concatanted data set aligns with the list of datasets
                # assert y == _y
                # from PIL import ImageChops
                # diff = ImageChops.difference(x, _x)  # https://stackoverflow.com/questions/35176639/compare-images-python-pil
                # assert diff.getbbox(), f'comparison of imgs failed: {diff.getbbox()=}'
                # assert list(x.getdata()) == list(_x.getdata()), f'\n{list(x.getdata())=}, \n{list(_x.getdata())=}'
                # tensor comparison
                x, _x = img2tensor(x), img2tensor(_x)
                print(f'{data_idx=}, {x.norm()=}, {_x.norm()=}')
                assert torch.equal(x, _x), f'Error for some reason, got: {data_idx=}, {x.norm()=}, {_x.norm()=}, {x=}, {_x=}'
                # - relabling
                new_label = y + offset
                self.indices_to_labels[new_idx] = new_label
                self.labels_to_indices[new_label] = new_idx
            num_labels_for_current_dataset: int = max([y for _, y in dataset])
            offset += num_labels_for_current_dataset
            new_idx += 1
        assert len(self.indices_to_labels.keys()) == len(self.concat_datasets)
        # contains the list of labels from 0 - total num labels after concat
        self.labels = range(offset)
        self.target_transform = lambda data: torch.tensor(data, dtype=torch.int)

    def __len__(self):
        return len(self.concat_datasets)

    def __getitem__(self, idx: int) -> tuple[Tensor, Tensor]:
        x = self.concat_datasets[idx]
        y = self.indices_to_labels[idx]
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y

Edit 3, clarification request:
My vision of the data set I want is a concatenation of a data sets in question -- where relabeling starting the first label commences. The curicial thing (according to me -- might be wrong on this) is that once concatenated we should verify in some way that the data set indeed behaves the way we want it. One check I thought is to index the data point from the list of data sets and also from the concatenation object of the data set. If the data set was correctly conatenated I'd expect the images to be correspond according to this indexing. So if the first image in the first data set had some unique identifier (e.g. the pixels) then the concatenation of the data sets should have the first image be the same as the first image in the list of data sets and so on...if this doesn't hold, if I start creating new labels -- how do I know I am even doing this correctly?

reddit link: https://www.reddit.com/r/pytorch/comments/xurnu9/why_dont_the_images_align_when_concatenating_two/
cross posted pytorch discuss: https://discuss.pytorch.org/t/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-torch-utils-data-concatdataset/162801?u=brando_miranda

Output:
Corrected code can be found here https://github.com/brando90/ultimate-utils/blob/master/ultimate-utils-proj-src/uutils/torch_uu/dataset/concate_dataset.py you can pip install the library pip install ultimate-utils.
Since only links is not a good way to answer I will copy paste the code too with it's test and expected output:
"""

do checks, loop through all data points, create counts for each label how many data points there are
do this for MI only

then check union and ur implementation?
compare the mappings of one & the other?

actually it's easy, just add the cummulative offset and that's it. :D the indices are already -1 indexed.

assert every image has a label between 0 --> n1+n2+... and every bin for each class is none empty

for it to work with any standard pytorch data set I think the workflow would be:

pytorch dataset -> l2l meta data set -> union data set -> .dataset field -> data loader
for l2l data sets:

l2l meta data set -> union data set -> .dataset field -> data loader
but the last one might need to make sure .indices or .labels is created or a get labels function that checking the attribute
gets the right .labels or remaps it correctly
"""
from collections import defaultdict
from pathlib import Path
from typing import Callable, Optional

import torch
import torchvision
from torch import Tensor
from torch.utils.data import Dataset, DataLoader

class ConcatDatasetMutuallyExclusiveLabels(Dataset):
    """
    Useful attributes:
        - self.labels: contains all new USL labels i.e. contains the list of labels from 0 - total num labels after concat.
        - len(self): gives number of images after all images have been concatenated
        - self.indices_to_labels: maps the new concat idx to the new label after concat.

    ref:
        - https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor
        - https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12
    """

    def __init__(self, datasets: list[Dataset],
                 transform: Optional[Callable] = None,
                 target_transform: Optional[Callable] = None,
                 compare_imgs_directly: bool = False,
                 verify_xs_align: bool = False,
                 ):
        """
        Concatenates different data sets assuming the labels are mutually exclusive in the data sets.

        compare_imgs_directly: adds the additional test that imgs compare at the PIL imgage level.
        """
        self.datasets = datasets
        self.transform = transform
        self.target_transform = target_transform
        # I think concat is better than passing data to a self.data = x obj since concat likely using the getitem method of the passed dataset and thus if the passed dataset doesnt put all the data in memory concat won't either
        self.concat_datasets = torch.utils.data.ConcatDataset(datasets)
        # maps a class label to a list of sample indices with that label.
        self.labels_to_indices = defaultdict(list)
        # maps a sample index to its corresponding class label.
        self.indices_to_labels = defaultdict(None)
        # - do the relabeling
        self._re_label_all_dataset(datasets, compare_imgs_directly, verify_xs_align)

    def __len__(self):
        return len(self.concat_datasets)

    def _re_label_all_dataset(self, datasets: list[Dataset],
                              compare_imgs_directly: bool = False,
                              verify_xs_align: bool = False,
                              ):
        """
        Relabels according to a blind (mutually exclusive) assumption.

        Relabling Algorithm:
        The zero index of the label starts at the number of labels collected so far. So when relabling we do:
            y =  y + total_number_labels
            total_number_labels += max label for current data set
        where total_number_labels always has the + 1 to correct for the zero indexing.

        :param datasets:
        :param compare_imgs_directly:
        :parm verify_xs_align: set to false by default in case your transforms aren't deterministic.
        :return:
        """
        self.img2tensor: Callable = torchvision.transforms.ToTensor()
        self.int2tensor: Callable = lambda data: torch.tensor(data, dtype=torch.int)
        total_num_labels_so_far: int = 0
        new_idx: int = 0
        for dataset_idx, dataset in enumerate(datasets):
            assert len(dataset) == len(self.concat_datasets.datasets[dataset_idx])
            assert dataset == self.concat_datasets.datasets[dataset_idx]
            for data_idx, (x, y) in enumerate(dataset):
                y = int(y)
                # - get data point from concataned data set (to compare with the data point from the data set list)
                _x, _y = self.concat_datasets[new_idx]
                _y = int(_y)
                # - sanity check concatanted data set aligns with the list of datasets
                assert y == _y
                if compare_imgs_directly:
                    # from PIL import ImageChops
                    # diff = ImageChops.difference(x, _x)  # https://stackoverflow.com/questions/35176639/compare-images-python-pil
                    # assert diff.getbbox(), f'comparison of imgs failed: {diff.getbbox()=}' # doesn't work :/
                    assert list(x.getdata()) == list(_x.getdata()), f'\n{list(x.getdata())=}, \n{list(_x.getdata())=}'
                    # tensor comparison
                if not isinstance(x, Tensor):
                    x, _x = self.img2tensor(x), self.img2tensor(_x)
                if isinstance(y, int):
                    y, _y = self.int2tensor(y), self.int2tensor(_y)
                if verify_xs_align:
                    # this might fails if there are random ops in the getitem
                    assert torch.equal(x,
                                       _x), f'Error for some reason, got: {dataset_idx=},' \
                                            f' {new_idx=}, {data_idx=}, ' \
                                            f'{x.norm()=}, {_x.norm()=}, ' \
                                            f'{x=}, {_x=}'
                # - relabling
                new_label = y + total_num_labels_so_far
                self.indices_to_labels[new_idx] = new_label
                self.labels_to_indices[new_label].append(new_idx)
                new_idx += 1
            num_labels_for_current_dataset: int = int(max([y for _, y in dataset])) + 1
            # - you'd likely resolve unions if you wanted a proper union, the addition assumes mutual exclusivity
            total_num_labels_so_far += num_labels_for_current_dataset
        assert len(self.indices_to_labels.keys()) == len(self.concat_datasets)
        # contains the list of labels from 0 - total num labels after concat, assume mutually exclusive
        self.labels = range(total_num_labels_so_far)

    def __getitem__(self, idx: int) -> tuple[Tensor, Tensor]:
        """
        Get's the data point and it's new label according to a mutually exclusive concatenation.

        For later?
        to do the relabling on the fly we'd need to figure out which data set idx corresponds to and to compute the
        total_num_labels_so_far. Something like this:
            current_data_set_idx = bisect_left(idx)
            total_num_labels_so_far = sum(max(_, y in dataset)+1 for dataset_idx, dataset in enumerate(self.datasets) if dataset_idx <= current_data_set_idx)
            new_y = total_num_labels_so_far
            self.indices_to_labels[idx] = new_y
        :param idx:
        :return:
        """
        x, _y = self.concat_datasets[idx]
        y = self.indices_to_labels[idx]
        # for the first data set they aren't re-labaled so can't use assert
        # assert y != _y, f'concat dataset returns x, y so the y is not relabeled, but why are they the same {_y}, {y=}'
        # idk what this is but could be useful? mnist had this.
        # img = Image.fromarray(img.numpy(), mode="L")
        if self.transform is not None:
            x = self.transform(x)
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y

def assert_dataset_is_pytorch_dataset(datasets: list, verbose: bool = False):
    """ to do 1 data set wrap it in a list"""
    for dataset in datasets:
        if verbose:
            print(f'{type(dataset)=}')
            print(f'{type(dataset.dataset)=}')
        assert isinstance(dataset, Dataset), f'Expect dataset to be of type Dataset but got {type(dataset)=}.'

def get_relabling_counts(dataset: Dataset) -> dict:
    """
    counts[new_label] -> counts/number of data points for that new label
    """
    assert isinstance(dataset, Dataset), f'Expect dataset to be of type Dataset but got {type(dataset)=}.'
    counts: dict = {}
    iter_dataset = iter(dataset)
    for datapoint in iter_dataset:
        x, y = datapoint
        # assert isinstance(x, torch.Tensor)
        # assert isinstance(y, int)
        if y not in counts:
            counts[y] = 0
        else:
            counts[y] += 1
    return counts

def assert_relabling_counts(counts: dict, labels: int = 100, counts_per_label: int = 600):
    """
    default values are for MI.

    - checks each label/class has the right number of expected images per class
    - checks the relabels start from 0 and increase by 1
    - checks the total number of labels after concat is what you expect

    ref: https://openreview.net/pdf?id=rJY0-Kcll
    Because the exact splits used in Vinyals et al. (2016)
    were not released, we create our own version of the Mini-Imagenet dataset by selecting a random
    100 classes from ImageNet and picking 600 examples of each class. We use 64, 16, and 20 classes
    for training, validation and testing, respectively.
    """
    # - check each image has the right number of total images
    seen_labels: list[int] = []
    for label, count in counts.items():
        seen_labels.append(label)
        assert counts[label] == counts_per_label
    # - check all labels are there and total is correct
    seen_labels.sort()
    prev_label = -1
    for label in seen_labels:
        diff = label - prev_label
        assert diff == 1
        assert prev_label < label
    # - checks the final label is the total number of labels
    assert label == labels - 1

def check_entire_data_via_the_dataloader(dataloader: DataLoader) -> dict:
    counts: dict = {}
    for it, batch in enumerate(dataloader):
        xs, ys = batch
        for y in ys:
            if y not in counts:
                counts[y] = 0
            else:
                counts[y] += 1
    return counts

# - tests

def check_xs_align_mnist():
    root = Path('~/data/').expanduser()
    import torchvision
    # - test 1, imgs (not the recommended use)
    train = torchvision.datasets.MNIST(root=root, train=True, download=True)
    test = torchvision.datasets.MNIST(root=root, train=False, download=True)
    concat = ConcatDatasetMutuallyExclusiveLabels([train, test], compare_imgs_directly=True)
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')
    # - test 2, tensor imgs
    train = torchvision.datasets.MNIST(root=root, train=True, download=True,
                                       transform=torchvision.transforms.ToTensor(),
                                       target_transform=lambda data: torch.tensor(data, dtype=torch.int))
    test = torchvision.datasets.MNIST(root=root, train=False, download=True,
                                      transform=torchvision.transforms.ToTensor(),
                                      target_transform=lambda data: torch.tensor(data, dtype=torch.int))
    concat = ConcatDatasetMutuallyExclusiveLabels([train, test], verify_xs_align=True)
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')
    assert len(concat) == 10 * 7000, f'Err, unexpected number of datapoints {len(concat)=} expected {100 * 700}'
    assert len(
        concat.labels) == 20, f'Note it should be 20 (since it is not a true union), but got {len(concat.labels)=}'

    # - test dataloader
    loader = DataLoader(concat)
    for batch in loader:
        x, y = batch
        assert isinstance(x, torch.Tensor)
        assert isinstance(y, torch.Tensor)

def check_xs_align_cifar100():
    from pathlib import Path
    root = Path('~/data/').expanduser()
    import torchvision
    # - test 1, imgs (not the recommended use)
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
    test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)
    concat = ConcatDatasetMutuallyExclusiveLabels([train, test], compare_imgs_directly=True)
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')
    # - test 2, tensor imgs
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True,
                                          transform=torchvision.transforms.ToTensor(),
                                          target_transform=lambda data: torch.tensor(data, dtype=torch.int))
    test = torchvision.datasets.CIFAR100(root=root, train=False, download=True,
                                         transform=torchvision.transforms.ToTensor(),
                                         target_transform=lambda data: torch.tensor(data, dtype=torch.int))
    concat = ConcatDatasetMutuallyExclusiveLabels([train, test], verify_xs_align=True)
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')
    assert len(concat) == 100 * 600, f'Err, unexpected number of datapoints {len(concat)=} expected {100 * 600}'
    assert len(
        concat.labels) == 200, f'Note it should be 200 (since it is not a true union), but got {len(concat.labels)=}'
    # details on cifar100: https://www.cs.toronto.edu/~kriz/cifar.html

    # - test dataloader
    loader = DataLoader(concat)
    for batch in loader:
        x, y = batch
        assert isinstance(x, torch.Tensor)
        assert isinstance(y, torch.Tensor)

def concat_data_set_mi():
    """
    note test had to be in MI where train, val, test have disjount/different labels. In cifar100 classic the labels
    in train, val and test are shared from 0-99 instead of being different/disjoint.
    :return:
    """
    # - get mi data set
    from diversity_src.dataloaders.hdb1_mi_omniglot_l2l import get_mi_datasets
    train_dataset, validation_dataset, test_dataset = get_mi_datasets()
    assert_dataset_is_pytorch_dataset([train_dataset, validation_dataset, test_dataset])
    train_dataset, validation_dataset, test_dataset = train_dataset.dataset, validation_dataset.dataset, test_dataset.dataset
    # - create usl data set
    union = ConcatDatasetMutuallyExclusiveLabels([train_dataset, validation_dataset, test_dataset])
    # union = ConcatDatasetMutuallyExclusiveLabels([train_dataset, validation_dataset, test_dataset],
    #                                              compare_imgs_directly=True)
    assert_dataset_is_pytorch_dataset([union])
    assert len(union) == 100 * 600, f'got {len(union)=}'
    assert len(union.labels) == 100, f'got {len(union.labels)=}'

    # - create dataloader
    from uutils.torch_uu.dataloaders.common import get_serial_or_distributed_dataloaders
    union_loader, _ = get_serial_or_distributed_dataloaders(train_dataset=union, val_dataset=union)
    for batch in union_loader:
        x, y = batch
        assert x is not None
        assert y is not None

if __name__ == '__main__':
    import time
    from uutils import report_times

    start = time.time()
    # - run experiment
    check_xs_align_mnist()
    check_xs_align_cifar100()
    concat_data_set_mi()
    # - Done
    print(f"\nSuccess Done!: {report_times(start)}\a")

expected correct output:
len(concat)=70000
len(concat.labels)=20
len(concat)=70000
len(concat.labels)=20
Files already downloaded and verified
Files already downloaded and verified
len(concat)=60000
len(concat.labels)=200
Files already downloaded and verified
Files already downloaded and verified
len(concat)=60000
len(concat.labels)=200
Success Done!: time passed: hours:0.16719497998555502, minutes=10.0316987991333, seconds=601.901927947998

warning:

if you have a transform that is random the verification that the data sets align might make it look as if the two data points are not algined. The code is correct so it's not an issue, but perhaps remove the randomness somehow. Note, I actually decided to not force the user to check all the images of their data set and trust my code works from running once my unit tests. Also note that it's slow to construct the data set since I do the re-labling at the beginning. Might be better to relabel on the fly. I outlined the code for it on how to do it but decided against it since we always see all the data set at least once so doing this amortized is the same as doing it on the fly (note the fly pseudo-code saves the labels to avoid recomputations).

This is better:
# int2tensor: Callable = lambda data: torch.tensor(data, dtype=torch.int)
int2tensor: Callable = lambda data: torch.tensor(data, dtype=torch.long)

class ConcatDatasetMutuallyExclusiveLabels(Dataset):
    """
    Useful attributes:
        - self.labels: contains all new USL labels i.e. contains the list of labels from 0 - total num labels after concat.
        - len(self): gives number of images after all images have been concatenated
        - self.indices_to_labels: maps the new concat idx to the new label after concat.

    ref:
        - https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor
        - https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12
    """

    def __init__(self, datasets: list[Dataset],
                 transform: Optional[Callable] = None,
                 target_transform: Optional[Callable] = None,
                 compare_imgs_directly: bool = False,
                 verify_xs_align: bool = False,
                 ):
        """
        Concatenates different data sets assuming the labels are mutually exclusive in the data sets.

        compare_imgs_directly: adds the additional test that imgs compare at the PIL imgage level.
        """
        self.datasets = datasets
        self.transform = transform
        self.target_transform = target_transform
        # I think concat is better than passing data to a self.data = x obj since concat likely using the getitem method of the passed dataset and thus if the passed dataset doesnt put all the data in memory concat won't either
        self.concat_datasets = torch.utils.data.ConcatDataset(datasets)
        # maps a class label to a list of sample indices with that label.
        self.labels_to_indices = defaultdict(list)
        # maps a sample index to its corresponding class label.
        self.indices_to_labels = defaultdict(None)
        # - do the relabeling
        self._re_label_all_dataset(datasets, compare_imgs_directly, verify_xs_align)

    def __len__(self):
        return len(self.concat_datasets)

    def _re_label_all_dataset(self, datasets: list[Dataset],
                              compare_imgs_directly: bool = False,
                              verify_xs_align: bool = False,
                              verbose: bool = False,
                              ):
        """
        Relabels according to a blind (mutually exclusive) assumption.

        Relabling Algorithm:
        The zero index of the label starts at the number of labels collected so far. So when relabling we do:
            y =  y + total_number_labels
            total_number_labels += max label for current data set
        where total_number_labels always has the + 1 to correct for the zero indexing.

        assumption: it re-lables the data points to have a concatenation of all the labels. If there are rebeated labels
        they are treated as different. So if dataset1 and dataset2 both have cats (represented as indices), then they
        will get unique integers representing these. So the cats are treated as entirely different labels.
        """
        print()
        self.img2tensor: Callable = torchvision.transforms.ToTensor()
        total_num_labels_so_far: int = 0
        global_idx: int = 0  # new_idx
        assert len(self.indices_to_labels.keys()) == 0
        assert len(self.labels_to_indices.keys()) == 0
        for dataset_idx, dataset in enumerate(datasets):
            print(f'{dataset_idx=} \n{len(dataset)=}')
            if hasattr(dataset, 'labels'):
                print(f'{len(dataset.labels)=}')
            assert len(dataset) == len(self.concat_datasets.datasets[dataset_idx])
            assert dataset == self.concat_datasets.datasets[dataset_idx]
            original_label2global_idx: defaultdict = defaultdict(list)
            for original_data_idx, (x, original_y) in enumerate(dataset):
                original_y = int(original_y)
                # - get data point from concataned data set (to compare with the data point from the data set list)
                _x, _y = self.concat_datasets[global_idx]
                _y = int(_y)
                # - sanity check concatanted data set aligns with the list of datasets
                assert original_y == _y, f'{original_y=}, {_y=}'
                if compare_imgs_directly:
                    # from PIL import ImageChops
                    # diff = ImageChops.difference(x, _x)  # https://stackoverflow.com/questions/35176639/compare-images-python-pil
                    # assert diff.getbbox(), f'comparison of imgs failed: {diff.getbbox()=}' # doesn't work :/
                    assert list(x.getdata()) == list(_x.getdata()), f'\n{list(x.getdata())=}, \n{list(_x.getdata())=}'
                # - tensor comparison of raw images
                if not isinstance(x, Tensor):
                    x, _x = self.img2tensor(x), self.img2tensor(_x)
                # if isinstance(original_y, int):
                #     original_y, _y = int2tensor(original_y), int2tensor(_y)
                if verify_xs_align:  # checks the data points after doing get item make them match.
                    # this might fails if there are random ops in the getitem
                    assert torch.equal(x,
                                       _x), f'Error for some reason, got: {dataset_idx=},' \
                                            f' {global_idx=}, {original_data_idx=}, ' \
                                            f'{x.norm()=}, {_x.norm()=}, ' \
                                            f'{x=}, {_x=}'
                # - collect original labels in dictionary keys
                original_label2global_idx[int(original_y)].append(global_idx)
                global_idx += 1
            print(f'{global_idx=}')
            local_num_dps: int = sum(len(global_indices) for global_indices in original_label2global_idx.values())
            assert len(dataset) == local_num_dps, f'Error: \n{local_num_dps=} \n{len(dataset)=}'
            # - do relabeling - original labeling to new global labels
            print(f'{total_num_labels_so_far=}')
            assert total_num_labels_so_far != len(dataset), f'Err:\n{total_num_labels_so_far=}\n{len(dataset)=}'
            new_local_label2global_indices: dict = {}
            global_label2global_indices: dict = {}
            # make sure to sort to avoid random looping of unordered data structures e.g. keys in a dict
            for new_local_label, original_label in enumerate(sorted(original_label2global_idx.keys())):
                global_indices: list[int] = original_label2global_idx[original_label]
                new_local_label2global_indices[int(new_local_label)] = global_indices
                new_global_label: int = total_num_labels_so_far + new_local_label
                global_label2global_indices[int(new_global_label)] = global_indices
            local_num_dps: int = sum(len(global_indices) for global_indices in original_label2global_idx.values())
            assert len(dataset) == local_num_dps, f'Error: \n{local_num_dps=} \n{len(dataset)=}'
            local_num_dps: int = sum(len(global_indices) for global_indices in new_local_label2global_indices.values())
            assert len(dataset) == local_num_dps, f'Error: \n{local_num_dps=} \n{len(dataset)=}'
            local_num_dps: int = sum(len(global_indices) for global_indices in global_label2global_indices.values())
            assert len(dataset) == local_num_dps, f'Error: \n{local_num_dps=} \n{len(dataset)=}'
            # - this assumes the integers in each data set is different, if there were unions you'd likely need semantic information about the label e.g. the string cat instead of absolute integers, or know the integers are shared between the two data sets
            print(f'{total_num_labels_so_far=}')
            # this is the step where classes are concatenated. Note due to the previous loops assuming each label is uning this should never have intersecting keys.
            print(f'{list(self.labels_to_indices.keys())=}')
            print(f'{list(global_label2global_indices.keys())=}')
            dup: list = get_duplicates(list(self.labels_to_indices.keys()) + list(global_label2global_indices.keys()))
            print(f'{list(self.labels_to_indices.keys())=}')
            print(f'{list(global_label2global_indices.keys())=}')
            assert len(dup) == 0, f'Error:\n{self.labels_to_indices.keys()=}\n{global_label2global_indices.keys()=}\n{dup=}'
            for global_label, global_indices in global_label2global_indices.items():
                # note g_idx might different to global_idx!
                global_indices: list[int]
                for g_idx in global_indices:
                    self.labels_to_indices[int(global_label)] = g_idx
                    self.indices_to_labels[g_idx] = int(global_label)
            # - update number of labels seen so far
            num_labels_for_current_dataset: int = len(original_label2global_idx.keys())
            print(f'{num_labels_for_current_dataset=}')
            total_num_labels_so_far += num_labels_for_current_dataset
            assert total_num_labels_so_far == len(self.labels_to_indices.keys()), f'Err:\n{total_num_labels_so_far=}' \
                                                                                f'\n{len(self.labels_to_indices.keys())=}'
            assert global_idx == len(self.indices_to_labels.keys()), f'Err:\n{global_idx=}\n{len(self.indices_to_labels.keys())=}'
            if hasattr(dataset, 'labels'):
                assert len(dataset.labels) == num_labels_for_current_dataset, f'Err:\n{len(dataset.labels)=}' \
                                                                              f'\n{num_labels_for_current_dataset=}'
        # - relabling done
        assert len(self.indices_to_labels.keys()) == len(
            self.concat_datasets), f'Err: \n{len(self.indices_to_labels.keys())=}' \
                                   f'\n {len(self.concat_datasets)=}'
        if all(hasattr(dataset, 'labels') for dataset in datasets):
            assert sum(len(dataset.labels) for dataset in datasets) == total_num_labels_so_far
        # contains the list of labels from 0 - total num labels after concat, assume mutually exclusive
        # - set & validate new labels
        self.labels = range(total_num_labels_so_far)
        labels = list(sorted(list(self.labels_to_indices.keys())))
        assert labels == list(labels), f'labels should match and be consecutive, but got: \n{labels=}, \n{self.labels=}'

    def __getitem__(self, idx: int) -> tuple[Tensor, Tensor]:
        """
        Get's the data point and it's new label according to a mutually exclusive concatenation.

        For later?
        to do the relabling on the fly we'd need to figure out which data set idx corresponds to and to compute the
        total_num_labels_so_far. Something like this:
            current_data_set_idx = bisect_left(idx)
            total_num_labels_so_far = sum(max(_, y in dataset)+1 for dataset_idx, dataset in enumerate(self.datasets) if dataset_idx <= current_data_set_idx)
            new_y = total_num_labels_so_far + y
            self.indices_to_labels[idx] = new_y
        :param idx:
        :return:
        """
        x, _y = self.concat_datasets[idx]
        y = self.indices_to_labels[idx]
        # for the first data set they aren't re-labaled so can't use assert
        # assert y != _y, f'concat dataset returns x, y so the y is not relabeled, but why are they the same {_y}, {y=}'
        # idk what this is but could be useful? mnist had this.
        # img = Image.fromarray(img.numpy(), mode="L")
        if self.transform is not None:
            x = self.transform(x)
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y


Instruction: Map tensor values to another tensor

Input:
Hi everyone I have two tensors, one tensor has as values the indicies to another tensor which contain the values I want mapped for example:
a = tensor([0, 2, 5, 1, 0, 0, 4, 3, 0, 0, 0, 2,2,1])
b = tensor([1.5, 2.7, 1.8, 7.0, 3.9, 10.0])

I would like to produce a new tensor in which the values of b replace the index values in a as such:
new_tensor = tensor([1.5, 1.8, 10.0, 2.7, 1.5, 1.5, ... ])

Id like to do it without performing some time of for loop.  If there is a pytorch/linear away to tackle the mapping that'd be great.

Output:
>>> b[a]
tensor([ 1.5000,  1.8000, 10.0000,  2.7000,  1.5000,  1.5000,  3.9000,  7.0000,
         1.5000,  1.5000,  1.5000,  1.8000,  1.8000,  2.7000])


Instruction: custom classifier pytorch, I want to add softmax

Input:
I have this classifier:
input_dim = 25088
h1_dim = 4096
h2_dim = 2048
h3_dim = 1024
h4_dim = 512
output_dim = len(cat_to_name) # 102
drop_prob = 0.2

model.classifier = nn.Sequential(nn.Linear(input_dim, h1_dim),
                                 nn.ReLU(),
                                 nn.Dropout(drop_prob),
                                 nn.Linear(h1_dim, h2_dim),
                                 nn.ReLU(),
                                 nn.Dropout(drop_prob),
                                 nn.Linear(h2_dim, h3_dim),
                                 nn.ReLU(),
                                 nn.Dropout(drop_prob),
                                 nn.Linear(h3_dim, h4_dim),
                                 nn.ReLU(),
                                 nn.Dropout(drop_prob),
                                 nn.Linear(h4_dim, output_dim),
                                 )

and I went with CrossEntropyLoss as the criterion. In the validation and testing how can I add Softmax? This is the validation loop:
model.eval()
            with torch.no_grad():
                for images, labels in valid_loader:
                    images, labels = images.to(device), labels.to(device)
                    images.requires_grad = True

                    logits = model.forward(images)
                    batch_loss = criterion(logits, labels)
                    valid_loss += batch_loss.item()

                    ps = torch.exp(logits)
                    top_p, top_class = ps.topk(1, dim=1)
                    equals = top_class == labels.view(*top_class.shape)

Output:
The CrossEntropyLoss already applies the softmax function. From the Pytorch doc:

Note that this case is equivalent to the combination of LogSoftmax and
NLLLoss.

So if you just want to use cross entropy loss, no need to apply SoftMax beforehand.

If you really wanted to use the SoftMax function anyway, you can do:

m = nn.Softmax(dim=1)
output = m(logits)

assuming your logits have a shape of (batch_size, number_classes)
You can check: https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html


Instruction: torch.sub cause cuda memory out

Input:
I have two tensors, a and b, with sizes a.shape=(10000,10000,120) and b.shape=(10000,10000,120).
I'm trying to get a cost matrix between a and b,
cost = torch.sub((a-b)**2,-1), where cost.shape=(10000,10000).
The problem is, when I tried to do a-b or torch.sum(a,b,alpha=1), a "CUDA MEMORY OUT" error occurs.
I don't think it should cost that much. It works when the size of the tensor is small, like 2000.
Using a for iteration is not an efficient way. How can I deal with it?

Output:
It does costs much (about 134 GB).
Let's do some calculations.
Assuming your data is of type torch.float32, a will occupy a memory size of:
32 bits (4 Bytes) * 10000 * 10000 * 120 = 4.8E10 bytes ≈ 44.7 G Bytes
So does b. When you do b-a, the result also has the same shape with a and thus occupies the same amount of memory, which means you need a total of 44.7 GB * 3 (≈ 134 GB) memory to do this operation.
Is your available memory size greater than 134GB?
Possible solution:
If you will no longer use a or b afterwards, you can store the result in one of them to prevents to allocating another 44.7 GB space like this:
torch.sub(a, b, out=a)  # In this case, the result goes to `a`


Instruction: Batch of n-dimensional vectors to batch of images with n channels

Input:
I have a batch of n-dimensional vectors, i.e. a tensor of size[batch_size, n]. I want this to be transformed into a batch of images of size [batch_size, n, H, W], i.e. each element of each vector in the batch must become a [1, W, H] image, thus each vector becomes a [n, H, W] image.
Now I'm doing it in a very ugly way:
vectors = torch.zeros((batch_size, n))

# This is the (batch_size, n, H, W) tensor that I will fill
channels = torch.empty((batch_size, n, H, W))

for i, vector in enumerate(vectors):
    for j, val in enumerate(vector):
        channels[i, j].fill_(val)

How can I do it properly, using pytorch functions?

Output:
You can add dimensions to the orginal tensor with vectors[:,:, None, None], then multiply by a (H, W) tensor of ones:
channels = vectors[:,:, None, None]*torch.ones((H, W))

This will give you a tensor of size (batch_size, n, H, W), with each channels[i][j] being a (H, W) map with constant values.


Instruction: Fine tuning freezing weights nnUNet

Input:
Good morning,
I've followed the instructions in this github issue:
https://github.com/MIC-DKFZ/nnUNet/issues/1108
to fine-tune an nnUNet model (pyTorch) on a pre-trained one, but this method retrain all weights, and i would like to freeze all weigths and retrain only the last layer's weights, changing the number of segmentation classes from 3 to 1.
Do you know a way to do that?
Thank you in advance

Output:
To freeze the weights you need to set parameter.requires_grad = False.
Example:
from nnunet.network_architecture.generic_UNet import Generic_UNet

model = Generic_UNet(input_channels=3, base_num_features=64, num_classes=4, num_pool=3)

for name, parameter in model.named_parameters():
    if 'seg_outputs' in name:
        print(f"parameter '{name}' will not be freezed")
        parameter.requires_grad = True
    else:
        parameter.requires_grad = False

To check parameter names you can use print:
print(model)

which produces:
Generic_UNet(
  (conv_blocks_localization): ModuleList(
    (0): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (instnorm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (instnorm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (conv_blocks_context): ModuleList(
    (0): StackedConvLayers(
      (blocks): Sequential(
        (0): ConvDropoutNormNonlin(
          (conv): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (dropout): Dropout2d(p=0.5, inplace=True)
          (instnorm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
        (1): ConvDropoutNormNonlin(
          (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          (dropout): Dropout2d(p=0.5, inplace=True)
          (instnorm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
        )
      )
    )
    (1): Sequential(
      (0): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (dropout): Dropout2d(p=0.5, inplace=True)
            (instnorm): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
      (1): StackedConvLayers(
        (blocks): Sequential(
          (0): ConvDropoutNormNonlin(
            (conv): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (dropout): Dropout2d(p=0.5, inplace=True)
            (instnorm): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (lrelu): LeakyReLU(negative_slope=0.01, inplace=True)
          )
        )
      )
    )
  )
  (td): ModuleList(
    (0): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
  )
  (tu): ModuleList(
    (0): Upsample()
  )
  (seg_outputs): ModuleList(
    (0): Conv2d(64, 4, kernel_size=(1, 1), stride=(1, 1), bias=False)
  )
)

Or you can visualize your network with netron:
https://github.com/lutzroeder/netron


Instruction: Using GPU with Pytorch

Input: I am confused on the usage of .to(device) in pytorch. I know that it loads the variables to the GPU. But after that, lets say we multiply two gpu tensors together, will our computer know to use the gpu to do that? Or will it cast it back to CPU and preform the computation? I guess I am just confused on when and how our computer will know to use gpu outside of us telling it to hold some variable in its memory.

Output:
All operations on tensors that are e.g. in the GPU memory will be performed on the GPU. Furthermore if multiple tensors are involved in some operation, all of them need to be set to the same .device.
The way this is done is that no matter what operation we do to tensors, we'll always call methods of the torch.Tensor class. So adding two tensors a + b does actually call torch.Tensor.add, so whenever some operation is executed, it is always done with the knowledge of what device is used.


Instruction: Laplacian positional encoding in a pytorch batch

Input:
I am trying to recode the laplacian positional encoding for a graph model in pytorch. A valid encoding in numpy can be found at https://docs.dgl.ai/en/0.9.x/_modules/dgl/transforms/functional.html#laplacian_pe .
I think I have managed to make an equivalent encoding to numpy in pytorch, but for performance issues I would like that function to be able to work with batches of data.
That is, the following function works with the parameters
with the form adj[N, N], degrees[N, N] and topk as an integer, where N is the number of nodes in the network.
def _laplacian_positional_encoding_th(self, adj, degrees, topk):
    number_of_nodes = adj.shape[-1].
    #degrees = th.clip(degrees, 0, 1) # not multigraph
    assert topk < number_of_nodes

    # Laplacian
    D = th.diag(degrees**-0.5)
    B = D * adj * D
    L = th.eye(number_of_nodes).to(B.device) * B

    # Eigenvectors
    EigVal, EigVec = th.linalg.eig(L)
    idx = th.argsort(th.real(EigVal)) # increasing order
    EigVal, EigVec = th.real(EigVal[idx]), th.real(EigVec[:,idx])

    # Only select [1,topk+1] EigenVectors as L is symmetric (Spectral decomposition)
    out = EigVec[:,1:topk+1]
    return out

However, when I try to perform the same efficient operations in batch form, I cannot code it. That is, the idea is that the parameters can come in the form adj[B, N, N], degrees[B, N, N] and topk as integer, B being the number of data in the batch.

Output:
How about:
def _laplacian_positional_encoding_th(self, adj, degrees, topk):
        number_of_nodes = adj.shape[-1]
        assert topk < number_of_nodes

        D = th.clip(degrees, 0, 1) # not multigraph
        B = D @ adj @ D
        L = th.eye(number_of_nodes).to(B.device)[None, ...] - B

        # Eigenvectors
        EigVal, EigVec = th.linalg.eig(L)
        idx = th.argsort(th.real(EigVal)) # increasing order

        out = th.real(th.gather(EigVec, dim=-1, index=idx[..., None]))
        return out

See th.diag_embed for creating a batch of diagonal matrices, and th.gather for selecting the right columns of EigVec according to the sorted indices.
Updata:
If you want to extract the topk vectors:
_, topk = th.topk(EigVal.real, k=5)  # get the top 5
out = th.gather(EigVec.real, dim=-1, index=topk[:, None, :].expand(-1, EigVec.shape[1], -1))


Instruction: How can I use larger input images when using a pre-trained CNN without resizing?

Input:
I have a ResNet18 model trained on the Places365 image dataset, and I'd like to use this pre-trained model to expedite the training needed to identify distressed houses. My dataset is images of size 800x800x3, but the inputs are of size 224x224x3. I do not want to resize the image to 224x224 since I'm worried it will lose house distress indicators (chipped paint and loose shingles) during conversion.
My idea was to add extra layers that can handle the larger images before feeding them into ResNet. I have the following pytorch model:
import torch
from torch import nn
from torchvision import models

class NewModel(nn.Model):
    def __init__(self, pretrain_model_path) -> None:
        # Not sure here
        self.pre_layers = None
        # import the trained model
        model = models.resnet18(num_classes=365)
        checkpoint = torch.load(pretrain_model_path, map_location=lambda storage, loc: storage)
        state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}
        model.load_state_dict(state_dict)
        # change prediction class count
        model.fc = nn.Linear(model.fc.in_features, 4)
        self.backbone = model

    def forward(self, x):
        x = self.pre_layers(x)
        x = self.backbone(x)
        return x

Is this a common practice or is it better to make one from scratch that is built for this size image specifically? How would I go about implementing it if so?

Output:
You can use an ordinary resnet18 model and pass 800x800 images to it. But it may be slow and consumes more memory.

import torch
from torchvision import models
model = models.resnet18(num_classes=4)
print(model(torch.zeros((1, 3, 800, 800))).shape)  # (1, 4)

You can add any lightweight module that reduces image resolution:

import torch
from torch import nn
from torchvision import models

class NewModel(nn.Module):
    def __init__(self, intermediate_features=64) -> None:
        super().__init__()
        model = models.resnet18(num_classes=4)
        self.backbone = model

        self.pre_model = nn.Sequential(
            nn.Conv2d(3, intermediate_features, 3, stride=2, padding=1),
            nn.ReLU(),
        )

        conv1 = self.backbone.conv1
        self.backbone.conv1 = nn.Conv2d(
            intermediate_features, conv1.out_channels,
            conv1.kernel_size, conv1.stride, conv1.padding)

    def forward(self, x):
        # 3x800x800
        x = self.pre_model(x)
        # 3x400x400
        x = self.backbone(x)
        # 4
        return x

model = NewModel()
x = torch.zeros((1, 3, 800, 800))
print(model(x).shape)

Depending on your data the different approaches may perform better or worse so you may need to experiment with model architectures.


Instruction: How to select top half of the elements in each row of a given tensor? Pytorch code

Input:
I have a tensor A with shape (Batch, sequence, dimension).
I calculated the importance weights for each element in a sequence. The weights can be represented as a tensor B with shape (Batch, sequence).
I need to select the first half of the elements with large weights. That means the result is expected to have a shape (Batch, sequence/2, dimension).
Note:

Each sequence of tensor A has different length. I pad the sequences to form the tensor A.
I have the mask matrix Mask_A with shape (Batch, sequence).
Therefore, I can't simply take the top sequence/2 elements because of padding.
I know it can be achieved by 'for' loop in python code. However, it will slow down the program. I hope it can be done in Pytorch functions to get GPU acceleration.
The result doesn't have to be sorted according to weights.
It's also acceptable to receive a result of mask containing top-half elements.

Here is an example code:
A = torch.randn((2, 4, 3))
# batch size = 2, sequence legnth = 4, dimension = 3
# it looks like
'''tensor([[[ 6.0840e-01,  4.5604e-01, -1.3264e+00],
         [-4.6437e-01,  1.6999e-01,  1.3551e+00],
         [-1.9888e+00, -2.3047e-01,  1.2347e-03],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00]],

        [[ 7.9035e-01, -5.5752e-01, -1.2477e+00],
         [-1.7801e-01,  4.6232e-01,  1.3019e+00],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00]]])'''
# [ 0.0000e+00,  0.0000e+00,  0.0000e+00] means padding

Mask_A = torch.tensor([[1., 1., 1., 0.], [1., 1., 0., 0.]])
# this means 1-st row of A has three elements, 2-nd row has two elements. '0.' means padding

B = torch.tensor([[0.3,0.1,0.6,0.0],[0.05, 0.95, 0.0, 0.0]])
# this is the unsorted importance weights matrix

# what I want is like:
'''
tensor([[[-1.9888e+00, -2.3047e-01,  1.2347e-03],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00]],

        [[-1.7801e-01,  4.6232e-01,  1.3019e+00],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00]]])
'''

# [ 0.0000e+00,  0.0000e+00,  0.0000e+00] means padding

# what I want can also be like:
Mask_result = torch.tensor([[0., 0., 1., 0.], [0., 1., 0., 0.]])

Output:
Making some assumptions here about exactly how many elements you want to keep, but this should work:
# Get a tensor marking the rank of each weight within the sequence
rank = torch.argsort(torch.argsort(B, axis=1, descending=True), axis=1)

# Determine how many elements per sequence we want to keep, based on the mask
n_to_keep = torch.floor(torch.sum(Mask_A, axis=1)/2)

# Make a mask marking the n elements per sequence with the top weights
# using broadcasting
mask_result = rank < n_to_keep[:,None]


Instruction: Einsum for shapes of different sizes or ranks

Input:
I have two PyTorch tensors. One is rank three and the other is rank four. Is there a way to get it so that it produce the rank and shape of the first tensor? For instance in this cross-attention bit:
q = torch.linspace(1, 192, steps=192)
q = q.reshape(2, 4, 3, 8)
k = torch.linspace(2, 193, steps=192)
k = k.reshape(2, 4, 3, 8)
v = torch.linspace(3, 194, steps=192)
v = v.reshape(2, 4, 24)

k = k.permute(0, 3, 2, 1)
attn = torch.einsum("nchw,nwhu->nchu", q, k)

# Below is what doesn't work. I would like to get it such that hidden_states is a tensor of rank 2, 4, 24
hidden_states = torch.einsum("chw,whu->chu", attn, v)

Is there a permutation/transpose I could apply to q, k, v, or attn that would allow me to multiply into (2, 4, 24)?
I have yet to find one.
I currently receive this error: "RuntimeError: einsum(): the number of subscripts in the equation (3) does not match the number of dimensions (4) for operand 0 and no ellipsis was given" so I'm wondering how to use the ellipsis in this case, if that could be a solution.
Any explanation as to why this is or isn't possible would also be an excepted answer!

Output:
It seems like your q and k are 4D tensors of shape batch-channel-height-width (2x4x3x8). However, when considering attention mechanism, one disregard the spatial arrangement of the features and only treat them as a "bag of features". That is, instead of q and k of shape 2x4x3x8 you should have 2x4x24:
q = torch.linspace(1, 192, steps=192)
q = q.reshape(2, 4, 3 * 8)  # collapse the spatial dimensions into a single one
k = torch.linspace(2, 193, steps=192)
k = k.reshape(2, 4, 3 * 8)  # collapse the spatial dimensions into a single one
v = torch.linspace(3, 194, steps=192)
v = v.reshape(2, 4, 24)

attn = torch.einsum("bcn,bcN->bnN", q, k)
# it is customary to convert the raw attn into probabilities using softmax
attn = torch.softmax(attn, dim=-1)
hidden_states = torch.einsum("bnN,bcN->bcn", attn, v)


Instruction: Rsizing The Spatial Dimensions Of a 5D Tensor

Input: For training a deep learning based model, I have an input tensor with the size of [batch_size=32, channels=3, Temporal=16, H=128, w=192] that includes frames of a video. I need to resize the spatial size of the tensor to (H=224, w=224). In other words, I need to have a tensor with the size of [batch_size=32, channels=3, Temporal=16, H=224, w=224]. How can I do that?

Output:
You can use torch.nn.functional.interpolate:
import torch.nn.functional as nnf

y = nnf.interpolate(x, size=(x.shape[2], 224, 224), mode='trilinear')


Instruction: How does a pytorch function (such as RoIPool) work?

Input:
For example, I'm trying to view the implementation of RoI Pooling in pytorch.
Here is a code fragment showing how to use RoIPool in pytorch
import torch
from torchvision.ops.roi_pool import RoIPool

device = torch.device('cuda')

# create feature layer, proposals and targets
num_proposals = 10
feature_map = torch.randn(1, 64, 32, 32)

proposals = torch.zeros((num_proposals, 4))
proposals[:, 0] = torch.randint(0, 16, (num_proposals,))
proposals[:, 1] = torch.randint(0, 16, (num_proposals,))
proposals[:, 2] = torch.randint(16, 32, (num_proposals,))
proposals[:, 3] = torch.randint(16, 32, (num_proposals,))

roi_pool_obj = RoIPool(3, 2**-1)
roi_pool = roi_pool_obj(feature_map, [proposals])

I'm using pychram, so when I follow RoIPool from the second line, it opens a file located at ~/anaconda3/envs/CV/lib/python3.8/site-package/torchvision/ops/roi_pool.py, which is exactly the same as codes in the documentation.
I pasted the code below without documentations.
from typing import List, Union

import torch
from torch import nn, Tensor
from torch.jit.annotations import BroadcastingList2
from torch.nn.modules.utils import _pair
from torchvision.extension import _assert_has_ops

from ..utils import _log_api_usage_once
from ._utils import convert_boxes_to_roi_format, check_roi_boxes_shape

def roi_pool(
    input: Tensor,
    boxes: Union[Tensor, List[Tensor]],
    output_size: BroadcastingList2[int],
    spatial_scale: float = 1.0,
) -> Tensor:

    if not torch.jit.is_scripting() and not torch.jit.is_tracing():
        _log_api_usage_once(roi_pool)
    _assert_has_ops()
    check_roi_boxes_shape(boxes)
    rois = boxes
    output_size = _pair(output_size)
    if not isinstance(rois, torch.Tensor):
        rois = convert_boxes_to_roi_format(rois)
    output, _ = torch.ops.torchvision.roi_pool(input, rois, spatial_scale, output_size[0], output_size[1])
    return output

class RoIPool(nn.Module):

    def __init__(self, output_size: BroadcastingList2[int], spatial_scale: float):
        super().__init__()
        _log_api_usage_once(self)
        self.output_size = output_size
        self.spatial_scale = spatial_scale

    def forward(self, input: Tensor, rois: Tensor) -> Tensor:
        return roi_pool(input, rois, self.output_size, self.spatial_scale)

    def __repr__(self) -> str:
        s = f"{self.__class__.__name__}(output_size={self.output_size}, spatial_scale={self.spatial_scale})"
        return s

So, in the code example:
When running roi_pool_obj = RoIPool(3, 2**-1) it will create an instance of RoIPool by calling its __init__ method, which only initialized two instance variables;
When running roi_pool = roi_pool_obj(feature_map, [proposals]), it must have called the forward() method (but I don't know how) which then called the roi_pool() function above;
When running the roi_pool() function, it did some checking first and then computed output with the line output, _ = torch.ops.torchvision.roi_pool(input, rois, spatial_scale, output_size[0], output_size[1]).
But this doesn't show details of how roi_pool is implemented and pycharm showed Cannot find declaration to go to when I tried to follow torch.ops.torchvision.roi_pool.
To summarize, I have two questions:

How does the forward() called by running roi_pool = roi_pool_obj(feature_map, [proposals])?
How can I view the source code of torch.ops.torchvision.roi_pool or where is the file containing it's implementaion located?

Last but not least, I've just started reading source code which is pretty difficult for me. I'd appreciate it if you can also provide some advice or tutorials.

Output:
RoIPool is a subclass of torch.nn.Module. Source code:

https://github.com/pytorch/vision/blob/07ae61bf9c21ddd1d5f65d326aa9636849b383ca/torchvision/ops/roi_pool.py#L56

nn.Module defines __call__ method which in turn calls forward method. Source code:

https://github.com/pytorch/pytorch/blob/b2311192e6c4745aac3fdd774ac9d56a36b396d4/torch/nn/modules/module.py#L1234

When you executing roi_pool = roi_pool_obj(feature_map, [proposals]) statement the __call__ method uses the forward() of RoiPool. Source code:

https://github.com/pytorch/vision/blob/07ae61bf9c21ddd1d5f65d326aa9636849b383ca/torchvision/ops/roi_pool.py#L67

RoiPool.forward calls torch.ops.torchvision.roi_pool.

https://github.com/pytorch/vision/blob/07ae61bf9c21ddd1d5f65d326aa9636849b383ca/torchvision/ops/roi_pool.py#L52

ops is a object which loads native libraries implemented in c++:

https://github.com/pytorch/pytorch/blob/b2311192e6c4745aac3fdd774ac9d56a36b396d4/torch/_ops.py#L537
so when you call torch.ops.torchvision it will use torchvision library.

Here the roi_pool function is registered:

https://github.com/pytorch/vision/blob/7947fc8fb38b1d3a2aca03f22a2e6a3caa63f2a0/torchvision/csrc/ops/roi_pool.cpp#L53

Here you can find the actual implementation of rol_pool

CPU:
https://github.com/pytorch/vision/blob/7947fc8fb38b1d3a2aca03f22a2e6a3caa63f2a0/torchvision/csrc/ops/cpu/roi_pool_kernel.cpp
GPU:
https://github.com/pytorch/vision/blob/7947fc8fb38b1d3a2aca03f22a2e6a3caa63f2a0/torchvision/csrc/ops/cuda/roi_pool_kernel.cu


Instruction: Problem installing matplotlib using conda and virtual environment. ModuleNotFoundError

Input:
I have tried to install matplotlib in my environment "cs323V2"with the following commands:
(cs323V2) conda install matplotlib
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Retrieving notices: ...working... done
(cs323V2) python
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
[3]+  Stopped                 python
>>> import matplotlib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'matplotlib'

Output:
Use the conda-forge channel.
conda install -c conda-forge matplotlib

I am not really sure why only
conda install matplotlib

does not work.


Instruction: How to combine two tensors of different shapes in pytorch?

Input:
I am currently doing this:
repeat_vals = [x.shape[0] // pfinal.shape[0]] + [-1] * (len(pfinal.shape) - 1)

x = torch.cat((x, pfinal.expand(*repeat_vals)), dim=-1)

the shape of x is[91,6] and of final is[6,6]
but I am getting this error:
RuntimeError: The expanded size of the tensor (15) must match the existing size (6) at non-singleton dimension 0.  Target sizes: [15, -1].  Tensor sizes: [6, 6]

Output:
You cannot expand non-singleton values. Furthermore, you cannot enforce len(x) to be a multiple of len(pfinal), so instead, depending on your needs, you could more over the desired number and then slice away the excess. Something that you can modify to fit your needs:
>>> reps = len(x) // len(pfinal) + 1
>>> res = pfinal.repeat(reps, *[1]*(pfinal.ndim - 1))[:len(x)]


Instruction: Slicing audio given video frames

Input:
I have audio from a video that I've loaded with PyTorch. Given a starting index and ending index corresponding to the video segment of interest, along with the video FPS and audio sampling rate, how would I go about extracting the slice of audio that matches the segment of interest of the video?
My intuition is to convert frames to time via:
start_time = frame_start / fps
end_time = frame_end / fps

the convert time to sample position with:
start_sample = int(math.floor(start_time * sr))
end_sample = int(math.floor(end_time * sr))

Is this correct? Or is there something I'm missing? I'm worried that there will be loss of information since I'm converting the samples into ints with floor.

Output:
Let's say you have
fs = 44100                # audio sampling frequency
vfr = 24                  # video frame rate
frame_start  = 10         # index of first frame
frame_end  = 10           # index of last frame
audio = np.arange(44100)  # audio in form of ndarray

you can calculate at which points in time you want to slice the audio
time_start = frame_start / vfr
time_end = frame_end / vfr         # or (frame_end + 1) / vfr for inclusive cut

and then to which samples those points in time correspond:
sample_start_idx = int(time_start * fs)
sample_end_idx = int(time_end * fs)

Its up to you if you want to be super-precise and take into account the fact that audio corresponding to a given frame should rather be starting half a frame before a frame and end half a frame after.
In such a case use:
time_start = np.clip((frame_start - 0.5) / vfr, 0, np.inf)
time_end = (frame_end + 0.5) / vfr


Instruction: Multi-target loss recommendations

Input:
I'm working on a classification problem. The number of classes is 5. I have a ground truth vector that has the shape (3) instead of 1. The values in this target vector are the possible classes and the predicted vector is of the shape (1x5) which holds the softmax scores for all the classes.
For example:
predicted_vector = tensor([0.0669, 0.1336, 0.3400, 0.3392, 0.1203]
ground_truth = tensor([3,2,5])

For the above illustration, a typical argmax operation would result in declaring class 3 as the predicted class (0.34) but I want the model to reward even if the argmax class is any of 3,2, or 5.
Which loss function is recommended for such a use case?

Output:
For this problem, a given sample is in exactly one class (say, class 3), but for training purposes, predicting class 2 or 5 is still okay so the model isn't penalised that heavily.
This is a typical single-label, multi-class problem, but with probabilistic (“soft”) labels, and CrossEntropyLoss should be used (and not use softmax()).
In this example, the (soft) target might be a probability of 0.7 for class 3, a probability of 0.2 for class 2, and a probability of 0.1 for class 5 (and zero for everything else).