Instruction: Error training ELMo - RuntimeError: The size of tensor a (5158) must match the size of tensor b (5000) at non-singleton dimension 1

Input:
I am trying to train my own custom ELMo model on AllenNLP.
The following bug RuntimeError: The size of tensor a (5158) must match the size of tensor b (5000) at non-singleton dimension 1 arises when training the model. There are instances where the size of tensor a is stated to be other values (e.g. 5300). When I tested on a small subset of files, I was able to train the model successfully.
Based on my intuition, this is something that deals with the number of tokens in my model. More specifically specific files which has tokens more than 5000. However, there is no parameter within the AllenNLP package which allows me to tweak this to bypass this error.
Any advice on how I can overcome this issue? Would tweaking the PyTorch code to set it at a 5000 size work (If yes, how can I do that)? Any insights will be deeply appreciated.
FYI, I am currently using a customised DatasetReader for tokenisation purposes. I've generated my own vocab list before training the model (to save some time) which is used to train the ELMo model via AllenNLP.
Update: I found out that there is this variable from AllenNLP max_len=5000 which is why the error is showing. See code here. I've tweaked the parameter to larger values and ended up with CUDA Out of Memory Error on many occasions instead. Making me believe this should not be touched.
Environment: Python 3.6.9, Linux Ubuntu, allennlp=2.9.1, allennlp-models=2.9.0
Traceback:
Traceback (most recent call last):
  File "/home/jiayi/.local/bin/allennlp", line 8, in <module>
    sys.exit(run())
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/__main__.py", line 34, in run
    main(prog="allennlp")
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/__init__.py", line 121, in main
    args.func(args)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/train.py", line 120, in train_model_from_args
    file_friendly_logging=args.file_friendly_logging,
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/train.py", line 179, in train_model_from_file
    file_friendly_logging=file_friendly_logging,
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/train.py", line 246, in train_model
    file_friendly_logging=file_friendly_logging,
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/train.py", line 470, in _train_worker
    metrics = train_loop.run()
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/commands/train.py", line 543, in run
    return self.trainer.train()
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/training/gradient_descent_trainer.py", line 720, in train
    metrics, epoch = self._try_train()
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/training/gradient_descent_trainer.py", line 741, in _try_train
    train_metrics = self._train_epoch(epoch)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/training/gradient_descent_trainer.py", line 459, in _train_epoch
    batch_outputs = self.batch_outputs(batch, for_training=True)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp/training/gradient_descent_trainer.py", line 352, in batch_outputs
    output_dict = self._pytorch_model(**batch)
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp_models/lm/models/language_model.py", line 257, in forward
    embeddings, mask
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp_models/lm/modules/seq2seq_encoders/bidirectional_lm_transformer.py", line 282, in forward
    token_embeddings = self._position(token_embeddings)
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jiayi/.local/lib/python3.6/site-packages/allennlp_models/lm/modules/seq2seq_encoders/bidirectional_lm_transformer.py", line 68, in forward
    return x + self.positional_encoding[:, : x.size(1)]
RuntimeError: The size of tensor a (5385) must match the size of tensor b (5000) at non-singleton dimension 1

AllenNLP training config file:
// For more info on config files generally, see https://guide.allennlp.org/using-config-files

local NUM_GRAD_ACC = 4;
local BATCH_SIZE = 1;

local BASE_LOADER = {
  "max_instances_in_memory": 8,
  "batch_sampler": {
    "type": "bucket",
    "batch_size": BATCH_SIZE,
    "sorting_keys": ["source"]
  }
};

{
    "dataset_reader" : {
        "type": "mimic_reader",
        "token_indexers": {
            "tokens": {
                "type": "single_id"
            },
            "token_characters": {
                "type": "elmo_characters"
            }
        },
        "start_tokens": ["<S>"],
        "end_tokens": ["</S>"],
    },
    "train_data_path": std.extVar("MIMIC3_NOTEEVENTS_DISCHARGE_PATH"),
    // Note: We don't set a validation_data_path because the softmax is only
    // sampled during training. Not sampling on GPUs results in a certain OOM
    // given our large vocabulary. We'll need to evaluate against the test set
    // (when we'll want a full softmax) with the CPU.
    "vocabulary": {
        // Use a prespecified vocabulary for efficiency.
        "type": "from_files",
        "directory": std.extVar("ELMO_VOCAB_PATH"),
        // Plausible config for generating the vocabulary.
        // "tokens_to_add": {
        //     "tokens": ["<S>", "</S>"],
        //     "token_characters": ["<>/S"]
        // },
        // "min_count": {"tokens": 3}
    },
    "model": {
        "type": "language_model",
        "bidirectional": true,
        "num_samples": 8192,
        # Sparse embeddings don't work with DistributedDataParallel.
        "sparse_embeddings": false,
        "text_field_embedder": {
        "token_embedders": {
            "tokens": {
            "type": "empty"
            },
            "token_characters": {
                "type": "character_encoding",
                "embedding": {
                    "num_embeddings": 262,
                    // Same as the Transformer ELMo in Calypso. Matt reports that
                    // this matches the original LSTM ELMo as well.
                    "embedding_dim": 16
                },
                "encoder": {
                    "type": "cnn-highway",
                    "activation": "relu",
                    "embedding_dim": 16,
                    "filters": [
                        [1, 32],
                        [2, 32],
                        [3, 64],
                        [4, 128],
                        [5, 256],
                        [6, 512],
                        [7, 1024]],
                    "num_highway": 2,
                    "projection_dim": 512,
                    "projection_location": "after_highway",
                    "do_layer_norm": true
                }
            }
        }
        },
        // Consider the following.
        // remove_bos_eos: true,
        // Applies to the contextualized embeddings.
        "dropout": 0.1,
        "contextualizer": {
            "type": "bidirectional_language_model_transformer",
            "input_dim": 512,
            "hidden_dim": 4096,
            "num_layers": 2,
            "dropout": 0.1,
            "input_dropout": 0.1
        }
    },
    "data_loader": BASE_LOADER,
    // "distributed": {
    //     "cuda_devices": [0, 1],
    // },
    "trainer": {
        "num_epochs": 10,
        "cuda_devices": [0, 1, 2, 3],
        "optimizer": {
        // The gradient accumulators in Adam for the running stdev and mean for
        // words not used in the sampled softmax would be decayed to zero with the
        // standard "adam" optimizer.
        "type": "dense_sparse_adam"
        },
        // "grad_norm": 10.0,
        "learning_rate_scheduler": {
        "type": "noam",
        // See https://github.com/allenai/calypso/blob/master/calypso/train.py#L401
        "model_size": 512,
        // See https://github.com/allenai/calypso/blob/master/bin/train_transformer_lm1b.py#L51.
        // Adjusted based on our sample size relative to Calypso's.
        "warmup_steps": 6000
        },
        "num_gradient_accumulation_steps": NUM_GRAD_ACC,
        "use_amp": true
    }
}

Output:
By setting the max_tokens variable for the custom DatasetReader built to below 5000, this error no longer persists. This was also suggested by one of AllenNLP's contributor to make sure the tokenizer truncates the input to 5000 tokens.
Same question was posted on AllenNLP: https://github.com/allenai/allennlp/discussions/5601


Instruction: Type Hinting torch.float for special form type hint

Input:
I have a function that can take a typing.Union of types including the type torch.float. But if I use the typing.Union type with torch.float as argument, I receive an error. Here is an example:
from typing import Union
import torch

def fct(my_float_or_tensor: Union[torch.float, torch.Tensor]):
    pass

And I get the error
TypeError: Union[t0, t1, ...]: each t must be a type. Got torch.float32.

What am I doing wrong?
Interestingly, the same problem occurs with the special type typing.Tuple but not if I use torch.float directly when type hinting.

Output: There is a difference between "dtypes" and "types". torch.float is a dtype. For type hinting, use torch.FloatTensor (there are also others, e.g., DoubleTensor, HalfTensor etc.)


Instruction: Create iterator from a Data Frame in Python

Input:
I am working on an NLP project using Seq2Seq. I created a data frame from my dataset then created a batch iterator using data loader, see the following code:
# creates lists containing each pair
original_word_pairs = [[w for w in l.split('\t')] for l in lines[:num_examples]]
data = pd.DataFrame(original_word_pairs, columns=["src", "trg"])

# conver the data to tensors and pass to the Dataloader
# to create a batch iterator

class MyData(Dataset):
    def __init__(self, X, y):
        self.data = X
        self.target = y
        # TODO: convert this into torch code is possible
        self.length = [ np.sum(1 - np.equal(x, 0)) for x in X]

    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        x_len = self.length[index]
        return x,y,x_len

    def __len__(self):
        return len(self.data)

train_dataset = MyData(input_tensor_train, target_tensor_train)
val_dataset = MyData(input_tensor_val, target_tensor_val)

train_dataset = DataLoader(train_dataset, batch_size = BATCH_SIZE,
                     drop_last=True,
                     shuffle=True)
test_dataset= DataLoader(val_dataset, batch_size = BATCH_SIZE,
                     drop_last=True,
                     shuffle=True)

That is a part of my code, the thing is I want to use the iterators like this
for i, batch in enumerate(iterator):

        src = batch.src
        trg = batch.trg

But I got an error "AttributeError: 'list' object has no attribute 'src'"
How can I use the iterator and access a specific column?

Output:
You can redefine __getitem__ in your Dataset to return a dictionary:
def __getitem__(self, index):
    x = self.data[index]
    y = self.target[index]
    x_len = self.length[index]
    return {"src": x, "trg": y, "x_len": x_len}

The default collate_fn of DataLoader will take care to provide a dictionary containing batches instead of single observations, but you need to convert x_len to a tensor into __getitem__ to make it work (or you can pass a custom collate_fn).


Instruction: Equivalent to torch.rfft() in newest PyTorch version

Input:
I want to estimate the fourier transform for a given image of size BxCxWxH
In previous torch version the following did the job:
fft_im = torch.rfft(img, signal_ndim=2, onesided=False)

and the output was of size:
BxCxWxHx2

However, with the new version of rfft :
fft_im = torch.fft.rfft2(img, dim=2, norm=None)

I do not get the same results. Do I miss something?

Output:
A few issues

The dim argument you provided is an invalid type, it should be a tuple of two numbers or should be omitted. Really PyTorch should raise an exception. I would argue that the fact this ran without exception is a bug in PyTorch (I opened a ticket stating as much).
PyTorch now supports complex tensor types, so FFT functions return those instead of adding a new dimension for the real/imaginary parts. You can use torch.view_as_real to convert to the old representation. Also worth pointing out that view_as_real doesn't copy data since it returns a view so shouldn't slow things down in any noticeable way.
PyTorch no longer gives the option of disabling one-sided calculation in RFFT. Probably because disabling one-sided makes the result identical to torch.fft.fft2, which is in conflict with the 13th aphorism of PEP 20. The whole point of providing a special real-valued version of the FFT is that you need only compute half the values for each dimension, since the rest can be inferred via the Hermition symmetric property.

So from all that you should be able to use
fft_im = torch.view_as_real(torch.fft.fft2(img))

Important If you're going to pass fft_im to other functions in torch.fft (like fft.ifft or fft.fftshift) then you'll need to convert back to the complex representation using torch.view_as_complex so those functions don't interpret the last dimension as a signal dimension.


Instruction: Not able to switch off batch norm layers for faster-rcnn (PyTorch)

Input:
I'm trying to switch off batch norm layers in a faster-rcnn model for evaluation mode.
I'm doing a sanity check atm:
@torch.no_grad()
def evaluate_loss(model, data_loader, device):
    val_loss = 0
    model.train()
    for images, targets in data_loader:
        # check that all layers are in train mode
        # for name, module in model.named_modules():
        #     if hasattr(module, 'training'):
        #         print('{} is training {}'.format(name, module.training))
        #         # set bn layers to eval
        for module in model.modules():
            if isinstance(module, torch.nn.BatchNorm2d):
                module.eval()
        # bn layers are now in eval
        for name, module in model.named_modules():
            if hasattr(module, 'training'):
                print('{} is training {}'.format(name, module.training))

However, all the batch norm layers are still in training mode. When I replace it with for example Conv2d, I get the expected behaviour of False. Here is an example snippet of the output:
backbone.body.layer4.0.conv1 is training True
backbone.body.layer4.0.bn1 is training True
backbone.body.layer4.0.conv2 is training True
backbone.body.layer4.0.bn2 is training True
backbone.body.layer4.0.conv3 is training True
backbone.body.layer4.0.bn3 is training True

Why is this happening? What can I do to switch off these layers? I have tried this with all variations of batch norm as provided by torch.nn.

Output:
So, after further investigation and after printing out all modules provided by the faster-rcnn, instead of BatchNorm2d, FrozenBatchNorm2d is used by the pretained model.
Furthermore, unlike what's currently stated by the documentation, you must call torchvision.ops.misc.FrozenBatchNorm2d instead of torchvision.ops.FrozenBatchNorm2d.
Additionally, as the layers are already frozen, there is no need to "switch off" these layers thus model.eval() is probably not required.


Instruction: How to remove a prediction head from pytorch model based on the output tensor?

Input:
I am working on a ViT (Vision Transformer) related project and some low level definition is deep inside timm library, which I can not change. The low level library definition involves a linear classification prediction head, which is not a part of my network.
Every thing was fine until I switched to DDP parallel implementation. Pytorch complained about some parameters which didn’t contribute to the loss, and it instructed me to use “find_unused_parameters=True”. In fact, it is a common scenario and it worked again if I added this “find_unused_parameters=True” to the training routine. However, I am only allowed to change the model definition in our code base, but I cannot modify anything related to training …
So I guess the only thing I can do right now, is to “remove” the linear head from the model.
Although I cannot dig into the low level definition of ViT, but I can output this tensor like this:
encoder_output,   linear_head_output =  ViT(input)

Is it possible to remove this linear prediction head based on this linear_head_output tensor?

Output:
Just set the num_classes=0 when you create your ViT model by calling timm.create_model().
Here is an example from TIMM documentation on Feature Extraction:
import torch
import timm
m = timm.create_model('resnet50', pretrained=True, num_classes=0, global_pool='')
o = m(torch.randn(2, 3, 224, 224))
print(f'Unpooled shape: {o.shape}')


Instruction: Manual Bidirectional torch.nn.RNN Implementation

Input:
I'm trying to reimplement the torch.nn.RNN module without C++/CUDA bindings, i.e., using simple tensor operations and associated logic. I have developed the following RNN class and associated testing logic, which can be used to compare output with a reference module instance:
import torch
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, bidirectional=False):
        super(RNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bidirectional = bidirectional
        self.w_ih = [torch.randn(hidden_size, input_size)]
        if bidirectional:
            self.w_ih_reverse = [torch.randn(hidden_size, input_size)]

        for layer in range(num_layers - 1):
            self.w_ih.append(torch.randn(hidden_size, hidden_size))
            if bidirectional:
                self.w_ih_reverse.append(torch.randn(hidden_size, hidden_size))

        self.w_hh = torch.randn(num_layers, hidden_size, hidden_size)
        if bidirectional:
            self.w_hh_reverse = torch.randn(num_layers, hidden_size, hidden_size)

    def forward(self, input, h_0=None):
        if h_0 is None:
            if self.bidirectional:
                h_0 = torch.zeros(2, self.num_layers, input.shape[1], self.hidden_size)
            else:
                h_0 = torch.zeros(1, self.num_layers, input.shape[1], self.hidden_size)

        if self.bidirectional:
            output = torch.zeros(input.shape[0], input.shape[1], 2 * self.hidden_size)
        else:
            output = torch.zeros(input.shape[0], input.shape[1], self.hidden_size)

        for t in range(input.shape[0]):
            print(input.shape, t)
            input_t = input[t]
            if self.bidirectional:
                input_t_reversed = input[-1 - t]

            for layer in range(self.num_layers):
                h_t = torch.tanh(torch.matmul(input_t, self.w_ih[layer].T) + torch.matmul(h_0[0][layer], self.w_hh[layer].T))
                h_0[0][layer] = h_t
                if self.bidirectional:
                    h_t_reverse = torch.tanh(torch.matmul(input_t_reversed, self.w_ih_reverse[layer].T) + torch.matmul(h_0[1][layer], self.w_hh_reverse[layer].T))
                    h_0[1][layer] = h_t_reverse

                input_t = h_t
                if self.bidirectional:
                    # This logic is incorrect for bidirectional RNNs with multiple layers
                    input_t = torch.cat((h_t, h_t_reverse), dim=-1)
                    input_t_reversed = input_t

            output[t, :, :self.hidden_size] = h_t
            if self.bidirectional:
                output[-1 - t, :, self.hidden_size:] = h_t_reverse

        return output

if __name__ == '__main__':
    input_size = 10
    hidden_size = 12
    num_layers = 2
    batch_size = 2
    bidirectional = True
    input = torch.randn(2, batch_size, input_size)
    rnn_val = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, bias=False, bidirectional=bidirectional, nonlinearity='tanh')
    rnn = RNN(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, bidirectional=bidirectional)
    for i in range(rnn_val.num_layers):
        rnn.w_ih[i] = rnn_val._parameters['weight_ih_l%d' % i].data
        rnn.w_hh[i] = rnn_val._parameters['weight_hh_l%d' % i].data
        if bidirectional:
            rnn.w_ih_reverse[i] = rnn_val._parameters['weight_ih_l%d_reverse' % i].data
            rnn.w_hh_reverse[i] = rnn_val._parameters['weight_hh_l%d_reverse' % i].data

    output_val, hn_val = rnn_val(input)
    output = rnn(input)
    print(output_val)
    print(output)

My implementation appears to work for vanilla RNNs with an arbitrary number of layers and different batch sizes/sequence lengths, in addition to single-layered bidirectional RNNs, however, it does not produce the correct result for multi-layered bidirectional RNNs.
For sake of simplicity, bias terms are not currently implemented, and only the tanh activation function is supported. I have narrowed the logic error down to the line input_t = torch.cat((h_t, h_t_reverse), dim=-1), as the first output sequence is incorrect.
It would be greatly appreciated if someone could point me in the correct direction, and let me know what the problem is!

Output:
There is two possible approaches to forward:

step through all layers with first elements, after that step through time (this one taken in snippet above)
sequentially step through layers, computing output (or equivalently input for next layer) for each index

So while first one good for RNN with one direction, it's not working properly for bidirectional multi-layered RNN. To illustrate, let's take 2 layers (same case in code) -- for computation output[0] input from previous layer is needed, and it's concatenation of:

hidden vector from normal pass of length 1 (because it's right at start of sequence)
and hidden vector from reverse pass of length seq_length (need to step through whole sequence, from end to start, to get it)

So when step through layers is taken first, it takes only one step in time (length of pass equal 1), and therefore output[0] takes garbage as input, because second part is not correct (there was no "whole pass from end to start").
To overcome this issue, I'd suggest to rewrite cycles in forward from:
for t in range(input.shape[0]):
    ...
    for layer in range(self.num_layers):
        ...

to something like:
for layer in range(self.num_layers):
    ...
    for t in range(input.shape[0]):
        ...

As alternative, leave forward for computing in other normal cases, but for bidirectional multilayer write another function forward_bidir, and write this cycles there.
It also worth noting, that w_ih[k] for k > 0 in bidirectional case have shape (hidden_size, 2 * hidden_size), as stated in pytorch documentation on RNN. Also, function torch.allclose should serve better for testing outputs than prints.
For code fixes check gist, no optimisations were taken, main aim to preserve original idea, seems to work for all configuration listed above (one-directional, multi-layered, bi-directional).


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (200x16 and 32x1)

Input:
I would like to connect two Fully Connected layers. Then, after connecting them, I want to build another neural net with the Fully Connected layer.
I can see that the error is caused by not setting cat_x = torch.cat([x, x1]) properly. However, I do not know how to solve this problem.
import torch
from torch import nn, optim
import numpy as np
from matplotlib import pyplot as plt

class Regression(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(2, 32)
        self.linear2 = nn.Linear(32, 16)
        self.linear3 = nn.Linear(32, 1)

    def forward(self, input):
        x = nn.functional.relu(self.linear1(input))
        x = nn.functional.relu(self.linear2(x))

        x1 = nn.functional.elu(self.linear1(input))
        x1 = nn.functional.elu(self.linear2(x1))

        cat_x = torch.cat([x, x1])

        cat_x = self.linear3(cat_x)

        return cat_x

def train(model, optimizer, E, iteration, x, y):
    losses = []
    for i in range(iteration):
        optimizer.zero_grad()                   # 勾配情報を0に初期化
        y_pred = model(x)                       # 予測
        loss = E(y_pred.reshape(y.shape), y)    # 損失を計算(shapeを揃える)
        loss.backward()                         # 勾配の計算
        optimizer.step()                        # 勾配の更新
        losses.append(loss.item())              # 損失値の蓄積
        print('epoch=', i+1, 'loss=', loss)
    return model, losses

def test(model, x):
    y_pred = model(x).data.numpy().T[0]  # 予測
    return y_pred

x = np.random.uniform(0, 10, 100)                                   # x軸をランダムで作成
y = np.random.uniform(0.9, 1.1, 100) * np.sin(2 * np.pi * 0.1 * x)  # 正弦波を作成
x = torch.from_numpy(x.astype(np.float32)).float()                  # xをテンソルに変換
y = torch.from_numpy(y.astype(np.float32)).float()                  # yをテンソルに変換
X = torch.stack([torch.ones(100), x], 1)                            # xに切片用の定数1配列を結合

net = Regression()

optimizer = optim.RMSprop(net.parameters(), lr=0.01)                # 最適化にRMSpropを設定
E = nn.MSELoss()
net, losses = train(model=net, optimizer=optimizer, E=E, iteration=5000, x=X, y=y)

error message
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849
   1850

RuntimeError: mat1 and mat2 shapes cannot be multiplied (200x16 and 32x1)

Output: since both x and x1 have dimensions of (100,16), the torch.cat operator concatenates in the 1st dimension (since they are of similar size in that direction). For your code to work change the cat_x = torch.cat([x, x1]) to cat_x = torch.cat([x, x1], dim=1)


Instruction: How to structure a cnn for fine-tuning?

Input:
I want to fine-tune a model so that I can experiment with various different hyper parameters. For example:

filters
Regularisation
Convolutional filter size
Learning rate
Optimizers

I have chose do do this in PyTorch and have created a base model (see below). However I am unsure of the best way to set up my code in order to do this. Specifically my ConvNet and train function. I will need to make comparisons as I go along using graphs. Could anyone offer any advice on the best way to structure my code/go about this.
class ConvNet(nn.Module):
  def __init__(self, num_classes=10):
    super().__init__()

    self.layer1 = nn.Sequential(
        nn.Conv2d(3, 16, kernel_size=3, stride=1),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )

    self.layer2 = nn.Sequential(
        nn.Conv2d(16, 32, kernel_size=3, stride=1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )

    self.fcl = nn.Sequential(
        nn.Flatten(),
        nn.Linear(1152, 100)
    )

  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.fcl(out)

    return out

Output:
If you're looking for a simple tutorial, PyTorch has one that is explained well for computer vision here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
This is section is most relevant to your question so you can see how fine-tuning is done when the optimizer is adjusted.
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

You have to train your model completely first. Then you can refer to this post: https://discuss.pytorch.org/t/how-to-perform-finetuning-in-pytorch/419/8?u=nullpointer
ignored_params = list(map(id, model.fc.parameters()))
base_params = filter(lambda p: id(p) not in ignored_params,
                     model.parameters())

optimizer = torch.optim.SGD([
            {'params': base_params},
            {'params': model.fc.parameters(), 'lr': opt.lr}
        ], lr=opt.lr*0.1, momentum=0.9)

If you'd like to change the weights of a layer without changing the weights of the network while you're fine-tuning, you can do something like the following:
    model = models.vgg16(pretrained=True)
    print list(list(model.classifier.children())[1].parameters())
    mod = list(model.classifier.children())
    mod.pop()
    mod.append(torch.nn.Linear(4096, 2))
    new_classifier = torch.nn.Sequential(*mod)
    print list(list(new_classifier.children())[1].parameters())
    model.classifier = new_classifier

If you're looking to add layers or filters to the current model:
class MyModel(nn.Module):
    def __init__(self, pretrained_model):
        self.pretrained_model = pretrained_model
        self.last_layer = ... # create layer

    def forward(self, x):
        return self.last_layer(self.pretrained_model(x))


Instruction: How can I send a Data Loader to the GPU in Google Colab?

Input:
I have two data loaders and I am trying to send them to the GPU using .to(device) but this is not working.
This is the code I am using:
# to create a batch iterator

class MyData(Dataset):
    def __init__(self, X, y):
        self.data = X
        self.target = y
        # TODO: convert this into torch code is possible
        self.length = [ np.sum(1 - np.equal(x, 0)) for x in X]

    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        x_len = self.length[index]
        xx_len = torch.tensor(x_len)
        return {"src": x, "trg": y, "x_len": xx_len}

    def __len__(self):
        return len(self.data)

dataset = DataLoader(train_dataset, batch_size = BATCH_SIZE,
                     drop_last=True,
                     shuffle=True)
test_Dataset= DataLoader(val_dataset, batch_size = BATCH_SIZE,
                     drop_last=True,
                     shuffle=True)

I also tried to use pin_memory = True but this is not working too.

Output:
You do not move the dataloader to the GPU. Instead, create batch tensors that store the data and then move these tensors to the GPU.
train_dataloader = DataLoader(MyData, batch_size=BS)
...
def train(nn, optim, train_dataloader, etc...):
    for batch in train_dataloader:
        batch = batch.to('cuda')
    ...


Instruction: How do non-image datatypes such as 3D objects, audio, video, etc work with Activeloop Hub?

Input:
I was using Hub, a dataset format for AI that allows data streaming to GPUs without sacrificing performance.
I have been using Hub for image datasets and would like to try to use the product for other data types.
How would Hub work for different data types such as 3D objects, audio, video, etc?
The following Activeloop Hub doc has an example of how to upload image datasets to Hub and I am using a similar approach for working with my image dataset.

Output:
You can currently upload any type of data as uncompressed arrays. Simply don't assign the Hub type (htype) which is a property of each tensor that gives info to the Hub and Activeloop Platform about how to optimally store, parse, and visualize Hub datasets.
If you do this you'll be able to .append any array you like, as long as len(ds.tensor[i].numpy().shape) is equal for all samples. So, the samples don't have to have the same shape, but they need to have the same number of dimensions.
Going forward, Activeloop Hub will be adding support for application-specific multi-dimensional htypes such as video, image, and audio, etc. These will include an appropriate compression.
Hub has htype docs that can help you understand dimensionality & conventions for each type.


Instruction: Weight tensor should be defined either for all 1000 classes or no classes but got weight tensor of shape: [5]

Input:
I'm trying to use VGG16 for ** 5 classes data set**.
I've already added 5 new layers to adjust the output for logit as 5.
model = models.vgg16(pretrained=True) #Downloads the vgg16 model which is pretrained on Imagenet dataset.
#Replace the Final layer of pretrained vgg16 with 5 new layers.
model.fc = nn.Sequential(nn.Linear(1000,512),
                         nn.ReLU(inplace=True),
                         nn.Linear(512,256),
                         nn.ReLU(inplace=True),
                         nn.Linear(256,128),
                         nn.ReLU(inplace=True),
                         nn.Linear(128,64),
                         nn.ReLU(inplace=True),
                         nn.Linear(64,5),
                    )

And my loss function is as follows
loss_fn   = nn.CrossEntropyLoss(weight=class_weights) #CrossEntropyLoss with class_weights.

where class_weights is defined as such
from sklearn.utils import class_weight #For calculating weights for each class.
class_weights = class_weight.compute_class_weight(class_weight='balanced',classes=np.array([0,1,2,3,4]),y=train_df['level'].values)
class_weights = torch.tensor(class_weights,dtype=torch.float).to(device)

print(class_weights) #Prints the calculated weights for the classes.

output: tensor([0.2556, 4.6000, 1.5333, 9.2000, 9.2000], device='cuda:0')
After first epoch I get the error given below.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [15], in <cell line: 5>()
      3 nb_epochs = 3
      4 #Call the optimize function.
----> 5 train_losses, valid_losses = optimize(train_dataloader,valid_dataloader,model,loss_fn,optimizer,nb_epochs)

Input In [14], in optimize(train_dataloader, valid_dataloader, model, loss_fn, optimizer, nb_epochs)
     21 print(f'\nEpoch {epoch+1}/{nb_epochs}')
     22 print('-------------------------------')
---> 23 train_loss = train(train_dataloader,model,loss_fn,optimizer, epoch) #Calls the train function.
     24 train_losses.append(train_loss)
     25 valid_loss = validate(valid_dataloader,model,loss_fn) #Calls the validate function.

Input In [12], in train(dataloader, model, loss_fn, optimizer, epoch)
     24 for batch,(x,y) in enumerate(dataloader): #Iterates through the batches.
     26     output = model(x.to(device)) #model's predictions.
---> 27     loss   = loss_fn(output,y.to(device)) #loss calculation.
     29     running_loss += loss.item()
     31     total        += y.size(0)

File ~/anaconda3/envs/Ammar/lib/python3.9/site-packages/torch/nn/modules/module.py:1113, in Module._call_impl(self, *input, **kwargs)
   1109 # If we don't have any hooks, we want to skip the rest of the logic in
   1110 # this function, and just call forward.
   1111 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1112         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1113     return forward_call(*input, **kwargs)
   1114 # Do not call functions when jit is used
   1115 full_backward_hooks, non_full_backward_hooks = [], []

File ~/anaconda3/envs/Ammar/lib/python3.9/site-packages/torch/nn/modules/loss.py:1163, in CrossEntropyLoss.forward(self, input, target)
   1162 def forward(self, input: Tensor, target: Tensor) -> Tensor:
-> 1163     return F.cross_entropy(input, target, weight=self.weight,
   1164                            ignore_index=self.ignore_index, reduction=self.reduction,
   1165                            label_smoothing=self.label_smoothing)

File ~/anaconda3/envs/Ammar/lib/python3.9/site-packages/torch/nn/functional.py:2961, in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2959 if size_average is not None or reduce is not None:
   2960     reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2961 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)

RuntimeError: weight tensor should be defined either for all 1000 classes or no classes but got weight tensor of shape: [5]

Output:
I faced the same problem as you. I started by changing the size of my final classifier layer (I copied the code from here):
model = models.mobilenet_v2(pretrained=True)

last_item_index = len(model.classifier) - 1
old_fc = model.classifier.__getitem__(last_item_index )
new_fc = nn.Linear(in_features=old_fc.in_features,
                   out_features= 129, bias=True)
model.classifier.__setitem__(last_item_index , new_fc)

After changing this, I printed the model architecture using the following code:
from torchsummary import summary
summary(model, (3, 224, 224))

And it's working (number of classes in my dataset is 129):
(classifier): Sequential(
  (0): Dropout(p=0.2, inplace=False)
  (1): Linear(in_features=1280, out_features=129, bias=True)
)


Instruction: AttributeError in Classification report

Input:
The thing is i want to output the precisio, recall, and f1-score using classification report. But when i run below code, that error occurs. How can i fix the AttributeError?
print(classification_report(test.targets.cpu().numpy(),
  File "C:\Users\Admin\PycharmProjects\ImageRotation\venv\lib\site-packages\torch\utils\data\dataset.py", line 83, in __getattr__
    raise AttributeError
AttributeError

This is where i load the data from my directory.
data_loader = ImageFolder(data_dir,transform = transformer)

lab = data_loader.classes
num_classes = int(len(lab))
print("Number of Classes: ", num_classes)

print("The classes are as follows : \n",data_loader.classes)

batch_size = 128
train_size = int(len(data_loader) * 0.8)
test_size = len(data_loader) - train_size

train,test = random_split(data_loader,[train_size,test_size])

train_size = int(len(train) * 0.8)
val_size = len(train) - train_size
train_data, val_data = random_split(train,[train_size,val_size])
#load the train and validation into batches.
print(f"Length of Train Data : {len(train_data)}")
print(f"Length of Validation Data : {len(val_data)}")
print(f"Length of Test Data : {len(test)}")

train_dl = DataLoader(train_data, batch_size, shuffle = True)
val_dl = DataLoader(val_data, batch_size*2)
test_dl = DataLoader(test, batch_size, shuffle=True)

model.evaL() code
with torch.no_grad():
    # set the model in evaluation mode
    model.eval()

    # initialize a list to store our predictions
    preds = []
    # loop over the test set
    for (x, y) in test_dl:
        # send the input to the device
        x = x.to(device)
        # make the predictions and add them to the list
        pred = model(x)
        preds.extend(pred.argmax(axis=1).cpu().numpy())
# generate a classification report
print(classification_report(test.targets.cpu().numpy(),
                            np.array(preds), target_names=test.classes))

Output:
It seems, ImageFolder is the your dataset object, but that is not inherited ted from torch.utils.data.Datasets.
torch Dataloader tries to call the __getitem__ method in the your Dataset object, but since it is not torch.utils.data.Dataset object it does not have has a function, then that causes to AttributeError now you are getting.
Convert ImageFolder to torch torch dataset. For further library details : torch doc
Practical implemtation : ast_dataloader
Also, you can use freeze model [without back propagation] to speedup the inference process.
   with torch.no_grad():
        # make the predictions and add them to the list
        pred = model(x)

Update>
sample torch dataset:
from torch.utils.data import Dataset
class Dataset_train(Dataset):
    def __init__(self, list_IDs, labels, base_dir):
        """self.list_IDs    : list of strings (each string: utt key),
           self.labels      : dictionary (key: utt key, value: label integer)"""
        self.list_IDs = list_IDs
        self.labels = labels
        self.base_dir = base_dir

    def __len__(self):
        return len(self.list_IDs)

    def __getitem__(self, index):
        key = self.list_IDs[index]
        X, _ = get_sample(f"{self.base_dir}/{key}", self.noises)
        y = self.labels[index]
        return X, y

[Note] get_sample is custom build function for .wav file read. you could replace it with any funtion.
torch example-1
torch example-2
medium example


Instruction: Get mean squared error between every element in two tensors

Input:
I have two single dimensional tensors, y_pred and y_true
where:
>>> y_pred.shape
torch.Size([2730441, 1])
>>> y_true.shape
torch.Size([2730441, 1])

To get the Mean Squared Error between the two tensors I can use
torch.nn.MSELoss() However, I want to get the loss between each row/ element in the tensors y_pred & y_true i.e. I want to run some function elementWiseMSE(y_pred, y_true) which will return loss_tensor of shape [2730441, 1] which contains the elementwise mean error of all the predictions.

Output:
The "function" you are looking for is literally
loss_tensor = (y_pred - y_true) ** 2


Instruction: Length of train image dataset and train image loader are different?

Input:
image_datasets Is a dictionary containing both train and test data.
Code below:
transforms= transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

image_datasets = {'train': datasets.CIFAR10(root=data_dir, train=True, download=True,
                            transform=transforms),
                  'test': datasets.CIFAR10(root=data_dir, train=False, download=True,
                            transform=transforms)
                  }

image_datasets

OUTPUT:
{'test': Dataset CIFAR10
     Number of datapoints: 10000
     Root location: ../Data
     Split: Test
     StandardTransform
 Transform: Compose(
                ToTensor()
                Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
            ), 'train': Dataset CIFAR10
     Number of datapoints: 50000
     Root location: ../Data
     Split: Train
     StandardTransform
 Transform: Compose(
                ToTensor()
                Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
            )}

#CREATING DATA LOADER
data_loaders = {
    'train': DataLoader(image_datasets['train'], 10, shuffle=True),
    'test': DataLoader(image_datasets['test'], 10)}

When I call len(data_loaders['train']) it returns
5000

When defining my data loader I am using a batch_size=10. Is the length of my data_loader being divided by my batch_size. New to coding and just wanted to double check.

Output:
In brief,

len(data_loaders['train'].dataset)gives you how many instances in the dataset e.g., 50000 in CIFAR10.
len(data_loaders['train']) gives you the number of batches in this dataloader e.g., 5000 in CIFA10 if your batch_size=10. Number of batches is calculated by len(dataset)/(batch_size).

Therefore, when we calculate the accuracy in each epoch we divide number of correct ones over len(data_loaders['train'].dataset) not len(data_loaders['train']) because I fixed this bug in Stack Overflow for someone whose accuracy goes beyond 100% and the cause is division over len(data_loaders['train']) .


Instruction: rearrange an array element based on list in python

Input:
I have got a 2D array a with size 2, 1403 and a list b which has 2 list.
a.shape = (2, 1403) # a is 2D array, each row has got unique elements.
len(b) = 2 # b is list
len(b[0]), len(b[1]) = 415, 452 # here also both the list inside b has got unique elements
all the elements present in b[0] and b[1] is present in a[0] and a[1] respectively
Now i want to rearrange elements of a based on elements of b. I want to rearrange such that all the elements in b[0] which is also present in a[0] should come in the ending of a[0], meaning new a should be such that a[0][:-len(b[0])] = b[0], similarly a[1][:-len(b[1])] = b[1].
Toy Example
a has got elements like [[1,2,3,4,5,6,7,8,9,10,11,12],[1,2,3,4,5,6,7,8,9,10,11,12]
b has got elements like [[5, 9, 10], [2, 6, 8, 9, 11]]
new_a becomes [[1,2,3,4,6,7,8,11,12,5,9,10], [1,3,4,5,7,10,12,2,6,8,9,11]]
I have written a code which loops over all the element which becomes very slow, it's shown below
a_temp = []
remove_temp = []
for i, array in enumerate(a):
    a_temp_inner = []
    remove_temp_inner = []
    for element in array:
        if element not in b[i]:
            a_temp_inner.append(element) # get all elements first which are not present in b
        else:
            remove_temp_inner.append(element) #if any element present in b, remove it from main array

    a_temp.append(a_temp_inner)
    remove_temp.append(b_temp_inner)

a_temp = torch.tensor(a_temp)
remove_temp = torch.tensor(remove_temp)
a = torch.cat((a_temp, remove_temp), dim = 1)

Can anyone please help me with some faster implementation that works better than this

Output:
Assuming a is a np.array, b is a list you can use
np.array([np.concatenate((i[~np.in1d(i, j)], j)) for i, j in zip(a,b)])

Output
array([[ 1,  2,  3,  4,  6,  7,  8, 11, 12,  5,  9, 10],
       [ 1,  3,  4,  5,  7, 10, 12,  2,  6,  8,  9, 11]])

Can be micro-optimized if b contains empty lists
np.array([np.concatenate((i[~np.in1d(i, j)], j)) if j else i for i, j in zip(a,b)])

In my benchmarks, for np.arrays with less than ~100 elements converting .tolist() is faster than np.concatenate
np.array([i[~np.in1d(i, j)].tolist() + j for i, j in zip(a,b)])

Data example and imports for this solution
import numpy as np

a = np.array([
        [1,2,3,4,5,6,7,8,9,10,11,12],
        [1,2,3,4,5,6,7,8,9,10,11,12]
    ])
b = [[5, 9, 10],
     [2, 6, 8, 9, 11]]


Instruction: Numpy: Filter segments in array based on how much overlap there is with a second array with different segmentation type

Input:
I am looking for the most computationally efficient way to filter out arrays from one array, based on segment overlap in a second array, and that array has a different segmentation type.
This is my first array
iob = np.array(
    [0, 1, 2, 2, 2, 2, 2, 2, 2, 0, 0, 1, 2, 2, 2, 2, 0, 0, 1, 2, 1, 2, 2, 0]
)

The number 1 is the start of each segment, the number 2 is the rest of the segment, and the number 0 indicates no segment. So for this array, the segments are 1, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 1, 2, and 1, 2, 2.
This is my second array
output = np.array(
    [0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0]
)

The segments are only defined by the number 1, and 0 indicates no segment. So the segments for this array are 1, 1, 1, 1, 1, 1, and 1, 1, 1.
In the first array, I want filter out segments whose contents to not overlap by at least 50% by a segment in the 2nd array. In other words, if at least half of the contents in a segment in the first array overlaps with a segment in the 2nd array, I want to keep that segment.
So this is the desired result
array([0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0])

I am looking for the most computationally efficient method to calculate this solution.
Current solution
I can get the segment indices using the technique described here https://stackoverflow.com/a/71297663/3259896
output_zero = np.concatenate((output, [0]))
zero_output = np.concatenate(([0], output))
iob_zero = np.concatenate((iob, [0]))

iob_starts = np.where(iob_zero == 1)[0]
iob_ends = np.where((iob_zero[:-1] != 0) & (iob_zero[1:] != 2))[0]
iob_pairs = np.column_stack((iob_starts, iob_ends))

output_starts = np.where((zero_output[:-1] == 0) & (zero_output[1:] == 1))[0]
output_ends = np.where((output_zero[:-1] == 1) & (output_zero[1:] == 0))[0]
output_pairs = np.column_stack((output_starts, output_ends))

Next, I directly compare all possible combination of segments to see which ones have at least a 50% overlap, and only keep those segments.
valid_pairs = []
for o_p in output_pairs:
    for i_p in iob_pairs:
        overlap = 1 + min(o_p[1], i_p[1]) - max(o_p[0], i_p[0])
        if overlap > np.diff(i_p)[0]/2:
            valid_pairs.append(i_p)
valid_pairs = np.array(valid_pairs)

Finally, I used the filtered indices to create the desired array
final = np.zeros_like(output)
for block in valid_pairs:
   final[block[0]:block[1]+1] = 1
final

array([0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0,
       0, 0])

I suspect that this may not be the most computationally efficient solution. It uses quite a few lines of code, and also used a nested for loop to do all possible comparisons, and uses another loop to create the desired array. I don't have a mastery on use of all of numpy's functions, and I am wondering if there is a more computationally efficient way to calculate this.

Output:
Here's a 4-liner:
output = np.concatenate((output, [0]))
iob = np.concatenate((iob, [0]))
idx = np.dstack(np.where(iob == 1) + tuple(np.array(np.where(np.diff(iob) < 0)) + 1)).flatten()
desired_array = np.concatenate([np.full(len(x), 1 if ((np.sum(x==2) >= np.sum(x==1)) and x.sum()>0) else 0) for x in np.split(np.where(iob>0, 1, 0) + output, idx)])[0:-1]

Output:
>>> desired_array
array([0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0])

And here is a more verbose version

output_0 = np.concatenate((output, [0]))
iob_0 = np.concatenate((iob, [0]))
overlap = np.where(iob_0>0, 1, 0) + output_0

iob_segment_starts = np.where(iob_0 == 1)
iob_segment_ends = tuple(np.array(np.where(np.diff(iob_0) < 0)) + 1)
iob_segment_indxs = np.dstack(iob_segment_starts + iob_segment_ends).flatten()
overlap_segments = np.split(overlap, iob_segment_indxs)
filtered_segment_params = (
    (
        len(x),
        np.sum(x == 2) >= np.sum(x == 1) and x.sum() > 0,
    )
    for x in overlap_segments
)
filtered_segment_pieces = [
    np.full(length, value, dtype=int)
    for length, value in filtered_segment_params
]
filtered_array = np.concatenate(filtered_segment_pieces)[:-1]
filtered_array

Pytorch Version
import torch
output = torch.tensor(
    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1]
)
iob = torch.tensor(
    [0, 1, 2, 2, 2, 2, 2, 2, 2, 0, 0, 1, 2, 2, 2, 2, 0, 0, 1, 2, 1, 2, 2, 1]
)
output_0 = torch.cat((output, torch.tensor([0])))
iob_0 = torch.cat((iob, torch.tensor([0])))
overlap = torch.where(iob_0>0, 1, 0) + output_0

iob_segment_starts = torch.where(iob_0 == 1)[0]
iob_segment_ends = torch.where(torch.diff(iob_0) < 0)[0] + 1
iob_segment_cuts = torch.cat([
    iob_segment_starts,
    iob_segment_ends,
    torch.tensor([0, overlap.shape[0]])
]).sort()[0]
iob_segment_sizes = torch.diff(iob_segment_cuts)
overlap_segments = torch.split(overlap, iob_segment_sizes.tolist(), dim=0)

filtered_segment_params = (
    (
        len(x),
        torch.sum(x == 2) >= torch.sum(x == 1) and x.sum() > 0,
    )
    for x in overlap_segments
)
filtered_segment_pieces = [
    torch.full((1, length), value, dtype=int).flatten(0)
    for length, value in filtered_segment_params
]
filtered_array = torch.cat(filtered_segment_pieces)[:-1]
filtered_array


Instruction: reorder columns in a tensor according to a dictionary

Input:
I don't know how to explain it correctly, so the title might be misleading.
What I want to do is to move columns from a 3d tensor t1 to another 3d tensor t2 according to the indices. There's a dictionary td, and a (k,v) pair in td means that kth column of t1 will be the vth column of t2
Currently, I'm doing it this way:
for k,v in td.items():
    t2[:,:,v] = torch.select(t1, 2, k)

but yes, it's super slow, as there are millions of them.
What would be the best way to do the work?

Output:
Assuming no repeated values then you can use
t2[:,:,list(td.values())] = t1[:,:,list(td.keys())]


Instruction: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([16, 2])) is deprecated

Input:
I am trying to build a multiclass text classification using Pytorch and torchtext. but I am receiving this error whenever output in last hidden layer is 2, but running fine on 1 outputdim. I know there is a problem with batchsize and Data shape. What to do? I don't know the fix.
Constructing iterator:
#set batch size
BATCH_SIZE = 16

train_iterator, valid_iterator = BucketIterator.splits(
    (train_data, valid_data),
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.text),
    sort_within_batch=True,
    device = device)

Model class:
class classifier(nn.Module):

    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,
                 bidirectional, dropout):
        super(classifier,self).__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)

        self.gru = nn.GRU(embedding_dim,
                           hidden_dim,
                           num_layers=n_layers,
                           bidirectional=bidirectional,
                           dropout=dropout,
                           batch_first=True)

        self.fc1 = nn.Linear(hidden_dim * 2, 128)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(128, 64)
        self.relu2 = nn.ReLU()
        self.fc3 = nn.Linear(64, 16)
        self.relu3 = nn.ReLU()
        self.fc4 = nn.Linear(16, output_dim)
        self.act = nn.Sigmoid()

    def forward(self, text, text_lengths):

        embedded = self.embedding(text)
        #embedded = [batch size, sent_len, emb dim]
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'),batch_first=True)

        packed_output, hidden = self.gru(packed_embedded)

        hidden = torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)

        dense_1=self.fc1(hidden)
        x = self.relu1(dense_1)
        x = self.fc2(x)
        x = self.relu2(x)
        x = self.fc3(x)
        x = self.relu3(x)
        dense_outputs = self.fc4(x)

        #Final activation function
        outputs=self.act(dense_outputs)

        return outputs

instantiating the model:
size_of_vocab = len(TEXT.vocab)
embedding_dim = 300
num_hidden_nodes = 256
num_output_nodes = 2
num_layers = 4
bidirection = True
dropout = 0.2

model = classifier(size_of_vocab, embedding_dim, num_hidden_nodes,num_output_nodes, num_layers,
                   bidirectional = True, dropout = dropout).to(device)

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

pretrained_embeddings = TEXT.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)

print(pretrained_embeddings.shape)

Optimizer and criterion used:
optimizer = optim.Adam(model.parameters())
criterion = nn.BCELoss()
model = model.to(device)
criterion = criterion.to(device)

Training function:
import torchmetrics as tm
metrics = tm.Accuracy()

def train(model, iterator, optimizer, criterion):

    #initialize every epoch
    epoch_loss = 0
    epoch_acc = 0

    #set the model in training phase
    model.train()

    for batch in iterator:

        #resets the gradients after every batch
        optimizer.zero_grad()

        #retrieve text and no. of words
        text, text_lengths = batch.text

        #convert to 1D tensor
        predictions = model(text, text_lengths).squeeze()

        #compute the loss
        loss = criterion(predictions, batch.label)

        #compute the binary accuracy
        # acc = binary_accuracy(predictions, batch.label)
        acc = metrics(predictions,batch.label)

        #backpropage the loss and compute the gradients
        loss.backward()

        #update the weights
        optimizer.step()

        #loss and accuracy
        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Full error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-60-eeabf5bacadf> in <module>()
      5
      6     #train the model
----> 7     train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
      8
      9     #evaluate the model

3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2906         raise ValueError(
   2907             "Using a target size ({}) that is different to the input size ({}) is deprecated. "
-> 2908             "Please ensure they have the same size.".format(target.size(), input.size())
   2909         )
   2910

ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([16, 2])) is deprecated. Please ensure they have the same size.

Output: What you want is CrossEntropyLoss instead of BCELoss.


Instruction: Is it possible to use PyTorch's `BatchNorm1d` with `BCELossWithLogits`?

Input:
I am attempting to normalize the outputs of my classifier that uses BCELossWithLogits as part of its loss function. As far as I know, this implements Sigmoid function internally and outputs the loss.
I want to normalize the output of the sigmoid function prior to calculating the loss. Is it possible to use BatchNorm1d with BCELossWithLogits? Or is passing the output tensor to torch.sigmoid to BatchNorm1d and separately calculating BCELoss the only possible solution?
Thanks.

Output:
You can use BCELoss instead of BCELossWithLogits which is described as:

This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss

For example,
m = nn.Sigmoid()
bn = nn.BatchNorm1d(3)
loss = nn.BCELoss()
input = torch.randn((2, 3), requires_grad=True)
target = torch.empty(2, 3).random_(2)
output = loss(m(bn(input)), target)
output.backward()


Instruction: Will the gradients be recorded when finding the model's accuracy in pytorch?

Input:
I starting to learn PyTorch and I am confused about something. From what I understand, if we set .requires_grad_() to our parameters then the necessary calculations to find the gradients of those parameters will be recorded. This way, we can perform gradient descent. However, the gradient values will be added on top of the previous gradient values, so after we perform a gradient descent step we should reset our gradients using param.grad.zero_(), where param is a weight or a bias term. I have a model which has just the input layer and one output neuron, so really simple stuff (since I have just one output neuron you can tell that I only have 2 possible classes). I have my parameters called weights and bias, it is on these 2 variables that I set requires_grad_(). Also I put my training data in a DataLoader called train_dl and my validation data in valid_dl. I use a subset of the MNIST dataset, but that is really not important to this question. These are the functions I use:
def forward_propagation(xb):
    z = xb @ weights + bias
    a = z.sigmoid()
    return a

def mse_loss(predictions, targets):
    loss = ((predictions - targets) ** 2).mean()
    return loss

def backward_propagation(loss):
    loss.backward()
    weights.data -= lr * weights.grad.data
    weights.grad.zero_()
    bias.data -= lr * bias.grad.data
    bias.grad.zero_()

def train_epoch():
    for xb, yb in train_dl:
        a = forward_propagation(xb)
        loss = mse_loss(a, yb)
        backward_propagation(loss)

As you can see I use the function train_epoch() to perform: forward propagation (where some of the calculations for the gradient will be recorded since that is where our parameters are first used), calculate the loss (this step will also be used to calculate the gradients), and then backward propagation where I update my parameters and then reset the gradients to 0, so that they don't accumulate. I used this code to train my model and it worked fine, I am satisfied with the accuracy I got. So I assume that it works, at least somewhat.
But I also use this code to find the validation data accuracy for my model:
def valid_accuracy():
    accuracies = []
    for xb, yb in valid_dl:
        a = forward_propagation(xb)
        correct = (a > 0.5) == yb
        accuracies.append(correct.float().mean())
    return round(torch.stack(accuracies).mean().item(), 4)

As you can see, in finding the model's accuracy I perform forward propagation (the above function, where I multiply the weights by the data and add the bias). My question is: will the gradients also be recorded here? So the next time when I use .backward() on loss will the gradients be influenced by the steps taken in finding the accuracy? I think that as it is right now, the gradient values will be added each time I find the accuracy of the model (which I do not want and doesn't make sense), but I am not sure. Should I have somewhere in the function valid_accuracy() another 2 lines with weights.grad.zero_() and bias.grad.zero_() so that this doesn't happen? Or is it the case that this doesn't happen automatically, so I get the desired behavior by default and I simply misunderstood something?

Output:
There are two things to consider: One is the gradients itself, and the other is the computational graph that is being built in each forward pass.
To compute the gradient after a forward pass, we need to record what operations have been done to what tensors in what order, that is, the computation graph. So whenever whe compute a new tensor form other tensors that have requires_grad==True, the new tensor has an attribute .grad_fn that points to the previous operation and the involved tensors. This is basically how backward() "knows" where to go. If you call backward() it will consider the this .grad_fn and recursively do the backward pass.
So currently the way you do it will actually build this computation graph, even when computing the accuracy. But if this graph is never accessed, the garbage collector will eventually destroy it.
The key thing to notice is that each separate evaluation will produce a new computation graph (depending on your model maybe sharing some parts), but the backward pass will only start from the "node" from which you called .backward, so in your snippets you won't ever get a gradient from the accuracy computation as you never call a.backward(), you only ever call loss.backward().
The recoding of the computation graph does require some overhead though, but this can be disabled using the torch.no_grad() context manager, which is made with this exact use case in mind. Unfortunately the name (as well as the documentation) mention the gradient, but it really is just about recording the (forward) computation graph. But obviously if you disable that, as a consequence you won't be able to compute a backward pass either.


Instruction: I have the code below which I want to translate into pytorch. I'm looking for a way to translate np.vectorize to any pytorch way in this case

Input:
I need to translate this code to pytorch. The code given below use np.vectorize. I am looking for a pytorch equivalent for this.
class SimplexPotentialProjection(object):
    def __init__(self, potential, inversePotential, strong_convexity_const, precision = 1e-10):
        self.inversePotential = inversePotential
        self.gradPsi = np.vectorize(potential)
        self.gradPsiInverse = np.vectorize(inversePotential)
        self.precision = precision
        self.strong_convexity_const = strong_convexity_const

Output:
The doc for numpy.vectorize clearly states that:

The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.

Therefore, in order to convert your numpy code to pytorch you'll simply need apply potential and inversePotential in a loop over their tensor arguments.
However, that might be very inefficient. You would better re-implement your functions to act "natively" in a vectorized manner on tensors.


Instruction: I do not split well in pytorch

Input:
I would like to do a tensor split in pytorch. However, I get an error message because I can't get the splitting to work.
The behavior I want is to split the input data into two Fully Connected layers. I then want to create a model that combines the two Fully Connected layers into one.
I believe the error is due to a wrong code in x1, x2 = torch.tensor_split(x,2)
import torch
from torch import nn, optim
import numpy as np
from matplotlib import pyplot as plt

class Regression(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(1, 32)
        self.linear2 = nn.Linear(32, 16)
        self.linear3 = nn.Linear(16*2, 1)

    def forward(self, x):
        x1, x2 = torch.tensor_split(x,2)
        x1 = nn.functional.relu(self.linear1(x1))
        x1 = nn.functional.relu(self.linear2(x1))

        x2 = nn.functional.relu(self.linear1(x2))
        x2 = nn.functional.relu(self.linear2(x2))
        cat_x = torch.cat([x1, x2], dim=1)

        cat_x = self.linear3(cat_x)
        return cat_x

def train(model, optimizer, E, iteration, x, y):
    losses = []
    for i in range(iteration):
        optimizer.zero_grad()                   # 勾配情報を0に初期化
        y_pred = model(x)                       # 予測
        loss = E(y_pred.reshape(y.shape), y)    # 損失を計算(shapeを揃える)
        loss.backward()                         # 勾配の計算
        optimizer.step()                        # 勾配の更新
        losses.append(loss.item())              # 損失値の蓄積
        print('epoch=', i+1, 'loss=', loss)
    return model, losses

x = np.random.uniform(0, 10, 100)                                   # x軸をランダムで作成
y = np.random.uniform(0.9, 1.1, 100) * np.sin(2 * np.pi * 0.1 * x)  # 正弦波を作成
x = torch.from_numpy(x.astype(np.float32)).float()                  # xをテンソルに変換
y = torch.from_numpy(y.astype(np.float32)).float()                  # yをテンソルに変換
X = torch.stack([torch.ones(100), x], 1)

net = Regression()

optimizer = optim.RMSprop(net.parameters(), lr=0.01)                # 最適化にRMSpropを設定
E = nn.MSELoss()                                                    # 損失関数にMSEを設定

net, losses = train(model=net, optimizer=optimizer, E=E, iteration=5000, x=X, y=y)

error message
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849
   1850

RuntimeError: mat1 and mat2 shapes cannot be multiplied (50x2 and 1x32)

Output:
Tl;dr
Specify dim=1 in torch.tensor_split(x,2) .
Explanation
The x comes from two tensors with the shape [100,1] stacked at dim 1, so its shape is [100, 2]. After applying tensor_split, you get two tensors both with shape [50, 2].
print(x.shape) # torch.Size([100, 2])
print(torch.tensor_split(X,2)[0].shape) # torch.Size([50, 2])

The error occurred because linear1 only accepts tensors with shape [BATCH_SIZE,1] as the input, but a tensor with shape [50, 2] was passed in.
If your intention was to split the array of random numbers and the array of all ones, change torch.tensor_split(x,2) to torch.tensor_split(x,2,dim=1), which produces two tensors with the shape [100,1].


Instruction: I can't import torchvision

Input:
I installed torchvision0.12.0, python3.8 and my OS is Windows.
I succeeded in importing torch, but I couldn't import torchvision and getting this error.
ImportError
DLL load failed while importing _imaging:
File "C:\Users'MyName'\Documents\GitHub\pytorch-cifar\main.py", line 8, in
import torchvision
Is there someone who can solve this problem?

Output:
This may be due to incompatible versions of torch and torchvision.You can get the information you want through the following link:
the corresponding torchvision versions and supported Python versions


Instruction: How to know the trained model is correct?

Input:
I use PyTorch Lightning for model training, during which I use ModelCheckpoint to save loading points. Finally, I would like to know whether the model is loaded correctly. Let me know if you require further information?
checkpoint_callback = ModelCheckpoint(
        filename='tb1000_{epoch: 02d}-{step}',
        monitor='val/acc@1',
        save_top_k=5,
        mode='max')

wandb_logger = pl.loggers.wandb.WandbLogger(
        name=run_name,
        project=args.project,
        entity=args.entity,
        offline=args.offline,
        log_model='all')

model = BYOL(**args.__dict__, num_classes=dm.num_classes)

trainer = pl.Trainer.from_argparse_args(args,
         logger=wandb_logger, callbacks=[checkpoint_callback])

trainer.fit(model, dm)

# Loading and testing
model_test = BYOL(**args.__dict__, num_classes=dm.num_classes)
path = "/tb100_epoch= 819-step=39359.ckpt"
model_test.load_from_checkpoint(path)

Output:
load_from_checkpoint() will return a model with trained weights, so you need to assign it to a new variable.
model_test = model_test.load_from_checkpoint(path)

or
model_test = BYOL.load_from_checkpoint(path)


Instruction: What does Tensor[batch_mask, ...] do?

Input:
I saw this line of code in an implementation of BiLSTM:
batch_output = batch_output[batch_mask, ...]

I assume this is some kind of "masking" operation, but found little information on Google about the meaning of ....  Please help:).
Original Code:
class BiLSTM(nn.Module):
    def __init__(self, vocab_size, tagset, embedding_dim, hidden_dim,
                 num_layers, bidirectional, dropout, pretrained=None):
         # irrelevant code ..........

    def forward(self, batch_input, batch_input_lens, batch_mask):
        batch_size, padding_length = batch_input.size()
        batch_input = self.word_embeds(batch_input)  # size: #batch * padding_length * embedding_dim
        batch_input = rnn_utils.pack_padded_sequence(
            batch_input, batch_input_lens, batch_first=True)
        batch_output, self.hidden = self.lstm(batch_input, self.hidden)
        self.repackage_hidden(self.hidden)
        batch_output, _ = rnn_utils.pad_packed_sequence(batch_output, batch_first=True)
        batch_output = batch_output.contiguous().view(batch_size * padding_length, -1)

        #######  HERE  ##########
        batch_output = batch_output[batch_mask, ...]
        #########################

        out = self.hidden2tag(batch_output)
        return out

Output:
I assume that batch_mask is a boolean tensor. In that case, batch_output[batch_mask] performs a boolean indexing that selects the elements corresponding to True in batch_mask.
... is usually referred as ellipsis, and in the case of PyTorch (but also other NumPy-like libraries), it is a shorthand for avoiding repeating the column operator (:) multiple times. For example, given a tensor v, with v.shape equal to (2, 3, 4), the expression v[1, :, :] can be rewritten as v[1, ...].
I performed some tests and using either batch_output[batch_mask, ...] or batch_output[batch_mask] seems to work identically:
t = torch.arange(24).reshape(2, 3, 4)

# mask.shape == (2, 3)
mask = torch.tensor([[False, True, True], [True, False, False]])

print(torch.all(t[mask] == t[mask, ...]))  # returns True


Instruction: How to load a fine tuned pytorch huggingface bert model from a checkpoint file?

Input:
I had fine tuned a bert model in pytorch and saved its checkpoints via torch.save(model.state_dict(), 'model.pt')
Now When I want to reload the model, I have to explain whole network again and reload the weights and then push to the device.
Can anyone tell me how can I save the bert model directly and load directly to use in production/deployment?
Following is the training code and you can try running there in colab itself! After training completion, you will notice in file system we have a checkpoint file. But I want to save the model itself.
LINK TO COLAB NOTEBOOK FOR SAMPLE TRAINING
Following is the current inferencing code I written.
import torch
import torch.nn as nn
from transformers import AutoModel, BertTokenizerFast
import numpy as np
import json

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
device = torch.device("cpu")

class BERT_Arch(nn.Module):

    def __init__(self, bert):
        super(BERT_Arch, self).__init__()

        self.bert = bert

        # dropout layer
        self.dropout = nn.Dropout(0.1)

        # relu activation function
        self.relu = nn.ReLU()

        # dense layer 1
        self.fc1 = nn.Linear(768, 512)

        # dense layer 2 (Output layer)
        self.fc2 = nn.Linear(512, 2)

        # softmax activation function
        self.softmax = nn.LogSoftmax(dim=1)

    # define the forward pass
    def forward(self, sent_id, mask):
        # pass the inputs to the model
        _, cls_hs = self.bert(sent_id, attention_mask=mask, return_dict=False)

        x = self.fc1(cls_hs)

        x = self.relu(x)

        x = self.dropout(x)

        # output layer
        x = self.fc2(x)

        # apply softmax activation
        x = self.softmax(x)

        return x

bert = AutoModel.from_pretrained('bert-base-uncased')
model = BERT_Arch(bert)
path = './models/saved_weights_new_data.pt'
model.load_state_dict(torch.load(path, map_location=device))
model.to(device)

def inference(comment):
    tokens_test = tokenizer.batch_encode_plus(
        list([comment]),
        max_length=75,
        pad_to_max_length=True,
        truncation=True,
        return_token_type_ids=False
    )
    test_seq = torch.tensor(tokens_test['input_ids'])
    test_mask = torch.tensor(tokens_test['attention_mask'])
    predictions = model(test_seq.to(device), test_mask.to(device))
    predictions = predictions.detach().cpu().numpy()
    predictions = np.argmax(predictions, axis=1)
    return predictions

I simply want to save a model from this notebook in a way such that I can use it for inferencing anywhere.

Output:
Just save your model using model.save_pretrained, here is an example:
model.save_pretrained("<path_to_dummy_folder>")

You can download the model from colab, save it on your gdrive or at any other location of your choice. While doing inference, you can just give path to this model (you may have to upload it) and start with inference.
To load the model
model = AutoModel.from_pretrained("<path_to_saved_pretrained_model>")

#Note: Instead of AutoModel class, you may use the task specific class as well.


Instruction: What's an elegant way to compare one-hot label results?

Input:
I'm trying to calculate the 'accuracy' of one-hot label encoded tensors, such that for the following example, I'd get 0.5.
tensor([[0,0,1], [1,0,0]]) == tensor([[0,0,1], [0,1,0]])

I want to know what proportion of the predictions are correctly labelled.
What's the most elegant way to achieve this with a pytorch tensor?

Output:
If I understand correctly. You want all values to match for each row to be considered as a correct prediction then it should be something like this.
(tensor([[0,0,1], [1,0,0]]) == tensor([[0,0,1], [0,1,0]])).all(dim=1).float().mean()


Instruction: invalid syntax when copying weight in pytorch

Input:
I tried this:
model2.down1.maxpool_conv.1.double_conv.0.weight.copy_(state_dict['layer1.0.conv1.weight'])

But I got below error message:
model2.down1.maxpool_conv.1.double_conv.0.weight.copy_(state_dict['layer1.0.conv1.weight'])
                             ^
SyntaxError: invalid syntax

But my model output show its has "down1.maxpool_conv.1.double_conv.0.weight"

Output:
Python does not allow numbers to be attributes, so if you somehow create an object that has such attributes (it can be done only by low level function calls) you will not have access to them through dot notation (and this is why you get an invalid syntax error, rather than something like Attribute not found error).
You should instead take the param reference that you already have in your loop and do the copy
for name, param in model2.named_parameters():
  if name == 'down1.maxpool_conv.1.double_conv.0.weight':
     param.copy_(state_dict['layer1.0.conv1.weight'])


Instruction: For loop optimization to create an adjacency matrix

Input:
I am currently working with graph with labeled edges.
The original adjacency matrix is a matrix with shape [n_nodes, n_nodes, n_edges] where each cell [i,j, k] is 1 if node i and j are connected via edge k.
I need to create a reverse of the original graph, where nodes become edges and edges become nodes, so i need a new matrix with shape [n_edges, n_edges, n_nodes], where each cell [i,j,k] is 1 if edges i and j have k as a common vertex.

The following code correctly completes the task, but the use of 5 nested for-loops is too slow, to process the amount of graphs with which I have to work seems to take about 700 hours.
Is there a better way to implement this?
n_nodes = extended_adj.shape[0]
n_edges = extended_adj.shape[2]
reversed_graph = torch.zeros(n_edges, n_edges, n_nodes, 1)
for i in range(n_nodes):
    for j in range(n_nodes):
        for k in range(n_edges):
            #If adj_mat[i][j][k] == 1 nodes i and j are connected with edge k
            #For this reason the edge k must be connected via node j to every outcoming edge of j
            if extended_adj[i][j][k] == 1:
                #Given node j, we need to loop through every other possible node (l)
                for l in range(n_nodes):
                    #For every other node, we need to check if they are connected by an edge (m)
                    for m in range(n_edges):
                        if extended_adj[j][l][m] == 1:
                            reversed_graph[k][m][j] = 1

Thanks is advance.

Output:
Echoing the comments above, this graph representation is almost certainly cumbersome and inefficient. But that notwithstanding, let's define a  vectorized solution without loops and that uses tensor views whenever possible, which should be fairly efficient to compute for larger graphs.
For clarity let's use [i,j,k] to index G (original graph) and [i',j',k'] to index G' (new graph). And let's shorten n_edges to e and n_nodes to n.
Consider the 2D matrix slice = torch.max(G,dim = 1). At each coordinate [a,b] of this slice, a 1 indicates that node a is connected by edge b to some other node (we don't care which).
slice = torch.max(G,dim = 1)                                     # dimension [n,e]

We're well on our way to the solution, but we need an expression that tells us whether a is connected to edge b and another edge c, for all edges c. We can map all combinations b,c by expanding slice, copying it and transposing it, and looking for intersections between the two.
expanded_dim = [slice.shape[0],slice.shape[1],slice.shape[1]]    # value [n,e,e]

# two copies of slice, expanded on different dimensions
expanded_slice = slice.unsqueeze(1).expand(expanded_dim)         # dimension [n,e,e]
transpose_slice = slice.unsqueeze(2).expand(expanded_dim)        # dimension [n,e,e]

G = torch.bitwise_and(expanded_slice,transpose_slice).int()      # dimension [n,e,e]

G[i',j',k'] now equals 1 iff node i' is connected by edge j' to some other node, AND  node i' is connected by edge k' to some other node. If j' = k' the value is 1 as long as one of the endpoints of that edge is i'.
Lastly, we reorder dimensions to get to your desired form.
G = torch.permute(G,(1,2,0))           # dimension [e,e,n]


Instruction: config a neural network with a JSON file

Input:
I want to config a neural network model (number of layers, number of neurons per layer, activation functions, ...) from a JSON file. But honestly, I have no idea how to do it. When I search the internet with "hyperparameters config PyTorch" or "hyperparameters tunning PyTorch" can't find anything interesting. The search result is more about hyperparameters optimization, not config from JSON.
Anyone has any idea how to do this (JSON file config) or do you know any useful tutorials that I can watch/read, please? That would be a great help!
Thank you in advance

Output:
Write your parameters to json file with proper name like following
{
   "number_layers":1,
   "number_neurons":2,
   "activation_function":"relu",
   "training":{
      "learning_rate":0.01
   }
}

Then read the json file
import json

with open('xxx.json', 'r', encoding='utf-8') as f:
    config = json.loads(f.read())

and access the parameter you want with
config['number_layers']
config['training']['learning_rate']


Instruction: torch model.load_state_dict *** AttributeError: 'ModelName' object has no attribute 'copy'

Input:
i previously saved a model like this:
trainedmodelpath = "model.th"
torch.save({'model': model,
            'scaler': scaler,
            'encoder': label_encoder,
            'config': config_parameters},
            trainedmodelpath)

But when i try to load it like this:
PreviousModelPath = "model.th"
TorchLoadedState = torch.load(PreviousModelPath)
TorchLoadedState_Model = TorchLoadedState['model']
model.load_state_dict(TorchLoadedState_Model)

i received this error
*** AttributeError: 'ModelName' object has no attribute 'copy'

Output:
Save model weigths with model.state_dict() instead:
torch.save({'model': model.state_dict(),
            'scaler': scaler,
            'encoder': label_encoder,
            'config': config_parameters},
            trainedmodelpath)


Instruction: How to reorder tensor based on indexes tensor from the same size

Input:
Say I have tensor A, and indexes Tensor: A = [1, 2, 3, 4], indexes = [1, 0, 3, 2]
I want to create a new Tensor from these two with the following result : [2, 1, 4, 3]
Each element of the result is element from A and the order is defined by the indexes Tensor.
Is there a way to do it with PyTorch tensor ops without loops?
My goal is to do it for 2D Tensor, but I don't think there is a way to do it without loops, so I thought to project it to 1D, do the work and project it back to the 2D.

Output:
You can use scatter:
A = torch.tensor([1, 2, 3, 4])
indices = torch.tensor([1, 0, 3, 2])
result = torch.tensor([0, 0, 0, 0])
print(result.scatter_(0, indices, A))


Instruction: DiffSharp sliding window implementation example

Input:
According to this issue Check view operations correspond to torch #199
It seems like it is not hard to implement sliding window function with DiffSharp Tensor. However I cannot get a hint searching DiffSharp official website.
In PyTorch, the unfold is like:
x = torch.arange(1., 20)
x.unfold(0,4,2)

tensor([[ 1.,  2.,  3.,  4.],
        [ 3.,  4.,  5.,  6.],
        [ 5.,  6.,  7.,  8.],
        [ 7.,  8.,  9., 10.],
        [ 9., 10., 11., 12.],
        [11., 12., 13., 14.],
        [13., 14., 15., 16.],
        [15., 16., 17., 18.]])

How do I correctly implement unfold operator with DiffSharp Tensor?

Output:
Using unfold provided by TensorSharp instead works fine.
Implement miscellaneous utility functions


Instruction: In Pytorch, how can i shuffle a DataLoader?

Input:
I have a dataset with 10000 samples, where the classes are present in an ordered manner. First I loaded the data into an ImageFolder, then into a DataLoader, and I want to split this dataset into a train-val-test set. I know the DataLoader class has a shuffle parameter, but thats not good for me, because it only shuffles the data when enumeration happens on it. I know about the RandomSampler function, but with it, i can only take n amount of data randomly from the dataset, and i have no control of what is being taken out, so one sample might be present in the train,test and val set at the same time.
Is there a way to shuffle the data in a DataLoader? The only thing i need is the shuffle, after that i can subset the data.

Output:
The Subset dataset class takes indices (https://pytorch.org/docs/stable/data.html#torch.utils.data.Subset). You can probably exploit that to get this functionality as below. Essentially, you can get away by shuffling the indices and then picking the subset of the dataset.
# suppose dataset is the variable pointing to whole datasets
N = len(dataset)

# generate & shuffle indices
indices = numpy.arange(N)
indices = numpy.random.permutation(indices)
# there are many ways to do the above two operation. (Example, using np.random.choice can be used here too

# select train/test/val, for demo I am using 70,15,15
train_indices = indices [:int(0.7*N)]
val_indices = indices[int(0.7*N):int(0.85*N)]
test_indices = indices[int(0.85*N):]

train_dataset = Subset(dataset, train_indices)
val_dataset = Subset(dataset, val_indices)
test_dataset = Subset(dataset, test_indices)


Instruction: Saving the weight of one layer in Pytorch

Input:
I would like to save the weight of a model, but not the whole model like this:
torch.save(model, 'model.pth')

But rather, just one layer. for example, suppose, I have defined one layer like this:
self.conv_up3 = convrelu(256 + 512, 512, 3, 1)

How do I save the weight of only this layer. And also how do I load it for this layer.

Output:
You can do the following to save/get parameters of the specific layer:
specific_params = self.conv_up3.state_dict()
# save/manipulate `specific_params` as you want

And similarly, to load the params to that specific layer:
self.conv_up3.load_state_dict(params)

You can do this  because each layer is a neural network (nn.Module instance) in itself.