Instruction: RuntimeError: Input, output and indices must be on the current device. (fill_mask("Random text .")

Input:
I am getting "RuntimeError: Input, output and indices must be on the current device."
when I run this line.
fill_mask("Auto Car .")
I am running it on Colab.
My Code:
from transformers import BertTokenizer, BertForMaskedLM
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer
from transformers import BertTokenizer, BertForMaskedLM

paths = [str(x) for x in Path(".").glob("**/*.txt")]
print(paths)

bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

from transformers import BertModel, BertConfig

configuration = BertConfig()
model = BertModel(configuration)
configuration = model.config
print(configuration)

model = BertForMaskedLM.from_pretrained("bert-base-uncased")

from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=bert_tokenizer,
    file_path="./kant.txt",
    block_size=128,
)

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=bert_tokenizer, mlm=True, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./KantaiBERT",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
    )

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=bert_tokenizer
)

fill_mask("Auto Car <mask>.")

The last line is giving me the error mentioned above. Please let me know what I am doing wrong or what I have to do in order to remove this error.

Output:
The trainer trains your model automatically at GPU (default value no_cuda=False). You can verify this by running:
model.device

after training. The pipeline does not this and this leads to the error you see (i.e. your model is on your GPU but your example sentence is on your CPU). You can fix that by either run the pipeline with GPU support as well:
fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=bert_tokenizer,
    device=0,
)

or by transferring your model to CPU before initializing the pipeline:
model.to('cpu')


Instruction: What does x[x!=x] mean?

Input:
I don't understand this line:
lprobs[lprobs != lprobs] = torch.tensor(-math.inf).to(lprobs)

There is no comment, so is it some well-known Python (or PyTorch?) idiom? Could someone explain what it means, or show a different way that makes the intent clearer?
lprobs is a pytorch Tensor, and it could contain any size float type (I doubt this code is intended to support int or complex types). As far as I know, the Tensor classes don't override the __ne__ function.

Output:
It's a combination of fancy indexing with a boolean mask, and a "trick" (although intended by design) to check for NaN: x != x holds iff x is NaN (for floats, that is).
They could alternatively have written
lprobs[torch.isnan(lprobs)] = torch.tensor(-math.inf).to(lprobs)

or, probably even more idiomatically, used torch.nan_to_num (but beware that the latter also has special behaviour towards infinities).
A non-updating variant of the above would be
torch.where(torch.isnan(lprobs), torch.tensor(-math.inf), lprobs)


Instruction: Can I unpack an `nn.ModuleList` inside `nn.Sequential`?

Input:
I am parametrizing the number of hidden layers of a simple ANN using nn.ModuleList. I am wondering if passing this list into a nn.Sequential module as follows would lead to any adverse effects with the execution graph.
nn.Sequential is not necessary, however it seems cleaner to me to have the entire architecture explicit in the constructor.
class ANN(nn.Module):

    def __init__(
        self,
        in_feats=3,
        in_hidden=5,
        n_hidden_layers=3,

    ):
        super(ANN, self).__init__()

        # ====== dynamically register hidden layers ======
        self.hidden_linears = nn.ModuleList()
        for i in range(n_hidden_layers):
            self.hidden_linears.append(nn.Linear(in_hidden, in_hidden))
            self.hidden_linears.append(nn.ReLU())

        # ====== sequence of layers ======
        self.layers = nn.Sequential(
            nn.Linear(in_feats, in_hidden),
            nn.ReLU(),
            *self.hidden_linears,
            nn.Linear(in_hidden, 1),
            nn.Sigmoid(),
        )

    def forward(self, X):
        return self.layers(X)

Also open to suggestions of neater ways to put this together.

Output: You could use nn.Sequential.modules() to get every single part of the Sequential and then pack them into a list.


Instruction: Pytorch nll_loss returning a constant loss during training loop

Input:
I've an image binary classification problem which i want to classify weather an image is of an ant or bee. I've scraped the images and i did all the cleaning, reshaping, converting to grayscale. The images are of size 200x200 one channel grayscale. I first wanted to solve this problem using Feed Forwad NN before i jump to Conv Nets..
My problem during the training loop  I am getting a constant loss I am using Adam Optimizer, F.log_softmax for the last layer in the network as well as the nll_loss function. My code so far looks as follows:
FF - Network
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(in_features , 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 2)

    def forward(self, X):
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = F.relu(self.fc3(X))
        X = F.log_softmax(self.fc4(X), dim=1)
        return X

net = Net()

Training loop.
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)
EPOCHS = 10
BATCH_SIZE = 5
for epoch in range(EPOCHS):
    print(f'Epochs: {epoch+1}/{EPOCHS}')
    for i in range(0, len(y_train), BATCH_SIZE):
        X_batch = X_train[i: i+BATCH_SIZE].view(-1,200 * 200)
        y_batch = y_train[i: i+BATCH_SIZE].type(torch.LongTensor)

        net.zero_grad() ## or you can say optimizer.zero_grad()

        outputs = net(X_batch)
        loss = F.nll_loss(outputs, y_batch)
        loss.backward()
        optimizer.step()
    print("Loss", loss)

I am suspecting that the problem maybe with my batching and the loss function. I will appreciate any help. Note: The images are gray-scale images of shape (200, 200).

Output:
I've been waiting for answers, but i couldn't even get a comment. I figured out myself the solution, maybe this can help someone in the future.
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(200 * 200 , 64) # 200 * 200 are in_features, which is an image of shape 200*200 (gray image)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 2)

    def forward(self, X):
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = F.relu(self.fc3(X))
        X = self.fc4(X) # I removed the activation function here,
        return X

net = Net()

# I changed the loss function to CrossEntropyLoss() since i didn't apply activation function on the last layer

loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)

EPOCHS = 10
BATCH_SIZE = 5
for epoch in range(EPOCHS):
    print(f'Epochs: {epoch+1}/{EPOCHS}')
    for i in range(0, len(y_train), BATCH_SIZE):
        X_batch = X_train[i: i+BATCH_SIZE].view(-1, 200 * 200)
        y_batch = y_train[i: i+BATCH_SIZE].type(torch.LongTensor)

        net.zero_grad() ## or you can say optimizer.zero_grad()

        outputs = net(X_batch)
        loss = loss_function(outputs, y_batch)
        loss.backward()
        optimizer.step()
    print("Loss", loss)


Instruction: Python(PyTorch): TypeError: string indices must be integers

Input:
I have written the following code to train a bert model on my dataset but when I execute it I get an error at the part where I implement tqdm. I have written the entire training code below with full description of the error. How to fix this?
Code
Model
TRANSFORMERS = {
    "bert-multi-cased": (BertModel, BertTokenizer, "bert-base-uncased"),
}

class Transformer(nn.Module):
    def __init__(self, model, num_classes=1):
        """
        Constructor

    Arguments:
        model {string} -- Transformer to build the model on. Expects "camembert-base".
        num_classes {int} -- Number of classes (default: {1})
    """
    super().__init__()
    self.name = model

    model_class, tokenizer_class, pretrained_weights = TRANSFORMERS[model]

    bert_config = BertConfig.from_json_file(MODEL_PATHS[model] + 'bert_config.json')
    bert_config.output_hidden_states = True

    self.transformer = BertModel(bert_config)

    self.nb_features = self.transformer.pooler.dense.out_features

    self.pooler = nn.Sequential(
        nn.Linear(self.nb_features, self.nb_features),
        nn.Tanh(),
    )

    self.logit = nn.Linear(self.nb_features, num_classes)

def forward(self, tokens):
    """
    Usual torch forward function

    Arguments:
        tokens {torch tensor} -- Sentence tokens

    Returns:
        torch tensor -- Class logits
    """
    _, _, hidden_states = self.transformer(
        tokens, attention_mask=(tokens > 0).long()
    )

    hidden_states = hidden_states[-1][:, 0] # Use the representation of the first token of the last layer

    ft = self.pooler(hidden_states)

    return self.logit(ft)

Training
def fit(model, train_dataset, val_dataset, epochs=1, batch_size=8, warmup_prop=0, lr=5e-4):

    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)

    optimizer = AdamW(model.parameters(), lr=lr)

    num_warmup_steps = int(warmup_prop * epochs * len(train_loader))
    num_training_steps = epochs * len(train_loader)

    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)

    loss_fct = nn.BCEWithLogitsLoss(reduction='mean').cuda()

    for epoch in range(epochs):
        model.train()
        start_time = time.time()

        optimizer.zero_grad()
        avg_loss = 0

        for step, (x, y_batch) in tqdm(enumerate(train_loader), total=len(train_loader)):

            y_pred = model(x.to(device))

            loss = loss_fct(y_pred.view(-1).float(), y_batch.float().to(device))
            loss.backward()
            avg_loss += loss.item() / len(train_loader)

            xm.optimizer_step(optimizer, barrier=True)
            #optimizer.step()
            scheduler.step()
            model.zero_grad()
            optimizer.zero_grad()

        model.eval()
        preds = []
        truths = []
        avg_val_loss = 0.

        with torch.no_grad():
            for x, y_batch in tqdm(val_loader):
                y_pred = model(x.to(device))
                loss = loss_fct(y_pred.detach().view(-1).float(), y_batch.float().to(device))
                avg_val_loss += loss.item() / len(val_loader)

                probs = torch.sigmoid(y_pred).detach().cpu().numpy()
                preds += list(probs.flatten())
                truths += list(y_batch.numpy().flatten())
            score = roc_auc_score(truths, preds)

        dt = time.time() - start_time
        lr = scheduler.get_last_lr()[0]
        print(f'Epoch {epoch + 1}/{epochs} \t lr={lr:.1e} \t t={dt:.0f}s \t loss={avg_loss:.4f} \t val_loss={avg_val_loss:.4f} \t val_auc={score:.4f}')

Error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<timed eval> in <module>

<ipython-input-19-e47eae808597> in fit(model, train_dataset, val_dataset, epochs, batch_size, warmup_prop, lr)
     22         for step, (x, y_batch) in tqdm(enumerate(train_loader), total=len(train_loader)):
     23
---> 24             y_pred = model(x.to(device))
     25
     26             loss = loss_fct(y_pred.view(-1).float(), y_batch.float().to(device))

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    724             result = self._slow_forward(*input, **kwargs)
    725         else:
--> 726             result = self.forward(*input, **kwargs)
    727         for hook in itertools.chain(
    728                 _global_forward_hooks.values(),

<ipython-input-11-2002cc7ec843> in forward(self, tokens)
     41         )
     42
---> 43         hidden_states = hidden_states[-1][:, 0] # Use the representation of the first token of the last layer
     44
     45         ft = self.pooler(hidden_states)

TypeError: string indices must be integers

Output:
Your code is designed for an older version of the transformers library:
AttributeError: 'str' object has no attribute 'dim' in pytorch
As such you will need to either downgrade to version 3.0.0, or adapt the code to deal with the new-format output of bert.


Instruction: How to save and load nn.Parameter() in pytorch, such that one can continue optimisation over it?

Input:
I know how to store and load nn.Model, but can not find how to make a checkpoint for nn.Parameter. I tried this version, but the optimizer is not changing the nn.Parameter value after restoring.
from torch import nn as nn
import torch
from torch.optim import Adam

alpha = torch.ones(10)
lr = 0.001
alpha = nn.Parameter(alpha)
print(alpha)
alpha_optimizer = Adam([alpha], lr=lr)

for i in range(10):
   alpha_loss = - alpha.mean()
   alpha_optimizer.zero_grad()
   alpha_loss.backward()
   alpha_optimizer.step()
   print(alpha)
path = "./test.pt"
state = dict(alpha_optimizer=alpha_optimizer.state_dict(),
               alpha=alpha)
torch.save(state, path)
checkpoint = torch.load(path)
alpha = checkpoint["alpha"]
alpha_optimizer.load_state_dict(checkpoint["alpha_optimizer"])
for i in range(10):
   alpha_loss = - alpha.mean()
   alpha_optimizer.zero_grad()
   alpha_loss.backward()
   alpha_optimizer.step()
   print(alpha)

Output:
The problem is that the optimizer is still with a reference to the old alpha (check id(alpha) vs. id(alpha_optimizer.param_groups[0]["params"][0]) before the last for loop), while a new one is set when you load it from the checkpoint in alpha = checkpoint["alpha"].
You need to update the params of the optimizer before loading its state:
# ....
torch.save(state, path)
checkpoint = torch.load(path)

# here's where the reference of alpha changes, and the source of the problem
alpha = checkpoint["alpha"]

# reset optim
alpha_optimizer = Adam([alpha], lr=lr)
alpha_optimizer.load_state_dict(checkpoint["alpha_optimizer"])

for i in range(10):
   alpha_loss = - alpha.mean()
   alpha_optimizer.zero_grad()
   alpha_loss.backward()
   alpha_optimizer.step()
   print(alpha)


Instruction: Pytorch batchwise matrix vector rowwise multiplication

Input:
I am using pytorch.
If I have a matrix M of size (d1, d2) and a vector V of size d2, doing M*V gives me an output OUT of size (d1, d2), where each row of M has been multiplied by V.
I need to do the same thing batch-wise, where the matrix M is fixed and I have a batch of dB vectors.
In practice, given a tensor M of size (d1, d2) and a tensor V of size (dB, d2), I need to get as output OUT a tensor of size (dB, d1, d2), so that OUT[i] = M*V[i].
How can I efficiently get it with pytorch?

Output:
This simple trick works for the problem:
M.unsqueeze(0) * V.unsqueeze(1)

This does multiplications of tensors having shapes (1, d1, d2) and (dB, 1, d2) and you get the desired output having shape (dB, d1, d2).


Instruction: Given multiple prediction vectors, how to efficiently obtain the label with most votes (in numpy/pytorch)?

Input:
I have 3 vectors representing the 3 different predictions of labels for the same data:
P1=[31, 22, 11, 10,  9, 9, 0, 0, 23 ....]  # length over 1M
P2=[31, 22, 12, 10,  8, 9, 0, 0, 30 ....]  # length over 1M
P3=[30, 22, 12, 11,  8, 9, 0, 1, 31 ....]  # length over 1M

Ans= [31, 22, 12, 10, 8, 9, 0, 0, 23, ....]

The basic idea is if a prediction has the highest vote count (e.g. "31" has count 2 in the first column), we pick it, but if all candidates have a different vote (e.g. "23", "30", "31" in the last column), we can pick any one of them.
These vectors may be numpy array, list or pytorch tensor. Considering the length of such vector is over 1000,000, what's the most efficient way (mainly runtime) to find Ans?

Output:
You can take the mode of the tensors:
t = torch.tensor([P1,P2,P3])
t.mode(0).values

tensor([31, 22, 12, 10,  8,  9,  0,  0, 23])


Instruction: Training PyTorch models on different machines leads to different results

Input:
I am training the same model on two different machines, but the trained models are not identical. I have taken the following measures to ensure reproducibility:
# set random number
random.seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)

# set the cudnn
torch.backends.cudnn.benchmark=False
torch.backends.cudnn.deterministic=True

# set data loader work threads to be 0
DataLoader(dataset, num_works=0)

When I train the same model multiple times on the same machine, the trained model is always the same. However, the trained models on two different machines are not the same. Is this normal? Are there any other tricks I can employ?

Output:
There are a number of areas that could additionally introduce randomness e.g:

PyTorch random number generator
You can use torch.manual_seed() to seed the RNG for all devices (both CPU and CUDA):

CUDA convolution determinism
While disabling CUDA convolution benchmarking (discussed above) ensures that CUDA selects the same algorithm each time an application is run, that algorithm itself may be nondeterministic, unless either torch.use_deterministic_algorithms(True) or torch.backends.cudnn.deterministic = True is set. The latter setting controls only this behavior, unlike torch.use_deterministic_algorithms() which will make other PyTorch operations behave deterministically, too.
CUDA RNN and LSTM
In some versions of CUDA, RNNs and LSTM networks may have non-deterministic behavior. See torch.nn.RNN() and torch.nn.LSTM() for details and workarounds.
DataLoader
DataLoader will reseed workers following Randomness in multi-process data loading algorithm. Use worker_init_fn() to preserve reproducibility:

https://pytorch.org/docs/stable/notes/randomness.html


Instruction: PipelineException: No mask_token ([MASK]) found on the input

Input:
I am getting this error "PipelineException: No mask_token ([MASK]) found on the input"
when I run this line.
fill_mask("Auto Car .")
I am running it on Colab.
My Code:
from transformers import BertTokenizer, BertForMaskedLM
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer
from transformers import BertTokenizer, BertForMaskedLM

paths = [str(x) for x in Path(".").glob("**/*.txt")]
print(paths)

bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

from transformers import BertModel, BertConfig

configuration = BertConfig()
model = BertModel(configuration)
configuration = model.config
print(configuration)

model = BertForMaskedLM.from_pretrained("bert-base-uncased")

from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=bert_tokenizer,
    file_path="./kant.txt",
    block_size=128,
)

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=bert_tokenizer, mlm=True, mlm_probability=0.15
)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./KantaiBERT",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
    )

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=bert_tokenizer,
    device=0,
)

fill_mask("Auto Car <mask>.").     # This line is giving me the error...

The last line is giving me the error mentioned above. Please let me know what I am doing wrong or what I have to do in order to remove this error.
Complete error: "f"No mask_token ({self.tokenizer.mask_token}) found on the input","

Output:
Even if you have already found the error, a recommendation to avoid it in the future. Instead of calling
fill_mask("Auto Car <mask>.")

you can do the following to be more flexible when you use different models:
MASK_TOKEN = tokenizer.mask_token

fill_mask("Auto Car {}.".format(MASK_TOKEN))


Instruction: What is bias node in googlenet and how to remove it?

Input: I am new to deep learning, i want to build a model that can identify similar images, i am reading classification is a Strong Baseline for Deep Metric Learning research paper. and here is they used the phrase: "remove the bias term inthe last linear layer". i have no idea what is bias term is and how to remove it from googlenet or other pretrained models. if someone help me out with this it would be great! :)

Output:
To compute the layer n outputs, a linear neural network computes a linear combination of the layer n-1 output for each layer n output, adds a scalar constant value to each layer n output (the bias term), and then applies an activation function. In pytorch, one could disable the bias in a linear layer using:
layer = torch.nn.Linear(n_in_features, n_out_features, bias=False)

To overwrite the existing structure of, say, the googlenet included in torchvision.models defined here, you can simply override the last fully connected layer after initialization:
from torchvision.models import googlenet
num_classes = 1000 # or whatever you need it to be

model = googlenet(num_classes)
model.fc = torch.nn.Linear(1000,num_classes,bias = False)


Instruction: How can I add a module into a network without change the main body of the network?

Input:
How can I add a module into a network without change the main body of the network? For example, I want to add an attention module after the layer1 of Resnet. I have tried to use forward_hook to add the new module, but after reading the source code of nn.Module:
def __call__(self, *input, **kwargs):
   result = self.forward(*input, **kwargs)
   for hook in self._forward_hooks.values():
       hook_result = hook(self, input, result)
   ...
   return result

I found that hooks are done after forward, so I can't use hook to add new modules. Are there any other way to realize this function?

Output:
The key point is make a new forward function and ditch the old one. You can inherit the ResNet directly and just add a new layer. Then specific how the forward should be.
Like
class AttentionResNet(torchvision.models.resnet.ResNet):
    def __init__(self, *args, **kwargs):
        super(AttentionResNet, self).__init__(*args, **kwargs)
        self.attention = Attention()

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)

        # this is what you asked
        x = self.attention(x)

        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        return x

Or if you want pretrained weights you can also make the resnet as a sub module and don't call the resnet's forward directly, like.
class AttentionResNet(nn.Module):
    def __init__(self):
        super(AttentionResNet, self).__init__()
        self.attention = Attention()
        self.resnet = torchvision.models.resnet.resnet50()

    def forward(self, x):
        x = self.resnet.conv1(x)
        x = self.resnet.bn1(x)
        x = self.resnet.relu(x)
        x = self.resnet.maxpool(x)

        x = self.resnet.layer1(x)

        # this is what you want
        x = self.attention(x)

        x = self.resnet.layer2(x)
        x = self.resnet.layer3(x)
        x = self.resnet.layer4(x)

        x = self.resnet.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.resnet.fc(x)
        return x

Note that the forward in ResNet looks like this. And for me using forward directly doesn't cause problem but you can also use the _forward_impl() like original if you want to.
    def _forward_impl(self, x):
        # See note [TorchScript super()]
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)

        return x

    def forward(self, x):
        return self._forward_impl(x)


Instruction: How to add layers to a pretrained model in PyTorch?

Input:
I want to add layer normalization function just after AdaptiveAvgPool2d layer and L2 normalization after fc layer. i wanted my fc layer  output to be 200 so tried not to include fc layer instead of it make new fc layer, but it did not remove fc layers comes with pretrained model, i am using googlenet.
my code:
class GoogleNet(nn.Module):
    def __init__(self):
        super(GoogleNet,self).__init__()
        self.model = googlenet_pytorch.GoogLeNet.from_pretrained('googlenet')

        self.fc = nn.Linear(1024,200, bias=False),

    def forward(self, x):
        batch_size ,_,_,_ =x.shape
        x = self.model.extract_features(x)
        x = self.model._avg_pooling(x)
        x = F.layer_norm(x,x.size[1:],elementwise_affine=False)
        x = self.fc(x)
        x = F.normalize(x, p=2, dim=1)
        return x

Output I am getting:
      .....
      .....
      .....

(aux1): InceptionAux(
      (conv): BasicConv2d(
        (conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
      (fc1): Linear(in_features=2048, out_features=1024, bias=True)
      (fc2): Linear(in_features=1024, out_features=1000, bias=True)
    )
    (aux2): InceptionAux(
      (conv): BasicConv2d(
        (conv): Conv2d(528, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
      (fc1): Linear(in_features=2048, out_features=1024, bias=True)
      (fc2): Linear(in_features=1024, out_features=1000, bias=True)
    )
    (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
    (dropout): Dropout(p=0.2, inplace=False)
    (fc): Linear(in_features=1024, out_features=1000, bias=True)
  )
)

Output I want:
  ......
  ......
  ......

  (aux1): None
  (aux2): None
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  ** layer normalization here**
  (dropout): Dropout(p=0.2, inplace=False)
  (fc): Linear(in_features=1024, out_features=200, bias=False)
  **L2 normalization here**
)

If someone needs to know the solution code for this problem, with the help of iacob's answer I solved it, I added it as an answer.

Output:
When printing a model, PyTorch will print every layer up until it encounters an error in the forward, irrespective of whether the model will run (on appropriately formatted input data).
You have a number of issues in your code after loading the backbone GoogLeNet model, and hence all layers you have added after this fail to display when printed. The immediately obvious issues are:

you must remove the , after self.fc = nn.Linear(1024,200, bias=False), or it will be interpreted as a tuple
GoogLeNet has no attribute _avg_pooling
x.size is not subscriptable. Either use a function call x.size()[1:] or use x.shape[1:]
F.layer_norm() has no argument elementwise_affine

You need to correct these in order to get the model to run, like so.


Instruction: Pytorch transfer learning error: The size of tensor a (16) must match the size of tensor b (128) at non-singleton dimension 2

Input:
Currently, I'm working on an image motion deblurring problem with PyTorch. I have two kinds of images: Blurry images (variable = blur_image) that are the input image and the sharp version of the same images (variable = shar_image), which should be the output. Now I wanted to try out transfer learning, but I can't get it to work.
Here is the code for my dataloaders:
train_loader = torch.utils.data.DataLoader(train_dataset,
                                           batch_size=batch_size,
                                           shuffle = True)
validation_loader = torch.utils.data.DataLoader(valid_dataset,
                                                batch_size=batch_size,
                                                shuffle = False)
test_loader = torch.utils.data.DataLoader(test_dataset,
                                          batch_size=batch_size,
                                          shuffle = False)

Their shape:
Trainloader - Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Trainloader - Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32
Validationloader - Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Validationloader - Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32
Testloader- Shape of blur_image [N, C, H, W]:  torch.Size([16, 3, 128, 128])
Testloader- Shape of sharp_image [N, C, H, W]:  torch.Size([16, 3, 128, 128]) torch.float32

The way I use transfer learning (I thought that for the 'in_features' I have to put in the amount of pixels):
model = models.alexnet(pretrained=True)
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 128)
device_string = "cuda" if torch.cuda.is_available() else "cpu"
device = torch.device(device_string)
model = model.to(device)

The way I define my training process:
# Define the loss function (MSE was chosen due to the comparsion of pixels
# between blurred and sharp images
criterion = nn.MSELoss()

# Define the optimizer and learning rate
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Learning rate schedule - If the loss value does not improve after 5 epochs
# back-to-back then the new learning rate will be:  previous_rate*0.5

#scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
        optimizer,
        mode='min',
        patience=5,
        factor=0.5,
        verbose=True
    )

def training(model, trainDataloader, epoch):
  """ Function to define the model training

    Args:
        model (Model object): The model that is going to be trained.
        trainDataloader (Dataloader object): Dataloader object of the trainset.
        epoch (Integer): Number of training epochs.

  """
  # Changing model into trainings mode
  model.train()
  # Supporting variable to display the loss for each epoch
  running_loss = 0.0
  running_psnr = 0.0
  for i, data in tqdm(enumerate(trainDataloader),
                      total=int(len(train_dataset)/trainDataloader.batch_size)):
    blur_image = data[0]
    sharp_image = data[1]

    # Transfer the blurred and sharp image instance to the device
    blur_image = blur_image.to(device)
    sharp_image = sharp_image.to(device)

    # Sets the gradient of tensors to zero
    optimizer.zero_grad()
    outputs = model(blur_image)
    loss = criterion(outputs, sharp_image)

    # Perform backpropagation
    loss.backward()
    # Update the weights
    optimizer.step()

    # Add the loss that was calculated during the trainigs run
    running_loss += loss.item()

    # calculate batch psnr (once every `batch_size` iterations)
    batch_psnr =  psnr(sharp_image, blur_image)
    running_psnr += batch_psnr

  # Display trainings loss
  trainings_loss = running_loss/len(trainDataloader.dataset)
  final_psnr = running_psnr/int(len(train_dataset)/trainDataloader.batch_size)
  final_ssim = ssim(sharp_image, blur_image, data_range=1, size_average=True)
  print(f"Trainings loss: {trainings_loss:.5f}")
  print(f"Train PSNR: {final_psnr:.5f}")
  print(f"Train SSIM: {final_ssim:.5f}")

  return trainings_loss, final_psnr, final_ssim

And here is my way to start the training:
train_loss  = []
val_loss = []
train_PSNR_score  = []
train_SSIM_score  = []
val_PSNR_score  = []
val_SSIM_score  = []

start = time.time()
for epoch in range(nb_epochs):
    print(f"Epoch {epoch+1}\n-------------------------------")
    train_epoch_loss = training(model, train_loader, nb_epochs)
    val_epoch_loss = validation(model, validation_loader, nb_epochs)
    train_loss.append(train_epoch_loss[0])
    val_loss.append(val_epoch_loss[0])

    train_PSNR_score.append(train_epoch_loss[1])
    train_SSIM_score.append(train_epoch_loss[2])

    val_PSNR_score.append(val_epoch_loss[1])
    val_SSIM_score.append(val_epoch_loss[2])

    scheduler.step(train_epoch_loss[0])
    scheduler.step(val_epoch_loss[0])
end = time.time()
print(f"Took {((end-start)/60):.3f} minutes to train")

But every time when I want to perform the training I receive the following error:
 0%|          | 0/249 [00:00<?, ?it/s]Epoch 1
-------------------------------
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([16, 3, 128, 128])) that is different to the input size (torch.Size([16, 128])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-195-ff0214e227cd> in <module>()
      9 for epoch in range(nb_epochs):
     10     print(f"Epoch {epoch+1}\n-------------------------------")
---> 11     train_epoch_loss = training(model, train_loader, nb_epochs)
     12     val_epoch_loss = validation(model, validation_loader, nb_epochs)
     13     train_loss.append(train_epoch_loss[0])

<ipython-input-170-dfa2c212ad23> in training(model, trainDataloader, epoch)
     25     optimizer.zero_grad()
     26     outputs = model(blur_image)
---> 27     loss = criterion(outputs, sharp_image)
     28
     29     # Perform backpropagation

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    526
    527     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 528         return F.mse_loss(input, target, reduction=self.reduction)
    529
    530

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2926         reduction = _Reduction.legacy_get_string(size_average, reduce)
   2927
-> 2928     expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   2929     return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   2930

/usr/local/lib/python3.7/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     72     if has_torch_function(tensors):
     73         return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 74     return _VF.broadcast_tensors(tensors)  # type: ignore
     75
     76

RuntimeError: The size of tensor a (16) must match the size of tensor b (128) at non-singleton dimension 2

model structure:
AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=128, bias=True)
  )
)

I'm a newbie in terms of using Pytorch (and image deblurring in general) and so I rather confused about the meaning of the error message and how to fix it. I tried to change my parameters and nothing worked. Does anyone have any advice for me on how to solve this problem?
I would appreciate every input :)

Output:
Here your you can't use alexnet for this task. becouse output from your model and sharp_image should be shame. because convnet encode your image as enbeddings you and fully connected layers can not convert these images to its normal size you can not use fully connected layers for decoding, for obtain the same size you need to use ConvTranspose2d() for this task.
your encoder should be:
class ConvEncoder(nn.Module):
    """
    A simple Convolutional Encoder Model
    """

    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(3, 16, (3, 3), padding=(1, 1))
        self.relu1 = nn.ReLU(inplace=True)
        self.maxpool1 = nn.MaxPool2d((2, 2))

        self.conv2 = nn.Conv2d(16, 32, (3, 3), padding=(1, 1))
        self.relu2 = nn.ReLU(inplace=True)
        self.maxpool2 = nn.MaxPool2d((2, 2))

        self.conv3 = nn.Conv2d(32, 64, (3, 3), padding=(1, 1))
        self.relu3 = nn.ReLU(inplace=True)
        self.maxpool3 = nn.MaxPool2d((2, 2))

        self.conv4 = nn.Conv2d(64, 128, (3, 3), padding=(1, 1))
        self.relu4 = nn.ReLU(inplace=True)
        self.maxpool4 = nn.MaxPool2d((2, 2))

    def forward(self, x):
        # Downscale the image with conv maxpool etc.
        x = self.conv1(x)
        x = self.relu1(x)
        x = self.maxpool1(x)

        x = self.conv2(x)
        x = self.relu2(x)
        x = self.maxpool2(x)

        x = self.conv3(x)
        x = self.relu3(x)
        x = self.maxpool3(x)

        x = self.conv4(x)
        x = self.relu4(x)
        x = self.maxpool4(x)

        return x

And your decoder should be:
class ConvDecoder(nn.Module):
    """
    A simple Convolutional Decoder Model
    """

    def __init__(self):
        super().__init__()
        self.deconv1 = nn.ConvTranspose2d(256, 128, (2, 2), stride=(2, 2))
        self.relu1 = nn.ReLU(inplace=True)

        self.deconv2 = nn.ConvTranspose2d(128, 64, (2, 2), stride=(2, 2))
        self.relu2 = nn.ReLU(inplace=True)

        self.deconv3 = nn.ConvTranspose2d(64, 32, (2, 2), stride=(2, 2))
        self.relu3 = nn.ReLU(inplace=True)

        self.deconv4 = nn.ConvTranspose2d(32, 16, (2, 2), stride=(2, 2))
        self.relu4 = nn.ReLU(inplace=True)

    def forward(self, x):
         # Upscale the image with convtranspose etc.
        x = self.deconv1(x)
        x = self.relu1(x)

        x = self.deconv2(x)
        x = self.relu2(x)

        x = self.deconv3(x)
        x = self.relu3(x)

        x = self.deconv4(x)
        x = self.relu4(x)
        return x

encoder = ConvEncoder()
decoder = ConvDecoder()

You can train your model like that:
    encoder.train()
    decoder.train()

    for batch_idx, (train_img, target_img) in enumerate(train_loader):
        # Move images to device
        train_img = train_img.to(device)
        target_img = target_img.to(device)

        # Zero grad the optimizer
        optimizer.zero_grad()
        # Feed the train images to encoder
        enc_output = encoder(train_img)
        # The output of encoder is input to decoder !
        dec_output = decoder(enc_output)

        # Decoder output is reconstructed image
        # Compute loss with it and orginal image which is target image.
        loss = loss_fn(dec_output, target_img)
        # Backpropogate
        loss.backward()
        # Apply the optimizer to network by calling step.
        optimizer.step()
    # Return the loss
    return loss.item()

you might want visit this for getting help in your project.


Instruction: PyTorch Boolean - Stop Backpropagation?

Input:
I need to create a Neural Network where I use binary gates to zero-out certain tensors, which are the output of disabled circuits.
To improve runtime speed, I was looking forward to use torch.bool binary gates to stop backpropagation along disabled circuits in the network. However, I created a small experiment using the official PyTorch example for the CIFAR-10 dataset, and the runtime speed is exactly the same for any values for gate_A and gate_B: (this means that the idea is not working)
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.pool = nn.MaxPool2d(2, 2)
        self.conv1a = nn.Conv2d(3, 6, 5)
        self.conv2a = nn.Conv2d(6, 16, 5)
        self.conv1b = nn.Conv2d(3, 6, 5)
        self.conv2b = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(32 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Only one gate is supposed to be enabled at random
        # However, for the experiment, I fixed the values to [1,0] and [1,1]
        choice  =  randint(0,1)
        gate_A  =  torch.tensor(choice   ,dtype = torch.bool)
        gate_B  =  torch.tensor(1-choice ,dtype = torch.bool)

        a = self.pool(F.relu(self.conv1a(x)))
        a = self.pool(F.relu(self.conv2a(a)))

        b = self.pool(F.relu(self.conv1b(x)))
        b = self.pool(F.relu(self.conv2b(b)))

        a *= gate_A
        b *= gate_B
        x  = torch.cat( [a,b], dim = 1 )

        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How can i define gate_A and gate_B in such a way that backpropagation effectively stops if they are zero?
PS. Changing concatenation dynamically at runtime would also change which weights are assigned to every module. (for example, the weights associated to a could be assigned to b in another pass, disrupting how the network operates).

Output:
You could use torch.no_grad (the code below can probably be made more concise):
def forward(self, x):
        # Only one gate is supposed to be enabled at random
        # However, for the experiment, I fixed the values to [1,0] and [1,1]
        choice  =  randint(0,1)
        gate_A  =  torch.tensor(choice   ,dtype = torch.bool)
        gate_B  =  torch.tensor(1-choice ,dtype = torch.bool)

        if choice:
            a = self.pool(F.relu(self.conv1a(x)))
            a = self.pool(F.relu(self.conv2a(a)))
            a *= gate_A

            with torch.no_grad(): # disable gradient computation
                b = self.pool(F.relu(self.conv1b(x)))
                b = self.pool(F.relu(self.conv2b(b)))
                b *= gate_B
        else:
            with torch.no_grad(): # disable gradient computation
                a = self.pool(F.relu(self.conv1a(x)))
                a = self.pool(F.relu(self.conv2a(a)))
                a *= gate_A

            b = self.pool(F.relu(self.conv1b(x)))
            b = self.pool(F.relu(self.conv2b(b)))
            b *= gate_B

        x  = torch.cat( [a,b], dim = 1 )

        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

On a second look, I think the following is a simpler solution to the specific problem:
def forward(self, x):
        # Only one gate is supposed to be enabled at random
        # However, for the experiment, I fixed the values to [1,0] and [1,1]
        choice  =  randint(0,1)

        if choice:
            a = self.pool(F.relu(self.conv1a(x)))
            a = self.pool(F.relu(self.conv2a(a)))
            b = torch.zeros(shape_of_conv_output) # replace shape of conv output here
        else:
            b = self.pool(F.relu(self.conv1b(x)))
            b = self.pool(F.relu(self.conv2b(b)))
            a = torch.zeros(shape_of_conv_output) # replace shape of conv output here

        x  = torch.cat( [a,b], dim = 1 )

        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


Instruction: Can I use a convolution filter instead of a dense layer for clasification?

Input:
I was reading a decent paper S-DCNet and I fell upon a section (page3,table1,classifier) where a convolution layer has been used on the feature map in order to produce a binary classification output as part of an internal process. Since I am a noob and when someone talks to me about classification I automatically make a synapse relating to FCs combined with softmax, I started wondering ... Is this a possible thing to do? Can indeed a convolutional layer be used to classify a binary outcome? The whole concept triggered my imagination so much that I insist on getting answers...
Honestly, how does this actually work? What is the difference between using a convolution filter instead of a fully connected layer for classification purposes?
Edit (Uncertain answer on how does it work): I asked a colleague and he told me that using a filter of the same shape as the length-width shape of the feature map at the current stage, may lead to a learnable binary output (considering that you also reduce the #channels of the feature map to a single channel). But I still don't understand the motivations behind such a technique ..

Output:
Using convolutions as FCs can be done (for example) with filters of spatial size (1,1) and with depth of the same size as the FC input size.
The resulting feature map would be of the same size as the input feature map, but each pixel would be the output of a "FC" layer whose weights are the weights of the shared 1x1 conv filter.
This kind of thing is used mainly for semantic segmentation, meaning classification per pixel. U-net is a good example if memory serves.

Also see this.
Also note that 1x1 convolutions have other uses as well.
paperswithcode probably some of the nets there use this trick.


Instruction: Bert Tokenizer add_token function not working properly

Input:
tokenizer add_tokens is not adding new tokens.
Here is my code:
  from transformers import BertTokenizer

  bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

  new_tokens = []
  text = open("parsed_data.txt", "r")
  for line in text:
     for word in line.split():
        new_tokens.append(word)

  print(len(new_tokens))      # 53966
  print(len(bert_tokenizer))  # 36369
  bert_tokenizer.add_tokens(new_tokens)
  print(len(bert_tokenizer))  # 36369

Output:
Yes, if a token already exists, it is skipped. By the way, after changing the tokenizer you have to also update your model. See the last line below.
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(my_new_tokens)

model.resize_token_embeddings(len(bert_tokenizer))


Instruction: Multiply by the factor the number of times that the position appears in the array (Python)

Input:
I'm trying to get a "pythonic" way or using a library (anything except "for" loops) for doing this.
my_array = np.array([2,4,6])
my_array[[0,2,0]] = my_array[[0,2,0]] * 2

Expected result:
[8,4,12]

However, I get:
[4,4,12]

This doesn't work for me as I need to multiply twice the factor (2) for the position 0 if, through the index, the 0 position repeats n amount of times.
"For" loops can't be used as there are billions of data points

Output:
You could try:
>>> num_occurrences_list = [0,2,0]
>>> factor = 2
>>> my_array * np.power(factor, np.bincount(num_occurrences_list))
array([ 8,  4, 12])

But I am not sure if it provides any advantage over using for loops explicitly.
Here, np.bincount provides an array where each index contains the number of occurrences of it in your list. np.power with base 2 specifies how many times you want to multiply by 2. Finally, you perform the actual multiplication with the computed weights for each element of the array.


Instruction: How to construct batch that return equal number of images for per classes

Input:
I am working an image retrieval project, for making model more fair i want to construct batches that return:

5 images per class, and
75 images and per batch

I have total 300 classes in my dataset, so it obvious that only 15 classes of images can be contained in each batch.data is balanced this mean there is equal number of images for per class,I am using pytorch.
I have create pytorch dataset and I want to add above functionality in my ImageFolderLoader class whose code I added below.
IMG_EXTENSIONS = [
   '.jpg', '.JPG', '.jpeg', '.JPEG',
   '.png', '.PNG', '.ppm', '.PPM', '.bmp', '.BMP',
]

def is_image_file(filename):
    return any(filename.endswith(extension) for extension in IMG_EXTENSIONS)

def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    classes = [clss.split('.')[1] for clss in classes]
    return classes, class_to_idx

def make_dataset(dir, class_to_idx):
    images = []
    for target in os.listdir(dir):
        d = os.path.join(dir, target)
        if not os.path.isdir(d):
            continue
        for filename in os.listdir(d):
            if is_image_file(filename):
                path = '{0}/{1}'.format(target, filename)
                item = (path, class_to_idx[target])
                images.append(item)

    return images

def default_loader(path):
    return Image.open(path).convert('RGB')

class ImageFolderLoader(Dataset):
    def __init__(self, root, transform=None, loader=default_loader,):
        classes, class_to_idx = find_classes(root)
        imgs = make_dataset(root, class_to_idx)

        self.root = root
        self.imgs = imgs
        self.classes = classes
        self.class_to_idx = class_to_idx
        self.transform = transform
        self.loader = loader

    def __getitem__(self, index):
        path, target = self.imgs[index]
        img = self.loader(os.path.join(self.root, path))
        if self.transform is not None:
            img = self.transform(img)

        return img, target

    def __len__(self):
        return len(self.imgs)

if there is way to do this then please let me know>.
edit:- Anyone want to see solution for this, i added the solution below after solving this problem.

Output:
I solved the problem by including batch_sampler in DataLoader module. for this i used pytorch-balanced-sampler git project, which allows  awesome customization for batch_sampler, you should visit this repo.
My custom dataset:
IMG_EXTENSIONS = [
   '.jpg', '.JPG', '.jpeg', '.JPEG',
   '.png', '.PNG', '.ppm', '.PPM', '.bmp', '.BMP',
]

def is_image_file(filename):
    return any(filename.endswith(extension) for extension in IMG_EXTENSIONS)

def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    classes = [clss.split('.')[1] for clss in classes]
    return classes, class_to_idx

def make_dataset(dir, class_to_idx):
    images = []
    for target in os.listdir(dir):
        d = os.path.join(dir, target)
        if not os.path.isdir(d):
            continue
        for filename in os.listdir(d):
            if is_image_file(filename):
                path = '{0}/{1}'.format(target, filename)
                item = (path, class_to_idx[target])
                images.append(item)

    data_dict = {}
    for item in images:
        cls = item[1]
        if cls not in data_dict.keys():
            data_dict[cls] = [item]
        else:
            data_dict[cls].append(item)

    return images,data_dict

def default_loader(path):
    return Image.open(path).convert('RGB')

class ImageFolderLoader(Dataset):
    def __init__(self, root, transform=None, loader=default_loader):
        classes, class_to_idx = find_classes(root)
        imgs,instance_labels = make_dataset(root, class_to_idx)

        self.instance_labels = instance_labels

        self.root = root
        self.imgs = imgs
        self.classes = classes
        self.class_to_idx = class_to_idx
        self.transform = transform
        self.loader = loader

    def __getitem__(self, index):
        path, target = self.imgs[index]
        img = self.loader(os.path.join(self.root, path))
        if self.transform is not None:
            img = self.transform(img)

        return img, target

    def __len__(self):
        return len(self.imgs)

Then i used SamplerFactory class from pytorch-balances-sampler project,  you need to visit this repository for understand the parameters,
train_data = ImageFolderLoader(root=TRAIN_PATH, transform=transform)
batch_sampler = SamplerFactory().get(
    class_idxs=my_list,
    batch_size=75,
    n_batches=146,
    alpha=1,
    kind="fixed"
)


Instruction: Reverse operation of torch.cat

Input:
Suppose I have a tensor like [A,A,A,A...,A].
How can I quickly obtain [[A],[A],[A],...,[A]] as a tensor in torch?

Output:
You can use torch.chunk as the inverse of cat, but it looks like you want unsqueeze(1):
A = torch.randn(2, 3)
A_rep = (A, A, A, A, A, A, A, A)
catted = torch.cat(A_rep)

#uncatted = torch.chunk(catted, len(A_rep))
catted.unsqueeze(1)


Instruction: Learnable scalar weight in PyTorch

Input:
I have two neural networks running in parallel. Each gives a features map of same size say Nx1. Now I want weighted average of these embedding like this w1 * embed1 + w2 * embed2. I have tried these 1 2.But the weights are not updating. Any help would be appreciated. Here is how I am trying to do it:
class LinearWeightedAvg(nn.Module):
      def __init__(self, n_inputs):
        super(LinearWeightedAvg, self).__init__()
        self.weight1 = Variable(torch.randn(1), requires_grad=True).cuda()
        self.weight2 = Variable(torch.randn(1), requires_grad=True).cuda()

    def forward(self, inp_embed):
        return self.weight1 * inp_embed[0] + self.weight2 * inp_embed[1]

class EmbedBranch(nn.Module):
    def __init__(self, feat_dim, embedding_dim):
        super(EmbedBranch, self).__init__()
        fc_layer1 = fc_layer
    def forward(self, x):
        x = self.fc_layer1(x)
        return x

class EmbeddingNetwork(nn.Module):
    def __init__(self, args, N):
        super(EmbeddingNetwork, self).__init__()
        embedding_dim = N

        self.embed1 = EmbedBranch(N, N)
        self.embed2 = EmbedBranch(N, N)
        self.comb_branch = LinearWeightedAvg(metric_dim)

        self.args = args
        if args.cuda:
            self.cuda()

    def forward(self, emb1, emb2):
        embeds1 = self.text_branch(emb1)
        embeds2 = self.image_branch(emb2)
        combined = self.comb_branch([embeds1, embeds2])
        return combined

    def train_forward(self, embed1, embed2):

        combined = self(embed1, embed2)

embeds = model.train_forward(embed1, embed2)
loss = loss_func(embeds, labels)
running_loss.update(loss.data.item())
optimizer.zero_grad()
loss.backward()

Also I want the weight to be within 0-1 range.
Thanks,

Output: You should use self.weightx = torch.nn.Parameter(your_inital_tensor) to register a tensor as a learnable parameter of the model.


Instruction: Apply hooks on inner layers of ResNet

Input:
The pytorch official implementation of resnet results in the following model:
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (shortcut): Sequential()
    )
  )
 ### Skipping layers 3 and 4
  (linear): Linear(in_features=512, out_features=10, bias=True)
)

I tried applying hook to the conv1 in the first BasicBlock of layer2 using
handle = model.layer2[0][0].register_forward_hook(batchout_pre_hook)

but got the following error :
TypeError: 'BasicBlock' object does not support indexing

I am able to apply hook to the BasicBlock using  handle = model.layer2[0].register_forward_hook(batchout_pre_hook) but cannot apply hook in the modules present inside the BasicBlock

Output:
For attaching a hook to conv1 in layer2's 0th block, you need to use
handle = model.layer2[0].conv1.register_forward_hook(batchout_pre_hook)

This is because inside the 0th block, the modules are named as conv1, bn1, etc. and are not a list to be accessed via an index.


Instruction: Predict new samples with PyTorch model

Input:
I am newbee in neural networks, i have teached my model and now i want to test it. I have wrote a code with google help, but it do not work.
The problem is that i do not understand from where i am getting the 4th dimension.
Code is the following:
import matplotlib.pyplot as plt
import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

transform = transforms.Compose([
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

main_path = 'D:/RTU/dataset/ready_dataset_2classes'
train_data_path = main_path + '/train'
#test_data_path = main_path + '/test'
weigths_path = 'D:/RTU/dataset/weights_done/weights_noise_original037-97%.pt'

train_data = torchvision.datasets.ImageFolder(root=train_data_path, transform=transform)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # convolutional layer (sees 32x32x3 image tensor)
        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
        # convolutional layer (sees 16x16x16 tensor)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        # convolutional layer (sees 8x8x32 tensor)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        # max pooling layer
        self.pool = nn.MaxPool2d(2, 2)
        # linear layer (64 * 4 * 4 -> 500)
        self.fc1 = nn.Linear(64 * 4 * 4, 500)
        # linear layer (500 -> 10)
        self.fc2 = nn.Linear(500, 250)
        self.fc3 = nn.Linear(250, 2)
        # dropout layer (p=0.25)
        self.dropout = nn.Dropout(0.25) #0.25

    def forward(self, x):
        # add sequence of convolutional and max pooling layers
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        # flatten image input
        x = x.view(-1, 64 * 4 * 4)
        # add dropout layer
        x = self.dropout(x)
        # add 1st hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        # add dropout layer
        x = self.dropout(x)
        # add 2nd hidden layer, with relu activation function
        x = F.relu(self.fc2(x))
        x = self.dropout(x)
        x = self.fc3(x)
        return x

# Disable grad
with torch.no_grad():

    # Retrieve item
    index = 1
    item = train_data[index]
    image = item[0]
    true_target = item[1]

    # Loading the saved model
    mlp = Net()
    optimizer = optim.SGD(mlp.parameters(), lr=0.01)
    epoch=5
    valid_loss_min = np.Inf
    checkpoint = torch.load(weigths_path , map_location=torch.device('cpu'))
    mlp.load_state_dict(checkpoint['model_state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    epoch = checkpoint['epoch']
    valid_loss_min = checkpoint['valid_loss_min']
    mlp.eval()
    # Generate prediction
    prediction = mlp(image)

    # Predicted class value using argmax
    predicted_class = np.argmax(prediction)

    # Reshape image
    #image = image.reshape(28, 28, 1)

    # Show result
    plt.imshow(image, cmap='gray')
    plt.title(f'Prediction: {predicted_class} - Actual target: {true_target}')
    plt.show()

The code seems working till the "mlp.eval()" and then i am getting an error Expected 4-dimensional input for 4-dimensional weight [16, 3, 3, 3], but got 3-dimensional input of size [3, 32, 32] instead.
What i am doing wrong?
Error

Output:
When you are training neural nets, you are feeding small batches of input data to your model. Indeed even it's not clearly specified when writting Layers in Pytorch, If you look at the documentation, here you can see that Layers receive 4D arrays

 with N corresponding to batch size and C to number of channels, here 3 because you are using RGB images

So when testing your model once trained, the testing data should  be built the same way to be fed into the network.
Thus if you want to feed 1 image to your network you must reshape it proprely
myimage.reshape(-1,3,32,32)
print(myimage.shape)
#(1,3,32,33)


Instruction: How to convert torchscript model in PyTorch to ordinary nn.Module?

Input:
I am loading the torchscript model in the following way:
model = torch.jit.load("model.pt").to(device)

The children modules of this model are identified as RecursiveScriptModule. I would like to finetune the uploaded weights and in order to make it simplier and cast them to torch.float32 It is preferable to convert all this stuff to ordinary PyTorch nn.Module.
In the official docs https://pytorch.org/docs/stable/jit.html it is told how to convert nn.Module to torchscript, but I have not found any examples in doing this in the opposite direction. Is there a way to do this?
P.S the example of loading model pretrained model is given here:  https://github.com/openai/CLIP/blob/main/notebooks/Interacting_with_CLIP.ipynb

Output:
You may try to load it as it e.g. state_dict = torch.load(src).state_dict().
Then manually convert every key and value new_v = state_dict[k].cpu().float().


Instruction: How to generate an attention mask after padding?

Input:
After padding a tensor A which has 4 dimensions [4, 5, 129, 24] in second and third dimension to [4, 6, 136, 24], how can I generate its' attention mask?
I have figured out two solutions:

The first one is to create a zero-tensor A_attention that likes A_pad，and then traverse A to fill 1 to relevant position in A_attention.
The second one is to create attention mask in the padding procedure. But it seems a little troublesome because the initial tensor A has 4 dimensions.

Are there any ways to generate attention mask more efficiently after padding? Is there an API?
Thank you very much.

Output:
You can try to use the Transformers library from Hugging Face, which provides a really useful tokenizer.
I suggest you go through the entire quickstart, but in principle, this is the part you are interested in.


Instruction: How to load pretrained pytorch weight for

Input:
I am following this blog https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html  want to run pytorch model in onnx runtime . here in example it has given a pretrained weight a URL how to load a pretrained weight from local disk .
# Load pretrained model weights
model_url = 'https://s3.amazonaws.com/pytorch/test_data/export/superres_epoch100-44c6958e.pth'
path = "/content/best.pt"
batch_size = 1    # just a random number

# Initialize model with the pretrained weights
map_location = lambda storage, loc: storage
if torch.cuda.is_available():
    map_location = None
torch_model.load_state_dict(model_zoo.load_url(model_url, map_location=map_location))

# set the model to inference mode
torch_model.eval()

I want to load the weight which is defined as Path .

Output:
If you want to load the state dict from a path, this is what you should do:
torch_model.load_state_dict(torch.load(path))

This should work.


Instruction: Python: BERT Tokenizer cannot be loaded

Input:
I am working on the bert-base-mutilingual-uncased model but when I try to set the TOKENIZER in the config it throws an OSError.
Model Config
class config:
    DEVICE = "cuda:0"
    MAX_LEN = 256
    TRAIN_BATCH_SIZE = 8
    VALID_BATCH_SIZE = 4
    EPOCHS = 1

    BERT_PATH = {"bert-base-multilingual-uncased": "workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased"}
    MODEL_PATH = "workspace/data/jigsaw-multilingual/model.bin"

    TOKENIZER = transformers.BertTokenizer.from_pretrained(
            BERT_PATH["bert-base-multilingual-uncased"],
            do_lower_case=True)

Error
    ---------------------------------------------------------------------------
    OSError                                   Traceback (most recent call last)
    <ipython-input-33-83880b6b788e> in <module>
    ----> 1 class config:
          2 #     def __init__(self):
          3
          4         DEVICE = "cuda:0"
          5         MAX_LEN = 256

    <ipython-input-33-83880b6b788e> in config()
         11         TOKENIZER = transformers.BertTokenizer.from_pretrained(
         12             BERT_PATH["bert-base-multilingual-uncased"],
    ---> 13             do_lower_case=True)

    /opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, *inputs, **kwargs)
       1138
       1139         """
    -> 1140         return cls._from_pretrained(*inputs, **kwargs)
       1141
       1142     @classmethod

    /opt/conda/lib/python3.6/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
       1244                     ", ".join(s3_models),
       1245                     pretrained_model_name_or_path,
    -> 1246                     list(cls.vocab_files_names.values()),
       1247                 )
       1248             )

    OSError: Model name 'workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased' was not
 found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking,
bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc,
bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1,
wietsedv/bert-base-dutch-cased).

We assumed 'workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased' was a path, a model   identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such
 vocabulary files at this path or url.

As I can interpret the error, it says that the vocab.txt file was not found at the given location but actually its present.
Following are the files available in the bert-base-multilingual-uncased folder:

config.json
pytorch_model.bin
vocab.txt

I am new to working with bert, so I am not sure if there is a different way to define the tokenizer.

Output:
I think this should work:
from transformers import BertTokenizer
TOKENIZER = BertTokenizer.from_pretrained('bert-base-multilingual-uncased', do_lower_case=True)

It will download the tokenizer from huggingface.


Instruction: How to create n-dimensional sparse tensor? (pytorch)

Input:
I want to initialize tensor to sparse tensor.
When tensor's dimensional is 2, I can use torch.nn.init.sparse(tensor, sparsity=0.1)
import torch
dim = torch.Size([3,2])
w = torch.Tensor(dim)
torch.nn.init.sparse_(w, sparsity=0.1)

Result
tensor([[ 0.0000,  0.0147],
        [-0.0190,  0.0004],
        [-0.0004,  0.0000]])

But when tensor dimensions > 2, this function isn't work.
v = torch.Tensor(torch.Size([5,5,30,2]))
torch.nn.init.sparse_(v, sparsity=0.1)

Result
ValueError: Only tensors with 2 dimensions are supported

I need this because I want to use it to initialize the convolution weights.
torch.nn.init.sparse_() function's def is below
def sparse_(tensor, sparsity, std=0.01):
    r"""Fills the 2D input `Tensor` as a sparse matrix, where the
    non-zero elements will be drawn from the normal distribution
    :math:`\mathcal{N}(0, 0.01)`, as described in `Deep learning via
    Hessian-free optimization` - Martens, J. (2010).

    Args:
        tensor: an n-dimensional `torch.Tensor`
        sparsity: The fraction of elements in each column to be set to zero
        std: the standard deviation of the normal distribution used to generate
            the non-zero values

    Examples:
        >>> w = torch.empty(3, 5)
        >>> nn.init.sparse_(w, sparsity=0.1)
    """
    if tensor.ndimension() != 2:
        raise ValueError("Only tensors with 2 dimensions are supported")

    rows, cols = tensor.shape
    num_zeros = int(math.ceil(sparsity * rows))

    with torch.no_grad():
        tensor.normal_(0, std)
        for col_idx in range(cols):
            row_indices = torch.randperm(rows)
            zero_indices = row_indices[:num_zeros]
            tensor[zero_indices, col_idx] = 0
    return tensor

How could I make n-dimensional sparse tensor?
Is there a way in pytorch to create this kind of tensor?
Or can I make it another way?

Output:
This function is an implementation of the following method:

The best random initialization scheme we found was one of our own design, "sparse initialization". In this scheme we hard limit the
number of non-zero incoming connection weights to each
unit (we used 15 in our experiments) and set the biases to
0 (or 0.5 for tanh units).

Deep learning via Hessian-free optimization - Martens, J. (2010).

The reason it is not supported for higher order tensors is because it maintains the same proportion of zeros in each column, and it is not clear which [subset of] dimensions this condition should be maintained across for higher order tensors.
You can implement this initialization strategy with dropout or an equivalent function e.g:
def sparse_(tensor, sparsity, std=0.01):
    with torch.no_grad():
        tensor.normal_(0, std)
        tensor = F.dropout(tensor, sparsity)
    return tensor

If you wish to enforce column, channel, etc-wise proportions of zeros (as opposed to just total proportion) you can implement logic similar to the original function.


Instruction: Derive the gradient through torch.topk

Input:
I want to derive the gradient through torch.topk function.
Suppose I the input is a vector
,
then it is transformed by a parameter matrix
 ,
and the top k values of the vector are selected
.
The result vector is further transformed by element-wise multiplication.

Finally the loss is computed by
.
I wonder, is the loss differentiable with respect to W? Formally, can we calculate the following gradient?

Output:
The topk() operation is simply a linear transformation to pick the top k elements of a tensor. Since this is a W @ X or matrix-vector multiplication kind of operation, this is also differentiable.
Example: Below I have computed the pipelined operation topk(Wx) in two ways and showed the gradients resulting from both are identical.
In [1]: import torch

In [2]: x1 = torch.rand(6, requires_grad = True)

In [3]: W1 = torch.rand(6, 6, requires_grad = True)

In [4]: x1
Out[4]: tensor([0.1511, 0.5990, 0.6338, 0.5137, 0.5203, 0.0560], requires_grad=True)

In [5]: W1
Out[5]:
tensor([[0.2541, 0.6699, 0.5311, 0.7801, 0.5042, 0.5475],
        [0.7523, 0.1331, 0.7670, 0.8132, 0.0524, 0.0269],
        [0.3974, 0.2880, 0.9142, 0.9906, 0.4401, 0.3984],
        [0.7956, 0.2071, 0.2209, 0.6192, 0.2054, 0.7693],
        [0.8587, 0.8415, 0.6033, 0.3812, 0.2498, 0.9813],
        [0.9033, 0.0417, 0.2272, 0.1576, 0.9087, 0.3284]], requires_grad=True)

In [6]: y1 = W1 @ x1

In [7]: y1
Out[7]: tensor([1.4699, 1.1260, 1.5721, 0.8523, 1.3969, 0.8776], grad_fn=<MvBackward>)

In [8]: yk, _ = torch.topk(y1, 3)

In [9]: yk
Out[9]: tensor([1.5721, 1.4699, 1.3969], grad_fn=<TopkBackward>)

In [10]: loss1 = (yk ** 2).sum()

In [11]: loss1.backward()

In [12]: W1.grad
Out[12]:
tensor([[0.4442, 1.7609, 1.8633, 1.5102, 1.5296, 0.1646],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4751, 1.8833, 1.9928, 1.6152, 1.6359, 0.1760],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4222, 1.6734, 1.7706, 1.4352, 1.4535, 0.1564],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])

Now let us evaluate the same set of operations but using topk() as a linear transformation explicitly. Note that the constructed Wk matrix selectively picks out top k (here 3) elements from the 6 element tensor through multiplication.
In [13]: x2 = torch.tensor([0.1511, 0.5990, 0.6338, 0.5137, 0.5203, 0.0560], req
    ...: uires_grad=True)

In [14]: W2 = torch.tensor([[0.2541, 0.6699, 0.5311, 0.7801, 0.5042, 0.5475],
    ...:         [0.7523, 0.1331, 0.7670, 0.8132, 0.0524, 0.0269],
    ...:         [0.3974, 0.2880, 0.9142, 0.9906, 0.4401, 0.3984],
    ...:         [0.7956, 0.2071, 0.2209, 0.6192, 0.2054, 0.7693],
    ...:         [0.8587, 0.8415, 0.6033, 0.3812, 0.2498, 0.9813],
    ...:         [0.9033, 0.0417, 0.2272, 0.1576, 0.9087, 0.3284]], requires_gra
    ...: d=True)

In [15]: y2 = W2 @ x2

In [16]: y2
Out[16]: tensor([1.4700, 1.1260, 1.5721, 0.8523, 1.3969, 0.8776], grad_fn=<MvBackward>)

# Use the indices obtained earlier to construct the matrix
In [19]: _
Out[19]: tensor([2, 0, 4])

In [20]: k = 3

In [21]: Wk = torch.zeros(k, y2.shape[0])

In [22]: Wk[torch.arange(k), _] = 1

In [23]: Wk.requires_grad = True

In [24]: Wk
Out[24]:
tensor([[0., 0., 1., 0., 0., 0.],
        [1., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 0.]], requires_grad=True)

In [25]: yk2 = Wk @ y2

In [26]: yk2
Out[26]: tensor([1.5721, 1.4700, 1.3969], grad_fn=<MvBackward>)

In [27]: loss2 = (yk2 ** 2).sum()

In [28]: loss2.backward()

Now compare the gradients obtained in both cases:
In [29]: W2.grad
Out[29]:
tensor([[0.4442, 1.7611, 1.8634, 1.5103, 1.5297, 0.1646],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4751, 1.8834, 1.9929, 1.6152, 1.6360, 0.1761],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4222, 1.6735, 1.7707, 1.4352, 1.4536, 0.1565],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])

In [30]: W1.grad
Out[30]:
tensor([[0.4442, 1.7609, 1.8633, 1.5102, 1.5296, 0.1646],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4751, 1.8833, 1.9928, 1.6152, 1.6359, 0.1760],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
        [0.4222, 1.6734, 1.7706, 1.4352, 1.4535, 0.1564],
        [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000]])

In [31]: x1.grad
Out[31]: tensor([4.3955, 5.2256, 6.1213, 6.4732, 3.5637, 5.6037])

In [32]: x2.grad
Out[32]: tensor([4.3957, 5.2261, 6.1215, 6.4733, 3.5641, 5.6040])

As you can see the results are identical upto some floating point errors which were introduced when I copied the values of x1 and W1 without taking their full precision.


Instruction: How to build YOLACT++ using Docker?

Input:
I have to build yolact++ in docker enviromment (i'm using sagemaker notebook). Like this
ARG PYTORCH="1.3"
ARG CUDA="10.1"
ARG CUDNN="7"

FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel

And i want to run this
COPY yolact/external/DCNv2/setup.py /opt/ml/code/external/DCNv2/setup.py
RUN cd /opt/ml/code/external/DCNv2 && \
python setup.py build develop

But i got this error :
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
Traceback (most recent call last):
File "setup.py", line 64, in <module>
ext_modules=get_extensions(),
File "setup.py", line 41, in get_extensions
raise NotImplementedError('Cuda is not available')
NotImplementedError: Cuda is not available

But the enviromment supports CUDA. Anyone have an idea where is the problem ?
Thank you.

Output:
SOLUTION :
i edit the /etc/docker/daemon.json with content:
{
"runtimes": {
    "nvidia": {
        "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
     }
},
"default-runtime": "nvidia"
}

Then i Restart docker daemon:
sudo system restart docker

it solved my problem.


Instruction: how can I load pretrained model by pytorch? ( mmfashion)

Input:
import io

import torch
import torch.nn as nn
from torchvision import models
from PIL import Image
import torchvision.transforms as transforms

checkpoint_path = 'C:/venvs/ai/aiproduct/latest.pth'
pretrained_weights = torch.load(checkpoint_path, map_location='cpu', strict=False)

model = models.resnet50(pretrained=True)
model.load_state_dict(pretrained_weights)

this bring
TypeError: 'strict' is an invalid keyword argument for load()

and
import io

import torch
import torch.nn as nn
from torchvision import models
from PIL import Image
import torchvision.transforms as transforms

checkpoint_path = 'C:/venvs/ai/aiproduct/latest.pth'
pretrained_weights = torch.load(checkpoint_path, map_location='cpu')

model = models.resnet50(pretrained=True)
model.load_state_dict(pretrained_weights)
# model.eval()
print(model)
# model.summary()

if I get rid of "strict"
Traceback (most recent call last):
  File "c:\venvs\ai\aiproduct\test.py", line 13, in <module>
    model.load_state_dict(pretrained_weights)
  File "C:\Python39\lib\site-packages\torch\nn\modules\module.py", line 1223, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ResNet:
        Missing key(s) in state_dict: "conv1.weight", "bn1.weight", "bn1.bias", "bn1.running_mean", "bn1.running_var", "layer1.0.conv1.weight", "layer1.0.bn1.weight", "layer1.0.bn1.bias", "layer1.0.bn1.running_mean", "layer1.0.bn1.running_var", "layer1.0.conv2.weight", "layer1.0.bn2.weight", "layer1.0.bn2.bias", "layer1.0.bn2.running_mean", "layer1.0.bn2.running_var", "layer1.0.conv3.weight", "layer1.0.bn3.weight", "layer1.0.bn3.bias", "layer1.0.bn3.running_mean", "layer1.0.bn3.running_var", "layer1.0.downsample.0.weight", "layer1.0.downsample.1.weight", "layer1.0.downsample.1.bias", "layer1.0.downsample.1.running_mean", "layer1.0.downsample.1.running_var", "layer1.1.conv1.weight", "layer1.1.bn1.weight", "layer1.1.bn1.bias", "layer1.1.bn1.running_mean", "layer1.1.bn1.running_var", "layer1.1.conv2.weight", "layer1.1.bn2.weight", "layer1.1.bn2.bias", "layer1.1.bn2.running_mean",
"layer1.1.bn2.running_var", "layer1.1.conv3.weight", "layer1.1.bn3.weight", "layer1.1.bn3.bias", "layer1.1.bn3.running_mean", "layer1.1.bn3.running_var", "layer1.2.conv1.weight", "layer1.2.bn1.weight", "layer1.2.bn1.bias", "layer1.2.bn1.running_mean", "layer1.2.bn1.running_var", "layer1.2.conv2.weight", "layer1.2.bn2.weight", "layer1.2.bn2.bias", "layer1.2.bn2.running_mean", "layer1.2.bn2.running_var", "layer1.2.conv3.weight", "layer1.2.bn3.weight", "layer1.2.bn3.bias", "layer1.2.bn3.running_mean", "layer1.2.bn3.running_var", "layer2.0.conv1.weight", "layer2.0.bn1.weight", "layer2.0.bn1.bias", "layer2.0.bn1.running_mean", "layer2.0.bn1.running_var", "layer2.0.conv2.weight", "layer2.0.bn2.weight", "layer2.0.bn2.bias", "layer2.0.bn2.running_mean", "layer2.0.bn2.running_var", "layer2.0.conv3.weight", "layer2.0.bn3.weight", "layer2.0.bn3.bias", "layer2.0.bn3.running_mean", "layer2.0.bn3.running_var", "layer2.0.downsample.0.weight", "layer2.0.downsample.1.weight", "layer2.0.downsample.1.bias", "layer2.0.downsample.1.running_mean", "layer2.0.downsample.1.running_var", "layer2.1.conv1.weight", "layer2.1.bn1.weight", "layer2.1.bn1.bias", "layer2.1.bn1.running_mean", "layer2.1.bn1.running_var", "layer2.1.conv2.weight", "layer2.1.bn2.weight", "layer2.1.bn2.bias", "layer2.1.bn2.running_mean", "layer2.1.bn2.running_var", "layer2.1.conv3.weight", "layer2.1.bn3.weight", "layer2.1.bn3.bias", "layer2.1.bn3.running_mean", "layer2.1.bn3.running_var", "layer2.2.conv1.weight", "layer2.2.bn1.weight", "layer2.2.bn1.bias", "layer2.2.bn1.running_mean", "layer2.2.bn1.running_var", "layer2.2.conv2.weight", "layer2.2.bn2.weight", "layer2.2.bn2.bias", "layer2.2.bn2.running_mean", "layer2.2.bn2.running_var", "layer2.2.conv3.weight", "layer2.2.bn3.weight", "layer2.2.bn3.bias", "layer2.2.bn3.running_mean", "layer2.2.bn3.running_var", "layer2.3.conv1.weight", "layer2.3.bn1.weight", "layer2.3.bn1.bias", "layer2.3.bn1.running_mean", "layer2.3.bn1.running_var", "layer2.3.conv2.weight", "layer2.3.bn2.weight", "layer2.3.bn2.bias", "layer2.3.bn2.running_mean", "layer2.3.bn2.running_var", "layer2.3.conv3.weight", "layer2.3.bn3.weight", "layer2.3.bn3.bias", "layer2.3.bn3.running_mean", "layer2.3.bn3.running_var", "layer3.0.conv1.weight", "layer3.0.bn1.weight", "layer3.0.bn1.bias", "layer3.0.bn1.running_mean", "layer3.0.bn1.running_var", "layer3.0.conv2.weight", "layer3.0.bn2.weight", "layer3.0.bn2.bias", "layer3.0.bn2.running_mean", "layer3.0.bn2.running_var", "layer3.0.conv3.weight", "layer3.0.bn3.weight", "layer3.0.bn3.bias", "layer3.0.bn3.running_mean", "layer3.0.bn3.running_var", "layer3.0.downsample.0.weight", "layer3.0.downsample.1.weight", "layer3.0.downsample.1.bias", "layer3.0.downsample.1.running_mean", "layer3.0.downsample.1.running_var", "layer3.1.conv1.weight", "layer3.1.bn1.weight", "layer3.1.bn1.bias", "layer3.1.bn1.running_mean", "layer3.1.bn1.running_var", "layer3.1.conv2.weight", "layer3.1.bn2.weight", "layer3.1.bn2.bias", "layer3.1.bn2.running_mean", "layer3.1.bn2.running_var", "layer3.1.conv3.weight", "layer3.1.bn3.weight", "layer3.1.bn3.bias", "layer3.1.bn3.running_mean", "layer3.1.bn3.running_var", "layer3.2.conv1.weight", "layer3.2.bn1.weight", "layer3.2.bn1.bias", "layer3.2.bn1.running_mean", "layer3.2.bn1.running_var", "layer3.2.conv2.weight", "layer3.2.bn2.weight", "layer3.2.bn2.bias", "layer3.2.bn2.running_mean", "layer3.2.bn2.running_var", "layer3.2.conv3.weight", "layer3.2.bn3.weight", "layer3.2.bn3.bias", "layer3.2.bn3.running_mean", "layer3.2.bn3.running_var", "layer3.3.conv1.weight", "layer3.3.bn1.weight", "layer3.3.bn1.bias", "layer3.3.bn1.running_mean", "layer3.3.bn1.running_var", "layer3.3.conv2.weight", "layer3.3.bn2.weight", "layer3.3.bn2.bias", "layer3.3.bn2.running_mean", "layer3.3.bn2.running_var", "layer3.3.conv3.weight", "layer3.3.bn3.weight", "layer3.3.bn3.bias", "layer3.3.bn3.running_mean", "layer3.3.bn3.running_var", "layer3.4.conv1.weight", "layer3.4.bn1.weight", "layer3.4.bn1.bias", "layer3.4.bn1.running_mean", "layer3.4.bn1.running_var", "layer3.4.conv2.weight", "layer3.4.bn2.weight", "layer3.4.bn2.bias", "layer3.4.bn2.running_mean", "layer3.4.bn2.running_var", "layer3.4.conv3.weight", "layer3.4.bn3.weight", "layer3.4.bn3.bias", "layer3.4.bn3.running_mean", "layer3.4.bn3.running_var",
"layer3.5.conv1.weight", "layer3.5.bn1.weight", "layer3.5.bn1.bias", "layer3.5.bn1.running_mean", "layer3.5.bn1.running_var", "layer3.5.conv2.weight", "layer3.5.bn2.weight", "layer3.5.bn2.bias", "layer3.5.bn2.running_mean", "layer3.5.bn2.running_var", "layer3.5.conv3.weight", "layer3.5.bn3.weight", "layer3.5.bn3.bias", "layer3.5.bn3.running_mean", "layer3.5.bn3.running_var", "layer4.0.conv1.weight", "layer4.0.bn1.weight", "layer4.0.bn1.bias", "layer4.0.bn1.running_mean", "layer4.0.bn1.running_var", "layer4.0.conv2.weight", "layer4.0.bn2.weight", "layer4.0.bn2.bias", "layer4.0.bn2.running_mean", "layer4.0.bn2.running_var", "layer4.0.conv3.weight", "layer4.0.bn3.weight", "layer4.0.bn3.bias", "layer4.0.bn3.running_mean", "layer4.0.bn3.running_var", "layer4.0.downsample.0.weight", "layer4.0.downsample.1.weight", "layer4.0.downsample.1.bias", "layer4.0.downsample.1.running_mean", "layer4.0.downsample.1.running_var", "layer4.1.conv1.weight", "layer4.1.bn1.weight", "layer4.1.bn1.bias", "layer4.1.bn1.running_mean", "layer4.1.bn1.running_var", "layer4.1.conv2.weight", "layer4.1.bn2.weight", "layer4.1.bn2.bias", "layer4.1.bn2.running_mean", "layer4.1.bn2.running_var", "layer4.1.conv3.weight", "layer4.1.bn3.weight", "layer4.1.bn3.bias",
"layer4.1.bn3.running_mean", "layer4.1.bn3.running_var", "layer4.2.conv1.weight", "layer4.2.bn1.weight", "layer4.2.bn1.bias", "layer4.2.bn1.running_mean", "layer4.2.bn1.running_var", "layer4.2.conv2.weight", "layer4.2.bn2.weight", "layer4.2.bn2.bias", "layer4.2.bn2.running_mean", "layer4.2.bn2.running_var", "layer4.2.conv3.weight", "layer4.2.bn3.weight", "layer4.2.bn3.bias", "layer4.2.bn3.running_mean", "layer4.2.bn3.running_var", "fc.weight", "fc.bias".
        Unexpected key(s) in state_dict: "meta", "state_dict", "optimizer".

what should I do?
I just want to make some cloth attribute predicting web app with pretrained model (mmfashion https://github.com/open-mmlab/mmfashion/blob/master/docs/MODEL_ZOO.md)
but I fail to use pretrained model.

Output:
Lets say if you downloaded weights for wide_resnet50_2 and you performing same task that the weights you downloaded trained then:
import torchvision
model = torchvision.models.wide_resnet50_2(pretrained=True)

for param in model.parameters():
    param.required_grad = False

and then the parameters you downloaded can be load as:
Load model state dict
model.load_state_dict(torch.load('./C:/venvs/ai/aiproduct/latest.pth')) # path of your weights
model.eval()
model.cuda()


Instruction: How can I pick a subset of a dataset in Pytorch?

Input: I'm trying to run https://github.com/menardai/FashionGenAttnGAN in Google Colab on GPU with the disk size of 30 GB. The code file with its dataset files are about 15 GB. After I extract this code, the disk remains is about 14 GB. When I try to run the Pretrain.py, I can see the captions are loading but suddenly I got the "Assertion Error". Since I've not got any proper answer for the cause of this error, I think that it is because of my lack of space in Colab environment. The solution that came to my mind is that write some code to tell the model to select just 30% of the train and test datasets to load. But I don't know how to do this. Can anyone help me please?

Output:
data is your total data, you can divide it how much you want just edit valid_size.
valid_size=0.3
num_train = len(data)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(valid_size * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)

# prepare data loaders (combine dataset and sampler)
train_loader = torch.utils.data.DataLoader(data, batch_size=4,
    sampler=train_sampler, num_workers=2)
valid_loader = torch.utils.data.DataLoader(data, batch_size=4,
    sampler=valid_sampler, num_workers=2)

IF there memory issue ocurres just reduce batch_size.


Instruction: What is the recommended way to sum a list in TorchScript

Input:
Because Python builtin <built-in function sum> is currently not supported in Torchscript: I'm looking for the recommended way to do something like:
class Model(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return sum(x.tolist())

model = Model()
model = torch.jit.script(model)
model(torch.arange(10))

Output:
Easiest approach would be to use PyTorch's sum directly:
class Model(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return torch.sum(x)

If, for some reason, this is not an option, you have to use type specification with explicit loop (please notice type hints!):
import typing

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x) -> int:
        x: typing.List[int] = x.tolist()
        result = 0
        for elem in x:
            result += elem
        return result


Instruction: Torchaudio C++ extension is not available - Python

Input:
Having just started learning Torchaudio, I got this error. I fixed the first part but even after some deep internet surfing, I can't find a fix. I use Windows 10.
C:\Users\bala006\Miniconda39\lib\site-packages\torchaudio\extension\extension.py:13: UserWarning: torchaudio C++ extension is not available.
  warnings.warn('torchaudio C++ extension is not available.')

Thanks for any help !

Output:
https://pytorch.org/audio/stable/backend.html
UserWarning: torchaudio C++ extension is not available
as @ex4 said this extencion is not avalible on windows


Instruction: Does torch.manual_seed include the operation of torch.cuda.manual_seed_all?

Input:
Does torch.manual_seed include the operation of torch.cuda.manual_seed_all?
If yes, we can just use torch.manual_seed to set the seed. Otherwise we should call both functions.

Output:
Yes, torch.manual_seed() does include CUDA:

You can use torch.manual_seed() to seed the RNG for all devices (both CPU and CUDA):

https://pytorch.org/docs/stable/notes/randomness.html


Instruction: Print the validation loss in each epoch in PyTorch

Input:
I want to print the model's validation loss in each epoch, what is the right way to get and print the validation loss?
Is it like this:
criterion = nn.CrossEntropyLoss(reduction='mean')
for x, y in validation_loader:
 optimizer.zero_grad()
 out = model(x)
 loss = criterion(out, y)
 loss.backward()
 optimizer.step()
 losses += loss

display_loss = losses/len(validation_loader)
print(display_loss)

or like this
criterion = nn.CrossEntropyLoss(reduction='mean')
for x, y in validation_loader:
     optimizer.zero_grad()
     out = model(x)
     loss = criterion(out, y)
     loss.backward()
     optimizer.step()
     losses += loss

display_loss = losses/len(validation_loader.dataset)
print(display_loss)

or something else? Thank you.

Output:
NO!!!!

Under no circumstances should you train your model (i.e., call loss.backward() + optimizer.step())  using validation / test data!!!
If you want to validate your model:
model.eval()  # handle drop-out/batch norm layers
loss = 0
with torch.no_grad():
  for x,y in validation_loader:
    out = model(x)  # only forward pass - NO gradients!!
    loss += criterion(out, y)
# total loss - divide by number of batches
val_loss = loss / len(validation_loader)

Note how optimizer has nothing to do with evaluating the model on the validation set.
You do not change the model according to the validation data - only validate it.


Instruction: Remove zero dimension in pytorch

Input:
I have torch.tensor which has dimension of 0 x 240 x 3 x 540 x 960
I just want to remove 0 dimension, so I want to have pytorch tensor
size of 240 x 3 x 540 x 960.
I used tensor= torch.squeeze(tensor) to try that, but zero dimension was not removed...
In my case, the size of tensor is variable so I can't hard code it to torch.squeeze..
Is there any simple way to remove undesired dimension in pytorch.tensor?

Output:
You can't.
The size (=number of elements) in a tensor of shape 0 x 240 x 3 x 540 x 960 is 0.
You can't reshape it to a tensor of shape 240 x 3 x 540 x 960 because that has 373248000 elements.
squeeze() doc:

Returns a tensor with all the dimensions of input of size 1 removed.

This removes 1 sized dims, not 0 sized dims, which makes sense.

Anyway, the amount of data in your original tensor is 0, which can be presented as any shape with at least one zero dim, but not as shapes whose size > 0.


Instruction: Pytorch Tensor::data_ptr() not working on Linux

Input:
I cannot link my program to pytorch under Linux, get the following error:
/tmp/ccbgkLx2.o: In function `long long* at::Tensor::data<long long>() const':
test.cpp:(.text._ZNK2at6Tensor4dataIxEEPT_v[_ZNK2at6Tensor4dataIxEEPT_v]+0x14): undefined reference to `long long* at::Tensor::data_ptr<long long>() const'

I am building a very simple minimal example:
#include "torch/script.h"
#include <iostream>

int main() {
    auto options = torch::TensorOptions().dtype(torch::kInt64);
    torch::NoGradGuard no_grad;
    auto T = torch::zeros(20, options).view({ 10, 2 });
    long long *data = (long long *)T.data<long long>();
    data[0] = 1;
    return 0;
}

The command used to build it:
g++ -w -std=c++17 -o test-torch test.cpp -D_GLIBCXX_USE_CXX11_ABI=1 -Wl,--whole-archive -ldl -lpthread -Wl,--no-whole-archive -I../libtorch/include -L../libtorch/lib -ltorch -ltorch_cpu -lc10 -Wl,-rpath,../libtorch/lib

Pytorch has been downloaded from the link https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcpu.zip and unzipped (so I have the libtorch folder next to the folder with test.cpp).
Any ideas how to solve this problem? Same program works just fine under Visual C++.
P.S. I know pytorch is kind of designed for cmake, but I have zero experience with cmake and no desire to write a cmake-based build system for my app. Also, the examples they give are seemingly supposed to only work if pytorch is "installed" in the system. So I cannot just download the .zip with libs? And if I "install" it (e.g. from sources or in whatever other way) on an AVX512 system, will the binary I link to it and distribute to end-users work on non-AVX512? The documentation is completely incomprehensible for newbies.
UPDATE: I tried to do this via CMake following the tutorial https://pytorch.org/cppdocs/installing.html and got exactly the same error. Specifically, I renamed my directory to example-app and the source file to example-app.cpp. Then I created CMakeLists.txt in this directory with the following contents:
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

Then
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=../../libtorch ..
cmake --build . --config Release

And here's the output:
CMakeFiles/example-app.dir/example-app.cpp.o: In function `long long* at::Tensor::data<long long>() const':
example-app.cpp:(.text._ZNK2at6Tensor4dataIxEEPT_v[_ZNK2at6Tensor4dataIxEEPT_v]+0x14): undefined reference to `long long* at::Tensor::data_ptr<long long>() const'

Makes me think, maybe I forgot to include some header or define some variable?
Oh, this is all on Mint 19.2 (equivalent to Ubuntu 18.04), g++ version is 7.5.0, glibc is 2.27. Compiling with g++-8 gives the same result.

Output:
This is not a cmake-related error, it's just how the library was implemented. I do not know why, but it appears that the specialization of T* at::Tensor::data<T> const with T = long long was forgotten/omitted.
If you want to get your signed 64-bits pointer, you can still get it with int64_t:
auto data = T.data<int64_t>();

It's good practice to use these types for which the size is explicit in general, in order to avoid compatibility issues.