Instruction: How do I use slicing as I pass a transformer dataset to Trainer?

Input:
In reference to this colab notebook (from Huggingface Transformer course here), if I run
tokenized_datasets["train"][:8]

the dtype is a dict instead of a Dataset and the slicing would return some data.
If I pass the slicing in here, I get a Key error, which I assume has to do with the fact I'm no longer passing a Dataset.
from transformers import Trainer

trainer = Trainer(
    model,
    training_args,
    train_dataset=tokenized_datasets["train"][:8],
    eval_dataset=tokenized_datasets["validation"],
    #data_collator=data_collator,
    tokenizer=tokenizer,
)

trainer.train()
***** Running training *****
  Num examples = 7
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 3
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-20-3435b262f1ae> in <module>()
----> 1 trainer.train()

Is there a simple way to just pass a subset of the Dataset rows for training or validation?

Output:
import transformers
from datasets import load_dataset

datasets = load_dataset('squad')
datasets

output:
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 87599
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 10570
    })
})

then we can sample partial dataset:
train_datasets_sampled =  datasets["train"].shuffle(seed=42).select(range(2000))
eval_dataset_sampled = datasets["validation"].shuffle(seed=42).select(range(500))

train_datasets_sampled, eval_dataset_sampled

get the following:
(Dataset({
     features: ['id', 'title', 'context', 'question', 'answers'],
     num_rows: 2000
 }), Dataset({
     features: ['id', 'title', 'context', 'question', 'answers'],
     num_rows: 500
 }))

source: https://github.com/huggingface/notebooks/blob/master/transformers_doc/training.ipynb


Instruction: What happens if optimal training loss is too high

Input:
I am training a Transformer. In many of my setups I obtain validation and training loss that look like this:

Then, I understand that I should stop training at around epoch 1. But then the training loss is very high. Is this a problem? Does the value of training loss actually mean anything?
Thanks

Output:
Regarding your first question - it is not necessarily a problem that your training loss is high, since there is no threshold for what is considered as a high training loss. It depends on your dataset, your actual test metrics and your business goals.
More specifically, the problems with the value of training loss:

The number isn't intuitive, since the loss objective is a metric optimized for gradient descent (i.e. a differentiable function, usually the log version of it).
You probably have intuitive business metrics (e.g., precision, recall) oriented towards your end goal, which you should use to decide if your model is good or not.

Your train loss is calculated on the training dataset, which is not always representative of a good model, as can be seen in the overfitted model you posted. You shouldn't use this number to make decisions for the goodness of the model.

It depends on what you are trying to achieve. Is 80% accuracy high or low?

Regarding your second question - Technically, the higher the number the worse the model did in converging, so you should always try to lower it (while taking into consideration overfitting).
Comparatively, you can say that one model has a higher loss than another and then try multiple hyperparameters (e.g., dropout, different optimizers) to minimize the point where the validation set diverges.


Instruction: torch.nn.CrossEntropyLoss over Multiple Batches

Input:
I am currently working with torch.nn.CrossEntropyLoss. As far as I know, it is common to compute the loss batch-wise. However, is there a possibility to compute the loss over multiple batches?
More concretely, assume we are given the data
import torch

features = torch.randn(no_of_batches, batch_size, feature_dim)
targets = torch.randint(low=0, high=10, size=(no_of_batches, batch_size))

loss_function = torch.nn.CrossEntropyLoss()

Is there a way to compute in one line
loss = loss_function(features, targets) # raises RuntimeError: Expected target size [no_of_batches, feature_dim], got [no_of_batches, batch_size]

?
Thank you in advance!

Output:
You can compute multiple cross-entropy losses but you'll need to do your own reduction. Since cross-entropy loss assumes the feature dim is always the second dimension of the features tensor you will also need to permute it first.
loss_function = torch.nn.CrossEntropyLoss(reduction='none')
loss = loss_function(features.permute(0,2,1), targets).mean(dim=1)

which will result in a loss tensor with no_of_batches entries.


Instruction: RuntimeError: Found dtype Long but expected Float when fine-tuning using Trainer API

Input:
I'm trying to fine-tune BERT model for sentiment analysis (classifying text as positive/negative) with Huggingface Trainer API. My dataset has two columns, Text and Sentiment, it looks like this.
Text                     Sentiment
This was good place          1
This was bad place           0

Here is my code:
from datasets import load_dataset
from datasets import load_dataset_builder
from datasets import Dataset
import datasets
import transformers
from transformers import TrainingArguments
from transformers import Trainer

dataset = load_dataset('csv', data_files='./train/test.csv', sep=';')
tokenizer = transformers.BertTokenizer.from_pretrained("TurkuNLP/bert-base-finnish-cased-v1")
model = transformers.BertForSequenceClassification.from_pretrained("TurkuNLP/bert-base-finnish-cased-v1", num_labels=1)
def tokenize_function(examples):
    return tokenizer(examples["Text"], truncation=True, padding='max_length')

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets = tokenized_datasets.rename_column('Sentiment', 'label')
tokenized_datasets = tokenized_datasets.remove_columns('Text')
training_args = TrainingArguments("test_trainer")
trainer = Trainer(
    model=model, args=training_args, train_dataset=tokenized_datasets['train']
)
trainer.train()

Running this throws error:
Variable._execution_engine.run_backward(
RuntimeError: Found dtype Long but expected Float

The error may come from dataset itself, but can I fix it with my code somehow? I searched the Internet and this error seems to have been previously solved by "converting tensors to float" but how would I do it with Trainer API? Any advise is very highly appreciated.
Some reference:
https://discuss.pytorch.org/t/run-backward-expected-dtype-float-but-got-dtype-long/61650/10

Output:
Most likely, the problem is with loss function. This can be fixed if you set up the model correctly, mainly by specifying the correct loss to use. Refer to this code to see the logic for deciding the proper loss.
Your problem has binary labels and thus should be framed as a single-label classification problem. As such, the code you have shared will be inferred as a regression problem, which explains the error that it expected float but found long type for target labels.
You need to pass the correct problem type.
model = transformers.BertForSequenceClassification.from_pretrained(
    "TurkuNLP/bert-base-finnish-cased-v1",
    num_labels=1,
    problem_type = "single_label_classification"
)

This will make use of BCE loss. For BCE loss, you need the target to float, so you also have to cast the labels to float. I think you can do that with the dataset API. See this.
The other way would be to use a multi-class classifier or CE loss. For that, just fixing num_labels should be fine.
model = transformers.BertForSequenceClassification.from_pretrained(
    "TurkuNLP/bert-base-finnish-cased-v1",
    num_labels=2,
)


Instruction: how effective is transfer learning? keeping only two specific output features without resetting features

Input:
I want to keep only two specific output features without resetting features.
Resetting features would lose the pre-trained weights.
For example, I don't want to do...
# https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html?highlight=transfer%20learning%20ant%20bees

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

Here is code (following the transfer learning tutorial on Pytorch)
I want to do this to see how effective transfer learning is.
Even without transfer learning, a model might be effective. Removing 998 out of 1000 categories and leaving only two categories, ant and bee, could be a great categorical model since you are left with only two choices.
I do not want to re-train the model, I want to use the weights as it is, otherwise, it will be the same as transfer learning.

Output:
You can certainly try this. You can reduce the model output to just the two logits you want to compare with:
                chosen_cats = torch.Tensor([ant_index, bee_index]).long()
                with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                outputs = torch.index_select(output, 1, chosen_cats)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

In this scenario, the preds will be 0 or 1, with 0 predicting ant and 1 predicting bee, so you will need to also modify your labels to reflect this.


Instruction: How to free GPU memory in PyTorch

Input:
I have a list of sentences I'm trying to calculate perplexity for, using several models using this code:
from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
import numpy as np
model_name = 'cointegrated/rubert-tiny'
model = AutoModelForMaskedLM.from_pretrained(model_name).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_name)

def score(model, tokenizer, sentence):
    tensor_input = tokenizer.encode(sentence, return_tensors='pt')
    repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
    mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
    masked_input = repeat_input.masked_fill(mask == 1, tokenizer.mask_token_id)
    labels = repeat_input.masked_fill( masked_input != tokenizer.mask_token_id, -100)
    with torch.inference_mode():
        loss = model(masked_input.cuda(), labels=labels.cuda()).loss
    return np.exp(loss.item())

print(score(sentence='London is the capital of Great Britain.', model=model, tokenizer=tokenizer))
# 4.541251105675365

Most models work well, but some sentences seem to throw an error:
RuntimeError: CUDA out of memory. Tried to allocate 10.34 GiB (GPU 0; 23.69 GiB total capacity; 10.97 GiB already allocated; 6.94 GiB free; 14.69 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Which makes sense because some are very long. So what I did was to add something like try, except RuntimeError, pass.
This seemed to work until around 210 sentences, and then it just outputs the error:
CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
I found this which had a lot of discussions and ideas, some were regarding potential faulty GPUs? But I know that my GPU works as this exact code works for other models. There's also talk about batch size here, which is why I thought it potentially relates to freeing up memory.
I tried running torch.cuda.empty_cache() to free the memory like in here after every some epochs but it didn't work (threw the same error).
Update:
I filtered sentences with length over 550 and this seems to remove the CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. error.

Output:
You need to apply gc.collect() before torch.cuda.empty_cache()
I also pull the model to cpu and then delete that model and its checkpoint. Try what works for you:
import gc

model.cpu()
del model, checkpoint
gc.collect()
torch.cuda.empty_cache()


Instruction: SHAP values with PyTorch - KernelExplainer vs DeepExplainer

Input:
I haven't been able to find much in the way of examples on SHAP values with PyTorch. I've used two techniques to generate SHAP values, however, their results don't appear to agree with each other.
SHAP KernelExplainer with PyTorch
import torch
from torch.autograd import Variable
import shap
import numpy
import pandas

torch.set_grad_enabled(False)

# Get features
train_features_df = ... # pandas dataframe
test_features_df = ... # pandas dataframe

# Define function to wrap model to transform data to tensor
f = lambda x: model_list[0]( Variable( torch.from_numpy(x) ) ).detach().numpy()

# Convert my pandas dataframe to numpy
data = test_features_df.to_numpy(dtype=np.float32)

# The explainer doesn't like tensors, hence the f function
explainer = shap.KernelExplainer(f, data)

# Get the shap values from my test data
shap_values = explainer.shap_values(data)

# Enable the plots in jupyter
shap.initjs()

feature_names = test_features_df.columns
# Plots
#shap.force_plot(explainer.expected_value, shap_values[0], feature_names)
#shap.dependence_plot("b1_price_avg", shap_values[0], data, feature_names)
shap.summary_plot(shap_values[0], data, feature_names)

SHAP DeepExplainer with PyTorch
# It wants gradients enabled, and uses the training set
torch.set_grad_enabled(True)
e = shap.DeepExplainer(model, Variable( torch.from_numpy( train_features_df.to_numpy(dtype=np.float32) ) ) )

# Get the shap values from my test data (this explainer likes tensors)
shap_values = e.shap_values( Variable( torch.from_numpy(data) ) )

# Plots
#shap.force_plot(explainer.expected_value, shap_values, feature_names)
#shap.dependence_plot("b1_price_avg", shap_values, data, feature_names)
shap.summary_plot(shap_values, data, feature_names)

Comparing results
As you can see from the summary plots, the value given to the features from the same PyTorch model, with the same test data, are noticeably different.
For example the feature b1_addresses_avg has value one from last with the KernelExplainer. But with the DeepExplainer is ranked third from top.
I'm not sure where to go from here.

Output:
Shapley values are very difficult to calculate exactly. Kernel SHAP and Deep SHAP are two different approximation methods to calculate the Shapley values efficiently, and so one shouldn't expect them to necessarily agree.
You can read the authors' paper for more details.

While Kernel SHAP can be used on any model, including deep models, it is natural to ask whether
there is a way to leverage extra knowledge about the compositional nature of deep networks to improve
computational performance. [...] This motivates our adapting DeepLIFT to become a compositional approximation
of SHAP values, leading to Deep SHAP.

In section 5, they compare the performance of Kernel SHAP and Deep SHAP. From their example it seems like Kernel SHAP performs better than Deep SHAP. So I guess if you aren't running into computational issues, you can stick with Kernel SHAP.

p.s. Just to make sure, you're inputting the exact same trained model to SHAP right? You shouldn't be training separate models, because they'll learn different weights.


Instruction: How do I retrieve the resultant image as a matrix(numpy array) from results given back by yolov5 in pytoch?

Input:
I have been learning how to implement pretrained yolo using pytorch, and I want to display the output image using openCV's cv2.imshow() method.
The output image can be displayed using .show() function and saved using .save() function, I however want to display it using cv2.imshow(), and for that I would need the image in the form of a numpy array.
I'm unaware about how we do that or even if that is at all possible.
Here's the code for it.
import torch

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True)
imgs = ['img.png']  # batch of images

results = model(imgs)

results.print()
results.show()  # or .save(), shows/saves the same image with bounding boxes around detected objects

# Show 'results' using openCV's cv2.imshow() method?

results.xyxy[0]  # img1 predictions (tensor)
print(results.pandas().xyxy[0])  # img1 predictions (pandas)

A longer way of solving this problem would be to create bounding boxes ourselves over the detected objects in the image and display it, but consider me lazy :p .

Output:
had the same issue, so I wrote a small method to do so quickly draw the image without saving it.
  def drawRectangles(image, dfResults):
    for index, row in dfResults.iterrows():
      print( (row['xmin'], row['ymin']))
      image = cv2.rectangle(image, (row['xmin'], row['ymin']), (row['xmax'], row['ymax']), (255, 0, 0), 2)
    cv2_imshow(image)

_
    results = model(image)
    dfResults = results.pandas().xyxy[0]
    self.drawRectangles(image, dfResults[['xmin', 'ymin', 'xmax','ymax']].astype(int))


Instruction: What is self referring to in this PyTorch derived nn.Module class method?

Input:
I am following this tutorial for Pytorch and there is a line of code that makes no sense to me in the derived class MnistModule method training_step of the nn.Module class.
The line is
out = self(images)
Please can someone explain to me what is happening here? Is this correct or not and if this is convention to follow.
Thanks
Here's the snippet

class MnistModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)

    def forward(self, xb):
        xb = xb.reshape(-1, 784)
        out = self.linear(xb)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        print(type(out))
        return loss

Output:
It refers to an instance of MnistModel, the same as in any other method defined by the class. The only thing odd is that self is called, but that's explained by the fact that nn.Module defines __call__, so all instances of MnistModel are themselves callable.
out = self(images) is equivalent to out = self.__call__(images).


Instruction: TRANSFORMERS: Asking to pad but the tokenizer does not have a padding token

Input:
In trying to evaluate a bunch of transformers models sequentially with the same dataset to check which one performs better.
The list of models is this one:
MODELS = [
      ('xlm-mlm-enfr-1024'   ,"XLMModel"),
      ('distilbert-base-cased', "DistilBertModel"),
      ('bert-base-uncased'     ,"BertModel"),
      ('roberta-base'        ,"RobertaModel"),
      ("cardiffnlp/twitter-roberta-base-sentiment","RobertaSentTW"),
      ('xlnet-base-cased'     ,"XLNetModel"),
      #('ctrl'                ,"CTRLModel"),
      ('transfo-xl-wt103'    ,"TransfoXLModel"),
      ('bert-base-cased'       ,"BertModelUncased"),
      ('xlm-roberta-base'     ,"XLMRobertaModel"),
      ('openai-gpt'           ,"OpenAIGPTModel"),
      ('gpt2'                 ,"GPT2Model")

All of them work fine until 'ctrl' model, which returns this error:
Asking to pad but the tokenizer does not have a padding token. Please select a token to use as 'pad_token' '(tokenizer.pad_token = tokenizer.eos_token e.g.)' or add a new pad token via 'tokenizer.add_special_tokens({'pad_token': '[PAD]'})'.
When tokenizing the sentences of my dataset.
The tokenizing code is
SEQ_LEN = MAX_LEN #(50)

for pretrained_weights, model_name in MODELS:

print("***************** INICIANDO " ,model_name,", weights ",pretrained_weights, "********* ")
print("carganzo el tokenizador ()")
tokenizer = AutoTokenizer.from_pretrained(pretrained_weights)
print("creando el modelo preentrenado")
transformer_model = TFAutoModel.from_pretrained(pretrained_weights)
print("aplicando el tokenizador al dataset")

##APLICAMOS EL TOKENIZADOR##

def tokenize(sentence):

  tokens = tokenizer.encode_plus(sentence, max_length=MAX_LEN,
                               truncation=True, padding='max_length',
                               add_special_tokens=True, return_attention_mask=True,
                               return_token_type_ids=False, return_tensors='tf')
  return tokens['input_ids'], tokens['attention_mask']

# initialize two arrays for input tensors
Xids = np.zeros((len(df), SEQ_LEN))
Xmask = np.zeros((len(df), SEQ_LEN))

for i, sentence in enumerate(df['tweet']):
    Xids[i, :], Xmask[i, :] = tokenize(sentence)
    if i % 10000 == 0:
        print(i)  # do this so we can see some progress

arr = df['label'].values  # take label column in df as array

labels = np.zeros((arr.size, arr.max()+1))  # initialize empty (all zero) label array
labels[np.arange(arr.size), arr] = 1  # add ones in indices where we have a value`

I have tried to define the padding tokens as the solution tells me, but then this error appears
could not broadcast input array from shape (3,) into shape (50,)

in line
Xids[i, :], Xmask[i, :] = tokenize(sentence)

I have also tried this solution and doesn't work neither.
If you have managed to read until here, thank you.
Any help is needed.

Output:
You can add the [PAD] token using add_special_tokens API.
tokenizer = AutoTokenizer.from_pretrained(pretrained_weights)
if tokenizer.pad_token is None:
    tokenizer.add_special_tokens({'pad_token': '[PAD]'})


Instruction: torch dataloader for large csv file - incremental loading

Input:
I am trying to write a custom torch data loader so that large CSV files can be loaded incrementally (by chunks).
I have a rough idea of how to do that. However, I keep getting some PyTorch error that I do not know how to solve.

import numpy as np
import pandas as pd
import torch
from torch.utils.data import Dataset, DataLoader

# Create dummy csv data
nb_samples = 110
a = np.arange(nb_samples)
df = pd.DataFrame(a, columns=['data'])
df.to_csv('data.csv', index=False)

# Create Dataset
class CSVDataset(Dataset):
    def __init__(self, path, chunksize, nb_samples):
        self.path = path
        self.chunksize = chunksize
        self.len = nb_samples / self.chunksize

    def __getitem__(self, index):
        x = next(
            pd.read_csv(
                self.path,
                skiprows=index * self.chunksize + 1,  #+1, since we skip the header
                chunksize=self.chunksize,
                names=['data']))
        x = torch.from_numpy(x.data.values)
        return x

    def __len__(self):
        return self.len

dataset = CSVDataset('data.csv', chunksize=10, nb_samples=nb_samples)
loader = DataLoader(dataset, batch_size=10, num_workers=1, shuffle=False)

for batch_idx, data in enumerate(loader):
    print('batch: {}\tdata: {}'.format(batch_idx, data))

I get 'float' object cannot be interpreted as an integer error

Output:
The error is caused by this line:
self.len = nb_samples / self.chunksize

When dividing using / the result is always a float. But you can only return an integer in the __len__() function. Therefore you have to round self.len and/or convert it to an integer. For example by simply doing this:
self.len = nb_samples // self.chunksize

the double slash (//) rounds down and converts to integer.
Edit:
You acutally CAN return a float in __len__() but when calling len(dataset) the error will occur. So I guess len(dataset) is called somewhere inside the DataLoader class.


Instruction: No attribute 'RRef' when loading .ckpt files on WIndows machine?

Input:
I generated ckpt files with Pytorch Lightning's ModelCheckpoint(save_last=True) on my cluster which uses linux.
On the cluster itself I can load them without problems, but on my Windows machine I cant and get this error:
AttributeError: module 'torch.distributed.rpc' has no attribute 'RRef'
I really need help, as I have a deadline in 3 hours. There has to be a way right so that my code is reproducible?

Output: It seems not to be possible to load the model on a windows system when it has been trained on a Linux system. The only work around I have found is install Ubuntu as a virtual machine on my windows system. This is quite easy. https://apps.microsoft.com/store/detail/ubuntu-2204-lts/9PN20MSR04DW


Instruction: User warning when exporting Pytorch model to ONNX

Input:
I have found some code that directly converts the pytorch model to onnx:
import torch.onnx
torch.onnx.export(
    model,
    input,
    "model.onnx",
    export_params=True,
    opset_version=10
)

But it throws UserWarning most of the time :-
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:2359: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
_verify_batch_size([input.size(0) * input.size(1) // num_groups, num_groups] + list(input.size()[2:]))
/usr/local/lib/python3.7/dist-packages/torch/onnx/symbolic_opset9.py:1934: UserWarning: ONNX export unsqueeze with negative axis -1 might cause the onnx model to be incorrect. Negative axis is not supported in ONNX. Axis is converted to 1 based on input shape at export time. Passing an tensor of different rank in execution will be incorrect.
"Passing an tensor of different rank in execution will be incorrect.")
/usr/local/lib/python3.7/dist-packages/torch/onnx/symbolic_opset9.py:1934: UserWarning: ONNX export unsqueeze with negative axis -1 might cause the onnx model to be incorrect. Negative axis is not supported in ONNX. Axis is converted to 2 based on input shape at export time. Passing an tensor of different rank in execution will be incorrect.
"Passing an tensor of different rank in execution will be incorrect.")
Can you explain why I am getting this error and is this method correct for exporting to onnx or can you suggest any better method for exporting complex pytorch model to onnx ?

Output:
The reason is given directly in the warning message. Since PyTorch1.10, the floordiv is deprecated. You need to update input.size(1) // num_groups to torch.div(input.size(1), num_groups, rounding_mode='floor') if you wish to avoid the warning.
But it is indeed weird that the // should be considered as torch. floor_divide only when a torch.Tensor is included as operand. This might has something to do with the onnx export logic. Hopefully someone more familar with the underlying logic could give a deeper answer.


Instruction: Is there any thing like 'TensorList’ in pytorch?

Input:
I would like to put some tensor in a list, and I know if I would like to put nn.Module class into a list, I must use ModuleList to wrap that list.
So, Is there anything like 'TensorList’ in pytorch, that I must use to wrap the list containing tensors?

Output:
What are these tensors? Are these tensors parameters of your nn.Module? If so, you need to use the proper container.
For example, using nn.ParameterList. This way calling your module's .paramters() methods will yield these tensors as well. Otherwise you'll get errors like this one.


Instruction: PyTorch RuntimeError t == DeviceType::CUDAINTERNAL ASSERT FAILED

Input:
A PyTorch Lightning model works perfectly well on CPU using this Trainer configuration:
trainer = Trainer(
    gpus=0,
    max_epochs=10,
    gradient_clip_val=2,
    callbacks=[pl.callbacks.progress.TQDMProgressBar(refresh_rate=5)],
)

trainer.fit(model)

But running the exact same model on GPU (by changing gpus=-1 or gpus=1 in the above code) triggers the following error:
RuntimeError: t == DeviceType::CUDAINTERNAL ASSERT FAILED
at "../c10/cuda/impl/CUDAGuardImpl.h":24, please report a bug to PyTorch.

The model is as follows:
class TorchModel(LightningModule):
    def __init__(self):
        super(TorchModel, self).__init__()
        self.cat_layers = ModuleList([TorchCatEmbedding(cat) for cat in columns_to_embed])
        self.num_layers = ModuleList([LambdaLayer(lambda x: x[:, idx:idx+1]) for _, idx in numeric_columns])
        self.ffo = TorchFFO(len(self.num_layers) + sum([embed_dim(l) for l in self.cat_layers]), y.shape[1])
        self.softmax = torch.nn.Softmax(dim=1)

    def forward(self, inputs):
        cats = [c(inputs) for c in self.cat_layers]
        nums = [n(inputs) for n in self.num_layers]
        concat = torch.cat(cats + nums, dim=1)
        out = self.ffo(concat)
        out = self.softmax(out)
        return out

    def training_step(self, train_batch, batch_idx):
        x, y = train_batch
        y_hat = self.forward(x)
        loss = cce(torch.log(torch.maximum(torch.tensor(1e-8), y_hat)), y.argmax(dim=1))
        acc = tm.functional.accuracy(y_hat.argmax(dim=1), y.argmax(dim=1))
        self.log("loss", loss)
        self.log("acc", acc, prog_bar=True)
        self.log("lr", self.scheduler.get_last_lr()[-1], prog_bar=True)
        return loss

with TorchCatEmbedding and TorchFFO being two sub-models.
Is there any way to solve this issue?
PyTorch version:
>>> torch.__version__
'1.10.1+cu113'

Cuda information:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+

Output:
This was due to a torch.tensor() declaration that wasn't transferred to GPU in the training step:
def training_step(self, train_batch, batch_idx):
    x, y = train_batch
    y_hat = self.forward(x)
    loss = cce(torch.log(torch.maximum(torch.tensor(1e-8), y_hat)), y.argmax(dim=1))
    acc = tm.functional.accuracy(y_hat.argmax(dim=1), y.argmax(dim=1))
    self.log("loss", loss)
    self.log("acc", acc, prog_bar=True)
    self.log("lr", self.scheduler.get_last_lr()[-1], prog_bar=True)
    return loss

Changing this:
loss = cce(
    torch.log(torch.maximum(torch.tensor(1e-8), y_hat)),
    y.argmax(dim=1)
)

to this by adding .type_as(y_hat):
loss = cce(
    torch.log(torch.maximum(torch.tensor(1e-8).type_as(y_hat), y_hat)),
    y.argmax(dim=1)
)

solved the issue.


Instruction: conv1d() received an invalid combination of arguments

Input:
I tried to repeat https://github.com/munhouiani/Deep-Packet and came across an error
This program uses CNN to classify network traffic. I decided to rewrite the program as I could not run the original on my computer. I am new to neural networks, so I cannot give a detailed description of the problem
TypeError: conv1d() received an invalid combination of arguments - got (list, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (!list!, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (!list!, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)

code:
import torch
from pathlib import Path
from torch import nn as nn
from torch.nn import functional as F
from torch.utils.data import DataLoader
from argparse import Namespace
from pytorch_lightning import Trainer
import pytorch_lightning as pl
import numpy as np
class CNN(pl.LightningModule):
    def __init__(self, hparams):
        super().__init__()
        # config
        self.save_hyperparameters(hparams)
        self.data_path = self.hparams.data_path

        # two convolution, then one max pool
        self.conv1 = nn.Sequential(
            nn.Conv1d(
                in_channels=1,
                out_channels=self.hparams.c1_output_dim,
                kernel_size=self.hparams.c1_kernel_size,
                stride=self.hparams.c1_stride
            ),
            nn.ReLU()
        )
        self.conv2 = nn.Sequential(
            nn.Conv1d(
                in_channels=self.hparams.c1_output_dim,
                out_channels=self.hparams.c2_output_dim,
                kernel_size=self.hparams.c2_kernel_size,
                stride=self.hparams.c2_stride
            ),
            nn.ReLU()
        )

        self.max_pool = nn.MaxPool1d(
            kernel_size=2
        )

        # flatten, calculate the output size of max pool
        # use a dummy input to calculate
        dummy_x = torch.rand(1, 1, self.hparams.signal_length, requires_grad=False)
        dummy_x = self.conv1(dummy_x)
        dummy_x = self.conv2(dummy_x)
        dummy_x = self.max_pool(dummy_x)
        max_pool_out = dummy_x.view(1, -1).shape[1]

        # followed by 5 dense layers
        self.fc1 = nn.Sequential(
            nn.Linear(
                in_features=max_pool_out,
                out_features=200
            ),
            nn.Dropout(p=0.05),
            nn.ReLU()
        )
        self.fc2 = nn.Sequential(
            nn.Linear(
                in_features=200,
                out_features=100
            ),
            nn.Dropout(p=0.05),
            nn.ReLU()
        )
        self.fc3 = nn.Sequential(
            nn.Linear(
                in_features=100,
                out_features=50
            ),
            nn.Dropout(p=0.05),
            nn.ReLU()
        )

        # finally, output layer
        self.out = nn.Linear(
            in_features=50,
            out_features=self.hparams.output_dim
        )

    def forward(self, x):
        # make sure the input is in [batch_size, channel, signal_length]
        # where channel is 1
        # signal_length is 1500 by default
        #batch_size = x.shape[0]
        batch_size = 16

        # 2 conv 1 max
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.max_pool(x)

        x = x.reshape(batch_size, -1)

        # 3 fc
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)

        # output
        x = self.out(x)

        return x
    def train_dataloader(self):
        reader = self.data_path
        dataloader = DataLoader(reader, batch_size=16)

        return dataloader
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())

    def training_step(self, batch, batch_idx):
        x = batch
        y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)

        if (batch_idx % 50) == 0:
            self.logger.log_metrics(loss, step=batch_idx)
        return loss
num_epochs = 6
num_classes = 10
batch_size = 100
learning_rate = 0.001
train_dataset = "D:\Deep-Packet-master\Deep-Packet-master\processed_data"
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
hparams = Namespace(**{
    'c1_kernel_size': 4,
    'c1_output_dim': 200,
    'c1_stride': 3,
    'c2_kernel_size': 5,
    'c2_output_dim': 200,
    'c2_stride': 1,
    'output_dim': 17,
    'data_path': train_dataset,
    'signal_length': 1500,
    'epoch': 6
})
model = CNN(hparams).float()
gpus = None
trainer = Trainer(val_check_interval=4, max_epochs=1)
trainer.fit(model)
trainer.save_checkpoint(str(train_dataset.absolute()))

Please, help

Output:
I'm going to guess that your training_step is incorrect:
def training_step(self, batch, batch_idx):
    x = batch[0]
    y = batch[1]
    y_hat = self(x)
    loss = F.cross_entropy(y_hat, y)

    if (batch_idx % 50) == 0:
        self.logger.log_metrics(loss, step=batch_idx)
    return loss

In your code, you set both x and y to batch which should be a tuple or a list, which conv1d's forward cannot interpret.


Instruction: CPU utilization when using ray and torch

Input:
I use ray and torch in my code and set one CPU core for each ray remote actor
to  compute gradient(use torch package).   But I find the CPU utilization of the actor
can go up to 300% in some time, This seems to be impossible since The actor is supposed to use
only one CPU core.
I want to know if the actor  is actually using more CPU resources since torch may open one or more
threads to compute gradient.
My OS is Win10 and CPU is Ryzen 5600H. Thanks.

Output:
Ray currently does not automatically pin the actor to specific CPU cores and prevent it from using other CPU cores. So what you're seeing makes sense.
It is possible to use a library like psutil to pin the actor to a specific core and prevent it from using other cores. This can be helpful if you have many parallel tasks/actors that are all multi-threaded and competing with each other for resources (e.g., because they use pytorch or numpy).


Instruction: How to retain node ordering when converting graph from networkx to pytorch geometric?

Input:
Question: How to retain the node ordering/labels when converting a graph from networkx to pytorch geometric?
Code: (to be run in Google Colab)
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import networkx as nx

import torch
from torch.nn import Linear
import torch.nn.functional as F
torch.__version__

# install pytorch geometric
!pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.10.0+cpu.html

from torch_geometric.nn import GCNConv
from torch_geometric.utils.convert import to_networkx, from_networkx

# Make the networkx graph
G = nx.Graph()

# Add some cars
G.add_nodes_from([
      ('Ford', {'y': 0, 'Name': 'Ford'}),
      ('Lexus', {'y': 1, 'Name': 'Lexus'}),
      ('Peugot', {'y': 2, 'Name': 'Peugot'}),
      ('Mitsubushi', {'y': 3, 'Name': 'Mitsubishi'}),
      ('Mazda', {'y': 4, 'Name': 'Mazda'}),
])

# Relabel the nodes
remapping = {x[0]: i for i, x in enumerate(G.nodes(data = True))}

G = nx.relabel_nodes(G, remapping, copy=False)

# Add some edges --> A = [(0, 1, 0, 1, 1), (1, 0, 1, 1, 0), (0, 1, 0, 0, 1), (1, 1, 0, 0, 0), (1, 0, 1, 0, 0)] as the adjacency matrix
G.add_edges_from([
                  (0, 1), (0, 3), (0, 4),
                  (1, 2), (1, 3),
                  (2, 1), (2, 4),
                  (3, 0), (3, 1),
                  (4, 0), (4, 2)
])

# Convert the graph into PyTorch geometric
pyg_graph = from_networkx(G)

pyg_graph.edge_index

When I print the edge indices in the last line of the code, I get different answers each time I run it. Most importantly, I am looking to consistently get the same (correct) answer whereby each node numbering is retained from networkx:
tensor([[0, 0, 1, 1, 1, 2, 2, 3, 3, 4, 4, 4],
        [4, 2, 4, 2, 3, 0, 1, 1, 4, 0, 1, 3]])

The form of this edge index tensor is:

the first list contains the node ids of the source node
the second list contains the node ids of the target node

For the node ids to be retained, we would expect node 0 to appear three times in the first (source) list instead of just twice.
Is there any way for me to force PyTorch Geometric to copy over the node ids?
Thanks
[EDIT] One possible work-around I have is using the following bit of code which is able to produce edge index and weight tensors for PyTorch geometric
# Create a dictionary of the mappings from company --> node id
mapping_dict = {x: i for i, x in enumerate(list(G.nodes()))}

# Get the number of nodes
num_nodes = len(mapping_dict)

# Now create a source, target, and edge list for PyTorch geometric graph
edge_source_list = []
edge_target_list = []
edge_weight_list = []

# iterate through all the edges
for e in G.edges():
  # first element of tuple is appended to source edge list
  edge_source_list.append(mapping_dict[e[0]])

  # last element of tuple is appended to target edge list
  edge_target_list.append(mapping_dict[e[1]])

  # add the edge weight to the edge weight list
  edge_weight_list.append(1)

# now create full edge lists for pytorch geometric - undirected edges need to be defined in both directions

full_source_list = edge_source_list + edge_target_list      # full source list
full_target_list = edge_target_list + edge_source_list      # full target list
full_weight_list = edge_weight_list + edge_weight_list      # full edge weight list

print(len(edge_source_list), len(edge_target_list), len(full_source_list))

# now convert these to torch tensors
edge_index_tensor = torch.LongTensor( np.concatenate([ [np.array(full_source_list)], [np.array(full_target_list)]] ))
edge_weight_tensor = torch.FloatTensor(np.array(full_weight_list))

Output:
It seems this issue was resolved in the comments (the solution proposed by @Sparky05 is to use copy=True, which is the default for nx.relabel_nodes), but below is the explanation for why the node order is changed.
When copy=False is passed, nx.relabel_nodes will re-add the nodes to the graph in the order they appear in the set of keys of remapping dict. The relevant lines in the code are here:
def _relabel_inplace(G, mapping):
    old_labels = set(mapping.keys())
    new_labels = set(mapping.values())
    if len(old_labels & new_labels) > 0:
        # skip codes for labels sets that overlap
    else:
        # non-overlapping label sets
        nodes = old_labels

    # skip lines
    for old in nodes: # this is now in the set order

By using set the order of nodes is modified, so to preserve the order the non-overlapping label sets should be treated as:
    else:
        # non-overlapping label sets
        nodes = mapping.keys()

A related PR is submitted here.


Instruction: Concatenate a tensor to another in PyTorch

Input:
I want to do the following thing: I have a tensor x
x

tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0.]])

and a tensor temp=torch.tensor([[298.]])
How can I quickly concatenate it to each of the above vectors obtaining something like
x

tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 298.],
        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 298.]])

what's the quickest way to do that?

Output:
perhaps not the most straight forward approach, but nonetheless - you can use torch.cat on x.T once you vectorized temp:
temp_vec = temp * torch.ones(x.shape[0])
torch.cat((x.T,temp_vec)).T

testing it out for a smaller x (to not clutter the answer):
x = torch.Tensor([[0.0, 0.0, 0.0],
                  [0.0, 0.0, 0.0]])

out:
tensor([[  0.,   0.,   0., 298.],
        [  0.,   0.,   0., 298.]])

last note: if I'm not mistaken, the transpose .T is a field of every tensor, so using it isn't inefficient per se


Instruction: Can we install Pytorch CUDA 11.3 when the system has CUDA 11.2

Input:
I have cuda 11.2 in my PC and want to install PyTorch.
PyTorch has only mentions of CUDA10.2 and 11.3 in it's website
Can I install torch==1.10.1+cu113 on my PC?
If not, how can I install PyTorch for CUDA11.2
I don't want to change my CUDA version as I have other applications using it.

Output:
I tried the one for 11.3 and so far it works fine w/ the GPU:
sudo pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113


Instruction: Can someone please clarify what happens in _getitem_ function? Thanks

Input:
I understand that output contains the all of encodings, token type ids, attention_mask, and corresponding labels as tensors. I would like to understand the inner working of getitem function and the need of getting label lengths with len function.
class NewsGroupsDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
        item["labels"] = torch.tensor([self.labels[idx]])
        return item

    def __len__(self):
        return len(self.labels)

# convert our tokenized data into a torch Dataset
train_dataset = NewsGroupsDataset(train_encodings, train_labels)
valid_dataset = NewsGroupsDataset(valid_encodings, valid_labels)

Output:
Python defines many special methods for classes. These methods define the behavior of the class in certain situations. You're probably already familiar with the __init__ special method, that gets invoked when a new instance of the class is created. __getitem__ is another special method that is called when you use subscription on an instance of the class (i.e. square brackets []), and __len__ is called when you use pass an instance of your class to the built-in len function.
As for Pytorch, we must implement these methods because that's what Pytorch's DataLoader object expects. It uses these methods to sample your dataset and know when it is done sampling the dataset. Though DataLoader uses many abstractions to support different sampling and multi-process operations, it basically needs __len__ in order to know the maximum index it can query from the dataset, and it uses __getitem__ to sample the indices it needs.
For example, when you're using random sampling with 0 workers, the following snippets do effectively the same thing
from torch.utils.data import DataLoader

train_dataset = NewsGroupsDataset(train_encodings, train_labels)
data_loader = DataLoader(train_dataset, batch_size=5, shuffle=True)
for items in data_loader:
    # items now contains batches of samples of size 5 from your dataset

and
# For demonstration purposes only, do NOT sample your datasets like this (use DataLoader)!
import random
from torch.utils.data import default_collate

def random_batches(dataset, batch_size, shuffle):
    indices = list(range(len(dataset)))  # uses Dataset.__len__
    if shuffle:
        random.shuffle(indices)
    batch = []
    for i in indices:
        batch.append(dataset[i])  # uses Dataset.__getitem__
        if len(batch) == batch_size:
            yield default_collate(batch)
            batch = []
    if batch:
        yield default_collate(batch)

train_dataset = NewsGroupsDataset(train_encodings, train_labels)
for items in random_batches(train_dataset, batch_size=5, shuffle=True):
    # items now contains batches of samples of size 5 from your dataset

Note that default_collate is a function that takes a list of samples and converts them into stacks of batch-sized tensors. The implementation can be found here if you're curious of the details.
DataLoader also supports a lot of other cool things, like multiple workers (probably most important), custom sampling schemes, custom data collation, pinned memory, dropping last non-full batch, and more. Pytorch does most of the work for you with this class, you just need to write the dataset object with __len__ and __getitem__ implementations.


Instruction: Training loss decreases dramatically after first epoch and validation loss unstable

Input: I am using EfficientNet-B0 as a subnet in Siamese network and contrastive loss as a loss function for an image similarity task. My dataset is quite large (27550 images for training) with 2 classes. After the first epoch, the training loss decreases dramatically while the validation loss is unstable. Can overfitting happen this early? Or is there something wrong with my data that is confusing? Here is the graph I get after training my model with 100 epochs

Output:
First, Draw the training and validation loss by setting up a lower and variable learning_rate. This might happen because of higher learning rate.
Secondly, we all knows that the model Overfits, when the training loss is way smaller than the testing loss. By using, dropout, regularization and deeper model (vgg, ResNet) you can improve it.


Instruction: how can i sum the size of this tensors?

Input:
I have different sizes or shapes for each tensor like
torch.Size([1, 12, 1000])
torch.Size([1, 12, 1000])
torch.Size([1, 10, 1000])
torch.Size([1, 11, 1000])
torch.Size([1, 11, 1000])
torch.Size([1, 15, 1000])
torch.Size([1, 10, 1000])
....

and need to be like torch.Size((12+12+10+11+11+15+ .... ),1000)
my code is
def extract():
     for y in sentences:
            tokenized_text = tokenizer.tokenize(y)
            indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
            tokens_tensor = torch.tensor([indexed_tokens])
            model.eval()
            outputs = model(tokens_tensor)
            hidden_states = outputs[2]
            my_tensor = torch.cat([hidden_states[i] for i in [-1,-2,-3,-4]], dim=-1)
     return my_tensor

Output:
Concatenate them:
tensors = [t1, t2, t3, ...]
result = torch.cat(tensors, dim=1)

# result.size(): torch.Size([1, 12+12+10+..., 1000])

If you also want to remove the first dimension, as it has size 1:
result = result.squeeze()
# result.size(): torch.Size([12+12+10+..., 1000])


Instruction: Formulae for calculating the shape of feature maps after convolutions

Input:
I know that Pytorch's documentation provides this, but I have difficulties in understanding their notation.
Is there any more accessible explanation (maybe also with graphical illustrations)?

Output:
I think you are looking for Receptive Field Arithmetics.
This webpage provides a detailed explanation of the various factors affecting the size of the receptive field, and the shape of the resulting feature maps.


Instruction: Workaround to successfully profile python script using scalene profiler on macOS? Just forget it and use machine with Windows or Linux?

Input:
Computational Science SE question How amenable is this 2D Frenkel–Kontorova-like energy minimization problem in Python to the use of a modest PC + GPU? (Heavy reliance on indexing) contains a short example script and note 3 links to my first attempt at profiling using scalene
The results were uninformative, so I followed a recommendation that I try the --profile-all option. After running for 30 minutes without finishing on a script that took seconds to run I added a CPU percent limit; what I assume means only things that used at least 2% of the CPU time would be profiled in depth.
scalene --html --outfile prof.html  --profile-all --cpu-percent-threshold 2 myscript.py

I received a two line error and was exited from python in a normal way.
Error getting real path: 2
Scalene error: received signal SIGABRT

This issue was closed in scalene issue #110 and there are links there to

https://github.com/plasma-umass/scalene/commit/6636d95d7ea9a16adedaedbd4ff0926374798f2b

Q: How do I use Scalene with PyTorch?
A: Scalene works with PyTorch version 1.5.1. There's a bug in newer versions of PyTorch (https://github.com/pytorch/pytorch/issues/57185) that interferes with Scalene (discussion here: https://github.com/plasma-umass/scalene/issues/110).

there's more information there as well
and

Pytorch open issue #57185 Segmentation fault with ITIMER_REAL PyTorch throws SIGSEGV when running alongside timer on MacOS x86

Question: Is there a workaround to successfully profile python script using scalene profiler on macOS? Or should I just move to a machine with Windows or Linux and forget about trying for now?

Output:
The message, Error getting real path: 2, seems to have to do with scalene finding your script or a path internal to your script possibly.
Ensure you're referencing a full, valid path for myscript.py, taking into account which directory you're at in the terminal. You may need to change directory.
The SIGABRT message is different from the SIGSEGV issue listed, though, if you wanted to test it, you could uninstall PyTorch (pip uninstall torch) and reinstall a specific version (pip install 'torch==1.5.1').


Instruction: Installing PyTorch on MacOS Big Sur

Input:
I am trying to figure out how to go about installing PyTorch on my computer which is a macOS Big Sur laptop (version 11.6.2). So far, I have installed Python 3.10.1 via the Python website, and pip 21.3.1 was installed along with it. At the moment, I’m stuck trying to figure out how to install PyTorch using pip?
I’m asking this question because I am partaking in a project that requires PyTorch and I need to install it as soon as possible.

Output:
pip3 install torch torchvision torchaudio

This command worked fine for me, you can find more information on the official website here


Instruction: AssertionError: Torch not compiled with CUDA enabled (depite several reinstallations)

Input:
Whenever I try to move a variable to cuda in pytorch (e.g. torch.zeros(1).cuda(), I get the error message "AssertionError: Torch not compiled with CUDA enabled". Besides,torch.cuda.is_available() returns False.
I have read several answers to approaching this error but for some reason several attempts to reinstall cuda and putorch didn't change anything. Here are some of the settings I used:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

conda install pytorch torchvision cudatoolkit=11 -c pytorch-nightly

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

Yet the same error remains. What could be the issue?
Some settings:
I'm using Ubuntu 20.04, GPU is RTX 2080, nvidia-smi works fine (NVIDIA-SMI 460.91.03, Driver Version: 460.91.03, (max possible) CUDA Version: 11.2)

Output:
Try installing with pip
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
You can go through this thread for detailed explanations
Pytorch for cuda 11.2


Instruction: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

Input:
I get the following error message which I tried to deal with it by throwing .to(self.device) everywhere but it doesn't work.
  ab = torch.lgamma(torch.tensor(a+b, dtype=torch.float, requires_grad=True).to(device=local_device))
Traceback (most recent call last):
  File "Script.py", line 923, in <module>
    average_epoch_loss, out , elbo2 =train(epoch)
  File "Script.py", line 848, in train
    loss_dict = net.get_ELBO(X)
  File "Script.py", line 546, in get_ELBO
    elbo -= compute_kumar2beta_kld(self.kumar_a[:, k].to(self.device), self.kumar_b[:, k].to(self.device), self.prior, (self.K-1-k)* self.prior).mean().to(self.device)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Here is a snippet of my script which is related to the error:
def compute_kumar2beta_kld(a, b, alpha, beta):
    SMALL = 1e-16
    EULER_GAMMA = 0.5772156649015329
    ab    = torch.mul(a,b)+ SMALL
    a_inv = torch.pow(a + SMALL, -1)
    b_inv = torch.pow(b + SMALL, -1)
    kl = torch.mul(torch.pow(1+ab,-1), beta_fn(a_inv, b))
    for idx in range(10):
        kl += torch.mul(torch.pow(idx+2+ab,-1), beta_fn(torch.mul(idx+2., a_inv), b))
    kl = torch.mul(torch.mul(beta-1,b), kl)
    psi_b = torch.digamma(b+SMALL)
    kl += torch.mul(torch.div(a-alpha,a+SMALL), -EULER_GAMMA - psi_b - b_inv)

    kl += torch.log(ab) + torch.log(beta_fn(alpha, beta) + SMALL)

    kl += torch.div(-(b-1),b +SMALL)
    return kl

class VAE(GMMVAE):
    def __init__(self, hyperParams, K, nchannel, base_channels, z_dim, w_dim, hidden_dim, device, img_width, batch_size, include_elbo2):
        global local_device
        local_device = device
        super(VAE, self).__init__(K, nchannel, base_channels, z_dim, w_dim, hidden_dim,  device, img_width, batch_size)

        self.prior      = hyperParams['prior']
        self.K          = hyperParams['K']
        self.z_dim      = hyperParams['latent_d']
        self.hidden_dim = hyperParams['hidden_d']
    def get_ELBO(self, X):

        elbo = torch.tensor(0, dtype=torch.float)
        if self.include_elbo2:
            for k in range(self.K-1):
                elbo -= compute_kumar2beta_kld(self.kumar_a[:, k], self.kumar_b[:, k], self.prior, (self.K-1-k)* self.prior).mean().to(self.device)

I appreciate it if someone can suggest how to fix the error.

Output:
I am not sure if this is the "only" problem, but one of the device-related problems is this:
 elbo = torch.tensor(0, dtype=torch.float) <- this will create the elbo tensor on CPU
and when you do, elbo -= <some result>,
The result is on cuda (or self.device). This will clearly cause a problem. To fix, this just do
elbo = torch.tensor(0, dtype=torch.float, device=self.device)


Instruction: How can I register a local model.mar to a running torchserve service?

Input:
I have a running torchserve service. According to the docs, I can register a new model at port 8081 with the ManagementAPI. When running curl -X OPTIONS http://localhost:8081, the output also states for the post request on /models:
    ...
    "post": {
        "description": "Register a new model in TorchServe.",
        "operationId": "registerModel",
        "parameters": [
          {
            "in": "query",
            "name": "url",
            "description": "Model archive download url, support local file or HTTP(s) protocol. For S3, consider use pre-signed url.",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
     ...

Now, I do have a local model.mar file and want to use it with the file protocol. I tried:
curl -X POST "http://localhost:8081/models?url=file:///path/to/model.mar"

But got:
{
  "code": 400,
  "type": "DownloadArchiveException",
  "message": "Failed to download archive from: file:///path/to/model.mar"
}

Is there anything that I am missing? How can I properly register a local model.mar to a running torchserve service?

Output:
I just figured it out. Everything I stated was completely correct and under normal circumstances, all of this would have worked.
The error is arising because I am running the torchserve instance in a docker container and the curl command is sent to this container which then looks in his local files for the model.mar. I thought for whatever reason that I can pass a file path from the machine that the docker container is running on.
Instead, I have to copy the file into the docker container first (or mount a directory) and then execute the registering command with the file path from the model.mar inside the docker container.


Instruction: two pytorch DistributedSampler same seeds different shuffling multiple GPU-s

Input:
I am trying to load two version (original and principal component pursuit (PCP) cleaned version) of the very same image data set for training a modell using pytorch on a multiple GPUs remote machine.
I would like to ensure the same shuffling order for both the original and the PCP cleaned data. To achieve this, I use torch.utils.data.DistributedSampler(datasetPCP, shuffle=True, seed=42) and torch.utils.data.DistributedSampler(dataset, shuffle=True, seed=42) samplers to pass these to the dataloaders for train my modell on 3 GPUs are present on the remote machine I use.
As far as I understood the same seed of the two sampler should ensure the exact same shuffling of the loaded data. However this is not the case.
Could anybody point me in the right direction?
Thanks a lot!

Output:
DistributedSampler is for distributed data training where we want different data to be sent to different processes so it is not what you need. Regular dataloader will do just fine.
Example:
import torch
from torch.utils.data.dataset import Dataset
from torch.utils.data import DataLoader, RandomSampler

class ToyDataset(Dataset):
    def __init__(self, type):
        self.type = type

    def __getitem__(self, idx):
        return f'{self.type}, {idx}'

    def __len__(self):
        return 10

def get_sampler(dataset, seed=42):
    generator = torch.Generator()
    generator.manual_seed(seed)
    sampler = RandomSampler(dataset, generator=generator)
    return sampler

original_dataset = ToyDataset('original')
pcp_dataset = ToyDataset('pcp')

original_loader = DataLoader(original_dataset, batch_size=2, sampler=get_sampler(original_dataset))
pcp_loader = DataLoader(pcp_dataset, batch_size=2, sampler=get_sampler(pcp_dataset))

for data in original_loader:
    print(data)

for data in pcp_loader:
    print(data)

Output:
['original, 2', 'original, 6']
['original, 1', 'original, 8']
['original, 4', 'original, 5']
['original, 0', 'original, 9']
['original, 3', 'original, 7']
['pcp, 2', 'pcp, 6']
['pcp, 1', 'pcp, 8']
['pcp, 4', 'pcp, 5']
['pcp, 0', 'pcp, 9']
['pcp, 3', 'pcp, 7']


Instruction: problems on google colab pytorch learning-RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

Input:
I'm very newbie on pytorch and deep learning, and I got some error while running a sample code from deep learning class.
When I run the code I attached below, There comes an errors like,
text = torch.from_numpy(data['text']).long().cuda(0)

# feature extraction
mel_gt = get_mel(audio)

# shift mel spectrogram -> the input of the network
mel_shift = torch.cat((torch.zeros_like(mel_gt)[:,:,:1], mel_gt[:,:,:-1]) ,axis=-1)

# inference
mel_est, attention = model(mel_shift, text)

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
I can hardly understand because I also check whether the tensors are on cuba with these code, they all return true..
print(mel_shift.is_cuda)
print(mel_gt.is_cuda)
print(text.is_cuda)

Can you guys figure out what's the problem?? I need big big help,,

Output:
Check if your model is loaded on cuda by running
next(model.parameters()).is_cuda.
If it returns False, load the model on CUDA using
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device)


Instruction: onnxruntime inference is way slower than pytorch on GPU

Input:
I was comparing the inference times for an input using pytorch and onnxruntime and I find that onnxruntime is actually slower on GPU while being significantly faster on CPU
I was tryng this on  Windows 10.

ONNX Runtime installed from source - ONNX Runtime version: 1.11.0 (onnx version 1.10.1)
Python version - 3.8.12
CUDA/cuDNN version - cuda version 11.5, cudnn version 8.2
GPU model and memory - Quadro M2000M, 4 GB

Relevant code -
import torch
from torchvision import models
import onnxruntime    # to inference ONNX models, we use the ONNX Runtime
import onnx
import os
import time

batch_size = 1
total_samples = 1000
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

def convert_to_onnx(resnet):
   resnet.eval()
   dummy_input = (torch.randn(batch_size, 3, 224, 224, device=device)).to(device=device)
   input_names = [ 'input' ]
   output_names = [ 'output' ]
   torch.onnx.export(resnet,
               dummy_input,
               "resnet18.onnx",
               verbose=True,
               opset_version=13,
               input_names=input_names,
               output_names=output_names,
               export_params=True,
               do_constant_folding=True,
               dynamic_axes={
                  'input': {0: 'batch_size'},  # variable length axes
                  'output': {0: 'batch_size'}}
               )

def infer_pytorch(resnet):
   print('Pytorch Inference')
   print('==========================')
   print()

   x = torch.randn((batch_size, 3, 224, 224))
   x = x.to(device=device)

   latency = []
   for i in range(total_samples):
      t0 = time.time()
      resnet.eval()
      with torch.no_grad():
         out = resnet(x)
      latency.append(time.time() - t0)

   print('Number of runs:', len(latency))
   print("Average PyTorch {} Inference time = {} ms".format(device.type, format(sum(latency) * 1000 / len(latency), '.2f')))

def to_numpy(tensor):
   return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

def infer_onnxruntime():
   print('Onnxruntime Inference')
   print('==========================')
   print()

   onnx_model = onnx.load("resnet18.onnx")
   onnx.checker.check_model(onnx_model)

   # Input
   x = torch.randn((batch_size, 3, 224, 224))
   x = x.to(device=device)
   x = to_numpy(x)

   so = onnxruntime.SessionOptions()
   so.execution_mode = onnxruntime.ExecutionMode.ORT_SEQUENTIAL
   so.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL

   exproviders = ['CUDAExecutionProvider', 'CPUExecutionProvider']

   model_onnx_path = os.path.join(".", "resnet18.onnx")
   ort_session = onnxruntime.InferenceSession(model_onnx_path, so, providers=exproviders)

   options = ort_session.get_provider_options()
   cuda_options = options['CUDAExecutionProvider']
   cuda_options['cudnn_conv_use_max_workspace'] = '1'
   ort_session.set_providers(['CUDAExecutionProvider'], [cuda_options])

   #IOBinding
   input_names = ort_session.get_inputs()[0].name
   output_names = ort_session.get_outputs()[0].name
   io_binding = ort_session.io_binding()

   io_binding.bind_cpu_input(input_names, x)
   io_binding.bind_output(output_names, device)

   #warm up run
   ort_session.run_with_iobinding(io_binding)
   ort_outs = io_binding.copy_outputs_to_cpu()

   latency = []

   for i in range(total_samples):
      t0 = time.time()
      ort_session.run_with_iobinding(io_binding)
      latency.append(time.time() - t0)
      ort_outs = io_binding.copy_outputs_to_cpu()
   print('Number of runs:', len(latency))
   print("Average onnxruntime {} Inference time = {} ms".format(device.type, format(sum(latency) * 1000 / len(latency), '.2f')))

if __name__ == '__main__':
   torch.cuda.empty_cache()
   resnet = (models.resnet18(pretrained=True)).to(device=device)
   convert_to_onnx(resnet)
   infer_onnxruntime()
   infer_pytorch(resnet)

Output
If run on CPU,
Average onnxruntime cpu Inference time = 18.48 ms
Average PyTorch cpu Inference time = 51.74 ms

but, if run on GPU, I see
Average onnxruntime cuda Inference time = 47.89 ms
Average PyTorch cuda Inference time = 8.94 ms

If I change graph optimizations to onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL, I see some improvements in inference time on GPU, but its still slower than Pytorch.
I use io binding for the input tensor numpy array and the nodes of the model are on GPU.
Further, during the processing for onnxruntime, I print device usage stats and I see this -
Using device: cuda:0
GPU Device name: Quadro M2000M
Memory Usage:
Allocated: 0.1 GB
Cached:    0.1 GB

So, GPU device is being used.
Further, I have used the resnet18.onnx model from the ModelZoo to see if it is a converted mode issue, but i get the same results.
What am I doing wrong or missing here?

Output:
When calculating inference time exclude all code that should be run once like resnet.eval() from the loop.
Please include imports in example
import torch
from torchvision import models
import onnxruntime    # to inference ONNX models, we use the ONNX Runtime
import onnx
import os
import time

After running your example GPU only I found that time differs only ~x2, so the speed difference may be caused by framework characteristics. For more details explore onnx conversion optimization
Onnxruntime Inference
==========================

Number of runs: 1000
Average onnxruntime cuda Inference time = 4.76 ms
Pytorch Inference
==========================

Number of runs: 1000
Average PyTorch cuda Inference time = 2.27 ms


Instruction: Is there a way to convert a python dictionary with some values on GPU memory to have everything on main memory?

Input:
I have a Deep Learning (Using PyTorch) model whose output is given in dictionary format. The dictionary has multiple arrays as values and all these arrays are on GPU memory (torch.tensors with device = 'cuda'). Is there any way to move every array in the dictionary to main memory in one go?
My current way of going about this is to write a loop and re-write the GPU arrays into variables and use those, but that seems quite ineffecient. My essential goal is to further process the outputs in a multi-threaded manner, but since these arrays are on GPU memory mutliple threads cannot access them at once.
Would appreciate any sort of help or suggestions for this!

Output:
You can use a dictionary comprehension on the output dictionary out:
out = {k: v.to(device='cpu', non_blocking=True) for k, v in out.items()}

If out has some elements that are not tensors, you can use:
out = {k: v.to(device='cpu', non_blocking=True) if hasattr(v, 'to') else v for k, v in out.items()}


Instruction: Is it possible to save a file from test_step() function?

Input:
I am trying to implement MNIST digits using PyTorch Lightning.
The train function is like the below one
def train(epochs, train_loader, test_loader, model):
        early_stopping = EarlyStopping('train_loss', mode='min', patience=5)
        model_checkpoint = ModelCheckpoint(dirpath=model_path/'mnist_{epoch}-{train_loss:.2f}',monitor='train_loss', mode='min', save_top_k=3)
        trainer = pl.Trainer(max_epochs=epochs, profiler=False, callbacks = [model_checkpoint],default_root_dir=model_path)
        trainer.fit(model, train_dataloader=train_loader)
        trainer.test(test_dataloaders=test_loader, ckpt_path=None)

The test_step function is like the below one
def test_step(self, test_batch):
    x, y = test_batch
    logits = self.forward(x)
    loss = self.mean_squared_error_loss(logits.squeeze(-1), y.float())

    # I want to calculate R2, MAPE, etc and want to save in a pandas df and
    # need to return to the train function

    self.log('test_loss', loss)
    return {'test_loss': loss}

I can do calculate R2, MAPE, etc using TorchMetrics. But, I am not sure how (or is it possible) to save them in a pandas df (or maybe in a list) for the whole test dataset. I have gone through this post but not sure how should I try!
Any suggestions are appreciated.

Output:
You can aggregate test result in test_epoch_end:
def test_step(self, test_batch):
    x, y = test_batch
    logits = self.forward(x)
    loss = self.mean_squared_error_loss(logits.squeeze(-1), y.float())

    self.log('test_loss', loss)
    return {'test_loss': loss, "logits":logits, "labels": y}

def test_epoch_end(self, outputs):
    all_preds, all_labels = [], []
    for output in outputs:
        probs = list(output['logits'].cpu().detach().numpy()) # predicted values
        labels = list(output['labels'].flatten().cpu().detach().numpy())
        all_preds.extend(probs)
        all_labels.extend(labels)

    # you can calculate R2 here or save results as file
    r2 = ...

Note that this only works on a single GPU. If you are using multiple GPUs, you need some function to gather results from different GPUs.
To get model predictions, you need to add a predict_step() in the model class.
def predict_step(self, test_batch):
    x, y = test_batch
    logits = self.forward(x)
    return {'logits': logits, 'labels':y}

And run:
outputs = trainer.predict(model, test_loader, return_predictions=True)


Instruction: Is there any reason for using the word "column" in the context of one-dimensional tensor?

Input:
Consider the following statements from the chapter named Tensors: Multidimensional arrays from the textbook titled Deep Learning with PyTorch by Eli Stevens et al.

Let’s construct our first PyTorch tensor and see what it looks like.
It won’t be a particularly meaningful tensor for now, just three ones
in a column:
# In[4]:
import torch
a = torch.ones(3)
a

In general, the notion of the column is used in the case of at least two dimensions. The tensor initialized is of a single dimension. So, I am guessing that it is immaterial if we use either row or column.
Am I true? Else, Is there any reason behind using the word "column" in this context?

Output:
Often, in linear algebra theory, an n-dimensional vector is considered as a n x 1 matrix, called a column vector.
Indeed, the behavior of a tensor t with shape (n,) is very similar to that of a tensor u of shape (n, 1). In mathematical terms, you can think of a vector t in R^n and a vector u in R^{n x 1}.
In conclusion, the author, perhaps, is suggesting to treat the tensor as a mathematical column vector.


Instruction: Vector functions in pytorch to apply autograd to them

Input:
If I have a tensor x.
Can I define a vector function, say f(x) = (3x, x+2), and obtain its derivative df/dx?
In a nutshell: I want a way to define such a vector function, from which I can get its gradient.

Output:
You can do so using torch.autograd.functional.jacobian, providing the function and input:
>>> jacobian(lambda x: (3*x, x+2), inputs=torch.tensor([3.]))

In this case the result is df/dx = (3, 1) for all x.


Instruction: How to build a vector of marginal probabilities, given a tensor in PyTorch

Input:
How to build a vector of marginal probabilities, given a tensor in PyTorch
I have a tensor 'A' of shape [ Dim1: <128>, Dim2: <64>], each element in Dim1 is drawn from a unknown distribution and I need to check if the Dim2 vector has appeared before in the other 128 samples. If it has, the marginal probability of this element is increased by 1 and recorded in another tensor 'B' which is of shape [DimB: <128>]. When the iteration is complete I divide all elements in B by 128 (the number of possibilities) to achieve the weighted delta, therefore the aim is to approximate the true distribution as Dim1 increases in size.
How can this be achieved directly in PyTorch? I attempted it with ordered dictionaries but it's too slow. I'm assuming a method exists to do it straight in PyTorch
my crude way to do it with ordered dictionaries if we have a tensor T1 of shape [Dim1: <6>, Dim2: <3>]:
from collections import OrderedDict
od = OrderedDict()
T1 = torch.Tensor([[2.4, 5.5,1],
                   [3.44,5.43,1],
                   [2.4, 5.5,1],
                   [3.44,8.43,1],
                   [3.44,5.43,9],
                   [3.44,5.43,1], ])
print ('T1 shape',T1.shape) # -> T1 shape torch.Size([6, 3])
for i in range(T1.shape[0]):
    key = ''.join([ str(int(j)) for j in T1[i].tolist()]) # creates a unique identifier (is there a better way to do this?)
    if key in od:
        od[key] +=1
        key_place_holder = key + str(od[key]) # unique identifier if we found duplicate to keep a 0 in the final tensor
        od[key_place_holder] = 0
    else:
        od[key] = 1
print ('len od',len(od)) # -> len od 6
list_values = [j/len(od) for i,j in od.items()]
T1_marginal_probabilities = torch.Tensor(list_values)
print ('Marginal Probs',T1_marginal_probabilities) # -> Marginal Probs tensor([0.3333, 0.3333, 0.0000, 0.1667, 0.1667, 0.0000])

The final output is as expected, as the probability of [2.4, 5.5,1] and [3.44,5.43,1] are both 2/6, since we have [2.4, 5.5,1] repeated 2 times at position 0 and 2. while [3.44,5.43,1] is repeated in position 1 and 5.

Output:
You can use torch.unique and torch.nonzero:
T1 = ...
values, inverse, counts = T1.unique(dim=0, return_inverse=True, return_counts=True)

ps = torch.zeros(inverse.numel())
for i, (v, c) in enumerate(zip(values, counts)):
    first_occurence = torch.nonzero(inverse == i)[0].item()
    ps[first_occurence] = c
ps /= ps.sum()


Instruction: Output of vgg16 layer doesn't make sense

Input:
I have a vgg16 network without the last max pooling, fully connected and softmax layers. The network summary says that the last layer's output is going to have a size of (batchsize, 512, 14, 14). Putting an image into the network gives me an output of (batchsize, 512, 15, 15). How do I fix this?
import torch
import torch.nn as nn
from torchsummary import summary

vgg16 = torch.hub.load('pytorch/vision:v0.10.0', 'vgg16', pretrained=True)
vgg16withoutLastFewLayers = nn.Sequential(*list(vgg16.children())[:-2][0][0:30]).cuda()

image = torch.zeros((1,3,244,244)).cuda()
output = vgg16withoutLastFewLayers(image)

summary(vgg16withoutLastFewLayers, (3,224,224))
print(output.shape)

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 224, 224]           1,792
              ReLU-2         [-1, 64, 224, 224]               0
            Conv2d-3         [-1, 64, 224, 224]          36,928
              ReLU-4         [-1, 64, 224, 224]               0
         MaxPool2d-5         [-1, 64, 112, 112]               0
            Conv2d-6        [-1, 128, 112, 112]          73,856
              ReLU-7        [-1, 128, 112, 112]               0
            Conv2d-8        [-1, 128, 112, 112]         147,584
              ReLU-9        [-1, 128, 112, 112]               0
        MaxPool2d-10          [-1, 128, 56, 56]               0
           Conv2d-11          [-1, 256, 56, 56]         295,168
             ReLU-12          [-1, 256, 56, 56]               0
           Conv2d-13          [-1, 256, 56, 56]         590,080
             ReLU-14          [-1, 256, 56, 56]               0
           Conv2d-15          [-1, 256, 56, 56]         590,080
             ReLU-16          [-1, 256, 56, 56]               0
        MaxPool2d-17          [-1, 256, 28, 28]               0
           Conv2d-18          [-1, 512, 28, 28]       1,180,160
             ReLU-19          [-1, 512, 28, 28]               0
           Conv2d-20          [-1, 512, 28, 28]       2,359,808
             ReLU-21          [-1, 512, 28, 28]               0
           Conv2d-22          [-1, 512, 28, 28]       2,359,808
             ReLU-23          [-1, 512, 28, 28]               0
        MaxPool2d-24          [-1, 512, 14, 14]               0
           Conv2d-25          [-1, 512, 14, 14]       2,359,808
             ReLU-26          [-1, 512, 14, 14]               0
           Conv2d-27          [-1, 512, 14, 14]       2,359,808
             ReLU-28          [-1, 512, 14, 14]               0
           Conv2d-29          [-1, 512, 14, 14]       2,359,808
             ReLU-30          [-1, 512, 14, 14]               0
================================================================
torch.Size([1, 512, 15, 15])

Output:
The output shape should be [512, 14, 14], assuming that the input image is [3, 224, 224]. Your input image size is [3, 244, 244]. For example,
image = torch.zeros((1,3,224,224))
# torch.Size([1, 512, 14, 14])
output = vgg16withoutLastFewLayers(image)

Therefore, by increasing the image size, the spatial size [W, H] of your output tensor also increases.


Instruction: Cast C++ PyTorch Tensor to Python PyTorch Tensor

Input:
For a project that I am working on, I need to call from C++ a Python function, which has as input a PyTorch Tensor. While searching for a way to achieve this, I found that using a function named THPVariable_Wrap (Information I have found link 1 and link 2) could transform a C++ Pytorch Tensor to a PyObject, which can be used as input for the call to the Python function. However, I have tried importing this function by including the header file directly in my code, but this will always return the error LNK2019, when calling the function, with the following description:

Severity  Code    Description Project File    Line    Suppression State
Error   LNK2019 unresolved external symbol "__declspec(dllimport) struct _object * __cdecl THPVariable_Wrap(class at::TensorBase)" (_imp?THPVariable_Wrap@@YAPEAU_object@@VTensorBase@at@@@Z) referenced in function main pythonCppTorchExp   C:\Users\MyName\source\repos\pythonCppTorchExp\pythonCppTorchExp\example-app.obj    1

I believe the problem is in how I import the THPVariable_Wrap function in my C++ file. However, I am still not that skilled with C++ and the information on this is limited. Besides Pytorch, I am also using Boost for calling Python and I am using Microsoft Visual Studio 2019 (v142), with C++ 14. I posted the code I used below.
C++ File
#include <iostream>
#include <iterator>
#include <algorithm>
#include <boost/python.hpp>
#include <Python.h>
#include <string.h>
#include <fstream>
#include <boost/filesystem.hpp>
#include <torch/torch.h>

#include <torch/csrc/autograd/python_variable.h> /* The header file where  */

namespace python = boost::python;
namespace fs = boost::filesystem;

using namespace std;

int main() {
    string module_path = "Path/to/python/folder";

    Py_Initialize();

    torch::Tensor cppTensor = torch::ones({ 100 });
    PyRun_SimpleString(("import sys\nsys.path.append(\"" + module_path + "\")").c_str());

    python::object module = python::import("tensor_test_file");
    python::object python_function = module.attr("tensor_equal");

    PyObject* castedTensor = THPVariable_Wrap(cppTensor) /* This function call creates the error.*/;

    python::handle<> boostHandle(castedTensor);
    python::object inputTensor(boostHandle);
    python::object result = python_function(inputTensor);

    bool succes = python::extract<bool>(result);
    if (succes) {
        cout << "The tensors match" << endl;
    }
    else {
        cout << "The tensors do not match" << endl;
    }
}

Python File
import torch

def tensor_equal(cppTensor):
    pyTensor = torch.ones(100)
    areEqual = cppTensor.equal(pyTensor)
    return areEqual

Output: This is linker problem. You probably have to link libtorch.python.so. It can be located in place like /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so. Or where you have your libtorch installed.


Instruction: How do I fix a Pytorch install error on a windows virtual environment with an error that says Pytorch could not be found from a pip command?

Input:
Thank you for taking the time to look at this thread.  I am running windows 11 and created a virtual environment that was setup with Python 3.10.2.  I installed jupyter notebook, tensorflow, CUDA 11.6 toolkit, and cuDNN 8.3.2.  I went to the PyTorch website and clicked the long term stable version of PyTorch for windows using Pip on the CUDA 11.1 option.
This produced the following Pip command:
pip3 install torch==1.8.2+cu111 torchvision==0.9.2+cu111 torchaudio===0.8.2 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html

I ran this command inside of my terminal as well as the jupyter notebook whose kernel was connected to my virtual environment and received the following error:
ERROR: Could not find a version that satisfies the requirement torch==1.8.2+cu111 (from versions: none)
ERROR: No matching distribution found for torch==1.8.2+cu111

Does anybody have any advice that could help me solve this issue?
I am running a machine with a GTX 2060 and an AMD Ryzen processor.  To my knowledge, I should be able to run CUDA with my GPU.  I attempted to troubleshoot the issue by using the stable version of PyTorch as well but still received the same error.
Thank you.

Output: OK, well now I feel silly.  I went back to the PyTorch website and saw that PyTorch only works up to Python 3.9 as of today in case anyone else runs into a similar issue.


Instruction: import torch.fx ModuleNotFoundError: No module named 'torch.fx'

Input:
After enabling torch and Cuda for my system according to my system GPU compatibility, whenever I am trying to run any program which needs to be run on GPU to enable the system, this error is coming. I could not able to find any solution for this. though I read about this that create another environment and this error will be solved but did not work for me. Please find the details of my system. for reference, I am using Python 3.7.2.
image include torch and Cuda version and device name
Details of NVIDIA Cuda compiler driver
nvidia-smi output
Can anyone help to solve this problem? Thank you

Output: torch.fx was added in PyTorch 1.8.0. Check release post. You're probably using an older version. Upgrade pytorch from website.


Instruction: PyTorch Inference High CPU Usage on Kubernetes

Input:
Problem
We are trying to create an inference API that load PyTorch ResNet-101 model on AWS EKS. Apparently, it always killed OOM due to high CPU and Memory usage. Our log shows we need around 900m CPU resources limit. Note that we only tested it using one 1.8Mb image. Our DevOps team didn't really like it.
What we have tried
Currently we are using standard PyTorch load model module. We also clean model state dict to clean up the memory usage.
Is there any method to reduce the CPU usage to load PyTorch model?

Output:
Have you tried limiting the CPU available to the pods?
  - name: pytorch-ml-model
    image: pytorch-cpu-hog-model-haha
    resources:
      limits:
        memory: "128Mi"
        cpu: "1000m" # Replace this with CPU amount your devops guys will be happy about

If your error is OOM, you might want to consider the adding more memory allocated per pod? We as outsiders have no idea how large of memory you would require to execute your models, I would suggest using debugging tools like PyTorch profiler to understand how much memory you need for your inferencing use-case.
You might also want to consider, using memory-optimized worker nodes and applying deployment-node affinity through labels to ensure that inferencing pods are allocated in memory-optimized nodes in your EKS clusters.


Instruction: Model name 'bert-base-uncased' was not found in tokenizers

Input:
My code that loads a pre-trained BERT model has been working alright until today I moved it to another, new server. I set up the environment properly, then when loading the 'bert-base-uncased' model, I got this error
Traceback (most recent call last):
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/.conda/envs/tensorflow-gpu/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/.conda/envs/tensorflow-gpu/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/wop_bert/code/python/src/exp/run_exp_bert_apply.py", line 74, in <module>
    input_text_fields)
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/wop_bert/code/python/src/classifier/classifier_bert_.py", line 556, in fit_bert_trainonly
    tokenizer = BertTokenizer.from_pretrained(bert_model, do_lower_case=True)
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/.conda/envs/tensorflow-gpu/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1140, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/jmain02/home/J2AD003/txk64/zzz70-txk64/.conda/envs/tensorflow-gpu/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1246, in _from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name 'bert-base-uncased' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). We assumed 'bert-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

And the line that triggered this error (classifier_bert_.py line 556) is very simple:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

Please can I have some help on how to solve this issue?
Thanks

Output:
You have to download it and put in the same directory:
You can download it from here: https://huggingface.co/bert-base-uncased


Instruction: How to get accuracy during/after training for Huggingface RobertaForMaskedLM model?

Input:
I am using HuggingFace Trainer to train a Roberta Masked LM. I am passing the following function for compute_metrics as other discussion threads suggest:
metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

I am loading my dataset from text files using load_dataset, and after applying a tokenizer, it returns attention_id, input_ids but no labels. I am training on this dataset using Trainer. Still, I cannot find the accuracy of my model during training after passing the compute_metrics function above or cannot evaluate my model after training on test data.
How to get accuracy for this model during training and evaluate it as we could do in Keras models using model.evaluate()? How is it measured?

Output: If you are using one of the example scripts, have you checked the --evaluation_strategy parameter? The default value is None, but it can be set to steps or epoch.


Instruction: Pytorch/Tensorflow: Compute gradient of Mixture of Gaussians log density

Input:
I have a mixture of three Gaussians and would like to compute the gradient of the log-density using Pytorch or Tensorflow. How can I do that?
from numpy import eye, log
from scipy.stats import multivariate_normal as MVN

μs   = [[0, 0], [2, 0], [0, 2]]              # Means
Σs   = [eye(2), eye(2), eye(2)]              # Covariance Matrices
cs   = [1 / 3] * 3                           # Mixture coefficients
MVNs = [MVN(μ, Σ) for (μ, Σ) in zip(μs, Σs)] # List of Gaussians

log_density = lambda x: log((sum([c * MVN.pdf(x) for (c, MVN) in zip(cs, MVNs)])))

Essentially I would like to compute the gradient of log_density. I tried using autograd.grad but it fails because of the array assignment.
Attempted Pytorch Solution
from torch import tensor, eye, sqrt, zeros, log, exp
from torch.distributions import MultivariateNormal as MVN

μs   = [tensor([0, 0]), tensor([2, 0]), tensor([0, 2])] # Means
Σs   = [eye(2), eye(2), eye(2)]                         # Covariance Matrices
cs   = [1 / 3] * 3                                      # Mixture coefficients
MVNs = [MVN(μ, Σ) for (μ, Σ) in zip(μs, Σs)]            # List of Gaussians

log_density = lambda x: log((sum([c * exp(MVN.log_prob(x)) for (c, MVN) in zip(cs, MVNs)])))

Attempted Autograd Solution (won't work)
from numpy import eye, log, zeros
from scipy.stats import multivariate_normal as MVN
from autograd import grad

μs   = [[0, 0], [2, 0], [0, 2]]              # Means
Σs   = [eye(2), eye(2), eye(2)]              # Covariance Matrices
cs   = [1 / 3] * 3                           # Mixture coefficients
MVNs = [MVN(μ, Σ) for (μ, Σ) in zip(μs, Σs)] # List of Gaussians

log_density = lambda x: log((sum([c * MVN.pdf(x) for (c, MVN) in zip(cs, MVNs)])))
gradient = grad(log_density)

# If you try using this gradient function you get an error
gradient(zeros(2))

The error I get is

ValueError: setting an array element with a sequence.

Naive Autograd Solution
There is, of course, a bad Autograd solution that won't scale well. For instance
from autograd.numpy import log, eye, zeros, array
from autograd.scipy.stats import multivariate_normal as MVN
from autograd import grad

μs   = [[0, 0], [2, 0], [0, 2]]              # Means
Σs   = [eye(2), eye(2), eye(2)]              # Covariance Matrices
cs   = [1 / 3] * 3                           # Mixture coefficients

def log_density(x):
    return log((1/3) * MVN.pdf(x, zeros(2), eye(2)) + (1/3) * MVN.pdf(x, array([2, 0]), eye(2)) + (1/3) * MVN.pdf(x, array([0, 2]), eye(2)))
grad(log_density)(zeros(2))  # Works!

Output:
You can do
from torch import tensor, eye, sqrt, zeros, log, exp
from torch.distributions import MultivariateNormal as MVN

μs   = [tensor([0, 0]), tensor([2, 0]), tensor([0, 2])] # Means
Σs   = [eye(2), eye(2), eye(2)]                         # Covariance Matrices
cs   = [1 / 3] * 3                                      # Mixture coefficients
MVNs = [MVN(μ, Σ) for (μ, Σ) in zip(μs, Σs)]            # List of Gaussians

x = tensor((0.0,0.0), requires_grad=True)
log_density = log((sum([c * exp(MVN.log_prob(x)) for (c, MVN) in zip(cs, MVNs)])))

log_density.backward()
print(x.grad)

which will print the gradient at (0.0,0.0). However as pytorch is not generating a static computation graph, I could not find an easy way to calculate the gradient at another point without rebuilding the computation graph. You could try to use tensorflow, which gives you more control on the computation graphs and allows you to construct a graph for the gradient computation.
Edit With tensorflow you could do something like
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import tensorflow_probability as tfp

@tf.function
def mygrad(x):
    print("building graph")
    us   = tf.stack([tf.constant([0.0, 0.0]), tf.constant([2., 0.]), tf.constant([0., 2.])])
    covs = tf.stack([tf.eye(2), tf.eye(2), tf.eye(2)])
    cs   = tf.constant([1 / 3] * 3)

    with tf.GradientTape() as gt:
        gt.watch(x)
        log_density = tf.math.log(tf.math.reduce_sum(tfp.distributions.MultivariateNormalTriL(us,covs).prob(x) * cs) )

    return gt.gradient(log_density,x)

print(mygrad(tf.constant([0.0,0.0])).numpy()) #gradient at 0.0,0.0
print(mygrad(tf.constant([1.0,0.0])).numpy()) #gradient at 1.0,0.0

Essentially you do automatic differentiation with the tf.GradientTape and capture the computation graph in a tf.function. There is more background information on the very extensive Tensorflow API documentation.


Instruction: Error downloading celebA dataset using torchvision

Input:
Using the torchvision module datasets, I can't download the celebA image dataset.
I am pretty sure that I am doing everything right.
dataset = datasets.CelebA(
    root='../datasets/celebA/train_images',
    split='train',
    target_type='identity',
    transform=transforms.Compose([transforms.ToTensor()]),
    download=True)

The error:
BadZipFile: File is not a zip file

Output:
It is a known issue that has been already reported in #1920, and it seems it was fixed in #4109 but the commit is not yet included in a stable release.
In the meanwhile you can do the following:

Look at the source code of datasets.CelebA, search for file_list and take note of the files in it.
Download those files from here, and copy the files in a new directory called celeba
Unzip img_align_celeba.zip in the same directory

Finally call datasets.CelebA() with download=False.


Instruction: TypeError: new(): data must be a sequence (got numpy.float64)

Input:
I  do not know what to do with this problem.  I am running a model training.  The following part is what I got
 mean_train = torch.Tensor(np.mean(train_vertices, axis=0))
 TypeError: new(): data must be a sequence (got numpy.float64)

My code is:
mean_train = torch.Tensor(np.mean(train_vertices, axis=0))
std_train = torch.Tensor(np.std(train_vertices, axis=0))

Output:
You have a numpy array and you want to create a pytorch tensor from it. You can use torch.from_numpy to achieve this. Note that torch.from_numpy expects an np.ndarray not a np.float64 so you'll need to figure out your shapes.
However, if you don't need numpy, you can just use pytorch from the jump. Pytorch will likely have the functions you needed from numpy anyway.