Instruction: RuntimeError: `lengths` array must be sorted in decreasing order when `enforce_sorted` is True. - Pytorch

Input:
 It have been 5 hours sitting here getting the same error:
RuntimeError: `lengths` array must be sorted in decreasing order when `enforce_sorted` is True. You can pass `enforce_sorted=False` to pack_padded_sequence and/or pack_sequence to sidestep this requirement if you do not need ONNX exportability.

I'm working on this simple sentiment classification task using RNN in pytorch. I'm loading the my custom data using torchtext. I'm loading it from a json file which looks as follows:
{"reviewText": "Da Silva takes the divine by ....", "overall": 4.0, "summary": "An amazing first novel"}

I created my field as follows. And i created a pre-processing get_sentiment() function that convert overalls that are greater than 2 to 1 0 otherwise:
get_sentiment = lambda x: 1 if x >=3 else 0

TEXT = data.Field(tokenize = 'spacy',
                  tokenizer_language = 'en_core_web_sm',
                  include_lengths=True
                  )
LABEL = data.Field(sequential=False, use_vocab=False, preprocessing=get_sentiment)

fields = {
    'reviewText': ('review', TEXT),
    'overall': ('sentiment', LABEL)
}

I loaded the data:
train_data, test_data = data.TabularDataset.splits(
    path="/content/",
    train="Books_small_10000.json",
    test="Books_small.json",
    format="json",
    fields=fields
)

I built the vocabularies:
MAX_VOCAB_SIZE = 25_000

TEXT.build_vocab(
    train_data,
    max_size = MAX_VOCAB_SIZE,
    vectors = "glove.6B.100d",
    unk_init = torch.Tensor.normal_
)

LABEL.build_vocab(train_data)

I created my iterators.
BATCH_SIZE = 64

train_iterator, validation_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, validation_data, test_data),
    device = device,
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.review),
)

This is how my Model looks.
class AmazonLSTMRNN(nn.Module):
  def __init__(self, vocab_size, embedding_size, hidden_size, output_size, num_layers
               , bidirectional, dropout, pad_idx):
    super(AmazonLSTMRNN, self).__init__()

    self.embedding = nn.Embedding(vocab_size, embedding_dim=embedding_size, padding_idx=pad_idx)
    self.lstm = nn.LSTM(embedding_size, hidden_size=hidden_size,
                        bidirectional=bidirectional, num_layers=num_layers,
                        dropout=dropout)
    self.fc = nn.Linear(hidden_size * 2, out_features=output_size)
    self.dropout = nn.Dropout(dropout)

  def forward(self, text, text_lengths):
    embedded = self.dropout(self.embedding(text))
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'))
    packed_output, (h_0, c_0) = self.rnn(packed_embedded)
    output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)
    h_0 = self.dropout(torch.cat((h_0[-2,:,:], h_0[-1,:,:]), dim = 1))
    return self.fc(h_0)

INPUT_DIM = len(TEXT.vocab) # # 25002
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token] # 0
amazon_model = AmazonLSTMRNN(INPUT_DIM,
            EMBEDDING_DIM,
            HIDDEN_DIM,
            OUTPUT_DIM,
            N_LAYERS,
            BIDIRECTIONAL,
            DROPOUT,
            PAD_IDX)

criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(amazon_model.parameters())
amazon_model = amazon_model.to(device)
criterion = criterion.to(device)

.....
Training function
def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    for batch in iterator:
        optimizer.zero_grad()
        text, text_lengths = batch.review
        predictions = model(text, text_lengths).squeeze(1)
        loss = criterion(predictions, batch.sentiment)
        acc = accuracy(predictions, batch.sentiment)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Training loop.
N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(amazon_model, train_iterator, optimizer, criterion)
    end_time = time.time()
    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(amazon_model.state_dict(), 'best-model.pt')
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')

If someone knows where am i wrong please correct me. Any help input will be appreciated.

Output:
After some few minutes I found the solution and I was able to get accuracy of aprox ~93% on a single training epoch.
I changed my LABEL field to:
LABEL = data.LabelField(preprocessing=get_sentiment, dtype = torch.float)

Then i changed my AmazonLSTMRNN model in the forward method by adding enforce_sorted=False to the pack_padded_sequence.
The forward method:
 def forward(self, text, text_lengths):
    embedded = self.dropout(self.embedding(text))
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths.to('cpu'), enforce_sorted=False)
    packed_output, (h_0, c_0) = self.lstm(packed_embedded)
    output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)
    h_0 = self.dropout(torch.cat((h_0[-2,:,:], h_0[-1,:,:]), dim = 1))
    return self.fc(h_0)


Instruction: Albumentations in Pytorch: Inconsistent Augmentation for multi-target datasets

Input:
I'm using Pytorch and want to perform the data augmentation of my images with Albumentations. My dataset object has two different targets: 'blurry' and 'sharp'. Each instance of both targets needs to have identical changes. When I try to perform the data augmentation with a Dataset object like this:
class ApplyTransform(Dataset):
def __init__(self, dataset, transformation):
    self.dataset = dataset
    self.aug = transformation

def __len__(self):
    return (len(self.dataset))

def __getitem__(self, idx):
    sample, target = self.dataset[idx]['blurry'], self.dataset[idx]['sharp']
    transformedImgs = self.aug(image=sample, target_image=target)
    sample_aug, target_aug = transformedImgs["image"], transformedImgs["target_image"]
    return {'blurry': sample_aug, 'sharp': target_aug}

Unfortunately, I receive two images with two different augmentations:

When I try the same without a Dataset object, I receive two images with the identical application of augmentations. Does anybody know how to make it work with a dataset object?
Here is my augmentation pipeline:
augmentation_transform = A.Compose(
        [
        A.Resize(1024,1024, p=1),
        A.HorizontalFlip(p=0.25),
        A.Rotate(limit=(-45, 65)),
        A.VerticalFlip(p=0.24),
        A.RandomContrast(limit=0.3, p=0.15),
        A.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
        A.pytorch.transforms.ToTensorV2(always_apply=True, p=1.0)
        ],
        additional_targets={"target_image": "image"}
        )

Output: You can stack your blurry and sharp images, apply your augmentation then unstack them


Instruction: Tensorboard in pytorch does not load anything in Browser

Input:
I am using tensorboard to monitor the training progress of the model from this codebase. To open tensorboard, I ran the command tensorboard --logdir=checkpoints/ as suggested in the codebase. I know that to open tensorboard, I need to pass the directory path in --logdir where the events file is present, which I did. It does seem to open tensorboard since it returns the following in the terminal.
I0620 12:52:16.737502 140647693104896 plugin.py:292] Monitor runs begin
TensorBoard 2.5.0 at http://localhost:8088/ (Press CTRL+C to quit)

But when I open the link in browser, this tensorboard loading screen occurs and it loads forever, and doesnt open any stats/plots that I want to visualize.
My tensorboard version is 2.5.0 and pytorch version is 1.8.1+cu102

Output: This issue got resolved once I uninstalled torch_tb_profiler and downgraded Tensorboard 2.5.0 to 1.15.0 as suggested in this answer


Instruction: "AssertionError: Cannot handle batch sizes > 1 if no padding token is > defined" and pad_token = eos_token

Input:
I am trying to finetune a pre-trained GPT2-model. When applying the respective tokenizer, I originally got the error message:

Using pad_token, but it is not set yet.

Thus, I changed my code to:
GPT2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
GPT2_tokenizer.pad_token = GPT2_tokenizer.eos_token

When calling the trainer.train() later, I end up with the following error:

AssertionError: Cannot handle batch sizes > 1 if no padding token is
defined.

Since I specifically defined the pad_token above, I expect these errors (or rather my fix of the original error and this new error) to be related - although I could be wrong. Is this a known problem that eos_token and pad_token somehow interfer? Is there an easy work-around?
Thanks a lot!

Output:
I've been running into a similar problem, producing the same error message you were receiving. I can't be sure if your problem and my problem were caused by the same issue, since I can't see your full stack trace, but I'll post my solution in case it can help you or someone else who comes along.
You were totally correct to fix the first issue you described with your tokenizer by setting its pad token with the code provided. However, I also had to set the pad_token_id of my model's configuration to get my GPT2 model to function properly. I did this in the following way:
# instantiate the configuration for your model, this can be imported from transformers
configuration = GPT2Config()
# set up your tokenizer, just like you described, and set the pad token
GPT2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
GPT2_tokenizer.pad_token = GPT2_tokenizer.eos_token
# instantiate the model
model = GPT2ForSequenceClassification(configuration).from_pretrained(model_name).to(device)
# set the pad token of the model's configuration
model.config.pad_token_id = model.config.eos_token_id

I suppose this is because the tokenizer and the model function separately, and both need knowledge of the ID being used for the pad token. I can't tell if this will fix your problem (since this post is 6 months old, it may not matter anyway), but hopefully my answer may be able to help someone else.


Instruction: How to get the size of a Hugging Face pretrained model?

Input:
I keep getting a CUDA out of memory error when trying to fine-tune a Hugging Face pretrained XML Roberta model. So, the first thing I want to find out is the size of the pretrained model.
model = XLMRobertaForCausalLM.from_pretrained('xlm-roberta-base', config=config)
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

model.to(device)

I have tried to get the size of the model with
sys.getsizeof(model)

and, unsurprisingly, I get an incorrect result.  I get 56 as a result, which is the size of the python object.
But then, I tried model. element_size(), and I get the error
ModuleAttributeError: 'XLMRobertaForCausalLM' object has no attribute 'element_size'

I have searched in the Hugging Face documentation, but I have not found how to do it. Does anyone here know how to do it?

Output:
If you facing CUDA out of memory errors, the problem is mostly not the model, rather than the training data. You can reduce the batch_size (number of training examples used in parallel), so your gpu only need to handle a few examples each iteration and not a ton of.
However, to your question:
I would recommend you objsize. It is a library that calculates the "real" size (also known as "deep" size). So a straightforward solution would be:
import objsize
objsize.get_deep_size(model)

However, the documentation says:

Excluding non-exclusive objects. That is, objects that are also referenced from somewhere else in the program. This is true for calculating the object's deep size and for traversing its descendants.

This shouldn't be a problem, but if it still gets a too small size for your model you can use Pympler, another Library that calculates the "deep" size via recursion.
Another approach would be implementing a get_deep_size() function by yourself, e.g. from this article:
import sys

def get_size(obj, seen=None):
    """Recursively finds size of objects"""
    size = sys.getsizeof(obj)
    if seen is None:
        seen = set()
    obj_id = id(obj)
    if obj_id in seen:
        return 0
    # Important mark as seen *before* entering recursion to gracefully handle
    # self-referential objects
    seen.add(obj_id)
    if isinstance(obj, dict):
        size += sum([get_size(v, seen) for v in obj.values()])
        size += sum([get_size(k, seen) for k in obj.keys()])
    elif hasattr(obj, '__dict__'):
        size += get_size(obj.__dict__, seen)
    elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
        size += sum([get_size(i, seen) for i in obj])
    return size


Instruction: Pytorch slowing down after few iterations

Input:
I am trying to implement a model in PyTorch. The training procedure is quite complex and take a while, but what I have noticed is that the model is very fast on the first few batches, and then suddenly gets about 500. I guess it is due to some memory leak issue, as if python was not really letting free the memory of released huge tensors.
At first I thought that the problem was linked to the storing gradient, but actually even with torch.no_grad() the same issue appears.
Here is an example to replicate the problem. (Note I am not trying to train this specific network, but the problem looks the same). To make things simpler I am not using the gradient and I am iterating on the same batch.
import torch
import torch.nn as nn
from torchvision.datasets import MNIST
import torchvision.transforms as T

dataset = MNIST(root='./MNIST', train=True, download=True,
                transform=T.Compose([T.ToTensor(), T.Lambda(lambda x: torch.flatten(x))]))
data_loader = torch.utils.data.DataLoader(dataset, batch_size=500)

X, _ = next(iter(data_loader))
X = X.to('cuda')

in_features = 28*28
out_features = 10
width= 15000

#defining huge network
NN = nn.Sequential(
          nn.Linear(in_features=28*28, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=width, bias=False),
          nn.ReLU(),
          nn.Linear(in_features=width, out_features=out_features, bias=False),
        ).to('cuda')

import time

iterations=100

X = X.to('cuda')

with torch.no_grad():
  for idx in range(iterations):
    print(f'Iteration {idx+1}')
    start = time.time()
    Y = NN(X)
    print(f'Time: {time.time() - start}')

The output shows that everything is very fast up to almost the 50th iteration, then it suddenly slows down.
Iteration 44
Time: 0.00035953521728515625
Iteration 45
Time: 0.00035309791564941406
Iteration 46
Time: 0.00035309791564941406
Iteration 47
Time: 0.048192501068115234
Iteration 48
Time: 0.1714644432067871
Iteration 49
Time: 0.16771984100341797
Iteration 50
Time: 0.1681973934173584
Iteration 51
Time: 0.16853046417236328
Iteration 52
Time: 0.16821908950805664

Why is there such a slow down? Is it possible to avoid it somehow?

Output:
Check out this page and scroll down to "Asynchronous execution".
Basically, you are measuring the time to enqueue your operation into the GPU not the time it actually takes to execute your operations. This is because GPU calls are asynchronous as described in the link. I copied the relevant part below:

By default, GPU operations are asynchronous. When you call a function that uses the GPU, the operations are enqueued to the particular device, but not necessarily executed until later. This allows us to execute more computations in parallel, including operations on CPU or other GPUs.
In general, the effect of asynchronous computation is invisible to the caller, because (1) each device executes operations in the order they are queued, and (2) PyTorch automatically performs necessary synchronization when copying data between CPU and GPU or between two GPUs. Hence, computation will proceed as if every operation was executed synchronously.
You can force synchronous computation by setting environment variable CUDA_LAUNCH_BLOCKING=1. This can be handy when an error occurs on the GPU. (With asynchronous execution, such an error isn’t reported until after the operation is actually executed, so the stack trace does not show where it was requested.)
A consequence of the asynchronous computation is that time measurements without synchronizations are not accurate. To get precise measurements, one should either call torch.cuda.synchronize() before measuring, or use torch.cuda.Event to record times as following:
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)
start_event.record()

# Run some things here

end_event.record()
torch.cuda.synchronize()  # Wait for the events to be recorded!
elapsed_time_ms = start_event.elapsed_time(end_event)


Instruction: How to move multiple tensors to the Cuda device concurrently?

Input:
policy_data, value_data, action_mask = policy_data.cuda(non_blocking=True), value_data.cuda(non_blocking=True), action_mask.cuda(non_blocking=True)

rewards, regret_probs = rewards.cuda(non_blocking=True), regret_probs.cuda(non_blocking=True)

return action_probs.cpu(), sample_probs.cpu(), sample_indices.cpu(), update

I am doing some RL work and am wondering whether it would be possible to speed up fragments like the above by launching the data transfer to the GPU on different streams before waiting on them together. Does PyTorch have any functions that would make this easier? I'd rather ask here before I dive into the minutiae of optimizing data transfers.

Output: Seems like one potential solution would be to pack all of the data into a single tensor (though of course you'd likely pay a small cost due to unused elements within this compacted representation.) An alternative would be to store this compact tensor as a sparse tensor (no additional data, but slightly more memory consumption per value). You'd have to test between these two to determine which was more efficient for your use case.


Instruction: PyTorch CUDA error: an illegal memory access was encountered

Input:
Relatively new to using CUDA. I keep getting the following error after a seemingly random period of time:
RuntimeError: CUDA error: an illegal memory access was encountered
I have seen people suggest things such as using cuda.set_device() rather than cuda.device(), setting torch.backends.cudnn.benchmark = False
but I can't seem to get the error to go away. Here are some pieces of my code:
torch.cuda.set_device(torch.device('cuda:0'))
torch.backends.cudnn.benchmark = False
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True, dropout=0.2)

        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        out = self.fc(out[:, -1, :])

        return out

    def pred(self, x):
        return self(x) > 0

def train(model, loss_fn, optimizer, num_epochs, x_train, y_train, x_val, y_val, loss_stop=60):
    cur_best_loss = 999
    loss_recur_count = 0
    best_model = None
    for t in range(num_epochs):
        model.train()

        y_train_pred = model(x_train)

        train_loss = loss_fn(y_train_pred, y_train)

        tr_l = train_loss.item()

        optimizer.zero_grad()

        train_loss.backward()
        optimizer.step()

        model.eval()
        with torch.no_grad():
            y_val_pred = model(x_val)

            val_loss = loss_fn(y_val_pred, y_val)

            va_l = val_loss.item()

            if va_l < cur_best_loss:
                cur_best_loss = va_l
                best_model = model
                loss_recur_count = 0
            else:
                loss_recur_count += 1

        if loss_recur_count == loss_stop:
            break
    if best_model is None:
        print("model is None.")
    return best_model

def lstm_test(cols, df, test_percent, test_bal, initial_shares_test, max_price, last_sell_day):
    wdw = 20
    x_train, y_train, x_test, y_test, x_val, y_val = load_data(df, wdw, test_percent, cols)

    x_train = torch.from_numpy(x_train).type(torch.Tensor).cuda()
    x_test = torch.from_numpy(x_test).type(torch.Tensor).cuda()
    x_val = torch.from_numpy(x_val).type(torch.Tensor).cuda()
    y_train = torch.from_numpy(y_train).type(torch.Tensor).cuda()
    y_test = torch.from_numpy(y_test).type(torch.Tensor).cuda()
    y_val = torch.from_numpy(y_val).type(torch.Tensor).cuda()

    input_dim = x_train.shape[-1]
    hidden_dim = 32
    num_layers = 2
    output_dim = 1
    y_preds_dict = {}
    for i in range(11):
        model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers).cuda()

        r = (y_train.cpu().shape[0] - np.count_nonzero(y_train.cpu()))/np.count_nonzero(y_train.cpu())/2
        pos_w = torch.tensor([r]).cuda()

        loss_fn = torch.nn.BCEWithLogitsLoss(pos_weight=pos_w).cuda()

        optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)

        best_model = train(model, loss_fn, optimizer, 300, x_train, y_train, x_val, y_val)

        y_test_pred = get_predictions(best_model, x_test)
        y_preds_dict[i] = y_test_pred.cpu().detach().numpy().flatten()

and here is the error msg:
<ipython-input-5-c52edc2c0508> in train(model, loss_fn, optimizer, num_epochs, x_train, y_train, x_val, y_val, loss_stop)
     19         model.eval()
     20         with torch.no_grad():
---> 21             y_val_pred = model(x_val)
     22
     23             val_loss = loss_fn(y_val_pred, y_val)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-4-9da8c811c037> in forward(self, x)
     10
     11     def forward(self, x):
---> 12         h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
     13         c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
     14

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Output:
It was partially said by the answer of the OP, but the problem under the hood with illegal memory access is that the GPU runs out of memory.
In my case, when I run a script on Windows I get the error message:
RuntimeError: CUDA out of memory. Tried to allocate 1.64 GiB (GPU 0; 4.00 GiB total capacity; 1.10 GiB already allocated; 1.27 GiB free; 1.12 GiB reserved in total by PyTorch)

but when run on Linux I get:
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Perhaps the message in Windows is more understandable :)
References:
https://forums.fast.ai/t/runtimeerror-cuda-error-an-illegal-memory-access-was-encountered/93899


Instruction: Get file names and file path using PyTorch dataloader

Input:
I am using PyTorch 1.8 and Python 3.8 to read images from a folder using the following code:
print(f"PyTorch version: {torch.__version__}")
# PyTorch version: 1.8.1

# Device configuration-
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"currently available device: {device}")
# currently available device: cpu

# Define transformations for training and test sets-
transform_train = transforms.Compose(
    [
      # transforms.RandomCrop(32, padding = 4),
      # transforms.RandomHorizontalFlip(),
      transforms.ToTensor(),
      # transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
     ]
     )

transform_test = transforms.Compose(
    [
      transforms.ToTensor(),
      # transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
     ]
     )

# Define directory containing images-
data_dir = 'My_Datasets/Cat_Dog_data/'

# Define datasets-
train_data = datasets.ImageFolder(data_dir + '/train',
                                  transform = train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test',
                                 transform = test_transforms)

print(f"number of train images = {len(train_data)} & number of validation images = {len(test_data)}")
# number of train images = 22500 & number of validation images = 2500

print(f"number of training classes = {len(train_data.classes)} & number of validation classes = {len(test_data.classes)}")
# number of training classes = 2 & number of validation classes = 2

# Define data loaders-
trainloader = torch.utils.data.DataLoader(train_data, batch_size = 32)
testloader = torch.utils.data.DataLoader(test_data, batch_size = 32)

len(trainloader), len(testloader)
# (704, 79)

# Sanity check-
len(train_data) / 32, len(test_data) / 32

You can iterate through the train data using 'train_loader' as follows:
for img, lab in train_loader:
   print(img.shape, lab.shape)
   pass

However, I am interested in getting the file name along with the file path from which the file was read. How can I achieve this?
Thanks!

Output: The default ImageFolder Dataset holds the paths of all images in self.samples. All you need to do is modify __getitem__ to return the paths as well.


Instruction: Good accuracy and loss on training vs bad accuracy on validation

Input:
I am learning pytorch and I have created binary classification algorithm. After having trained the model I have very low loss and quite good accuracy. However, on validation the accuracy is exactly 50%. I am wondering if I loaded samples incorrectly or the algorithm does not perform well.
Here you can find the plot of Training loss and accuracy.
Here is my training method:
epochs = 15
itr = 1
p_itr = 100
model.train()
total_loss = 0
loss_list = []
acc_list = []
for epoch in range(epochs):
    for samples, labels in train_loader:
        samples, labels = samples.to(device), labels.to(device)
        optimizer.zero_grad()
        output = model(samples)
        labels = labels.unsqueeze(-1)
        labels = labels.float()
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        scheduler.step()

    #if itr%p_itr == 0:
    pred = torch.round(output)
    correct = pred.eq(labels)
    acc = torch.mean(correct.float())
    print('[Epoch {}/{}] Iteration {} -> Train Loss: {:.4f}, Accuracy: {:.3f}'.format(epoch+1, epochs, itr, total_loss/p_itr, acc))
    loss_list.append(total_loss/p_itr)
    acc_list.append(acc)
    total_loss = 0

    itr += 1

Here, I am loading data from the path:
train_list_cats = glob.glob(os.path.join(train_cats_dir,'*.jpg'))
train_list_dogs = glob.glob(os.path.join(train_dogs_dir,'*.jpg'))
train_list = train_list_cats + train_list_dogs

val_list_cats = glob.glob(os.path.join(validation_cats_dir,'*.jpg'))
val_list_dogs = glob.glob(os.path.join(validation_dogs_dir,'*.jpg'))

val_list = val_list_cats + val_list_dogs

I am not attaching the model architecture, however I can add it if required.
I think that my training method is correct, although, I am not sure about training/validation data processing.
Edit:
The network params are as follow:
optimizer = torch.optim.RMSprop(model.parameters(), lr=0.001)
criterion = nn.BCELoss()
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[500,1000,1500], gamma=0.5)

Activation function is sigmoid.
The network architecture:
 self.layer1 = nn.Sequential(
        nn.Conv2d(3,16,kernel_size=3),
        nn.ReLU(),
        nn.MaxPool2d(2),
        nn.Dropout(p=0.2)
    )

    self.layer2 = nn.Sequential(
        nn.Conv2d(16,32, kernel_size=3),
        nn.ReLU(),
        nn.MaxPool2d(2),
        nn.Dropout(p=0.2)
        )

    self.layer3 = nn.Sequential(
        nn.Conv2d(32,64, kernel_size=3),
        nn.ReLU(),
        nn.MaxPool2d(2),
        nn.Dropout(p=0.2)
    )

    self.fc1 = nn.Linear(17*17*64,512)
    self.fc2 = nn.Linear(512,1)
    self.relu = nn.ReLU()
    self.sigmoid = nn.Sigmoid()

def forward(self,x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    out = out.view(out.size(0),-1)
    out = self.relu(self.fc1(out))
    out = self.fc2(out)
    return torch.sigmoid(out)

Output: Going by your "Training loss and accuracy" plot your model is overfitting. Your train loss is near zero after 25 epochs and you continue training for 200+ epochs. This is wrong way to train a model. You should rather be doing early stopping based on the validation set. ie. Run one epoch of train and one epoch of eval and repeat. Stop when your train epoch is improving and the corresponding eval epoch is not improving.


Instruction: difference in code between using nn.RNN or not

Input:
hi im new to rnn's and I found RNN NLP FROM SCRATCH from pytorch official tutorials, and I think it's named "from scartch" because it didn't use the nn.RNN built in nn in pytorch some line like this self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True) in the def __init__(self, input_size, hidden_size, output_size): segment. so how to the code would have been evolved if the nn.RNN was been used?
class RNN(nn.Module):
    # implement RNN from scratch rather than using nn.RNN
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input_tensor, hidden_tensor):
        combined = torch.cat((input_tensor, hidden_tensor), 1)

        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def init_hidden(self):
        return torch.zeros(1, self.hidden_size)
def train(line_tensor, category_tensor):
    hidden = rnn.init_hidden()

    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)

    loss = criterion(output, category_tensor)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    return output, loss.item()

another equivalent to this question is how to rewrite the code with using self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True) or if it's not possible how internal nn.RNN structure look like?

Output:
This model is referring the implementation of RNN before autograde module introduce, it is a pure implementation of RNN. In this example hidden state and gradient entirely handled by graph.
def init_hidden(self):
        return torch.zeros(1, self.hidden_size)

the line above initializes the hidden state(which is zeros at first). and after first step we get the output and next hidden state which later feed in the next step.
All this process handle by graph.


Instruction: What is the purpose of optimizer's state_dict in PyToch Big Graph's embedding dataset?

Input: The documentation for PyTorch Big Graph (PBG) states that "An additional dataset may exist, optimizer/state_dict, which contains the binary blob (obtained through torch.save()) of the state dict of the model’s optimizer." When inspecting this dataset, it seems to be stored as an array of bytes. Could someone conceptually explain the point of state_dict and why it's stored as an array rather than a dictionary?

Output:
Could someone conceptually explain the point of state_dict

If you know about Adam or SGD's momentum you probably know that there're some parameters in the optimizer that change in every step. When resume training on top of loading the model weights it'll make convergence faster if you load these parameters too.
You can get away without it, just that sometime it'll almost as if you start training from scratch.

why it's stored as an array rather than a dictionary?

If it's really obtained through torch.save() then it is in fact stored in dictionary or at least a list of dictionaries. Just that your process of "inspecting" it is wrong. Try
print(torch.load('path_to_the_file'))


Instruction: Slice a list with two other lists in tensorflow / pytorch

Input:
How can I slice a list with two other lists? In another word, how can I do a vectorized slicing in tensorflow?
indptr = [0 2 2 5 7]
values = [2 4 3 2 1 1 5]

values[indptr[:-1]:indptr[1:]] # --> throws exception

expected output:
[[2, 4],
 [],
 [3, 2, 1],
 [1, 5]]

More specifically, I wanna vectorized the following loop:
import numpy as np

# sparse representation in CSR format
indptr = [0, 2, 2, 5, 7]
indices = [1, 3, 0, 1, 2, 2, 3]
values = [2, 4, 3, 2, 1, 1, 5]

m, n = 4, 4
out = np.zeros((m, n))

for i in range(m):
    out[i][indices[indptr[i]:indptr[i + 1]]] = values[indptr[i]:indptr[i + 1]]

expected output:
# dense representation
[[0, 2, 0, 4],
 [0, 0, 0, 0],
 [3, 2, 1, 0],
 [0, 0, 1, 5]])

Output:
In tensorflow, tf.scatter_nd can be used for the purpose.
@tf.function
def csr_to_dense(indptr,indices,values,m,n):
  repeats=indptr[1:]-indptr[:-1]
  ind1=tf.repeat(tf.range(m),repeats)
  indices=tf.stack([ind1,indices],1)
  return tf.scatter_nd(indices,values,(m,n))

indptr=tf.constant([0,2,2,5,7])
indices=tf.constant([1,3,0,1,2,2,3])
values=tf.constant([2,4,3,2,1,1,5],dtype=tf.float32)
m=tf.constant(4)
n=tf.constant(4)
print(csr_to_dense(indptr,indices,values,m,n))

#from scipy docs, "Duplicate entries are summed together"
indptr=tf.constant([0,9,9,9,9,9])
indices=tf.constant([0,0,0,0,0,0,0,1,2])
values=tf.constant([2,4,3,2,1,-1,5,-1,-3],dtype=tf.float32)
m=tf.constant(5)
n=tf.constant(3)
print(csr_to_dense(indptr,indices,values,m,n))

target=csr_matrix(np.random.randint(9,size=(5,4)))
indptr=tf.constant(target.indptr)
indices=tf.constant(target.indices)
values=tf.constant(target.data,dtype=tf.float32)
m=tf.constant(target.shape[0])
n=tf.constant(target.shape[1])
print(np.allclose(csr_to_dense(indptr,indices,values,m,n),target.toarray()))
'''
tf.Tensor(
[[0. 2. 0. 4.]
 [0. 0. 0. 0.]
 [3. 2. 1. 0.]
 [0. 0. 1. 5.]], shape=(4, 4), dtype=float32)
tf.Tensor(
[[16. -1. -3.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]], shape=(5, 3), dtype=float32)
True
'''


Instruction: pytorch's grid_sample return an incorrect value

Input:
I have a 3D matrix: img[i, j, k] = i+j+k.
In my opinion, if I want the value of (1, 2, 3), the grid_sample should return 6. But it not.
The code is:
import torch
from torch.nn import functional as F
import numpy as np
X, Y, Z = 10, 20, 30
img = np.zeros(shape=[X, Y, Z], dtype=np.float32)
for i in range(X):
    for j in range(Y):
        for k in range(Z):
            img[i,j,k] = i+j+k
inp = torch.from_numpy(img).unsqueeze(0).unsqueeze(0)
grid = torch.from_numpy(np.array([[1, 2, 3]], dtype=np.float32)).unsqueeze(1).unsqueeze(1).unsqueeze(1)
grid[..., 0] /= (X-1)
grid[..., 1] /= (Y-1)
grid[..., 2] /= (Z-1)
grid = 2*grid - 1
outp = F.grid_sample(inp, grid=grid, mode='bilinear', align_corners=True)
print(outp)

The grid_sample return 6.15. Is there anything wrong with my code?

Output:
Finally, I find the solution. The reason why the above code return an incorrect value is that the torch.grid_sample accept (z, y, x) point.
Thus, the correct code should be:
import torch
from torch.nn import functional as F
import numpy as np
X, Y, Z = 10, 20, 30
img = np.zeros(shape=[X, Y, Z], dtype=np.float32)
for i in range(X):
    for j in range(Y):
        for k in range(Z):
            img[i,j,k] = i+j+k
inp = torch.from_numpy(img).unsqueeze(0).unsqueeze(0)
grid = torch.from_numpy(np.array([[1, 2, 3]], dtype=np.float32)).unsqueeze(1).unsqueeze(1).unsqueeze(1)
grid[..., 0] /= (X-1)
grid[..., 1] /= (Y-1)
grid[..., 2] /= (Z-1)

grid = 2*grid - 1

newgrid = grid.clone()
newgrid[..., 0] = grid[..., 2]
newgrid[..., 1] = grid[..., 1]
newgrid[..., 2] = grid[..., 0]

outp = F.grid_sample(inp, grid=newgrid, mode='bilinear', align_corners=True)
print(outp)


Instruction: huggingface-hub 0.0.12 requires packaging>=20.9, but you'll have packaging 20.4 which is incompatible

Input:
huggingface-hub 0.0.12 requires packaging>=20.9, but you'll have packaging 20.4 which is incompatible
enter image description here

Output:
You will have to update the huggingface-hub through
pip install --upgrade huggingface-hub


Instruction: I want to get feature value of an object with YOLOv5

Input:
I want to get feature value of an object with YOLOv5. I'm guessing there is a hint in "detect.py" in opensource.
How can I get feature value of the object used for inference?Please tell me how to resolve.

Output:
The variable 'det' inside the def run in detect.py(line 181), you can know the xyxy value, the confidence score, and the number of class name of the object.
Since 'det' is a tensor data type, you will need to converting 'det'.
If you want to get only the number of class name of the object, you can easily get it by converting cls in detect.py(line 205) like 'int(cls)'.


Instruction: convert pytorch model with multiple networks to onnx

Input:
I am trying to convert pytorch model with multiple networks to ONNX, and encounter some problem.
The git repo:  https://github.com/InterDigitalInc/HRFAE
The Trainer Class:
class Trainer(nn.Module):
    def __init__(self, config):
        super(Trainer, self).__init__()
        # Load Hyperparameters
        self.config = config
        # Networks
        self.enc = Encoder()
        self.dec = Decoder()
        self.mlp_style = Mod_Net()
        self.dis = Dis_PatchGAN()
        ...

Here is how the trained model process image:
    def gen_encode(self, x_a, age_a, age_b=0, training=False, target_age=0):
        if target_age:
            self.target_age = target_age
            age_modif = self.target_age*torch.ones(age_a.size()).type_as(age_a)
        else:
            age_modif = self.random_age(age_a, diff_val=25)

        # Generate modified image
        self.content_code_a, skip_1, skip_2 = self.enc(x_a)
        style_params_a = self.mlp_style(age_a)
        style_params_b = self.mlp_style(age_modif)

        x_a_recon = self.dec(self.content_code_a, style_params_a, skip_1, skip_2)
        x_a_modif = self.dec(self.content_code_a, style_params_b, skip_1, skip_2)

        return x_a_recon, x_a_modif, age_modif

And as following is how I did to convert to onnx:
enc = Encoder()
dec = Decoder()
mlp = Mod_Net()
layers = [enc, mlp, dec]
model = torch.nn.Sequential(*layers)
# here is my confusion: how do I specify the inputs of each layer??
# E.g. one of the outputs of 'enc' layer should be input of 'mlp' layer,
# or the outputs of 'enc' layer should be part of inputs of 'dec' layer...

params = torch.load('./logs/001/checkpoint')
model[0].load_state_dict(params['enc_state_dict'])
model[1].load_state_dict(params['mlp_style_state_dict'])
model[2].load_state_dict(params['dec_state_dict'])

torch.onnx.export(model, torch.randn([1, 3, 1024, 1024]), 'trained_hrfae.onnx', do_constant_folding=True)

Maybe the convert-part code is in wrong way??
Could anyone help, many thanks!
#20210629-11:52GMT Edit:
I found there's constraint of using torch.nn.Sequential. The output of former layer in Sequential should be consistent with latter input.
So my code shouldn't work at all because the output of 'enc' layer is not consistent with input of 'mlp' layer.
Could anyone help how to convert this type of pytorch model to onnx? Many thanks, again :)

Output:
After research and try, I found a method which maybe in correct way:
Convert each net(Encoder, Mod_Net, Decoder) to onnx model, and handle their input/output in latter logic-process or any further procedure (e.g convert to tflite model).
I'm trying to port onto Android using this method.
#Edit 20210705-03:52GMT#
Another approach may be better: write a new net combines the three nets. I've prove the output is same as origin pytorch model.
class HRFAE(nn.Module):
    def __init__(self):
        super(HRFAE, self).__init__()
        self.enc = Encoder()
        self.mlp_style = Mod_Net()
        self.dec = Decoder()

    def forward(self, x, age_modif):
        content_code_a, skip_1, skip_2 = self.enc(x)
        style_params_b = self.mlp_style(age_modif)

        x_a_modif = self.dec(content_code_a, style_params_b, skip_1, skip_2)

        return x_a_modif

and then convert use following:
net = HRFAE()

params = torch.load('./logs/002/checkpoint')
net.enc.load_state_dict(params['enc_state_dict'])
net.mlp_style.load_state_dict(params['mlp_style_state_dict'])
net.dec.load_state_dict(params['dec_state_dict'])

net.eval()
torch.onnx.export(net, (torch.randn([1, 3, 512, 512]), torch.randn([1]).type(torch.long)), 'test_hrfae.onnx')

This should be the answer.


Instruction: Read data from numpy array into a pytorch tensor without creating a new tensor

Input:
Let's say I have a numpy array arr = np.array([1, 2, 3]) and a pytorch tensor tnsr = torch.zeros(3,)
Is there a way to read the data contained in arr to the tensor tnsr, which already exists rather than simply creating a new tensor like tnsr1 = torch.tensor(arr).
This is a simplified example of the problem, since I am using a dataset that contains nearly 17 million entries.
EDIT: I know I can manually loop through each entry in the array. With 17 million entries, that would take quite a while I believe...

Output:
You can do that using torch.from_numpy(arr). Here is an example that shows that it's not being copied.
import numpy as np
import torch

arr = np.random.randint(0,high=10**6,size=(10**4,10**4))
%timeit arr.copy()

tells me that it took 492 ms ± 6.54 ms to copy the array of random integers.
On the other hand
%timeit torch.from_numpy(arr)

tells me that it took 1.14 µs ± 131 ns to turn it into a tensor. So there is no way that the 100 mio integers could have been copied. Pytorch is still using the same data.
Finally your version i.e.
%timeit torch.tensor(arr)

gives 201 ms ± 4.08 ms. Which is quite surprising to me. Since it should not be faster than numpy's copy in copying. But when it's not copying what takes it 1/5 or a second? Maybe it's doing a shallow copy. Maybe somebody else can tell us what's going on exactly.


Instruction: Create a CNN that has a Kernel that is 1xD, where D is number of columns that slides vertically over a MxD matrix?

Input:
Create a CNN that has a Kernel that is 1xD, where D is number of columns that slides vertically over a MxD matrix?
I'm trying to create a CNN in Pytorch that has a kernel that slides a 1xD kernel over a 2D image vertically so the output should be Mx1. As in the CNN convolves each row of the image then produces a single value for each row. Also having the ability to change from a 1xD to a NxD where N is some predefined number of rows would be nice as well. The input is purely just a matrix not a 3D matrix representing an image.

Output:
Kernels in nn.Conv2d do not have to be square, they can also be rectangular:
class MyModel(nn.Module):
  def __init__(self, in_channels, out_channels, N, D):
    super(MyModel, self).__init__()
    self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=(N, D), padding=0, stride=1)

  def forward(self, x):
    return self.conv(x)

Note that your input x has to be 4 dimensional: B-C-H-W. Where the number of channels C must match in_channels defined when constructing MyModel.
If you have a single image with only one channel, then the input should have two leading singleton dimensions: that is, x should have the shape 1-1-M-D.
See this answer for more information about why x should be 4D.


Instruction: "torch.relu_(input) unknown parameter type" from pytorch

Input:
I am trying to run this 3D pose estimation repo in Google Colab on a GPU, but after doing all of the steps and putting in my own left/right cam vids, I get this error in Colab:
infering thread started
1 1
: cannot connect to X server
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/content/Stereo-3D-Pose-Estimation/poseinferscheduler.py", line 59, in infer_pose_loop
    l_pose_t = infer_fast(self.net, l_img, height, self.stride, self.upsample_ratio, self.cpu)
  File "/content/Stereo-3D-Pose-Estimation/pose3dmodules.py", line 47, in infer_fast
    stages_output = net(tensor_img)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/Stereo-3D-Pose-Estimation/models/with_mobilenet.py", line 115, in forward
    backbone_features = self.model(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/activation.py", line 102, in forward
    return F.relu(input, inplace=self.inplace)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 1296, in relu
    result = torch.relu_(input)
RuntimeError: unknown parameter type

I am a bit confused as to why I am seeing it, I have already installed all necessary prerequisites; also can't interpret what it means either.

Output:
Since the traceback happens in the pytorch library, I checked the code there on the pytorch github.
What the error means is that you are calling an inplace activation function in torch.relu_ to some object called input. However, what is happening is that the type of input is not recognized by the torch backend which is why it is a runtime error.
Therefore, I would suggest to print out input and also run
type(input)

to find out what object input represents and what that variable is. As a further reference, this is the particular script that Pytorch runs in the backend that leads it to throw an unknown parameter type error. From a quick look, it seems to be a switch statement that confirms if a value falls into a list of types. If it is not in the list of types, then it will run the default block which throws unknown parameter type error.
https://github.com/pytorch/pytorch/blob/aacc722aeca3de1aedd35adb41e6f8149bd656cd/torch/csrc/utils/python_arg_parser.cpp#L518-L541
EDIT:
If type(input) returns a torch.tensor then it is probably an issue with the version of python you are using. I know you said you have the prerequisites but I think it would be good to double check if you have python 3.6, and maybe but less preferably python 3.5 or 3.7. These are the python versions that work with the repo you just sent me.
You can find the python version on your collab by typing
!python --version on one of cells. Make sure that it returns a correct version supported by the software you are running. This error might come from the fact that instead of torch, python itself is expressing this error in its backend.
I found this stackoverflow useful as it shows how some code was unable to recognize a built in type dictionary in python: "TypeError: Unknown parameter type: <class 'dict_values'>"
The solution to this was to check python versions.
Sarthak


Instruction: How to implement this equation in pytorch?

Input:
I'm trying to implement GNNs from a research paper and I have to code the following equations to get some sort of relatedness scores sj.
Since I am new to pytorch, I'm having some difficulties implementing equations.
Following are the set of equations that I want to code:

I have the following inputs h(t) = input_a = hidden[:,0:8,:]
hc(t) = input_b = hidden[:,8:13,:]   The dimensions of 'hidden' is (1000, 13, 128) #(Batch size, inputs(context-0:8,  and candidates 8:13), embedding dimension)
Also, the subscripts i in h(t) belong to 0:8, and j in hc(t) belong to 8:13. So, hi(t) would be hidden[:,i,:].
To provide some context,

Could someone please help me code the equations in the first picture ? I can even implement the function g() by myself, but I'm confused about the first two. Could someone help ?

Output:
All the operations you could do to pytorch tensors are documented over here:
https://pytorch.org/docs/stable/torch.html
I suggest command + F to search for the operation you need.
For the 1st equations you gave:

You can find torch.tanh() as shown here: https://pytorch.org/docs/stable/generated/torch.tanh.html#torch.tanh
You can find torch.transpose() over here
https://pytorch.org/docs/stable/generated/torch.transpose.html#torch.transpose (Also has 2 aliases torch.swapaxes and torch.swapdims that you can find in the documentation I linked above)

For the second equation

You can find the exponential with torch.exp() as shown here https://pytorch.org/docs/stable/generated/torch.exp.html#torch.exp
The summation can be coded with a for loop or clever indexing in pytorch.

There are many operations you can do to a torch.tensor object that are pretty useful for papers. As a rule of thumb, when implementing papers just use this pytorch doc as a way to find out many of those operations are.
Additionally if you are less comfortable with pytorch, another suggestion can be to do the equations in numpy then convert that value to a tensor.
Sarthak


Instruction: Why the method of log_prob in my Pytorch doesn't work

Input:
For example, I have a Beta distribution in Pytorch, and the parameter a=0.01 and b=1.4709.
The density function is as below:
Density function of the Beta distribution
Then I sample an action from this distribution which is 1.1754943508222875e-38.
Now, there is something happened, after I calculate the log_prob of this action, what I get is 81.83833312988281.
We know that the sampled action is extremely small and the probability of this action should be very close to 1. However, the log_prob becomes very large and more than 0.
Firstly, shouldn't the log_prob be between [-inf, 0] ??

Output:
log_prob returns the log of the probability density/mass function (pdf) evaluated at the given sample value. Probability density mass is not the same as probability since for a continuous distribution like the Beta distribution  the probability of any single value is actually 0. As such, there's no stipulation that the log-pdf evaluated at a given point should be between [-inf,0].
In fact, for your linked example you can see that the pdf evaluated at  1.1754943508222875e-38 would be extremely large - hence the large positive log_prob value of 81.83833312988281.


Instruction: Graph Neural Network Regression

Input:
I am trying to implement a regression on a Graph Neural Network. Most of the examples that I see are that of classification in this area, none so far of regression. I saw one for classification as follows:
from torch_geometric.nn import GCNConv
class GCN(torch.nn.Module):
def __init__(self, hidden_channels):
    super(GCN, self).__init__()
    torch.manual_seed(12345)
    self.conv1 = GCNConv(dataset.num_features, hidden_channels)
    self.conv2 = GCNConv(hidden_channels, dataset.num_classes)

def forward(self, x, edge_index):
    x = self.conv1(x, edge_index)
    x = x.relu()
    x = F.dropout(x, p=0.5, training=self.training)
    x = self.conv2(x, edge_index)
    return x

model = GCN(hidden_channels=16)
print(model)

I am trying to modify it for my task, which basically includes performing a regression on a network with 30 nodes, each having 3 features and the edge has one feature.
If anyone could point me to examples to do the same, that would be very helpful.

Output:
add a linear layer,and don't forget use a regression loss function
class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = GCNConv(dataset.num_features, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, dataset.num_classes)
        self.linear1 = torch.nn.Linear(100,1)
    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        x = self.linear1(x)
        return x


Instruction: What are the expected values in the input in Pytorch?

Input:
I am new to Pytorch, following the tutorials. I want to implement a regresor for a nonlinear function with 4 real inputs and 2 real outputs. I cannot find anywhere what is the supposed range for the inputs and outputs. They should go between -1 and 1? Between 0 and 1? Can it be anything?
More details
I have written the following simple parameterized model for experimenting:
class Net(torch.nn.Module):
    def __init__(self, n_inputs: int, n_outputs: int, n_hidden_layers: int, n_nodes_per_hidden_layer: int):
        n_inputs = int(n_inputs)
        n_outputs = int(n_outputs)
        n_hidden_layers = int(n_hidden_layers)
        n_nodes_per_hidden_layer = int(n_nodes_per_hidden_layer)
        if any([i<=0 for i in [n_inputs,n_outputs,n_hidden_layers,n_nodes_per_hidden_layer]]):
            raise ValueError(f'All n_inputs, n_outputs, n_hidden_layers and n_nodes_per_hidden_layer must be greater than 0.')
        super().__init__()
        self.input_layer = torch.nn.Linear(n_inputs, n_nodes_per_hidden_layer)
        self.hidden_layers = [torch.nn.Linear(n_nodes_per_hidden_layer, n_nodes_per_hidden_layer) for i in range(n_hidden_layers)]
        self.output_layer = torch.nn.Linear(n_nodes_per_hidden_layer, n_outputs)

    def forward(self, x):
        x *= .1
        activation_function = torch.nn.functional.relu
        x = activation_function(self.input_layer(x))
        for idx,layer in enumerate(self.hidden_layers):
            x = activation_function(layer(x))
        x = self.output_layer(x)
        return x

and I instantiate it in this way:
dnn = Net(
    n_inputs = 4, # Defined by the number of observables (charge of each channel now).
    n_outputs = 2, # x and y.
    n_hidden_layers = 3, # Choose yourself.
    n_nodes_per_hidden_layer = 66, # Choose yourself.
)

My input x is data that distributes in a weird way from 0 to 1, my outputs are 2 values in the range 1e-2 +- 100e-6. I have tried x -= .5 and different scalings too, but cannot make it work. I don't get any error, just it does not seem to learn what it is supposed to learn.
I know that this model should work because I have used it with similar data that distributes in a similar way but the inputs in the range 0-100e-12 using x *= 1e9 and it was performing reasonably well. I don't know why, however.

Output:
Data transformation

They should go between -1 and 1? Between 0 and 1? Can it be anything?

They can be any real valued numbers, but in general we standardize input values using mean and standard deviation (so the result has 0 mean and 1 variance) like this (for two dimensional data that you have, assuming samples are zeroth dimension and features are the first dimension):
import torch

samples, features = 128, 4
data = torch.randn(samples, features)
std, mean = torch.std_mean(data, dim=0, keepdim=True)

normalized = (data - mean) / std

In general neural networks work best with normalized input with similar ranges.
And this transformation could (and should as it will make it easier for nn.Linear layer output) also be applied to your regression target as it's reversible.
Other things

Use torch.nn.MSELoss for regression
Use standard optimizer (like Adam) with default learning rates (you can fine tune it later)
Make sure your pipeline works correctly


Instruction: Where can I get official PyTorch documentation in pdf form?

Input:
I want to learn PyTorch in great detail.
I have read all the docs and tutorials on the main site. I learn better from paper. When I print from website pages, I am getting very small letters. This makes the printouts difficult to read them.
The packages offer PDF documentation, but I cannot find a similar file for the main PyTorch site.
Where can I find a similar resource for official PyTorch documentation?

Output:
I don't think there is an official pdf. The pytorch documentation uses sphinx to generate the web version of the documentation. But sphinx can also generate PDFs.
So you could download the git repo of pytorch, install sphinx, and then generate the PDF yourself using sphinx.
The instructions to built the HTML can be found here, and generating the PDF should be no different.


Instruction: Why transformations go into the dataset and not into the NN itself in Pytorch?

Input:
I am new to Pytorch and I am now following the tutorial on transforms. I see that the transformations are configured into the dataset object. I am wondering, however, why aren't they configured within the neural network itself. My naive point of view is that the transformations should be in any case the most external layers of the network, in the same way as the eye comes before the brain to transform light into signals for the brain, and you don't modify the world instead to adapt it to the brain.
So, is there any technical reason for putting the transformations in the dataset instead of the net? Is it a good/bad practice to put the transformations within my neural network instead? Why?

Output:
These are some of the reason that can explain why one would do this.

We would like to use the same NN code for training as well as testing / inference. Typically during inference, we don't want to do any transformation and hence one might want to keep it out of the network. However, you may argue that one can just simply use model.training flag to skip the transformation.
Most of the transformations happen on CPU. Doing transformations in dataset allows to easily use multi-processing and prefetching. The dataset code can prefetch the data, transform, and keep it ready to be fed into the NN in a separate thread. If instead, we do it inside the forward function, GPUs will idle during the transformations (as these happen on CPU), likely leading to a longer training time.


Instruction: nvcc not found but cuda runs fine?

Input:
I was trying to run nvcc -V to check cuda version but I got the following error message.
Command 'nvcc' not found, but can be installed with:
sudo apt install nvidia-cuda-toolkit
But gpu acceleration is working fine for training models on cuda. Is there another way to find out cuda compiler tools version. I know nvidia-smi doesn't give the right version.
Is there a way to install or configure nvcc. So I don't have to install a whole new toolkit.

Output:
Most of the time, nvcc and other CUDA SDK binaries are not in the environment variable PATH. Check the installation path of CUDA; if it is installed under /usr/local/cuda, add its bin folder to the PATH variable in your ~/.bashrc:
export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

You can apply the changes with source ~/.bashrc, or the next time you log in, everything is set automatically.


Instruction: RuntimeError: Function AddmmBackward returned an invalid gradient

Input:
RuntimeError: Function AddmmBackward returned an invalid gradient at index 2 - got [100, 80] but expected shape compatible with [80, 80]

And my NN :

Output:
It could be because of your neural network shape is not compatible to the previous shape.
Try changing your fc1 from nn.Linear(in_feature=80, out_feature=80) to nn.Linear(in_feature=100, out_feature=80)


Instruction: pytorch cifar10 dataset - cannot get first item

Input:
I have selected the CIFAR 10 dataset using the torchvision library:
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transforms.ToTensor())

Then I try to select the first item in the dataset, which as I understand implements the get_item method of the dataset class:
trainset[0]

and I get
File "env\lib\site-packages\torchvision\transforms\functional.py", line 129, in to_tensor
    np.array(pic, mode_to_nptype.get(pic.mode, np.uint8), copy=True)
TypeError: __array__() takes 1 positional argument but 2 were given

Any ideas why I would get this error?
Python 3.7.9, torch==1.9.0, torchvision==0.10.0

Output:
I was hitting this error too:
def get_transformations():
    return transforms.Compose([transforms.ToTensor()])

...

self.transforms = get_transformations()

...

# Load the image + augment
img = Image.open(img_path).convert("RGB")
img = self.transforms(img)

...

Original Traceback (most recent call last):
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torch\utils\data\_utils\worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\src\screenshot_dataset.py", line 112, in __getitem__
    img = self.transforms(img)
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torchvision\transforms\transforms.py", line 60, in __call__
    img = t(img)
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torchvision\transforms\transforms.py", line 97, in __call__
    return F.to_tensor(pic)
  File "c:\2021-mcm-master\src\PyTorch-RCNN\ui-prediction\env\lib\site-packages\torchvision\transforms\functional.py", line 129, in to_tensor
    np.array(pic, mode_to_nptype.get(pic.mode, np.uint8), copy=True)
TypeError: __array__() takes 1 positional argument but 2 were given

As @Phil suggested, downgrading Pillow from 8.3.0 to 8.2.0 solved the issue:
pip install pillow==8.2.0


Instruction: Almost non-existent training accuracy and low test accuracy

Input:
I am really new to Machine Learning and I am not so well versed in coding in general. However there is need to look through the customers feedback at our store, that average quite a lot each year, yet we cannot tell % of positive, negative and neutral.
Currently I am trying to train a Bert Model to do simple multi labeled sentiment analysis. The input is our store's customers feedback. The customers feedback is not always so clearly defined since customers do tend to tell long and long about their experience and their sentiment is not always so clear. However we managed to get positive, negative and neutral, each set 2247 samples.
But when I try to train it the training accuracy is around 0.4% which is super low. Validation score is around 60%. F1-score is around 60% for each of the label. I wonder what can be done to improve this training accuracy. I have been stuck for a while. Please take a look at my codes and help me out with this.
I have tried changing learning rate (tried all learning rate Bert suggested and 1e-5),changing Max LEN, changing amount of EPOCH, changing drop out rate (0.1, 0.2, 0.3, 0.4, 0.5), but so far nothing yielded results.
#read dataset
df = pd.read_csv("data.csv",header=None, names=['content', 'sentiment'], sep='\;', lineterminator='\r',encoding = "ISO-8859-1",engine="python")

from sklearn.utils import shuffle

df = shuffle(df)

df['sentiment'] = df['sentiment'].replace(to_replace = [-1, 0, 1], value = [0, 1, 2])

df.head()

#Load pretrained FinBert model and get bert tokenizer from it
PRE_TRAINED_MODEL_NAME = 'TurkuNLP/bert-base-finnish-cased-v1'
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME)

#Choose sequence Length
token_lens = []

for txt in df.content:

 tokens = tokenizer.encode(txt, max_length=512)

 token_lens.append(len(tokens))

sns.distplot(token_lens)

plt.xlim([0, 256]);

plt.xlabel('Token count');

MAX_LEN = 260

#Make a PyTorch dataset
class FIDataset(Dataset):

 def __init__(self, texts, targets, tokenizer, max_len):

   self.texts = texts

   self.targets = targets

   self.tokenizer = tokenizer

   self.max_len = max_len

 def __len__(self):

   return len(self.texts)

 def __getitem__(self, item):

   text = str(self.texts[item])

   target = self.targets[item]

   encoding = self.tokenizer.encode_plus(

     text,

     add_special_tokens=True,

     max_length=self.max_len,

     return_token_type_ids=False,

     pad_to_max_length=True,

     return_attention_mask=True,

     return_tensors='pt',

   )

   return {

     'text': text,

     'input_ids': encoding['input_ids'].flatten(),

     'attention_mask': encoding['attention_mask'].flatten(),

     'targets': torch.tensor(target, dtype=torch.long)

   }

#split test and train
df_train, df_test = train_test_split(

 df,

 test_size=0.1,

 random_state=RANDOM_SEED

)

df_val, df_test = train_test_split(

 df_test,

 test_size=0.5,

 random_state=RANDOM_SEED

)

df_train.shape, df_val.shape, df_test.shape

#data loader function
def create_data_loader(df, tokenizer, max_len, batch_size):

 ds = FIDataset(

   texts=df.content.to_numpy(),

   targets=df.sentiment.to_numpy(),

   tokenizer=tokenizer,

   max_len=max_len

 )

 return DataLoader(

   ds,

   batch_size=batch_size,

   num_workers=4

 )

#Load data into train, test, val
BATCH_SIZE = 16

train_data_loader = create_data_loader(df_train, tokenizer, MAX_LEN, BATCH_SIZE)

val_data_loader = create_data_loader(df_val, tokenizer, MAX_LEN, BATCH_SIZE)

test_data_loader = create_data_loader(df_test, tokenizer, MAX_LEN, BATCH_SIZE)

# Sentiment Classifier based on Bert model just loaded
class SentimentClassifier(nn.Module):

 def __init__(self, n_classes):

   super(SentimentClassifier, self).__init__()

   self.bert = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)

   self.drop = nn.Dropout(p=0.1)

   self.out = nn.Linear(self.bert.config.hidden_size, n_classes)

 def forward(self, input_ids, attention_mask):
   returned = self.bert(

       input_ids=input_ids,
       attention_mask=attention_mask
   )
   pooled_output = returned["pooler_output"]
   output = self.drop(pooled_output)

   return self.out(output)

#Create a Classifier instance and move to GPU
model = SentimentClassifier(3)

model = model.to(device)

#Optimize with AdamW
EPOCHS = 5

optimizer = AdamW(model.parameters(), lr= 2e-5, correct_bias=False)

total_steps = len(train_data_loader) * EPOCHS

scheduler = get_linear_schedule_with_warmup(

 optimizer,

 num_warmup_steps=0,

 num_training_steps=total_steps

)

loss_fn = nn.CrossEntropyLoss().to(device)

#Train each Epoch function
def train_epoch(

 model,

 data_loader,

 loss_fn,

 optimizer,

 device,

 scheduler,

 n_examples

):

 model = model.train()

 losses = []

 correct_predictions = 0

 for d in data_loader:

   input_ids = d["input_ids"].to(device)

   attention_mask = d["attention_mask"].to(device)

   targets = d["targets"].to(device)

   outputs = model(

     input_ids=input_ids,

     attention_mask=attention_mask

   )

   _, preds = torch.max(outputs, dim=1)

   loss = loss_fn(outputs, targets)

   correct_predictions += torch.sum(preds == targets)

   losses.append(loss.item())

   loss.backward()

   nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

   optimizer.step()

   scheduler.step()

   optimizer.zero_grad()

   return correct_predictions.double() / n_examples, np.mean(losses)

#Eval model function
def eval_model(model, data_loader, loss_fn, device, n_examples):

 model = model.eval()

 losses = []

 correct_predictions = 0

 with torch.no_grad():

   torch.cuda.empty_cache()

   for d in data_loader:

     input_ids = d["input_ids"].to(device)

     attention_mask = d["attention_mask"].to(device)

     targets = d["targets"].to(device)

     outputs = model(

       input_ids=input_ids,

       attention_mask=attention_mask

     )

     _, preds = torch.max(outputs, dim=1)

     loss = loss_fn(outputs, targets)

     correct_predictions += torch.sum(preds == targets)

     losses.append(loss.item())

 return correct_predictions.double() / n_examples, np.mean(losses)

#training loop through each epochs

import torch

torch.cuda.empty_cache()

history = defaultdict(list)

best_accuracy = 0

if __name__ == '__main__':

   for epoch in range(EPOCHS):

     print(f'Epoch {epoch + 1}/{EPOCHS}')

     print('-' * 10)

     train_acc, train_loss = train_epoch(

       model,

       train_data_loader,

       loss_fn,

       optimizer,

       device,

       scheduler,

       len(df_train)

     )

     print(f'Train loss {train_loss} accuracy {train_acc}')

     val_acc, val_loss = eval_model(

       model,

       val_data_loader,

       loss_fn,

       device,

       len(df_val)

     )

     print(f'Val   loss {val_loss} accuracy {val_acc}')

     print()

     history['train_acc'].append(train_acc)

     history['train_loss'].append(train_loss)

     history['val_acc'].append(val_acc)

     history['val_loss'].append(val_loss)

     if val_acc > best_accuracy:

       torch.save(model.state_dict(), 'best_model_state.bin')

       best_accuracy = val_acc

-- Edit: I have printed out preds and targets as well as train and val accuracy

Output:
Here _, preds = torch.max(outputs, dim=1), you probably want argmax, not max?
Print out preds and targets to better see what's going on.
Edit after preds and targets printed out. For epochs 4 and 5, preds matches targets exactly, so train accuracy should be 1. I think the issue is that the accuracy is divided by n_examples, which is a number of examples in the whole train dataset, while it should be divided by the number of examples in the epoch.


Instruction: Making predictions on new images using a CNN in pytorch

Input:
I'm new in pytorch, and i have been stuck for a while on this problem. I have trained a CNN for classifying X-ray images. The images can be found in  this Kaggle page https://www.kaggle.com/prashant268/chest-xray-covid19-pneumonia/ .
I managed to get good accuracy both on training and test data, but when i try to make predictions on new images i get the same (wrong class) output for every image. Here's my model in detail.
import os
import matplotlib.pyplot as plt
import numpy as np
import torch
import glob
import torch.nn.functional as F
import torch.nn as nn
from torchvision.transforms import transforms
from torch.utils.data import DataLoader
from torch.optim import Adam
from torch.autograd import Variable
import torchvision
import pathlib
from google.colab import drive
drive.mount('/content/drive')

epochs = 20
batch_size = 128
learning_rate = 0.001

#Data Transformation
transformer = transforms.Compose([
                                  transforms.Resize((224,224)),
                                  transforms.RandomHorizontalFlip(),
                                  transforms.ToTensor(),
                                  transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])
                                  ])

#Load data with DataLoader
train_path = '/content/drive/MyDrive/Chest X-ray (Covid-19 & Pneumonia)/Data/train'
test_path = '/content/drive/MyDrive/Chest X-ray (Covid-19 & Pneumonia)/Data/test'

train_loader = DataLoader(torchvision.datasets.ImageFolder(train_path,transform = transformer), batch_size= batch_size, shuffle= True)
test_loader = DataLoader(torchvision.datasets.ImageFolder(test_path,transform = transformer), batch_size= batch_size, shuffle= False)

root = pathlib.Path(train_path)
classes = sorted([j.name.split('/')[-1] for j in root.iterdir()])
print(classes)
train_count = len(glob.glob(train_path+'/**/*.jpg')) + len(glob.glob(train_path+'/**/*.png')) + len(glob.glob(train_path+'/**/*.jpeg'))
test_count = len(glob.glob(test_path+'/**/*.jpg')) + len(glob.glob(test_path+'/**/*.png')) + len(glob.glob(test_path+'/**/*.jpeg'))
print(train_count,test_count)

#Create the CNN
class CNN(nn.Module):
  def __init__(self):
    super(CNN,self).__init__()
    '''nout = [(width + 2*padding - kernel_size) / stride] + 1 '''
    # [128,3,224,224]
    self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 12, kernel_size = 5)
    # [4,12,220,220]
    self.pool1 = nn.MaxPool2d(2,2) #reduces the images by a factor of 2
    # [4,12,110,110]
    self.conv2 = nn.Conv2d(in_channels = 12, out_channels = 24, kernel_size = 5)
    # [4,24,106,106]
    self.pool2 = nn.MaxPool2d(2,2)
    # [4,24,53,53] which becomes the input of the fully connected layer
    self.fc1 = nn.Linear(in_features = (24 * 53 * 53), out_features = 120)
    self.fc2 = nn.Linear(in_features = 120, out_features = 84)
    self.fc3 = nn.Linear(in_features = 84, out_features = len(classes)) #final layer, output will be the number of classes

  def forward(self, x):
    x = self.pool1(F.relu(self.conv1(x)))
    x = self.pool2(F.relu(self.conv2(x)))
    x = x.view(-1, 24 * 53 * 53)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

# Training the model
model = CNN()
loss_function = nn.CrossEntropyLoss() #includes the softmax activation function
optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

n_total_steps = len(train_loader)
for epoch in range(epochs):
  n_correct = 0
  n_samples = 0
  for i, (images, labels) in enumerate(train_loader):
    # Forward pass
    outputs = model(images)
    _, predicted = torch.max(outputs, 1)
    n_samples += labels.size(0)
    n_correct += (predicted == labels).sum().item()

    loss = loss_function(outputs, labels)
    # Backpropagation and optimization
    optimizer.zero_grad() #empty gradients
    loss.backward()
    optimizer.step()

    acc = 100.0 * n_correct / n_samples

  print(f'Epoch [{epoch+1}/{epochs}], Step [{i+1}/{n_total_steps}], Accuracy: {round(acc,2)} %, Loss: {loss.item():.4f}')
print('Done!!')

# Testing the model
with torch.no_grad():
  n_correct = 0
  n_samples = 0
  n_class_correct = [0 for i in range(3)]
  n_class_samples = [0 for i in range(3)]
  for images, labels in test_loader:
    outputs = model(images)
    # max returns (value ,index)
    _, predicted = torch.max(outputs, 1)
    n_samples += labels.size(0)
    n_correct += (predicted == labels).sum().item()

  acc = 100.0 * n_correct / n_samples
  print(f'Accuracy of the network: {acc} %')

torch.save(model.state_dict(),'/content/drive/MyDrive/Chest X-ray (Covid-19 & Pneumonia)/model.model')

For loading the model and trying to make predictions on new images, the code is as follows:
checkpoint = torch.load('/content/drive/MyDrive/Chest X-ray (Covid-19 & Pneumonia)/model.model')
model = CNN()
model.load_state_dict(checkpoint)
model.eval()

#Data Transformation
transformer = transforms.Compose([
                                  transforms.Resize((224,224)),
                                  transforms.ToTensor(),
                                  transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])
                                  ])

#Making preidctions on new data
from PIL import Image
def prediction(img_path,transformer):
  image = Image.open(img_path).convert('RGB')
  image_tensor = transformer(image)
  image_tensor = image_tensor.unsqueeze_(0) #so img is not treated as a batch
  input_img = Variable(image_tensor)
  output = model(input_img)
  #print(output)
  index = output.data.numpy().argmax()
  pred = classes[index]
  return pred

pred_path = '/content/drive/MyDrive/Chest X-ray (Covid-19 & Pneumonia)/Test_images/Data/'
test_imgs = glob.glob(pred_path+'/*')

for i in test_imgs:
    print(prediction(i,transformer))

I'm guessing the problem must be in the way that i am preprocessing the data, although i cannot find my mistake. Any help will be deeply appreciated, since i have been stuck on this for a while now.
p.s. i can share my notebook as well, if it is of any help

Output:
Regarding your problem, I have a  really good way to debug this to target where the problem most likely will be and so it will be really easy to fix your issue.
So, my debugging process would be based on the fact that your CNN performs well on the test set. Firstly set your test loader batch size to 1 temporarily. After that, One thing to do is in your test loop when you calculate the amount correct, you can run the following code:
#Your code
outputs = model(images) # Really only one image and 1 output.

#Altered Code:
correct =  (predicted == labels).sum().item() # This will be either 1 or 0 since you have only one image per batch

# My new code:
if correct:
   # if value is 1 instead of 0 then turn value into a single image with no batch size
   single_correct_image = images.squeeze(0)
   # Then convert tensor image into PIL image
   pil_image = transforms.ToPILImage()(single_correct_image)
   # Save the pil image to any directory specified in quotes.
   pil_image = pil_image.save("/content")

   #Terminate testing process. Ignore Value Error if it says terminating process
   raise ValueError("terminating process")

Now you have an image saved to disk that you know is correct in the test set. The next step would be to open such image and run it to your predict function. Couple of things can happen and thus give info about your situation

If your model returns the wrong answer then there is something wrong with the different code you have within the prediction and testing code. One uses a torch.sum and torch.max the other uses np.argmax.Then you can use print statements to debug what is going on there. Perhaps some conversion error or your expectation of the output's format is different.
If your code return the right answer then your model is just failing to predict on new images. I suggest running more trial cases with the above process.

For additional reference, if you still get very stuck to the point where you feel like you can't solve it, then I suggest using this notebook to guide and give some suggestions on what code to atleast inspect.
https://www.kaggle.com/salvation23/xray-cnn-pytorch
Sarthak Jain


Instruction: How does pytorch.autograd.Function calculate dL_dy?

Input:
This code calculate grad of y=x**2.
In this code, dL_dy is  [0., 2., 8.]  How do they calculate dL_dy? Where did this tensor came from ?
import torch
from torch.autograd import Function
class Square(Function):
    @staticmethod
    def forward(ctx,input):
        ctx.save_for_backward (input)
        return torch.square(input)

    @staticmethod
    def backward(ctx,dL_dy):
        print('dL_dy',dL_dy)
        print('ctx.saved_tensors',ctx.saved_tensors)
        x, = ctx.saved_tensors
        return dL_dy * 2 * x

square = Square.apply
x = torch.arange(3).to(torch.float64).requires_grad_(True)
y = square(x)
L = torch.sum(y*y)
L.backward()
print(x.grad)

Output:
Let say your f(y) = x**2, so f'(y) = 2*x
x = torch.arange(3).to(torch.float64).requires_grad_(True) means x=[0,1,2], we can compute y = x**2 = [0,1,4].
So when you call L.backward(), it will apply f'(y) (calculating the gradient of y), and stored in dL_dy.
That's why dL_dy = f'(y) = 2 * [0,1,4] = [0,2,8]


Instruction: Error in creating an offline PDF documentatin for PyTorch

Input:
I wanted to make an offline PDF on my system for PyTorch documentation. After reading from several resources  #1, #2, #3
git clone https://github.com/pytorch/pytorch

cd pytorch/docs/

make latexpdf

First two commands are working fine. Third command leads to the following error
Traceback (most recent call last):
  File "source/scripts/build_activation_images.py", line 70, in <module>
    function = torch.nn.modules.activation.__dict__[function_name]()
KeyError: 'SiLU'

How to overcome this error and make a PDF document of PyTorch?

1.4.0 is the version of PyTorch in my system
print(torch.__version__)
1.4.0

3.8.3 is the version of Python in my system
python -V
Python 3.8.3

Output:
The PyTorch version installed in your machine (1.4.0) is older than the one you cloned (most recent). Two ways to fix it:

Checkout to the version you have installed (if you want the doc of 1.4 version):

git clone https://github.com/pytorch/pytorch

# move back to the 1.4 release, which you have installed in your machine
cd pytorch
git checkout release/1.4

cd docs
make latexpdf

Upgrade to the most-recent PyTorch version (if you want the most recent doc):

# upgrade PyTorch to the nightly release (change it accordingly)
python -m pip install --pre torch torchvision torchaudio -f https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html

git clone https://github.com/pytorch/pytorch

cd pytorch/docs/

make latexpdf


Instruction: RuntimeError: expected scalar type Float but found Double (LSTM classifier)

Input:
I'm training my LSTM classifier.
epoch_num = 30

train_log = []
test_log = []
set_seed(111)
for epoch in range(1, epoch_num+1):

running_loss = 0
train_loss = []
lstm_classifier.train()
for (inputs, labels) in tqdm(train_loader, desc='Training epoch ' + str(epoch), leave=False):
    inputs, labels = inputs.to(device), labels.to(device)
    optimizer.zero_grad()
    outputs = lstm_classifier(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    train_loss.append(loss.item())
train_log.append(np.mean(train_loss))

running_loss = 0
test_loss = []
lstm_classifier.eval()
with torch.no_grad():
    for (inputs, labels) in tqdm(test_loader, desc='Test', leave=False):
        inputs, labels = inputs.to(device), labels.to(device)
        outputs = lstm_classifier(inputs)
        loss = criterion(outputs, labels)
        test_loss.append(loss.item())
test_log.append(np.mean(test_loss))
plt.plot(range(1, epoch+1), train_log, color='C0')
plt.plot(range(1, epoch+1), test_log, color='C1')
display.clear_output(wait=True)
display.display(plt.gcf())

error is:
RuntimeError                              Traceback (most recent call last)
 in ()
     23         print((labels.dtype))
     24         print(outputs[:,0].dtype)
---> 25         loss = criterion(outputs, labels)
     26         loss.backward()
     27         optimizer.step()

2 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2822     if size_average is not None or reduce is not None:
   2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2825
   2826

RuntimeError: expected scalar type Float but found Double
How to fix it?

Output:
RuntimeError: expected scalar type Float but found Double

The error at line loss = criterion(outputs, labels) is quite clear in that it requites your datatype to be float rather than double, but it doesn't explicitly say whether outputs or label is creating this.
My guess is its because of labels. Try converting it to float by doing labels.float()


Instruction: Easy way to convert a tensor shape In pytorch

Input:
Input
I have torch tensor as fallow.
The shape for this input_tensor is torch.size([4,4])
input_tensor =
tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11],
        [12, 13, 14, 15]])

I'm going to create a tensor that stacks up the tensor that comes out of the above input_tensor by moving the Window in size (2,2).
output
my desired output is as follow
The shape for this output_tensor is torch.size([8,2])
output =
tensor([[ 0,  1],
        [ 4,  5],
        [ 2,  3],
        [ 6,  7],
        [ 8,  9],
        [12, 13],
        [10, 11],
        [14, 15]])

My code is as follows.
x = torch.chunk(input_tensor, chunks=2, dim=0)
x = list(x)
for i, t in enumerate(x):
    x[i] = torch.cat(torch.chunk(t, chunks=2 ,dim=1))
output_tensor = torch.cat(x)

Is there a simpler or easier way to get the result I want?

Output:
You can use torch.split() together with torch.cat() as follows:
output_tensor = torch.cat(torch.split(input_tensor, 2, dim=1))

The ouput with be:
output =
tensor([[ 0,  1],
        [ 4,  5],
        [ 8,  9],
        [12, 13],
        [ 2,  3],
        [ 6,  7],
        [10, 11],
        [14, 15]])


Instruction: About pytorch tensor caculation

Input: I want to ask a question about torch calculation that if I only want to subtract the elements on the diagonal of the matrix without changing the elements in the remaining positions, is there any way to achieve it?

Output:
One way to do this is by getting the diagonal, doint the required operation on its elements and replacing the original one. Example code:
x = torch.rand(3, 3)
#get the original diagonal and for example substract 3
replaced_diag = x.diagonal() - 3
#replace the original diagonal
x.diagonal().copy_(replaced_diag)

For reference look at this: Replace diagonal elements with vector in PyTorch


Instruction: Pytorch - RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

Input:
I was trying & experimenting something with PyTorch, where I created my own inputs & targets. I fed these inputs to the model (which is a basic ANN with 2 hidden layers, nothing wrong with that). But for some reason I am not being able to calculate the CrossEntropyLoss(). I am not being able to figure out why. I know some of the other questions on StakcOverflow have the same title of mine or have a similar problem. I have gone through that but nothing worked out for me. Alot of people had an issue with the dataset, which does not seem to be the problem with me.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

class Net(nn.Module):
    def __init__(self) -> None:
        super(Net, self).__init__()
        self.layer1 = nn.Linear(2, 10)
        self.layer2 = nn.Linear(10, 1)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = self.layer2(x)
        return x

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net().to(device=device)
loss_fn = nn.CrossEntropyLoss()
learning_rate = 1e-3
epochs = 20
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
inputs = torch.Tensor([
    [0,0],
    [0,1],
    [1,0],
    [1,1]
], ).to(device=device)

targets = torch.Tensor([
    0,
    1,
    1,
    0
]).to(device=device)

model.train()
for epoch in range(epochs):

    pred_output = model(inputs)
    print(pred_output.dtype)
    print(targets.dtype)
    loss = loss_fn(pred_output, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print()
    break

The error that I see is,
torch.float32
torch.float32
Traceback (most recent call last):
  File ".\main.py", line 57, in <module>
    loss = loss_fn(pred_output, targets)
  File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 1047, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\functional.py", line 2693, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "C:\Users\user\anaconda3\lib\site-packages\torch\nn\functional.py", line 2388, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward

Output:
I could replicate you error using this code.
import torch.nn as nn
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.tensor([1., 2., 3.])
loss(input, target)

Error:

RuntimeError: expected scalar type Long but found Float

changed the datatype of target to target = torch.tensor([1., 2., 3.], dtype=torch.long) and everything worked fine. I believe the target variable does require long datatype because changing the input to float will also work.
#this will also work
input = torch.randn(3, 5, requires_grad=True, dtype=torch.float)
target = torch.tensor([1., 2., 3.], dtype=torch.long)
loss(input, target)

Note the documentation also has this torch.long dtype in example code. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
#Edit 1
The reason it's not working is because of the way you defined input/target tensors in your code. Use torch.tensor with a small 't' instead of torch.Tensor. For a detailed discussion see What is the difference between torch.tensor and torch.Tensor?.
#this will work. Also notice the decimal. otherwise it will be interpreted differently by pytorch
inputs = torch.tensor([[0.,0.],[0.,1.],[1.,0.],[1.,1.]]).to(device=device)
targets = torch.tensor([0.,1.,1.,0.], dtype=torch.long).to(device=device)


Instruction: Pytorch: gradient computation fails when in-place operation follows certain functions

Input:
Consider the following piece of code:
import torch
from torch import nn
a = torch.tensor([1.], requires_grad=True)
b = nn.Tanh()(a)
# b = nn.Linear(1,1)(a)
b *= 1
# b = b * 1
b.sum().backward()

Running the code results in RuntimeError:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead.

However, if I change Tanh() to Linear(1,1) or change b*=1 to b=b*1 (as in the commented lines), the code will run successfully and get the correct gradient.
Why is that?
My environment:

Python 3.8.5 (anaconda) on Windows
Pytorch 1.8.0, running on CPU

Output:
I just found some text from the Pytorch official site:

In-place correctness checks
Every tensor keeps a version counter, that is incremented every time it is marked dirty in any operation. When a Function saves any tensors for backward, a version counter of their containing Tensor is saved as well. Once you access self.saved_tensors it is checked, and if it is greater than the saved value an error is raised. This ensures that if you’re using in-place functions and not seeing any errors, you can be sure that the computed gradients are correct.

Source: https://pytorch.org/docs/stable/notes/autograd.html#in-place-correctness-checks
According to this, an in-place operation may or may not break the gradient computation, depending on the actual situation.
Therefore, as long as the code contains in-place operations, it should not be a surprise when (as what is observed in the question) changing something in the context magically fixes the gradient computation.
Also according to the same passage, in-place operations may disrupt the gradient calculation even if they do not break it, in which case the speed of gradient calculation may be decreased. So it seems like a good strategy to avoid any in-place operation.


Instruction: Installing pytorch with conda

Input:
I've just been trying to dabble in AI in the past few weeks, I've tried installing pytorch with conda and it all seems to work but then I get the error:
ImportError: /home/lp35791/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so: cannot read file data
I've been trawling through the web but can't seem to find the answer to this error. I've uninstalled and reinstalled anaconda and when I made a new environment and installed numpy along with pytorch, numpy imported successfully but pytorch did not. I'm just wondering what the problem is. Any help would be greatly appreciated!

Output:
You seem to have installed PyTorch in your base environment, you therefore cannot use it from your other "pytorch" env.
Either:
directly create a new environment (let's call it pytorch_env) with PyTorch: conda create -n pytorch_env -c pytorch pytorch torchvision
switch to the pytorch environment you have already created with: source activate pytorch_env and then install PyTorch in it: conda install -c pytorch pytorch torchvision