Instruction: How to access the save results of yolov5 in different folder?

Input:
I am using the below code to load the trained custom Yolov5 model and perform detections.
import cv2
import torch
from PIL import Image

model = torch.hub.load('ultralytics/yolov5', 'custom',
 path='yolov5/runs/train/exp4/weights/best.pt', force_reload=True)

img = cv2.imread('example.jpeg')[:, :, ::-1]  # OpenCV image (BGR to RGB)

results = model(img, size=416)

#To display and save results I am using:
results.print()
results.save()
results.show()

My question is how can I save the results in different directory so that I can use them in my web-based application. For your reference I am using Streamlit. For instance, at the moment, results (image) are being saved in runs\detect\exp*. I want to change it. Can anyone please guide me.

Output:
You can make changes in the function definition of results.save(), the function can be found in the file yolov5/models/common.py. By default the definition is:
def save(self, labels=True, save_dir='runs/detect/exp'):
        save_dir = increment_path(save_dir, exist_ok=save_dir != 'runs/detect/exp', mkdir=True)  # increment save_dir
        self.display(save=True, labels=labels, save_dir=save_dir)  # save results

You can make changes in the save_dir argument to the desired save location and the files should be saved in the new directory.


Instruction: PyTorch - FineTuning bert - Oscillating loss - Very bad accuracy

Input:
I have been trying to train a model on vulnerability detection through source code. And, after a little bit of searching, I thought a very good starting point could be using a pre-trained transformer model from HuggingFace with PyTorch and pl.lightning torch. I chose DistilBert because it was the fastest one.
I have an imbalanced dataset, approximately 70% non-vulnerable and 30% vulnerable functions.
However, my results have been very bad. The model does not seem to learn and generalize. Specifically, during training the train loss is heavily oscillating, accuracy is around 70 percent and recall is extremely low (implying that the model always predicts one label).
I was wondering if there is anything that might be obviously problematic with my code. This is the first time I am using a pre-trained model and pl lightning and I cannot really tell what might have gone wrong.
class Model(pl.LightningModule):
    def __init__(self, n_classes, n_training_steps, n_warmup_steps, lr, fine_tune=False):
        super().__init__()
        self.save_hyperparameters()
        self.bert = DistilBert.from_pretrained(BERT_MODEL_NAME, return_dict=True)
        for name, param in self.bert.named_parameters():
            param.requires_grad = False
        self.classifier = nn.Linear(self.bert.config.hidden_size, self.hparams.n_classes)
        self.criterion = nn.BCELoss()

    def finetune(self):
        self.fine_tune = True
        for name, param in self.bert.named_parameters():
            if  'layer.5' in name:
                param.requires_grad = True

    def forward(self, input_ids, attention_mask, labels=None):
        x = self.bert(input_ids, attention_mask=attention_mask)
        x = x.last_hidden_state[:,0,:]
        x = self.classifier(x)
        x = torch.sigmoid(x)
        x = x.squeeze(dim=-1)
        loss = 0
        if labels is not None:
            loss = self.criterion(x, labels.float())
        return loss, x

    def training_step(self, batch, batch_idx):
        enc, labels = batch
        input_ids, attention_mask = enc
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        return {'loss': loss, 'predictions': outputs, 'labels': labels}

    def validation_step(self, batch, batch_idx):
        enc, labels = batch
        input_ids, attention_mask = enc
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        r = recall(outputs[:], labels[:])
        self.log("val_loss", loss, prog_bar=True, logger=True)
        self.log("val_recall", r, prog_bar=True, logger=True)
        return {'loss': loss, 'predictions': outputs, 'labels': labels}

    def test_step(self, batch, batch_idx):
        enc, labels = batch
        input_ids, attention_mask = enc
        loss, outputs = self.forward(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        return {'loss': loss, 'predictions': outputs, 'labels': labels}

    def training_epoch_end(self, outputs):
        labels = []
        predictions = []
        for o in outputs:
            for o_labels in o['labels'].detach().cpu():
                labels.append(o_labels)
            for o_preds in o['predictions'].detach().cpu():
                predictions.append(o_preds)
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        class_recall = recall(predictions[:], labels[:])
        self.logger.experiment.add_scalar("recall/Train", class_recall, self.current_epoch)

    def validation_epoch_end(self, outputs):
        labels = []
        predictions = []
        for o in outputs:
            for o_labels in o['labels'].detach().cpu():
                labels.append(o_labels)
            for o_preds in o['predictions'].detach().cpu():
                predictions.append(o_preds)
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        class_recall = recall(predictions[:], labels[:])
        self.logger.experiment.add_scalar("recall/Validation", class_recall, self.current_epoch)

    def test_epoch_end(self, outputs):
        labels = []
        predictions = []
        for o in outputs:
            for o_labels in o['labels'].detach().cpu():
                labels.append(o_labels)
            for o_preds in o['predictions'].detach().cpu():
                predictions.append(o_preds)
        labels = torch.stack(labels).int()
        predictions = torch.stack(predictions)
        class_recall = recall(predictions[:], labels[:])
        self.logger.experiment.add_scalar("recall/Test", class_recall, self.current_epoch)

    def configure_optimizers(self):
        optimizer = AdamW(self.parameters(), lr=self.hparams.lr if self.hparams.fine_tune == False else self.hparams.lr // 100)
        scheduler = get_linear_schedule_with_warmup(
            optimizer,
            num_warmup_steps=self.hparams.n_warmup_steps,
            num_training_steps=self.hparams.n_training_steps
        )
        return dict(
            optimizer=optimizer,
            lr_scheduler=dict(
                scheduler=scheduler,
                interval='step'
            )
        )

if __name__ == "__main__":

    data_module = SourceCodeDataModule(batch_size=BATCH_SIZE)
    steps_per_epoch = len(train_loader) // BATCH_SIZE
    total_training_steps = steps_per_epoch * N_EPOCHS
    warmup_steps = total_training_steps // 5

    model = Model(
        n_classes=1,
        n_warmup_steps = warmup_steps,
        n_training_steps=total_training_steps,
        lr=2e-5
    )

    logger = TensorBoardLogger("lightning_logs", name="bert_predictor")
    early_stopping_callback = EarlyStopping(monitor='val_loss', patience=2)
    trainer = pl.Trainer(
    logger=logger,
    checkpoint_callback=checkpoint_callback,
    callbacks=[early_stopping_callback],
    max_epochs=N_EPOCHS,
    gpus=1 if str(device).startswith('cuda') else 0,
    progress_bar_refresh_rate=30
    )

    # First just train the final layer.
    trainer.fit(model, datamodule=data_module)
    result = trainer.test(model, datamodule=data_module)
    print(f"Result when training classifier only: {result}")

    # Then train the whole model
    model = Model.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)
    model.finetune()
    trainer.fit(model, datamodule=data_module)
    result = trainer.test(model, datamodule=data_module)
    print(f"Result when fine tuning: {result}")

Output:
Here,
def finetune(self):
        self.fine_tune = True
        for name, param in self.bert.named_parameters():
            if  'layer.5' in name:
                param.requires_grad = True

try to unfreeze more layers at the end of the neural net, maybe the weights are saturated and not learning enough. Also, pay attention to the loss you are using, as well as the activation function at the output.


Instruction: TypeError: backward() got an unexpected keyword argument 'variables'

Input:
I am using the recurrent Gaussian Process library. I believe the code is developed by older versions of python and pytorch. I ran one of the experiments of the model after cloning the repository
python ./testing/rnn_rgp_test.py

I got this error message from this line of the rnn_encoder.py script:
./RGP/autoreg/rnn_encoder.py", line 274, in backward_computation
    torch.autograd.backward( variables=self.forward_means_list + self.forward_vars_list,
TypeError: backward() got an unexpected keyword argument 'variables'

I will be grateful if someone can point out how I can fix this error?

Output:
Version 0.3.1 of PyTorch seems to be the last version with the variables parameter. Ideally, the RGP library should have documented which version of their dependencies they use but they didn't. Given that their Git repo seems to be inactive, you have several choices:

Use old versions of whatever libraries they require. You will have to go from one error to the next, hoping that things work as intended.
Fork RGP and re-implement the logic with current libraries. This will likely involve significant coding and may not even be possible at all.
Try to find a different library that implements RGPs.


Instruction: Why is RandomCrop with size 84 and padding 8 returning an image size of 84 and not 100 in pytorch?

Input:
I was using the mini-imagenet data set and noticed this line of code:
    elif data_augmentation == 'lee2019:
        normalize = Normalize(
            mean=[120.39586422 / 255.0, 115.59361427 / 255.0, 104.54012653 / 255.0],
            std=[70.68188272 / 255.0, 68.27635443 / 255.0, 72.54505529 / 255.0],
        )
        train_data_transforms = Compose([
            ToPILImage(),
            RandomCrop(84, padding=8),
            ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4),
            RandomHorizontalFlip(),
            ToTensor(),
            normalize,
        ])
        test_data_transforms = Compose([
            normalize,
        ])

but when I checked the image size it was 84 instead of 100 (after adding padding):
X.size()=torch.Size([50, 3, 84, 84])

what is going on with this? Shouldn't it be 100?

reproduction:
import random
from typing import Callable

import learn2learn as l2l
import numpy as np
import torch
from learn2learn.data import TaskDataset, MetaDataset, DataDescription
from learn2learn.data.transforms import TaskTransform
from torch.utils.data import Dataset

class IndexableDataSet(Dataset):

    def __init__(self, datasets):
        self.datasets = datasets

    def __len__(self) -> int:
        return len(self.datasets)

    def __getitem__(self, idx: int):
        return self.datasets[idx]

class SingleDatasetPerTaskTransform(Callable):
    """
    Transform that samples a data set first, then creates a task (e.g. n-way, k-shot) and finally
    applies the remaining task transforms.
    """

    def __init__(self, indexable_dataset: IndexableDataSet, cons_remaining_task_transforms: Callable):
        """

        :param: cons_remaining_task_transforms; constructor that builds the remaining task transforms. Cannot be a list
        of transforms because we don't know apriori which is the data set we will use. So this function should be of
        type MetaDataset -> list[TaskTransforms] i.e. given the dataset it returns the transforms for it.
        """
        self.indexable_dataset = MetaDataset(indexable_dataset)
        self.cons_remaining_task_transforms = cons_remaining_task_transforms

    def __call__(self, task_description: list):
        """
        idea:
        - receives the index of the dataset to use
        - then use the normal NWays l2l function
        """
        # - this is what I wish could have gone in a seperate callable transform, but idk how since the transforms take apriori (not dynamically) which data set to use.
        i = random.randint(0, len(self.indexable_dataset) - 1)
        task_description = [DataDescription(index=i)]  # using this to follow the l2l convention

        # - get the sampled data set
        dataset_index = task_description[0].index
        dataset = self.indexable_dataset[dataset_index]
        dataset = MetaDataset(dataset)

        # - use the sampled data set to create task
        remaining_task_transforms: list[TaskTransform] = self.cons_remaining_task_transforms(dataset)
        description = None
        for transform in remaining_task_transforms:
            description = transform(description)
        return description

def sample_dataset(dataset):
    def sample_random_dataset(x):
        print(f'{x=}')
        i = random.randint(0, len(dataset) - 1)
        return [DataDescription(index=i)]
        # return dataset[i]

    return sample_random_dataset

def get_task_transforms(dataset: IndexableDataSet) -> list[TaskTransform]:
    """
    :param dataset:
    :return:
    """
    transforms = [
        sample_dataset(dataset),
        l2l.data.transforms.NWays(dataset, n=5),
        l2l.data.transforms.KShots(dataset, k=5),
        l2l.data.transforms.LoadData(dataset),
        l2l.data.transforms.RemapLabels(dataset),
        l2l.data.transforms.ConsecutiveLabels(dataset),
    ]
    return transforms

def print_datasets(dataset_lst: list):
    for dataset in dataset_lst:
        print(f'\n{dataset=}\n')

def get_indexable_list_of_datasets_mi_and_cifarfs(root: str = '~/data/l2l_data/') -> IndexableDataSet:
    from learn2learn.vision.benchmarks import mini_imagenet_tasksets
    datasets, transforms = mini_imagenet_tasksets(root=root)
    mi = datasets[0].dataset

    from learn2learn.vision.benchmarks import cifarfs_tasksets
    datasets, transforms = cifarfs_tasksets(root=root)
    cifarfs = datasets[0].dataset

    dataset_list = [mi, cifarfs]

    dataset_list = [l2l.data.MetaDataset(dataset) for dataset in dataset_list]
    dataset = IndexableDataSet(dataset_list)
    return dataset

# -- tests

def loop_through_l2l_indexable_datasets_test():
    """

    :return:
    """
    # - for determinism
    random.seed(0)
    torch.manual_seed(0)
    np.random.seed(0)

    # - options for number of tasks/meta-batch size
    batch_size: int = 10

    # - create indexable data set
    indexable_dataset: IndexableDataSet = get_indexable_list_of_datasets_mi_and_cifarfs()

    # - get task transforms
    def get_remaining_transforms(dataset: MetaDataset) -> list[TaskTransform]:
        remaining_task_transforms = [
            l2l.data.transforms.NWays(dataset, n=5),
            l2l.data.transforms.KShots(dataset, k=5),
            l2l.data.transforms.LoadData(dataset),
            l2l.data.transforms.RemapLabels(dataset),
            l2l.data.transforms.ConsecutiveLabels(dataset),
        ]
        return remaining_task_transforms
    task_transforms: TaskTransform = SingleDatasetPerTaskTransform(indexable_dataset, get_remaining_transforms)

    # -
    taskset: TaskDataset = TaskDataset(dataset=indexable_dataset, task_transforms=task_transforms)

    # - loop through tasks
    for task_num in range(batch_size):
        print(f'{task_num=}')
        X, y = taskset.sample()
        print(f'{X.size()=}')
        print(f'{y.size()=}')
        print(f'{y=}')
        print()

    print('-- end of test --')

# -- Run experiment

if __name__ == "__main__":
    import time
    from uutils import report_times

    start = time.time()
    # - run experiment
    loop_through_l2l_indexable_datasets_test()
    # - Done
    print(f"\nSuccess Done!: {report_times(start)}\a")

context: https://github.com/learnables/learn2learn/issues/333
crossposted:

https://discuss.pytorch.org/t/why-is-randomcrop-with-size-84-and-padding-8-returning-an-image-size-of-84-and-not-100-in-pytorch/151463
https://www.reddit.com/r/pytorch/comments/uno1ih/why_is_randomcrop_with_size_84_and_padding_8/

Output:
The padding is applied to the input image or tensor before applying the random crop. Ultimately, the output image has a spatial size equal to that of the provided size(s) given to the T.RandomCrop function since the operation is performed after.
After all, it makes more sense to pad the input image rather than the cropped image, doesn't it?


Instruction: TypeError: forward() takes 1 positional argument but 2 were given while inferencing of PyTorch model

Input:
My mode likes the following:
    class RankingModel(nn.Module):
        def __init__(self, conf: Dict[Text, Any], **kwargs: Any):
            super(RankingModel, self).__init__()
            self.conf = deepcopy(conf)
            ......

    def forward(self, **_features):   # the model input is a torch.utils.data.Dataset()
        ### model body part.
        return prob

Then I train my model using:
trainer = Trainer(
                model=model,
                args=training_args,
                train_dataset=training_dataset,
                eval_dataset=valid_dataset,
                compute_metrics=compute_metrics_for_binary_classification,
                callbacks=[callback],
          )

trainer.train()

Then I predict the result with the model.
test_predict = model(x_test)

I get the error:
TypeError                                 Traceback (most recent call last)
Input In [18], in <cell line: 9>()
     17 x_test.from_dict(feature_test)
     18 x_test.set_format(tensor_type="torch")
---> 20 test_predict = model(x_test) # trainer.predict(x_test).predictions
     21 if np.argmax(test_predict) < 5:
     22     recall_counter = recall_counter + 1

File ~/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py:1051, in Module._call_impl(self, *input, **kwargs)
   1047 # If we don't have any hooks, we want to skip the rest of the logic in
   1048 # this function, and just call forward.
   1049 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051     return forward_call(*input, **kwargs)
   1052 # Do not call functions when jit is used
   1053 full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() takes 1 positional argument but 2 were given

But all is OK if I predict the result through:
test_predict = trainer.predict(x_test).predictions

Why may I not use model(x_test) to get inference result? Could you please give me any suggestions? Thanks.

Output:
Your forward expect argument with key like forward(data=myarray) because you used double asterix when defining it and didn't give positional argument.
either use def forward(self, input, **kwargs)which  would read the first argument of the call and then use other argument as kwargs
or call it with:
model(keyword=x_test) and then in your foward function you can access it with _features['keyword']


Instruction: Why won't PyTorch RNN accept unbatched input?

Input:
I'm trying to train a PyTorch RNN to predict the next value in a 1D sequence. According to the PyTorch documentation page, I think I should be able to feed unbatched input to the RNN with shape [L,H_in] where L is the length of the sequence and H_in is the input length. That is, a 2D vector.
https://pytorch.org/docs/stable/generated/torch.nn.RNN.html
import torch

x1 = torch.tensor([[1.0], [2.0], [3.0], [4.0], [5.0], [6.0], [7.0], [8.0]])
x1_input = x1[0:-1, :]
x1_target = x1[1:, :]

rnn = torch.nn.RNN(1, 1, 1)

optimizer_prediction = torch.optim.Adam(rnn.parameters())
prediction_loss = torch.nn.L1Loss()
rnn.train()

epochs = 100
for i in range(0, epochs):
    output_x1 = rnn(x1_input)
    print(output_x1)
    error_x1 = prediction_loss(output_x1, x1_target)
    error_x1.backward()
    optimizer_prediction.step()
    optimizer_prediction.zero_grad()

However, PyTorch is complaining that it expects a 3D input vector (i.e. including a dimension for the batch):
RuntimeError: input must have 3 dimensions, got 2

What is the correct method for feeding unbatched input to an RNN?

Output:
I would recommend turning your input into a 3d array by adding a batch size of one with:
torch.unsqueeze(x1_input, dim=0).


Instruction: How do I calculate the mean of values with the same label given by a mask for multi-dimensional data?

Input:
My input x is a tensor of [B, C, H, W] dim. B is batch size, C number of channels, H height and W width. I have a mask m of [H, W] dim. For each batch size and each channel I want to use the mask m to calculate the mean of all values in [H, W] with the same label.
For example:
B = 2
C = 2
H = 2
W = 3
x = torch.tensor([[[[1,2,3][4,5,6]][[7,8,9][0,1,2]]][[[1,2,3][4,5,6]][[7,8,9][0,1,2]]]])

m = torch.tensor([[0,0,1][0,2,2]])

I expect as output:
mean = torch.tensor([[2.3,3,5.5][5,9,1.5][2.3,3,5.5][5,9,1.5]])

out = torch.tensor([[[[2.3,2.3,3][2.3,5.5,5.5]][[5,5,9][5,1.5,1.5]]][[[2.3,2.3,3][2.3,5.5,5.5]][[5,5,9][5,1.5,1.5]]]])

What I want is the following:
1.) The mean value for each label [mean of label 0, mean of label 1, mean of label 2, mean of label 3, mean of label 4]
2.) I want to replace each value in x with the mean value of its associated label. Output should have the size [B, C, H, W].
I thought to somehow use torch_scatter.scatter_mean.

Output:
I figured out how to get the mean values for multi-dimensional data. I flatten H and W and then call scatter_mean from torch_scatter. That gives me the mean values per label for each channel and batch.
x = x.view(B, C, H*W)
m = m.view(B, 1, H*W)
mean = torch_scatter.scatter_mean(x, m, dim=-1)


Instruction: Steps for Machine Learning in Pytorch

Input:
When we define our model in PyTorch. We run through different #epochs. I want to know that in the iteration of epochs.
What is the difference between the two following snippets of code in which the order is different? These two snippet versions are:

I found over tutorials
The code provided by my supervisor for the project.

Tutorial Version
for i in range(epochs):
    logits = model(x)
    loss = loss_fcn(logits,lables)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Supervisor Version
for i in range(epochs):
    logits = model(x)
    loss = loss_fcn(logits,lables)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Output: The only difference is when the gradients are cleared. (when you call optimizer.zero_grad()) the first version zeros out the gradients after updating the weights (optimizer.step()), the second one zeroes out the gradient after updating the weights. both versions should run fine. The only difference would be the first iteration, where the second snippet is better as it makes sure the residue gradients are zero before calculating the gradients. Check this link that explains why you would zero the gradients


Instruction: pytorch: simple recurrent neural network for image classification

Input:
I am making a simple recurrent neural network architecture for CIFAR10 image classification. I am not interested not use pre-defined RNN class in PyTorch because i am implementing from scratch according to figure. I am getting input tensor errors in the same device.  I am not sure whether my code is right or wrong. Any simple way to write FC layer without defining shape and hard coded parameters.
Figure

Code
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.input_to_hidden = nn.Linear(in_features=input_size + hidden_size, out_features=output_size)
        self.input_to_output = nn.Linear(in_features=input_size + hidden_size, out_features=output_size)
        self.softmax = nn.LogSoftmax(dim=1)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, input_tensor):
        combined = torch.cat((input_tensor, torch.zeros(input_tensor.size(0))), 1)
        hidden = self.input_to_hidden(combined)
        output = self.input_to_output(combined)
        output = self.softmax(output)
        return output, hidden

Trackback
Traceback (most recent call last):
  File "/media/cvpr/CM_1/tutorials/rnn.py", line 81, in <module>
    outputs = model(images)
  File "/home/cvpr/anaconda3/envs/tutorials/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/cvpr/CM_1/tutorials/rnn.py", line 33, in forward
    combined = torch.cat((input_tensor, torch.zeros(input_tensor.size(0))), 1)
RuntimeError: All input tensors must be on the same device. Received cuda:0 and cpu

Output:
You need to make sure the tensors are on the same device (cpu/gpu) before you are contacting them
you can add a device parameter to your class and use it:
    class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_classes, device='cuda'):
        super(RNN, self).__init__()
        self.device = device
        self.hidden_size = hidden_size
        self.input_to_hidden = nn.Linear(in_features=input_size + hidden_size, out_features=output_size)
        self.input_to_output = nn.Linear(in_features=input_size + hidden_size, out_features=output_size)
        self.softmax = nn.LogSoftmax(dim=1)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, input_tensor):
        combined = torch.cat((input_tensor.to(device), torch.zeros(input_tensor.size(0), device=self.device)), 1)
        hidden = self.input_to_hidden(combined)
        output = self.input_to_output(combined)
        output = self.softmax(output)
        return output, hidden


Instruction: Transforms.Normalize returns values higher than 255 Pytorch

Input:
I am working on an video dataset, I read the frames as integers and convert them to a numpy array float32.
After being loaded, they appear in a range between 0 and 255:
   [165., 193., 148.],
   [166., 193., 149.],
   [167., 193., 149.],
   ...

Finally, to feed them to my model and stack the frames I do the "ToTensor()" plus my transformation [transforms.Resize(224), transforms.Normalize([0.454, 0.390, 0.331], [0.164, 0.187, 0.152])]
and here the code to transform and stack the frames:
res_vframes = []
for i in range(len(v_frames)):
    res_vframes.append(self.transforms((v_frames[i])))
res_vframes = torch.stack(res_vframes, 0)

The problem is that after the transformation the values appears in this way, which has values higher than 255:
[tensor([[[1003.3293, 1009.4268, 1015.5244,  ..., 1039.9147, 1039.9147,
          1039.9147],...

Any idea on what I am missing or doing wrong?

Output:
The behavior of torchvision.transforms.Normalize:
output[channel] = (input[channel] - mean[channel]) / std[channel]

Since the numerator of the lefthand of the above equation is greater than 1 and the denominator of it is smaller than 1, the computed value gets larger.
The class ToTensor() maps a tensor's value to [0, 1] only if some condition is satisfied. Check this code from official Pytorch docs:
if isinstance(pic, np.ndarray):
        # handle numpy array
        if pic.ndim == 2:
            pic = pic[:, :, None]

        img = torch.from_numpy(pic.transpose((2, 0, 1))).contiguous()
        # backward compatibility
        if isinstance(img, torch.ByteTensor):
            return img.to(dtype=default_float_dtype).div(255)
        else:
            return img

Therefore you need to divide tensors explicitly or make to match the above condition.


Instruction: No CMAKE_CUDA_COMPILER could be found when installing pytorch

Input:
I am trying to install pytorch from source. The reason why I am doing this (instead of just pip install pytorch) is because I need the sm_86 support for my GPU (NVIDIA GTX 3060 Ti) and so I have set the TORCH_CUDA_ARCH_LIST=8.6 variable. I've read that this variable affects only the source installation.
Basically I am following this guide (linux system, using pip instead of conda) but I was not able to understand how to correctly set the CMAKE_PREFIX_PATH variable.
Despite this I tried in any case to install pytorch with the python3 setup.py install command but it returned this error:
CMake Error at cmake/public/cuda.cmake:47 (enable_language):
  No CMAKE_CUDA_COMPILER could be found.

  Tell CMake where to find the compiler by setting either the environment
  variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:43 (include)
  CMakeLists.txt:696 (include)

The log file shows this:
Checking whether the CUDA compiler is NVIDIA using "" did not match "nvcc: NVIDIA \(R\) Cuda compiler driver":
Checking whether the CUDA compiler is Clang using "" did not match "(clang version)":
Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed.
Compiler: CMAKE_CUDA_COMPILER-NOTFOUND
Build flags: ;-Xfatbin;-compress-all
Id flags:  -v

The output was:
No such file or directory

Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed.
Compiler: CMAKE_CUDA_COMPILER-NOTFOUND
Build flags:
Id flags:  -v

The output was:
No such file or directory

Can anybody help me solve this?
Update
Cuda seems to be installed. With apt list --installed|grep cuda this is the output (I am Italian "sconosciuto"=unknown :)
cuda-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-cccl-11-7/sconosciuto,now 11.7.58-1 amd64 [installato, automatico]
cuda-command-line-tools-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-compiler-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-cudart-11-7/sconosciuto,now 11.7.60-1 amd64 [installato, automatico]
cuda-cudart-dev-11-7/sconosciuto,now 11.7.60-1 amd64 [installato, automatico]
cuda-cuobjdump-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-cupti-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-cupti-dev-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-cuxxfilt-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-demo-suite-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-documentation-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-driver-dev-11-7/sconosciuto,now 11.7.60-1 amd64 [installato, automatico]
cuda-drivers-515/sconosciuto,now 515.43.04-1 amd64 [installato, automatico]
cuda-drivers/sconosciuto,now 515.43.04-1 amd64 [installato, automatico]
cuda-gdb-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-libraries-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-libraries-dev-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-memcheck-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nsight-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nsight-compute-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-nsight-systems-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-nvcc-11-7/sconosciuto,now 11.7.64-1 amd64 [installato, automatico]
cuda-nvdisasm-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvml-dev-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvprof-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvprune-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvrtc-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvrtc-dev-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvtx-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-nvvp-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-runtime-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-sanitizer-11-7/sconosciuto,now 11.7.50-1 amd64 [installato, automatico]
cuda-toolkit-11-7-config-common/sconosciuto,now 11.7.60-1 all [installato, automatico]
cuda-toolkit-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-toolkit-11-config-common/sconosciuto,now 11.7.60-1 all [installato, automatico]
cuda-toolkit-config-common/sconosciuto,now 11.7.60-1 all [installato, automatico]
cuda-tools-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda-visual-tools-11-7/sconosciuto,now 11.7.0-1 amd64 [installato, automatico]
cuda/sconosciuto,now 11.7.0-1 amd64 [installato]
libcuda1/sconosciuto,now 515.43.04-1 amd64 [installato, automatico]
nvidia-cuda-mps/sconosciuto,now 515.43.04-1 amd64 [installato, automatico]

Output:
My guess is that the CUDA installation is somehow messed up / invisible - otherwise CMake should have noticed it. You can overcome the issue more "manually" by running CMake like so:
CUDACXX=/usr/local/cuda-11.7/bin/nvcc cmake -S /path/to/source/dir -B /path/to/build/dir

(as you have installed CUDA under /usr/local/cuda-11.7)
A less-likely cause is that CMake doesn't properly recognize the just-released CUDA 11.7, although I doubt it.


Instruction: Massive neural network training time increase by inverting images in a data set

Input:
I have been working with neural networks for a few months now and I have a little mystery that I can't solve on my own.
I wanted to create and train a neural network which can identify simple geometric shapes (squares, circles, and triangles) in 56*56 pixel greyscale images. If I use images with a black background and a white shape, everything work pretty well. The training time is about 18 epochs and the accuracy is pretty close to 100% (usually 99.6 % - 99.8%).
But all that changes when I invert the images (i.e., now a white background and black shapes). The training time skyrockets to somewhere around 600 epochs and during the first 500-550 epochs nothing really happens. The loss barely decreases in those first 500-550 epochs and it just seems like something is "stuck".
Why does the training time increase so much and how can I reduce it (if possible)?

Output:
Color inversion

You have to essentially “switch” WxH pixels, hence touching every possible pixel during augmentation for every image, which amounts to lots of computation.
In total it would be DxWxH operations per epoch (D being size of your dataset).
You might want to precompute these and feed your neural network with them afterwards.

Loss

It is harder for neural networks as white is encoded with 1, while black is encoded with 0. Inverse giving us 0 for white and 1 for black.
This means most of neural network weights are activated by background pixels!
What is more, every sensible signal (0 in case of inversion) is multiplied by zero value and has not effect on final loss.
With hard {0, 1} encoding neural network tries to essentially get signal from the background (now 1 for black pixels) which is mostly meaningless (each weight will tend to zero or almost zero as it bears little to no information) and what it does instead is fitting distribution to your labels (if I predict 1 only I will get smaller loss, no matter the input).

Experiment if you are bored

Try to encode your pixels with smooth values, e.g. white being 0.1 and black being 0.9 (although 1.0 might work okayish, although more epochs might be needed as 0 is very hard to obtain via backprop) and see what the results are.


Instruction: AttributeError: 'Upsample' object has no attribute 'recompute_scale_factor'

Input:
I get error on line x_stats = dec(z).float().
import torch.nn.functional as F

z_logits = enc(x)
z = torch.argmax(z_logits, axis=1)
z = F.one_hot(z, num_classes=enc.vocab_size).permute(0, 3, 1, 2).float()

x_stats = dec(z).float()
x_rec = unmap_pixels(torch.sigmoid(x_stats[:, :3]))
x_rec = T.ToPILImage(mode='RGB')(x_rec[0])

display_markdown('Reconstructed image:')
display(x_rec)

I tried to downgrade and reinstall the torch package but that didn't help the issue. My package version is torch==1.11.0
Full traceback:
AttributeError                            Traceback (most recent call last)
/Users/hanpham/github/DALL-E/notebooks/usage.ipynb Cell 4' in <cell line: 7>()
      4 z = torch.argmax(z_logits, axis=1)
      5 z = F.one_hot(z, num_classes=enc.vocab_size).permute(0, 3, 1, 2).float()
----> 7 x_stats = dec(z).float()
      8 x_rec = unmap_pixels(torch.sigmoid(x_stats[:, :3]))
      9 x_rec = T.ToPILImage(mode='RGB')(x_rec[0])

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/homebrew/lib/python3.9/site-packages/dall_e/decoder.py:94, in Decoder.forward(self, x)
     91 if x.dtype != torch.float32:
     92     raise ValueError('input must have dtype torch.float32')
---> 94 return self.blocks(x)

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/container.py:141, in Sequential.forward(self, input)
    139 def forward(self, input):
    140     for module in self:
--> 141         input = module(input)
    142     return input

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/container.py:141, in Sequential.forward(self, input)
    139 def forward(self, input):
    140     for module in self:
--> 141         input = module(input)
    142     return input

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/upsampling.py:154, in Upsample.forward(self, input)
    152 def forward(self, input: Tensor) -> Tensor:
    153     return F.interpolate(input, self.size, self.scale_factor, self.mode, self.align_corners,
--> 154                          recompute_scale_factor=self.recompute_scale_factor)

File /opt/homebrew/lib/python3.9/site-packages/torch/nn/modules/module.py:1185, in Module.__getattr__(self, name)
   1183     if name in modules:
   1184         return modules[name]
-> 1185 raise AttributeError("'{}' object has no attribute '{}'".format(
   1186     type(self).__name__, name))

AttributeError: 'Upsample' object has no attribute 'recompute_scale_factor'

Output:
Install Torch version, this will solve the issue

pip install torchvision==0.10.1
pip install torch==1.9.1


Instruction: Pytorch-forecasting:: Univariate AssertionError: filters should not remove entries all entries

Input:
I tried to do univariate forecasting with Pytorch-Forecasting.
But I got following error on TimeSeriesDataSet

AssertionError: filters should not remove entries all entries - check
encoder/decoder lengths and lags

I have tried googling for the error, read the suggestion and make sure my training_df has sufficient number of rows. (I have plenty: 196). Also I only has 1 group_id which is 1. No other group_id, so all those 196 should be in same group.
My dataframes sample:
note: all rows has same group value = 1

    PutCall_Ratio_Total  time_idx  group

Date
2006-02-24    11119.140000    0         1
2006-02-25    7436.316667     1         1
2006-02-26    3753.493333     2         1

I have training_df with length of 196

len(training_df)
196

And here is my TimeSeriesDataSet portion:
 context_length = 28*7
 prediction_length = 7
     # setup Pytorch Forecasting TimeSeriesDataSet for training data
     training_data = TimeSeriesDataSet(
         training_df,
         time_idx="time_idx",
         target="PutCall_Ratio_Total",
         group_ids=["group"],
         time_varying_unknown_reals=["PutCall_Ratio_Total"],
         max_encoder_length=context_length,
         max_prediction_length=prediction_length
     )```

Output:
After some experiment, it seems that the training_df length (196) should be larger
than or equal to (context_length + prediction_length).
So for example above it works once I update the context_length to 27 * 7 instead of 28 * 7.
Since 27 * 7 + 7 = 196.
While 28 * 7 + 7 > 196.


Instruction: Google Colab: torch cuda is true but No CUDA GPUs are available

Input: I use Google Colab to train the model, but like the picture shows that when I input 'torch.cuda.is_available()' and the ouput is 'true'. And then I run the code but it has the error that RuntimeError: No CUDA GPUs are available.

Output:
Try to install cudatoolkit version you want to use
"conda install pytorch torchvision cudatoolkit=10.1 -c pytorch"


Instruction: Pytorch: Dataloader shuffle=False producing same batches

Input:
class DataSet(torch.utils.data.Dataset):
    def __init__(self,dataframe,n_classes=3,w=384,h=384,apply_aug=False):
        self.data=dataframe
        self.n_classes=n_classes
        self.apply_aug=apply_aug
        self.w=w
        self.h=h
        self.transform=A.Compose([A.Rotate(limit=30,p=0.8),
          A.HorizontalFlip(),
          A.CoarseDropout(max_height=0.1,max_width=0.1,p=1.0),
          A.ShiftScaleRotate(shift_limit=0.09,scale_limit=0.2,rotate_limit=0)
          ])

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self,idx):
        IMG=np.zeros((1,self.w,self.h),dtype=np.float32)
        MASK=np.zeros((self.n_classes,self.w,self.h),dtype=np.float32)
        path=self.data.iloc[idx]["image_path"]
        encoded_list=self.data.iloc[idx]["segmentation"]
        width=int(self.data.iloc[idx]["width"][0])
        heigth=int(self.data.iloc[idx]["heigth"][0])
        class_list=self.data.iloc[idx]["class"]
        img=implt.imread(path)
        print(f"idx is {idx}")
        IMG[:,:width,:heigth]=img
        for class_idx,c in enumerate(class_list):
            if str(encoded_list[class_idx])=="nan":
                mask=np.zeros((1,width,heigth))
            else:
                mask=RLE_TO_MASK(encoded_list[class_idx],width,heigth)
            MASK[c,:width,:heigth]=mask[0]
        if self.apply_aug:
            transformed=self.transform(image=IMG,mask=MASK)
            IMG,MASK=transformed['image'],transformed['mask']
        IMG=IMG/255.0
        return IMG,MASK

Above is the dataset function created. It outputs the images and masks.
When I change shuffle=True for data loader, it's working fine but when I change shuffle=False. For the next batch the data loader providing the same batch which is produced before.
dataloader=torch.utils.data.DataLoader(DataSet(df,apply_aug=True),batch_size=BATCH_SIZE,shuffle=False)
for i in range(2):
    images,masks=next(iter(dataloader))
    print()
print(images.shape,masks.shape)

idx is 0
idx is 1
idx is 2
idx is 3
idx is 4
idx is 5
idx is 6
idx is 7
idx is 8
idx is 9
idx is 10
idx is 11
idx is 12
idx is 13
idx is 14
idx is 15
idx is 16
idx is 17
idx is 18
idx is 19
idx is 20
idx is 21
idx is 22
idx is 23
idx is 24
idx is 25
idx is 26
idx is 27
idx is 28
idx is 29
idx is 30
idx is 31
idx is 0
idx is 1
idx is 2
idx is 3
idx is 4
idx is 5
idx is 6
idx is 7
idx is 8
idx is 9
idx is 10
idx is 11
idx is 12
idx is 13
idx is 14
idx is 15
idx is 16
idx is 17
idx is 18
idx is 19
idx is 20
idx is 21
idx is 22
idx is 23
idx is 24
idx is 25
idx is 26
idx is 27
idx is 28
idx is 29
idx is 30
idx is 31
torch.Size([32, 1, 384, 384]) torch.Size([32, 3, 384, 384])
When shuffle=True
for i in range(2):
    images,masks=next(iter(dataloader))
    print()
print(images.shape,masks.shape)
idx is 25498
idx is 15357
idx is 11275
idx is 36247
idx is 33223
idx is 8566
idx is 14229
idx is 23999
idx is 28883
idx is 8847
idx is 35485
idx is 36647
idx is 22422
idx is 3693
idx is 32525
idx is 19464
idx is 22187
idx is 38244
idx is 7795
idx is 3690
idx is 7461
idx is 36806
idx is 22455
idx is 6817
idx is 8789
idx is 37809
idx is 33157
idx is 22828
idx is 35858
idx is 38320
idx is 2684
idx is 29708

idx is 38240
idx is 28020
idx is 10356
idx is 20215
idx is 18561
idx is 30083
idx is 30997
idx is 14020
idx is 20896
idx is 25551
idx is 2735
idx is 19138
idx is 23026
idx is 30677
idx is 26664
idx is 2731
idx is 14150
idx is 16735
idx is 28621
idx is 18268
idx is 11793
idx is 35654
idx is 4470
idx is 11312
idx is 37349
idx is 27501
idx is 5389
idx is 34019
idx is 24120
idx is 38311
idx is 14880
idx is 9533

torch.Size([32, 1, 384, 384]) torch.Size([32, 3, 384, 384])

Output:
You use iterator incorrectly:
next(iter(dataloader))

Every step you create a new iterator and take the first element (hence it's always the same because the iterator is actually the same). Instead you should create the iterator before for-loop  and call next() in every step.
But why not simply iterate over your dataloader this way:
for images,masks in dataloader:
   # do sth with data


Instruction: How to calculate mutual information in PyTorch (differentiable estimator)

Input: I am training a model with pytorch, where I need to calculate the degree of dependence between two tensors (let's say they are the two tensors each containing values very close to zero or one, e.g. v1 = [0.999, 0.998, 0.001, 0.98] and v2 = [0.97, 0.01, 0.997, 0.999]) as a part of my loss function. I am trying to calculate mutual information, but I can't find any mutual information estimation implementation in PyTorch. Has such a thing been provided anywhere?

Output:
Mutual information is defined for distribution and not individual points. So, I will write the next part assuming v1 and v2 are samples from a distribution, p. I will also take that you have n samples from p, n>1.
You want a method to estimate mutual information from samples. There are many ways to do this. One of the simplest ways to do this would be to use a non-parametric estimator like NPEET (https://github.com/gregversteeg/NPEET). It works with numpy (you can convert from torch to numpy for this). There are more involved parametric models for which you may be able to find implementation in pytorch (See https://arxiv.org/abs/1905.06922).
If you only have two vectors and want to compute a similarity measure, a dot product similarity would be more suitable than mutual information as there is no distribution.


Instruction: How to require gradient only for some tensor elements in a pytorch tensor?

Input:
I like to use a tensor with only a few variable elements which are considered during the backpropagation step. Consider for example:
self.conv1 = nn.Conv2d(3, 16, 3, 1, padding=1, bias=False)
mask = torch.zeros(self.conv1.weight.data.shape, requires_grad=False)
self.conv1.weight.data[0, 0, 0, 0] += mask[0, 0, 0, 0]
print(self.conv1.weight.data[0, 0 , 0, 0].requires_grad)

It will be output False

Output:
You can only switch on and off gradient computation at the tensor level which means that the requires_grad is not element-wise. What you observe is different because you have accessed the requires_grad attribute of conv1.weight.data which is not the same object as its wrapper tensor conv1.weight!
Notice the difference:
>>> conv1 = nn.Conv2d(3, 16, 3) # requires_grad=True by default

>>> conv1.weight.requires_grad
True

>>> conv1.weight.data.requires_grad
False

conv1.weight is the weight tensor while conv1.weight.data is the underlying data tensor which never requires gradient because it is at a different level.

Now onto how to solve the partially requiring gradient computation on a tensor. Instead of looking at solving it as "only require gradient for some elements of tensor", you can think of it as "don't require gradient for some elements of tensor". You can do so by overwriting the gradient values on the tensor at the desired positions after the backward pass:
>>> conv1 = nn.Conv2d(3, 1, 2)
>>> mask = torch.ones_like(conv1.weight)

For example, to prevent the update of the first component of the convolutional layer:
>>> mask[0,0,0,0] = 0

After the backward pass, you can mask the gradient on conv1.weight:
>>> conv1(torch.rand(1,3,10,10)).mean().backward()
>>> conv1.weight.grad *= mask


Instruction: How can I change my environment architecture to arm64 from x86-64?

Input:
I am an MacBook M1 user and I am trying to use M1 GPU (MPS) supported by Pytorch. I read that I need to make sure my system is arm64 rather than x86 so I created my env as below:
  CONDA_SUBDIR=osx-arm64 conda create -n nlp2 --clone nlp

  (nlp2) twang20@C02G82XRQ05N ~ % python --version
  Python 3.9.7
  (nlp2) twang20@C02G82XRQ05N ~ % conda config --env --set subdir
  osx-arm64
  (nlp2) twang20@C02G82XRQ05N ~ % uname -m
  arm64

However, in torch, when I checked the environment info, it still tells me my architecture is x86-64. I cannot find a way to change it to arm64.
get_pretty_env_info()

Out[2]:
PyTorch version: 1.12.0.dev20220520
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 11.6.5 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: Could not collect
Libc version: N/A
Python version: 3.9.7 (default, Sep 16 2021, 08:50:36)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.3
[pip3] numpydoc==1.1.0
[pip3] torch==1.12.0.dev20220520
[pip3] torchaudio==0.12.0.dev20220520
[pip3] torchvision==0.13.0.dev20220520
[conda] blas

I would expect to see something like this:
OS: macOS 11.6.5 (arm64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.30)
CMake version: Could not collect
Libc version: N/A

Python version: 3.9.7 (default, Sep 16 2021, 08:50:36)  [Clang 10.0.0 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit

How can I make it happen? Thanks.

Output:
Cloning is going to copy/link the packages from the previous environment, which is already x86_64. Instead, you would need to recreate the environment. Something like:
## dump previous environment
conda env export -n nlp --from-history > nlp_x86.yaml

## create new one with temp subdir
CONDA_SUBDIR=osx-arm64 conda env create -n nlp_arm -f nlp_x86.yaml

## permanently set subdir after creation
conda activate nlp_arm
conda config --env --set subdir osx-arm64

However, you'll likely need to edit the YAML to add channels, adjust packages, etc.. For example, some packages may not yet be available.
In particular, the M1 support from PyTorch is still only on nightly builds, so you'll need the pytorch-nightly channel.
Also, note they aren't yet building other PyTorch packages (e.g., torchvision) for osx-arm64, so at the time of this writing, I wouldn't expect full environments to simply swap out to M1 support. Might need to wait for an official release.


Instruction: Input Shape of Deep learning model

Input:
I have created deep learning models in different input shapes.
For Testing ,  I am resizing the images according to the model's input shape manually
I need to resize the image with input shape of the deep model
Any Command to find the input shape of the model in PYTORCH
model = torch.load(config.MODEL_PATH).to(config.DEVICE)
im = cv2.resize(im, (INPUT_IMAGE_HEIGHT,INPUT_IMAGE_HEIGHT))

How can I find the INPUT_IMAGE_HEIGHT,INPUT_IMAGE_HEIGHT from model
Thank You

Output:
This is a tricky question because your input size will can depend on several components of your model. The short answer is you can't.

Concerning the number of channels in your input tensor, you can infer this solely based on the first convolutional layer. Assuming your model is a two dimensional convolutional network, then you can get the input channel number based on :
for child in model.modules():
    if type(child).__name__ == 'Conv2d':
        print(child.weight.size(1))
        break

Now for the input size, as I said you may not be able to infer this information at all. Indeed some convolutional networks, such as classification networks, may require specific dimensions such that the bottleneck can be flattened and fed into a fully-connected network. This is not always true though, networks that use some sort of pulling operation (average, or maximum pulling) will alleviate the need to provide fixed input shapes. Other networks such as dense prediction networks may not need to get a specific shape as the input, given the symmetry between input and output...
This all depends on the network's design and as such, I'm afraid there is no definitive answer to your question.


Instruction: Why does a call to torch.tensor inside of apply_async fail to complete (seems to block execution)?

Input:
I'm trying to understand why the following simple example doesn't successfully complete execution and seems to get stuck on the first line of really_simple_func (on Ubuntu machines, but not Windows). The code is:
import torch as t
import numpy as np
import multiprocessing as mp          # I've tried both multiprocessing
# import torch.multiprocessing as mp  # and torch.multiprocessing

def really_simple_func():
    temp_val_2 = t.tensor(np.zeros(425447)[0:400000])  # this is the line that blocks.
    return 4.3

if __name__ == "__main__":
    print("Run brief starting")
    some_zeros = np.zeros(425447)
    temp_val = t.tensor(some_zeros[0:400000]) # DELETE THIS LINE TO MAKE IT WORK

    pool = mp.Pool(processes=1)
    job = pool.apply_async(really_simple_func)
    print("just before job.get()")
    result = job.get()
    print("Run brief completed. Reward = {}".format(result))

I have torch 1.11.0 installed, numpy 1.22.3 and have tried both CPU and GPU versions of Torch. When I run this code on two different Ubuntu machines, I get the following output:
Run brief starting
just before job.get()

However, the code never successfully completes (doesn't print the "Run brief completed" line). (It does complete on a third Windows box).
On the Ubuntu machines, if I delete the line with the comment "#DELETE THIS LINE TO MAKE IT WORK" the execution DOES complete, printing the final line as expected. Similarly, if I leave the line defining temp_val in but delete the line with the comment "This is the line that blocks" it will also complete. Moreover, if I reduce the size of the temp_val tensor (say from 400000 to 4000) it will also complete successfully. Finally, it is worth noting that while I can reproduce this behaviour on two different Ubuntu machines, this code does actually complete on my Windows machine - though, as far as I can tell, the versions of key packages, such as torch, are the same.
I don't understand this behaviour. I suspect it is something to do with the way torch allocates memory or stores information. I've tried calling del temp_val to free up memory, but that doesn't seem to fix things. It seems to me that the async call to t.tensor within really_simple_func is stopped from completing if there has already been a call to t.tensor in the main code block, creating a sufficiently large tensor.
I don't understand why this is happening, or even if that is the correct explanation. In any case, what would be best practice if I do need to do some tensor processing within apply_async as well as in the main thread? More generally, what is Torch waiting on when I make a call to t.tensor?
(Obviously, this is just the simplest version of the real code I'm trying to get to work that reproduced this issue. I realise that calling mp.Pool with only one process doesn't really make sense...nor, indeed, does using apply_async to call a function that returns a constant!)

Output:
Unfortunately, I cannot provide any answers to your questions.
I can, however, share experiences with seemingly the same issue. I use a Linux machine with torch 1.8.1 and numpy 1.19.2.
When I run the following code on my machine:
with Pool(max_pool) as p:
    pool_outputs = list(
        tqdm(
            p.imap(lambda f: get_model_results_per_query_file(get_preds, tokenizer, f), query_files),
            total=len(query_files)
        )
    )

For which the function get_model_results_per_query_file contains operations similar to the following:
feats = features.unsqueeze(0).repeat(batch_size, 1, 1).to(device)

(features is a torch tensor)
The first round of jobs automatically fail, and new ones are immediately started (that do not fail for some reason). The whole process never completes though, since the main process still seems to be waiting for the first failed jobs.
If I remove the lines in my code involving the repeat function, no jobs fail.
I managed to solve my issue and preserve the same results by adapting a similar solution to yours:
feats = torch.as_tensor(np.tile(features, (batch_size, 1, 1))).to(device)

I believe as_tensor works in a similar fashion to from_numpy in this case.
I only managed to find this solution thanks to your post and your proposed workaround, so thank you!


Instruction: the same pretrained model with same input , running multiple times gives different outputs

Input:
I load a pretrained Resnet152 from torchvision. I evaluate the model multiple times with the same input image, but each time the output is different. It's very strange. Anyone knows what is the reason? My code is
from torchvision import transforms
import torch
from torchvision import models
from PIL import Image

# load a pretrained model
model = models.resnet152(pretrained=True)

# load a image and preprocess it
preprocessor = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(
 mean=[0.485, 0.456, 0.406],
 std=[0.229, 0.224, 0.225]
 )])

input_image = Image.open('lion2.jpg')
input_tensor = preprocessor(input_image)
input_batch = torch.unsqueeze(input_tensor, 0)

# run multiple times and print output
for k in range(5):
    model.train()
    out = model(input_batch)

    model.eval()
    out2 = model(input_batch)

    print(out2[0][:10].cpu().detach())

The output are

tensor([ 0.4722, -2.1463, -0.5993, -0.3880, -2.6292,  1.9123, -1.7939, -0.3289,
        -0.3189,  0.5306])
tensor([ 0.4407, -2.0370, -0.7397, -0.4447, -2.6059,  1.9052, -1.9715, -0.6495,
        -0.5361,  0.2618])
tensor([ 0.3874, -1.9249, -0.8254, -0.5408, -2.5266,  1.8302, -2.1151, -0.8739,
        -0.7206,  0.0478])
tensor([ 0.3150, -1.8490, -0.9004, -0.6544, -2.4615,  1.7409, -2.2083, -1.0194,
        -0.8352, -0.1017])
tensor([ 0.2310, -1.7754, -0.8858, -0.7081, -2.3238,  1.5943, -2.2625, -1.1185,
        -0.9551, -0.2954])

(If I remove either model.train() or model.eval(), the output keep constant. )

Output:
The model torchvision.models.resnet152 contains batch normalization layers with track_running_stats set to True. This means that whenever the model is called in training mode (i.e., when model.train() is set), the running_mean and running_var parameters of such batch normalization layers get updated to include the data of the batch passed in that call.
In your example, each time you call the model under model.train() in the loop, this causes the running_mean and running_var parameters of all the batch normalization layers to be updated. These are then freezed at the updated values when you call model.eval(), and used in the second forward pass, which causes the outputs to be different.
Since you are passing exactly the same inputs every time in the loop, it implies that the running_mean and running_var will converge to a constant after a large number of iterations. You can check that as a result the outputs in eval() mode eventually become identical.
The standard way of evaluating networks typically involves calling model.eval() once and using it over the entire test set, and so does not exhibit this apparent discrepancy.


Instruction: Detect Apple silicon GPU core count

Input:
Similar to this question, I am interested in detecting the exact GPU inside a Mac equipped with Apple silicon.
I am interested in knowing the exact GPU core count.
sysctl -a | grep gpu

or
sysctl -a | grep core

does not seem to provide anything useful.

Output:
You can use ioreg like this:
 ioreg -l | grep gpu-core-count

You can also look up an object with class that's named something like AGXAcceleratorG13X and see all of its properties, gpu-core-count will also be there.


Instruction: How to not break differentiability with a model's output?

Input:
I have an autoregressive language model in Pytorch that generates text, which is a collection of sentences, given one input:
output_text = ["sentence_1. sentence_2. sentence_3. sentence_4."]

Note that the output of the language model is in the form of logits (probability over the vocabulary), which can be converted to token IDS or strings.
Some of these sentences need to go into another model to get a loss that should affect only those sentences:
loss1 = model2("sentence_2")
loss2 = model2("sentence_4")
loss_total = loss1+loss2

What is the correct way to break/split the generated text from the first model without breaking differentiability? That is, so the corresponding text (from above) will look like a pytorch tensor of tensors (in order to then use some of them in the next model):
"[["sentence_1."]
["sentence_2."]
["sentence_3."]
["sentence_4."]]

For example, Python's split(".") method will most likely break differentiability, but will allow me to take each individual sentence and insert it into the second model to get a loss.

Output:
Okay solved it. Posting answer for completion.
Since the output is in the form of logits, I can take the argmax to get the indices of each token. This should allow me to know where each period is (to know where the end of the sentence is). I can then split the sentences in the following way to maintain the gradients:
sentences_list = []
r = torch.rand(50) #imagine that this is the output logits (though instead of a tensor of values it will be a tensor of tensors)
period_indices = [10,30,49]
sentences_list.append(r[0:10])
sentences_list.append(r[10:30])
sentences_list.append(r[30:])

Now each element in sentences_list is a sentence, that I can send to another model to get a loss


Instruction: How to write many and similar python scripts at once?

Input:
I need to have 100 of those similar python scripts that have MyData class from MyData_1 to MyData_100.
import torch
import numpy as np
from torch_geometric.data import InMemoryDataset, Data
from torch_geometric.utils import to_undirected

class MyData_1(InMemoryDataset):
    def __init__(self, root, transform=None):
        super(MyData_1, self).__init__(root, transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return "mydata_1.npz"

    @property
    def processed_file_names(self):
        return "data_1.pt"

    def process(self):
        raw_data = np.load(self.raw_paths[0])
        cluster_data = torch.load('./raw/total_clusters.pt')
        x = torch.from_numpy(raw_data['x'])
        y = torch.from_numpy(raw_data['y'])

        pos = torch.stack([x,y], dim=-1)
        cp = torch.from_numpy(raw_data['cp'])

        data_list = []
        for i in range(cp.size(0)):
            data = Data(x=cp[i].view(-1,1),pos=pos.view(-1,2),  cluster=cluster_data[0])
            data_list.append(data)

        torch.save(self.collate(data_list), self.processed_paths[0])

I'm trying to do this because each MyData class calls different mydata_1,2,...100.npz to generate dataset.
Is there any way to make this fast?
Thanks in advance!

Output:
I didn't fully understand the reason why you need to create 100 different classes.
Is it because you need to return mydata_1.npz to mydata_100.npz? If then, You can create a single class like this:
class Myclass:
  def __init__(self, index):
    self.index = index

  def raw_file_names(self):
    return "mydata_{}.npz".format(self.index)

Then, at another script like main.py, you can create/assign it like:
for i in range(100):
  exec('dataset_{} = MyData_{}({})'.format(i, i, i))

I believe you can build your own code that fits your problem with above examples.


Instruction: How to free all GPU memory from pytorch.load?

Input:
This code fills some GPU memory and doesn't let it go:
def checkpoint_mem(model_name):
    checkpoint = torch.load(model_name)
    del checkpoint
    torch.cuda.empty_cache()

Printing memory with the following code:
print(torch.cuda.memory_reserved(0))
print(torch.cuda.memory_allocated(0))

shows BEFORE running checkpoint_mem:
0
0

and AFTER:
121634816
97332224

This is with torch.__version__ 1.11.0+cu113 on Google colab.
Does torch.load leak memory?  How can I get the GPU memory completely cleared?

Output:
It probably doesn't. Also, it depends on what you call memory leak. In this case, after the program ends all memory should be freed, python has a garbage collector, so it might not happen immediately (your del or after leaving the scope) like it does in C++ or similar languages with RAII.
del

del is called by Python and only removes the reference (same as when the object goes out of scope in your function).
torch.nn.Module does not implement del, hence its reference is simply removed.
All of the elements within torch.nn.Module have their references removed recursively (so for each CUDA torch.Tensor instance their __del__ is called).
del on each tensor is a call to release memory

More about __del__
Caching allocator
Another thing - caching allocator occupies part of the memory so it doesn't have to rival other apps in need of CUDA when you are going to use it.
Also, I assume PyTorch is loaded lazily, hence you get 0 MB used at the very beginning, but AFAIK PyTorch itself, during startup, reserves some part of CUDA memory.
The short story is given here, longer one here in case you didn’t see it already.
Possible experiments

You may try to run time.sleep(5) after your function and measure afterwards.
You can get snapshot of the allocator state via torch.cuda.memory_snapshot to get more info about allocator’s reserved memory and inner workings.
You might set the environment variable PYTORCH_NO_CUDA_MEMORY_CACHING=1 and see whether and if anything changes.

Disclaimer
Not a CUDA expert by any means, so someone with more insight could probably expand (and/or correct) my current understanding as I am sure way more things happen under the hood.


Instruction: Torch neural network does not train

Input:
I have implemented a very simple neural network in the torch framework
def mlp(sizes, activation, output_activation=torch.nn.Identity):
layers = []
for j in range(len(sizes)-1):
    act = activation if j < len(sizes)-1 else output_activation
    layers += [torch.nn.Linear(sizes[j], sizes[j+1]), act()]
return torch.nn.Sequential(*layers)

In order to train a network to make regression on the function y=sin(x)
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)

the training code is here
size = [1,20,20,1]
activation = torch.nn.ReLU
model = mlp(size, activation)

optimizer = torch.optim.SGD(model.parameters(), lr=0.002)

n_epoch = 600
mse_loss = torch.nn.MSELoss()
X = x.unsqueeze(-1)
for i in range(n_epoch):
    y_pred = model(X)
    step_loss = mse_loss(y_pred, y)
    optimizer.zero_grad()
    step_loss.backward()
    optimizer.step()

Unfortunately, the network only learn an almost constant function $y=0$.
I have already tried many things

Change Hyperparameters of the network
Add mini batches in training
Change the number of epochs and learning rate

But nothing seems to work. The problem is so simple that I think there is an error in the code.

Output:
I am not sure if this is the main cause, but the statement
act = activation if j < len(sizes)-1 else output_activation

appears to be logically incorrect. In the loop, j can take values from 0 to len(sizes)-1, so the condition is always true. This means that your network has a ReLU right at the end, and so can only ever give non-negative outputs. This can be corrected by changing that statement to:
act = activation if j < len(sizes)-2 else output_activation


Instruction: How to apply differential privacy on list of data?

Input:
How to apply differential privacy on a list of data.
OpenMined release a differential privacy project called PyDP 2 years ago.
On the examples provided, they showed how to compute the PyDP on the data by computing some statistical features such as the mean, Max, Median.

Is there a way to apply a differential privacy to the list of dataset, and get the list of data back, without computing any statistical feature yet ?

e.g. input_list = [1.03,2.23,3.058,4.97]

out_put_differential_privacy_list = dp_function(input_list)
out_put_differential_privacy_list
>> [1.01,2.03,3.8,4.04]

How is the noise added to the data (they use laplacian)?
Is the noise added taking into account the whole data set, or is it added considering each single value at a time ?

I couldn't fine the github code for pydp.algorithms.laplacian.
These are the statistical features they showed how to compute.
from pydp.algorithms.laplacian import (
    BoundedSum,
    BoundedMean,
    BoundedStandardDeviation,
    Count,
    Max,
    Min,
    Median,
)

Are they also functions to compute differential privacy percentiles ?
Any other resources will also be welcome.

Output:
Here are my two cents on the question,
The idea of differential privacy is to publish aggregated information of sensitive values only if noise is added to the aggregated info. This will in terms make it infeasible to match sensitive values to their owners, and also make the dataset not highly dependent on any particular data from the dataset.
The way that noise is added to the differential privacy is by injecting Laplace noise into each pieces of data at a time which in terms will add noise to the overall dataset, the essential idea of DP would be the following:
A(D,f) = f(D) + noise

A = some randomised algorithm

This is to ensure that the result each time will be slightly different.

f = sensitivity of a function such that this sensitivity will be used to determine to what degree an individual piece of data will affect the output.
D = the dataset you want to 'mask', the overall thing, in your case it would be the list of numbers.
noise = Laplace noise, i.e. lambda = delta f/epsilon = 1/epsilon

The epsilon value here is sort of indicates the privacy loss on adding/removing an entry from the dataset, i.e. making adjusments to the dataset. The smaller the epsilon is, the less privacy loss on the adjustments made on the dataset, which means better protection for privacy.

And as you can see now, the noise are only dependent on the sentitivyt and epsilon value, and has nothing to do with the underlying dataset.

... they showed how to compute the PyDP on the data by computing some statistical features such as the mean, Max, Median.

lets say for example, say we have a bunch of numbers like you have here, we could first find the max number out of the list, which would be 4.97, then we can just try to draw the eta from Lap(4.7/epsilon). I believe the idea is to sort of anchor the data around some certain statistical feature.
Hope this is somewhat useful :)


Instruction: Adding in the weight parameter for PyTorch's cross-entropy loss causes datatype RuntimeError

Input:
I'm currently using PyTorch to train a neural network. The dataset that I'm using is a binary classification dataset with a large number of 0's.
I decided to try and use the weight parameter of PyTorch's cross-entropy loss. I calculated the weights via sklearn.utils.class_weight.compute_class_weight and got weight values of [0.58479532, 3.44827586].
When I added this class_weights tensor into the weight parameter of my loss (i.e., criterion = nn.CrossEntropyLoss(weight=class_weights), I'm suddenly getting a RuntimeError: expected scalar type Float but found Double. The outputs and labels that I'm feeding into my loss are of types float32 and int64, respectively. The loss was working fine but when I add the weight parameter I'm getting this error. Attempting to cast my data via outputs.float() doesn't seem to work either.
Does anybody know why this error may be occurring and how I might fix this? Thanks.

Output:
You can create a tensor from your weights as follows.
Also, remember to match the devices between the weights and the rest of your tensors.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
weights =  torch.tensor([0.58479532, 3.44827586],dtype=torch.float32).to(device)


Instruction: How to train pytorch model using large data file while using Dataloader?

Input:
I am using pytorch Dataset class and Dataloader to load data. The class and loader looks like the following.
class Dataset(Dataset):
    def __init__(self):

        self.input_and_label   = json.load(open(path_to_large_json_file)) # Large file
        self.dataset_size      = len(self.input_and_label)

    def __getitem__(self, index):
        # Convert to tensor
        input_data    = torch.LongTensor(self.input_and_label[index][0]) # Input X
        label_data    = torch.LongTensor(self.input_and_label[index][1]) # Label y

        return input_data, label_data

    def __len__(self):
        return self.dataset_size

And the iterator is generated like,
train_loader = torch.utils.data.DataLoader(
    Dataset(),
    # Batch size
    batch_size = 8, # This is expected to be large, 8 is for trial -- didn't work
    shuffle = True,
    pin_memory = False #True
)

The data-file is a large (json) file. But I am getting memory error as,
<RuntimeError: CUDA out of memory. Tried to allocate... ... ... >

Note:
The large json file content is list of numbers like,

    [0, 1 , 0, 0,..... 4000 numbers]  <-- this is the input_data
    [0, 2, 2, ... 50 numbers ]        <-- this is the label
So, probably batch size 8 (that means 8 such pairs), or 800 ... should not matter much

Can someone please help me, how can I get the iterator without loading the large file at once? Or, any other solution welcome. Thank you very much for your support.

Output:
You get a CUDA OOM error, it is not related to the file itself being large, but single example being large.
JSON file loads correctly to RAM, but 8 examples cannot fit on your GPU (which is often the case for images/videos, especially with high resolution).
Solutions

Use a larger GPU (e.g. cloud provided)
Use a smaller batch size (even of size 1) and use gradient accumulation. It will run slow, but the results will be as good as those of larger batches.


Instruction: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same

Input:
I am trying to run this notebook on Apple M1 (1st gen) running MacOS 12.4,
libs freeze:

>pip3 freeze
anyio @ file:///private/tmp/jupyterlab--anyio-20211211-70040-1yv1wmx/anyio-3.4.0
appnope==0.1.2
argon2-cffi @ file:///private/tmp/jupyterlab--argon2-cffi-20211211-70040-1er07d0/argon2-cffi-21.2.0
argon2-cffi-bindings @ file:///private/tmp/jupyterlab--argon2-cffi-bindings-20211211-70040-o64kwi/argon2-cffi-bindings-21.2.0
asttokens==2.0.5
attrs @ file:///private/tmp/jupyterlab--attrs-20211211-70040-6u3qxt/attrs-21.2.0
Babel==2.9.1
backcall @ file:///private/tmp/jupyterlab--backcall-20211211-70040-acdr42/backcall-0.2.0
beniget==0.4.1
black==21.12b0
bleach==4.1.0
certifi==2022.5.18.1
cffi==1.15.0
charset-normalizer==2.0.12
click==8.0.3
cycler==0.10.0
Cython==0.29.24
debugpy @ file:///private/tmp/jupyterlab--debugpy-20211211-70040-2j9lay/debugpy-1.5.1
decorator==5.1.0
defusedxml @ file:///private/tmp/jupyterlab--defusedxml-20211211-70040-uowur4/defusedxml-0.7.1
entrypoints @ file:///private/tmp/jupyterlab--entrypoints-20211211-70040-1r2y5g4/entrypoints-0.3
et-xmlfile==1.1.0
executing==0.8.2
finnhub-python==2.4.5
gast==0.5.2
GDAL==3.4.0
gensim==4.1.2
graphviz==0.19.1
idna==3.3
imageio==2.13.5
ipykernel==6.6.0
ipython==7.30.1
ipython-genutils==0.2.0
ipywidgets==7.6.5
jedi==0.18.1
Jinja2==3.0.3
joblib==1.1.0
json5==0.9.6
jsonschema @ file:///private/tmp/jupyterlab--jsonschema-20211211-70040-1np642r/jsonschema-4.2.1
jupyter==1.0.0
jupyter-client==7.1.0
jupyter-console==6.4.0
jupyter-core==4.9.1
jupyter-server @ file:///private/tmp/jupyterlab--jupyter-server-20211211-70040-1u7h7vl/jupyter_server-1.13.1
jupyterlab @ file:///private/tmp/jupyterlab-20211211-70040-1ltrjpx/jupyterlab-3.2.5
jupyterlab-pygments==0.1.2
jupyterlab-server @ file:///private/tmp/jupyterlab--jupyterlab-server-20211211-70040-iufjhi/jupyterlab_server-2.8.2
jupyterlab-widgets==1.0.2
kiwisolver==1.3.2
lxml==4.6.3
MarkupSafe==2.0.1
matplotlib==3.4.3
matplotlib-inline==0.1.3
midi @ git+https://github.com/vishnubob/python-midi.git@abb85028c97b433f74621be899a0b399cd100aaa
midi-to-dataframe @ git+https://github.com/TaylorPeer/midi-to-dataframe@35347f787f01a2326234ad278d8c40bee3817f1d
mido==1.2.10
mistune==0.8.4
multitasking==0.0.9
mypy-extensions==0.4.3
nbclassic @ file:///private/tmp/jupyterlab--nbclassic-20211211-70040-1fah2fe/nbclassic-0.3.4
nbclient @ file:///private/tmp/jupyterlab--nbclient-20211211-70040-ptwp5d/nbclient-0.5.9
nbconvert==6.3.0
nbformat==5.1.3
nest-asyncio @ file:///private/tmp/jupyterlab--nest-asyncio-20211211-70040-72pz5e/nest_asyncio-1.5.4
networkx==2.6.3
notebook==6.4.6
numpy==1.23.0rc1
openpyxl==3.0.9
packaging @ file:///private/tmp/jupyterlab--packaging-20211211-70040-1f14ddt/packaging-21.3
pandas==1.4.2
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.1
platformdirs==2.4.1
ply==3.11
prometheus-client==0.12.0
prompt-toolkit @ file:///private/tmp/jupyterlab--prompt-toolkit-20211211-70040-hcpjwc/prompt_toolkit-3.0.24
ptyprocess @ file:///private/tmp/jupyterlab--ptyprocess-20211211-70040-wjbvpa/ptyprocess-0.7.0
pure-eval==0.2.1
pybind11==2.8.0
pycparser==2.21
Pygments==2.10.0
pyparsing==3.0.6
pyrsistent @ file:///private/tmp/jupyterlab--pyrsistent-20211211-70040-1fnadg/pyrsistent-0.18.0
python-dateutil==2.8.2
pythran==0.10.0
pytz==2022.1
PyWavelets==1.2.0
PyYAML==6.0
pyzmq @ file:///private/tmp/jupyterlab--pyzmq-20211211-70040-2xtuon/pyzmq-22.3.0
qtconsole==5.2.2
QtPy==2.0.0
requests==2.27.1
scikit-image==0.19.1
scikit-learn==1.1.dev0
scipy==1.8.1
seaborn==0.11.2
Send2Trash==1.8.0
six==1.16.0
smart-open==5.2.1
sniffio @ file:///private/tmp/jupyterlab--sniffio-20211211-70040-wu3dri/sniffio-1.2.0
squarify==0.4.3
stack-data==0.1.4
terminado @ file:///private/tmp/jupyterlab--terminado-20211211-70040-dw1vl6/terminado-0.12.1
testpath @ file:///private/tmp/jupyterlab--testpath-20211211-70040-895z1/testpath-0.5.0
threadpoolctl==3.0.0
tifffile==2021.11.2
tomli==1.2.3
torch==1.13.0.dev20220528
torchaudio==0.11.0
torchsummary==1.5.1
torchtext==0.10.0
torchvision==0.14.0a0+f0f8a3c
torchviz==0.0.2
tornado==6.1
tqdm==4.62.3
traitlets @ file:///private/tmp/jupyterlab--traitlets-20211211-70040-ru76xv/traitlets-5.1.1
typing_extensions==4.2.0
urllib3==1.26.9
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.2.3
wget==3.2
widgetsnbextension==3.5.2
yfinance==0.1.64

in the code , am setting device = torch.device('mps')
at this line: history = [evaluate(model, valid_dl)] am getting runtime error
Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same
Trace:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.clone():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29

/var/folders/mz/qfpvpvf550s039lrnxg70whh0000gn/T/ipykernel_11483/1143432410.py in evaluate(model, val_loader)
      3 def evaluate(model, val_loader):
      4     model.eval()
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

/var/folders/mz/qfpvpvf550s039lrnxg70whh0000gn/T/ipykernel_11483/1143432410.py in <listcomp>(.0)
      3 def evaluate(model, val_loader):
      4     model.eval()
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

/var/folders/mz/qfpvpvf550s039lrnxg70whh0000gn/T/ipykernel_11483/446280773.py in validation_step(self, batch)
     16     def validation_step(self, batch):
     17         images, labels = batch
---> 18         out = self(images)                   # Generate prediction
     19         loss = F.cross_entropy(out, labels)  # Calculate loss
     20         acc = accuracy(out, labels)          # Calculate accuracy

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

/var/folders/mz/qfpvpvf550s039lrnxg70whh0000gn/T/ipykernel_11483/3789274317.py in forward(self, xb)
     29
     30     def forward(self, xb): # xb is the loaded batch
---> 31         out = self.conv1(xb)
     32         out = self.conv2(out)
     33         out = self.res1(out) + out

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/container.py in forward(self, input)
    137     def forward(self, input):
    138         for module in self:
--> 139             input = module(input)
    140         return input
    141

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1128         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130             return forward_call(*input, **kwargs)
   1131         # Do not call functions when jit is used
   1132         full_backward_hooks, non_full_backward_hooks = [], []

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/conv.py in forward(self, input)
    457
    458     def forward(self, input: Tensor) -> Tensor:
--> 459         return self._conv_forward(input, self.weight, self.bias)
    460
    461 class Conv3d(_ConvNd):

/opt/homebrew/Cellar/jupyterlab/3.2.5/libexec/lib/python3.9/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    453                             weight, bias, self.stride,
    454                             _pair(0), self.dilation, self.groups)
--> 455         return F.conv2d(input, weight, bias, self.stride,
    456                         self.padding, self.dilation, self.groups)
    457

RuntimeError: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same

MPS is still new and am trying to figure out the cause here, any suggestions are welcome,
the code runs fine if torch device is set to CPU - just takes so much time.
Thanks,
Deep Kamal Singh

Output:
My guess is that the model has not been placed onto the MPS device.
If you place your model onto the MPS device (by calling model.to(device)), does your code work?


Instruction: How can I efficiently mask out certain pairs in (2, N) tensor?

Input:
I have a torch tensor edge_index of shape (2, N) that represents edges in a graph. For each (x, y) there is also a (y, x), where x and y are node IDs (ints). During the forward pass of my model I need to mask out certain edges. So, for example, I have:
n1 = [0, 3, 4]   # list of node ids as x
n2 = [1, 2, 1]   # list of node ids as y
edge_index = [[1, 2, 0, 1, 3, 4, 2, 3, 1, 4, 2, 4],  # actual edges as (x, y) and (y, x)
              [2, 1, 1, 0, 4, 3, 3, 2, 4, 1, 4, 2]]

# do something that efficiently removes (x, y) and (y, x) edges as formed by n1 and n2

Final edge_index should look like:
>>> edge_index
[[1, 2, 3, 4, 2, 4],
 [2, 1, 4, 3, 4, 2]]

Preferably we need to efficiently make some kind of boolean mask that I can apply to edge index e.g. as edge_index[:, mask] or something like that.
Could also be done in numpy but I'd like to avoid converting back and forth.
Edit #1:
If that can't be done, then I can think of a way so that, instead of n1 and n2, I have access to the indices of the positions I need to exclude in one tensor e.g. _except=[2, 3, 6, 7, 8, 9] (by making a dict/index once in the beginning).
Is there a way to get the desired result by "telling" edge_index to drop the indices in except? edge_index[:, _except] gives me the ones I want to get rid of. I need its complement operation.
Edit #2:
I managed to do it like this:
mask = torch.ones(edge_index.shape[1], dtype=torch.bool)
for i in range(len(n1)):
    mask = mask &  ~(torch.tensor([n1[i], n2[i]], dtype=torch.long) == edge_index.T).all(dim=1) & ~(torch.tensor([n2[i], n1[i]], dtype=torch.long) == edge_index.T).all(dim=1)

edge_index[:, mask]

but it is too slow and I can't use it. How can I speed it up?
Edit #3: I managed to solve this Edit#1 efficiently with:
mask = torch.ones(edge_index.shape[1], dtype=torch.bool)
mask[_except] = False
edge_index[:, mask]

Still interested in solving the original problem if someone comes up with something...

Output:
If you're ok with the way you suggested at Edit#1,
you get the complement result by:
edge_index[:, [i for i in range(edge_index.shape[1]) if not (i in _except)]]
hope this is fast enough for your requirement.

Edit 1:
from functools import reduce

ids = torch.stack([torch.tensor(n1), torch.tensor(n2)], dim=1)
ids = torch.cat([ids, ids[:, [1,0]]], dim=0)
res = edge_index.unsqueeze(0).repeat(6, 1, 1) == ids.unsqueeze(2).repeat(1, 1, 12)
mask = ~reduce(lambda x, y: x | (reduce(lambda p, q: p & q, y)), res, reduce(lambda p, q: p & q, res[0]))
edge_index[:, mask]

Edit 2:
ids = torch.stack([torch.tensor(n1), torch.tensor(n2)], dim=1)
ids = torch.cat([ids, ids[:, [1,0]]], dim=0)
res = edge_index.unsqueeze(0).repeat(6, 1, 1) == ids.unsqueeze(2).repeat(1, 1, 12)
mask = ~(res.sum(1) // 2).sum(0).bool()
edge_index[:, mask]


Instruction: Difference between sum and mean in .backward()

Input:
I know we are converting the tensor in scaler than applying backward(), but when to sum and when to mean?
some_loss_function.sum().backward()
-OR-
some_loss_function.mean().backward()

Output:
There is no canonical answer to your question. Essentially what you're asking is should I average or sum my loss, as readers we have no knowledge of your problem and what this loss function corresponds to. It all depends on your use case.
Generally though, you would average over summation because you often don't wish the loss value to scale with the dimensionality of the output. Indeed high dimensionality of your output would lead to a higher value to your loss than a summation which is meant to be constant w.r.t. the dimensions of your output tensor. If you sum your loss you will end up scaling your loss value and the gradients that are inferred from it uncontrollably.


Instruction: Difference between forward and train_step in Pytorch Lightning?

Input:
I have a transfer learning Resnet set up in Pytorch Lightning. the structure is borrowed from this wandb tutorial  https://wandb.ai/wandb/wandb-lightning/reports/Image-Classification-using-PyTorch-Lightning--VmlldzoyODk1NzY
and from looking at the documentation https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html
I am confused about the difference between the def forward () and the def training_step() methods.
Initially in the PL documentation, the model is not called in the training step, only in forward. But forward is also not called in the training step. I have been running the model on data and the outputs look sensible (I have an image callback and I can see that the model is learning, and getting a good accuracy result at the end). But I am worried that given the forward method is not being called, the model is somehow not being implemented?
Model code is:
class TransferLearning(pl.LightningModule):
    "Works for Resnet at the moment"
    def __init__(self, model, learning_rate, optimiser = 'Adam', weights = [ 1/2288  , 1/1500], av_type = 'macro' ):
        super().__init__()
        self.class_weights = torch.FloatTensor(weights)
        self.optimiser = optimiser
        self.thresh  =  0.5
        self.save_hyperparameters()
        self.learning_rate = learning_rate

        #add metrics for tracking
        self.accuracy = Accuracy()
        self.loss= nn.CrossEntropyLoss()
        self.recall = Recall(num_classes=2, threshold=self.thresh, average = av_type)
        self.prec = Precision( num_classes=2, average = av_type )
        self.jacq_ind = JaccardIndex(num_classes=2)

        # init model
        backbone = model
        num_filters = backbone.fc.in_features
        layers = list(backbone.children())[:-1]
        self.feature_extractor = nn.Sequential(*layers)

        # use the pretrained model to classify damage 2 classes
        num_target_classes = 2
        self.classifier = nn.Linear(num_filters, num_target_classes)

    def forward(self, x):
        self.feature_extractor.eval()
        with torch.no_grad():
            representations = self.feature_extractor(x).flatten(1)
        x = self.classifier(representations)
        return x

    def training_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = self.loss(logits, y)

        # training metrics
        preds = torch.argmax(logits, dim=1)
        acc = self.accuracy(preds, y)
        recall = self.recall(preds, y)
        precision = self.prec(preds, y)
        jac = self.jacq_ind(preds, y)

        self.log('train_loss', loss, on_step=True, on_epoch=True, logger=True)
        self.log('train_acc', acc, on_step=True, on_epoch=True, logger=True)
        self.log('train_recall', recall, on_step=True, on_epoch=True, logger=True)
        self.log('train_precision', precision, on_step=True, on_epoch=True, logger=True)
        self.log('train_jacc', jac, on_step=True, on_epoch=True, logger=True)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = self.loss(logits, y)

        # validation metrics
        preds = torch.argmax(logits, dim=1)
        acc = self.accuracy(preds, y)
        recall = self.recall(preds, y)
        precision = self.prec(preds, y)
        jac = self.jacq_ind(preds, y)

        self.log('val_loss', loss, prog_bar=True)
        self.log('val_acc', acc, prog_bar=True)
        self.log('val_recall', recall, prog_bar=True)
        self.log('val_precision', precision, prog_bar=True)
        self.log('val_jacc', jac, prog_bar=True)

        return loss

    def test_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        loss = self.loss(logits, y)

        # validation metrics
        preds = torch.argmax(logits, dim=1)
        acc = self.accuracy(preds, y)
        recall = self.recall(preds, y)
        precision = self.prec(preds, y)
        jac = self.jacq_ind(preds, y)

        self.log('test_loss', loss, prog_bar=True)
        self.log('test_acc', acc, prog_bar=True)
        self.log('test_recall', recall, prog_bar=True)
        self.log('test_precision', precision, prog_bar=True)
        self.log('test_jacc', jac, prog_bar=True)

        return loss

    def configure_optimizers(self,):
        print('Optimise with {}'.format(self.optimiser) )
        # optimizer = self.optimiser_dict[self.optimiser](self.parameters(), lr=self.learning_rate)

                # Support Adam, SGD, RMSPRop and Adagrad as optimizers.
        if self.optimiser == "Adam":
            optimiser = optim.AdamW(self.parameters(), lr = self.learning_rate)
        elif self.optimiser == "SGD":
            optimiser = optim.SGD(self.parameters(), lr = self.learning_rate)
        elif self.optimiser == "Adagrad":
            optimiser = optim.Adagrad(self.parameters(), lr = self.learning_rate)
        elif self.optimiser == "RMSProp":
            optimiser = optim.RMSprop(self.parameters(), lr = self.learning_rate)
        else:
            assert False, f"Unknown optimizer: \"{self.optimiser}\""

        return optimiser

Output:
I am confused about the difference between the def forward () and the
def training_step() methods.

Quoting from the docs:
"In Lightning we suggest separating training from inference. The training_step defines the full training loop. We encourage users to use the forward to define inference actions."
So forward() defines your prediction/inference actions. It doesn't even need to be part of your training_step in which you would define you whole training loop. Nonetheless you can choose to have it in your training_step if you want it that way. An example where forward() isn't part of the training_step:
    def forward(self, x):
        # in lightning, forward defines the prediction/inference actions
        embedding = self.encoder(x)
        return embedding

    def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # in this case it is independent of forward
        x, y = batch
        x = x.view(x.size(0), -1)
        z = self.encoder(x)
        x_hat = self.decoder(z)
        loss = F.mse_loss(x_hat, x)
        # Logging to TensorBoard by default
        self.log("train_loss", loss)
        return loss

the model is not called in the training step, only in forward. But
forward is also not called in the training step

The fact that forward() is not called in your train_step is because self(x) does it for you. You can alternatively call forward() explicitly instead of using call(x).

I am worried that given the forward method is not being called, the
model is somehow not being implemented?

As long as you see your metrics logged with self.log move in the right direction you will know that you model gets called correctly and its learning.