Instruction: How to strip a pretrained network and add some layers to it using pytorch lightning?

Input:
I am trying to use transfer learning for an image segmentation task, and my plan is to use the first few layers of a pretrained model (VGG16 for example) as an encoder and then will add my own decoder.
So, I can load the model and see the structure by printing it:
model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True)
print(model)

I get like this:
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  .....
  .....
  .....

I can also access the specific layers with model.layer3 for example. Now, I am struggling with certain things.

How to cut the model and take every module from the beginning to the end of any layer (model.layer3 for example)?
How to freeze only this stripped part, and keep the newly added modules available for training?

Output:
For 1): Initialize the ResNet in your LightningModule and slice it until the part that you need. Then add your own head after that, and define forward in the order that you need. See this example, based on the transfer learning docs:
import torchvision.models as models

class ImagenetTransferLearning(LightningModule):
    def __init__(self):
        super().__init__()

        # init a pretrained resnet
        backbone_tmp = models.resnet50(pretrained=True)
        num_filters = backbone_tmp.fc.in_features
        layers = list(backbone_tmp.children())[:-1]
        self.backbone = nn.Sequential(*layers)

        # use the pretrained model to classify cifar-10 (10 image classes)
        num_target_classes = 10
        self.classifier = nn.Linear(num_filters, num_target_classes)

For 2): Pass a  BackboneFinetuning callback to your trainer. This requires that your LightningModule has a self.backbone attribute containing the modules that you want to be frozen, as shown on the snippet above. You can also use the BaseFinetuning callback if you need different freeze-unfreeze behavior.
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import BackboneFinetuning
multiplicative = lambda epoch: 1.5
backbone_finetuning = BackboneFinetuning(200, multiplicative)
trainer = Trainer(callbacks=[backbone_finetuning])


Instruction: Unable to load weights from pytorch checkpoint after splitting pytorch_model.bin into chunks

Input:
I need to transfer a pytorch_model.bin of a pretrained deeppavlov ruBERT model but I have a file size limit. So I split it into chunks using python, transferred and reassembled in the correct order. However, the size of the file increased, and when I tried to load the resulting file using BertModel.from_pretrained(pytorch_model.bin) I received an error:
During handling of the above exception, another exception occurred:
OSError: Unable to load weights from pytorch checkpoint <...>

So my question is: is it actually possible to split the file like that? I could possibly have a mistake in the way I split and reassemble the file. However, this could also be some version mismatch.
My python code to get chunks:
chunk_size = 40000000
file_num = 1
with open("pytorch_model.bin", "rb") as f:
    chunk = f.read(chunk_size)
    while chunk:
        with open("chunk_" + str(file_num), "wb") as chunk_file:
            chunk_file.write(chunk)
        file_num += 1
        chunk = f.read(chunk_size)

Code to reassemble one file:
chunks = !ls | grep chunk_
chunks = sorted(chunks, key=lambda x: int(x.split("_")[-1]))

for chunk in chunks:
    with open(chunk, "rb") as f:
        contents = f.read()
    if chunk == chunks[0]:
        write_mode = "wb"
    else:
        write_mode = "ab"
    with open("pytorch_model.bin", write_mode) as f:
        f.write(contents)

python 3.7.0, torch 1.5.1, transformers 4.2.2. I have no way to move files bigger than 40 MB.
TIA for your help!

Output:
Those who are new to this issue I just figured it out and save your time
What is this error about?
==> When you run the model for the first time it downloads some files { pytorch_model.bin } and if your internet is broken accidentally between processes it will continue running the pipeline file without completely downloading that pytorch_model.bin file so it will raise this issue.
Steps :
1 ] Go to C:// Users / UserName / .cache
2 ] Delete .cache folder
3 ] And Done Just Run The Model Once Again......


Instruction: BERT embeddings in batches

Input:
I am following this post to extract embeddings for sentences and for a single sentence the steps are described as follows:
    text = "After stealing money from the bank vault, the bank robber was seen " \
           "fishing on the Mississippi river bank."

    # Add the special tokens.
    marked_text = "[CLS] " + text + " [SEP]"

    # Split the sentence into tokens.
    tokenized_text = tokenizer.tokenize(marked_text)

    # Mark each of the 22 tokens as belonging to sentence "1".
    segments_ids = [1] * len(tokenized_text)

    # Convert inputs to PyTorch tensors
    tokens_tensor = torch.tensor([indexed_tokens])
    segments_tensors = torch.tensor([segments_ids])

    # Load pre-trained model (weights)
    model = BertModel.from_pretrained('bert-base-uncased',
                                 output_hidden_states = True,
                                      )

    # Put the model in "evaluation" mode, meaning feed-forward operation.
    model.eval()

    with torch.no_grad():
        outputs = model(tokens_tensor, segments_tensors)
        hidden_states = outputs[2]

And I want to do this for a batch of sequences. Here is my example code:
seql = ['this is an example', 'today was sunny and', 'today was']
encoded = [tokenizer.encode(seq, max_length=5, pad_to_max_length=True) for seq in seql]

encoded
[[2, 2511, 1840, 3251, 3],
 [2, 1663, 2541, 1957, 3],
 [2, 1663, 2541, 3, 0]]

But since I'm working with batches, sequences need to have same length. So I introduce a padding token (3rd sentence) which confuses me about several points:

What should the segment id for pad_token (0) will be?

Should I use attention masking when feeding the tensors to the model so that padding is ignored? In the example only token and segment tensors are used.
outputs = model(tokens_tensor, segments_tensors)

If I don't work with batches but with individual sentences, then I might not need a padding token. Would it be better to do that compared to batches?

Output:
You could do all the work you need using one function ( padding,truncation)

encode_plus

check the parameters: the docs
The same you could do with a list of sequences

batch_encode_plus

docs


Instruction: torch matmul two matrix row by row

Input:
I want to find a decent way to write the below function in torch. Appreciate for clean solution to complete this.
import torch
a=torch.randn(3,100)
b=torch.randn(3,100)
row_num = a.size()[0] # 3

# Given two matrix with shape (n1,n2)
# I want to have the row-wise `matmul` results which will result in a tensor with size (n1, )

scores = []
for i in range(row_num):
    score_i = a[i,:].matmul(b[i,:])
    scores.append(score_i)
expected_result = torch.tensor(scores)

Output:
The operation you are trying to do is essentially the values of a dot product (matmul, a @ b.T) which lie on its diagonal.
You can get the same using torch.matmul or @ operator between a and b.T and then get the torch.diagonal -
np.diagonal(a @ b.T)

You can also use torch.einsum directly to get the same result -
torch.einsum('ij,ij->i',a,b)


Instruction: How pytorch implements back propagation from the output layer to the input layer

Input:
I am having difficulty implementing the following functions.
Assuming that we have trained a network model, I want to backpropagate from the output layer to the input layer (not the first layer) to obtain a new input data. I want to know if there is a function in pytorch or other existing functions that can achieve this function, I did not find the relevant function in the pytorch tutorials.

Output:
If you want the gradient w.r.t to the input, you can simply get it from the .grad:
x.requires_grad_(True)  # explicitly ask pytorch to estimate the gradient w.r.t x
# forward pass:
pred = model(x)  # make a prediction
loss = criterion(pred, y)  # compute the loss
# backward pass - compute gradients:
loss.bacward()

# now you have access to the gradient of loss w.r.t the input:
x_grad = x.grad

if you are interested in inspecting gradients of specific layers, you'll need to use hooks.


Instruction: How to convert PyTorch tensor to C++ torch::Tensor vice versa?

Input:
I want to receive a dictionary includes PyTorch Tensor in C++ module using pybind11, and return the result dictionary with some modification that includes C++ torch::Tensor back. As far as I was looking for, there seems no clear way to convert PyTorch Tensor to C++ Tensor, and C++ Tensor to PyTorch Tensor. For a last trial, I tried to convert PyObject to torch::Tensor but seems not working as well. (https://discuss.pytorch.org/t/is-it-possible-to-get-pyobject-from-a-torch-tensor/85980/2) I want to know if it is correct and there are there any workarounds. I share my code snippet on the below.
py::dict quantize(py::dict target) {
    ...
    for (auto item: target) {
        py::str key(item.first);
        torch::Tensor test = item.second.ptr(); // it fails to compile
    }
    ...
    return py::dict("name"_a="test", "tensor"_a=torch::rand({3, 3, 3})); // it fails on runtime
}

Output:
PyObject * THPVariable_Wrap(at::Tensor t);

at::Tensor& THPVariable_Unpack(PyObject* obj);

Those two are what you are looking for i guess.


Instruction: NameError: name 'utils' is not defined in Pytorch

Input:
I have pytorch 1.7. The following code is same as from Pytorch's tutorial page for Object detection and finetuning.
But I have error for the following line
data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4, collate_fn=utils.collate_fn)

as NameError: name 'utils' is not defined
What could be wrong?
The whole code is as follows.
import os
import numpy as np
import torch
from PIL import Image

class PrepareDataset(object):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "images"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "masks"))))
        self.annotations = list(sorted(os.listdir(os.path.join(root, "annotations"))))

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        mask_path = os.path.join(self.root, "masks", self.masks[idx])
        annotation_path = os.path.join(self.root, "annotations", self.annotations[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)
        # convert the PIL Image into a numpy array
        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        # convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

# load a model pre-trained pre-trained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

# replace the classifier with a new one, that has
# num_classes which is user-defined
num_classes = 2  # 1 class (person) + background
# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

import torchvision
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator

# load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features
# FasterRCNN needs to know the number of
# output channels in a backbone. For mobilenet_v2, it's 1280
# so we need to add it here
backbone.out_channels = 1280

# let's make the RPN generate 5 x 3 anchors per spatial
# location, with 5 different sizes and 3 different aspect
# ratios. We have a Tuple[Tuple[int]] because each feature
# map could potentially have different sizes and
# aspect ratios
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),), aspect_ratios=((0.5, 1.0, 2.0),))

# let's define what are the feature maps that we will
# use to perform the region of interest cropping, as well as
# the size of the crop after rescaling.
# if your backbone returns a Tensor, featmap_names is expected to
# be [0]. More generally, the backbone should return an
# OrderedDict[Tensor], and in featmap_names you can choose which
# feature maps to use.
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0], output_size=7, sampling_ratio=2)

# put the pieces together inside a FasterRCNN model
model = FasterRCNN(backbone, num_classes=5, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

    return model

from torchvision import transforms as T

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
dataset = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=True))
data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4, collate_fn=utils.collate_fn)
# For Training
images,targets = next(iter(data_loader))
images = list(image for image in images)
targets = [{k: v for k, v in t.items()} for t in targets]
output = model(images,targets)   # Returns losses and detections
# For inference
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)           # Returns predictions

from engine import train_one_epoch, evaluate
import utils

def main():
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    # our dataset has two classes only - background and person
    num_classes = 2
    # use our dataset and defined transformations
    dataset = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=True))
    dataset_test = PrepareDataset('/home/centos/atic-nyan/Traffic', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = get_model_instance_segmentation(num_classes)

    # move model to the right device
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    print("That's it!")

Output:
I just put
def collate_fn(batch):
    data_list, label_list = [], []
    for _data, _label in batch:
        data_list.append(_data)
        label_list.append(_label)
    return torch.Tensor(data_list), torch.LongTensor(label_list)

in my code and it works.


Instruction: take the output from a specific layer in pytorch

Input:
I have implemented an autoencoder in Pytorch and wish to extract the representations (output) from a specified encoding layer. This setup is similar to making predictions using sub-models that we used to have in Keras.
However, implementing something similar in Pytorch looks a bit challenging. I tried forward hooks as explained in How to get the output from a specific layer from a PyTorch model? and https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html but to no avail.
Could you help me getting outputs from a specific layer?
I have attached my code below:
class Autoencoder(torch.nn.Module):

    # Now defining the encoding and decoding layers.

    def __init__(self):
        super().__init__()
        self.enc1 = torch.nn.Linear(in_features = 784, out_features = 256)
        self.enc2 = torch.nn.Linear(in_features = 256, out_features = 128)
        self.enc3 = torch.nn.Linear(in_features = 128, out_features = 64)
        self.enc4 = torch.nn.Linear(in_features = 64, out_features = 32)
        self.enc5 = torch.nn.Linear(in_features = 32, out_features = 16)
        self.dec1 = torch.nn.Linear(in_features = 16, out_features = 32)
        self.dec2 = torch.nn.Linear(in_features = 32, out_features = 64)
        self.dec3 = torch.nn.Linear(in_features = 64, out_features = 128)
        self.dec4 = torch.nn.Linear(in_features = 128, out_features = 256)
        self.dec5 = torch.nn.Linear(in_features = 256, out_features = 784)

    # Now defining the forward propagation step

    def forward(self,x):
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))
        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))

        return x

autoencoder_network = Autoencoder()

I have to take the output from encoder layers marked enc1, enc2 .., enc5.

Output:
The simplest way is to explicitly return the activations you need:
    def forward(self,x):
        e1 = F.relu(self.enc1(x))
        e2 = F.relu(self.enc2(e1))
        e3 = F.relu(self.enc3(e2))
        e4 = F.relu(self.enc4(e3))
        e5 = F.relu(self.enc5(e4))
        x = F.relu(self.dec1(e5))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        x = F.relu(self.dec5(x))

        return x, e1, e2, e3, e4, e5


Instruction: How to create a submodel from a pretrained model in pytorch without having to rewrite the whole architecture?

Input:
So, I have been working on neural style transfer in Pytorch, but I'm stuck at the point where we have to run the input image through limited number of layers and minimize the style loss. Long story short, I want to find a way in Pytorch to evaluate the input at different layers of the architecture(I'm using vgg16). I have seen this problem solved very simply in keras, but I wanted to see if there is a similar way in pytorch as well or not.
from keras.applications.vgg16 import VGG16

model = VGG16()
model = Model(inputs=model.inputs, outputs=model.layers[1].output)

Output:
Of course you can do that:
import torch
import torchvision

pretrained = torchvision.models.vgg16(pretrained=True)
features = pretrained.features

# First 4 layers
model = torch.nn.Sequential(*[features[i] for i in range(4)])

You can always print your model and see how it's structured. If it is torch.nn.Sequential (or part of it is, as above), you can always use this approach.


Instruction: ValueError: Target size (torch.Size([10, 1])) must be the same as input size (torch.Size([10, 2]))

Input:
A binary classification problem with Batch Size = 10. Trying to use torch.nn.BCEWithLogitsLoss().
~\Anaconda3\envs\notebook\lib\site-packages\torch\nn\functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2578
   2579     if not (target.size() == input.size()):
-> 2580         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2581
   2582     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([1, 10])) must be the same as input size (torch.Size([10, 2]))

Here is my training code:
def train(epochs):
    print('Starting training..')
    for e in range(0, epochs):
        exp_lr_scheduler.step()
        print('='*20)
        print(f'Starting epoch {e + 1}/{epochs}')
        print('='*20)
        train_loss = 0.
        val_loss = 0.
        resnet18.train() # set model to training phase
        for train_step, (images, labels) in enumerate(dl_train):
            optimizer.zero_grad()
            outputs = resnet18(images)
            outputs = outputs.float()
            loss = loss_fn(outputs, labels.unsqueeze(0))
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if train_step % 20 == 0:
                print('Evaluating at step', train_step)
                accuracy = 0
                resnet18.eval() # set model to eval phase
                for val_step, (images, labels) in enumerate(dl_val):
                    outputs = resnet18(images)
                    outputs = outputs.float()
                    loss = loss_fn(outputs, labels.unsqueeze(0))
                    val_loss += loss.item()
                    _, preds = torch.max(outputs, 1)
                    accuracy += sum((preds == labels).numpy())
                val_loss /= (val_step + 1)
                accuracy = accuracy/len(val_dataset)
                print(f'Validation Loss: {val_loss:.4f}, Accuracy: {accuracy:.4f}')
                show_preds()
                resnet18.train() #set model to training phase
                if accuracy >= 0.95:
                    print('Performance condition satisfied, stopping..')
                    return
        train_loss /= (train_step + 1)
        print(f'Training Loss: {train_loss:.4f}')
    print('Training complete..')**

train(epochs=30)

Output:
Target size (torch.Size([1, 10])) must be the same as input size (torch.Size([10, 2]))

Seems to me you have two issues:

target size (a.k.a. ground truth tensor) should have the batch on the first axis: (1, 10).

From what you've described you are dealing with a binary classification task not a multi-label (2-class) classification task. Therefore input size (a.k.a. model's output) should have a shape of (10, 1).

In a binary classification task you should only have a single logit coming out of your model, i.e. your last nn.Linear layer should have a single neuron. The output will define which class has been predicted. Since you are using nn.BCEWithLogitsLoss, the loss input should be the raw output (since it includes a Sigmoid layer, cf. documentation) and should have a shape matching (batch_size=10, 1). Similarly, the target tensor should have the same shape. Its content would be 0s and 1s in shape (batch_size=10, 1).


Instruction: torch: minimally pad tensor such that num elements divisible by x

Input:
Suppose I have a tensor t of arbitrary ndim
I want to pad (with zeroes) it such that
a) I introduce the fewest possible # elements
b) after padding, (t.numel() % x) == 0
Is there a better algorithm for doing this than
find the largest dimension and increase it by 1 until condition (b) is satisfied?
Maybe working code:
def pad_minimally(t, x):
    largest_dim = np.argmax(t.shape)
    buffer_shape = list(t.shape)
    new_t = t.clone()
    print(t.shape)
    for n_to_add in range(x):
        if new_t.numel() % x == 0:
            break
        buffer_shape[largest_dim] = n_to_add
        new_buffer = torch.zeros(*buffer_shape)
        new_t = torch.cat([t, new_buffer], axis=largest_dim)
    assert new_t.numel() % x == 0
    return new_t
assert pad_minimally(torch.rand(3,1), 7).shape == (7,1)
assert pad_minimally(torch.rand(3,2), 7).shape == (7,2)
assert pad_minimally(torch.rand(3,2, 6), 7).shape == (3,2,7)

Output:
First off, simply adding one to the largest dimension until numel is divisible by x doesn't work in all cases. For example if the shape of t is (3, 2) and x = 9 then we would want to pad t to be (3, 3), not (9, 2).
Even more concerning is that there's no guarantee that only one dimension needs to be padded. For example if t has shape (13, 17, 25) and x = 8 then the optimally padded t would be either (14, 18, 26) or (13, 18, 28).
Distilling this into the mathematics, the problem becomes

Given positive integers s[1], ..., s[D] find positive integers q[1], ..., q[D] that minimize prod(q[i], i=1 to D) subject to the constraints that prod(q[i], i=1 to D) is divisible by x and q[i] >= s[i] for all i=1 to D.

I wasn't able to develop an efficient solution (see update for more efficient solution), though I'm not particularly well versed in non-linear integer programming. Perhaps an efficient solution to this problem exists. If it does I imagine it would involve the prime factors of x and q and/or better memoization. That said, it is possible to solve the problem using an exhaustive search, provided that x and D (i.e. len(t.shape)) are sufficiently small (otherwise the algorithm may run for a really really long time).
The brute force search algorithm I came up with iterates over each multiple of x greater-than or equal to t.numel() and uses depth-first search to see if a padding exists for that multiple. As soon as a valid padding is found the algorithm finishes. The python code for this algorithm is:
import numpy as np

def search(shape, target_numel, memory):
    numel = np.prod(shape)
    if numel == target_numel:
        return True
    elif numel < target_numel:
        for idx in range(len(shape)):
            shape[idx] += 1
            if tuple(shape) not in memory:
                if search(shape, target_numel, memory):
                    return True
                memory.add(tuple(s for s in shape))
            shape[idx] -= 1
    return False

def minimal_shape(shape, target_multiple):
    shape = [s for s in shape]
    target_numel = target_multiple * int(np.ceil(max(1, np.prod(shape)) / target_multiple))
    while not search(shape, target_numel, set()):
        target_numel += target_multiple
    return shape

Once you have the minimal shape, the pad_minimal function can be implemented pretty succinctly as
def pad_minimally(t, x):
    new_shape = minimal_shape(t.shape, x)
    new_t = t.new_zeros(new_shape)
    new_t[[slice(0, s) for s in t.shape]] = t
    return new_t

I'm not sure if this will be fast enough for your needs. Hopefully someone else can come along with a more efficient version.

Some test cases for minimal_shape
assert minimal_shape([2, 2], 9) == [3, 3]
assert minimal_shape([2, 8], 6) == [2, 9]
assert minimal_shape([13, 17, 25], 8) in [[14, 18, 26], [13, 18, 28]]
assert minimal_shape([5, 13, 19], 6) == [5, 14, 21]

Update
I asked about this algorithm on CS.SE. Based on the answer I received there and the subsequent update to the question the following is a much more efficient implementation of minimal_shape.
from functools import reduce
from operator import mul
from copy import deepcopy

def prod(x):
    return reduce(mul, x, 1)

def argsort(x, reverse=False):
    return sorted(range(len(x)), key=lambda idx: x[idx], reverse=reverse)

def divisors(v):
    """ does not include 1 """
    d = {v} if v > 1 else set()
    for n in range(2, int(v**0.5) + 1):
        if v % n == 0:
            d.add(n)
            d.add(v // n)
    return d

def update_memory(b, c_rem, memory):
    tuple_m = tuple(b + [c_rem])
    if tuple_m in memory:
        return False
    memory.add(tuple_m)
    return True

def dfs(a, b, c, c_rem, memory, p_best=float('inf'), b_best=None):
    ab = [ai + bi for ai, bi in zip(a, b)]
    p = prod(ab)
    if p >= p_best:
        return p_best, b_best
    elif p % c == 0:
        return p, deepcopy(b)

    dc = divisors(c_rem)
    for i in argsort(ab):
        for d in dc:
            db = (d - ab[i]) % d
            b[i] += db
            if update_memory(b, c_rem // d, memory):
                p_best, b_best = dfs(a, b, c, c_rem // d, memory, p_best, b_best)
            b[i] -= db

    return p_best, b_best

def minimal_shape(shape, target_multiple):
    a = list(shape)
    b = [0 for _ in range(len(a))]
    c = target_multiple
    _, b = dfs(a, b, c, c, set())
    return [ai + bi for a, b in zip(a, b)]


Instruction: compute accuracy of Band RNN

Input:
So I am trying to figure out how to compute the accuracy of a BandRNN.
BandRnn is a diagonalRNN model with a different number of connections per neuron. For example:

here C is the number of connections per neuron.
My current model training is as follows:
model = ModelLSTM(m, k).to(device)

model.train()

opt = torch.optim.Adam(model.parameters(), lr=args.lr)

best_test = 1e7
best_validation = 1e7

for ep in range(1, args.epochs + 1):

init_time = datetime.now()
processed = 0
step = 1

for batch_idx, (batch_x, batch_y, len_batch) in enumerate(train_loader):
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)

    opt.zero_grad()

    logits = model(batch_x)

    loss = model.loss(logits, batch_y, len_batch)

    acc = sum(logits == batch_y) * 1.0 / len(logits)
    print(acc)

    loss.backward()

    if args.clip > 0:
        nn.utils.clip_grad_norm_(model.parameters(), args.clip)

    opt.step()

    processed += len(batch_x)
    step += 1
    print("   batch_idx {}\tLoss: {:.2f} ".format(batch_idx, loss))

print("Epoch {}, LR {:.5f} \tLoss: {:.2f} ".format(ep, opt.param_groups[0]['lr'], loss))

And my model test is as follows:
model.eval()
with torch.no_grad():

for batch_x, batch_y, len_batch in test_loader:
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
    logits = model(batch_x)
    loss_test = model.loss(logits, batch_y, len_batch)

    acc = sum(logits == batch_y) * 1.0 / len(logits)

for batch_x, batch_y, len_batch in val_loader:
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
    logits = model(batch_x)
    loss_val = model.loss(logits, batch_y, len_batch)

if loss_val < best_validation:
    best_validation = loss_val.item()
    best_test = loss_test.item()

print()
print("Val:  Loss: {:.2f}\tBest: {:.2f}".format(loss_val, best_validation))
print("Test: Loss: {:.2f}\tBest: {:.2f}".format(loss_test, best_test))
print()

model.train()

I am struggling with thinking about a way to compute the accuracy of this model and I would like to receive some suggestions about a way to do so.
Thank you.

Output:
I believe this line in your code is already attempting to calculate accuracy:
acc = sum(logits == batch_y) * 1.0 / len(logits)

Though you probably want to argmax the logits before comparing with the labels:
preds = logits.argmax(dim=-1)
acc = sum(preds == batch_y) * 1.0 / len(logits)


Instruction: ValueError: All bounding boxes should have positive height and width

Input:
Any help solving this will be highly appreciated.
I have an idea why the error is happening, it is because the xmin == xmax and ymin == ymax which should not be. However I seem not to know how this is happening. Here is how I load my custom dataset with pytorch Dataset class.
`class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, root, transforms=None):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "seg_image_use"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "seg_mask_use"))))

    def __getitem__(self, idx):
        # load one image and mask using idx
        img_path = os.path.join(self.root, "seg_image_use", self.imgs[idx])
        mask_path = os.path.join(self.root, "seg_mask_use", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.asarray(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)[1:] # first id is the background, so remove it
        masks = mask == obj_ids[:, None, None]  # split the color-encoded mask into a set of binary masks
        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

       # convert everything into torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        target = {}
        target["boxes"] = boxes
        target["labels"] = torch.as_tensor(obj_ids, dtype=torch.int64) - 1 # corrected by Rawi
        target["masks"] = torch.as_tensor(masks, dtype=torch.uint8) #uint8
        target["image_id"] = torch.tensor([idx])
        target["area"] = area
        target["iscrowd"] = torch.zeros((num_objs,), dtype=torch.int64) # suppose all instances are not crowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)
`
And then when I call it to see the dataset, I get the first index of each dataset showing this; take note of the first index in the 'boxes' tensor. (italic)

`dataset_sample = CustomDataset('C:/Users/LENOVO/Desktop/clothme/Train')
img, target = dataset_sample[2]
print(target)

result: {'boxes': tensor(*[[  0.,   0., 286., 403.]*,
         [ 30., 240.,  52., 241.],
         [ 25., 183.,  31., 204.],
         [ 26., 224.,  34., 240.],
         [ 30., 169.,  88., 181.],
         [ 32., 239.,  85., 251.],
`

Here is the error I get when I try to train the model.
`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-81-c798930961c1> in <module>
      4 for epoch in range(num_epochs):
      5     # train for one epoch, printing every 10 iterations
----> 6     train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
      7     # update the learning rate
      8     lr_scheduler.step()

~\measurement_model_dev\engine.py in train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq)
     28         targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
     29
---> 30         loss_dict = model(images, targets)
     31
     32         losses = sum(loss for loss in loss_dict.values())

~\anaconda3\envs\measurement_py37\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\anaconda3\envs\measurement_py37\lib\site-packages\torchvision\models\detection\generalized_rcnn.py in forward(self, images, targets)
     92                     raise ValueError("All bounding boxes should have positive height and width."
     93                                      " Found invalid box {} for target at index {}."
---> 94                                      .format(degen_bb, target_idx))
     95
     96         features = self.backbone(images.tensors)

ValueError: All bounding boxes should have positive height and width. Found invalid box [790.0323486328125, 359.0328369140625, 790.0323486328125, 359.0328369140625] for target at index 0.
`

Output:
TLDR;you have to check your ground truth first and reassure that any zero_area boxes are discarded..
Imagine that what you provide as a bounding box is a zero_area box.
Considering that the data format is [x1,y1,x2,y2] which actually indicates the [left,top..] and the [..,right,bottom] edge of the ground truth box, in your case these two spots, are actually overlapping.
This is a good example of a valid input , since x1<x2 and y1<y2 in all boxes
result: {'boxes': tensor(*[[  0.,   0., 286., 403.]*,
         [ 30., 240.,  52., 241.],
         [ 25., 183.,  31., 204.],
         [ 26., 224.,  34., 240.],
         [ 30., 169.,  88., 181.],
         [ 32., 239.,  85., 251.],

but the error message says that:
[790.0323486328125, 359.0328369140625, 790.0323486328125, 359.0328369140625]

which indicates that x1=x2 and y1=y2.


Instruction: Supplying weights to nn.functional.conv2d in PyTorch

Input:
I am trying to learn the weights of a 3x3 conv2d layer accepting 3 channels and outputting 3 channels. For this discussion consider bias=0 in each case. However, the weights of the conv layer are learned indirectly. I have a 2 layered Multi layer perception having 9 nodes in first layer and 9 in the second. The weights of the 2d conv layer are then precisely the weights learned using this MLP i.e. nn.Linear(9,9). I understand in this case I will have to use nn.functional.conv2d(input,weight). But how exactly to extract the weights from MLP and use it for convolution is not clear and can think of the following.
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        m=nn.Linear(9,9)
    def forward(self, x):
        # some operations involving MLP `m`
        return nn.Functional.conv2d(x,m.weight)

Can some one provide a short, dummy code in PyTorch to achieve this training configuration allowing backpropagation?

Output:
A convolution from 3 input channels to 3 output channels with kernel_size=3 has 81 weights (and not 9). You can reduce this number to 27 if you use groups=3.
you can do the following:
class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hyper = nn.Linear(9, 9)  # output the required number of parameters

  def forward(self, x):
    # do stuff with self.hyper(x)
    y = nn.Functional.conv2d(x, self.hyper.weight.reshape((3, 3, 3, 3)))  # add padding and other parameters
    return y


Instruction: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! when resuming training

Input:
I saved a checkpoint while training on gpu. After reloading the checkpoint and continue training I get the following error:
Traceback (most recent call last):
  File "main.py", line 140, in <module>
    train(model,optimizer,train_loader,val_loader,criteria=args.criterion,epoch=epoch,batch=batch)
  File "main.py", line 71, in train
    optimizer.step()
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/optim/sgd.py", line 106, in step
    buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

My training code is as follows:
def train(model,optimizer,train_loader,val_loader,criteria,epoch=0,batch=0):
    batch_count = batch
    if criteria == 'l1':
        criterion = L1_imp_Loss()
    elif criteria == 'l2':
        criterion = L2_imp_Loss()
    if args.gpu and torch.cuda.is_available():
        model.cuda()
        criterion = criterion.cuda()

    print(f'{datetime.datetime.now().time().replace(microsecond=0)} Starting to train..')

    while epoch <= args.epochs-1:
        print(f'********{datetime.datetime.now().time().replace(microsecond=0)} Epoch#: {epoch+1} / {args.epochs}')
        model.train()
        interval_loss, total_loss= 0,0
        for i , (input,target) in enumerate(train_loader):
            batch_count += 1
            if args.gpu and torch.cuda.is_available():
                input, target = input.cuda(), target.cuda()
            input, target = input.float(), target.float()
            pred = model(input)
            loss = criterion(pred,target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            ....

The saving process happened after finishing each epoch.
torch.save({'epoch': epoch,'batch':batch_count,'model_state_dict': model.state_dict(),'optimizer_state_dict':
                    optimizer.state_dict(),'loss': total_loss/len(train_loader),'train_set':args.train_set,'val_set':args.val_set,'args':args}, f'{args.weights_dir}/FastDepth_Final.pth')

I can't figure why I get this error.
args.gpu == True, and I'm passing the model, all data, and loss function to cuda, somehow there is still a tensor on cpu, could anyone figure out what's wrong?
Thanks.

Output:
There might be an issue with the device parameters are on:

If you need to move a model to GPU via .cuda() , please do so before constructing optimizers for it. Parameters of a model after .cuda() will be different objects with those before the call.
In general, you should make sure that optimized parameters live in consistent locations when optimizers are constructed and used.


Instruction: looking for an equivalent of Tensorflow normalization layer in Pytorch

Input:
I was using 'tf.keras.layers.experimental.preprocessing.Normalization'. This layer is cool since you can save weights in this layer to normalize any input data to this layer.
However, I couldn't find any normalization layer in Pytorch.
Is there a layer that functions the same role?

Output:
There is no built-in that achieves this is PyTorch. However, you can measure the mean and standard deviation yourself (keeping only the relevant axes), then use torchvision.transform.Normalize with those statistics.
For instance in order to measure mean and std over the channels:
>>> x = torch.rand(16, 3, 10, 10)
>>> mean, std = x.mean((0, 2, 3)), x.std((0, 2, 3))
(tensor(0.4941), tensor(0.2899))

Then initialize a transform:
>>> t = torchvision.transform.Normalize(mean, std)

You can use this function on a new dataset to normalize it based on the initial dataset's statistics:
>>> z_normalized = t(z)


Instruction: AllenNLP DatasetReader: only loads a single instance, instead of iterating over all instances in the training dataset

Input:
I am using AllenNLP to train a hierarchical attention network model. My training dataset consists of a list of JSON objects (eg, each object in the list is a JSON object with keys := ["text", "label"]. The value associated with the text key is a list of lists, eg:
[{"text":[["i", "feel", "sad"], ["not", "sure", "i", "guess", "the", "weather"]], "label":0} ... {"text":[[str]], "label":int}]

My DatasetReader class looks like:
@DatasetReader.register("my_reader")
class TranscriptDataReader(DatasetReader):
    def __init__(self,
                 token_indexers: Optional[Dict[str, TokenIndexer]] = None,
                 lazy: bool = True) -> None:
        super().__init__(lazy)
        self._token_indexers = token_indexers or {'tokens': SingleIdTokenIndexer()}

    def _read(self, file_path: str) -> Iterator[Instance]:
        with open(file_path, 'r') as f:
            data = json.loads(f.read())
            for _,data_json in enumerate(data):
                sent_list = []
                for segment in data_json["text"]:
                    sent_list.append(self.get_text_field(segment))
                yield self.create_instance(sent_list, str(data_json["label"]))

    def get_text_field(self, segment):
        return TextField([Token(token.lower()) for token in segment],self._token_indexers)

    def create_instance(self, sent_list, label):
        label_field = LabelField(label, skip_indexing=False)
        fields = {'tokens': ListField(sent_list), 'label': label_field}
        return Instance(fields)

and in my config file, I have:
{
  dataset_reader: {
    type: 'my_reader',
  },

  train_data_path: 'data/train.json',
  validation_data_path: 'data/dev.json',

 data_loader: {
    batch_sampler: {
      type: 'bucket',
      batch_size: 10
    }
 },

I have tried (alternatively) setting the lazy param for the dataset reader to True and False.

When set to True, the model is able to train, however, I observe that only one train and one dev instance actually get loaded, when my dataset contains ~100.
When set to False, I've modified the yield line in _read to be return; however, this causes a type error in the base vocabulary class. I've also tried keeping the yield as is when set to False; in this case, no instances get loaded at all, and since the set of instances is empty, the vocabulary does not get instantiated, and the embedding class throws an error.

Would appreciate pointers, and/or tips for debugging.

Output: If you are using allennlp>=v2.0.0, the lazy parameter in the DatasetReader constructor is deprecated. Therefore, your super().__init__(lazy) would be instead interpreted as the new constructor parameter max_instances, i.e. max_instances=True which is equivalent to max_instances=1.


Instruction: Storing a dictionary with random indices as keys and simulated values as values in hdf5 possibly using pytorch?

Input:
UPDATED QUESTION:
Each entry in an nd-array (say Sim_nDArray) correspond to a combination of parameters chosen from 8D search space. I have used Sim_nDArray.ravel() to convert it to 1D equivalent. Since I can not search from ~100 million entries, I decided to choose ~1 million random entries. I have corresponding ~1 million simulated values.
I have been able to simulate and save it. However, it seems that I have not been able to load data properly. I am getting error while overloading "len" during the declaration of the object: "dataset".
I am planning to use hdf5 to store and read data. Can someone please guide me how to achieve this?
def add_trace(arrInd, arr):
    """ Add one trace to the dataset, keeping count of the # of traces written """
    global ntraces
    dset1[ntraces, :] = arrInd
    dset2[ntraces, :] = arr
    ntraces += 1

def done():
    """ After all calls to add_trace_2, trim the dataset to size """
    dset1.resize((ntraces, 1000))
    dset2.resize((ntraces, 1000))

import torch
from torch.utils.data import Dataset, DataLoader

class Dataset(torch.utils.data.Dataset):
    # Characterizes a dataset for PyTorch
    def __init__(self, dset1, dset2):
        'Initialization'
        self.dset1 = dset1
        self.dset2 = dset2
        self._data_len = len(dset1)

def __len__(self):
    # Denotes the total number of samples
    return len(self._data_len)

def __getitem__(self, index):
    # Generates one sample of data
    # Select sample
    ID = self.dset1[index]
    SimData = self.dset2[index]
    return ID, SimData

# Running the main.
if __name__ == '__main__':
    import h5py
    import numpy as np
    import timeit

    """ Re-initialize both datasets for the tests """
    global data, N, dset1, dset2, ntraces
    N = 1000
    ################ WRITE #############################################################################################
    ## Creating two datasets
    f = h5py.File("randomDataset2.hdf5", 'w')
    dset1 = f.create_dataset('dataset1', (5000, 1000), maxshape=(None, 1000), dtype="float32", chunks=(1, 1000))
    dset2 = f.create_dataset('dataset2', (5000, 1000), maxshape=(None, 1000),
                             dtype="float32")  # DK: why faster if I do not define chunk

    dset1.resize((10001, 1000))  # Allocating extra space
    dset2.resize((10001, 1000))  # Allocating extra space

    ## TEST 1: Less efficient way of writing to hdf5
    ntraces = 0
    start1 = timeit.default_timer()
    for idx in range(N):
        IndxVec1 = np.random.randint(low=0, high=1000, size=1000);
        DataVec1 = np.random.random(1000)
        add_trace(IndxVec1, DataVec1)
    done()

    # All the program statements
    stop1 = timeit.default_timer()
    execution_time = stop1 - start1
    print("Program Executed in " + str(execution_time))  # It returns time in seconds
    f.close()
    ##################################
    ## READING HDF files
    fr = h5py.File("randomDataset2.hdf5", 'r')
    dset10 = fr['dataset1']
    dset20 = fr['dataset2']
    fr.close()

    # Parameters
    params = {'batch_size': 64, 'shuffle': True, 'num_workers': 6}
    max_epochs = 100

    # Generators
    training_set = Dataset(dset10, dset20)
    training_generator = torch.utils.data.DataLoader(training_set, batch_size= 64, shuffle=True, num_workers= 6)

Error:
(PipInConda_DKU) dushyant20@DESKTOP-U96RKFC:/mnt/c/PyImageSearch/Sim_Write_n_Read$ python3 main.
py
Program Executed in 2.265893899995717
Traceback (most recent call last):
  File "main.py", line 89, in <module>
    training_set = Dataset(dset10, dset20)
  File "main.py", line 30, in __init__
    self._data_len = len(dset1)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/dushyant20/miniconda3/envs/PipInConda_DKU/lib/python3.8/site-packages/h5py/_hl/dat
aset.py", line 447, in __len__
    size = self.len()
  File "/home/dushyant20/miniconda3/envs/PipInConda_DKU/lib/python3.8/site-packages/h5py/_hl/dat
aset.py", line 459, in len
    shape = self.shape
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/dushyant20/miniconda3/envs/PipInConda_DKU/lib/python3.8/site-packages/h5py/_hl/dataset.py", line 286, in shape
    return self.id.shape
  File "h5py/h5d.pyx", line 132, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/h5d.pyx", line 133, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 289, in h5py.h5d.DatasetID.get_space
ValueError: Not a dataset (not a dataset)

Output:
I don't use PyTorch, so can't comment on that (or run the entire code). Observations:

I noticed 2 methods for Class Dataset are not indented properly:
def __len__(self): and def __getitem__(self, index):. I assume
that's an error from cut-n-paste to your SO post...but you should
double check.
I ran your code (after commenting out the PyTorch stuff), and it runs
to completion and creates randomDataset2.hdf5 with 2 1000x1000
datasets. So, the problem is not in HDF5 file creation.
You are passing h5py datasets to your PyTorch generators. However, you close the HDF5 file BEFORE you call them. So, the data isn't available at that point in time. That could be the problem. Also, do your generators expect NumPy arrays or dataset objects? That could also cause a problem (once you fix the file close issue).

Other observations:

Use with / as : context manager when working with files to avoid
open/close issues.
Recommended practice is to put all imports at the top of the file.
Use this call if you want a NumPy array instead of a h5py dataset: arr10 = fr['dataset1'][:]

Modified code to reflect above is shown below.
I don't know if this will solve your problem...but it might get you pointed in the right direction.
import h5py
import numpy as np
import timeit
import torch
from torch.utils.data import Dataset, DataLoader

def add_trace(arrInd, arr):
    """ Add one trace to the dataset, keeping count of the # of traces written """
    global ntraces
    dset1[ntraces, :] = arrInd
    dset2[ntraces, :] = arr
    ntraces += 1

def done():
    """ After all calls to add_trace_2, trim the dataset to size """
    dset1.resize((ntraces, 1000))
    dset2.resize((ntraces, 1000))

class Dataset(torch.utils.data.Dataset):
    # Characterizes a dataset for PyTorch
    def __init__(self, dset1, dset2):
        'Initialization'
        self.dset1 = dset1
        self.dset2 = dset2
        self._data_len = len(dset1)

    def __len__(self):
        # Denotes the total number of samples
        return len(self._data_len)

    def __getitem__(self, index):
        # Generates one sample of data
        # Select sample
        ID = self.dset1[index]
        SimData = self.dset2[index]
        return ID, SimData

# Running the main.
if __name__ == '__main__':

    """ Re-initialize both datasets for the tests """
    global data, N, dset1, dset2, ntraces
    N = 1000
    ################ WRITE #############################################################################################
    ## Creating two datasets
    with h5py.File("randomDataset2.hdf5", 'w') as f:
        dset1 = f.create_dataset('dataset1', (5000, 1000), maxshape=(None, 1000), dtype="float32", chunks=(1, 1000))
        dset2 = f.create_dataset('dataset2', (5000, 1000), maxshape=(None, 1000),
                                 dtype="float32")  # DK: why faster if I do not define chunk

        dset1.resize((10001, 1000))  # Allocating extra space
        dset2.resize((10001, 1000))  # Allocating extra space

        ## TEST 1: Less efficient way of writing to hdf5
        ntraces = 0
        start1 = timeit.default_timer()
        for idx in range(N):
            IndxVec1 = np.random.randint(low=0, high=1000, size=1000);
            DataVec1 = np.random.random(1000)
            add_trace(IndxVec1, DataVec1)
        done()

        # All the program statements
        stop1 = timeit.default_timer()
        execution_time = stop1 - start1
        print("Program Executed in " + str(execution_time))  # It returns time in seconds

    ##################################
    ## READING HDF files
    with h5py.File("randomDataset2.hdf5", 'r') as fr:
        dset10 = fr['dataset1']
        arr10 = fr['dataset1'][:]
        dset20 = fr['dataset2']
        arr20 = fr['dataset2'][:]

    # Parameters
    params = {'batch_size': 64, 'shuffle': True, 'num_workers': 6}
    max_epochs = 100

    # Generators
    training_set = Dataset(dset10, dset20)
    training_generator = torch.utils.data.DataLoader(training_set, batch_size= 64, shuffle=True, num_workers= 6)


Instruction: How to allow complex inputs, and complex weights to a Pytorch model?

Input:
Assume even the simplest model (taken from here)
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        output = F.log_softmax(x, dim=1)
        return output

When feeding complex data to the model,
output = model(data.complex())

it gives

ret = torch.addmm(bias, input, weight.t())

RuntimeError: expected scalar type Float but found ComplexDouble

(I didn't copy the entire stack trace, nor the entire training code, for question simplicity)

doing self.complex() after the model's __init__, as I normally would do self.double(), doesn't work, with
torch.nn.modules.module.ModuleAttributeError: 'Net' object has no attribute 'complex'

How to allow model's weights to be complex?
How to allow complex input to a model?
Which built-in activation functions support this?
Is anything also supported for 1d operations?

EDIT:
In the meantime, I found
this paper. Still reading it.

Output:
As you normally did self.double(), I found self.type(dst_type) from https://pytorch.org/docs/stable/generated/torch.nn.Module.html
In my case, self.type(torch.complex64) is working for me.


Instruction: Reducing size of pytorch library

Input:
I've made a conversation telegram bot with pytorch and I'm trying to host it onto Github. The large pytorch file prevents me from doing so as its too large and I get this error:
remote: error: File env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.dylib is 233.61 MB; this exceeds GitHub's file size limit of 100.00 MB
Is there anyway to reduce the size of the torch file? Or is it possible to find and delete unused dependencies?

Output:
Do not host that file on github.
Make a requirements.txt file and add required versions there, you can even fix version required to run your code.
Whoever downloads it can create a virtual environment (venv) or a docker image and install it as
pip install -r requirements.txt

For example:
requirements.txt
https://download.pytorch.org/whl/cpu/torch-1.5.1%2Bcpu-cp38-cp38-linux_x86_64.whl
transformers==3.5


Instruction: How to use a neural network (Pytorch- or Tensorflow-based) in Fortran?

Input: Python is popular and optimal for neural network development and training. However, many scientific codes are written in the Fortran language. How I can call a trained network in my Fortran program?

Output:
It would not make sense. You are not training the network in Fortran, you are just trying to run the C++ or Python code from Fortran.
You should abstract the training/inference from your Fortran code. You could do the orchestration in Fortran.

Create your model in Python
Expose your model thru an API that you can access from Fortran via an httpRequest.

By doing that, you could expose anything you want to your Fortran app.


Instruction: GPU showing no speed up over CPU

Input:
I'm training a neural network with 100*100 hidden nodes, four inputs/one output, and batch size of 32, and I am seeing no speed improvement in using the GPU vs. CPU. I only have a limited data set (1067 samples, copied all to the GPU at the beginning), but I would have thought the 33 batches could have run in parallel, more than making up for the time in copying to the GPU. Is my data set too small, or is there potentially some other issue? Here is my code snippet:
def train_for_regression(X, T):
    BATCH_SIZE = 32
    n_epochs = 1000
    learning_rate = 0.01
    device = torch.device("cuda:0")
    Xt = torch.from_numpy(X).float().to(device) #Training inputs are 4 * 1067 samples
    Tt = torch.from_numpy(T).float().to(device) #Training outputs are 1 * 1067 samples

    nnet = torch.nn.Sequential(torch.nn.Linear(4, 100),
                               torch.nn.Tanh(),
                               torch.nn.Linear(100, 100),
                               torch.nn.Tanh(),
                               torch.nn.Linear(100, 1))
    nnet.to(device)
    mse_f = torch.nn.MSELoss()
    optimizer = torch.optim.Adam(nnet.parameters(), lr=learning_rate)

    for epoch in range(n_epochs):
        for i in range(0, len(Xt), BATCH_SIZE):
            batch_Xt = Xt[i:i+BATCH_SIZE,:]
            batch_Tt = Tt[i:i+BATCH_SIZE,:]
            optimizer.zero_grad()
            Y = nnet(batch_Xt)
            mse = mse_f(Y, batch_Tt)
            mse.backward()
            optimizer.step()
    return nnet

Output:
Chances are the time required for the data to get to the GPU negates the benefit of the GPU. In this case the size of the network seems so small that the CPU should be efficient enough and the speedup from the GPU shouldn't be that big.
Also, GPUs are usually used for matrix computations in parallel, or in this case - a single batch's data multiplied by the weights of the network. So batches shouldn't be processed in parallel unless you take extra steps, like using additional libraries and/or GPUs.


Instruction: Neural network graph visualization

Input:
I would like to generate visualization of my neural network (PyTorch or ONNX model) similar to this using Graphcore Poplar.
I have looked in the documentation but I cannot find where this visualization feature is.
How can I achieve such a task ? Is there any other existing library ?

Output:
that visualization is not part of the Graphcore Poplar software. It is "data art" generated by the team at GraphCore.
It is a tough work and requires many hours to get to that fine quality, but if you are decided, I would suggest to start looking at graph visualization tools looking for "graph network visualization" (and get inspiration from galleries like https://cytoscape.org/screenshots.html).
The NN architecture can be converted into a common graph format (neurons as nodes, connections as edges) and then you may start trying.
Some ideas:
Start with a simple NN with three layers. Place the input layer at the outer circle, there is a inner circle for the hidden layer and the output layer is placed in the center. Each neuron is a dot, with radius relative to the weight and color with the bias, and you can displace it towards/away the neurons in the previous layers based on the weight. Check this image for inspiration if you are looking for a "biological" style: https://cytoscape.org/images/screenshots/edge_bundling3_1400px.png


Instruction: TypeError: 'NoneType' object cannot be interpreted as an integer

Input:
I want to classify cat and dog with pytorch. So I downloaded dataset from Kaggle, and separate train/validate set. I changed the file name from 00001.jpg to cat.00001.jpg..
But when I try to use enumerate(dataset), this error occurs:
My dataset code is:
class TrainImageFolder(Dataset):
    def __init__(self, path, transform=None):
        self.transform = transform
        self.path = path
        self.image = []
        self.label = []
        for i in os.listdir(self.path):
            self.image.append(i)
            if i.startswith("cat"):
                self.label.append(0)
            elif i.startswith("dog"):
                self.label.append(1)
        assert len(self.label) == len(self.image)

    def __len__(self):
        len(self.image)

    def __getitem__(self, index):
        label = self.label[index]
        img = Image.open(self.image[index]).convert("RGB")
        if self.transform:
            img = self.transform(img)
        return img, label

train_transform = transforms.Compose([transforms.Resize((224, 224)),
                                transforms.RandomHorizontalFlip(),
                                transforms.ToTensor(),
                                transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

train_dataset = TrainImageFolder('train', transform=train_transform)
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=False)

for i, (imgs, labels) in tqdm(enumerate(train_dataloader)):
    print(labels)

and error is:
Traceback (most recent call last):
  File "C:/Users/ge971/PycharmProjects/myVGG16/dataset.py", line 148, in <module>
    for i, (imgs, labels) in tqdm(enumerate(train_dataloader)):
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\tqdm\std.py", line 1166, in __iter__
    for obj in iterable:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 435, in __next__
    data = self._next_data()
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 474, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 67, in __iter__
    return iter(range(len(self.data_source)))
TypeError: 'NoneType' object cannot be interpreted as an integer

Process finished with exit code 1

Could you let me know how to fix the error?

Output: I just had the same error, you forgot to return the length of the data in len function.


Instruction: How to detect source of under fitting and vanishing gradients in pytorch?

Input:
How to detect source of vanishing gradients in pytorch?
By vanishing gradients, I mean then the training loss doesn't go down below some value, even on limited sets of data.
I am trying to train some network, and I have the above problem, in which I can't even get the network to over fit, but can't understand the source of the problem.
I've spent a long time googling this, and only found ways to prevent over fitting, but nothing about under fitting, or specifically, vanishing gradients.

What I did find:
Pytorch forum discussion about "bad gradients". It only refers to exploding gradients, and nan gradients, and leads to here and here which is more of the same.
I know that "making the network larger or more complex" is a general suggested way of causing over fitting (which is desired right now).
I also know that very deep networks can have their gradients vanish.
It is not clear to me that a larger network would solve the problem because it could create its own problem, as I just stated, and again I would not know how to debug this, while still seeing roughly the same behavior.
Changing the architecture to some res-net could help, but also could not, because the problem was not pinpointed to be caused by network depth.
Dead Relu can cause underfitting, and indeed moving to LeakyRelu helps, but still not enough.

How would one debug sources of under fitting in Pytorch, specifically, caused by vanishing gradients?
Instead of shooting blindly, trying things, I would like to be able to properly visualize the gradients in my network to know what I am actually trying to solve instead of guessing.
Surely, I am not the first one to have this requirement, and tools and methodologies were created for this purpose.
I would like to read about them, but don't know what to look for.
The specific net I have right now is irrelevant, as this is a general question about methodology.

Output:
You can use tensorboard with Pytorch to visualize the training gradients. Add the gradients to a tensorboard histogram during training.

For example...
Let:

model be your pytorch model
model_input be an example input to your model
run_name be a string identifier for your training session

from torch.utils.tensorboard import SummaryWriter

summary_writer = SummaryWriter(comment=run_name)
summary_writer.add_graph(model, model_input, verbose=True)

# Training loop

for step_index in ...:

    # Calculate loss etc

    for name, param in model.named_parameters():
        summary_writer.add_histogram(f'{name}.grad', param.grad, step_index)

References:

https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html
https://discuss.pytorch.org/t/is-there-a-way-to-visualize-the-gradient-path-of-the-back-propagation-of-the-entire-network/44322/4
https://debuggercafe.com/track-your-pytorch-deep-learning-project-with-tensorboard/


Instruction: Changing config and loading Hugging Face model fine-tuned on a downstream task

Input:
I am using HuggingFace models for TokenClassification task. I have the following label2id mapping. I am using version 3.3.0 of the library
label2id = {
    "B-ADD": 4,
    "B-ARRESTED": 7,
    "B-CRIME": 2,
    "B-INCIDENT_DATE": 3,
    "B-SUSPECT": 9,
    "B-VICTIMS": 1,
    "B-WPN": 5,
    "I-ADD": 8,
    "I-ARRESTED": 13,
    "I-CRIME": 11,
    "I-INCIDENT_DATE": 10,
    "I-SUSPECT": 14,
    "I-VICTIMS": 12,
    "I-WPN": 6,
    "O": 0
  }

The following scenario works well and the model gets loaded correctly.
from transformers import AutoModelForTokenClassification, AutoTokenizer, AutoConfig

pretrained_model_name = "bert-base-cased"
config = AutoConfig.from_pretrained(pretrained_model_name)

id2label = {y:x for x,y in label2id.items()}
config.label2id = label2id
config.id2label = id2label
config._num_labels = len(label2id)

model = AutoModelForTokenClassification.from_pretrained(pretrained_model_name, config=config)

model

I get the following output. The last layer have been correctly initialized with 15 neurons (numer of token category to predict).
.....................
      (dropout): Dropout(p=0.1, inplace=False)
      (classifier): Linear(in_features=768, out_features=15, bias=True)
    )

but if I changed the pretrained_model_name to "dbmdz/bert-large-cased-finetuned-conll03-english", I am getting the following error
loading weights file https://cdn.huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english/pytorch_model.bin from cache at C:\Users\anu10961/.cache\torch\transformers\4b02c1fe04cf7f7e6972536150e9fb329c7b3d5720b82afdac509bd750c705d2.6dcb154688bb97608a563afbf68ba07ae6f7beafd9bd98b5a043cd269fcc02b4
All model checkpoint weights were used when initializing BertForTokenClassification.

All the weights of BertForTokenClassification were initialized from the model checkpoint at dbmdz/bert-large-cased-finetuned-conll03-english.
If your task is similar to the task the model of the checkpoint was trained on, you can already use BertForTokenClassification for predictions without further training.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-2969a8092bf4> in <module>
----> 1 model = AutoModelForTokenClassification.from_pretrained(pretrained_model_name, config=config)

C:\ProgramData\Anaconda3\envs\arcgis183\lib\site-packages\transformers\modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1372         if type(config) in MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING.keys():
   1373             return MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING[type(config)].from_pretrained(
-> 1374                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
   1375             )
   1376

C:\ProgramData\Anaconda3\envs\arcgis183\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1047                 raise RuntimeError(
   1048                     "Error(s) in loading state_dict for {}:\n\t{}".format(
-> 1049                         model.__class__.__name__, "\n\t".join(error_msgs)
   1050                     )
   1051                 )

RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
    size mismatch for classifier.weight: copying a param with shape torch.Size([9, 1024]) from checkpoint, the shape in current model is torch.Size([15, 1024]).
    size mismatch for classifier.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([15]).

The only difference I could see is the model dbmdz/bert-large-cased-finetuned-conll03-english is already finetuned on TokenClassification task and it model config has these label2id mappings
label2id = {
    "B-LOC": 7,
    "B-MISC": 1,
    "B-ORG": 5,
    "B-PER": 3,
    "I-LOC": 8,
    "I-MISC": 2,
    "I-ORG": 6,
    "I-PER": 4,
    "O": 0
  }

But still I feel that we can change the last layer of this model and use it for my specific task (although I need to train the model first before using it for inferencing)

Output:
Once a part of the model is in the saved pre-trained model, you cannot change its hyperparameters. By setting the pre-trained model and the config, you are saying that you want a model that classifies into 15 classes and that you want to initialize with a model that uses 9 classes and that does not work.
If I understand correctly you want to initialize the underlying BERT from a different classifier. A workaround that can do it is:

Load only the underlying BERT without the classification layer;
Initialize a classification model from scratch;
Replace the randomly initialized BERT in the new classifier with the pre-trained one.

from Transformers import AutoModel, AutoModelForTokenClassification
bert = AutoModel.from_pretrained('dbmdz/bert-large-cased-finetuned-conll03-english')
classifier = AutoModelForTokenClassification.from_config(config)
classifier.bert = bert


Instruction: PyTorch one of the variables needed for gradient computation has been modified by an inplace operation

Input:
I'm doing a policy gradient method in PyTorch. I wanted to move the network update into the loop and it stopped working. I'm still a PyTorch newbie so sorry if the explanation is obvious.
Here is the original code that works:
self.policy.optimizer.zero_grad()
G = T.tensor(G, dtype=T.float).to(self.policy.device)

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss += -g * logprob

loss.backward()
self.policy.optimizer.step()

And after the change:
G = T.tensor(G, dtype=T.float).to(self.policy.device)

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss = -g * logprob
    self.policy.optimizer.zero_grad()

    loss.backward()
    self.policy.optimizer.step()

I get the error:
File "g:\VScode_projects\pytorch_shenanigans\policy_gradient.py", line 86, in learn
    loss.backward()
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I read that this RuntimeError often has to do with having to clone something, because we're using the same tensor to compute itself but I can't make heads of tails of what is wrong in my case.

Output:
This line, loss += -g * logprob, is what is wrong in your case.
Change it to this:
loss = loss + (-g * logprob)

And Yes, they are different. They  perform the  same operations but in different ways.


Instruction: What distribution is used when you make a tensor with torch.Tensor constructor in Pytorch?

Input:
I typed and ran torch.Tensor(2, 3) in Google Colab. It did work but it returned an weird-valued 2x3 tensor which includes even nan.
tensor([[3.8202e-36, 0.0000e+00, 3.9236e-44],
        [0.0000e+00,        nan, 1.8750e+00]])

I searched Pytorch(1.7.1)'s tensor.Tensor Doc to find out what distribution the default constructor has but the case when you create a tensor with Tensor class constructor was not written.
what happens when you use Tensor class constructor and what are the parameters for it?

Output:
I believe that torch.Tensor is identical to the torch.empty creation operator. It doesn't use a distribution to draw from, it's just a tensor filled with uninitialized values. Essentially used to allocate memory.
>>> torch.empty(2, 3)
tensor([[5.5699e-35, 0.0000e+00, 1.5975e-43],
        [1.3873e-43, 1.4574e-43, 6.4460e-44]])


Instruction: PyTorch installation issues on MacOS through Anaconda

Input:
I am trying to install PyTorch on my Macbook Pro. I had no issues installing NumPy or Matplotlib using the following commands:
conda install numpy
conda install matplotlib

When I then import those into Python console, they work correctly. However, when I try to import PyTorch I get the following error:
(myenv) $ % python
Python 3.9.1 (default, Dec 11 2020, 06:28:49)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/__init__.py", line 189, in <module>
    _load_global_deps()
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/__init__.py", line 142, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.dylib, 10): Library not loaded: @rpath/libomp.dylib
  Referenced from: /Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.dylib
  Reason: image not found

I have absolutely no idea what is causing this, even after looking through several forums for answers. When I got to try and reinstall PyTorch, I get this:
(myenv) $ % conda install pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

So it seems like it is all downloaded correctly - I just can't import it when in the Python console.
Any idea how to get this working correctly?

Output:
OP indicates use of Python 3.9 from Anaconda, but the PyTorch installer tool explicitly notes that one must use Python from the Conda Forge channel:

I have no issue with the following environment YAML:
File: pytorch.yaml
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - python=3.9
  - pytorch
  - torchvision
  - torchaudio
  - numpy
  - matplotlib

created with
conda env create -f pytorch.yaml -n foo


Instruction: PyTorch - Where are kernels launched?

Input:
I need to get information about kernels that PyTorch launches. For example, a callstack information such as "main.py:24 -> ... -> callkernel.py:53" would be beneficial. Is there anyway I can gather this information out out a PyTorch application execution? I also am currently searching through the source code of PyTorch but I still could not find a line where a CUDA kernel is launched. My questions are twofold:

Can I get callstack at the time of kernel launch?
Can someone show me an example of kernel launch in the source of PyTorch?

Output:
To get a helpful stack trace, you would most likely need to build pytorch with debug symbols (build instructions are here). I'm not sure if there are any debug builds available to download. But a stack trace might not make very much sense without some background, so here's a general outline of where things are defined in the codebase:
Most operators in PyTorch are implemented in the codebase as a C++ at::native namespace function within pytorch/aten/src/ATen/native. When PyTorch is built, codegen scripts automatically generate the Python functions and the Python-to-C++ bindings for the operators defined in native_functions.yaml, and the generated code is not checked into the repo (so you would have to either read the scripts or build PyTorch yourself if you want to see what's going on in codegen).
An at::native operator will usually call a device dispatch function for that operator, which is often suffixed with _stub. The dispatch function checks what device (cpu, cuda, etc) the arguments are on, and then runs a device-specific implementation. From there, another dispatch happens, which calls a datatype-specific implementation.
To go through an example, the add.out operator (which is called when you do torch.add(..., out=...) in Python) is declared here. Codegen generates everything needed to bind the Python function to at::native::add_out, which is defined here. Notice that that function calls add_stub, which is the device dispatch function.
A CPU implementation for add_stub is registered here and implemented here as add_kernel. A CUDA implementation is registered here and implemented here as add_kernel_cuda. Notice that both of these use a TensorIteratorBase object. Long story short, this object will iterate through each pair of elements in the tensor inputs that should be added together.
There is another dispatch within add_kernel and add_kernel_cuda which chooses a separate implementation based on the data type of the arguments. The separate data type implementations are generated from a shared template function. You can see that the CPU function also has a different implementation for a vectorized and a non-vectorized operation, while the CUDA implementation just has the one here.
If you want to see a full stack trace, you could run a script with gdb --args python <script name>, and create a break point for the specific kernel you want. Again, debug symbols would be needed to make sense of it.


Instruction: regarding the trick of using 1*1 convolution

Input:
I once read the following statement on using 1*1 convolution, which can help connect the input and output with different dimensions:

For example, to reduce the activation dimensions (HxW) by a factor of 2, you can use a 1x1 convolution with a stride of 2.

How to understand this example?

Output:
You can use a stride of 2. However, I wouldn't say this is a trick, not like a magic solution to retain information. You will lose half of the information. I wouldn't qualify this method as a pooling method either.
The kernel size is one pixel high and one pixel wide, and will move (stride) two pixels at a time. As a consequence, for every pixel there is on a row, the kernel will output a single value every two pixels, i.e. will output half the number of pixels on that row. Equivalently for the height, the kernel will completely discard half of the rows.

Here is the example of a 2D convolution of size 1x1 and stride 2 over a 6x6 input. On the left, the 1x1 patches in dark yellow are the successive positions of the kernel. On the right is the resulting image shaped 3x3.