Instruction: Input image size of Faster-RCNN model in Pytorch

Input:
I'm Trying to implement of Faster-RCNN model with Pytorch.
In the structure, First element of model is Transform.
from torchvision.models.detection import fasterrcnn_resnet50_fpn

model = fasterrcnn_resnet50_fpn(pretrained=True)

print(model.transform)
GeneralizedRCNNTransform(
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
    Resize(min_size=(800,), max_size=1333, mode='bilinear')
)

When images pass forward of Resize(), They come out with (800,h) or (w, 1333) according to ratio of Width and Height.
for i in range(2):
    _, image, target = testset.__getitem__(i)
    img = image.unsqueeze(0)
    output, _ = model.transform(img)

Before Transform : torch.Size([512, 640])
After Transform : [(800, 1000)]
Before Transform : torch.Size([315, 640])
After Transform : [(656, 1333)]

My question is how to get those resized output and why they use This method? I can't find the information in the paper and I can't understand the source code about transform in fasterrcnn_resnet50_fpn.
Sorry for my English

Output:
GeneralizedRCNN data transform:
https://github.com/pytorch/vision/blob/922db3086e654871c35cd80c2c01eabb65d78475/torchvision/models/detection/generalized_rcnn.py#L15
performs the data transformation on the inputs to feed into the model
min_size: minimum size of the image to be rescaled before feeding it to the backbone.
max_size: maximum size of the image to be rescaled before feeding it to the backbone
https://github.com/pytorch/vision/blob/main/torchvision/models/detection/faster_rcnn.py#L256
I couldn't either find out why it was generalize for min 800 and max 1333, didn't find anything in research paper either.
but as the 1st layer is a Conv layer, the  input to the network is fixed size, I apply many other augmentations such as mirror, random cropping etc, inspired by SSD based networks. Hence I would prefer to do all augmentation in a separate place once instead of twice.
I would assume the model should work the best during validation using images with shapes and other properties as close as possible to the training data.
though you can experiment with custom min_size and max_size...
`
from .transform import GeneralizedRCNNTransform

min_size = 900 #changed from default
max_size = 1433 #changed from default
image_mean = [0.485, 0.456, 0.406]
image_std = [0.229, 0.224, 0.225]

model = fasterrcnn_resnet50_fpn(pretrained=True, min_size, max_size, image_mean, image_std)

#batch of 4 image, 4 bboxes
images, boxes = torch.rand(4, 3, 600, 1200), torch.rand(4, 11, 4)
labels = torch.randint(1, 91, (4, 11))
images = list(image for image in images)
targets = []
for i in range(len(images)):
    d = {}
    d['boxes'] = boxes[i]
    d['labels'] = labels[i]
    targets.append(d)

output = model(images, targets)

`
or you can completely write your transforms
https://pytorch.org/vision/stable/transforms.html
'
from torchvision.transforms import transforms as T
model = fasterrcnn_resnet50_rpn()
model.transform = T.Compose([*check torchvision.transforms for more*])

'
Hope this helps.


Instruction: How do I fix Pytorch-Forecasting model fit ValueError about sequence element

Input:
I am new with Pytorch_Forecasting. I followed exactly same as described in 'Demand forecasting with the Temporal Fusion Transformer' (https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html). Everything went find until the last model fitting step (trainer.fit(...)). I keep getting the error, "ValueError: dictionary update sequence element #0 has length 1; 2 is required". I cannot figure out where this is happening... I will appreciate any help, any clue, anything to fix my problem
Thanks!!

Output: I faced the same issue. Pytorch_lightning, recently launched a new version, and pytorch-forecasting is built on top of it. I changed the version of torchmetrics to 0.5.0. pip install torchmetrics==0.5.0


Instruction: Have Euclidean distance between two inputs populate a matrix within one loop

Input:
I'm taking two tensors of dimensions (batch size, D1, D2, D3) and flattening them to (batch size, D1). I'm then trying to take the Euclidean distance between every row of one (train) tensor with every row of the second tensor (test). I'm having trouble understanding how to populate the distance combinations between tensors into each row element of the ith column.
  # Flatten
  train = x_train.view(num_train, x_train[1].view(1, -1).shape[1])
  test = x_test.view(num_test, x_test[1].view(1, -1).shape[1])

  # 1 Loop
  for i in range(num_test):
    dists[:,i] = torch.sqrt(torch.sum(torch.square(train-test[i])))

Doing it in one loop, every ith column is being populated with the same scalar value. I'm trying to make it such that
[1,i] = Euclidean distance between 1st image of train and 1st image of test ...
[2,i] = Euclidean distance between 1st image of train and 2nd image of test ...
[3,i] = Euclidean distance between 1st image of train and 3rd image of test
...
[(last element),i] = Euclidean distance between 1st image of train and (last element) image of test ...
[1,i+1] = Euclidean distance between 2nd image of train and 1st image of test

Output: Okay, I was missing a very important parameter in torch.sum() that I did not know about that solves this issue. adding a 1, so that it looks like torch.sqrt(torch.sum(torch.square(train-test[i]), 1)) outputs to what I want.


Instruction: How to Load Fastai model and predict it on single image

Input:
I have trained a fastai model using Kaggle notebook, it has saved the model but how to load the model is the problem, i have tried different methods like the method given below. Even it does load the model it doesn't have any predict function only thing I can see is model.eval().
The second problem is when the model was trained on google collab it didn't even get the single image, I did try to convert the image to NumPy way and another way but both didn't work out.
I am attaching the kaggle link of model training, the saved model and the test images in last after this code
#Code for Loading model
from fastai import *
from fastai.vision import *
import torch
loc = torch.load('/content/gdrive/MyDrive/Data Exports/35k data/stage-1.pth')
body = create_body(models.resnet18, True, None)
data_classes = 4
nf = callbacks.hooks.num_features_model(body) * 2
head = create_head(nf, data_classes, None, ps=0.5, bn_final=False)
model = nn.Sequential(body, head)

Kaggle Model
Test Images From Kaggle Dataset
Saved Model

Output:
How to load pytorch models:
loc = torch.load('/content/gdrive/MyDrive/Data Exports/35k data/stage-1.pth')
model = ... # build your model
model.load_state_dict(loc)
model.eval()

Now you should be able to simply use the forward pass to generate your predictions:
input = ... # your input image
pred = model(input) # your class predictions

Don't forget to convert your inputs to torch tensors first, you might want to use a DataLoader for ease of use.


Instruction: how to get max value by the indices return by torch.max?

Input:
The interface torch.max will return value and indices， how can i use the indices to get the according elements  from another tensor?
for example:
a = torch.rand(2,3,4)
b = torch.rand(2,3,4)
# indices shape is [2, 4]
indices = torch.max(a, 1)[1]
# how to get elements by indices ?
b_max = ????

Output:
keepdim=True when calling torch.max() and torch.take_along_dim() should do the trick.
>>> import torch
>>> a=torch.rand(2,3,4)
>>> b=torch.rand(2,3,4)
>>> indices=torch.max(a,1,keepdim=True)[1]
>>> b_max = torch.take_along_dim(b,indices,dim=1)

2D example:
>>> a=torch.rand(2,3)
>>> a
tensor([[0.0163, 0.0711, 0.5564],
        [0.4507, 0.8675, 0.5974]])
>>> b=torch.rand(2,3)
>>> b
tensor([[0.7542, 0.1793, 0.5399],
        [0.2292, 0.5329, 0.2084]])
>>> indices=torch.max(a,1,keepdim=True)[1]
>>> torch.take_along_dim(b,indices,dim=1)
tensor([[0.5399],
        [0.5329]])


Instruction: LibTorch and OpenCV Libs not working in same cmakelist file

Input:
cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
project(Detect)
#set(Torch "/home/somnath/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED)
find_package(OpenCV REQUIRED)
message(STATUS "CVINCLUDE: ${OpenCV_INCLUDE_DIRS}")
include_directories(${OpenCV_INCLUDE_DIRS})
add_executable(Detect main.cpp)
target_link_libraries(Detect  ${TORCH_LIBRARIES}; ${OpenCV_LIBS})

${TORCH_LIBRARIES}; ${OpenCV_LIBS} both are not working at the same time if want to use to build the code.
enter image description here

Output:
Try to change the libTorch from Pre-cxx11 ABI to cxx11 ABI
https://pytorch.org/get-started/locally/
All thanks to Jacob HM from
https://stackoverflow.com/a/61459156/13045595


Instruction: Pytorch Geometric sparse adjacency matrix to edge index tensor

Input: My data object has the data.adj_t parameter, giving me the sparse adjacency matrix. How can I get the edge_index tensor of size [2, num_edges] from this?

Output:
As you can see in the docs:

Since this feature is still experimental, some operations, e.g., graph pooling methods, may still require you to input the edge_index format. You can convert adj_t back to (edge_index, edge_attr) via:
row, col, edge_attr = adj_t.t().coo()
edge_index = torch.stack([row, col], dim=0)


Instruction: Having Issues Loading the CelebA dataset on Google Colab Using Pytorch

Input:
I need to load the CelebA dataset for a Python (Pytorch) implementation of the following paper: https://arxiv.org/pdf/1908.10578.pdf
The original code for loading the CelebA dataset was written in MATLAB using MatConvNet with autonn (source 15 paper). I have the source code but I'm not sure if I can share it.
It's my first time using Pytorch(version 1.9.0+cu102) and doing a paper implementation in Computer Vision.
I looked at the following relevant question: How do I load the CelebA dataset on Google Colab, using torch vision, without running out of memory?
and tested out the solution suggested by user anurag: https://stackoverflow.com/a/65528710/15087536
Unfortunately, I'm still getting a syntax error.
Here's the code below:
import torchvision
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
from torchvision import transforms

# Root directory for the dataset
data_root = 'data/celeba'
# Spatial size of training images, images are resized to this size.
image_size = 64
# batch size
batch_size = 50000

transform=transforms.Compose([transforms.Resize(image_size),
transforms.CenterCrop(image_size),transforms.ToTensor(),transforms.Normalize(mean=
[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])

dataset = ImageFolder(data_root,transform)   **syntax error**

Output:
Since we do not know the syntax error in your case, I cannot comment on it.
Below I will share one possible way to do it.

You can download the celebA dataset from Kaggle using this link. Alternatively, you can also create a Kaggle kernel using this data (no need to download data then)

If you are using google colab, upload this data accessible from your notebook.

Next you can write a PyTorch dataset which will load the images based on the partition (train, valid, test).

I am pasting an example below. You can always customize this to suit your needs.

    from torch.utils.data import Dataset, DataLoader
    import pandas as pd
    from skimage import io
    class CelebDataset(Dataset):
        def __init__(self,data_dir,partition_file_path,split,transform):
            self.partition_file = pd.read_csv(partition_file_path)
            self.data_dir = data_dir
            self.split = split
            self.transform = transform
        def __len__(self):
            self.partition_file_sub = self.partition_file[self.partition_file["partition"].isin(self.split)]
            return len(self.partition_file_sub)
        def __getitem__(self,idx):
            img_name = os.path.join(self.data_dir,
                                    self.partition_file_sub.iloc[idx, 0])
            image = io.imread(img_name)
            if self.transform:
                image = self.transform(image)
            return image

Next, you can create your train and test loaders. Change the IMAGE_PATH to your directory which contains images.

batch_size = celeba_config['batch_size']

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

IMAGE_PATH = '../input/celeba-dataset/img_align_celeba/img_align_celeba'

trainset = CelebDataset(data_dir=IMAGE_PATH,
                        partition_file_path='../input/celeba-dataset/list_eval_partition.csv',
                        split=[0,1],
                        transform=transform)
trainloader = DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = CelebDataset(data_dir=IMAGE_PATH,
                        partition_file_path='../input/celeba-dataset/list_eval_partition.csv',
                        split=[2],
                        transform=transform)
testloader = DataLoader(testset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)


Instruction: Which loss function to use for training sparse multi-label text classification problem and class skewness/imbalance

Input:
I am training a sparse multi-label text classification problem using Hugging Face models which is one part of SMART REPLY System. The task which I am doing is mentioned below:
I classify Customer Utterances as input to the model and classify to which Agent Response clusters it belongs to. I have 60 clusters and Customer Utterances can map to one or more clusters.
Input to Model
Input                             Output

My account is blocked             [0,0,0,1,1,0....0,0,0,0,0]

The Output is Encoding Vector for Cluster labels. In the above example the customer query maps into  cluster 4 and cluster 5 of agent responses.
Problem:
The model always predict the cluster numbers which are very frequent. It doesn't take the rare clusters.
Only few 1's are present at a time in output labels and the rest are 0.
Code:
#Dividing the params into those which needs to be updated and rest
param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [
    {
        'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.01
    },
    {
        'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.0
    }
]

optimizer = BertAdam(optimizer_grouped_parameters, lr =0.05, warmup = .1)

Model Training
#Empty the GPU memory as it might be memory and CPU intensive while training
torch.cuda.empty_cache()
#Number of times the whole dataset will run through the network and model is fine-tuned
epochs = 10
epoch_count = 1
#Iterate over number of epochs
for _ in trange(epochs, desc = "Epoch"):
    #Switch model to train phase where it will update gradients
    model.train()
    #Initaite train and validation loss, number of rows passed and number of batches passed
    tr_loss = 0
    nb_tr_examples, nb_tr_steps = 0, 0
    val_loss = 0
    nb_val_examples, nb_val_steps = 0, 0

    #Iterate over batches within the same epoch
    for batch in tqdm(train_dataloader):
        #Shift the batch to GPU for computation
        #pdb.set_trace()
        batch = tuple(t.to(device) for t in batch)
        #Load the input ids and masks from the batch
        b_input_ids, b_input_mask, b_labels = batch
        #Initiate gradients to 0 as they tend to add up
        optimizer.zero_grad()
        #Forward pass the input data
        logits = model(b_input_ids, token_type_ids = None, attention_mask = b_input_mask)
        #We will be using the Binary Cross entropy loss with added sigmoid function after that in BCEWithLogitsLoss
        loss_func = BCEWithLogitsLoss()
        #Calculate the loss between multilabel predicted outputs and actuals
        loss = loss_func(logits, b_labels.type_as(logits))

        #Backpropogate the loss and calculate the gradients
        loss.backward()
        #Update the weights with the calculated gradients
        optimizer.step()
        #Add the loss of the batch to the final loss, number of rows and batches
        tr_loss += loss.item()
        nb_tr_examples += b_input_ids.size(0)
        nb_tr_steps += 1
    #Print the current training loss
    print("Train Loss: {}".format(tr_loss/nb_tr_examples))

    # Save the trained model after each epoch.
#     pickle.dump(model, open("conv_bert_model_"+str(epoch_count)+".pkl", "wb"))
    epoch_count=epoch_count+1

I am using this loss function currently:
loss_func = BCEWithLogitsLoss()
#Calculate the loss between multilabel predicted outputs and actuals
loss = loss_func(logits, b_labels.type_as(logits))

Is there any way to improve models output.( Recall and Precision) by using different loss function?
How we tackle the cluster imbalance problem in Hugging face models in case of MULTI LABLES classification .

Output:
You can use a weighted cross entropy applied to each index of the output. You'd have to go through the training set to calculate the weights of each cluster.
criterion = nn.BCEWithLogitsLoss(reduction='none')
loss = criterion(output, target)
loss = (loss * weights).mean()
loss.backward()

By doing so the losses for different indexes are not combined immediately, but kept separate. They are first multiplied with the weights and then combined.
To calculate the weights, assuming outputs is a Tensor:
weights = torch.sum(outputs, 0)/torch.sum(outputs)

And assuming numpy arrays:
weights = np.sum(outputs, 0)/np.sum(outputs)


Instruction: Not able to save model to gs bucket using torch.save()

Input:
I am trying to save a PyTorch model to my google cloud bucket but it is always showing a "FileNotFoundError error".
I already have a gs bucket and the file path I am providing is also correct. My code is running on a GCP notebook instance.
path = "gs://bucket_name/model/model.pt"
torch.save(model,path)

It would really help me if someone tries to upload a model to gs bucket and let me know if you could or not? It would also help me if you shared the right way to put models into gs bucket using torch.save().

Output:
try this:
import gcsfs
fs = gcsfs.GCSFileSystem(project = '<enter your gc project>')
with fs.open("gs://bucket_name/"+f'model/model.pt', 'wb') as f:
    torch.save(model, f)


Instruction: How to solve issue : grad_fn = None

Input: Could anyone advise why grad_fn = None for this pytorch coding ?

Output:
Parameter weights are leaf nodes, i.e. those tensor are not the result of an operation, in other words, there is no other tensor node preceding them. You can check using the is_leaf attribute:
>>> nn.Linear(1,1).weight.is_leaf
True

The grad_fn attribute essentially holds the callback function to backpropagate from a given tensor node. By definition, leaf tensors do not have such function because there is nothing to backpropagate on.


Instruction: tflite quantized mobilenet v2 classifier not working

Input:
My goal is to convert a PyTorch Model into a quantized tflite model that can be used for inference on the Edge TPU.
I was able to convert a fairly complex depth estimation model from PyTorch to tflite and I successfully ran it on the Edge TPU. But because not all operations were supported, inference was pretty slow (>800ms).
Number of operations that will run on Edge TPU: 87
Number of operations that will run on CPU: 47

Depth Estimation
Because I want a model that runs fully on the TPU, I tried converting the simplest model I could think of, a MobilenetV2 classification model. But when running the quantized model, I get strangely inaccurate results.

PyTorch
TFLite

Samoyed:0.8303
missile: 0.184565

Pomeranian: 0.06989
kuvasz: 0.184565

keeshond: 0.01296
stupa: 0.184565

collie: 0.0108
Samoyed: 0.184565

Great Pyrenees: 0.00989
Arctic fox: 0.184565

Is this caused by quantizing the model from float32 to uint8 or am I doing something wrong? And if it is caused by quantization, how can I mitigate that? The classification example from corral works fine and, as far as I know, is uses the same model.
Conversion Process
PyTorch -> ONNX -> OpenVINO -> TensorFlow -> TensorFlowLite
I wrote my own code to convert the model from PyTorch to ONNX and from TensorFlow(pd) into TFlite. For the other conversion steps, I used the OpenVINO mo.py script and the openvino2tensorflow toll because of the nchw nhwc mismatch between PyTorch and TensorFlow.
Downloads
Depth Estimation Model: https://github.com/AaronZettler/miscellaneous/blob/master/mobilenet_v2_depth_est.pth?raw=true
Classification Model: https://github.com/AaronZettler/miscellaneous/blob/master/mobilenetv2.tflite?raw=true
Labels: https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt
Image: https://raw.githubusercontent.com/pytorch/hub/master/images/dog.jpg
Code
This code does not require the Edge TPU to be run, but it does require the google coral libraries.
If I use different parameters for mean and std, like (2.0, 76.0), I get a solid result for the dog.jpg image but if I try to classify something else I have the same problem.

import numpy as np
from PIL import Image
from pycoral.adapters import classify
from pycoral.adapters import common
from pycoral.utils.dataset import read_label_file
from torchvision import transforms

from tensorflow.lite.python.interpreter import Interpreter

def cropPIL(image, new_width, new_height):
    width, height = image.size

    left = (width - new_width)/2
    top = (height - new_height)/2
    right = (width + new_width)/2
    bottom = (height + new_height)/2

    return image.crop((left, top, right, bottom))

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

def classify_img(image_dir, lables_dir, model_dir, mean, std):
    #loading lables and model
    labels = read_label_file(lables_dir)
    interpreter = Interpreter(model_path=model_dir)
    interpreter.allocate_tensors()

    #load an resize image
    size = (256, 256)
    image = Image.open(image_dir).convert('RGB')
    image = image.resize(((int)(size[0]*image.width/image.height), size[1]), Image.ANTIALIAS)
    image = cropPIL(image, 224, 224)
    image = np.asarray(image)

    #normalizing the input image
    params = common.input_details(interpreter, 'quantization_parameters')
    scale = params['scales']
    zero_point = params['zero_points']

    normalized_input = (image - mean) / (std * scale) + zero_point
    np.clip(normalized_input, 0, 255, out=normalized_input)

    #setting the image as input
    common.set_input(interpreter, normalized_input.astype(np.uint8))

    #run inference
    interpreter.invoke()

    #get output tensor and run softmax
    output_details = interpreter.get_output_details()[0]
    output_data = interpreter.tensor(output_details['index'])().flatten()
    scores = softmax(output_data.astype(float))

    #get the top 10 classes
    classes = classify.get_classes_from_scores(scores, 5, 0.0)

    print('-------RESULTS--------')
    for c in classes:
       print('%s: %f' % (labels.get(c.id, c.id), c.score))

image_dir  = 'data/dog.jpg'
lables_dir = 'data/imagenet_classes.txt'
model_dir  = 'models/mobilenetv2.tflite'

classify_img(image_dir, lables_dir, model_dir, 114.0, 57.0)

To run the PyTorch model on google colab I had to replace
model = torch.hub.load('pytorch/vision:v0.9.0', 'mobilenet_v2', pretrained=True)
with
model = torchvision.models.mobilenet_v2(pretrained=True)
to make it work.
This is the code I used to test The PyTorch model on my machine.
import torch
from PIL import Image
from torchvision import transforms
import torchvision

import numpy as np
import matplotlib.pyplot as plt

def inference(model, input_image, lables_dir):
    preprocess = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])

    input_tensor = preprocess(input_image)
    input_batch = input_tensor.unsqueeze(0)

    # move the input and model to GPU for speed if available
    if torch.cuda.is_available():
        input_batch = input_batch.to('cuda')
        model.to('cuda')

    with torch.no_grad():
        output = model(input_batch)

    probabilities = torch.nn.functional.softmax(output[0], dim=0)

    # Read the categories
    with open(lables_dir, "r") as f:
        categories = [s.strip() for s in f.readlines()]

    # Show top categories per image
    top5_prob, top5_catid = torch.topk(probabilities, 5)
    result = {}
    for i in range(top5_prob.size(0)):
        result[categories[top5_catid[i]]] = top5_prob[i].item()
    return result

def classify(image_dir, lables_dir):
    model = torchvision.models.mobilenet_v2(pretrained=True)
    model.eval()

    im = Image.open(image_dir)
    results = inference(model, im, lables_dir)
    for result in results:
        print(f'{result}: {round(results[result], 5)}')

classify('data/dog.jpg', 'data/imagenet_classes.txt')

Output:
EdgeTPU mapping of PReLU (LeakyReLU) is now supported in openvino2tensorflow v1.20.4.
However, due to the large size of the model, it is not possible to map all operations to the EdgeTPU. Therefore, the part of the EdgeTPU that does not fit in RAM is offloaded to the CPU for inference, which is very slow. In this case, inference by the CPU alone is 4 to 5 times faster. EdgeTPU does not support PReLU (LeakyReLU), so the operations must be replaced. However, openvino2tensorflow v1.20.4 automatically replaces the operations in the conversion process.

Converted model
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/149_depth_estimation

Convert sample

docker run --gpus all -it --rm \
-v `pwd`:/home/user/workdir \
pinto0309/openvino2tensorflow:latest

cd workdir

MODEL=depth_estimation_mbnv2

H=180
W=320
$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model ${MODEL}_${H}x${W}.onnx \
--data_type FP32 \
--output_dir ${H}x${W}/openvino/FP32
$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model ${MODEL}_${H}x${W}.onnx \
--data_type FP16 \
--output_dir ${H}x${W}/openvino/FP16
mkdir -p ${H}x${W}/openvino/myriad
${INTEL_OPENVINO_DIR}/deployment_tools/inference_engine/lib/intel64/myriad_compile \
-m ${H}x${W}/openvino/FP16/${MODEL}_${H}x${W}.xml \
-ip U8 \
-VPU_NUMBER_OF_SHAVES 4 \
-VPU_NUMBER_OF_CMX_SLICES 4 \
-o ${H}x${W}/openvino/myriad/${MODEL}_${H}x${W}.blob

openvino2tensorflow \
--model_path ${H}x${W}/openvino/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_no_quant_float32_tflite \
--output_weight_quant_tflite \
--output_float16_quant_tflite \
--output_integer_quant_tflite \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8' \
--output_tfjs \
--output_coreml \
--output_tftrt
mv saved_model saved_model_${H}x${W}

openvino2tensorflow \
--model_path ${H}x${W}/openvino/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_edgetpu \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8'
mv saved_model/model_full_integer_quant.tflite saved_model_${H}x${W}/model_full_integer_quant.tflite
mv saved_model/model_full_integer_quant_edgetpu.tflite saved_model_${H}x${W}/model_full_integer_quant_edgetpu.tflite

mv ${H}x${W}/openvino saved_model_${H}x${W}/openvino
mv ${MODEL}_${H}x${W}.onnx saved_model_${H}x${W}/${MODEL}_${H}x${W}.onnx

H=240
W=320
$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model ${MODEL}_${H}x${W}.onnx \
--data_type FP32 \
--output_dir ${H}x${W}/openvino/FP32
$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model ${MODEL}_${H}x${W}.onnx \
--data_type FP16 \
--output_dir ${H}x${W}/openvino/FP16
mkdir -p ${H}x${W}/openvino/myriad
${INTEL_OPENVINO_DIR}/deployment_tools/inference_engine/lib/intel64/myriad_compile \
-m ${H}x${W}/openvino/FP16/${MODEL}_${H}x${W}.xml \
-ip U8 \
-VPU_NUMBER_OF_SHAVES 4 \
-VPU_NUMBER_OF_CMX_SLICES 4 \
-o ${H}x${W}/openvino/myriad/${MODEL}_${H}x${W}.blob

openvino2tensorflow \
--model_path ${H}x${W}/openvino/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_no_quant_float32_tflite \
--output_weight_quant_tflite \
--output_float16_quant_tflite \
--output_integer_quant_tflite \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8' \
--output_tfjs \
--output_coreml \
--output_tftrt
mv saved_model saved_model_${H}x${W}

openvino2tensorflow \
--model_path ${H}x${W}/openvino/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_edgetpu \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8'
mv saved_model/model_full_integer_quant.tflite saved_model_${H}x${W}/model_full_integer_quant.tflite
mv saved_model/model_full_integer_quant_edgetpu.tflite saved_model_${H}x${W}/model_full_integer_quant_edgetpu.tflite

mv ${H}x${W}/openvino saved_model_${H}x${W}/openvino
mv ${MODEL}_${H}x${W}.onnx saved_model_${H}x${W}/${MODEL}_${H}x${W}.onnx

PReLU (LeakyReLU) to Maximum (ReLU), Minimum, Mul, Add
From:

To:

EdgeTPU model


Instruction: How do I use separate types of gpus (e.g. 1080Ti vs 2080Ti) on the same docker image without needing to re-run `python setup.py develop`?

Input:
I'm using a pytorch-based repository where the installation step specifies to run python setup.py develop with this setup.py file. I have been running the repository fine with 1080Ti and 1080 GPUs using a docker image which clones the repo and runs the setup.py script in the build process. The following are files copied from my Dockerfile.
RUN git clone https://github.com/CVMI-Lab/ST3D.git
WORKDIR /ST3D
RUN nvidia-smi
RUN pip install -r requirements.txt
RUN python setup.py develop

Upon entering the container, I only mount specific folders within the repo as follows:
GPU_ID = 0

ENVS="  --env=NVIDIA_VISIBLE_DEVICES=$GPU_ID
        --env=CUDA_VISIBLE_DEVICES=$GPU_ID
        --env=NVIDIA_DRIVER_CAPABILITIES=all"

VOLUMES="       --volume=$DATA_PATH:/ST3D/data
                --volume=$CODE_PATH/pcdet:/ST3D/pcdet
                --volume=$CODE_PATH/tools:/ST3D/tools
                --volume=$CODE_PATH/output:/ST3D/output"

docker  run -d -it --rm \
        $VOLUMES \
        $ENVS \
        --runtime=nvidia \
        --gpus $GPU_ID \
        --privileged \
        --net=host \
        --workdir=/ST3D \
        darrenjkt/st3d:v0.3.0

Recently we installed a 2080Ti in the same computer. When I enter the same docker container with solely the 2080Ti gpu, using the same python script, I get the following error:
RuntimeError: CUDA error: no kernel image is available for execution on the device

This error pertains to one of the cpp modules installed in the setup.py.
I can solve this by running python setup.py develop again which would then enable it to work with the 2080Ti. I've tried then committing the docker container to a 2080Ti specific docker image, and the 1080 docker container to a 1080 specific image. However I noticed that once I run python setup.py develop on the 2080Ti container, it then gives me the CUDA error for the 1080 images. And if I run the setup.py on the 1080 gpu again, it'll then give me the CUDA error back on that 2080Ti image. This baffles me as I have not mounted the build files but rather kept them solely in the container and committed it to a new image.
So my question is, how can I set up my environment/docker image such that it doesn't require a rebuild of setup.py each time?

Output:
The problem was solved by building the docker image with the following:
RUN git clone https://github.com/CVMI-Lab/ST3D.git
WORKDIR /ST3D
RUN nvidia-smi
RUN pip install -r requirements.txt
RUN TORCH_CUDA_ARCH_LIST="6.1 7.5" python setup.py develop

Where the TORCH_CUDA_ARCH_LIST="6.1 7.5" where it's 6.1 for 1080/1080Ti and 7.5 for 2080Ti. This overwrites the default cpp_extension behaviour to explicitly specify which compute_capability to support.


Instruction: understanding pytorch conv2d internally

Input:
I'm trying to understand what does the nn.conv2d do internally.
so lets assume we are applying Conv2d to a 32*32 RGB image.
torch.nn.Conv2d(3, 49, 4, bias=True)

so :

when we initialize the conv layer how many weights and in which shapes would it have, please tell this for biases apart?
before applying it the conv the image would have 3 * 32 * 32 shape and after applying it would have 49 * 29 * 29 so what happens in between?

I define "slide" operation (don't know real name) as multiplying first to element of kernel to first element of box in shape of image going till last element of kernel corresponding making one the 1of29 * 1of29 is calculated.
and "slide all" doing this horizontally and vertically till the all 29 * 29 are calculated.
so I understand how a kernel would act but I don't understand how many kernels would be created by the torch.nn.Conv2d(3, 49, 4, bias=True) and which of them would be apllying on R,G,B channels.

Output:
I understand how a kernel would act but I don't understand how many
kernels would be created by the nn.Conv2d(3, 49, 4, bias=True) and
which of them would be applying on R, G, and B channels.

Calling nn.Conv2d(3, 49, 4, bias=True) will initialize 49 4x4-kernels, each having a total of three channels and a single bias parameter. That's a total of 49*(4*4*3 + 1) parameters, i.e. 2,401 parameters.
You can check that it is indeed correct with:
>>> conv2d = nn.Conv2d(3, 49, 4, bias=True)

Parameters list will contain the weight tensor shaped (n_filters=49, n_channels=3, kernel_height=4, kernel_width=4), and a bias tensor shaped (49,):
>>> [p.shape for p in conv2d.parameters()]
[torch.Size([49, 3, 4, 4]), torch.Size([49])]

If we get a look at the total number of parameters, we indeed find:
>>> nn.utils.parameters_to_vector(conv2d.parameters()).numel()
2401

Concerning how they are applied: each one of the 49 kernels will be applied 'independently' to the input map. For each filter operation, you are convolving the input of a three-channel tensor, with a three-channel kernel. Each one of those 49 convolutions gets its respective bias added. At the end, you are left with a number of 49 single-channel maps which are concatenated to make up a single 49-channel map. In practice, everything is done in one go using a window view of the input.
I am certainly biased towards my own posts: here you will find another explanation of shapes in convolutional neural networks.


Instruction: estimator.fit hangs on sagemaker on local mode

Input:
I am trying to train a pytorch model using Sagemaker on local mode, but whenever I call estimator.fit the code hangs indefinitely and I have to interrupt the notebook kernel. This happens both in my local machine and in Sagemaker Studio. But when I use EC2, the training runs normally.
Here the call to the estimator, and the stack trace once I interrupt the kernel:
import sagemaker
from sagemaker.pytorch import PyTorch

bucket = "bucket-name"
role = sagemaker.get_execution_role()
training_input_path = f"s3://{bucket}/dataset/path"

sagemaker_session = sagemaker.LocalSession()
sagemaker_session.config = {"local": {"local_code": True}}

output_path = "file://."

estimator = PyTorch(
    entry_point="train.py",
    source_dir="src",
    hyperparameters={"max-epochs": 1},
    framework_version="1.8",
    py_version="py3",
    instance_count=1,
    instance_type="local",
    role=role,
    output_path=output_path,
    sagemaker_session=sagemaker_session,
)

estimator.fit({"training": training_input_path})

Stack trace:
    ---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-9-35cdd6021288> in <module>
----> 1 estimator.fit({"training": training_input_path})

/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    678         self._prepare_for_training(job_name=job_name)
    679
--> 680         self.latest_training_job = _TrainingJob.start_new(self, inputs, experiment_config)
    681         self.jobs.append(self.latest_training_job)
    682         if wait:

/opt/conda/lib/python3.7/site-packages/sagemaker/estimator.py in start_new(cls, estimator, inputs, experiment_config)
   1450         """
   1451         train_args = cls._get_train_args(estimator, inputs, experiment_config)
-> 1452         estimator.sagemaker_session.train(**train_args)
   1453
   1454         return cls(estimator.sagemaker_session, estimator._current_job_name)

/opt/conda/lib/python3.7/site-packages/sagemaker/session.py in train(self, input_mode, input_config, role, job_name, output_config, resource_config, vpc_config, hyperparameters, stop_condition, tags, metric_definitions, enable_network_isolation, image_uri, algorithm_arn, encrypt_inter_container_traffic, use_spot_instances, checkpoint_s3_uri, checkpoint_local_path, experiment_config, debugger_rule_configs, debugger_hook_config, tensorboard_output_config, enable_sagemaker_metrics, profiler_rule_configs, profiler_config, environment, retry_strategy)
    572         LOGGER.info("Creating training-job with name: %s", job_name)
    573         LOGGER.debug("train request: %s", json.dumps(train_request, indent=4))
--> 574         self.sagemaker_client.create_training_job(**train_request)
    575
    576     def _get_train_request(  # noqa: C901

/opt/conda/lib/python3.7/site-packages/sagemaker/local/local_session.py in create_training_job(self, TrainingJobName, AlgorithmSpecification, OutputDataConfig, ResourceConfig, InputDataConfig, **kwargs)
    184         hyperparameters = kwargs["HyperParameters"] if "HyperParameters" in kwargs else {}
    185         logger.info("Starting training job")
--> 186         training_job.start(InputDataConfig, OutputDataConfig, hyperparameters, TrainingJobName)
    187
    188         LocalSagemakerClient._training_jobs[TrainingJobName] = training_job

/opt/conda/lib/python3.7/site-packages/sagemaker/local/entities.py in start(self, input_data_config, output_data_config, hyperparameters, job_name)
    219
    220         self.model_artifacts = self.container.train(
--> 221             input_data_config, output_data_config, hyperparameters, job_name
    222         )
    223         self.end_time = datetime.datetime.now()

/opt/conda/lib/python3.7/site-packages/sagemaker/local/image.py in train(self, input_data_config, output_data_config, hyperparameters, job_name)
    200         data_dir = self._create_tmp_folder()
    201         volumes = self._prepare_training_volumes(
--> 202             data_dir, input_data_config, output_data_config, hyperparameters
    203         )
    204         # If local, source directory needs to be updated to mounted /opt/ml/code path

/opt/conda/lib/python3.7/site-packages/sagemaker/local/image.py in _prepare_training_volumes(self, data_dir, input_data_config, output_data_config, hyperparameters)
    487             os.mkdir(channel_dir)
    488
--> 489             data_source = sagemaker.local.data.get_data_source_instance(uri, self.sagemaker_session)
    490             volumes.append(_Volume(data_source.get_root_dir(), channel=channel_name))
    491

/opt/conda/lib/python3.7/site-packages/sagemaker/local/data.py in get_data_source_instance(data_source, sagemaker_session)
     52         return LocalFileDataSource(parsed_uri.netloc + parsed_uri.path)
     53     if parsed_uri.scheme == "s3":
---> 54         return S3DataSource(parsed_uri.netloc, parsed_uri.path, sagemaker_session)
     55     raise ValueError(
     56         "data_source must be either file or s3. parsed_uri.scheme: {}".format(parsed_uri.scheme)

/opt/conda/lib/python3.7/site-packages/sagemaker/local/data.py in __init__(self, bucket, prefix, sagemaker_session)
    183             working_dir = "/private{}".format(working_dir)
    184
--> 185         sagemaker.utils.download_folder(bucket, prefix, working_dir, sagemaker_session)
    186         self.files = LocalFileDataSource(working_dir)
    187

/opt/conda/lib/python3.7/site-packages/sagemaker/utils.py in download_folder(bucket_name, prefix, target, sagemaker_session)
    286                 raise
    287
--> 288     _download_files_under_prefix(bucket_name, prefix, target, s3)
    289
    290

/opt/conda/lib/python3.7/site-packages/sagemaker/utils.py in _download_files_under_prefix(bucket_name, prefix, target, s3)
    314             if exc.errno != errno.EEXIST:
    315                 raise
--> 316         obj.download_file(file_path)
    317
    318

/opt/conda/lib/python3.7/site-packages/boto3/s3/inject.py in object_download_file(self, Filename, ExtraArgs, Callback, Config)
    313     return self.meta.client.download_file(
    314         Bucket=self.bucket_name, Key=self.key, Filename=Filename,
--> 315         ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
    316
    317

/opt/conda/lib/python3.7/site-packages/boto3/s3/inject.py in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
    171         return transfer.download_file(
    172             bucket=Bucket, key=Key, filename=Filename,
--> 173             extra_args=ExtraArgs, callback=Callback)
    174
    175

/opt/conda/lib/python3.7/site-packages/boto3/s3/transfer.py in download_file(self, bucket, key, filename, extra_args, callback)
    305             bucket, key, filename, extra_args, subscribers)
    306         try:
--> 307             future.result()
    308         # This is for backwards compatibility where when retries are
    309         # exceeded we need to throw the same error from boto3 instead of

/opt/conda/lib/python3.7/site-packages/s3transfer/futures.py in result(self)
    107         except KeyboardInterrupt as e:
    108             self.cancel()
--> 109             raise e
    110
    111     def cancel(self):

/opt/conda/lib/python3.7/site-packages/s3transfer/futures.py in result(self)
    104             # however if a KeyboardInterrupt is raised we want want to exit
    105             # out of this and propogate the exception.
--> 106             return self._coordinator.result()
    107         except KeyboardInterrupt as e:
    108             self.cancel()

/opt/conda/lib/python3.7/site-packages/s3transfer/futures.py in result(self)
    258         # possible value integer value, which is on the scale of billions of
    259         # years...
--> 260         self._done_event.wait(MAXINT)
    261
    262         # Once done waiting, raise an exception if present or return the

/opt/conda/lib/python3.7/threading.py in wait(self, timeout)
    550             signaled = self._flag
    551             if not signaled:
--> 552                 signaled = self._cond.wait(timeout)
    553             return signaled
    554

/opt/conda/lib/python3.7/threading.py in wait(self, timeout)
    294         try:    # restore state no matter what (e.g., KeyboardInterrupt)
    295             if timeout is None:
--> 296                 waiter.acquire()
    297                 gotit = True
    298             else:

KeyboardInterrupt:

Output:
SageMaker Studio does not natively support local mode. Studio Apps are themselves docker containers and therefore they require privileged access if they were to be able to build and run docker containers.
As an alternative solution, you can create a remote docker host on an EC2 instance and setup docker on your Studio App. There is quite a bit of networking and package installation involved, but the solution will enable you to use full docker functionality. Additionally, as of version 2.80.0 of SageMaker Python SDK, it now supports local mode when you are using remote docker host.
sdockerSageMaker Studio Docker CLI extension (see this repo) can simplify deploying the above solution in simple two steps (only works for Studio Domain in VPCOnly mode) and it has an easy to follow example here.
UPDATE:
There is now a UI extension (see repo) which can make the experience much smoother and easier to manage.


Instruction: PyTorch | getting "RuntimeError: Found dtype Long but expected Float" with dataset Omniglot

Input:
Im a real newbie on PyTorch and Neural Networks. I have started to work on these suubjects this week and my mentor has gave me a code and with some tasks to work on the code.
But the code that he gave me is not working. I have tried to fix this all day but got no result. Because i do not know the background of the NN's and PyTorch it is harder to understand the problem.
Need your help on that.
Thank you !
import torch
import numpy as np
import torchvision.datasets as datasets
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
from torchsummary import summary
#DEFINE YOUR DEVICE
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device) #if cpu, go Runtime-> Change runtime type-> Hardware accelerator GPU -> Save -> Redo previous steps
#DOWNLOAD DATASET
train_data = datasets.Omniglot('./data', background=True, download = True, transform = transforms.ToTensor())
test_data = datasets.Omniglot('./data',background = False, download = True, transform = transforms.ToTensor())
#DEFINE DATA GENERATOR
batch_size = 50
train_generator = torch.utils.data.DataLoader(train_data, batch_size = batch_size, shuffle = True)
test_generator = torch.utils.data.DataLoader(test_data, batch_size = batch_size, shuffle = False)
#DEFINE NEURAL NETWORK MODEL
class CNN(torch.nn.Module):
  def __init__(self):
    super(CNN, self).__init__()
    self.conv1 = torch.nn.Conv2d(1, 8, kernel_size = 4, stride = 1)
    self.conv2 = torch.nn.Conv2d(8, 16, kernel_size = 4, stride = 1)
    self.mpool = torch.nn.MaxPool2d(2)
    self.fc1 = torch.nn.Linear(18432, 256)
    self.fc2 = torch.nn.Linear(256, 64)
    self.fc3 = torch.nn.Linear(64, 50)
    self.relu = torch.nn.ReLU()
    self.sigmoid = torch.nn.Sigmoid()
  def forward(self, x):
    hidden = self.mpool(self.relu(self.conv1(x)))
    hidden = self.mpool(self.relu(self.conv2(hidden)))
    hidden = hidden.view(-1,18432)
    hidden = self.relu(self.fc1(hidden))
    hidden = self.relu(self.fc2(hidden))
    output = self.fc3(hidden)
    return output
# CREATE MODEL
model = CNN()
model.to(device)
summary(model, (1, 105, 105))
# DEFINE LOSS FUNCTION AND OPTIMIZER
learning_rate = 0.001
loss_fun = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# TRAIN THE MODEL
model.train()
epoch = 10
num_of_batch = np.int(len(train_generator.dataset) / batch_size)
loss_values = np.zeros(epoch * num_of_batch)
for i in range(epoch):
      for batch_idx, (x_train, y_train) in enumerate(train_generator):
          x_train, y_train = x_train.to(device), y_train.to(device)
          optimizer.zero_grad()
          y_pred = model(x_train)
          loss = loss_fun(y_pred, y_train)
          loss_values[num_of_batch * i + batch_idx] = loss.item()
          loss.backward()
          optimizer.step()
          if (batch_idx + 1) % batch_size == 0:
              print('Epoch: {}/{} [Batch: {}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                  i + 1, epoch, (batch_idx + 1) * len(x_train), len(train_generator.dataset),
                  100. * (batch_idx + 1) / len(train_generator), loss.item()))
#PLOT THE LEARNING CURVE
iterations = np.linspace(0,epoch,num_of_batch*epoch)
plt.plot(iterations, loss_values)
plt.title('Learning Curve')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.grid('on')
#TEST THE MODEL
model.eval()
correct=0
total=0
for x_val, y_val in test_generator:
  x_val = x_val.to(device)
  y_val = y_val.to(device)
  output = model(x_val)
  y_pred = output.argmax(dim=1)
  for i in range(y_pred.shape[0]):
    if y_val[i]==y_pred[i]:
      correct += 1
    total +=1
print('Validation accuracy: %.2f%%' %((100*correct)//(total)))

Here is the error code that i receive.
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([50])) that is different to the input size (torch.Size([25, 50])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-bffd863688df> in <module>()
     13     loss = loss_fun(y_pred, y_train)
     14     loss_values[num_of_batch*i+batch_idx] = loss.item()
---> 15     loss.backward()
     16     optimizer.step()
     17     if (batch_idx+1) % batch_size == 0:

1 frames
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    147     Variable._execution_engine.run_backward(
    148         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    150
    151

RuntimeError: Found dtype Long but expected Float

Output:
Your dataset is returning integers for your labels, you should cast them to floating points. One way of solving it is to do:
loss = loss_fun(y_pred, y_train.float())


Instruction: How to optimize cudaHostAlloc and cudaLaunchKernel times in pytorch training

Input:
I am trying to profile my model with pytorch profiler. I used the below code to profile
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    with record_function("model_inference"):
        output_batch = self.model(input_batch)
print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))

The profiler output is as follows
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
                                        model_inference         3.17%      83.011ms        63.97%        1.675s        1.675s       0.000us         0.00%     373.844ms     373.844ms             1
                                            aten::copy_         0.24%       6.333ms        39.93%        1.046s       1.504ms      28.758ms         7.69%      29.035ms      41.777us           695
                                          cudaHostAlloc        36.02%     943.053ms        36.02%     943.053ms      30.421ms       0.000us         0.00%       0.000us       0.000us            31
                                       cudaLaunchKernel        35.93%     940.773ms        35.93%     940.773ms      86.619us       0.000us         0.00%       0.000us       0.000us         10861
                                           aten::repeat         0.04%     979.000us        33.77%     884.170ms      30.489ms       0.000us         0.00%     204.000us       7.034us            29
                                           aten::conv2d         0.06%       1.481ms         8.71%     228.183ms     695.680us       0.000us         0.00%     145.688ms     444.171us           328
                                      aten::convolution         0.05%       1.391ms         8.66%     226.702ms     691.165us       0.000us         0.00%     145.688ms     444.171us           328
                                     aten::_convolution         0.10%       2.742ms         8.61%     225.311ms     686.924us       0.000us         0.00%     145.688ms     444.171us           328
                                aten::cudnn_convolution         0.53%      13.803ms         8.33%     218.051ms     664.790us     137.822ms        36.87%     137.822ms     420.189us           328
                                               cudaFree         7.46%     195.373ms         7.46%     195.373ms      48.843ms       0.000us         0.00%       0.000us       0.000us             4
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
Self CPU time total: 2.618s
Self CUDA time total: 373.844ms

I notice that a large part of the time (self CPU) is taken by cudaHostAlloc and cudaLaunchKernel. What are these cudaHostAlloc and cudaLaunchKernel? Is it possible to reduce this time? If yes how? Are there any standard operations that I'm missing which is leading to this high time consumption?
PS: I'm new to profiling as such. Kindly let me know if any other information is needed.

Output:
I am not an expert but I think cudaLaunchKernel is called at every operation done with cuda. So I think you cannot optimize it.
If you plot a detailed tracing https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html#using-tracing-functionality, you see that it is called everytime you do a cuda operation like here for a linear layer.

One note on you profiler output: aten::copy_ cudaHostAlloc cudaLaunchKernel and aten::repeat all take roughly 40% of the CPU total time. I think it may be related to ProfilerActivity.CUDA that records CUDA operation but it also add a lot of CPU time on your first CUDA operation that is profiled. In my case a simple torch.ones(1000, device="cuda") took a full second of CPU time because it was the first cuda operation.
It may be the problem in your case, try to remove ProfilerActivity.CUDA and maybe aten::copy_ cudaHostAlloc cudaLaunchKernel and aten::repeat will have a much smaller CPU time and will disappear from the table.


Instruction: Weights & Biases with Transformers and PyTorch?

Input:
I'm training an NLP model at work (e-commerce SEO) applying a BERT variation for portuguese language (BERTimbau) through Transformers by Hugging Face.
I didn't used the Trainer from Transformers API. I used PyTorch to set all parameters through DataLoader.utils and adamW. I trained my model using run_glue.py.
I'm training with a VM on GCP using Jupyterlab. I know that I can use Weights & Biases both for PyTorch and Transformers. But I don't know exactly how to set it using run_glue.py. It's my first time using Weights & Biases.
After preprocessing and splitting train and test through Sklearn, my code is as it follows:
from transformers import BertTokenizer
import torch
#import torchvision
from torch.utils.data import Dataset, TensorDataset
import collections.abc as container_abcs

# To feed our text to BERT, it must be split into tokens, and then these tokens must be mapped to their index in the tokenizer vocabulary.
# Constructs a BERT tokenizer. Based on WordPiece.
# The tokenization must be performed by the tokenizer included with BERT
tokenizer = BertTokenizer.from_pretrained('neuralmind/bert-large-portuguese-cased',
                                          do_lower_case=True)

# Tokenize all of the sentences and map the tokens to thier word IDs. To convert all the titles from text into encoded form.
# We will use padding and truncation because the training routine expects all tensors within a batch to have the same dimensions.
encoded_data_train = tokenizer.batch_encode_plus(
    df[df.data_type=='train'].text.values,
    add_special_tokens=True,                            # Add '[CLS]' and '[SEP]'. Sequences encoded with special tokens relative to their model
    return_attention_mask=True,                         # Return mask according to specific tokenizer defined by max_length
    pad_to_max_length=True,                             # Pad & truncate all sentences. Pad all titles to certain maximum length
    max_length=128,                                     # Do not need to set max_length=256
    return_tensors='pt'                                 # Set to use PyTorch tensors
)

encoded_data_val = tokenizer.batch_encode_plus(
    df[df.data_type=='val'].text.values,
    add_special_tokens=True,
    return_attention_mask=True,
    pad_to_max_length=True,
    max_length=128,
    return_tensors='pt'
)

# Split the data into input_ids, attention_masks and labels.
# Converting the input data to the tensor , which can be feeded to the model
input_ids_train = encoded_data_train['input_ids']                    # Add the encoded sentence to the list.
attention_masks_train = encoded_data_train['attention_mask']         # And its attention mask (simply differentiates padding from non-padding).
labels_train = torch.tensor(df[df.data_type=='train'].label.values)

input_ids_val = encoded_data_val['input_ids']
attention_masks_val = encoded_data_val['attention_mask']
labels_val = torch.tensor(df[df.data_type=='val'].label.values)

# Create training data and validation data
dataset_train = TensorDataset(input_ids_train, attention_masks_train, labels_train)
dataset_val = TensorDataset(input_ids_val, attention_masks_val, labels_val)

from transformers import BertForSequenceClassification
model = BertForSequenceClassification.from_pretrained("neuralmind/bert-large-portuguese-cased",  # Select your pretrained Model
                                                      num_labels=len(label_dict),                # Labels tp predict
                                                      output_attentions=False,                   # Whether the model returns attentions weights. We don’t really care about output_attentions.
                                                      output_hidden_states=False)                # Whether the model returns all hidden-states. We also don’t need output_hidden_states

from torch.utils.data import DataLoader, RandomSampler, SequentialSampler

batch_size = 32                                                              # Set your batch size according to your GPU memory

dataloader_train = DataLoader(dataset_train                                  # Use DataLoader to Optimize your model
                              ,sampler=RandomSampler(dataset_train)          # Random Sampler from your dataset
                              ,batch_size=batch_size)                        # If your batch_size is too high you will get a warning when you run the model
                              #,num_workers=4                                # Number of cores
                              #,pin_memory=True)                             # Use GPU to send your batch

dataloader_validation = DataLoader(dataset_val
                                   ,sampler=SequentialSampler(dataset_val)   # For validation the order doesn't matter. Sequential Sampler consumes less GPU.
                                   ,batch_size=batch_size)
                                   #,num_workers=4
                                   #,pin_memory=True)
from transformers import AdamW, get_linear_schedule_with_warmup
# hyperparameters
# To construct an optimizer, we have to give it an iterable containing the parameters to optimize.
# Then, we can specify optimizer-specific options such as the learning rate, epsilon, etc.
optimizer = AdamW(model.parameters(),  # AdamW is a class from the huggingface library (as opposed to pytorch)
                  lr=2e-5,             # args.learning_rate - default is 5e-5
                  eps=1e-8)            # args.adam_epsilon  - default is 1e-8

# Number of training epochs. The BERT authors recommend between 2 and 4.
epochs = 2

# Create the learning rate scheduler that decreases linearly from the initial learning rate set in the optimizer to 0,
# after a warmup period during which it increases linearly from 0 to the initial learning rate set in the optimizer.
scheduler = get_linear_schedule_with_warmup(optimizer,
                                            num_warmup_steps=0,                               # Default value in run_glue.py
                                            num_training_steps=len(dataloader_train)*epochs)  # Total number of training steps is [number of batches] x [number of epochs].
                                                                                # Note that this is not the same as the number of training samples).

from sklearn.metrics import f1_score

def f1_score_func(preds, labels):
    preds_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return f1_score(labels_flat, preds_flat, average='weighted')

def accuracy_per_class(preds, labels):
    label_dict_inverse = {v: k for k, v in label_dict.items()}

    preds_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()

    for label in np.unique(labels_flat):
        y_preds = preds_flat[labels_flat==label]
        y_true = labels_flat[labels_flat==label]
        print(f'Class: {label_dict_inverse[label]}')
        print(f'Accuracy: {len(y_preds[y_preds==label])}/{len(y_true)}\n')

And here follows run_glue.py:
import random
from tqdm import tqdm
import torch
import numpy as np
# from tqdm.notebook import trange, tqdm

'''
This training code is based on the 'run_glue.py' script here:
https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128
'''

# Just right before the actual usage select your hardware
device = torch.device('cuda') # or cpu
model = model.to(device)      # send your model to your hardware

# Set the seed value all over the place to make this reproducible.
seed_val = 17
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# We'll store a number of quantities such as training and validation loss, validation and timings.
def evaluate(dataloader_val):
    '''
    Put the model in evaluation mode--the dropout layers behave differently
    during evaluation.
    '''

    model.eval()

    loss_val_total = 0 # Tracking variables

    predictions, true_vals = [], []

    for batch in dataloader_val:
        '''
         Unpack this training batch from our dataloader.
         As we unpack the batch, we'll also copy each tensor to the GPU using the
         `to` method.

         `batch` contains three pytorch tensors:
           [0]: input ids
           [1]: attention masks
           [2]: labels
        '''
        batch = tuple(b.to(device) for b in batch)

        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }

        '''
        Tell pytorch not to bother with constructing the compute graph during
        the forward pass, since this is only needed for backprop (training).
        '''
        with torch.no_grad():
            outputs = model(**inputs)

        '''
        Perform a forward pass (evaluate the model on this training batch).
        The documentation for this `model` function is here:
        https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
        This will return the loss (rather than the model output)
        because we have provided the `labels`.
        It returns different numbers of parameters depending on what arguments
        arge given and what flags are set. For our useage here, it returns
        the loss (because we provided labels) and the "logits"--the model
        outputs prior to activation.
        '''
        loss = outputs[0]
        logits = outputs[1]
        loss_val_total += loss.item() # Accumulate the validation loss.

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = inputs['labels'].cpu().numpy()
        predictions.append(logits)
        true_vals.append(label_ids)

    loss_val_avg = loss_val_total/len(dataloader_val) # Calculate the average loss over all of the batches.

    predictions = np.concatenate(predictions, axis=0)
    true_vals = np.concatenate(true_vals, axis=0)

    return loss_val_avg, predictions, true_vals

    # ========================================
    #               Training
    # ========================================

# For each epoch...
for epoch in tqdm(range(1, epochs+1)):
    '''
    Put the model into training mode. Don't be mislead--the call to
    `train` just changes the *mode*, it doesn't *perform* the training.
    `dropout` and `batchnorm` layers behave differently during training
    vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
    '''

    model.train()        # Put the model into training mode.

    loss_train_total = 0 # Reset the total loss for this epoch.

    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)
    for batch in progress_bar:

        model.zero_grad()
        '''
        Always clear any previously calculated gradients before performing a
        backward pass. PyTorch doesn't do this automatically because
        accumulating the gradients is "convenient while training RNNs".
        (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
        '''

        batch = tuple(b.to(device) for b in batch)
        '''
        Unpack this training batch from our dataloader.

         As we unpack the batch, we'll also copy each tensor to the GPU using the
         `to` method.

         `batch` contains three pytorch tensors:
           [0]: input ids
           [1]: attention masks
           [2]: labels
         '''

        inputs = {'input_ids':      batch[0], #.to(device)
                  'attention_mask': batch[1], #.to(device)
                  'labels':         batch[2], #.to(device)
                 }

        outputs = model(**inputs)

        loss = outputs[0] # The call to `model` always returns a tuple, so we need to pull the loss value out of the tuple.
        loss_train_total += loss.item()         # Accumulate the training loss over all of the batches so that we can
                                                # calculate the average loss at the end. `loss` is a Tensor containing a
                                                # single value; the `.item()` function just returns the Python value
                                                # from the tensor.
        loss.backward() # Perform a backward pass to calculate the gradients.

        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)    # Clip the norm of the gradients to 1.0.
                                                                   # This is to help prevent the "exploding gradients" problem.
                                                                   # modified based on their gradients, the learning rate, etc.

        optimizer.step()        # Update parameters and take a step using the computed gradient.
                                # The optimizer dictates the "update rule"--how the parameters are
                                # modified based on their gradients, the learning rate, etc.

        scheduler.step()        # Update the learning rate.

        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})

    torch.save(model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model') # Save Model

    tqdm.write(f'\nEpoch {epoch}') # Show running epoch

    loss_train_avg = loss_train_total/len(dataloader_train) # Calculate the average loss over all of the batches.

    tqdm.write(f'Training loss: {loss_train_avg}') # Show loss average

        # ========================================
        #               Validation
        # ========================================
        # After the completion of each training epoch, measure our performance on
        # our validation set.

    # Record all statistics from this epoch.
    val_loss, predictions, true_vals = evaluate(dataloader_validation)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'Validation loss: {val_loss}')
    tqdm.write(f'F1 Score (Weighted): {val_f1}')

Output:
Scott from W&B here. Although you're not using the HuggingFace WandbCallback, you can still take advantage of wandb easily using our Python API.
All you need to do is call wandb.log({'val_loss': val_loss, 'train_loss': train_avg}) with whatever you want to log after you call wandb.init before training.
Here's an example training loop:
wandb.init(
      # Set entity to specify your username or team name
      # ex: entity="carey",
      # Set the project where this run will be logged
      project="huggingface",
      # Track hyperparameters and run metadata
      config={
      "learning_rate": 0.02,
      "architecture": "BERT",
      "dataset": "my-dataset",})

  # This simple block simulates a training loop logging metrics
  for x in range(50):
      loss, accuracy = # calculate this yourself
      # Log metrics from your script to W&B
      wandb.log({"acc":acc, "loss":loss})

  # Mark the run as finished
  wandb.finish()

If you want to log Datasets or Models, you can do that using wandb Artifacts.
This Quickstart guide is a good place to start for more information:
W&B Quickstart


Instruction: Sequential batch processing vs parallel batch processing?

Input: In deep learning based model training, in general batch of inputs are passed. For example for training a deep learning model with [512] dimensional input feature vector, say for batch size= 4, we mainly pass [4,512] dimenional input. I am curious what are the logical significance of passing the same input after flattening the input across the batch and channel dimenions [2048]. Logically the locality structure will be destroyed but will it significanlty speed up my implementation? And can it affect the performance?

Output:
In supervised learning, you would usually be working with data points (e.g. a feature vector or a multi-dimensional input such as an image) paired with some kind of ground-truth (a label for classifications tasks, or another multi-dimensional object altogether). Feeding to your model a flattened tensor containing multiple data points would not make sense in terms of supervision. Assuming you do an inference this way, what would be the supervision signal at the output level of your model? Would you combine the labels as well? All of this seem to depend heavily on the use case: is there some kind of temporal coherence between the elements of the batch?
Performance-wise, this has no implications whatsoever. Tensors are already 'flattened' by design since their memory is laid out in contiguous memory buffers. The idea of multi-dimensionality is an abstraction layer provided by those libraries (namely NumPy's arrays and Torch's tensors) to allow for easier and more flexible control over data.


Instruction: How to make Python libraries accessible in PYTHONPATH?

Input:
I am trying to implement StyleGAN2-ADA PyTorch: https://github.com/NVlabs/stylegan2-ada-pytorch.
On the GitHub repo, it states the following:

The above code requires torch_utils and dnnlib to be accessible via
PYTHONPATH. It does not need source code for the networks themselves —
their class definitions are loaded from the pickle via
torch_utils.persistence.

What does this mean, and how can I do this?

Output:
Let's say you have the source code of this repo cloned to /somepath/stylegan2-ada-pytorch which means that the directories you quoted are at /somepath/stylegan2-ada-pytorch/torch_utils and /somepath/stylegan2-ada-pytorch/dnnlib, respectively.
Now let's say you have a python script that you want to access this code. It can be anywhere on your machine, as long as you add this to the top of your python script:
import os
import sys

#save the literal filepath to both directories as strings
tu_path = os.path.join('somepath','stylegan2-ada-pytorch','torch_utils')
dnnlib_path = os.path.join('somepath','stylegan2-ada-pytorch','dnnlib')

#add those strings to python path
sys.path.append(tu_path)
sys.path.append(dnnlib_path )

Note that this only adds those location to PYTHONPATH for the duration of that python script running, so you need this at the top of any python script that intends to use those libraries.


Instruction: Pytorch customized dataloader

Input:
I am trying to train a classifier with MNIST dataset using pytorch-lightening.
import pytorch_lightning as pl
from torchvision import transforms
from torchvision.datasets import MNIST, SVHN
from torch.utils.data import DataLoader, random_split

class MNISTData(pl.LightningDataModule):

    def __init__(self, data_dir='./', batch_size=256):
        super().__init__()
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.transform = transforms.ToTensor()

    def download(self):
        MNIST(self.data_dir, train=True, download=True)
        MNIST(self.data_dir, train=False, download=True)

    def setup(self, stage=None):
        if stage == 'fit' or stage is None:
            mnist_train = MNIST(self.data_dir, train=True, transform=self.transform)
            self.mnist_train, self.mnist_val = random_split(mnist_train, [55000, 5000])
        if stage == 'test' or stage is None:
            self.mnist_test = MNIST(self.data_dir, train=False, transform=self.transform)

    def train_dataloader(self):
        mnist_train = DataLoader(self.mnist_train, batch_size=self.batch_size)
        return mnist_train

    def val_dataloader(self):
        mnist_val = DataLoader(self.mnist_val, batch_size=self.batch_size)
        return mnist_val

    def test_dataloader(self):
        mnist_test = DataLoader(self.mnist_test, batch_size=self.batch_size)

After using MNISTData().setup(), I gained MNISTData().mnist_train, MNISTData().mnist_val, MNISTData().mnist_test whose length are 55000, 5000, 10000 with type of torch.utils.data.dataset.Subset.
But when i call dataloader w.r.t MNISTData().train_dataloader, MNISTData().val_dataloader, MNISTData().test_dataloader I only get DataLoader with 215, 20, None datas in them.
Can someone know the reason or could fix the problem?

Output:
As I told in the comments, and Ivan posted in his answer, there was missing return statement:
def test_dataloader(self):
    mnist_test = DataLoader(self.mnist_test, batch_size=self.batch_size)
    return mnist_test  # <<< missing return

As per your comment, if we try:
a = MNISTData()
# skip download, assuming you already have it
a.setup()

b, c, d = a.train_dataloader(), a.val_dataloader(), a.test_dataloader()
# len(b)=215, len(c)=20, len(d)=40

I think your question is why the length of b, c, d are different from the length of the datasets. The answer is that the len() of a DataLoader is equal to the number of batches, not the number of samples, therefore:
import math

batch_size = 256
len(b) = math.ceil(55000 / batch_size) = 215
len(c) = math.ceil(5000 / batch_size) = 20
len(d) = math.ceil(10000 / batch_size) = 40

BTW, we're using math.ceil because DataLoader has drop_last=False by default, otherwise it would be math.floor.


Instruction: Is it possible to retrain a trained model on fewer classes?

Input:
I am working on image detection where I am detecting and classifying an image into one of 14 different thoric diseases (multi-label classification problem).
The model is trained on NIH dataset with which I get 80% AUC. Now I want to improve the model by training on a second dataset. But the main problem is both dataset's classes are not matched.
The second dataset contains 10 classes that overlap with the first dataset with which I trained the model.
Questions:

Is it possible to retrain a model on fewer classes.

Will retraining my model on a new dataset impact the AUC of other non-similar classes?

How big is the chance that this will improve the model?

The model and code are based on fast.ai and PyTorch.

Output:
Based on discussion in the comments:

Yes, if the classes overlap (with different data points from a different dataset) you can train the same classifier layer with two datasets. This would mean in one of the datasets, 4 out of 14 classes are simply not trained. What this means is that you are basically making your existing 14-class dataset more imbalanced by adding more samples for only 10 out of 14 classes.
Training on 10 out of 14 classes will introduce a forgetting effect on the 4 classes that are not trained additionally. You can counteract this somewhat by using the suggested alternate training, or by combining all the data into one big dataset, but this does not solve the fact that the new combined dataset is then probably more imbalanced than the original 14-class dataset. Unless the 4 classes not in the 10-class dataset are for some reason over represented in the 14-class dataset, but I assume you're not going to get that lucky.
Because both your dataset and your model will focus heavier on 10 out of the 14 classes, your accuracy may go up. However, this means that the 4 classes that do not overlap are simply being ignored in favor of higher accuracy on the remaining 10 classes. On paper, the numbers may look better, but in practice you're making your model less useful for a 14-class classification task.


Instruction: DropPath in TIMM seems like a Dropout?

Input:
The code below (taken from here) seems to implement only a simple Dropout, neither the DropPath nor DropConnect. Is that true?
def drop_path(x, drop_prob: float = 0., training: bool = False):
    """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
    This is the same as the DropConnect impl I created for EfficientNet, etc networks, however,
    the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
    See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for
    changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use
    'survival rate' as the argument.
    """
    if drop_prob == 0. or not training:
        return x
    keep_prob = 1 - drop_prob
    shape = (x.shape[0],) + (1,) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNets
    random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
    random_tensor.floor_()  # binarize
    output = x.div(keep_prob) * random_tensor
    return output

Output:
No, it is different from Dropout:
import torch
from torch.nn.functional import dropout

torch.manual_seed(2021)

def drop_path(x, drop_prob: float = 0., training: bool = False):
    if drop_prob == 0. or not training:
        return x
    keep_prob = 1 - drop_prob
    shape = (x.shape[0],) + (1,) * (x.ndim - 1)
    random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
    random_tensor.floor_()  # binarize
    output = x.div(keep_prob) * random_tensor
    return output

x = torch.rand(3, 2, 2, 2)

# DropPath
d1_out = drop_path(x, drop_prob=0.33, training=True)

# Dropout
d2_out = dropout(x, p=0.33, training=True)

Let's compare the outputs (I removed the line break between channel dimension for readability):
# DropPath
print(d1_out)
#  tensor([[[[0.1947, 0.7662],
#            [1.1083, 1.0685]],
#           [[0.8515, 0.2467],
#            [0.0661, 1.4370]]],
#
#          [[[0.0000, 0.0000],
#            [0.0000, 0.0000]],
#           [[0.0000, 0.0000],
#            [0.0000, 0.0000]]],
#
#          [[[0.7658, 0.4417],
#            [1.1692, 1.1052]],
#           [[1.2014, 0.4532],
#            [1.4840, 0.7499]]]])

# Dropout
print(d2_out)
#  tensor([[[[0.1947, 0.7662],
#            [1.1083, 1.0685]],
#           [[0.8515, 0.2467],
#            [0.0661, 1.4370]]],
#
#          [[[0.0000, 0.1480],
#            [1.2083, 0.0000]],
#           [[1.2272, 0.1853],
#            [0.0000, 0.5385]]],
#
#          [[[0.7658, 0.0000],
#            [1.1692, 1.1052]],
#           [[1.2014, 0.4532],
#            [0.0000, 0.7499]]]])

As you can see, they are different. DropPath is dropping an entire sample from the batch, which effectively results in stochastic depth when used as in Eq. 2 of their paper. On the other hand, Dropout is dropping random values, as expected (from the docs):

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.

Also note that both scale the output values based on the probability, i.e., the non-zeroed out elements are identical for the same p.


Instruction: Can't install GPU-enabled Pytorch in Conda environment from environment.yml

Input:
I'm on Ubuntu 20.04 LTS with CUDA 11.1 installed (and working, with PATH and LD_LIBRARY_PATH configured correctly), and I'm trying to define a reusable conda environment (i.e., in an environment.yml file) that successfully installs PyTorch with CUDA support.
However, when I use the environment file, I get a message that Torch wasn't compiled with CUDA support:
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> device = torch.device("cuda:0")
>>> t = torch.tensor(device=device, data=[0,1,2,3])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jdr2160/anaconda3/envs/foo/lib/python3.8/site-packages/torch/cuda/__init__.py", line 166, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

My environment.yml is pretty bare-bones:
name: foo
channels:
  - conda-forge
  - nvidia
  - pytorch
dependencies:
  - cudatoolkit=11.1
  - python=3.8
  - pytorch

When I create an 'empty' python 3.8 environment and install the Conda packages from the command line instead of from an environment file, everything works fine:
$ conda env create --name bar python=3.8
...
$ conda activate bar
$ conda install pytorch cudatoolkit=11.1 -c pytorch -c nvidia
...
$ python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> device = torch.device("cuda:0")
>>> t = torch.tensor(device=device, data=[0,1,2,3])
>>>

Can anyone tell what's going on here? It seems that Conda doesn't see the cudatoolkit=11.1 dependency while installing PyTorch from the environment file, but I have no idea how to fix it.

Output:
Just a few minutes after posting this question, I was able to figure out the solution. It turns out that it has to do with prioritizing Conda channels. The solution (which isn't well-documented by Anaconda) is to specify the correct channel for cudatoolkit and pytorch in environment.yml:
name: foo
channels:
  - conda-forge
  - nvidia
  - pytorch
dependencies:
  - nvidia::cudatoolkit=11.1
  - python=3.8
  - pytorch::pytorch


Instruction: Pytorch DataLoder Is Very Slow

Input:
I have a problem with the DataLoader form Pytorch, because is very slow.
I did a test to show this, here is the code:
data = np.load('slices.npy')
data = np.reshape(data, (-1, 1225))
data = torch.FloatTensor(data).to('cuda')
print(data.shape)
# ==> torch.Size([273468, 1225])

class UnlabeledTensorDataset(TensorDataset):
    def __init__(self, data_tensor):
        self.data_tensor = data_tensor
        self.samples = data_tensor.shape[0]

    def __getitem__(self, index):
        return self.data_tensor[index]

    def __len__(self):
        return self.samples

test_set = UnlabeledTensorDataset(data)
test_loader = DataLoader(test_set, batch_size=data.shape[0])

start = datetime.datetime.now()
with torch.no_grad():
    for batch in test_loader:
        print(batch.shape)     # ==> torch.Size([273468, 1225])
        y_pred = model(batch)
        loss = torch.sqrt(criterion(y_pred, batch))
        avg_loss = loss
print(round((datetime.datetime.now() - start).total_seconds() * 1000, 2))
# ==> 1527.57  (milliseconds)   !!!!!!!!!!!!!!!!!!!!!!!!

start = datetime.datetime.now()
with torch.no_grad():
    print(data.shape)     # ==> torch.Size([273468, 1225])
    y_pred = model(data)
    loss = torch.sqrt(criterion(y_pred, data))
    avg_loss = loss
print(round((datetime.datetime.now() - start).total_seconds() * 1000, 2))
# ==> 2.0     (milliseconds)    !!!!!!!!!!!!!!!!!!!!!!!!

I will like to use the DataLoader but I want a way to fix the slow issue, dose anyone know why this is happening?

Output:
The time difference seems logical to me:

On one end you're looping over test_loader and doing 1225 inferences.

On the other, you are doing a single inference.


Instruction: How to convert a tensor into a list of tensors

Input: How can I convert a tensor into a list of tensors. For instance: P1 is a torch.Tensor with 60 values in it and I want a list of tensors with 60 tensors in it.

Output:
You can coerce the torch.Tensor to a list with list:
>>> P1 = torch.rand(60)
>>> list(P1)
[tensor(0.5987),
 tensor(0.5321),
 tensor(0.6590),
 ...
 tensor(0.1381)]

This works with multi-dimensional tensors too:
>>> P1 = torch.rand(60, 2)
>>> list(P1)
[tensor([0.4675, 0.0430]),
 tensor([0.2175, 0.6271]),
 tensor([0.3378, 0.8516]),
 ...,
 tensor([0.5099, 0.3411]


Instruction: PyTorch's nn.Conv2d with half-precision (fp16) is slower than fp32

Input:
I have found that a single 2D convolution operation with float16 is slower than with float32.
I am working with a Gtx 1660 Ti with torch.1.8.0+cu111 and cuda-11.1 (also tried with torch.1.9.0)

Dtype
in=1,out=64
in=1,out=128
in=64,out=128

Fp16
3532 it/s
632 it/s
599it/s

Fp32
2160 it/s
1311 it/s
925it/s

I am measuring the convolution speed with the following code.
inputfp16 = torch.arange(0,ch_in*64*64).reshape(1, ch_in, 64, 64).type(torch.float16).to('cuda:0')
inputfp32 = torch.arange(0,ch_in*64*64).reshape(1, ch_in, 64, 64).type(torch.float32).to('cuda:0')

conv2d_16 = nn.Conv2d(ch_in,ch_out, 3, 1, 1).eval().to('cuda:0').half()
conv2d_32 = nn.Conv2d(ch_in,ch_out, 3, 1, 1).eval().to('cuda:0')

for i in tqdm(range(0, 50)):
    out = conv2d_16(inputfp16)
    out.cpu()

for i in tqdm(range(0, 50)):
    out = conv2d_32(inputfp32)
    out.cpu()

It would be great if you let me know whether you have had the same problem, even better if you can provide a solution.

Output:
Well, the problem lays on the fact that Mixed/Half precision tensor calculations are accelerated via Tensor Cores.
Theoretically (and practically) Tensor Cores are designed to handle lower precision matrix calculations, where, for instance you add the fp32 multiplication product of 2 fp16 matrix calculation to the accumulator.
As long as GTX 1660 TI doesn't come with Tensor Cores, we can conclude that CUDA won't be able to utilize acceleration with Mixed/Half precisions on that GPU.


Instruction: LSTM.weight_ih_l[k] dimensions with proj_size

Input:
According to Pytorch LSTM documentation :-

~LSTM.weight_ih_l[k] – the learnable input-hidden weights of the kth layer (W_ii|W_if|W_ig|W_io), of shape (4*hidden_size, input_size) for k = 0. Otherwise, the shape is (4 * hidden_size, num_directions * hidden_size)

My doubt is, why for k > 0  the shape for each weight is (hidden_size, num_directions * hidden_size), according to me, shouldn't be (hidden_size, num_directions * proj_size)  because the layer above the lowest layer is receiving the input which is the output of the lowest layer which have the shape of (L, N, num_directions*proj_size)
UPDATE
Indeed, the documentation have mentioned wrong shapes. It will be fixed soon.
Github Issue

Output:
This is now fixed. The OP opened the Issue #65053 that was fixed by PR #65102 (commit 83878e1).

It just happens that the documentation isn't providing all the details in this case, and you're correct. In fact, you can check in the source code that the shape of W_ih is (4*hidden_size, num_directions * proj_size) when proj_size > 0 for k > 0:
# [...]
if mode == 'LSTM':
    gate_size = 4 * hidden_size
elif mode == 'GRU':
    gate_size = 3 * hidden_size
elif mode == 'RNN_TANH':
    gate_size = hidden_size
elif mode == 'RNN_RELU':
    gate_size = hidden_size
else:
    raise ValueError("Unrecognized RNN mode: " + mode)

self._flat_weights_names = []
self._all_weights = []
for layer in range(num_layers):
    for direction in range(num_directions):
        real_hidden_size = proj_size if proj_size > 0 else hidden_size
        layer_input_size = input_size if layer == 0 else real_hidden_size * num_directions

        w_ih = Parameter(torch.empty((gate_size, layer_input_size), **factory_kwargs))
        # [...]

As you can see, w_ih has the shape (gate_size, layer_input_size), where:

gate_size is 4 * hidden_size for LSTM, and
layer_input_size is

input_size if layer == 0 (layer is equivalent to k in the docs), else
real_hidden_size * num_directions for k > 0, and

real_hidden_size = proj_size if proj_size > 0, else it is hidden_size.

That is: if proj_size > 0 and layer > 0, layer_input_size = proj_size * num_directions, and the shape of w_ih will be equal to (4 * hidden_size, proj_size * num_directions.
It is worth noting that they do have the following in the documentation:

If proj_size > 0 is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of h_t will be changed from hidden_size to proj_size (dimensions of W_hi will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: h_t = W_hr * h_t. Note that as a consequence of this, the output of LSTM network will be of different shape as well.


Instruction: Mismatched size on BertForSequenceClassification from Transformers and multiclass problem

Input:
I just trained a BERT model on a Dataset composed by products and labels (departments) for an e-commerce website. It's a multiclass problem. I used BertForSequenceClassification to predict the department for each product. I split it in train and evaluation, I used dataloader from pytorch, and I've got a good score with no overfit.
Now I want to try it on a new Dataset to check how it works on unseen data. But I can't achieve to load the model and apply on the new Dataset. I get the following error:
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
    size mismatch for classifier.weight: copying a param with shape torch.Size([59, 1024]) from checkpoint, the shape in current model is torch.Size([105, 1024]).
    size mismatch for classifier.bias: copying a param with shape torch.Size([59]) from checkpoint, the shape in current model is torch.Size([105]).

I see that the problem probably is a mismatch from labels size between both Datasets. I've searched a bit and I've found a recommendation to use ignore_mismatched_sizes=True as and argument for pretrained. But I keep receiving the same error.
Here is part of my code when trying to predict on unseen data:
from transformers import BertForSequenceClassification

# Just right before the actual usage select your hardware
device = torch.device('cuda') # or cpu
model = model.to(device)      # send your model to your hardware

model = BertForSequenceClassification.from_pretrained("neuralmind/bert-large-portuguese-cased",
                                                      num_labels=len(label_dict),
                                                      output_attentions=False,
                                                      output_hidden_states=False,
                                                      ignore_mismatched_sizes=True)

model.to(device)

model.load_state_dict(torch.load('finetuned_BERT_epoch_2_full-Copy1.model', map_location=torch.device('cuda')))

_, predictions, true_vals = evaluate(dataloader_validation)
accuracy_per_class(predictions, true_vals)

Could someone help me how could I deal with it? I don't know what more can I do!
Any help I'm very grateful!

Output:
Your new dataset has 105 classes while your model was trained for 59 classes.  As you have already mentioned, you can use ignore_mismatched_sizes to load your model. This parameter will load the the embedding and encoding layers of your model, but will randomly initialize the classification head:
model = BertForSequenceClassification.from_pretrained("finetuned_BERT_epoch_2_full-Copy1.model",
                                                      num_labels=105,
                                                      output_attentions=False,
                                                      output_hidden_states=False,
                                                      ignore_mismatched_sizes=True)

In case you want to keep the classification layer of the 59 labels and add 46 labels, you can refer to this answer. Please also note the comments of this answer, because this approach does not provide any meaningful results due to the random initialization for the new labels.


Instruction: Unable to access batch items in iterator - Torchtext Attribute Error: 'Field' object has no attribute 'vocab'

Input:
I am unable to access batch items from the Iterator object in Torchtext.
Following is the error

AttributeError: 'Field' object has no attribute 'vocab'

Code to Recreate Problem
#Access to Drive
from google.colab import drive
drive.mount ('/content/gdrive')

import numpy as np
import spacy
spacy_en = spacy.load("en")

def tokenize(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

import torch
from torchtext.legacy.data import Field, LabelField, Iterator

TEXT = Field(sequential=True, use_vocab=True, tokenize=tokenize, lower=True)
LABEL = LabelField(dtype = torch.long, use_vocab=False)

fields = {"text": ("txt", TEXT), "label": ("lbl", LABEL)}

from torchtext.legacy.data import TabularDataset
train_data, test_data = TabularDataset.splits(path="/content/gdrive/MyDrive/Colab Notebooks/",
                                              train="Strong_Train.csv",
                                              test="Strong_Test.csv", format="csv", fields=fields)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

TEXT = Field(sequential=True, use_vocab=True, tokenize=tokenize, lower=True)
LABEL = LabelField(dtype = torch.long, use_vocab=False)

TEXT.build_vocab(train_data, max_size=10000 )
LABEL.build_vocab(train_data)

train_iterator = Iterator(train_data, batch_size=1, device=device)
for batch in train_iterator:
    print('hello')

My Analysis of Problem
As per the error description, 'Field' object is causing the problem; TEXT is the Field object in this code.
type(TEXT)
Output: torchtext.legacy.data.field.Field
Since the error says,

Field object has no attribute 'vocab'

accessing 'vocab' should give an error, but it doesn't give an error
TEXT.vocab
Output: <torchtext.legacy.vocab.Vocab at 0x7fce37b91950>
I am also able to get the length of vocab
len(TEXT.vocab)
Output: 2
So the question still remains, if there exists 'vocab' attribute in the Field object, why am I getting this error? and how do I resolve it?
Environment Specifics
Running the code on Google Colab
Torchtext version is 0.10.0

Output:
After you set your device, you are redefining TEXT and LABEL. That isn't necessary, and could cause issues. Also, you are setting use_vocab=False for your LabelField, and then building a vocab for it.
From the torchtext 0.8 docs:

use_vocab: Whether to use a Vocab object. If False, the data in this field should already be numerical. Default: True.

I would start by clearing up those issues and see if that resolves your error.


Instruction: How do I split an iterable dataset into training and test datasets?

Input:
I have an iterable dataset object with all of my data files. How can I split it into train and validation set. I have seen a few solutions for custom datasets but iterable does not support len() operator.
torch.utils.random_sample() and torch.utils.SubsetRandomSample() don't work.
def __init__(self):
    bla bla
def __iter__(self):
    bla bla
    yield batch

Output:
Technically you can just set a goal ratio, and start collecting items into two lists randomly using that ratio. The result won't be perfect, but asymptotically it will keep the ratio.
The example is JavaScript, as it can be run here:

{
  let a = [],
      b = [];
  function addsample(x) {
    if (Math.random() < 0.2) // aims for 20%-80% split
      a.push(x);
    else
      b.push(x);
    return {a, b};
  }
}

for(let i=0;i<20;i++)
  console.log(JSON.stringify(addsample(i)));

If you run the snippet a couple times, you will see that the output varies a lot, but even with such a small sample size it's quite visible, that usually there is a suitable split available all the time where a really has around 1/4 the size of b. Sometimes it even manages to end up exactly 4:16, but many times it will be something else. And there can be "unlucky" runs too, when a has more elements than b at the end.