Instruction: ImportError: Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training

Input:
cannot install apex for distributed and fp16 training of bert model
i have tried to install by cloning the apex from github and tried to install packages using pip

i have tried to install apex by cloning from git hub using following command:

git clone https://github.com/NVIDIA/apex.git

and cd apex to goto apex directory and tried to install package using following pip command:

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext"

full code is:

def main(server_ip,server_port,local_rank,no_cuda,fp16,train_batch_size,gradient_accumulation_steps,seed,do_train,do_eval,output_dir,task_name,data_dir,do_lower_case,bert_model,num_train_epochs,cache_dir,learning_rate,warmup_proportion,loss_scale,max_seq_length):
        if server_ip and server_port:
            # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
            import ptvsd
            print("Waiting for debugger attach")
            ptvsd.enable_attach(address=(server_ip, server_port), redirect_output=True)
            ptvsd.wait_for_attach()

        processors = {"ner":NerProcessor}
        print(processors)

        if local_rank == -1 or no_cuda:
            device = torch.device("cuda" if torch.cuda.is_available() and not no_cuda else "cpu")
            n_gpu = torch.cuda.device_count()
        else:
            torch.cuda.set_device(local_rank)
            device = torch.device("cuda", local_rank)
            n_gpu = 1
            # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
            torch.distributed.init_process_group(backend='nccl')
        logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
            device, n_gpu, bool(local_rank != -1), fp16))

        if gradient_accumulation_steps < 1:
            raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
                                args.gradient_accumulation_steps))

        train_batch_size = train_batch_size // gradient_accumulation_steps

        random.seed(seed)
        np.random.seed(seed)
        torch.manual_seed(seed)

        if not do_train and not do_eval:
            raise ValueError("At least one of `do_train` or `do_eval` must be True.")

        if os.path.exists(output_dir) and os.listdir(output_dir) and do_train:
            raise ValueError("Output directory ({}) already exists and is not empty.".format(output_dir))
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)

        task_name = task_name.lower()

        if task_name not in processors:
            raise ValueError("Task not found: %s" % (task_name))

        processor = processors[task_name]()
        label_list = processor.get_labels()
        num_labels = len(label_list) + 1

        tokenizer = BertTokenizer.from_pretrained(bert_model, do_lower_case=do_lower_case)

        train_examples = None
        num_train_optimization_steps = None
        if do_train:
            train_examples = processor.get_train_examples(data_dir)
            num_train_optimization_steps = int(
                len(train_examples) / train_batch_size / gradient_accumulation_steps) * num_train_epochs
            if local_rank != -1:
                num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()

    #     # Prepare model
        cache_dir = cache_dir if cache_dir else os.path.join(str(PYTORCH_PRETRAINED_BERT_CACHE), 'distributed_{}'.format(local_rank))
        model = Ner.from_pretrained(bert_model,
                  cache_dir=cache_dir,
                  num_labels = num_labels)
        if fp16:
            model.half()
        # model.cuda()
        model.to(device)
        if local_rank != -1:
            try:
                from apex.parallel import DistributedDataParallel as DDP
            except ImportError:
                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")

            model = DDP(model)
        elif n_gpu > 1:
            model = torch.nn.DataParallel(model)

        param_optimizer = list(model.named_parameters())
        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
        optimizer_grouped_parameters = [
            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
             ]
        if fp16:
            try:
                from apex.optimizers import FP16_Optimizer
                from apex.optimizers import FusedAdam
            except ImportError:
                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")

            optimizer = FusedAdam(optimizer_grouped_parameters,
                                  lr=learning_rate,
                                  bias_correction=False,
                                  max_grad_norm=1.0)
            if loss_scale == 0:
                optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
            else:
                optimizer = FP16_Optimizer(optimizer, static_loss_scale=loss_scale)

        else:
            optimizer = BertAdam(optimizer_grouped_parameters,
                                 lr=learning_rate,
                                 warmup=warmup_proportion,
                                 t_total=num_train_optimization_steps)

        global_step = 0
        nb_tr_steps = 0
        tr_loss = 0
        label_map = {i : label for i, label in enumerate(label_list,1)}
        if do_train:
            train_features = convert_examples_to_features(
                train_examples, label_list, max_seq_length, tokenizer)
            logger.info("***** Running training *****")
            logger.info("  Num examples = %d", len(train_examples))
            logger.info("  Batch size = %d", train_batch_size)
            logger.info("  Num steps = %d", num_train_optimization_steps)
            all_input_ids = torch.tensor([f.input_ids for f in train_features], dtype=torch.long)
            all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)
            all_segment_ids = torch.tensor([f.segment_ids for f in train_features], dtype=torch.long)
            all_label_ids = torch.tensor([f.label_id for f in train_features], dtype=torch.long)
            all_valid_ids = torch.tensor([f.valid_ids for f in train_features], dtype=torch.long)
            all_lmask_ids = torch.tensor([f.label_mask for f in train_features], dtype=torch.long)
            train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids,all_valid_ids,all_lmask_ids)
            if local_rank == -1:
                train_sampler = RandomSampler(train_data)
            else:
                train_sampler = DistributedSampler(train_data)
            train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=train_batch_size)

            model.train()
            for _ in trange(int(num_train_epochs), desc="Epoch"):
                tr_loss = 0
                nb_tr_examples, nb_tr_steps = 0, 0
                for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")):
                    batch = tuple(t.to(device) for t in batch)
                    input_ids, input_mask, segment_ids, label_ids, valid_ids,l_mask = batch
                    loss = model(input_ids, segment_ids, input_mask, label_ids,valid_ids,l_mask)
                    del loss
                    if n_gpu > 1:
                        loss = loss.mean() # mean() to average on multi-gpu.
                    if gradient_accumulation_steps > 1:
                        loss = loss / gradient_accumulation_steps

                    if fp16:
                        optimizer.backward(loss)
                    else:
                        loss.backward()

                    tr_loss += loss.item()
                    nb_tr_examples += input_ids.size(0)
                    nb_tr_steps += 1
                    if (step + 1) % gradient_accumulation_steps == 0:
                        if fp16:
                            # modify learning rate with special warm up BERT uses
                            # if args.fp16 is False, BertAdam is used that handles this automatically
                            lr_this_step = learning_rate * warmup_linear(global_step/num_train_optimization_steps, warmup_proportion)
                            for param_group in optimizer.param_groups:
                                param_group['lr'] = lr_this_step
                        optimizer.step()
                        optimizer.zero_grad()
                        global_step += 1

main('','',-1,True,True,8,1,42,True,True,'jpt','ner','data/',True,'bert-base-cased',5,'cache_dir',5e-5,0.4,0,128)

Output:
This worked for me:

import os, sys, shutil
import time
import gc
from contextlib import contextmanager
from pathlib import Path
import random
import numpy as np, pandas as pd
from tqdm import tqdm, tqdm_notebook

@contextmanager
def timer(name):
    t0 = time.time()
    yield
    print(f'[{name}] done in {time.time() - t0:.0f} s')

USE_APEX = True

if USE_APEX:
            with timer('install Nvidia apex'):
                # Installing Nvidia Apex
                os.system('git clone https://github.com/NVIDIA/apex; cd apex; pip install -v --no-cache-dir' +
                          ' --global-option="--cpp_ext" --global-option="--cuda_ext" ./')
                os.system('rm -rf apex/.git') # too many files, Kaggle fails
                from apex import amp


Instruction: Multivariate input LSTM in pytorch

Input:
I would like to implement LSTM for multivariate input in Pytorch.

Following this article https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/ which uses keras, the input data are in shape of (number of samples, number of timesteps, number of parallel features)

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
. . .
Input     Output
[[10 15]
 [20 25]
 [30 35]] 65
[[20 25]
 [30 35]
 [40 45]] 85
[[30 35]
 [40 45]
 [50 55]] 105
[[40 45]
 [50 55]
 [60 65]] 125
[[50 55]
 [60 65]
 [70 75]] 145
[[60 65]
 [70 75]
 [80 85]] 165
[[70 75]
 [80 85]
 [90 95]] 185

n_timesteps = 3
n_features = 2

In keras it seems to be easy:

model.add(LSTM(50, activation='relu', input_shape=(n_timesteps, n_features)))

Can it be done in other way, than creating n_features of LSTMs as first layer and feed each separately (imagine as multiple streams of sequences) and then flatten their output to linear layer?

I'm not 100% sure but by nature of LSTM the input cannot be flattened and passed as 1D array, because each sequence "plays by different rules" which the LSTM is supposed to learn.

So how does such implementation with keras equal to PyTorch
input of shape (seq_len, batch, input_size)(source https://pytorch.org/docs/stable/nn.html#lstm)

Edit:

  Can it be done in other way, than creating n_features of LSTMs as first layer and feed each separately (imagine as multiple streams of sequences) and then flatten their output to linear layer?

According to PyTorch docs the input_size parameter actually means number of features (if it means number of parallel sequences)

Output:
I hope that problematic parts are commented to make sense:
Data preparation
import random
import numpy as np
import torch

# multivariate data preparation
from numpy import array
from numpy import hstack

# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
    X, y = list(), list()
    for i in range(len(sequences)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(sequences):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

# define input sequence
in_seq1 = array([x for x in range(0,100,10)])
in_seq2 = array([x for x in range(5,105,10)])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
# convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
# horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))

Multivariate LSTM Network
class MV_LSTM(torch.nn.Module):
    def __init__(self,n_features,seq_length):
        super(MV_LSTM, self).__init__()
        self.n_features = n_features
        self.seq_len = seq_length
        self.n_hidden = 20 # number of hidden states
        self.n_layers = 1 # number of LSTM layers (stacked)

        self.l_lstm = torch.nn.LSTM(input_size = n_features,
                                 hidden_size = self.n_hidden,
                                 num_layers = self.n_layers,
                                 batch_first = True)
        # according to pytorch docs LSTM output is
        # (batch_size,seq_len, num_directions * hidden_size)
        # when considering batch_first = True
        self.l_linear = torch.nn.Linear(self.n_hidden*self.seq_len, 1)

    def init_hidden(self, batch_size):
        # even with batch_first = True this remains same as docs
        hidden_state = torch.zeros(self.n_layers,batch_size,self.n_hidden)
        cell_state = torch.zeros(self.n_layers,batch_size,self.n_hidden)
        self.hidden = (hidden_state, cell_state)

    def forward(self, x):
        batch_size, seq_len, _ = x.size()

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)
        # lstm_out(with batch_first = True) is
        # (batch_size,seq_len,num_directions * hidden_size)
        # for following linear layer we want to keep batch_size dimension and merge rest
        # .contiguous() -> solves tensor compatibility error
        x = lstm_out.contiguous().view(batch_size,-1)
        return self.l_linear(x)

Initialization
n_features = 2 # this is number of parallel inputs
n_timesteps = 3 # this is number of timesteps

# convert dataset into input/output
X, y = split_sequences(dataset, n_timesteps)
print(X.shape, y.shape)

# create NN
mv_net = MV_LSTM(n_features,n_timesteps)
criterion = torch.nn.MSELoss() # reduction='sum' created huge loss value
optimizer = torch.optim.Adam(mv_net.parameters(), lr=1e-1)

train_episodes = 500
batch_size = 16

Training
mv_net.train()
for t in range(train_episodes):
    for b in range(0,len(X),batch_size):
        inpt = X[b:b+batch_size,:,:]
        target = y[b:b+batch_size]

        x_batch = torch.tensor(inpt,dtype=torch.float32)
        y_batch = torch.tensor(target,dtype=torch.float32)

        mv_net.init_hidden(x_batch.size(0))
    #    lstm_out, _ = mv_net.l_lstm(x_batch,nnet.hidden)
    #    lstm_out.contiguous().view(x_batch.size(0),-1)
        output = mv_net(x_batch)
        loss = criterion(output.view(-1), y_batch)

        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    print('step : ' , t , 'loss : ' , loss.item())

Results
step :  499 loss :  0.0010267728939652443 # probably overfitted due to 500 training episodes


Instruction: Does batch normalisation work with a small batch size?

Input:
I'm using batch normalization with a batch size of size 10 for face detection, I wanted to know if it is better to remove the batch norm layers or keep them.
And if it is better to remove them what can I use instead?

Output:
Yes, it works for the smaller size, it will work even with the smallest possible size you set.

The trick is the bach size also adds to the regularization effect, not only the batch norm.
I will show you few pics:

We are on the same scale tracking the bach loss. The left-hand side is a module without the batch norm layer (black), the right-hand side is with the batch norm layer.
Note how the regularization effect is evident even for the bs=10.

When we set the bs=64 the batch loss regularization is super evident. Note the y scale is always [0, 4].

My examination was purely on nn.BatchNorm1d(10, affine=False) without learnable parameters gamma and beta i.e. w and b.

This is why when you have low batch size, it has sense to use the BatchNorm layer.


Instruction: ModuleNotFoundError: No module named 'tools.nnwrap'

Input:
I tried to install torch using:
pip install torch

Installation started, but after a few seconds I got the error:
from tools.nnwrap import generate_wrappers as generate_nn_wrappers

ModuleNotFoundError: No module named 'tools.nnwrap'

OS: Windows

Output:
Anyone who is looking for the solution refer below:
It seems command to install torch not is working as expected, instead, you can try to install PyTorch using below command.
It's working and solved my above-mentioned issue.
Run below command(for below-specified OS, package-manager, Language):
# for OS: Windows, package-manager: pip, Language: python3.6 (below command is valid for only mentioned python 3.6)

pip3 install https://download.pytorch.org/whl/cu90/torch-1.1.0-cp36-cp36m-win_amd64.whl
pip3 install https://download.pytorch.org/whl/cu90/torchvision-0.3.0-cp36-cp36m-win_amd64.whl

For another version/type of the software (OS, package, Language) installed, the command must be generated from the below-mentioned link.
https://pytorch.org/get-started/locally/
Also, look for the Python version in your IDE(If you are using PyCharm) from the terminal using the command: python. If it returns 32bit this could happen, instead install Python 64-bit.


Instruction: Image data cannot be converted to float

Input:
I have a code for predicting dog breed after training on CNN model, I get the class index from the below function. I want to display an random image from the class idx folder obtained from the function.
    class_name = [item for item in loaders['train'].dataset.classes]

      def predict_dog_breed(img,model,class_names):
           image = Image.open(img).convert('RGB')
           transform = transforms.Compose([
                          transforms. RandomResizedCrop(224),
                          transforms.ToTensor(),
                         transforms.Normalize(mean=[0.485,0.456,0.406],
                         std=[0.229, 0.224, 0.225])])
           image = transform(image)
           test_image = image.unsqueeze(0)
           net.eval()
           output = net(test_image)
          idx = torch.argmax(output)
          a = random.choice(os.listdir("./dogImages/train/{}/".format (class_name[idx])))
          imshow(a)
          return class_name[idx]

When I tried to display the random image, I am getting the below error:

TypeError Traceback (most recent call last) in 1 for img_file in os.listdir('./images'): 2 image = os.path.join('./images', img_file) ----> 3 dog_or_human(image)
in dog_or_human(img) 5 plt.show() 6 if dog_detector(img) == True: ----> 7 predict_dog = predict_dog_breed(img, net, class_name) 8 print("Dog Detected!The breed is {}".format(predict_dog)) 9 elif face_detector(img) > 0:
in predict_dog_breed(img, model, class_name) 18 a = random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx]))) 19 print(a) ---> 20 imshow(a) 21 #subdir = ''.join(["/dogImages/train/", class_name[idx]]) 22 #print(file)
~/Library/Python/3.7/lib/python/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs) 2697 filternorm=filternorm, filterrad=filterrad, imlim=imlim, 2698 resample=resample, url=url, **({"data": data} if data is not -> 2699 None else {}), **kwargs) 2700 sci(__ret) 2701 return __ret
~/Library/Python/3.7/lib/python/site-packages/matplotlib/init.py in inner(ax, data, *args, **kwargs) 1808 "the Matplotlib list!)" % (label_namer, func.name), 1809 RuntimeWarning, stacklevel=2) -> 1810 return func(ax, *args, **kwargs) 1811 1812 inner.doc = _add_data_doc(inner.doc,
~/Library/Python/3.7/lib/python/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs) 5492 resample=resample, **kwargs) 5493 -> 5494 im.set_data(X) 5495 im.set_alpha(alpha) 5496 if im.get_clip_path() is None:
~/Library/Python/3.7/lib/python/site-packages/matplotlib/image.py in set_data(self, A) 632 if (self._A.dtype != np.uint8 and 633 not np.can_cast(self._A.dtype, float, "same_kind")): --> 634 raise TypeError("Image data cannot be converted to float") 635 636 if not (self._A.ndim == 2
TypeError: Image data cannot be converted to float

Any help on this would be appreciated!

Output:
So, I tried to reproduced the error in your code here and was successful in doing that. You are getting error because of these lines in your code:

a = random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx])))
imshow(a)

random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx]))) basically returns an image filename, which is a string. You are not reading the image, just passing the filename to imshow function, which is incorrect. Check below figures for clarification.

Code with error:

Code without error:

Hence, change your predict_do_breed function to following:

def predict_dog_breed(img,model,class_name):
    image = Image.open(img).convert('RGB')
    transform = transforms.Compose([transforms.RandomResizedCrop(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                            std=[0.229, 0.224, 0.225])])
    image = transform(image)
    test_image = image.unsqueeze(0)
    net.eval()
    output = net(test_image)
    idx = torch.argmax(output)
    a = random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx])))
    print(a)
    img = cv2.imread("./dogImages/train/{}/".format(class_name[idx])+a)
    imshow(img)
    return class_name[idx]

In the above code, cv2.imread function has been used to read the image filename, outputted by random.choice(os.listdir("./dogImages/train/{}/".format(class_name[idx]))).


Instruction: PyTorch batch Size suddenly reduced after n epochs

Input:
I have a pytorch nn model running on win 10 cpu.
batch size is 42
After 67 iterations, a strange thing happens: batch size is suddenly reduced to 28, and I get

RuntimeError: Expected hidden[0] size (1, 28, 256), got (1, 42, 256)

Output:
Is it possible the number of training examples in the dataset is not divisible by 42? Could it be that the reminder is 28?

If your model cannot handle online change of batch size, you should consider setting drop_last=True in your torch.utils.data.DataLoader, thus only full batches will be processed during training.


Instruction: How to fix "“TypeError: img should be PIL Image. Got ”?

Input:
I am a beginner and I am learning to code an image classifier. My goal is to create a predict function.

Any suggestion to fix it?

In this project, I want to use the predict function to recognize different flower species. So I could check their labels later.

Attempt to fix: I have already used the unsqueeze_(0) method and changing from numpy to torch method . I usually get the same error message shown below:

  TypeError: img should be PIL

Code:

    # Imports here
    import pandas as pd
    import numpy as np

    import torch
    from torch import nn
    from torchvision import datasets, transforms, models
    import torchvision.models as models
    import torch.nn.functional as F
    import torchvision.transforms.functional as F
    from torch import optim
    import json

    from collections import OrderedDict
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns
    %matplotlib inline
    from PIL import Image

    def process_image(image):
     #Scales, crops, and normalizes a PIL image for a PyTorch model,
            #returns an Numpy array
        # Process a PIL image for use in a PyTorch model
        process = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
        ])
        image = process(image)
        return image

    # Predict
    #Predict the class (or classes) of an image using a trained deep learning model.
    def predict(image, model, topk=5):

        img = process_image(image)
        img = img.unsqueeze(0)

        output = model.forward(img)
        probs, labels = torch.topk(output, topk)
        probs = probs.exp()

        # Reverse the dict
        idx_to_class = {val: key for key, val in model.class_to_idx.items()}
        # Get the correct indices
        top_classes = [idx_to_class[each] for each in classes]

        return labels, probs

    #Passing
    probs, classes = predict(image, model)
    print(probs)
    print(classes)

Error:

TypeError                                 Traceback (most recent call last)
<ipython-input-92-b49fdcab5791> in <module>()
----> 1 probs, classes = predict(image, model)
     2 print(probs)
     3 print(classes)

<ipython-input-91-05809355bfe0> in predict(image, model, topk)
     2     ‘’' Predict the class (or classes) of an image using a trained deep learning model.
     3     ‘’'
----> 4     img = process_image(image)
     5     img = img.unsqueeze(0)
     6

<ipython-input-20-02663a696e34> in process_image(image)
    11                              std=[0.229, 0.224, 0.225])
    12     ])
---> 13     image = process(image)
    14     return image

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/transforms.py in __call__(self, img)
    47     def __call__(self, img):
    48         for t in self.transforms:
---> 49             img = t(img)
    50         return img
    51

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/transforms.py in __call__(self, img)
   173             PIL Image: Rescaled image.
   174         “”"
--> 175         return F.resize(img, self.size, self.interpolation)
   176
   177     def __repr__(self):

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/functional.py in resize(img, size, interpolation)
   187     “”"
   188     if not _is_pil_image(img):
--> 189         raise TypeError(‘img should be PIL Image. Got {}’.format(type(img)))
   190     if not (isinstance(size, int) or (isinstance(size, collections.Iterable) and len(size) == 2)):
   191         raise TypeError(‘Got inappropriate size arg: {}’.format(size))

TypeError: img should be PIL Image. Got <class ‘str’>

All I want is to get these similar result. Thank you!

    predict(image,model)
    print(probs)
    print(classes)

    tensor([[ 0.5607,  0.3446,  0.0552,  0.0227,  0.0054]], device='cuda:0')
    tensor([[  8,   1,  31,  24,   7]], device='cuda:0')

Output:
You are getting the above error because of the below line in predict function:

img = process_image(image)

The input to the process_image function should be Image.open(image), not image which is basically the path to an image(string) and hence the error message TypeError: img should be PIL Image. Got <class ‘str’>.

So, change img = process_image(image) to img = process_image(Image.open(image))

Modified predict function:

def predict(image, model, topk=5):
    '''
      Predict the class (or classes) of an image using a trained deep learning model.
      Here, image is the path to an image file, but input to process_image should be
      Image.open(image)
    '''
    img = process_image(Image.open(image))
    img = img.unsqueeze(0)

    output = model.forward(img)
    probs, labels = torch.topk(output, topk)
    probs = probs.exp()

    # Reverse the dict
    idx_to_class = {val: key for key, val in model.class_to_idx.items()}
    # Get the correct indices
    top_classes = [idx_to_class[each] for each in classes]

    return labels, probs


Instruction: Looping over pytorch LSTM

Input:
I am training a seq2seq model for machine translation in pytorch. I would like to gather the cell state at every time step, while still having the flexibility of multiple layers and bidirectionality, that you can find in the LSTM module of pytorch, for example.

To this end, I have the following encoder and forward method, where I loop over the LSTM module. The problem is, that the model does not train very well. Right after the loop terminates, you can see the normal way to use the LSTM module and with that, the model trains.

So, is the loop not a valid way to do this?

class encoder(nn.Module):
    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
        super().__init__()

        self.input_dim = input_dim
        self.emb_dim = emb_dim
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.dropout = dropout

        self.embedding = nn.Embedding(input_dim, emb_dim)

        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)

        self.dropout = nn.Dropout(dropout)

    def forward(self, src):

        #src = [src sent len, batch size]

        embedded = self.dropout(self.embedding(src))

        #embedded = [src sent len, batch size, emb dim]
        hidden_all = []

        for i in range(len(embedded[:,1,1])):
            outputs, hidden = self.rnn(embedded[i,:,:].unsqueeze(0))
            hidden_all.append(hidden)

        #outputs, hidden = self.rnn(embedded)

        #outputs = [src sent len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]
        None
        #outputs are always from the top hidden layer

        return hidden

Output: Okay, so the fix is very simple, you can just run the first timestep outside, to get a hidden tuple to input in the LSTM module.


Instruction: Multiply every row of a matrix with every row of another matrix

Input:
In numpy / PyTorch, I have two matrices, e.g. X=[[1,2],[3,4],[5,6]], Y=[[1,1],[2,2]]. I would like to dot product every row of X with every row of Y, and have the results

[[3, 6],[7, 14], [11,22]]

How do I achieve this?, Thanks!

Output:
I think this is what you are looking for:

import numpy as np

x= [[1,2],[3,4],[5,6]]
y= [[1,1],[2,2]]

x = np.asarray(x) #convert list to numpy array
y = np.asarray(y) #convert list to numpy array

product = np.dot(x, y.T)

.T transposes the matrix, which is neccessary in this case for the multiplication (because of the way dot products are defined). print(product) will output:

[[ 3  6]
 [ 7 14]
 [11 22]]


Instruction: Copy a chunk of one tensor into another one in C++ API

Input:
I need to copy a row of one tensor (in c++ API) into some part of another tensor, form which the begin and end indexes are available. Within C++ we can use something like:

int myints[] = {10, 20, 30, 40, 50, 60, 70};
std::vector<int> myvector(18);

std::copy(myints, myints + 3, myvector.begin() + 4);

to copy three values from myints into myvector, starting at the fourth index.
I was wondering if there is a similar API in libtorch (i.e., C++)?

Output:
The C++ API provides the Python slice equivalent function

at::Tensor at::Tensor::slice(int64_t dim, int64_t start, int64_t end, int64_t step);

You can thus do something like:

auto myints = torch::Tensor({10, 20, 30, 40, 50, 60, 70});
auto myvector = torch::ones({18});

myvector.slice(0, 3, 7) = myints.slice(0, 0, 3);

in your case using dim=0 first dimension


Instruction: Why there's no .train() method in this Pytorch official tutorial?

Input:
So, I just learned Pytorch and they say that you must put the NN in train mode by .train() method, then while inferencing .eval() mode. I'm reading this tutorial and there's no .train() at all. Why is that?

https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

Output:
.train() sets the self.training property of the module to True. .eval() sets it to False.
As can be seen in the source for nn.Module, this property is initially set to True. So, unless you have called eval() before you start training, you do not necessarily (as per the current implementation) need to call train(). But the important point is that the module should be in the self.training=True state at the time of training, so it is probably a good practice to do so anyway.
Also, currently, only some modules (such as dropout and batchnorm) change their behavior based on the self.training property. So you don't necessarily have to call .train() and .eval() if you are not using those specific modules, but again, it is probably good practice to do so anyway for future-proofing your code.


Instruction: Pytorch - how do I index a 2-D matrix row wise?

Input:
I would like to index a 2-D matrix row-wise and re-assign values.

For example, first consider a 1-D vector case where we have three 1-D tensors t1, indexes, t2 with the same shape. We can do this indexing and re-assignment as follows:

indexes = torch.tensor([0, 2, 1, 3])
t1 = torch.tensor([0.0, 0.0, 0.0, 0.0])
t2 = torch.tensor([0.1, 0.2, 0.3, 0.4])

t1[indexes] = t2

Now, say that t1, indexes, t2 are 2-D matrices instead of 1-D vectors and have the same shape (R X C). I would like to do similar indexing as above for every row in these matrices where:

for i in range(R):
    t1[i][indexes[i]] = t2[i]

I would like to vectorize this operation instead of using a for loop. How do I do this?

Output:
Similiar to @Anubhav's answer with a slight change in dimension of scatter_, this did the job. Source: PyTorch Discussion

indexes = torch.tensor([[0, 2, 1, 3],
                        [1, 0, 3, 2]])
t1 = torch.zeros_like(indexes).float()
t2 = torch.tensor([[0.1, 0.2, 0.3, 0.4],
                   [0.5, 0.6, 0.7, 0.8]])
t1.scatter_(1, indexes, t2)


Instruction: Why does the pytorch Docker image not come with torch?

Input:
I am trying to build a docker container that will enable me to run my code that requires the python torch module.

I have chosen to start my image from the pytorch/pytorch:latest base image and add a few required modules manually. The build, push, and pull to the remote server were successful (no error messages at least).

Currently my Dockerfile looks like this:

FROM pytorch/pytorch:latest

RUN apt-get update \
     && apt-get install -y \
        libgl1-mesa-glx \
        libx11-xcb1 \
     && apt-get clean all \
     && rm -r /var/lib/apt/lists/*

RUN /opt/conda/bin/conda install --yes \
    astropy \
    matplotlib \
    pandas \
    glob2 \
    PIL \
    scikit-learn \
    scikit-image \
    numpy

However, when running my python script within the container, I get ImportError: No module named torch. This strikes me as rather weird, as it leads me to assume that the pytorch base image does not include the torch module...?

I have nevertheless tried to add torch to the list of modules to install in the Dockerfile, but then the initial build will already fail with the error message PackagesNotFoundError: The following packages are not available from current channels:  - torch. Following the advice given here did unfortunately not help me.

I'd appreciate any explanation as to why the torch module could not be found within the built container, and of course any help to fix this problem! Thanks!

Output:
First thing, your assumption is false, to verify this you just need to run the container from the base image, as you can check the official Dockerfile  or run first the base image that is pytorch/pytorch:latest
and verify does the base image working as you need?
Here is the list of installed modules in the official image and at bottom of the list, you can see the torch.

Here is a simple example from the torch using the base image.

As for your Dockerfile, so the package PIL is breaking the docker build from scratch, but this not visible if PyTorch is the base image.

For some reason, I am failed in the child image to find the torch so installed it using pip install and then its able to work.

Here is the Dockerfile:
FROM pytorch/pytorch:latest

RUN apt-get update \
     && apt-get install -y \
        libgl1-mesa-glx \
        libx11-xcb1 \
     && apt-get clean all \
     && rm -r /var/lib/apt/lists/*

RUN /opt/conda/bin/conda install --yes \
    astropy \
    matplotlib \
    pandas \
    glob2 \
    scikit-learn \
    scikit-image \
    numpy \
    torch

Updated
Here is the way to make torch available
FROM pytorch/pytorch:latest

RUN apt-get update \
     && apt-get install -y \
        libgl1-mesa-glx \
        libx11-xcb1 \
     && apt-get clean all \
     && rm -r /var/lib/apt/lists/*

RUN /opt/conda/bin/conda install --yes \
    astropy \
    matplotlib \
    pandas \
    scikit-learn \
    scikit-image

RUN pip install torch


Instruction: How can torch multiply two 10000*10000 matrices in almost zero time? Why does the speed change so much from 349 ms down to 999 µs?

Input:
Here is an excerpt from Jupyter:
In [1]:
import torch, numpy as np, datetime
cuda = torch.device('cuda')

In [2]:
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))

Wall time: 349 ms
tensor(17.0374, device='cuda:0') tensor(17.0376, device='cuda:0')

The time is low but still reasonable (0.35 sec for 1e12 multiplications)
But if we repeat the same:
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))

Wall time: 999 µs
tensor(-78.7172, device='cuda:0') tensor(-78.7173, device='cuda:0')

1e12 multiplications in 1ms?!
Why did the time change from 349ms to 1ms?
Info:

Tested on GeForce RTX 2070;
Can be reproduced on Google Colab.

Output:
There is already a discussion about this on Discuss PyTorch: Measuring GPU tensor operation speed.
I'd like to highlight two comments from that thread:

From @apaszke:

[...] the GPU executes all operations asynchronously, so you need to insert proper barriers for your benchmarks to be correct

From @ngimel:

I believe cublas handles are allocated lazily now, which means that first operation requiring cublas will have an overhead of creating cublas handle, and that includes some internal allocations. So there’s no way to avoid it other than calling some function requiring cublas before the timing loop.

Basically, you have to synchronize() to have a proper measurement:
import torch

x = torch.randn(10000, 10000).to("cuda")
w = torch.randn(10000, 10000).to("cuda")
# ensure that context initialization finish before you start measuring time
torch.cuda.synchronize()

%time y = x.mm(w.t()); torch.cuda.synchronize()

CPU times: user 288 ms, sys: 191 ms, total: 479 ms
Wall time: 492 ms

x = torch.randn(10000, 10000).to("cuda")
w = torch.randn(10000, 10000).to("cuda")
# ensure that context initialization finish before you start measuring time
torch.cuda.synchronize()

%time y = x.mm(w.t()); torch.cuda.synchronize()

CPU times: user 237 ms, sys: 231 ms, total: 468 ms
Wall time: 469 ms


Instruction: Reinforcement Learning with Pytorch. [Error: KeyError ]

Input:
I am new to reinforcement learning as well as pytorch. I am learning it from Udemy. However, the code I have is the same as it is shown but I am having an error. I guess it is a pytorch error but can't debug it. If anyone helps it is really appreciated.

import gym
import time
import torch
import matplotlib.pyplot as plt
from gym.envs.registration import register
register(id='FrozenLakeNotSlippery-v0',entry_point='gym.envs.toy_text:FrozenLakeEnv',kwargs={'map_name': '4x4', 'is_slippery':False})
env = gym.make('FrozenLakeNotSlippery-v0')
number_of_states = env.observation_space.n
number_of_actions = env.action_space.n
Q = torch.zeros([number_of_states,number_of_actions])
num_episodes = 1000
steps_total = []
gamma = 1
for i in range(num_episodes):
    state = env.reset()
    step = 0
    while True:
        step += 1
        #action = env.action_space.sample()
        random_values = Q[state]+torch.rand(1,number_of_actions)/1000
        action = torch.max(random_values,1)[1][0]
        new_state, reward, done, info = env.step(action)
        Q[state, action] = reward + gamma * torch.max(Q[new_state])
        state = new_state
        #time.sleep(0.4)
        #env.render()
        if done:
            steps_total.append(step)
            print ("Episode Finished after %i steps" %step)
            break

print ("Average Num Steps: %2f" %(sum(steps_total)/num_episodes))
plt.plot(steps_total)
plt.show()

The error I am having is the below one

KeyError                                  Traceback (most recent call last)
<ipython-input-11-a6aa419c3767> in <module>
  8         random_values = Q[state]+torch.rand(1,number_of_actions)/1000
  9         action = torch.max(random_values,1)[1][0]
---> 10         new_state, reward, done, info = env.step(action)
 11         Q[state, action] = reward + gamma * torch.max(Q[new_state])
 12         state = new_state

c:\users\souradip\appdata\local\programs\python\python36\lib\site-packages\gym\envs\toy_text\discrete.py in step(self, a)
 53
 54     def step(self, a):
---> 55         transitions = self.P[self.s][a]
 56         i = categorical_sample([t[0] for t in transitions], self.np_random)
 57         p, s, r, d= transitions[i]

KeyError: tensor(3)

Output:
The below code

action = torch.max(random_values,1)[1][0]

results in a 0-dim tensor, but env.step() expects a python number, which is basically an action from the action space. So, as @a_guest mentioned in the comment, use a.item() to convert a 0-dim tensor to a python number like below:

new_state, reward, done, info = env.step(action.item())


Instruction: python - Machine Learning 2D Regression with confidence bands

Input:
I have measures of a variable versus time.
I want to obtain a regression with confidence bands so that the plot sounds like this:

Given an arbitrary x, I "predicted" the y and its confidence by evaluating mean and std of N closest measured points. But I feel like this is quite naive.

I was wondering if it is possible to obtain a similar result with Machine Learning, Neural Network, ...

I'm quite familiar with python, sklearn and pytorch so if you please could suggest a solution that implements those tools I would be very thankfull.

Output:
You can achieve something like that with Gaussian Processes. For regression problems, you can use GaussianProcessRegressor from sklearn.

This is an example of how to use it to obtain the plot you are looking for.


Instruction: Combination of torch.cat and torchvision.transforms leeds to zero tensor

Input:
I want to add some more information to an image in a fourth layer of a tensor which first three layers are based on an image. Afterwards I want to cut a peace out of an image (data augmentation) and have to resize the image to a given size.

For this I created a tensor from a picture and joined it with a tensor with one layer of additional information using torch.cat. (Almost but not all entries of the second tensor have been zeros.)

I sent the result through a transforms.compose (to cut and resize the tensor) But after that the tensor completely consisted out of zeros.

Here I have built a reproducible example.

import torch
from torchvision import transforms

height = 2
width = 4
resize = 2
tensor3 = torch.rand(3,height,width)
tensor1 = torch.zeros(1,height,width)
#tensor1 = torch.rand(1,height,width)

imageToTensor = transforms.ToTensor()
tensorToImage = transforms.ToPILImage()

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(resize, scale=(0.9, 1.0)),
    transforms.ToTensor(),
])

tensor4 = torch.cat((tensor3,tensor1),0)

image4 = tensorToImage(tensor4)
transformed_image4 = train_transform(image4)

print(tensor4)
print(transformed_image4)

tensor([[[0.6774, 0.5293, 0.4420, 0.2463],
         [0.1391, 0.7481, 0.3436, 0.9391]],

        [[0.0652, 0.2061, 0.2931, 0.6126],
         [0.2618, 0.3506, 0.5095, 0.7351]],

        [[0.8555, 0.6320, 0.9461, 0.0928],
         [0.2094, 0.3944, 0.0528, 0.7900]],

        [[0.0000, 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000, 0.0000]]])

tensor([[[0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.]],

        [[0., 0.],
         [0., 0.]]])

If I choose “tensor1 = torch.rand(1,height,width)” I do not have this problem. But if most entries are zero I have.
With scale=(0.5, 1.0) I don't have the problem either.

No some questions:

How may I get the first three layers just resized with non zero entries?
Did I misunderstand something, or is it really weird?

Output:
I created an issue:

https://github.com/pytorch/pytorch/issues/22611

And the answer was that only PIL-Images are supported in Torchvision.

An alternative is the albumentations-library for transformations.


Instruction: LSTM model implementation in PyTorch

Input: I was trying to implement CNN+LSTM model in PyTorch, but I have problem with LSTM part (I never used LSTM before). Could you write Many-to-one-LSTM model class (Image-link: https://i.ibb.co/SRGWT5j/lstm.png  )...

Output:
For nn.LSTM in Pytorch , as per docs https://pytorch.org/docs/stable/nn.html?highlight=lstm#torch.nn.LSTM

it takes input as (embedding_size_dimension , hidden_size_dimension , number_of_layers)
(currently ignoring bidirectional parameter , we can also pass initial hidden_state and cell_state )

so we need to pass a tensor of shape [max sentence length , batch size , embedding size ]

just a sample model

class Model(nn.Module):
    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        super(Model, self).__init__()
        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=drop_prob)

    def forward(self, sentence):
        batch_size = sentence.size(0)
        sentence = sentence.long()
        embeds = self.embedding(sentence)
        lstm_out, hidden = self.lstm(embeds)
        # so here lstm_out will be of [max sentence length , batch size , hidden size]
        # so for simple many-to-one we can just use output of last cell of LSTM
        out = lstm_out[-1,:,:]
        return out

You can refer this link , is has really nicely explained about LSTM in pytorch , it also has one sample example of SentimentNet model

https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/


Instruction: No matching distribution found for numpy (from torch==1.10)

Input:
Working on Ubuntu 18.04 in a VM.

I have python3 installed at /usr/bin/python3.6 according to whereis python3.
When I run pip3 freeze, I see numpy installed, with version 1.15.4, and when I enter into python3 console, I can import numpy as np perfectly fine with no errors.

What I am trying to do now is to install PyTorch, specifically torch==1.1.0, without GPU / CUDA.
I got the following installation instructions from https://pytorch.org/get-started/locally/:

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
pip3 install https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl

I downloaded these wheel files, and tried to run them from local directory as such:

/usr/bin/python3.6 -m pip install ~/Desktop/torch-1.1.0-cp36-cp36m-linux_x86_64.whl

and got the following error:

No matching distribution found for numpy (from torch==1.1.0)

I don't understand why this is the case, cause my numpy library is clearly already installed and working (READ ABOVE: I am able to import numpy in python3 console). Also, I had no issues installing PyTorch with GPU on my Windows machine, with torch 1.1.0 and numpy 1.15.4.

Alternatively, I tried to build PyTorch from source by moving into the PyTorch source directory and executing /usr/bin/python3 setup.py install, but it didn't work since I am not using a GPU, and CUDA is not installed. I am not able to find instructions on how I can build PyTorch from source without GPU in a Linux environment?

Can anyone kindly advise? Thank you!

Some additional information
(1) Tried to install PyTorch with the --no-dependencies flag as suggested by @Jan, and it installed, but I am not sure if Torch will really work perfectly.
(2) which pip3 points to /usr/bin/pip3
(3) pip3 show numpy tells me that the location is at /home/name/.local/lib/python3.6/site-packages

Output: If you are sure that all dependencies are manually installed, you could try --no-dependencies flag for pip3. Otherwise, I would point you to Anaconda as python environment provider, it is pretty convenient.


Instruction: PyTorch broadcasting: how this worked?

Input:
I'm new to Deep Learning. I'm studying from Udacity.

I came across one of the codes to build up a neural network, where 2 tensors are being added, specifically the 'bias' tensor with the output of the tensor-multiplication product.

It was kind of...

def activation(x):
return (1/(1+torch.exp(-x)))

inputs = images.view(images.shape[0], -1)
w1 = torch.randn(784, 256)
b1 = torch.randn(256)
h = activation(torch.mm(inputs,w1) + b1)

After flattening the MNIST, it came out as [64,784] (inputs).

I'm not getting how the bias tensor (b1) of dimension [256], could be added to the multiplication product of 'inputs' and 'w1' which comes out to be the dimensions of [256, 64].

Output:
In simple terms, whenever we use "broadcasting" from a Python library (Numpy or PyTorch), what we are doing is treating our arrays (weight, bias) dimensionally compatible.

In other words, if you are operating with W of shape [256,64], and your bias is only [256]. Then, broadcasting will complete that lacking dimension.

As you can see in the image above, the dimension left is being filled so that our operations can be done successfully. Hope this is helpful


Instruction: Apply preprocessing to the dataset

Input:
I am implementing a paper on image segmentation in pytorch. I am required to do some preprocessing steps but as I am trying it the first time so I am unable to incorporate them in the traditional pipeline.
Following are the preprocessing steps-

1)   N(w, h) = I(w, h) − G(w, h), (1)
where N is the normalized image, I is the original image, and G is the Gaussian blurred image with kernel size 65*65 and 0 mean and standard deviation 10.

2)Normalizing the mean image and dividing each pixel by average standard deviation.

Following is my code snippet for the above steps-

def gaussian_blur(img):
    image = cv2.GaussianBlur(image,(65,65),10)
    new_image = img - image
return image

def normalise(img):
    img_normalised = np.empty(img.shape)
    img_std = np.std(img)
    img_mean = np.mean(img)
    img_normalized = (img-img_mean)/imgs_std

    for i in range(img.shape[1]):
        img_normalized[i] = (img_normalized -
         np.mean(img_normalized))/np.std(img_normalized)
return img_normalized

I am really not sure how to add above functions in the traditional pytorch data-loaders pipeline like first I should load the dataset using ImageFolder and then apply or first apply and then use ImageFolder method.

Output:
This is how I did it-

The solution of the first part is first defining the required function and then calling in the transforms using the generic transforms in the following way-

def gaussian_blur(img):
   image = np.array(img)
   image_blur = cv2.GaussianBlur(image,(65,65),10)
   new_image = image - image_blur
   im = Image.fromarray(new_image)
return im

Solution of second part is to go through every image and calculate the mean and std deviation and then finally calling the mean and std deviation values in the transforms.-

   train_mean = []
   train_std = []

   for i,image in enumerate(train_loader,0):
       numpy_image = image[0].numpy()
       batch_mean = np.mean(numpy_image, axis=(0, 2, 3))
       batch_std = np.std(numpy_image, axis=(0, 2, 3))

       train_mean.append(batch_mean)
       train_std.append(batch_std)

   train_mean = torch.tensor(np.mean(train_mean, axis=0))
   train_std = torch.tensor(np.mean(train_std, axis=0))

    print('Mean:', train_mean)
    print('Std Dev:', train_std)

Final transform calling looks like this-

data_transforms = transforms.Compose([transforms.RandomCrop(512,512),
                                 transforms.Lambda(gaussian_blur),
                                 transforms.RandomRotation([+90,+180]),
                                 transforms.RandomRotation([+180,+270]),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.ToTensor(),
                                 transforms.Normalize(mean=train_mean, std=train_std)
                               ])


Instruction: Can't change the Anchors in Faster RCNN

Input:
I'm a newbie in pytorch and I was trying to put some custom anchors on my Faster RCNN network in pytorch. Basically, I'm using a resnet50 backbone and when I try to put the anchors, I got a mismatch error.

This is the code that I have:

backbone = torchvision.models.detection.backbone_utils.resnet_fpn_backbone('resnet50', True)
backbone.out_channels = 256

anchor_generator = AnchorGenerator(sizes=((4, 8, 16, 32, 64, 128),),
                                    aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)

model = FasterRCNN(backbone,
                   num_classes=10,
                   rpn_anchor_generator=anchor_generator,
                   box_roi_pool=roi_pooler)

The error that I got is the following: shape '[1440000, -1]' is invalid for input of size 7674336.

Output:
Alright, after some digging into the source code of PyTorch Faster RCNN, I found how they initialize the anchors:

anchor_sizes = ((32,), (64,), (128,), (256,), (512,))
            aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)

rpn_anchor_generator = AnchorGenerator(
                anchor_sizes, aspect_ratios
            )

Following the same pattern for my custom anchors, the code will be:

anchor_sizes = ((4,), (8,), (16,), (32,), (64,), (128,))
            aspect_ratios = ((0.5, 1.0, 2.0),) * len(anchor_sizes)

rpn_anchor_generator = AnchorGenerator(
                anchor_sizes, aspect_ratios
            )

It will work!


Instruction: Cartpole-v0 loss increasing using DQN

Input:
Hi I'm trying to train a DQN to solve gym's Cartpole problem.
For some reason the Loss looks like this (orange line). Can y'all take a look at my code and help with this? I've played around with the hyperparameters a decent bit so I don't think they're the issue here.

class DQN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(DQN, self).__init__()
        self.linear1 = nn.Linear(input_dim, 16)
        self.linear2 = nn.Linear(16, 32)
        self.linear3 = nn.Linear(32, 32)
        self.linear4 = nn.Linear(32, output_dim)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        return self.linear4(x)

final_epsilon = 0.05
initial_epsilon = 1
epsilon_decay = 5000
global steps_done
steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
    if sample > eps_threshold:
        with torch.no_grad():
            state = torch.Tensor(state)
            steps_done += 1
            q_calc = model(state)
            node_activated = int(torch.argmax(q_calc))
            return node_activated
    else:
        node_activated = random.randint(0,1)
        steps_done += 1
        return node_activated

class ReplayMemory(object): # Stores [state, reward, action, next_state, done]

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = [[],[],[],[],[]]

    def push(self, data):
        """Saves a transition."""
        for idx, point in enumerate(data):
            #print("Col {} appended {}".format(idx, point))
            self.memory[idx].append(point)

    def sample(self, batch_size):
        rows = random.sample(range(0, len(self.memory[0])), batch_size)
        experiences = [[],[],[],[],[]]
        for row in rows:
            for col in range(5):
                experiences[col].append(self.memory[col][row])
        return experiences

    def __len__(self):
        return len(self.memory[0])

input_dim, output_dim = 4, 2
model = DQN(input_dim, output_dim)
target_net = DQN(input_dim, output_dim)
target_net.load_state_dict(model.state_dict())
target_net.eval()
tau = 2
discount = 0.99

learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

memory = ReplayMemory(65536)
BATCH_SIZE = 128

def optimize_model():
    if len(memory) < BATCH_SIZE:
        return 0
    experiences = memory.sample(BATCH_SIZE)
    state_batch = torch.Tensor(experiences[0])
    action_batch = torch.LongTensor(experiences[1]).unsqueeze(1)
    reward_batch = torch.Tensor(experiences[2])
    next_state_batch = torch.Tensor(experiences[3])
    done_batch = experiences[4]

    pred_q = model(state_batch).gather(1, action_batch)

    next_state_q_vals = torch.zeros(BATCH_SIZE)

    for idx, next_state in enumerate(next_state_batch):
        if done_batch[idx] == True:
            next_state_q_vals[idx] = -1
        else:
            # .max in pytorch returns (values, idx), we only want vals
            next_state_q_vals[idx] = (target_net(next_state_batch[idx]).max(0)[0]).detach()

    better_pred = (reward_batch + next_state_q_vals).unsqueeze(1)

    loss = F.smooth_l1_loss(pred_q, better_pred)
    optimizer.zero_grad()
    loss.backward()
    for param in model.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
    return loss

points = []
losspoints = []

#save_state = torch.load("models/DQN_target_11.pth")
#model.load_state_dict(save_state['state_dict'])
#optimizer.load_state_dict(save_state['optimizer'])

env = gym.make('CartPole-v0')
for i_episode in range(5000):
    observation = env.reset()
    episode_loss = 0
    if episode % tau == 0:
        target_net.load_state_dict(model.state_dict())
    for t in range(1000):
        #env.render()
        state = observation
        action = select_action(observation)
        observation, reward, done, _ = env.step(action)

        if done:
            next_state = [0,0,0,0]
        else:
            next_state = observation

        memory.push([state, action, reward, next_state, done])
        episode_loss = episode_loss + float(optimize_model(i_episode))
        if done:
            points.append((i_episode, t+1))
            print("Episode {} finished after {} timesteps".format(i_episode, t+1))
            print("Avg Loss: ", episode_loss / (t+1))
            losspoints.append((i_episode, episode_loss / (t+1)))
            if (i_episode % 100 == 0):
                eps = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
                print(eps)
            if ((i_episode+1) % 5001 == 0):
                save = {'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
                torch.save(save, "models/DQN_target_" + str(i_episode // 5000) + ".pth")
            break
env.close()

x = [coord[0] * 100 for coord in points]
y = [coord[1] for coord in points]

x2 = [coord[0] * 100 for coord in losspoints]
y2 = [coord[1] for coord in losspoints]

plt.plot(x, y)
plt.plot(x2, y2)
plt.show()

I basically followed the tutorial pytorch has, except using the state returned by the env rather than the pixels. I also changed the replay memory because I was having issues there. Other than that, I left everything else pretty much the same.

Edit:

I tried overfitting on a small batch and the Loss looks like this without updating the target net and this when updating it

Edit 2:

This is definitely an issue with the target net, I tried removing it and loss seemed to not increase exponentially

Output:
Your tau value is too small, small target network update cause DQN traning unstable. You can try to use 1000 (OpenAI Baseline's DQN example) or 10000 (Deepmind's Nature paper).

In Deepmind's 2015 Nature paper, it states that:

  The second modification to online Q-learning aimed at further improving the stability of our method with neural networks is to use a separate network for generating the traget yj in the Q-learning update. More precisely, every C updates we clone the network Q to obtain a target network Q' and use Q' for generating the Q-learning targets yj for the following C updates to Q.
  This modification makes the algorithm more stable compared to standard online Q-learning, where an update that increases Q(st,at) often also increases Q(st+1, a) for all a and hence also increases the target yj, possibly leading to oscillations or divergence of the policy. Generating the targets using the older set of parameters adds a delay between the time an update to Q is made and the time the update affects the targets yj, making divergence or oscillations much more unlikely.

Human-level control through deep reinforcement
learning, Mnih et al., 2015

I've run your code with settings of tau=2, tau=10, tau=100, tau=1000 and tau=10000. The update frequency of tau=100 solves the problem (reach maximum steps of 200).

tau=2

tau=10

tau=100

tau=1000

tau=10000

Below is the modified version of your code.

import random
import math
import matplotlib.pyplot as plt

import torch
from torch import nn
import torch.nn.functional as F
import gym

class DQN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(DQN, self).__init__()
        self.linear1 = nn.Linear(input_dim, 16)
        self.linear2 = nn.Linear(16, 32)
        self.linear3 = nn.Linear(32, 32)
        self.linear4 = nn.Linear(32, output_dim)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        return self.linear4(x)

final_epsilon = 0.05
initial_epsilon = 1
epsilon_decay = 5000
global steps_done
steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
    if sample > eps_threshold:
        with torch.no_grad():
            state = torch.Tensor(state)
            steps_done += 1
            q_calc = model(state)
            node_activated = int(torch.argmax(q_calc))
            return node_activated
    else:
        node_activated = random.randint(0,1)
        steps_done += 1
        return node_activated

class ReplayMemory(object): # Stores [state, reward, action, next_state, done]

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = [[],[],[],[],[]]

    def push(self, data):
        """Saves a transition."""
        for idx, point in enumerate(data):
            #print("Col {} appended {}".format(idx, point))
            self.memory[idx].append(point)

    def sample(self, batch_size):
        rows = random.sample(range(0, len(self.memory[0])), batch_size)
        experiences = [[],[],[],[],[]]
        for row in rows:
            for col in range(5):
                experiences[col].append(self.memory[col][row])
        return experiences

    def __len__(self):
        return len(self.memory[0])

input_dim, output_dim = 4, 2
model = DQN(input_dim, output_dim)
target_net = DQN(input_dim, output_dim)
target_net.load_state_dict(model.state_dict())
target_net.eval()
tau = 100
discount = 0.99

learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

memory = ReplayMemory(65536)
BATCH_SIZE = 128

def optimize_model():
    if len(memory) < BATCH_SIZE:
        return 0
    experiences = memory.sample(BATCH_SIZE)
    state_batch = torch.Tensor(experiences[0])
    action_batch = torch.LongTensor(experiences[1]).unsqueeze(1)
    reward_batch = torch.Tensor(experiences[2])
    next_state_batch = torch.Tensor(experiences[3])
    done_batch = experiences[4]

    pred_q = model(state_batch).gather(1, action_batch)

    next_state_q_vals = torch.zeros(BATCH_SIZE)

    for idx, next_state in enumerate(next_state_batch):
        if done_batch[idx] == True:
            next_state_q_vals[idx] = -1
        else:
            # .max in pytorch returns (values, idx), we only want vals
            next_state_q_vals[idx] = (target_net(next_state_batch[idx]).max(0)[0]).detach()

    better_pred = (reward_batch + next_state_q_vals).unsqueeze(1)

    loss = F.smooth_l1_loss(pred_q, better_pred)
    optimizer.zero_grad()
    loss.backward()
    for param in model.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
    return loss

points = []
losspoints = []

#save_state = torch.load("models/DQN_target_11.pth")
#model.load_state_dict(save_state['state_dict'])
#optimizer.load_state_dict(save_state['optimizer'])

env = gym.make('CartPole-v0')
for i_episode in range(5000):
    observation = env.reset()
    episode_loss = 0
    if i_episode % tau == 0:
        target_net.load_state_dict(model.state_dict())
    for t in range(1000):
        #env.render()
        state = observation
        action = select_action(observation)
        observation, reward, done, _ = env.step(action)

        if done:
            next_state = [0,0,0,0]
        else:
            next_state = observation

        memory.push([state, action, reward, next_state, done])
        episode_loss = episode_loss + float(optimize_model())
        if done:
            points.append((i_episode, t+1))
            print("Episode {} finished after {} timesteps".format(i_episode, t+1))
            print("Avg Loss: ", episode_loss / (t+1))
            losspoints.append((i_episode, episode_loss / (t+1)))
            if (i_episode % 100 == 0):
                eps = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
                print(eps)
            if ((i_episode+1) % 5001 == 0):
                save = {'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
                torch.save(save, "models/DQN_target_" + str(i_episode // 5000) + ".pth")
            break
env.close()

x = [coord[0] * 100 for coord in points]
y = [coord[1] for coord in points]

x2 = [coord[0] * 100 for coord in losspoints]
y2 = [coord[1] for coord in losspoints]

plt.plot(x, y)
plt.plot(x2, y2)
plt.show()

Here's the result of your plotting code.

tau=100

tau=10000


Instruction: Fastest way to split tensors in groups of two and for each group, randomly assign one to 0 and the other to 1

Input:
I have a big tensor with size (128, 64, 1, 1). For every two small tensors of size (1, 1) in that big tensor, I want to randomly assign one to be 1 and the other to be 0.

My following code works but it's awfully slow.

# a is a big tensor of size (128, 64, 1, 1)
for i in range(a.size()[0]):
    j = 0
    while j < a.size()[1] - 1:
        r = int(torch.randint(0, 2, (1,)))
        a[i][j + r] = 1
        a[i][j + 1 - r] = 0
        j += 2

Is there any way to avoid for loops and do every assignment in parallel? Thank you.

Output:
I think this might be faster :

import torch
x = torch.randn(128, 32, 2, 1, 1)
y = x.max(dim=2, keepdim=True)[0]
z = (y-x) > 0
r = torch.flatten(z, 1, 2)

The idea is that you split your 64 dimension into 2, pick random values, and chose the one that is minimal to be 1 and the other to be 0, then you recombine your 32*2 into a 64 size.


Instruction: Inplace add with list selectors

Input:
I'm facing a somehow inconsistent behaviour of pythorch according to weither the index is a list or an integer. Take a look at this snippet of code :

# First example, integer selector ==> Ok
t = torch.tensor([[0, 1], [1, 0]])
t[0, 0].add_(10)
print(t)
tensor([[10,  1],
        [ 1,  0]])

# Second example, list selector ==> ???
t = torch.tensor([[0, 1], [1, 0]])
t[[0], [0]].add_(10) # notice the list selector
print(t)
tensor([[0, 1],
        [1, 0]])

#Third example, list selector with inplace add operator ==> Ok
t = torch.tensor([[0, 1], [1, 0]])
t[[0], [0]] += 10
print(t)
tensor([[10,  1],
        [ 1,  0]])

I can't understand why pytorch was unable to update tin the second example !

Output:
See the difference between the two indexing:

In []: t[0, 0].shape

Out[]: torch.Size([])

In []: t[[0], [0]].shape

Out[]: torch.Size([1])

When you index directly the (0, 0)th element of t you have a reference to that entry and you can inplace add_ to it. The shape of t[0,0] is [] - that is you get a scalar back - the content of the (0,0) entry.
However, when you use list indices ([0], [0]) you get back a 1-dim tensor, shape is [1]. That is, you get a copy of a sub-tensor of t. You then inplace add_ to that copy of sub-tensor, you have no effect over the original t:

In []: r = t[[0], [0]].add_(10)
In []: t
Out[]:
tensor([[0, 1],
        [1, 0]])

In []: r
Out[]: tensor([10])

Perhaps you want to look into index_add_() to accomplish your task.

Update
When you assign to t using list indices, you are not creating a copy (it makes no sense. So,

t[[0], [0]] += 10

Translates to

t[[0], [0]] = t[[0], [0]] + 10

That is, on the right hand side we have a copy of the (0,0) sub-tensor of t and we are adding 10 to that sub-tensor, resulting with a shape [1] tensor with value [10]. On the left hand side we assign this [10] to the (0,0) sub-tensor of t (not to a copy of it - it makes no sense).
Therefore the output of t[[0], [0]] += 10 is

tensor([[10,  1],
        [ 1,  0]])


Instruction: NotImplementedError: when i try to create a DataLoader object in Pytorch

Input:
I created a custom Dataset class that inherits from PyTorch's Dataset class, in order to handle my custom dataset which i already preprocessed.

When i try to create a DataLoader object, i get this error:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn)
    174             if sampler is None:
    175                 if shuffle:
--> 176                     sampler = RandomSampler(dataset)
    177                 else:
    178                     sampler = SequentialSampler(dataset)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py in __init__(self, data_source, replacement, num_samples)
     62                              "since a random permute will be performed.")
     63
---> 64         if not isinstance(self.num_samples, int) or self.num_samples <= 0:
     65             raise ValueError("num_samples should be a positive integer "
     66                              "value, but got num_samples={}".format(self.num_samples))

/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py in num_samples(self)
     70         # dataset size might change at runtime
     71         if self._num_samples is None:
---> 72             return len(self.data_source)
     73         return self._num_samples
     74

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in __len__(self)
     18
     19     def __len__(self):
---> 20         raise NotImplementedError
     21
     22     def __add__(self, other):

NotImplementedError:

So, the error message regards to the not implementation of the len() function in the dataset.py, right?  But i did implement it and the getitem(), init() as well .

How can i overcome this?
Thank you

Output: Make sure the name is correct in your code. It should be __len__.


Instruction: What does this warning mean in PATE analysis?

Input:
Got warning while doing PATE analysis:

Warning: May not have used enough values of l. Increase 'moments' variable and run again.

from syft.frameworks.torch.differential_privacy import pate

data_dep_eps, data_ind_eps = pate.perform_analysis(teacher_preds=preds, indices=indices, noise_eps=0.1)
print("Data Independent Epsilon:", data_ind_eps)
print("Data Dependent Epsilon:", data_dep_eps)

It has gone after increasing the value of the "moment" parameter in the "pate.perform_analysis" analysis function. But I want to know why this was so.
data_dep_eps, data_ind_eps = pate.perform_analysis(teacher_preds=preds, indices=indices, noise_eps=0.1,moments=20)
print("Data Independent Epsilon:", data_ind_eps)
print("Data Dependent Epsilon:", data_dep_eps)

Output:
TL;DR: perform_analysis wants to double-check unusually small epsilon results by using more granular computation.

The pate.perform_analysis function iterates through the data (technically the privacy loss random variable) and computes various epsilons.  It uses the moments parameter to know how granular this iteration should be.  When using the default 8 moments, it will compute 8 epsilons.  Then it returns the minimum of the computed epsilons, as you can see in the source code.

When this function returns a very small data-dependent epsilon, it could be because A) the data has a high amount of agreement, or B) the computation wasn't granular enough, and the true epsilon is higher.  When only 8 epsilons are computed, it's possible that they happened to be anomalies in the data that paint an overly-optimistic picture of the overall epsilon!  So the function sees a surprisingly small epsilon and warns you - may want to increase the moments variable to compute more epsilons and make sure you've found the real minimum.  If you still get the same result when you increase your moments parameter, your data probably has a high amount of agreement, so it truly has a small data-dependent epsilon compared to its data-independent epsilon.

Hopefully that makes sense to you at a high level.  If you want more details on the math behind this, you can check out the research paper that inspired the source code.


Instruction: Using pypi pretrained models vs PyTorch

Input:
I have two setups - one takes approx. 10 minutes to run the other is still going after an hour:

10 m:

import pretrainedmodels

def resnext50_32x4d(pretrained=False):
    pretrained = 'imagenet' if pretrained else None
    model = pretrainedmodels.se_resnext50_32x4d(pretrained=pretrained)
    return nn.Sequential(*list(model.children()))

learn = cnn_learner(data, resnext50_32x4d, pretrained=True, cut=-2, split_on=lambda m: (m[0][3], m[1]),metrics=[accuracy, error_rate])

Not finishing:

import torchvision.models as models

def get_model(pretrained=True, model_name = 'resnext50_32x4d', **kwargs ):
    arch = models.resnext50_32x4d(pretrained, **kwargs )
    return arch

learn = Learner(data, get_model(), metrics=[accuracy, error_rate])

This is all copied and hacked from other peoples code so there are parts that I do not understand. But the most perplexing one is why one would be so much faster than the other. I would like to use the second option because its easier for me to understand and I can just swap out the pretrained model to test different ones.

Output:
Both architectures are different. I assume you are using pretrained-models.pytorch.

Please notice you are using SE-ResNeXt in your first example and ResNeXt in second (standard one from torchvision).

The first version uses faster block architecture (Squeeze and Excitation), research paper describing it here.

I'm not sure about exact differences between both architectures and implementations except different building block used, but
you could print both models and check for differences.

Finally here is a nice article summarizing what Squeeze And Excitation is. Basically you do GlobalAveragePooling on all channels (im pytorch it would  be torch.nn.AdaptiveAvgPoo2d(1) and flatten afterwards), push it through two linear layers (with ReLU activation in-between) finished by sigmoid in order to get weights for each channel. Finally you multiply the channels by those.

Additionally you are doing something strange with modules transforming them to torch.nn.Sequential. There may be some logic in forward call of pretrained network you are removing by copying modules, it may play a part as well.


Instruction: Need help regarding Transfer Learning a Faster RCNN ResNet50FPN in PyTorch

Input:
I am new to PyTorch. I'm trying to use a pre-trained Faster RCNN torchvision.models.detection.fasterrcnn_resnet50_fpn() for object detection project. I have created a CustomDataset(Dataset) class to handle the custom dataset.

Here is the custom class implementation

class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['meta_data']

        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose((2, 0, 1))
        return {'image': torch.from_numpy(image),
                'meta_data': landmarks}

class CustomDataset(Dataset):
    """Custom Landmarks dataset."""
    def __init__(self, data_dir, root_dir, transform=None):
        """
        Args:
            data_dir (string): Directory with all the labels(json).
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data_dir = data_dir
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(os.listdir(self.data_dir))

    def __getitem__(self, idx):
        img_name = sorted(os.listdir(self.root_dir))[idx]
        image = io.imread(self.root_dir+'/'+img_name, plugin='matplotlib')
        json_file = sorted(os.listdir(self.data_dir))[idx]
        with open(self.data_dir+'/'+json_file) as f:
          meta_data = json.load(f)

        meta_data = meta_data['annotation']['object']
        sample = {'image': image, 'meta_data': meta_data}

        to_tensor = ToTensor()
        transformed_sample = to_tensor(sample)

        if self.transform:
            sample = self.transform(sample)

        return transformed_sample

Here is the train_model function

def train_model(model, criterion, optimizer, lr_scheduler, num_epochs=25):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'test']:
            if phase == 'train':
                optimizer = lr_scheduler(optimizer, epoch)
                model.train()  # Set model to training mode
            else:
                model.eval()  # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            for data in dset_loaders[phase]:
                # get the inputs

                inputs, labels = data['image'], data['meta_data']

                inputs= inputs.to(device) # ,

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                outputs = model(inputs, labels)
                _, preds = torch.max(outputs.data, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

                # statistics
                running_loss += loss.item()
                running_corrects += torch.sum(preds == labels).item()

            epoch_loss = running_loss / dset_sizes[phase]
            epoch_acc = running_corrects / dset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'test' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model = copy.deepcopy(model)

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    return best_model

While performing model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25) I am getting "RuntimeError: _thnn_upsample_bilinear2d_forward not supported on CUDAType for Byte"

Output:
It appears your datapoints are byte tensors, i.e type uint8. Try casting your data into float32

# Replace this
inputs = inputs.to(device)
# With this
inputs = inputs.float().to(device)

Note that the torchvision models expect data to be normalized in a specific way. Check here for the procedure, which basically entails using

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])

for normalizing your data.


Instruction: How the standard normal distribution works in practice in NumPy and PyTorch?

Input:
I have two points to ask about:

1)
I would like to understand what is precisely returned from the np.random.randn from NumPy and torch.randn from PyTorch. They both return a tensor with random numbers from a normal distribution with mean 0 and std 1, hence, a standard normal distribution. However, it is not the same thing as puting x  values in the standard normal distribution function here and getting its respective image values y. The values returned by PyTorch and NumPy does not seem like this.

For me, it seems that both np.random.randn and torch.randn from these libraries returns the x values from the functions, not the image y as I calculated below. Is that correct?

normal = np.array([(1/np.sqrt(2*np.pi))*np.exp(-(1/2)*(i**2)) for i in range(-38,39)])

Printing the normal variable shows me something like this.

array([1.10e-314, 2.12e-298, 1.51e-282, 3.94e-267, 3.79e-252, 1.34e-237,
       1.75e-223, 8.36e-210, 1.47e-196, 9.55e-184, 2.28e-171, 2.00e-159,
       6.45e-148, 7.65e-137, 3.34e-126, 5.37e-116, 3.17e-106, 6.90e-097,
       5.52e-088, 1.62e-079, 1.76e-071, 7.00e-064, 1.03e-056, 5.53e-050,
       1.10e-043, 8.00e-038, 2.15e-032, 2.12e-027, 7.69e-023, 1.03e-018,
       5.05e-015, 9.13e-012, 6.08e-009, 1.49e-006, 1.34e-004, 4.43e-003,
       5.40e-002, 2.42e-001, 3.99e-001, 2.42e-001, 5.40e-002, 4.43e-003,
       1.34e-004, 1.49e-006, 6.08e-009, 9.13e-012, 5.05e-015, 1.03e-018,
       7.69e-023, 2.12e-027, 2.15e-032, 8.00e-038, 1.10e-043, 5.53e-050,
       1.03e-056, 7.00e-064, 1.76e-071, 1.62e-079, 5.52e-088, 6.90e-097,
       3.17e-106, 5.37e-116, 3.34e-126, 7.65e-137, 6.45e-148, 2.00e-159,
       2.28e-171, 9.55e-184, 1.47e-196, 8.36e-210, 1.75e-223, 1.34e-237,
       3.79e-252, 3.94e-267, 1.51e-282, 2.12e-298, 1.10e-314])

2) Also, if we ask these libraries that I want a matrix of values from a standard normal distribution, it means that all rows and columns are draw from the same standard distribution? If I want i.i.d distributions in every row, I would need to call np.random.randn over a for loop for each row and then vstack them?

Output:
1) Yes, they give you x and not phi(x) since the formula for phi(x) gives the probability density of sampling a value x. If you want to know the probability of getting values in an interval [a,b] you need to integrate phi(x) between a and b. Intuitively, if you look at the function phi(x) you'll see that you're more likely to get values near zero than, say, values near 1.
An easy way to see it, is look at the histogram of the sampled values.

import numpy as np
import matplotlib.pyplot as plt
samples = np.random.normal(size=[1000])
plt.hist(samples)

2) they're iid. Just use a 2d size like so:

samples = np.random.normal(size=[10, 10])


Instruction: how to concatenate embedding layer in pytorch

Input:
I am trying to concatenate embedding layer with other features. It doesn’t give me any error, but doesn’t do any training either. Is anything wrong with this model definition, how to debug this?

Note: The last column (feature) in my X is feature with word2ix (single word).
Note: The net works fine without the embedding feature/layer

originally posted on pytorch forum

class Net(torch.nn.Module):
    def __init__(self, n_features, h_sizes, num_words, embed_dim, out_size, dropout=None):
        super().__init__()

        self.num_layers = len(h_sizes)  # hidden + input

        self.embedding = torch.nn.Embedding(num_words, embed_dim)
        self.hidden = torch.nn.ModuleList()
        self.bnorm = torch.nn.ModuleList()
        if dropout is not None:
            self.dropout = torch.nn.ModuleList()
        else:
            self.dropout = None
        for k in range(len(h_sizes)):
            if k == 0:
                self.hidden.append(torch.nn.Linear(n_features, h_sizes[0]))
                self.bnorm.append(torch.nn.BatchNorm1d(h_sizes[0]))
                if self.dropout is not None:
                    self.dropout.append(torch.nn.Dropout(p=dropout))

            else:
                if k == 1:
                    input_dim = h_sizes[0] + embed_dim
                else:
                    input_dim = h_sizes[k-1]

                self.hidden.append(torch.nn.Linear(input_dim, h_sizes[k]))
                self.bnorm.append(torch.nn.BatchNorm1d(h_sizes[k]))
                if self.dropout is not None:
                    self.dropout.append(torch.nn.Dropout(p=dropout))

        # Output layer
        self.out = torch.nn.Linear(h_sizes[-1], out_size)

    def forward(self, inputs):

        # Feedforward

        for l in range(self.num_layers):
            if l == 0:
                x = self.hidden[l](inputs[:, :-1])
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x= self.dropout[l](x)

                embeds = self.embedding(inputs[:,-1])#.view((1, -1)
                x = torch.cat((embeds, x),dim=1)

            else:
                x = self.hidden[l](x)
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x = self.dropout[l](x)
            x = F.relu(x)
        output= self.out(x)

        return output

Output:
There were a few issues. The key one was data type. I mixed float features and int indices.

sample data and training before fix:

NUM_TARGETS = 4
NUM_FEATURES = 3
NUM_TEXT_FEATURES = 1

x = np.random.rand(5, NUM_FEATURES)
y = np.random.rand(5, NUM_TARGETS)

word_ix = np.arange(5).reshape(-1,1).astype(int)
x_train = np.append(x, word_ix, axis=1)

x_train = torch.from_numpy(x).float().to(device)
y_train = torch.from_numpy(y).float().to(device)

h_sizes = [2,2]

net = Net(x_train.shape[1] , h_sizes=h_sizes, num_words=5, embed_dim=2, out_size=y_train.shape[1],dropout=.01)     # define the network
print(net)  # net architecture
net = net.float()
net.to(device)

optimizer = torch.optim.Adam(net.parameters(), lr=0.0001, weight_decay=.01)
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss

# one training loop
prediction = net(x_train)     # input x and predict based on x

loss = loss_func(prediction, y_train)     # must be (1. nn output, 2. target)

optimizer.zero_grad()   # clear gradients for next train
loss.backward()         # backpropagation, compute gradients
optimizer.step()        # apply gradients
# train_losses.append(loss.detach().to('cpu').numpy())

To resolve this, I separated word index feature from x, and also removed net.float().

changed the dtypes conversion to:

x_train = torch.from_numpy(x).float().to(device)
y_train = torch.from_numpy(y).float().to(device)

# NOTE: word index needs to be long
word_ix = torch.from_numpy(word_ix).to(torch.long).to(device)

and forward method changed to :

    def forward(self, inputs, word_ix):

        # Feedforward

        for l in range(self.num_layers):
            if l == 0:
                x = self.hidden[l](inputs)
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x = self.dropout[l](x)

                embeds = self.embedding(word_ix)
                # NOTE:
                # embeds has a shape of (batch_size, 1, embed_dim)
                # inorder to merge this change this with x, reshape this to
                # (batch_size, embed_dim)
                embeds = embeds.view(embeds.shape[0], embeds.shape[2])
                x = torch.cat((x, embeds.view(x.shape)),dim=1)

            else:
                x = self.hidden[l](x)
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x = self.dropout[l](x)
            x = F.relu(x)
        output= self.out(x)

        return output


Instruction: how to know model's input size in onnx?

Input: The size of the input is not specified in the pytorch. Just size the kernel to make the output. The WinMLDashboard shows the width and height of the image input. How is that possible?

Output:
Do you mean when you serialize the network from pytorch to onnx? Because when you export from pytorch you need to define the size of the input as per documentation

dummy_input = torch.randn(10, 3, 224, 224, device='cuda')
model = torchvision.models.alexnet(pretrained=True).cuda()
input_names = [ "actual_input_1" ]
output_names = [ "output1" ]

torch.onnx.export(model, dummy_input, "alexnet.onnx", verbose=True,
input_names=input_names, output_names=output_names)


Instruction: "No route to host" error in torch.distributed.init_process_group

Input:
I'm trying to train a simple pytorch model on two GPU servers in parallel. I have compiled pytorch from source. The program gives the "RuntimeError: No route to host" when the process runs on the second server. How do I fix it?

I have tried the anaconda installation and source installation of pytorch, cuda, NCCL.

I have copied the following code from https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html

import os
from datetime import datetime
import argparse
import torch.multiprocessing as mp
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.distributed as dist
#from apex.parallel import DistributedDataParallel as DDP
#from apex import amp

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-n', '--nodes', default=1, type=int, metavar='N')
    parser.add_argument('-g', '--gpus', default=1, type=int, help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int, help='ranking within the nodes')
    parser.add_argument('--epochs', default=2, type=int, metavar='N', help='number of total epochs to run')
    args = parser.parse_args()

    args.world_size = args.gpus * args.nodes
    os.environ['MASTER_ADDR'] = '192.168.0.238'
    os.environ['MASTER_PORT'] = '8888'
    mp.spawn(train, nprocs=args.gpus, args=(args,))

    #train(0, args)

def train(gpu, args):
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(
            backend='nccl',
            init_method='env://',
            world_size=args.world_size,
            rank=rank
            )
    model = ConvNet()
    print('gpu:', gpu)
    torch.cuda.set_device(gpu)
    model.cuda(gpu)
    batch_size= 100

    criterion = nn.CrossEntropyLoss().cuda(gpu)
    optimizer = torch.optim.SGD(model.parameters(), 1e-4)

    model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])

    train_dataset = torchvision.datasets.MNIST(root='./data',
                                              train=True,
                                              transform=transforms.ToTensor(),
                                              download=True)

    train_sampler = torch.utils.data.distributed.DistributedSample(
            train_dataset,
            num_replicas=args.world_size,
            rank=rank
            )

    train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                              batch_size=batch_size,
                                              shuffle=False,
                                              num_workers=0,
                                              pin_memory=True,
                                              sampler=train_sampler)

    start = datetime.now()
    total_step = len(train_loader)
    for epoch in range(args.epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.cuda(non_blocking=True)
            labels = labels.cuda(non_blocking=True)

            outputs = model(images)
            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i+1) % 100 == 0 and gpu == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(
                    epoch + 1,
                    args.epochs,
                    i + 1,
                    total_step,
                    loss.item()
                ))

    if gpu == 0:
        print("Training complete in:" + str(datetime.now() - start))

if __name__ == '__main__':
    main()

Output:
TL;DR

This may be caused by the firewall.

Follow https://serverfault.com/a/890421 to trust your slave node.

A step-by-step checklist:

1. Install the netcat to help us check the network

For Ubuntu:

apt-get install netcat

2. Check if your master process is available

On node0 (or the master node):

nc -vv localhost:<port>

The output Connection to localhost <port >port [tcp/tproxy] succeeded! means your main process is running correctly. Otherwise, check if your program on the master node is running correctly.

3. Close the Firewall
If the master node program is working, and two nodes are supposed to connect well, this may be a firewall issue. For firewalld, consider set your node as trusted zone. See https://serverfault.com/a/890421 for more details.


Instruction: Save model with updated weights in pytorch

Input:
I have a model that I want to train for 5 epochs. Then, I would like to see where the model was wrong and increase the training set accordingly. How can I save the following model with the learnt weight?

trainer_ = Trainer(network = network,
                       optimizer = optim.Adam(network.parameters(), lr=0.001),
                       loss_function = loss_function,
                       train_loader = train_loader,
                       valid_every = 100,
                       print_every = 50,
                       save_every = 15000,
                       save_path = ".",
                       cudaok = is_cuda_available)

trainer_.run(4,is_cuda_available)

I have tried this:

path = os.path.join(project_path, 'model.pth')
torch.save(network.cpu().state_dict(), path) # saving model

But I don't really think that the object network contains weights.

I am very confused here. Can anyone help? Thank you!

Output:
network.state_dict() is a dictionary; try this to see your weights:

for param in network.state_dict():
    print(param, "\n", network.state_dict()[param])


Instruction: Pytorch LSTM vs LSTMCell

Input:
What is the difference between LSTM and LSTMCell in Pytorch (currently version 1.1)? It seems that LSTMCell is a special case of LSTM (i.e. with only one layer, unidirectional, no dropout).

Then, what's the purpose of having both implementations? Unless I'm missing something, it's trivial to use an LSTM object as an LSTMCell (or alternatively, it's pretty easy to use multiple LSTMCells to create the LSTM object)

Output:
Yes, you can emulate one by another, the reason for having them separate is efficiency.

LSTMCell is a cell that takes arguments:

Input of shape batch × input dimension;
A tuple of LSTM hidden states of shape batch x hidden dimensions.

It is a straightforward implementation of the equations.

LSTM is a layer applying an LSTM cell (or multiple LSTM cells) in a "for loop", but the loop is heavily optimized using cuDNN. Its input is

A three-dimensional tensor of inputs of shape batch × input length × input dimension;
Optionally, an initial state of the LSTM, i.e., a tuple of hidden states of shape batch × hidden dim (or tuple of such tuples if the LSTM is bidirectional)

You often might want to use the LSTM cell in a different context than apply it over a sequence, i.e. make an LSTM that operates over a tree-like structure. When you write a decoder in sequence-to-sequence models, you also call the cell in a loop and stop the loop when the end-of-sequence symbol is decoded.