Instruction: How does one use 3D convolutions on standard 3 channel images?

Input:
I am trying to use 3d conv on cifar10 data set (just for fun). I see the docs that we usually have the input be 5d tensors (N,C,D,H,W). Am I really forced to pass 5 dimensional data necessarily?

The reason I am skeptical is because 3D convolutions simply mean my conv moves across 3 dimensions/directions. So technically I could have 3d 4d 5d or even 100d tensors and then should all work as long as its at least a 3d tensor. Is that not right?

I tried it real quick and it did give an error:

import torch

def conv3d_example():
    N,C,H,W = 1,3,7,7
    img = torch.randn(N,C,H,W)
    ##
    in_channels, out_channels = 1, 4
    kernel_size = (2,3,3)
    conv = torch.nn.Conv3d(in_channels, out_channels, kernel_size)
    ##
    out = conv(img)
    print(out)
    print(out.size())

##
conv3d_example()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-29c73923cc64> in <module>
     15
     16 ##
---> 17 conv3d_example()

<ipython-input-3-29c73923cc64> in conv3d_example()
     10     conv = torch.nn.Conv3d(in_channels, out_channels, kernel_size)
     11     ##
---> 12     out = conv(img)
     13     print(out)
     14     print(out.size())

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    474                             self.dilation, self.groups)
    475         return F.conv3d(input, self.weight, self.bias, self.stride,
--> 476                         self.padding, self.dilation, self.groups)
    477
    478

RuntimeError: Expected 5-dimensional input for 5-dimensional weight 4 1 2 3, but got 4-dimensional input of size [1, 3, 7, 7] instead

cross posted:

https://discuss.pytorch.org/t/how-does-one-use-3d-convolutions-on-standard-3-channel-images/53330
How does one use 3D convolutions on standard 3 channel images?

Output:
Consider the following scenario. You have a 3 channel NxN image. This image will have size of 3xNxN in pytorch (ignoring the batch dimension for now).

Say you pass this image to a 2D convolution layer with no bias, kernel size 5x5, padding of 2, and input/output channels of 3 and 10 respectively.

What's actually happening when we apply this layer to the input image?

You can think of it like this...

For each of the 10 output channels there is a kernel of size 3x5x5. A 3D convolution is applied to the 3xNxN input image using this kernel, which can be thought of as unpadded in the first dimension. The result of this convolution is a 1xNxN feature map.

Since there are 10 output layers, there are 10 of the 3x5x5 kernels. After all kernels have been applied the outputs are stacked into a single 10xNxN tensor.

So really, in the classical sense, a 2D convolution layer is already performing a 3D convolution.

Similarly for a 3D convolution layer, its really doing a 4D convolution, which is why you need 5 dimensional input.


Instruction: Why did I get 2 different results from two models with same parameters and inputs?

Input:
I loaded resnet18 into my two models (model1 and model2), with pretrained weights.

I want to use them as feature extractors
For model1: I freezed the parameters except the last linear layer model1.fc, then train it. After training, I set model1.fc into torch.nn.Identity()
For model2: I directly set model2.fc into torch.nn.Identity()

Then these 2 models should be the same, but I get different forward result from the same inputs.

If the training of model1 is not done, they will have the same result, maybe something wrong with the freezing of  parameter.
However, I checked their parameters after the training of model1 and setting the last layer of both models to identity layer, and they seems to be the same.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms, models

# Load weights pretrained on ImageNet
def load_weights(model):
    model_dir = "....."
    model.load_state_dict(tor`enter code here`ch.utils.model_zoo.load_url("https://download.pytorch.org/models/resnet18-5c106cde.pth", model_dir=model_dir))
    return model

model1=models.resnet18()
model1=load_weights(model1)
for param in model1.parameters():
    param.requires_grad = False
model1.fc=nn.Linear(512, 2)
model1.cuda()
optimizer = optim.SGD(model1.fc.parameters(), lr=1e-2, momentum=0.9)
result_freeze = \
    run_training(model1, optimizer, device, train_loader, val_loader,num_epochs=10)

model2=models.resnet18()
model2=load_weights(model2)
model2.fc=nn.Identity()
model2.cuda()

model1.fc=nn.Identity()
model1.cuda()

# checking forward results(extracting features)

# The batch size is one here
for batch_idx, (data, target) in enumerate(X_train):
        data, target = data.to(device), target.to(device)
        d=data
        X_train_feature[batch_idx]=model1(data).cpu().detach().numpy()
        y_train[batch_idx]=target.cpu().detach().numpy()
        X_train2_feature[batch_idx]=model2(d).cpu().detach().numpy()
        y_train2[batch_idx]=target.cpu().detach().numpy()
        print(sum(X_train_feature[batch_idx]==X_train2_feature[batch_idx]))
        print(sum(y_train[batch_idx]==y_train2[batch_idx]))
        print(torch.sum(d==data))
for batch_idx, (data, target) in enumerate(X_test):
        data, target = data.to(device), target.to(device)
        d=data
        X_test_feature[batch_idx]=model1(data).cpu().detach().numpy()
        y_test[batch_idx]=target.cpu().detach().numpy()
        X_test2_feature[batch_idx]=model2(d).cpu().detach().numpy()
        y_test2[batch_idx]=target.cpu().detach().numpy()
        print(sum(X_test_feature[batch_idx]==X_test2_feature[batch_idx]))
        print(sum(y_test[batch_idx]==y_test2[batch_idx]))
        print(torch.sum(d==data))

# checking parameters
for a,b in zip(model1.parameters(),model2.parameters()):
    print(torch.sum(a!=b))

Expect to get the same forward results from model1 and model2, but they are different. And If they produce different forward results, why do they have exactly the same parameters?

Output:
Have you taken into account changes that might occur to BatchNorm layers?
Batch norm layers do not behave like normal layers - their internal parameters are modified by computing running mean and std of the data, and not by gradient descent.
Try setting model1.eval() before the finetune and then check.


Instruction: Pyinstaller unable to find dlls for the project dependencies while creating exe

Input:
Pyinstaller failed to find certain dlls that are required for binding in dependencies into one exe.

Please find the error logs below.

We have tried installing these libraries:

pip3 install intel-openmp mkl

Tried adding --paths to the command, but as there are no dlls in the system, pyinstaller is unable to find them:

pyinstaller --onefile --paths <Paths-where-dll-could-be> -c main.py

These libs are missing and showed in logs as WARNINGS.

364427 WARNING: lib not found: impi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_intelmpi_ilp64.dll
365396 WARNING: lib not found: mpich2mpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_mpich2_lp64.dll
366241 WARNING: lib not found: msmpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_msmpi_lp64.dll
368089 WARNING: lib not found: msmpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_msmpi_ilp64.dll
369270 WARNING: lib not found: pgf90.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
369997 WARNING: lib not found: pgc14.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
370791 WARNING: lib not found: pgf90rtl.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
373039 WARNING: lib not found: mpich2mpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_mpich2_ilp64.dll
374289 WARNING: lib not found: impi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_intelmpi_lp64.dll
377030 WARNING: lib not found: torch_python.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torch\_C.cp36-win_amd64.pyd
378792 WARNING: lib not found: c10_cuda.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
379568 WARNING: lib not found: torch.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
380290 WARNING: lib not found: caffe2.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
381126 WARNING: lib not found: c10.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
382053 WARNING: lib not found: torch_python.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd

As the missing lib dlls are not in the system, kindly assist the efficient way to build the exe.

Output:
Your log of warnings is very similar to mine: PyInstaller .exe file terminates early without an error message
I am therefore assuming, despite these warnings, PyInstaller still successfully builds your executable? These steps (as per above link) worked for me:

Use PyInstaller to generate a one-folder bundle.
From within your python environment that you used to write your program, search for all "dll" files, and copy & paste them into the dist folder of your bundle.
In my case, this enabled my executable file to run successfully.
You can then experiment by systematically removing the DLL files you pasted into your dist folder, in order to identify which are essential and which are redundant.
(optional) if you prefer a one-file bundle, you will need to do step 4 to identify the essential DLL files that are required for your executable file to run, so that you can add just these DLLs as data files to your one-file build (in my case, it was just the one! - "libiomp5md.dll").

Appreciate this is not a very elegant solution, it's more about perseverance! Good luck :)


Instruction: Pytorch: Visualize model while training

Input:
I am training a neural network by regression but it is predicting a constant value during testing. Which is why I want to visualize the weights of the neural network change during training and see the weights change dynamically in the jupyter notebook.
Currently, my model looks like this:

import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.inp = nn.Linear(2, 40)
        self.act1 = nn.Tanh()
        self.h1 = nn.Linear(40, 40)
        self.act2 = nn.Tanh()
        self.h2 = nn.Linear(40, 2)
        self.act3 = nn.Tanh()
        #self.h3 = nn.Linear(20, 20)
        #self.act4=nn.Tanh()
        self.h4 = nn.Linear(2, 1)

    def forward_one_pt(self, x):
        out = self.inp(x)
        out = self.act1(out)
        out = self.h1(out)
        out = self.act2(out)
        out = self.h2(out)
        out = self.act3(out)
        #out = self.h3(out)
        #out = self.act4(out)
        out = self.h4(out)
        return out

    def forward(self, config):
        E = torch.zeros([config.shape[0], 1])
        for i in range(config.shape[0]):
            E[i] = self.forward_one_pt(config[i])
            # print("config[",i,"] = ",config[i],"E[",i,"] = ",E[i])
        return torch.sum(E, 0)

and my main function looks like this:

def main()  :
    learning_rate = 0.5
    n_pts = 1000
    t_pts = 100
    epochs = 15

    coords,E = load_data(n_pts,t_pts)

    #generating my data to NN
    G = get_symm(coords,save,load_symmetry,symmtery_pickle_file,eeta1,eeta2,Rs,ex,lambdaa,zeta,boxl,Rc,pi,E,scale)
    net = Net()
    if(cuda_flag):
        net.cuda()
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

    net_trained = train(save,text_output,epochs,n_pts,G,E,net,t_pts,optimizer,criterion,out,cuda_flag)
    test(save,n_pts,t_pts,G,E,net_trained,out,criterion,cuda_flag)

    torch.save(net,save_model)

any tutorials or answers would be helpful

Output:
You can use model.state_dict() to see if your weights are updating across epochs:

old_state_dict = {}
for key in model.state_dict():
    old_state_dict[key] = model.state_dict()[key].clone()

output = model(input)

new_state_dict = {}
for key in model.state_dict():
    new_state_dict[key] = model.state_dict()[key].clone()

for key in old_state_dict:
    if not (old_state_dict[key] == new_state_dict[key]).all():
        print('Diff in {}'.format(key))
    else:
        print('NO Diff in {}'.format(key))

On a side note, you can vectorize your forward function instead of looping over it. Following would do the same job as your original forward function but much faster:

def forward(self, config):
    out= self.forward_one_pt(config)
    return torch.sum(out, 0)


Instruction: Why is the memory in GPU still in use after clearing the object?

Input:
Starting with zero usage:

>>> import gc
>>> import GPUtil
>>> import torch
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Then I create a big enough tensor and hog the memory:

>>> x = torch.rand(10000,300,200).cuda()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Then I tried several ways to see if the tensor disappears.

Attempt 1: Detach, send to CPU and overwrite the variable

No, doesn't work.

>>> x = x.detach().cpu()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 2: Delete the variable

No, this doesn't work either

>>> del x
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 3: Use the torch.cuda.empty_cache() function

Seems to work, but it seems that there are some lingering overheads...

>>> torch.cuda.empty_cache()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 4: Maybe clear the garbage collector.

No, 5% is still being hogged

>>> gc.collect()
0
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 5: Try deleting torch altogether (as if that would work when del x didn't work -_- )

No, it doesn't...*

>>> del torch
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

And then I tried to check gc.get_objects() and it looks like there's still quite a lot of odd THCTensor stuff inside...

Any idea why is the memory still in use after clearing the cache?

Output:
It looks like PyTorch's caching allocator reserves some fixed amount of memory even if there are no tensors, and this allocation is triggered by the first CUDA memory access
(torch.cuda.empty_cache() deletes unused tensor from the cache, but the cache itself still uses some memory).

Even with a tiny 1-element tensor, after del and torch.cuda.empty_cache(), GPUtil.showUtilization(all=True) reports exactly the same amount of GPU memory used as for a huge tensor (and both torch.cuda.memory_cached() and torch.cuda.memory_allocated() return zero).


Instruction: How to customize pytorch data

Input:
I am trying to make a customized Dataloader using pytorch.

I've seen some codes like (omitted the class sorry.)

def __init__(self, data_root, transform=None, training=True, return_id=False):
    super().__init__()
    self.mode = 'train' if training else 'test'

    self.data_root = Path(data_root)
    csv_fname = 'train.csv' if training else 'sample_submission.csv'
    self.csv_file = pd.read_csv(self.data_root / csv_fname)
    self.transform = transform
    self.return_id = return_id
def __getitem__():
    """ TODO
    """
def __len__():
    """ TODO
    """

The problem here is that the datas I've dealt with before contains all the training data in one csv file, and all the testing data in the other csv file, with total 2 csv files for training and testing. (For example like in MNIST, the last column is the labeling, and the all the previous columns are each different features.)

However, the problem I've been facing is that I've got very many, (about 200,000) csv files for training, each one smaller than 60,000 sized MNIST, but still quite big. All these csv files contain different number of rows.

To inherit torch.util.data, how can I make customized class? MNIST dataset is quite small, so can be uploaded on RAM at once. However, the data I'm dealing with is super big, so I need some help.

Any ideas? Thank you in advance.

Output:
First, you want to customize (overload) data.Dataset and not data.DataLoader which is perfectly fine for your use case.

What you can do, instead of loading all data to RAM, is to read and store "meta data" on __init__ and read one relevant csv file whnever you need to __getitem__ a specific entry.
A pseudo-code of your Dataset will look something like:

class ManyCSVsDataset(data.Dataset):
  def __init__(self, ...):
    super(ManyCSVsDataset, self).__init__()
    # store the paths for all csvs and the number of items in each one
    self.metadata = ...
    self.num_items = total_number_of_items

  def __len__(self):
    return self.num_items

  def __getitem__(self, index):
    # based on the index, use self.metadata to determine what csv file to open
    with open(relevant_csv_file, 'r') as R:
      # read from R the specific line matching item index
    return item

This implementation is not efficient in the sense that it reads the same csv file over and over and does not cache anything. On the other hand, you can take advantage of data.DataLoader's multi processing support to have many parallel sub-processes doing all these file access at the background while you actually use the data for training.


Instruction: How can I deploy my Pytorch model into IOS?

Input: I have a deep learning neural network I built on Pytorch I am seeking to deploy onto IOS.

Output:
Native support doesn't exist still I think, but what some do is to export the ONNX model and then open this in Caffe2 which has the support for IOS device (also Androids)

So use ONNX export tutorial and this mobile integration helper.

There is also a path converting ONNX to CoreML but depending on your project it may not be super fast.

There is also an option to port the ONNX to TensorFlow Lite and to integrate with your Swift or Objective C app from there on.


Instruction: Pytorch custom dataset: ValueError: some of the strides of a given numpy array are negative

Input:
I wrote a custom pytorch dataset, but ran into an error thhat seems quite unintelligible.

My custom dataset,

class data_from_xlsx(Dataset):
    def __init__(self, xlsx_fp, path_col, class_cols_list):
        self.xlsx_file = pd.read_excel(xlsx_fp)
        self.path_col = path_col
        self.class_cols_list = class_cols_list

    def __len__(self):
        return get_xlsx_length(self.xlsx_file)

    def __getitem__(self, index):
        file_path = cols_from_xlsx(self.xlsx_file, index, 1, self.path_col)
        feature = load_nii_file(file_path) # get 3D volume (x, y, z)
        feature = np.expand_dims(feature, axis=0) # add channel (c, x, y, z)
        label = cols_from_xlsx(self.xlsx_file, index, 1, self.class_cols_list) # get label
        return feature, label.astype(np.bool)

def main():
dataset = data_from_xlsx("train.xlsx", "file_path", ["pos", "neg"], transformations, aug=True)
    data_loader = DataLoader(dataset, batch_size=4, shuffle=True)

    for (f, l) in data_loader:
        print("f shape", f.shape)
        print("l shape", l.shape)

An error is reported when I ran main(),

 File "d:\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 346, in __next__
    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 80, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 80, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 65, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 65, in <listcomp>
    return default_collate([torch.as_tensor(b) for b in batch])
ValueError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future release

The reported error does't make sense to me, so I googled it. At first I thought I didn't change the feature from numpy.array to tensor, so I tried feature = torch.from_array(feature.copy()) and also tried transforms.TOTensor() but both attempts failed.

Output:
Thanks to the advice from @jodag and @UsmanAli, I sovled this by return torch.from_numpy(feature.copy()) and torch.tensor(label.astype(np.bool))
So the whole thing should be,

class data_from_xlsx(Dataset):
    def __init__(self, xlsx_fp, path_col, class_cols_list):
        self.xlsx_file = pd.read_excel(xlsx_fp)
        self.path_col = path_col
        self.class_cols_list = class_cols_list

    def __len__(self):
        return get_xlsx_length(self.xlsx_file)

    def __getitem__(self, index):
        file_path = cols_from_xlsx(self.xlsx_file, index, 1, self.path_col)
        feature = load_nii_file(file_path) # get 3D volume (x, y, z)
        feature = np.expand_dims(feature, axis=0) # add channel (c, x, y, z)
        label = cols_from_xlsx(self.xlsx_file, index, 1, self.class_cols_list) # get label
        return torch.from_numpy(feature.copy()), torch.tensor(label.astype(np.bool))


Instruction: constants in Pytorch Linear Module Class Definition

Input:
What is __constants__ in pytorch class Linear(Module): defined in https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html?

What is its functionality and why is it used?

I have been searching around, but did not find any documentation. Please note that this does not mean the __constants__ in torch script.

Output:
The __constants__ you're talking about is, in fact, the one related to TorchScript. You can confirm it by using git blame (when it was added and by who) on GitHub. For example, for torch/nn/modules/linear.py, check its git blame.

TorchScript also provides a way to use constants that are defined in Python. These can be used to hard-code hyper-parameters into the function, or to define universal constants.
-- Attributes of a ScriptModule can be marked constant by listing them as a member of the constants property of the class:

class Foo(torch.jit.ScriptModule):
    __constants__ = ['a']

    def __init__(self):
        super(Foo, self).__init__(False)
        self.a = 1 + 4

   @torch.jit.script_method
   def forward(self, input):
       return self.a + input


Instruction: What is the meaning of keep_vars in state_dict?

Input:
state_dict(destination=None, prefix='', keep_vars=False)

what does changing keep_vars to True do?

Output:
In PyTorch >=0.4, it has no use.

keep_vars was added in the commit: Add keep_vars parameter to state_dict stating that

  When keep_vars is true, it returns a Variable for each parameter
          (rather than a Tensor).

In state_dict function, _save_to_state_dict is called internally, which contains the following code

for name, param in self._parameters.items():
            if param is not None:
                destination[prefix + name] = param if keep_vars else param.data
        for name, buf in self._buffers.items():
            if buf is not None:
                destination[prefix + name] = buf if keep_vars else buf.data

The portion param if keep_vars else param.data made difference prior to PyTorch 0.4.0 when Variable and Tensor were separate, but now as they are merged, keep_vars is probably present only for backward compatibility. Check Is .data still useful in pytorch?


Instruction: What is the difference between register_parameter and register_buffer in PyTorch?

Input:
Module's parameters get changed during training, that is, they are what is learnt during training of a neural network, but what is a buffer?
and is it learnt during neural network training?

Output:
Pytorch doc for register_buffer() method reads

  This is typically used to register a buffer that should not to be considered a model parameter. For example, BatchNorm’s running_mean is not a parameter, but is part of the persistent state.

As you already observed, model parameters are learned and updated using SGD during the training process.
However, sometimes there are other quantities that are part of a model's "state" and should be
 - saved as part of state_dict.
 - moved to cuda() or cpu() with the rest of the model's parameters.
 - cast to float/half/double with the rest of the model's parameters.
Registering these "arguments" as the model's buffer allows pytorch to track them and save them like regular parameters, but prevents pytorch from updating them using SGD mechanism.

An example for a buffer can be found in _BatchNorm module where the running_mean , running_var and num_batches_tracked are registered as buffers and updated by accumulating statistics of data forwarded through the layer. This is in contrast to weight and bias parameters that learns an affine transformation of the data using regular SGD optimization.


Instruction: Filter data in pytorch tensor

Input:
I have a tensor X like [0.1, 0.5, -1.0, 0, 1.2, 0], and I want to implement a function called filter_positive(), it can filter the positive data into a new tensor and return the index of the original tensor. For example:

new_tensor, index = filter_positive(X)

new_tensor = [0.1, 0.5, 1.2]
index = [0, 1, 4]

How can I implement this function most efficiently in pytorch?

Output:
Take a look at torch.nonzero which is roughly equivalent to np.where. It translates a binary mask to indices:
>>> X = torch.tensor([0.1, 0.5, -1.0, 0, 1.2, 0])
>>> mask = X >= 0
>>> mask
tensor([1, 1, 0, 1, 1, 1], dtype=torch.uint8)

>>> indices = torch.nonzero(mask)
>>> indices
tensor([[0],
        [1],
        [3],
        [4],
        [5]])

>>> X[indices]
tensor([[0.1000],
        [0.5000],
        [0.0000],
        [1.2000],
        [0.0000]])

A solution would then be to write:
mask = X >= 0
new_tensor = X[mask]
indices = torch.nonzero(mask)


Instruction: Difference in Model Performance when using Validation set/ Testing set

Input:
I have implemented a PyTorch NN code for classification and regression.

Classification:
a) Use stratifiedKfolds for cross-validation (K=10- means 10 fold-cross validation)

I divided the data: as follows:
Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.

b) Loss function = CrossEntropy

c) Optimization = SGD

d) Early Stopping where waittime = 100 epochs.

Problem is:
Baseline Accuracy = 51%
Accuracy on Training set = 100%
Accuracy on validation set = 90%
Accuracy on testing set = 72%

I don’t understand what are the reasons behind the huge performance difference in Testing data/ Validation data?

How can I solve this problem?

Regression:

a) use the same network structure

b) loss function = MSELoss

c) Optimization = SGD

d) Early Stopping where wait-time = 100 epochs.

e) Use K-fold for cross-validation.

I divided the data: as follows:

Suppose I have 100 data: 10 for testing, 18 for validation, 72 for training.

Problem is:
Baseline MSE= 14.0
Accuracy on Training set = 0.0012
Accuracy on validation set = 6.45
Accuracy on testing set = 17.12

I don’t understand what are the reasons behind the huge performance difference in Testing data/ Validation data?

How can I solve these problems? or Is this an obvious thing for NN/ depend on particular dataset?

Output:
You have a large gap between training and validation performance, and between validation and test performance. There are two issues to explore:

Differences in the distribution. We assume that train / val / test sets are all drawn from the same distribution, and so have similar characteristics. A well trained model should perform equally well on the val and test datasets. If you are dataset really is only 10 samples for test and 18 for val, there is a high chance that the samples selected will skew one/both of these datasets, so that they no longer have similar characteristics. Therefore the difference between your val and test performance could just be chance: Your test set just happens to be much harder. You could test this by manual inspection.
Overfitting to val: However, I think it is more likely that you have experimented with different architectures, training regimes, etc, and have tweaked parameters to get the best performance on you validation set. This means that you have overfit your model to your val set. The test set is a truer reflection of your model's accuracy.

Your training accuracy is very high for both problems, and there is a large gap between training and validation performance. You are therefore overfitting to the training data, so need to train less, or introduce more stringent regularisation.


Instruction: ModuleNotFoundError: No module named 'past' when installing tensorboard with pytorch 1.2

Input:
I'm trying out tensorboard with pytorch by following this: https://pytorch.org/docs/stable/tensorboard.html

I've installed tensorboard with pip install tb-nightly

The command tensorboard --logdir=runs starts ok.

But the line self.writer = SummaryWriter()

Gives the following error:

ModuleNotFoundError: No module named 'past'

How do I fix this?

Output:
Following this issue: https://github.com/pytorch/pytorch/issues/22389,

Adding future to the list of requirements solved the problem

# requirements.txt:
tb-nightly
future

pip install -r requirements.txt


Instruction: Difference between PyTorch, PyTorchModel in sagemaker.pytorch

Input:
I am trying to create a model using pytorch in sagemaker. I tried deploying using - PyTorch module in sagemaker.pytorch [from sagemaker.pytorch import PyTorch].

But, I want to understand what is PyTorchModel in sagemaker.pytorch [from sagemaker.pytorch import PyTorchModel]. They both have deploy() . And I followed the link https://sagemaker.readthedocs.io/en/stable/using_pytorch.html to create and deploy the model . Where I don't see using "PyTorchModel" anywhere. I would like to know the difference and when to use what.

I tried the following so far.

Step 1 : I called pytorch estimator

pytorch_model = PyTorch(entry_point='entry_v1.py',
                        train_instance_type='ml.m5.4xlarge',
                        role = role,
                        train_instance_count=1,
                        output_path = "s3://model-output-bucket/test",
                        framework_version='1.1',
                        hyperparameters = {'epochs': 10,'learning-rate': 0.01})

Step2  : I called fit method

pytorch_model.fit({'train': 's3://training-data/train_data.csv',
                  'test':'s3://testing-data/test_data.csv'})

Step3: I called deploy method.

predictor = pytorch_model.deploy(instance_type='ml.m4.xlarge', initial_instance_count=1)

I would like to know when to call create_model() here.

I got some understanding over here. We use [from sagemaker.pytorch import PyTorch] for end to end process where we train a model with .fit() and then we can deploy the model with .deploy()

But, with [from sagemaker.pytorch import PyTorchModel] we can use the model which is already trained.

Step1:

pytorch_model = PyTorchModel(model_data='s3://model-output-bucket/sagemaker-pytorch-2019-08-20-16-54-32-500/output/model.tar.gz', role=role,entry_point=entry_v1.py,sagemaker_session=sagemaker_session)

Step2:

predictor = pytorch_model.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1)

Also, .create_model() of PyTorch Estimator will return an object of PyTorchModel.

Please correct me if i am wrong anywhere.

Output:
PyTorch class is inherited from the Framework class whereas PyTorchModel is inherited from the FrameworkModel class.

The difference between these two is that:
Framework is used to perform the end to end training and deployment of a model
FrameworkModel is used to create an Estimator from a pretrained model and then use it to deploy an endpoint using the deploy() method. This does not involve training of the model.

In the PyTorch class, you cannot directly call the deploy model. You'll first have to invoke the fit() method which will be followed by the deploy() method.

You can read the following blog on how you can bring your own Pretrained model to Sagemaker
https://aws.amazon.com/blogs/machine-learning/bring-your-own-pre-trained-mxnet-or-tensorflow-models-into-amazon-sagemaker/

Regarding the create_model() method, you need not invoke it in your script if you'd like to directly deploy an endpoint after training.
It is generally used in scenarios where you require to create a pipeline for inferences through multiple models


Instruction: pytorch TypeError: '<' not supported between instances of 'Example' and 'Example' when referring to iterator

Input:
I am trying to use my own dataset to classify text according to https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb. My dataset is a csv of sentences and a class associated with it. there are 6 different classes:

sent                      class
'the fox is brown'        animal
'the house is big'        object
'one water is drinkable'  water
...

When running:

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    print(start_time)
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    #print(train_loss.type())
    #print(train_acc.type())
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut5-model.pt')

    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

, I obtained the following error

1566460706.977012
epoch_loss
<torchtext.data.iterator.BucketIterator object at 0x000001FABE907E80>
TypeError: '<' not supported between instances of 'Example' and 'Example'

pointing to:

TypeError                                 Traceback (most recent call last)
<ipython-input-22-19e8a7eb204e> in <module>()
     10     #print(train_loss.type())
     11     #print(train_acc.type())
---> 12     valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
     13
     14     end_time = time.time()

<ipython-input-21-83b02f99bca7> in evaluate(model, iterator, criterion)
      9         print('epoch_loss')
     10         print(iterator)
---> 11         for batch in iterator:
     12             print('batch')
     13             predictions = model(batch.text)

I am new to pytorch, so only added a line to idenify iterator data type and got:

<torchtext.data.iterator.BucketIterator object at 0x000001FABE907E80>

I tried to determine specific attributes following  https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py to no avail.

Any suggestions are appreciated.

The code for the evaluate (where the iterator is) method is:

def evaluate(model, iterator, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.eval()

    with torch.no_grad():
        print('epoch_loss')
        print(iterator)
        for batch in iterator:
            print('batch')
            predictions = model(batch.text)

            loss = criterion(predictions, batch.label)

            acc = categorical_accuracy(predictions, batch.label)

            epoch_loss += loss.item()
            epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Output:
I faced a similar issue and got it resolved by using sort_key and sort_within_batch while creating iterators.
train_iterator, valid_iterator = BucketIterator.splits(
    (train, valid),
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.sent),
    sort_within_batch=True,
    device = device)


Instruction: Taking a norm of matrix rows/cols in pytorch

Input:
The norm of a vector can be taken by

torch.norm(vec)

However, how to take a norm of a set of vectors grouped as a matrix (either as rows or columns)?

For example, if a matrix size is (5,8), then the rows norms should return a vector of norms of size (5).

Output:
torch.norm without extra arguments performs what is called a Frobenius norm which is effectively reshaping the matrix into one long vector and returning the 2-norm of that. To take the norm along a particular dimension provide the optional dim argument.

For example torch.norm(mat, dim=1) will compute the 2-norm along the columns (i.e. this will compute the 2-norm of each row) thus converting a mat of size [N,M] to a vector of norms of size [N].

To compute the norm of the columns use dim=0.


Instruction: PyTorch not downloading

Input:
I go to the PyTorch website and select the following options

PyTorch Build: Stable (1.2)

Your OS: Windows

Package: pip

Language: Python 3.7

CUDA: None

(All of these are correct)

Than it displays a command to run

pip3 install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

I have already tried to mix around the the different options but none of them has worked.

ERROR: ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.2.0+cpu

I tried to do pip install pytorch but pytorch doesn't support pypi

Output:
I've been in same situation.
My prob was,  the python version... I mean, in the 'bit' way.

It was 32 bit that the python I'd installed.
You should check which bit of python you installed.
you can check in the app in setting, search python, then you will see the which bit you've installed.

After I installed the 64 bit of Python, it solved.

I hope you figure it out!

environment : win 10


Instruction: How to set the output size of an RNN?

Input:
I want to have an RNN with input size 7, hidden size 10 and output size 2.
So for an input of, say, shape 99x1x7 I expect an output of shape 99x1x2.
For an RNN alone, I get:

model = nn.RNN(input_size=7, hidden_size=10, num_layers=1)

output,hn=model(torch.rand(99,1,7))
print(output.shape) #torch.Size([99, 1, 10])
print(hn.shape)     #torch.Size([ 1, 1, 10])

So I assume I still have to put a Linear behind it:

model = nn.Sequential(nn.RNN(input_size=7, hidden_size=10, num_layers=1),
                      nn.Linear(in_features=10, out_features=2))
model(torch.rand(99,1,7))

Traceback (most recent call last):
  File "train_rnn.py", line 80, in <module>
    main()
  File "train_rnn.py", line 25, in main
    model(torch.rand(99,1,7))
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/functional.py", line 1404, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

I guess this is because Linear receives the tuple that RNN.forward yields. But how am I supposed to combine the two?

Output:
From the pytorch doc https://pytorch.org/docs/stable/nn.html?highlight=rnn#torch.nn.RNN

the output is of shape seq_len, batch, num_directions * hidden_size

So depending on what you want you might add a fc layer to get an output of size 2.
Basically, a Sequential will apply each model on top of the output of the next_one so you must either not use a Sequential or create a special Linear Layer that works on sequence, the following should work :

class seq_Linear(nn.module):
  def __init__(self, linear):
    self.linear = linear
  # To apply on every hidden state
  def forward(self, x):
    return torch.stack([self.linear(hs) for hs in x])
  # To apply on the last hidden state
  def forward(self, x):
    return self.linear(x[-1])

and replace your nn.Linear by a seq_Linear(nn.Linear) in your code.

Edit : If you want to create a sequence of outputs of size 2, perhaps the best way would be to stack another RNN on top of your first one with input_size 10 and output_size 2, they should be stackable inside a Sequential without any trouble.


Instruction: What is the difference between these two neural network structures?

Input:
first using nn.Parameter

class ModelOne(nn.Module):
  def __init__(self):
    super().__init__()
    self.weights = nn.Parameter(torch.randn(300, 10))
    self.bias = nn.Parameter(torch.zeros(10))
  def forward(self, x):
    return x @ self.weights + self.bias

when I do

mo = ModelOne()
[len(param) for param in mo.parameters()]

it gives
[300, 10]

second using nn.Linear

class ModelTwo(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(300, 10)

  def forward(self, x):
    return self.linear(x)

same thing here gives
[10, 10]

Output:
The difference lies in how nn.Linear initializes weights and bias:

class Linear(Module):

    def __init__(self, in_features, out_features, bias=True):
        super(Linear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        if bias:
            self.bias = Parameter(torch.Tensor(out_features))
        ...

So, when you write nn.Linear(300, 10) the weight is (10, 300) and bias is (10). But, in the ModelOne, weights has dimension (300, 10).

You can confirm it using

for name, param in mo.named_parameters():
    print(name, param.shape)

The output in ModelOne:

weights torch.Size([300, 10])
bias torch.Size([10])

In ModelTwo:

linear.weight torch.Size([10, 300])
linear.bias torch.Size([10])

Now, the reason you're getting [300, 10] in first case and [10, 10] in second case is because if you print the length of a 2d Tensor, then it'll only give its first dimension i.e.

a = torch.Tensor(10, 300)
b = torch.Tensor(10)
print(len(a), len(b))

(10, 10)


Instruction: DataLoader crashes when shuffling

Input:
I'm using DataLoader to read from a custom Dataset object based on numpy memmap.
As long as I read the data without shuffling everything works fine but, as I set shuffle=True, the runtime crash.

I tried implementing the shuffling mechanism in the Dataset class by using a permutation vector and setting shuffle=False in the DataLoader but the issue persists.
I also noticed that, when shuffling, the __getitem__() function of the Dataset object is called n times, where n is the batch_size.

Here's the Dataset code:

class CustomDataset(Dataset):
  num_pattern = 60112
  base_folder = 'dataset'

  def __init__(self, root):
    self.root = os.path.expanduser(root)

    self.output_ = np.memmap('{0}/output'.format(root), 'int64', 'r', shape=(60112, 62))
    self.out_len = np.memmap('{0}/output-lengths'.format(root), 'int32', 'r', shape=(60112))
    self.input_ = np.memmap('{0}/input'.format(root), 'float32', 'r', shape=(60112, 512, 1024))
    self.in_len = np.memmap('{0}/input-lengths'.format(root), 'int32', 'r', shape=(60112))

  def __len__(self):
    return self.num_pattern

  def __getitem__(self, index):
    return (self.in_len[index], torch.from_numpy(self.input_[index])), (self.out_len[index], torch.from_numpy(self.output_[index]))

if __name__ == '__main__':
  dataset = CustomDataset(root='/content/')
  data_loader = data.DataLoader(dataset, batch_size=32, shuffle=False, num_workers=1)
  for i, data in enumerate(data_loader, 0):
    # training

The error stack is the following:

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _try_get_batch(self, timeout)
    510         try:
--> 511             data = self.data_queue.get(timeout=timeout)
    512             return (True, data)

9 frames
/usr/lib/python3.6/multiprocessing/queues.py in get(self, block, timeout)
    103                     timeout = deadline - time.monotonic()
--> 104                     if not self._poll(timeout):
    105                         raise Empty

/usr/lib/python3.6/multiprocessing/connection.py in poll(self, timeout)
    256         self._check_readable()
--> 257         return self._poll(timeout)
    258

/usr/lib/python3.6/multiprocessing/connection.py in _poll(self, timeout)
    413     def _poll(self, timeout):
--> 414         r = wait([self], timeout)
    415         return bool(r)

/usr/lib/python3.6/multiprocessing/connection.py in wait(object_list, timeout)
    910             while True:
--> 911                 ready = selector.select(timeout)
    912                 if ready:

/usr/lib/python3.6/selectors.py in select(self, timeout)
    375             try:
--> 376                 fd_event_list = self._poll.poll(timeout)
    377             except InterruptedError:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/signal_handling.py in handler(signum, frame)
     62         # Python can still get and update the process status successfully.
---> 63         _error_if_any_worker_fails()
     64         if previous_handler is not None:

RuntimeError: DataLoader worker (pid 3978) is killed by signal: Bus error.

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-b407a8532808> in <module>()
      5   data_loader = data.DataLoader(dataset, batch_size=4, shuffle=True, num_workers=1)
      6
----> 7   for i, data in enumerate(data_loader, 0):
      8     print(i)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    574         while True:
    575             assert (not self.shutdown and self.batches_outstanding > 0)
--> 576             idx, batch = self._get_batch()
    577             self.batches_outstanding -= 1
    578             if idx != self.rcvd_idx:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _get_batch(self)
    551         else:
    552             while True:
--> 553                 success, data = self._try_get_batch()
    554                 if success:
    555                     return data

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _try_get_batch(self, timeout)
    517             if not all(w.is_alive() for w in self.workers):
    518                 pids_str = ', '.join(str(w.pid) for w in self.workers if not w.is_alive())
--> 519                 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
    520             if isinstance(e, queue.Empty):
    521                 return (False, None)

RuntimeError: DataLoader worker (pid(s) 3978) exited unexpectedly

Output:
RuntimeError: DataLoader worker (pid(s) 3978) exited unexpectedly

This error is because, In data.DataLoader(dataset, batch_size=32, shuffle=False, num_workers=1) make num_workers=0 , its saying there are no subprocesses in your cpu


Instruction: What is the difference between feature and classifier?

Input:
I saw code like this

self.feature = model_func()
if loss_type == 'softmax':
    self.classifier = nn.Linear(self.feature.final_feat_dim, num_class)
    self.classifier.bias.data.fill_(0)
elif loss_type == 'dist': #Baseline ++
    self.classifier = backbone.distLinear(self.feature.final_feat_dim, num_class)

where model_func is a ConvNet 4/6 or ResNet 10/18/34/101

What here is classifier?

I know that in neural networks we have parameters that we learn, buffers that are used to store something that gets updated during training, activations that are the results after each layer.

Is feature same as activation, and what is a classifier, where is the end of features, and the beginning of classifier in a neural network? Is the result of a classifier also activation?

Output:
I find the question a little messy, but I'll give my best from what I understand you're asking.

  What here is classifier?

The classifier would be the model itself. The model is the one who will, after being trained, be able to classify new data.

  Is feature same as activation

I don't know what kind of feature you have in mind. In data science context, a feature is understood to be one of the variables of the data one has. For example, if you have a dataset about houses, you may have features such as latitude, long., if it has a pool, how many bedrooms it has, etc.

Activation functions are mathematical equations that determine the output of a neural network. The function is attached to each neuron in the network, and determines whether it should be activated (“fired”) or not, based on whether each neuron’s input is relevant for the model’s prediction. [1]

I'm not sure I'm truly understanding what you're asking.

  Is the result of a classifier also activation?

The result of a classifier is the label, the class to which each data point belongs. Activation functions are used by neural networks in the process of classifying.

Hope this helps!

[1] https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/


Instruction: Not able to load the saved graph using torch.utils.tensorboard.SummaryWriter.add_graph method

Input:
I am saving the scalar summary along with model graph using  add_scalar and add_graph methods from torch.utils.tensorboard.SummaryWriter.

While running tensorboard on the summary file, it doesnt show the model graph. Just 2 small rectangle at the bottom right, However, it is able to show the scalar variable and images.

Sample code from the pytorch documentation attached

import torch
import torchvision
from torch.utils.tensorboard import SummaryWriter
from torchvision import datasets, transforms

# Writer will output to ./runs/ directory by default
writer = SummaryWriter()

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = datasets.MNIST('mnist_train', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
model = torchvision.models.resnet50(False)

print(model)

# Have ResNet model take in grayscale rather than RGB
model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
images, labels = next(iter(trainloader))

grid = torchvision.utils.make_grid(images)
writer.add_image('images', grid, 0)
writer.add_graph(model, images)
writer.close()

Any pointer to the solution would be appreciated

  PyTorch version - 1.2.0 Tensorboard version - 1.15.0a20190828

Output:
I had the same problem.
Check out this thread:
https://github.com/pytorch/pytorch/issues/24157

TLDR: Update PyTorch to PyTorch-nightly and the problem should be solved.
https://pytorch.org/get-started/locally/


Instruction: How can I find a solution for the "FileNotFoundError"

Input:
I'm currently working on an image classifier project. During the testing of the predict function, I receive the error: FileNotFoundError: [Errno 2] No such file or directory: 'flowers/test/1/image_06760'

the path of the file is correct.
You can find the whole notebook here:
https://github.com/MartinTschendel/image-classifier-1/blob/master/190829-0510-Image%20Classifier%20Project.ipynb

Here is the respective predict function and the test of this function:

def predict(image_path, model, topk=5):
    ''' Predict the class (or classes) of an image using a trained deep learning model.
    '''

    #loading model
    loaded_model = load_checkpoint(model)

    # implement the code to predict the class from an image file
    img = Image.open(image_name)
    img = process_image(img)

    # convert 2D image to 1D vector
    img = np.expand_dims(img, 0)

    img = torch.from_numpy(img)

    #set model to evaluation mode and turn off gradients
    loaded_model.eval()
    with torch.no_grad():
        #run image through network
        output = loaded_model.forward(img)

    #calculate probabilities
    probs = torch.exp(output)
    probs_top = probs.topk(topk)[0]
    index_top = probs.topk(topk)[1]

    # Convert probabilities and outputs to lists
    probs_top_list = np.array(probs_top)[0]
    index_top_list = np.array(index_top[0])

    #load index and class mapping
    loaded_model.class_to_idx = train_data.class_to_idx
    #invert index-class dictionary
    idx_to_class = {x: y for y, x in class_to_idx.items()}

    #convert index list to class list
    classes_top_list = []
    for index in index_top_list:
        classes_top_list += [idx_to_class[index]]

    return probs_top_list, classes_top_list

# test predict function

# inputs equal paths to saved model and test image
model_path = '190824_checkpoint.pth'
image_name = data_dir + '/test' + '/1/' + 'image_06760'

probs, classes = predict(image_name, model_path, topk=5)

print(probs)
print(classes)

This is the error I receive:

FileNotFoundError: [Errno 2] No such file or directory: 'flowers/test/1/image_06760'

Output:
These are the images you have

You should have it as

image_name = data_dir + '/test' + '/1/' + 'image_06760.jpg'

for it to work as you were not specifying the image extension.


Instruction: Getting truth value of array with more than one element in ambigous for albuementation transform

Input:
I am using albumentations for applying transform to a Pytorch model but getting this error and I m not getting any clue of what this error is about. Only thing I know is this is occuring due to transform which is being applied but not sure what is wrong with that.

ValueError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "<ipython-input-23-119ea6bc360e>", line 24, in __getitem__
    image = self.transform(image)
  File "/opt/conda/lib/python3.6/site-packages/albumentations/core/composition.py", line 164, in __call__
    need_to_run = force_apply or random.random() < self.p
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This is are the snippets of code.
Dataloader getitem() method:

        image = cv2.imread(p_path)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        image = crop_image_from_gray(image)
        image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))
        image = cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , 10) ,-4 ,128)
        print(image.shape)
        image = self.transform(image)

transforms applied :

val_transform = albumentations.Compose([
            Normalize(
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225],
            ),
            ToTensor()
        ])

The class is called by:

valset       = MyDataset(val_df, transform = val_transform)

Output:
From official albumentation documentation, you can apply transformation to image

from PIL import Image
import cv2
import numpy as np
from torch.utils.data import Dataset
from torchvision import transforms
from albumentations import Compose, RandomCrop, Normalize, HorizontalFlip, Resize
from albumentations.pytorch import ToTensor

class AlbumentationsDataset(Dataset):
    """__init__ and __len__ functions are the same as in TorchvisionDataset"""
    def __init__(self, file_paths, labels, transform=None):
        self.file_paths = file_paths
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, idx):
        label = self.labels[idx]
        file_path = self.file_paths[idx]

        # Read an image with OpenCV
        image = cv2.imread(file_path)

        # By default OpenCV uses BGR color space for color images,
        # so we need to convert the image to RGB color space.
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        image = crop_image_from_gray(image)
        image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))
        image = cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , 10) ,-4 ,128)

        image = Img.fromarray(image, mode='RGB')
        if self.transform:
            augmented = self.transform(image=np.array(image))
            image = augmented['image']

        image = np.transpose(image, (2, 0, 1))

        return image, label

albumentations_transform = Compose([

    Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
    ToTensor()
])

albumentations_dataset = AlbumentationsDataset(
    file_paths=['./images/image_1.jpg', './images/image_2.jpg', './images/image_3.jpg'],
    labels=[1, 2, 3],
    transform=albumentations_transform,
)

test_loader = DataLoader(dataset = albumentations_dataset, batch_size=4, drop_last=False, shuffle=False).


Instruction: TypeError although same shape: if not (target.size() == input.size()): 'int' object is not callable

Input:
This is the error message I get. In the first line, I output the shapes of predicted and  target. From my understanding, the error arises from those shapes not being the same but here they clearly are.

torch.Size([6890, 3]) torch.Size([6890, 3])

Traceback (most recent call last):
  File "train.py", line 251, in <module>
    main()
  File "train.py", line 230, in main
    train(net, training_dataset, targets, device, criterion, optimizer, epoch, args.epochs)
  File "train.py", line 101, in train
    loss = criterion(predicted, target.detach().cpu().numpy())
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 443, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2244, in mse_loss
    if not (target.size() == input.size()):
TypeError: 'int' object is not callable

I hope all the relevant context information is provided and if not, please let me know. Thanks for any suggestions!

EDIT: This is the part of the code where this error occurs:

        target = torch.from_numpy(np.load(file_dir + '/points/points{:03}.npy'.format(i))).to(device)
        rv = torch.zeros(12 * outputs.shape[0])

        for j in [x for x in range(10) if x != i]:
            source = torch.from_numpy(np.load(file_dir + '/points/points{:03}.npy'.format(j))).to(device)
            rv = factor.ransac(source, target, prob, n_iter, tol, device) # some self-written RANSAC-like method

        predicted = factor.predict(source, rv, outputs)
        print(target.shape, predicted.shape)
        loss = criterion(predicted, target.detach().cpu().numpy()) ## error occurs here

criterion is nn.MSELoss().

Output:
A little bit late but maybe it will help someone else. Just solved the same problem for myself.

As Alpha said in his answer we cannot call .size() for a numpy array.
But we can call .size() for a tensor.
Therefore, we need to make our target a tensor. You can do it like this:

target = torch.from_numpy(target)

I'm using GPU, so I also needed to send my target to GPU. You can do it like this:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
target = target.to(device)

And then the loss function must work perfectly.


Instruction: Binary cross entropy Vs categorical cross entropy with 2 classes

Input:
When considering the problem of classifying an input to one of 2 classes, 99% of the examples I saw used a NN with a single output and sigmoid as their activation followed by a binary cross-entropy loss.  Another option that I thought of is having the last layer produce 2 outputs and use a categorical cross-entropy with C=2 classes, but I never saw it in any example.
Is there any reason for that?

Thanks

Output:
If you are using softmax on top of the two output network you get an output that is mathematically equivalent to using a single output with sigmoid on top.
Do the math and you'll see.

In practice, from my experience, if you look at the raw "logits" of the two outputs net (before softmax) you'll see that one is exactly the negative of the other. This is a result of the gradients pulling exactly in the opposite direction each neuron.

Therefore, since both approaches are equivalent, the single output configuration has less parameters and requires less computations, thus it is more advantageous to use a single output with a sigmoid ob top.


Instruction: torch.cat but create a new dimension

Input:
I would like to concatenate tensors, not along a dimension, but by creating a new dimension.

For example:

x = torch.randn(2, 3)
x.shape # (2, 3)

torch.cat([x,x,x,x], 0).shape # (8, 3)
# This concats along dim 0, not what I want

torch.cat([x,x,x,x], -1).shape # (2, 10)
# This concats along dim 1, not what I want

torch.cat([x[None, :, :],x[None, :, :],x[None, :, :],x[None, :, :]], 0).shape
# => (4, 2, 3)
# This is what I want, but unwieldy

Is there a simpler way?

Output:
Just use torch.stack:

torch.stack([x,x,x,x]).shape # (4, 2, 3)


Instruction: How to ensure that a batch contains samples from all workers with PyTorch's DataLoader?

Input:
I want to know how to use torch.utils.data.DataLoader in PyTorch, especially in a multi-worker case.

I found that one batch output from DataLoader always comes from a single worker.
I expected that there is a queue in the DataLoader which stores data from all of the workers and DataLoader shuffles them in the queue to output the random batch data. I think this is the way in tf.data.Dataset in Tensorflow.
Can we implement a similar function in PyTorch? I want to load a dataset from big serialized files (like Tfrecord) by using multi workers. In this case, mixing the source file in one batch, which means mixing the source of the worker, is important.

Please refer to following code:

import random
import time

import torch

class MyDataset(torch.utils.data.Dataset):
    def __len__(self):
        return 50

    def __getitem__(self, idx):
        info = torch.utils.data.get_worker_info()

        time.sleep(random.uniform(0, 1))
        print("[{}]:{}".format(info.id, idx))
        return idx, info.id

if __name__ == '__main__':
    dataset = MyDataset()
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=5, shuffle=False, num_workers=2)
    for batch in dataloader:
        print(batch)

Output:

[0]:0
[1]:5
[0]:1
[1]:6
[0]:2
[0]:3
[1]:7
[0]:4
[tensor([0, 1, 2, 3, 4]), tensor([0, 0, 0, 0, 0])]
[1]:8
[1]:9
[tensor([5, 6, 7, 8, 9]), tensor([1, 1, 1, 1, 1])]
[0]:10
[0]:11
[1]:15
[1]:16
[0]:12
[1]:17
...

Here, [0, 1, 2, 3, 4] and [0, 0, 0, 0, 0] in [tensor([0, 1, 2, 3, 4]), tensor([0, 0, 0, 0, 0])] mean that this batch includes index 0-th to 4-th data came from worker id 0.
Note that shuffle=True does not solve this problem which only change the indices of data.

In this case, I want to get a batch like: [tensor([0, 5, 1, 6, 2]), tensor([0, 1, 0, 1, 0])].

Output:
I've implemented something simple to solve a similar problem, where I have large video files as training data and each worker is responsible for loading and preprocessing a single file and then yielding samples from it. Problem is that as OP describes, with Pytorch's default data loading mechanism, each batch contains samples only from a single video file.

First, let's review the problem. In this simplified code example each worker yields a single Tensor containing its zero-indexed worker id. With a batch size of 32 and 4 workers, we want each batch to contain 8 zeros, 8 ones, 8 twos and 8 threes.

from collections import defaultdict

import torch as T
import torch.utils.data as tdata

class Dataset(tdata.IterableDataset):
    def __init__(self, batch_size: int):
        self._bs = batch_size

    def __iter__(self):
        worker_info = tdata.get_worker_info()
        if not worker_info:
            raise NotImplementedError('Not implemented for num_workers=0')
        for _ in range(self._bs):
            yield T.tensor([worker_info.id])

batch_size = 32
num_workers = 4
dataset = Dataset(batch_size)
loader = tdata.DataLoader(dataset,
                          batch_size=batch_size,
                          num_workers=num_workers)

for batch in loader:
    counts = defaultdict(int)
    for n in batch.numpy().flatten():
        counts[n] += 1
    print(dict(counts))

Instead the code prints:

{0: 32}
{1: 32}
{2: 32}
{3: 32}

Meaning that the first batch contains samples only from worker 0, the second only from worker 1, etc. To remedy this, we will set the batch size in the DataLoader to batch_size // num_workers and use a simple wrapper over the DataLoader to pool samples from each worker for our batch:

def pooled_batches(loader):
    loader_it = iter(loader)
    while True:
        samples = []
        for _ in range(loader.num_workers):
            try:
                samples.append(next(loader_it))
            except StopIteration:
                pass
        if len(samples) == 0:
            break
        else:
            yield T.cat(samples, dim=0)

batch_size = 32
num_workers = 4
dataset = Dataset(batch_size)
per_worker = batch_size // num_workers
loader = tdata.DataLoader(dataset,
                          batch_size=per_worker,
                          num_workers=num_workers)

for batch in pooled_batches(loader):
    counts = defaultdict(int)
    for n in batch.numpy().flatten():
        counts[n] += 1
    print(dict(counts))

And the code now prints

{0: 8, 1: 8, 2: 8, 3: 8}
{0: 8, 1: 8, 2: 8, 3: 8}
{0: 8, 1: 8, 2: 8, 3: 8}
{0: 8, 1: 8, 2: 8, 3: 8}

as expected.


Instruction: Can't import torch in jupyter notebook

Input:
System: macOS 10.13.6
Python: 3.7
Anaconda3
I have trouble when import torch in jupyter notebook.
ModuleNotFoundError: No module named 'torch'

Here is how I install pytorch:
conda install pytorch torchvision -c pytorch

I've checked PyTorch is installed in my anaconda environment:

When I command python3 in my terminal and import torch, it works. But not work in jupyter notebook
I've tried:
conda update conda
conda install mkl=2018

But still the same error.

Output:
You have to install jupyter in addition to pytorch inside your activated conda env. Here is installation steps:

1. Create conda env

for example: pytorch_p37 with python 3.7:

user@pc:~$ conda create -n pytorch_p37 python=3.7

2. Activate it

user@pc:~$ conda activate pytorch_p37

Or with (for older conda versions):

user@pc:~$ source activate pytorch_p37

Now you should see (pytorch_p37) before your shell prompt:

(pytorch_p37) user@pc:~$

3. Go to PyTorch website and choose appropriate installation command via conda. Run it in your shell, for example

(pytorch_p37) user@pc:~$ conda install pytorch torchvision -c pytorch

4. Install jupyter inside your activated env as well

(pytorch_p37) user@pc:~$ conda install jupyter

5. Verify the installation

(pytorch_p37) user@pc:~$ conda list

# packages in environment at /home/user/anaconda3/envs/pytorch_p37:
#
# Name
...
jupyter                   1.0.0
jupyter_client            5.3.1
jupyter_console           6.0.0
jupyter_core              4.5.0
...
python                    3.7.4
pytorch                   1.2.0
...

6. Run jupyter

(pytorch_p37) user@pc:~$ jupyter notebook


Instruction: What does the `model.parameters()` include?

Input:
In Pytorch,

What will be registered into the model.parameters().

As far as now, what I know are as belows:

1.  Conv layer: weight  bias
2.  BN layers: weight(gamma)  bias(beta)
3.  nn.Parameter()
    such as:   self.alpha = nn.Parameter(torch.rand(10))  defined in the model.

My question is:
And are there some parameters else that are registered in the model.parameters() ?

PS. The most common case for the model.parameters() is in the optimizer,
e.g.  pytorch resnet example

optimizer = torch.optim.SGD(model.parameters(), args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

Thank you in advance.

Output:
Like you wrote there, model.parameters() stores the weight and bias (if set to true) values of the model.
It is given as an argument to an optimizer to update the weight and bias values of the model with one line of code optimizer.step(), which you then use when next you go over your dataset.


Instruction: Data Normalization using Pytorch

Input:
I'm starting to work on the classification of images dataset, as many tutorials I followed; it starts by normalizing the data (train and test data)

My question is: if I want to normalize the data by shifting and scaling it with a factor of 0.5

What does this mean 'the factor of something x'?

I know that it will be used within .Normalize():

transform_train = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(),
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(),
])

But I'm a little bit confused about the meaning shift and scale (maybe it's like resize?) by the factor of X

Thanks.

Output:
Shifting and scaling refers to the color space. What you do is you subtract the mean (shifting to the mean of the pixel values of the whole dataset to 0) and divide by the standard deviation (scaling the pixel values to [0, 1].

It has nothing to do with modifying the size of the image or the like.

In numy you would do something like:

mean, std = np.mean(image), np.std(image)
image = image - mean
image = image / std

Note: You wouldn't want to normalize the data bz just 0.5 but by its mean and standard deviation.


Instruction: Multiple outputs in Pytorch, Keras style

Input:
How could you implement these 2 Keras models (inspired by the Datacamp course 'Advanced Deep Learning with Keras in Python') in Pytorch:

Classification with 1 input, 2 outputs:

from keras.layers import Input, Concatenate, Dense
from keras.models import Model

input_tensor = Input(shape=(1,))
output_tensor = Dense(2)(input_tensor)

model = Model(input_tensor, output_tensor)
model.compile(optimizer='adam', loss='categorical_crossentropy')
X = ... # e.g. a pandas series
y = ... # e.g. a pandas df with 2 columns
model.fit(X, y, epochs=100)

A model with classification and regression:

from keras.layers import Input, Dense
from keras.models import Model

input_tensor = Input(shape=(1,))
output_tensor_reg = Dense(1)(input_tensor)
output_tensor_class = Dense(1, activation='sigmoid')(output_tensor_reg)

model.compile(loss=['mean_absolute_error','binary_crossentropy']
X = ...
y_reg = ...
y_class = ...
model.fit(X, [y_reg, y_class], epochs=100)

Output:
This ressource was particularly helpful.

Basically, the idea is that, contrary to Keras, you have to explicitly say where you're going to compute each output in your forward function and how the global loss is gonna be computed from them.

For example, regarding the 1st example:

def __init__(self, ...):
    ... # define your model elements

def forward(self, x):
    # Do your stuff here
    ...
    x1 = F.sigmoid(x) # class probabilities
    x2 = F.sigmoid(x) # bounding box calculation
    return x1, x2

Then you compute the losses:

out1, out2 = model(data)
loss1 = criterion1(out1, target1)
loss2 = criterion2(out2, targt2)
alpha = ... # define the weights of each sub-loss in the global loss
loss = alpha * loss1 + (1-alpha) * loss2
loss.backward()

For the 2nd one, it's almost the same but you compute the loss at different point in the forward pass:

def __init__(self, ...):
    ... # define your model elements

def forward(self, main_input, aux_input):
    aux = F.relu(self.dense_1(main_input))
    x = F.relu(self.input_layer(aux))
    x = F.sigmoid(x)
    return x, aux


Instruction: TypeError: hook() takes 2 positional arguments but 3 were given

Input:
I'm new to pytorch and I'm trying to use hook() and register_forward_pre_hook in my project

What I've tried is

def get_features_hook(module,input):
    print(input)

handle_feat = alexnet.features[0].register_forward_pre_hook(get_features_hook)

a = alexnet(input_data)

And I got belows error at a = alexnet(input_data)

  TypeError: get_features_hook() takes 2 positional arguments but 3 were given

I've lost few hours on this problem and I just can't able to figure it out.

Anyone likes to help me?

With Shai's help, I tried his codes, and I got this

Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
get_features_hook called with 2 args:
    arg of type Conv2d
    arg of type tuple

File "<input>", line 2, in get_features_hook
NameError: name 'args' is not defined

Output: If get_features_hook is defined inside your torch.nn.Module, it should be annotated as @staticmethod, otherwise self is implicitly passed to it


Instruction: Specify network to not backpropagate the gradients in a given function

Input:
In my neural network, in order to compute the loss, I need to do some intermediate computations during training to first obtain some transformations rv.

rv = factor.ransac(source, target, prob, device)
predicted = factor.predict(source, rv, outputs, device)
loss = criterion(predicted, target)

I want to backpropagate the gradients only through predicted but not through factor.ransac. How do I do this?

Output:
You can use detach().

rv = factor.ransac(source, target, prob, device).detach()

That way, gradients on rv will be discarded.


Instruction: Is indexing with a bool array or index array faster in numpy/pytorch?

Input: I can index my numpy array / pytorch tensor with a boolean array/tensor of the same shape or an array/tensor containing integer indexes of the elements I'm after. Which is faster?

Output:
The following tests indicate that it's generally 3x to 20x faster with an index array in both numpy and pytorch:

In [1]: a = torch.arange(int(1e5))
idxs = torch.randint(len(a), (int(1e4),))
ind = torch.zeros_like(a, dtype=torch.uint8)
ind[idxs] = 1
ac, idxsc, indc = a.cuda(), idxs.cuda(), ind.cuda()

In [2]: %timeit a[idxs]
73.4 µs ± 1 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [3]: %timeit a[ind]
622 µs ± 8.99 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [4]: %timeit ac[idxsc]
9.51 µs ± 475 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [5]: %timeit ac[indc]
59.6 µs ± 313 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: idxs = torch.arange(len(a)-1, dtype=torch.long)
ind = torch.zeros_like(a, dtype=torch.uint8)
ind[idxs] = 1
ac, idxsc, indc = a.cuda(), idxs.cuda(), ind.cuda()

In [7]: %timeit a[idxs]
146 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [8]: %timeit a[ind]
4.59 ms ± 106 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [9]: %timeit ac[idxsc]
33 µs ± 15.1 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [10]: %timeit ac[indc]
85.9 µs ± 56.9 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)


Instruction: mask first k elements in a 3D tensor in PyTorch (different k for each row)

Input:
I have a tensor M of dimensions [NxQxD] and a 1d tensor of indices idx (of size N). I want to efficiently create a tensor mask of dimensions [NxQxD] such that mask[i,j,k] = 1 iff j <= idx[i], i.e. I want to keep only the idx[i] first dimensions out of Q in the second dimension (dim=1) of M, for every row i.

Thanks!

Output:
It turns out this can be done via a broadcasting trick:

mask_2d = torch.arange(Q)[None, :] < idx[:, None] #(N,Q)
mask_3d = mask[..., None] #(N,Q,1)
masked = mask.float() * data


Instruction: torch.utils.data.random_split() is not splitting the data

Input:
I’m not getting split when I use torch.utils.data.random_split.

I get correct numbers for train_size and val_size, but when I do random_split, both train_data and val_data get full_data. There is no split happening.

Please help me with this issue.

class DeviceLoader(Dataset):

def __init__(self, root_dir, train=True, transform=None):
    self.file_path = root_dir
    self.train = train
    self.transform = transform
    self.file_names = ['%s/%s'%(root,file) for root,_,files in os.walk(root_dir) for file in files]
    self.len = len(self.file_names)
    self.labels = {'BP_Raw_Images':0, 'DT_Raw_Images':1, 'GL_Raw_Images':2, 'PO_Raw_Images':3, 'WS_Raw_Images':4}

def __len__(self):
    return(len(self.file_names))

def __getitem__(self, idx):
    file_name = self.file_names[idx]
    device = file_name.split('/')[5]
    img = self.pil_loader(file_name)
    if(self.transform):
        img = self.transform(img)
    cat = self.labels[device]
    if(self.train):
        return(img, cat)
    else:
        return(img, file_name)
full_data = DeviceLoader(root_dir=’/kaggle/input/devices/dataset/’, transform=transforms, train=True)
train_size = int(0.7*len(full_data))
val_size = len(full_data) - train_size
train_data, val_data = torch.utils.data.random_split(full_data,[train_size,val_size])

Expected result is to split the full_data into train_data(2000) and val_data(500). But instead, I get full_data(2500) in both train and val.

Output:
From the image below you could see, it actually makes a subset of data but the original dataset is still there. This might be confusing. I did the following on mnist dataset
train, validate, test = data.random_split(training_set, [50000, 10000, 10000])
print(len(train))
print(len(validate))
print(len(test))

output:
50000
10000
10000


Instruction: Pytorch : how to run code on several machines in cluster

Input:
I am using a cluster to train a recurrent neural network developed using PyTorch. PyTorch automatically threads, which allows to use all the cores of a machine in parallel without having to explicitly program for it. This is great !

Now when I try to use several nodes at the same time using a script like this one :

#$ -S /bin/bash
#$ -N comparison_of_architecture
#$ -pe mvapich2-rostam 32
#4 -tc 4
#$ -o /scratch04.local/cnelias/Deep-Jazz/logs/out_comparison_training.txt
#$ -e /scratch04.local/cnelias/Deep-Jazz/logs/err_comparison_training.txt
#$ -t 1
#$ -cwd

I see that 4 nodes are being used but only one is actually doing work, so "only" 32 cores are in use.

I have no knowledge of parallel programming and I don't understand a thing in the tutorial provided on PyTorch's website, I am afraid this is completely out of my scope.

Are you aware of a simple way to let a PyTorch program run on several machines without having to explicitly program the exchanges of the messages and computation between these machines ?

PS : I unfortunately don't have a GPU and the cluster I am using also doesn't, otherwise I would have tried it.

Output:
tl;dr There is no easy solution.

There are two ways how you can parallelize training of a deep learning model. The most commonly used is data parallelism (as opposed to model parallelism). In that case, you have a copy of the model on each device, run the model and back-propagation on each device independently and get the weight gradients. Now, the tricky part begins. You need to collect all gradients at a single place, sum them (differentiation is linear w.r.t. summation) and do the optimizer step. The optimizer computes the weight updates and you need to tell each copy of your model how to update the weights.

PyTorch can somehow do this for multiple GPUs on a single machine, but as far as I know, there is no ready-made solution to do this on multiple machines.