Instruction: Force installing torchvision 0.4.2 when I am forced to use pytorch 1.3.1 due to hardware constraints (ppc64le IBM)

Input:
I am in a weird scenario were I am forced to use torch 1.3.1 (due to hardware see: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/). I read from the pytorch docs that it's corresponding version of torchvision is 0.4.1 (https://pypi.org/project/torchvision/):

Installation
We recommend Anaconda as Python package management system. Please refer to pytorch.org for the detail of PyTorch (torch) installation. The following is the corresponding torchvision versions and supported Python versions.
Installation
We recommend Anaconda as Python package management system. Please refer to pytorch.org for the detail of PyTorch (torch) installation. The following is the corresponding torchvision versions and supported Python versions.

torch torchvision python
master / nightly  master / nightly    >=3.6
1.5.0 0.6.0   >=3.5
1.4.0 0.5.0   ==2.7, >=3.5, <=3.8
1.3.1 0.4.2   ==2.7, >=3.5, <=3.7
1.3.0 0.4.1   ==2.7, >=3.5, <=3.7
1.2.0 0.4.0   ==2.7, >=3.5, <=3.7
1.1.0 0.3.0   ==2.7, >=3.5, <=3.7
<=1.0.1   0.2.2   ==2.7, >=3.5, <=3.7

but for some reason I have the wrong version of it:
torchvision 0.2.2 pypi_0 pypi

is there a way to install the right version of torchvision?

What I've tried:
First I tried force installing the right version with conda. Conda couldn't find the version of torchvision that I need:
$ conda install torchvision==0.4.2
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.4.2

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-ppc64le
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-ppc64le
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Then I proceeded to try to install it regardless with pip
$ pip install torchvision==0.4.2
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement torchvision==0.4.2 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
ERROR: No matching distribution found for torchvision==0.4.2

got an error too.
Is there anything else to try?

I tried but it failed:
conda install torchvision==0.4.2 -c pytorch

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.4.2

Current channels:

  - https://conda.anaconda.org/pytorch/linux-ppc64le
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/linux-ppc64le
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-ppc64le
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

related:

crossposted SO: Force installing torchvision 0.4.2 when I am forced to use pytorch 1.3.1 due to hardware constraints (ppc64le IBM)
crossposted pytorch forum: https://discuss.pytorch.org/t/force-installing-torchvision/97279
crossposted reddit pytorch: https://www.reddit.com/r/pytorch/comments/iyf2qn/force_installing_torchvision/
crossposted reddit ibm: https://www.reddit.com/r/IBM/comments/iyhzex/force_installing_torchvision_042_when_i_am_forced/
real problem is installing torchmeta: https://github.com/tristandeleu/pytorch-meta/issues/95
https://www.ibm.com/mysupport/s/forumsquestion?id=0D50z00006gaxV9CAI
quora: https://www.quora.com/unanswered/How-does-one-install-specific-Python-packages-in-Conda-from-IBM-architectures
reddit ibm2: https://www.reddit.com/r/newIBM/comments/iyij10/force_installing_torchvision_042_when_i_am_forced/
gitissue for ibm: https://github.com/IBM/powerai/issues/268
gitissue in pytorch

Output:
For all details check (https://github.com/IBM/powerai/issues/268).
Make sure you have the right conda channel prepended:
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/

then install the powerai wmlce that you want e.g. 1.7.0 (most recent as of this writing):
conda create -n my_new_env python=3.7 powerai=1.7.0
conda activate my_new_env


Instruction: Why does requires_grad turns from true to false when doing torch.nn.conv2d operation?

Input:
I have Unet network which takes in MRI images of the brain, where the goal is to segment white substance in the brain. The images has the shape 256x256x183 (reshaped to 183x256x256) (FLAIR and T1 images). The problem I am having is that before sending the input to the Unet network, I have requires_grad=True on my pytorch tensor, but after one torch.nn.conv2d operation the requires_grad=False. This is a huge problem since the gradient will not update and learn.
from collections import OrderedDict

import torch
import torch.nn as nn

class UNet(nn.Module):

    def __init__(self, in_channels=3, out_channels=1, init_features=32):
        super(UNet, self).__init__()

        features = init_features
        self.encoder1 = UNet._block(in_channels, features, name="enc1")
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder2 = UNet._block(features, features * 2, name="enc2")
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder3 = UNet._block(features * 2, features * 4, name="enc3")
        self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder4 = UNet._block(features * 4, features * 8, name="enc4")
        self.pool4 = nn.MaxPool2d(kernel_size=2, stride=2)

        self.bottleneck = UNet._block(features * 8, features * 16, name="bottleneck")

        self.upconv4 = nn.ConvTranspose2d(
            features * 16, features * 8, kernel_size=2, stride=2
        )
        self.decoder4 = UNet._block((features * 8) * 2, features * 8, name="dec4")
        self.upconv3 = nn.ConvTranspose2d(
            features * 8, features * 4, kernel_size=2, stride=2
        )
        self.decoder3 = UNet._block((features * 4) * 2, features * 4, name="dec3")
        self.upconv2 = nn.ConvTranspose2d(
            features * 4, features * 2, kernel_size=2, stride=2
        )
        self.decoder2 = UNet._block((features * 2) * 2, features * 2, name="dec2")
        self.upconv1 = nn.ConvTranspose2d(
            features * 2, features, kernel_size=2, stride=2
        )
        self.decoder1 = UNet._block(features * 2, features, name="dec1")

        self.conv = nn.Conv2d(
            in_channels=features, out_channels=out_channels, kernel_size=1
        )

    def forward(self, x):

        print(x.requires_grad) #<---- here it is true
        enc1 = self.encoder1(x)#<---- where the problem happens

        print(enc1.requires_grad) #<---- here it is false
        enc2 = self.encoder2(self.pool1(enc1))
        print(enc2.requires_grad)
        enc3 = self.encoder3(self.pool2(enc2))
        print(enc3.requires_grad)
        enc4 = self.encoder4(self.pool3(enc3))
        print(enc4.requires_grad)

        bottleneck = self.bottleneck(self.pool4(enc4))
        print(bottleneck.requires_grad)

        dec4 = self.upconv4(bottleneck)
        print(dec4.requires_grad)
        dec4 = torch.cat((dec4, enc4), dim=1)
        print(dec4.requires_grad)
        dec4 = self.decoder4(dec4)
        print(dec4.requires_grad)
        dec3 = self.upconv3(dec4)
        print(dec3.requires_grad)
        dec3 = torch.cat((dec3, enc3), dim=1)
        print(dec3.requires_grad)
        dec3 = self.decoder3(dec3)
        print(dec3.requires_grad)
        dec2 = self.upconv2(dec3)
        print(dec2.requires_grad)
        dec2 = torch.cat((dec2, enc2), dim=1)
        print(dec2.requires_grad)
        dec2 = self.decoder2(dec2)
        print(dec2.requires_grad)
        dec1 = self.upconv1(dec2)
        print(dec1.requires_grad)
        dec1 = torch.cat((dec1, enc1), dim=1)
        print(dec1.requires_grad)
        dec1 = self.decoder1(dec1)
        print(dec1.requires_grad)
        print("going out")
        return torch.sigmoid(self.conv(dec1))

    @staticmethod
    def _block(in_channels, features, name):
        return nn.Sequential(
            OrderedDict(
                [
                    (
                        name + "conv1",
                        nn.Conv2d(
                            in_channels=in_channels,
                            out_channels=features,
                            kernel_size=3,
                            padding=1,
                            bias=False,
                        ),
                    ),
                    (name + "norm1", nn.BatchNorm2d(num_features=features)),
                    (name + "relu1", nn.ReLU(inplace=True)),
                    (
                        name + "conv2",
                        nn.Conv2d(
                            in_channels=features,
                            out_channels=features,
                            kernel_size=3,
                            padding=1,
                            bias=False,
                        ),
                    ),
                    (name + "norm2", nn.BatchNorm2d(num_features=features)),
                    (name + "relu2", nn.ReLU(inplace=True)),
                ]
            )
        )

Edit:
This is the training code
class run_network:
def __init__(self, eta, epoch, batch_size, train_file_path, validation_file_path, shuffle_after_epoch = True):
    self.eta = eta
    self.epoch = epoch
    self.batch_size = batch_size
    self.train_file_path = train_file_path
    self.validation_file_path = validation_file_path
    self.shuffle_after_epoch = shuffle_after_epoch

def __call__(self, is_train = False):

    device = torch.device("cpu" if not torch.cuda.is_available() else torch.cuda())
    unet = torch.hub.load('mateuszbuda/brain-segmentation-pytorch', 'unet',
    in_channels=3, out_channels=1, init_features=32, pretrained=True)
    unet.to(device)
    unet = unet.double()

    optimizer = optim.Adam(unet.parameters(), lr=self.eta)
    dsc_loss = DiceLoss()

    Load_training   = NiftiLoader(self.train_file_path)
    Load_validation = NiftiLoader(self.validation_file_path)

    mean_flair, mean_t1, std_flair, std_t1 = Load_training.average_mean_and_std(20, 79,99)

    total_mean = [mean_flair, mean_t1]
    total_std = [std_flair, std_t1]

    loss_train = []
    loss_validation = []

    for current_epoch in tqdm(range(self.epoch)):
        for phase in ["train", "validation"]:

            if phase == "train":
                mini_batch = Load_training.create_batch(self.batch_size, self.shuffle_after_epoch)
                unet.train()
                print("her22")

            if phase == "validation":
                print("her")
                mini_batch = Load_validation.create_batch(self.batch_size, self.shuffle_after_epoch)
                unet.eval()

            dim1, dim2, dim3 = mini_batch.shape

            for iteration in range(1):
                if phase == "train":
                    current_batch = Load_training.Load_Image_batch(mini_batch, iteration)
                    image_batch = Load_training.image_zero_mean_normalizer(current_batch)
                if phase == "validation":
                    current_batch = Load_validation.Load_Image_batch(mini_batch, iteration)
                    image_batch = Load_training.image_zero_mean_normalizer(current_batch, False, mean_list, std_list)

                image_dim0, image_dim1, image_dim2, image_dim3, image_dim4 = image_batch.shape
                image_batch = image_batch.reshape((
                                                    image_dim0,
                                                    image_dim1*image_dim2,
                                                    image_dim3,
                                                    image_dim4
                                                    ))

                image_batch = np.swapaxes(image_batch, 0,1)
                image_batch = torch.as_tensor(image_batch)#.requires_grad_(True) #, requires_grad=True)
                image_batch = image_batch.to(device)
                print(image_batch.requires_grad)
                optimizer.zero_grad()

                with torch.set_grad_enabled(is_train == "train"):
                    for j in range(0, 10, 1):
                        # [183*5, 3, 256, 256] -> [12, 3, 256, 256]
                        # ANTALL ITERASJONER: (183*5/12) -> en chunk

                        input_image = image_batch[j:j+2,0:3,:,:]
                        print(input_image.requires_grad)
                        print("går inn")
                        y_predicted = unet(input_image)

                        print(y_predicted.requires_grad)
                        print(image_batch[j:j+2,3,:,:].requires_grad)
                        loss = dsc_loss(y_predicted.squeeze(1), image_batch[j:j+2,3,:,:])
                        print(loss.requires_grad)

                        if phase == "train":
                            loss_train.append(loss.item())

                            loss.backward()
                            print(loss.item())
                            exit()
                            optimizer.step()
                            print(loss.item())
                            exit()
                        if phase == "validation":
                            loss_validation.append(loss.item())

Number of iteration and print statement are for experimenting what the cause could be.

Output:
The training code is fine and the input doesn't need a gradient at all, if you just want to train and update the weights.
The real problem is this line here
 with torch.set_grad_enabled(is_train == "train"):

So you want to disable the gradients if you are not training. The thing is is_train is a bool (judging form this: def __call__(self, is_train=False):), so the comparisons will be always false and no gradients will bet set. Just change it to
with torch.set_grad_enabled(is_train):

and you will be fine.


Instruction: undefined symbol: THPVariableClaload_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

Input:
Do you know how I could fix this? I am trying to use https://github.com/benjiebob/SMALViewer/issues/3 repo however I get error on neural_renderer port:
$ python smal_viewer.py
Traceback (most recent call last):
  File "smal_viewer.py", line 2, in <module>
    import pyqt_viewer
  File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 13, in <module>
    from smal.smal3d_renderer import SMAL3DRenderer
  File "/home/mona/research/3danimals/SMALViewer/smal/smal3d_renderer.py", line 6, in <module>
    import neural_renderer as nr
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/__init__.py", line 3, in <module>
    from .load_obj import load_obj
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/load_obj.py", line 8, in <module>
    import neural_renderer.cuda.load_textures as load_textures_cuda
ImportError: /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

Here is some details:
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.is_available()
True

$ gcc --version
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

Here is the neural renderer git repo: https://github.com/daniilidis-group/neural_renderer
I installed the neural renderer using pip install neural_renderer_pytorch

Output:
It looks like you didn't build the neural_renderer_pytorch yourself, but used a wheel. However, this wheel was built with an older pytorch version and doesn't work with the current pytorch version on your machine.
Build neural_renderer from the source (after deinstalling neural_renderer you have now) using your current pytorch-version  i.e.
$ pip uninstall neural-renderer-pytorch

$ pip install https://github.com/daniilidis-group/neural_renderer/zipball/master

and it should work.

Until pytorch 1.5, it used a someway brittle way of building extensions on Linux: despite depending on torch, extensions didn't link explicitly against libtorch.so. The missing symbols were provided only because import torch loaded libtorch.so with RTLD_GLOBAL, thus making its symbols globally visibile/accessible - this is the reason why prior to loading those extensions (e.g. neural_renderer_pytorch like here) one had to import torch.
One could enforce the old behavior setting RTLD_GLOBAL prior to importing torch for the very first time it is happening:
import sys; import ctypes;
sys.setdlopenflags(sys.getdlopenflags() | ctypes.RTLD_GLOBAL)
import torch # now all symbols of torch
             # have global visibility and can be used in
             # other extensions

However, using RTLD_GLOBAL is quite dangerous as it could possibly interpose symbols that are unrelated and lead to subtle bugs or even crashes.
Thus, since 1.5 pytorch no longer uses RTLD_GLOBAL, but links explicitly against libpytorch.so (see this commit) and extensions built with older pytorch versions will not work.


Instruction: How to make a Trainer pad inputs in a batch with huggingface-transformers?

Input:
I'm trying to train a model using a Trainer, according to the documentation (https://huggingface.co/transformers/master/main_classes/trainer.html#transformers.Trainer) I can specify a tokenizer:

tokenizer (PreTrainedTokenizerBase, optional) – The tokenizer used to
preprocess the data. If provided, will be used to automatically pad
the inputs the maximum length when batching inputs, and it will be
saved along the model to make it easier to rerun an interrupted
training or reuse the fine-tuned model.

So padding should be handled automatically, but when trying to run it I get this error:

ValueError: Unable to create tensor, you should probably activate
truncation and/or padding with 'padding=True' 'truncation=True' to
have batched tensors with the same length.

The tokenizer is created this way:
tokenizer = BertTokenizerFast.from_pretrained(pretrained_model)

And the Trainer like that:
trainer = Trainer(
    tokenizer=tokenizer,
    model=model,
    args=training_args,
    train_dataset=train,
    eval_dataset=dev,
    compute_metrics=compute_metrics
)

I've tried putting the padding and truncation parameters in the tokenizer, in the Trainer, and in the training_args. Nothing does. Any idea?

Output:
Look at the columns your tokenizer is returning. You might wanna limit it to only the required columns.
For Example
def preprocess_function(examples):
   #function to tokenize the dataset.
   if sentence2_key is None:
       return tokenizer(examples[sentence1_key], truncation=True, padding=True)
   return tokenizer(examples[sentence1_key], examples[sentence2_key], truncation=True, padding=True)

encoded_dataset = dataset.map(preprocess_function, batched=True, load_from_cache_file=False)

#Thing you should do is

columns_to_return = ['input_ids', 'label', 'attention_mask']
encoded_dataset.set_format(type='torch', columns=columns_to_return)


Instruction: How does one install torchmeta for a ppc64le architecture in pytorch?

Input:
I was trying to use torchmeta in a ppc64le architecture. Unfortunately it's not been easy to install since ppc64le requires special binaries to work.
I eventually managed to get the right binaries for pytorch and torchvision by following these instructions (that prepend the right ibm channel with the conda binaries, plus installs all the required files too):
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
conda create -n my_new_env python=3.7 powerai=1.7.0
conda activate my_new_env

after that I proceeded to install the right version of torchmeta, which was 1.3.1 since ppc64le only has pytorch 1.3.1 and torchvision 0.4.2. So I did:
pip install torchmeta==1.3.1

but now I have a new error that it cannot find the right version of h5py compatible with what I want to do. The error message is to large to paste but I will paste what I hope are useful part of it:
(my_new_env) [miranda9@hal-login ~]$ pip install torchmeta==1.3.1
Collecting torchmeta==1.3.1
  Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB)
Requirement already satisfied: requests in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (2.22.0)
Requirement already satisfied: torchvision<0.6.0,>=0.4.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (0.4.2)
Requirement already satisfied: torch<1.5.0,>=1.3.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.3.1)
Processing ./.cache/pip/wheels/87/f5/ad/9f04a48453875e8054c19f9fe3f50cbbe0c09b956835555019/Pillow-6.2.2-cp37-cp37m-linux_ppc64le.whl
Requirement already satisfied: numpy>=1.14.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.17.4)
Requirement already satisfied: tqdm>=4.0.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (4.36.1)
Collecting h5py~=2.9.0
  Using cached h5py-2.9.0.tar.gz (287 kB)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2020.6.20)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (1.25.10)
Requirement already satisfied: idna<2.9,>=2.5 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2.8)
Requirement already satisfied: six in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchvision<0.6.0,>=0.4.0->torchmeta==1.3.1) (1.13.0)
Building wheels for collected packages: h5py
  Building wheel for h5py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ccg1oj0n
       cwd: /tmp/pip-install-bpmeop26/h5py/
  Complete output (1321 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-ppc64le-3.7
  creating build/lib.linux-ppc64le-3.7/h5py
  copying h5py/__init__.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/h5py_warnings.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/highlevel.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/ipy_completer.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/version.py -> build/lib.linux-ppc64le-3.7/h5py
  creating build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/attrs.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/base.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/compat.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dataset.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/datatype.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dims.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/files.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/filters.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/group.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections2.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/vds.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/common.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  creating build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_base.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dataset.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file_image.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_group.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5f.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5p.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5t.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_objects.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_selections.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_slicing.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_filters.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_threads.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  running build_ext
  Autodetected HDF5 1.10.2
  ********************************************************************************
                         Summary of the h5py configuration

      Path to HDF5: None
      HDF5 Version: '1.10.2'
       MPI Enabled: False
  Rebuild Required: True

  ********************************************************************************
  Executing api_gen rebuild of defs
  Executing cythonize()
  [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
  /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd
    tree = Parsing.p_module(s, pxd, full_module_name)

...

  /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   #warning "Using deprecated NumPy API, disable it with " \
    ^
  In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
  /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
   #include "hdf5.h"
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for h5py
  Running setup.py clean for h5py
Failed to build h5py
DEPRECATION: Could not build wheels for h5py which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: Pillow, h5py, torchmeta
  Attempting uninstall: Pillow
    Found existing installation: Pillow 7.1.2
    Uninstalling Pillow-7.1.2:
      Successfully uninstalled Pillow-7.1.2
  Attempting uninstall: h5py
    Found existing installation: h5py 2.8.0
    Uninstalling h5py-2.8.0:
      Successfully uninstalled h5py-2.8.0
    Running setup.py install for h5py ... error
    ERROR: Command errored out with exit status 1:
     command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py

...

    copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    running build_ext
    Autodetected HDF5 1.10.2
    ********************************************************************************
                           Summary of the h5py configuration

        Path to HDF5: None
        HDF5 Version: '1.10.2'
         MPI Enabled: False
    Rebuild Required: True

    ********************************************************************************
    Executing cythonize()
    [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
    /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd

...

    warning: h5py/api_types_hdf5.pxd:730:6: 'H5Z_ERROR_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:731:6: 'H5Z_DISABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:732:6: 'H5Z_ENABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:733:6: 'H5Z_NO_EDC' redeclared
    building 'h5py.defs' extension
    creating build/temp.linux-ppc64le-3.7
    creating build/temp.linux-ppc64le-3.7/tmp
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py
    gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-bpmeop26/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-bpmeop26/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py/defs.o
    In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:
    /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it with " \
      ^
    In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
    /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
     #include "hdf5.h"
                      ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of h5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info
ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.

anyone know how I can successfully install a working torchmeta version in a ppc64le (using wmcle 1.7.0)?

related:

gitissue for torchmeta: https://github.com/tristandeleu/pytorch-meta/issues/95

IBM gitissue for torchmeta support: https://github.com/IBM/powerai/issues/269

h5py gitissue for torchmeta: https://github.com/h5py/h5py/issues/1678

IBM h5py support for torchmeta: https://github.com/IBM/powerai/issues/270

Output:
Because there are not wheels for powerpc for h5py you are installing h5py from source (from the tarball).  This requires both the Python and h5py development headers to be available, see https://docs.h5py.org/en/stable/build.html#source-installation.
Either install h5py from conda or install the required build dependencies.


Instruction: how "data" and "target" are choosen in a federated learning? (PySyft)

Input:
i can't understand how in function train() below, the variable (data, target) are choosen.
def train(args, model, device, federated_train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(federated_train_loader): # <-- now it is a distributed dataset
        model.send(data.location) # <-- NEW: send the model to the right location`

i guess they are 2 tensor representing 2 random images of dataset train, but then the loss function
loss = F.nll_loss(output, target)

is calculated at every interaction with different target?
Also i have different question: i trained the network with images of cats, then i test it with images of cars and the accuracy reached is 97%. How is this possible? is a proper value or i'm doing something wrong?
here is the entire code:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms

import syft as sy  # <-- NEW: import the Pysyft library
hook = sy.TorchHook(torch)  # <-- NEW: hook PyTorch ie add extra functionalities to support Federated Learning
bob = sy.VirtualWorker(hook, id="bob")  # <-- NEW: define remote worker bob
alice = sy.VirtualWorker(hook, id="alice")  # <-- NEW: and alice

class Arguments():
    def __init__(self):
        self.batch_size = 64
        self.test_batch_size = 1000
        self.epochs = 2
        self.lr = 0.01
        self.momentum = 0.5
        self.no_cuda = False
        self.seed = 1
        self.log_interval = 30
        self.save_model = False

args = Arguments()

use_cuda = not args.no_cuda and torch.cuda.is_available()

torch.manual_seed(args.seed)

device = torch.device("cuda" if use_cuda else "cpu")

kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}

federated_train_loader = sy.FederatedDataLoader( # <-- this is now a FederatedDataLoader
    datasets.MNIST("C:\\users...\\train", train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))
    .federate((bob, alice)), # <-- NEW: we distribute the dataset across all the workers, it's now a FederatedDataset
    batch_size=args.batch_size, shuffle=True, **kwargs)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST("C:\\Users...\\test", train=False, download=True, transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.test_batch_size, shuffle=True, **kwargs)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

def train(args, model, device, federated_train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(federated_train_loader): # <-- now it is a distributed dataset
        model.send(data.location) # <-- NEW: send the model to the right location
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        model.get() # <-- NEW: get the model back
        if batch_idx % args.log_interval == 0:
            loss = loss.get() # <-- NEW: get the loss back
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * args.batch_size, len(federated_train_loader) * args.batch_size,
                100. * batch_idx / len(federated_train_loader), loss.item()))

def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss
            pred = output.argmax(1, keepdim=True) # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

model = Net().to(device)
optimizer = optim.SGD(model.parameters(), lr=args.lr) # TODO momentum is not supported at the moment

for epoch in range(1, args.epochs + 1):
    train(args, model, device, federated_train_loader, optimizer, epoch)
    test(args, model, device, test_loader)

if (args.save_model):
    torch.save(model.state_dict(), "mnist_cnn.pt")

Output:
Consider it like this. When you hook torch, all your torch tensors will get additional functionality - methods like .send(), .federate(), and attributes like .location and ._objects. Your data and target, which were once torch tensors, became pointers to tensors residing in different VirtualWorker objects due to .federate((bob, alice)).
Now data and target have additional attributes that includes .location, which will return the location of that tensor - data/target pointed by the pointer called data/target.
Federated learning sends the global model to this location, as seen in model.send(data.location).
Now, model is a pointer residing at the same location and data is also a pointer residing there. Hence when you take the output as output = model(data), output will also reside there and all we (the central server or in other words, the VirtualWorker called 'me') will get is a pointer to that output.
Now, regarding your doubt on loss calculation, since output and target are both residing in that same location, calculation of loss will also happen there. Same goes for backprop and step.
Finally, you can see model.get(), here is where the central server pulls the remote model using the pointer called model. (I'm not sure if it should be model = model.get() though).
So anything with .get() will be pulled from that worker and will be returned in our python statement. Also note that .get() will remove that object from it's location when called. Hence use .copy().get() if you are going to need it further.


Instruction: Using both vCPU's with google cloud computing. Python code. PyTorch

Input:
I am new to cloud computing. I made a virtual machine in google cloud computing, machine type:
e2-highcpu-2 (2 vCPU's, 2 GB geheugen)
I run a script with running the command
python3 simulation1.py
When I look at the output control screen, I note that only 50% of the CPU power is used. So I just use one of my 2 CPU's. Is there a way to make full use of the computing power ?

Output:
Looks like your question can be resumed to "is Python capable of running on multiple cores?"
And you can find the answer to that question perfectly explained in this post.
Basically:
Python threads cannot take advantage of many cores. This is due to an internal implementation detail called the GIL (global interpreter lock) in the C implementation of python.
You can either use something like multiprocessing, celery or mpi4py to split the parallel work into another process;
Or you can use something like Jython or IronPython to use an alternative interpreter that doesn't have a GIL.
If you are already using any of the tools mentioned above. you could also add more details about your code.


Instruction: PyTorch linear regression model

Input:
I have a multivariate linear regression problem in which each data point looks like this:
y_i = 3                             # Some integer between 0 and 20
X_i = [0.5, 80, 0.004, 0.5, 0.789]  # A 5 dimensional vector

I can train a simple linear model by using sklearn, something like:
from sklearn import linear_model
ols = linear_model.LinearRegression()
model = ols.fit(X, y)

This gets me an accuracy of ~55 % (a linear model is not suitable for the problem, but this is a baseline to demonstrate the feasibility of modelling the problem, and a way for me to learn PyTorch, having use TensorFlow previously).
When I try to train a linear model using PyTorch I am defining the model as:
class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, D_out):

        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, D_out)

    def forward(self, x):

        y_pred = self.linear1(x)
        return y_pred

D_in, D_out = 5, 1
model = TwoLayerNet(D_in, D_out)

And training as:
epochs = 10
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
for epoch in range(epochs):
    for n, batch in enumerate(batches):
        X = []
        y = []
        for values in batch:
            X.append(values[0])
            y.append(values[1])

        X = torch.from_numpy(np.asarray(X))
        y = torch.from_numpy(np.asarray(y))
        # Forward pass: Compute predicted y by passing x to the model
        optimizer.zero_grad()
        y_pred = model(X)
        # Compute and print loss
        loss = criterion(y_pred, y)
        if n % 100 == 99:
            print(n, loss.item())

        # Zero gradients, perform a backward pass, and update the weights.

        loss.backward()
        optimizer.step()

This is just some code from the PyTorch documentation which I have adjusted. The current set up only achieves ~25%, no where near the accuracy that I would expect from the linear model. Am I doing something incorrect in the model training wrt PyTorch?

Output:
tam63,
you are missing activation function in the model definition. replace
y_pred = self.linear1(x)

with
y_pred =  F.relu(self.linear1(x))

there are few more things that may go wrong.
For instance (1) too low a learning rate, (2) too few layers (add one more). If you are familiar with TF as you say, try same problem in TF and once you have good results - translate it into Pytorch with same network structure and same hyperparameters.


Instruction: PyTorch: difference between type(a), a.type, a.type()

Input:
suppose a is a tensor, then what's the difference between:
type(a)
a.type
a.type()
I couldn't find a document differentiating these.

Output:
type is the python in-built method.

It will return type of object. like <class 'torch.Tensor'>

torch.Tensor.type (x.type()) is pytorch in-built method.

It will return type of data stored inside tensor. like torch.DoubleTensor, etc.

Edit:
And about x.type() vs x.type -
When you write a function name with parentheses x.type () it will actually execute the function and return its value. Whereas without parentheses x.type it is simply a reference to function.


Instruction: Jupyter doesn't recognize torchaudio

Input:
I am trying to install torchaudio to use in a Jupyter notebook but when i import it i get the error:
ModuleNotFoundError: No module named 'torchaudio'

I tried to import it in a .py file that the notebook uses but to with no success. I thought maybe it wasnt installed properly but when i try to install it using pip install torchaudio i get "requirement already satisfied".
Im lost, how can i import it successfully?

Output:
pip install torchaudio

should return:
Collecting torchaudio
  Downloading https://files.pythonhosted.org/packages/96/34/c651430dea231e382ddf2eb5773239bf4885d9528f640a4ef39b12894cb8/torchaudio-0.6.0-cp36-cp36m-manylinux1_x86_64.whl (6.7MB)

     |████████████████████████████████| 6.7MB 2.4MB/s
Requirement already satisfied: torch==1.6.0 in /usr/local/lib/python3.6/dist-packages (from torchaudio) (1.6.0+cu101)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch==1.6.0->torchaudio) (0.16.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch==1.6.0->torchaudio) (1.18.5)
Installing collected packages: torchaudio
Successfully installed torchaudio-0.6.0

And everything should work as expected.


Instruction: matching PyTorch tensor dimensions

Input:
I am having some issues with regards to the dimensionality of my tensors in my training function at present. I am using the MNIST dataset, so 10 possible targets, and originally wrote the prototype code using a training batch size of 10, which was in retrospect not the wisest choice. It gave poor results during some earlier tests, and increasing the amount of training iterations saw no benefit. Upon trying to then increase the batch size, I realised that what I had written was not that general, and I was likely never training it on the proper data. Below is my training function:
def Train(tLoops, Lrate):
    for _ in range(tLoops):
        tempData = train_data.view(batch_size_train, 1, 1, -1)
        output = net(tempData)
        trainTarget = train_targets
        criterion = nn.MSELoss()
        print("target:", trainTarget.size())
        print("Output:", output.size())
        loss = criterion(output, trainTarget.float())
        # print("error is:", loss)
        net.zero_grad()  # zeroes the gradient buffers of all parameters
        loss.backward()
        for j in net.parameters():
            j.data.sub_(j.grad.data * Lrate)

of which the print functions return
target: torch.Size([100])
Output: torch.Size([100, 1, 1, 10])

before the error message on the line where loss is calculated;
RuntimeError: The size of tensor a (10) must match the size of tensor b (100) at non-singleton dimension 3

The first print, target, is a 1-dimensional list of the respective ground truth values for each image. Output contains the output of the neural net for each of those 100 samples, so a 10 x 100 list, however from skimming and reshaping the data from 28 x 28 to 1 x 784 earlier, I seem to have extra dimensions unnecessarily. Does PyTorch provide a way to remove these? I couldn't find anything in the documentation, or is there something else that could be my issue?

Output:
There are several problems in your training script. I will address each of them below.

First, you should NOT do data batching by hand. Pytorch/torchvision have functions for that, use a dataset and a data loader: https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html.

You should also NEVER update the parameters of you network by hand. Use an Optimizer: https://pytorch.org/docs/stable/optim.html. In your case, SGD without momentum will have the same effect.

The dimensionality of your input seems to be wrong, for MNIST an input tensor should be (batch_size, 1, 28, 28) or (batch_size, 784) if you're training a MLP. Furthermore, the output of your network should be (batch_size, 10)


Instruction: Pytorch: What happens to memory when moving tensor to GPU?

Input:
I’m trying to understand what happens to both RAM and GPU memory when a tensor is sent to the GPU.
In the following code sample, I create two tensors - large tensor arr = torch.Tensor.ones((10000, 10000)) and small tensor c = torch.Tensor.ones(1). Tensor c is sent to GPU inside the target function step which is called by multiprocessing.Pool. In doing so, each child process uses 487 MB on the GPU and RAM usage goes to 5 GB. Note that the large tensor arr is just created once before calling Pool and not passed as an argument to the target function. Ram usage does not explode when everything is on the CPU.
I have the following questions on this example:
I’m sending torch.Tensor.ones(1) to GPU and yet it consumes 487 MB of GPU memory. Does CUDA allocate a minimum amount of memory on the GPU even if the underlying tensor is very small? GPU memory is not a problem for me, and this is just for me to understand how the allocation is done.
The problem lies in RAM usage. Even though I am sending a small tensor to the GPU, it appears as if everything in memory (large tensor arr) is copied for every child process (possibly to pinned memory). So when a tensor is sent to the GPU, what objects are copied to pinned memory? I’m missing something here as it does not make sense to prepare everything to be sent to GPU when I’m only sending a particular object.
Thanks!
from multiprocessing import get_context
import time
import torch

dim = 10000
sleep_time = 2
npe = 4  # number of parallel executions

# cuda
if torch.cuda.is_available():
    dev = 'cuda:0'
else:
    dev = "cpu"
device = torch.device(dev)

def step(i):
    c = torch.ones(1)
    # comment the line below to see no memory increase
    c = c.to(device)
    time.sleep(sleep_time)

if __name__ == '__main__':
    arr = torch.ones((dim, dim))

    # create list of inputs to be executed in parallel
    inp = list(range(npe))

    # sleep added before and after launching multiprocessing to monitor the memory consumption
    print('before pool')  # to check memory with top or htop
    time.sleep(sleep_time)

    context = get_context('spawn')
    with context.Pool(npe) as pool:
        print('after pool')  # to check memory with top or htop
        time.sleep(sleep_time)

        pool.map(step, inp)

    time.sleep(sleep_time)

Output:
I’m sending torch.Tensor.ones(1) to GPU and yet it consumes 487 MB of GPU memory. Does CUDA allocate a minimum amount of memory on the GPU even if the underlying tensor is very small?

The CUDA device runtime reserves memory for all sorts of things at context establishment, some of which are fixed in size and some of which are variable and can be controlled by API calls (see here for some more information). It is completely normal that the first API call which explicitly or lazily establishes a context on the device will produce a jump in GPU memory consumption. In this case I imagine the first tensor creation is triggering this memory overhead allocation. This is a property of the CUDA runtime and not PyTorch or the tensor.


Instruction: What does this strange C++ syntax do?

Input:
From what I understand, 3 arguments are being passed to m.def(). I don't understand the syntax of the second argument passed, i.e. [] (Observable...){} What does this mean?
m.def(
  "DumpHistogramFile",
  [](Observable<NetBase>::Observer* ob) {
    HistogramNetObserver* hist_ob =
        dynamic_cast_if_rtti<HistogramNetObserver*>(ob);
    hist_ob->DumpHistogramFile();
  },
  pybind11::arg("ob"));

I found this syntax in the pytorch source code

Output: The second argument is a function that takes a pointer to an Observer and doesn't return anything. When executed, it calls DumpHistogramFile. This is called a lambda expression.


Instruction: Pytorch for Raspberry Pi 3 B+

Input:
I have successfully cross compiled Pytorch 1.7 for Raspberry Pi3 B+ and the following wheel was generated as the result:
torch-1.7.0a0-cp37-cp37m-linux_armv7l.whl

However, when I try to install the wheel on the pi, I get this error:
ERROR: torch-1.7.0a0-cp37-cp37m-linux_armv7l.whl is not a supported wheel on this platform.

When I do lscpu, I see that the arm7l was correctly used as the target hw:
    Architecture:          armv7l
    Byte Order:            Little Endian
    CPU(s):                4
    On-line CPU(s) list:   0-3
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             1
    Model:                 4
    Model name:            ARMv7 Processor rev 4 (v7l)
    CPU max MHz:           1400.0000
    CPU min MHz:           600.0000
    BogoMIPS:              38.40
    Flags:                 half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

And here is the output of pep425tags:
[('cp38', 'cp38', 'linux_armv7l'), ('cp38', 'abi3', 'linux_armv7l'),
('cp38', 'none', 'linux_armv7l'), ('cp37', 'abi3', 'linux_armv7l'),
('cp36', 'abi3', 'linux_armv7l'), ('cp35', 'abi3', 'linux_armv7l'),
('cp34', 'abi3', 'linux_armv7l'), ('cp33', 'abi3', 'linux_armv7l'),
('cp32', 'abi3', 'linux_armv7l'), ('py3', 'none', 'linux_armv7l'),
('cp38', 'none', 'any'), ('cp3', 'none', 'any'), ('py38', 'none',
'any'), ('py3', 'none', 'any'), ('py37', 'none', 'any'), ('py36',
'none', 'any'), ('py35', 'none', 'any'), ('py34', 'none', 'any'),
('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none',
'any'), ('py30', 'none', 'any')]

So I am wondering, what am I missing?

Output: I eventually found the answer, I had python 3.8 on my target hw (raspberry pi 3B+) and I used had python 3.7 on my build system. Downgrading python 3.8 to python 3.7 on target hw fixed the issue.


Instruction: AttributeError: 'Function' object has no attribute 'block_variable'

Input:
I have written a subclass of torch_fenics. In this, the input is a vector from DG space. I use this input in the weak formulation and then calculate the solution. Further, I need the gradient of the solution with respect to the given input.

I get the following error log on running the same

~/miniconda3/envs/py37/lib/python3.7/site-packages/torch/autograd/function.py in apply(self, *args)
75
76     def apply(self, *args):
---> 77         return self._forward_cls.backward(self, *args)
78
79
~/miniconda3/envs/py37/lib/python3.7/site-packages/torch_fenics/torch_fenics.py in backward(ctx, *grad_outputs)
88         # Check which gradients need to be computed
89         controls = list(map(fenics_adjoint.Control,
---> 90                             (c for g, c in zip(ctx.needs_input_grad[1:], ctx.fenics_inputs) if g)))
91
92         # Compute and accumulate gradient for each output with respect to each input
~/miniconda3/envs/py37/lib/python3.7/site-packages/pyadjoint/control.py in init(self, control)
38     def init(self, control):
39         self.control = control
---> 40         self.block_variable = control.block_variable
41
42     def data(self):
AttributeError: 'Function' object has no attribute 'block_variable'

Output: Don't import dolfin in your code. It will resolve the issue.


Instruction: Creating a Pseudo-Cyclic signal using a cyclic signal

Input:
I am looking for a way using numpy or pytorch to skew a tensor.
For example given an array of samples of sin(x), i hope to get a skewed version of it (preferably the same size) such that the cycle of the function is either stretched or shrinks or even both (if it can interpolate it randomly), so in some parts the frequency is higher and in some parts lower.
I need to create "pseudo-cyclic" signals which means that they are almost cyclic but not perfectly.

Output:
You can play around with (for instance) locally varying frequency. Considering a sine function as a base periodic function, using a locally varying frequency can give "stretched" and/or "dilated" effect.
Example 1: chirp function with linear frequency change (check the wiki page for more information):
import matplotlib.pyplot as plt
import numpy as np

# initialize
n = 1000
x = np.linspace(0, 10, n)

# variying frequency between 2 and 4
f = np.linspace(2, 4, n)
y = np.sin(f * x)

# plot local frequency values and signal
plt.subplot(211)
plt.plot(x, f)
plt.ylabel('Local frequency')
plt.subplot(212)
plt.plot(x, y)
plt.ylabel('signal')
plt.xlabel('samples')
plt.show()

leading to increasing frequency pseudo-periodic signal:

Example 2: varying frequency with arbitrary function (here polynomial):
import matplotlib.pyplot as plt
import numpy as np

# initialize
n = 1000
x = np.linspace(0, 10, n)

# variying frequency between 2 and 4
f = .5 * (2 + (x - 5) ** 2 + 2 * x)
y = np.sin(f * x)

# plot local frequency values and signal
plt.subplot(211)
plt.plot(x, f)
plt.ylabel('Local frequency')
plt.subplot(212)
plt.plot(x, y)
plt.ylabel('signal')
plt.xlabel('samples')
plt.show()

with the following output

Playing around with the frequency function can help getting other pseudo-periodic signal shapes. Also note that scipy comes with a bank of signal generators like sweep_poly from which you can start with in order to get to randomly varying signal frequencies.


Instruction: deep neural network model stops learning after one epoch

Input:
I am training a unsupervised NN model and for some reason, after exactly one epoch (80 steps), model stops learning.
]
Do you have any idea why it might happen and what should I do to prevent it?
This is more info about my NN:
I have a deep NN that tries to solve an optimization problem. My loss function is customized and it is my objective function in the optimization problem.
So if my optimization problems is min f(x) ==> loss, now in my DNN loss = f(x). I have 64 input, 64 output, 3 layers in between :
self.l1 = nn.Linear(input_size, hidden_size)
self.relu1 = nn.LeakyReLU()
self.BN1 = nn.BatchNorm1d(hidden_size)

and last layer is:
self.l5 = nn.Linear(hidden_size, output_size)
self.tan5 = nn.Tanh()
self.BN5 = nn.BatchNorm1d(output_size)

to scale my network.
with more layers and nodes(doubles: 8 layers each 200 nodes), I can get a little more progress toward lower error, but again after 100 steps training error becomes flat!

Output:
The symptom is that the training loss stops being improved relatively early. Suppose that your problem is learnable at all, there are many reasons for the for this behavior. Following are most relavant:

Improper preprocessing of input: Neural network prefers input with
zero mean. E.g., if the input is all positive, it will restrict the
weights to be updated in the same direction, which may not be
desirable (https://youtu.be/gYpoJMlgyXA).

Therefore, you may want to subtract the mean from all the images (e.g., subtract 127.5 from each of the 3 channels). Scaling to make unit standard deviation in each channel may also be helpful.

Generalization ability of the network: The network is not complicated
or deep enough for the task.
This is very easy to check. You can train the network on just a few
images (says from 3 to 10). The network should be able to overfit the
data and drives the loss to almost 0. If it is not the case, you may
have to add more layers such as using more than 1 Dense layer.

Another good idea is to used pre-trained weights (in applications of Keras documentation). You may adjust the Dense layers at the top to fit with your problem.

Improper weight initialization. Improper weight initialization can
prevent the network from converging (https://youtu.be/gYpoJMlgyXA,
the same video as before).
For the ReLU activation, you may want to use He initialization
instead of the default Glorot initialiation. I find that this may be
necessary sometimes but not always.

Lastly, you can use debugging tools for Keras such as keras-vis, keplr-io, deep-viz-keras. They are very useful to open the blackbox of convolutional networks.
I faced the same problem then I followed the following:
After going through a blog post, I managed to determine that my problem resulted from the encoding of my labels. Originally I had them as one-hot encodings which looked like [[0, 1], [1, 0], [1, 0]] and in the blog post they were in the format [0 1 0 0 1]. Changing my labels to this and using binary crossentropy has gotten my model to work properly. Thanks to Ngoc Anh Huynh and rafaelvalle!


Instruction: Char RNN classification with batch size

Input:
I'm replicating this example for a classification with a Pytorch char-rnn.
for iter in range(1, n_iters + 1):
    category, line, category_tensor, line_tensor = randomTrainingExample()
    output, loss = train(category_tensor, line_tensor)
    current_loss += loss

I see that every epoch only 1 example is taken and random. I would like that each epoch all the dataset is taken with a specific batch size of examples. I can adjust the code to do this myself but I was wondering if some flags already exist.
Thank you

Output:
If you construct a Dataset class by inheriting from the PyTorch Dataset class and then feed it into the PyTorch DataLoader class, then you can set a parameter batch_sizeto determine how many examples you will get out in each iteration of your training loop.
I have followed the same tutorial as you. I can show you how I have used the PyTorch classes above to get the data in batches.
# load data into a DataFrame using the findFiles function as in the tutorial
files = findFiles('data/names') # load the files as in the tutorial into a dataframe
df_names = pd.concat([
    pd.read_table(f, names = ["names"], header = None)\
      .assign(lang = f.stem)\
    for f in files]).reset_index(drop = True)
print(df_names.head())

# output:
#      names      lang
# 0      Abe  Japanese
# 1  Abukara  Japanese
# 2   Adachi  Japanese
# 3     Aida  Japanese
# 4   Aihara  Japanese

# Make train and test data
from sklearn.model_selection import train_test_split
X_train, X_dev, y_train, y_dev = train_test_split(df_names.names, df_names.lang,
                                                   train_size = 0.8)
df_train = pd.concat([X_train, y_train], axis=1)
df_val = pd.concat([X_dev, y_dev], axis=1)

Now I construct a modified Dataset class using the dataframe(s) above by inheriting from the PyTorch Dataset class.
import torch
from torch.utils.data import Dataset, DataLoader
class NameDatasetReader(Dataset):
    def __init__(self, df: pd.DataFrame):
        self.df = df

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx: int):
        row = self.df.loc[idx] # gets a row from the df
        input_name = list(row.names) # turns name into a list of chars
        len_name = len(input_name) # length of name (used to pad packed sequence)
        labels = row.label # target
        return input_name, len_name, labels
train_dat =  NameDatasetReader(df_train) # make dataset from dataframe with training data

Now, the thing is that when you want to work with batches and sequences you need the sequences to be of equal length in each batch. That is why I also get the length of the extracted name from the dataframe in the __getitem__() function above. This is to be used in function that modifies the training examples used in each batch.
This is called a collate_batch function and in this example it modifies each batch of your training data such that the sequences in a given batch are of equal length.
# Dictionary of all letters (as in the original tutorial,
#  I have just inserted also an entry for the padding token)
all_letters_dict= dict(zip(all_letters, range(1, len(all_letters) +2)))
all_letters_dict['<PAD>'] = 0

# function to turn name into a tensor
def line_to_tensor(line):
    """turns name into a tensor of one hot encoded vectors"""
    tensor = torch.zeros(len(line),
                         len(all_letters_dict.keys())) # (name_len x vocab_size) - <PAD> is part of vocab
    for li, letter in enumerate(line):
        tensor[li][all_letters_dict[letter]] = 1
    return tensor

def collate_batch_lstm(input_data: Tuple) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """
    Combines multiple name samples into a single batch
    :param input_data: The combined input_ids, seq_lens, and labels for the batch
    :return: A tuple of tensors (input_ids, seq_lens, labels)
    """

    # loops over batch input and extracts vals
    names = [i[0] for i in input_data]
    seq_names_len = [i[1] for i in input_data]
    labels = [i[2] for i in input_data]

    max_length = max(seq_names_len) # longest sequence aka. name

    # Pad all of the input samples to the max length
    names = [(name + ["<PAD>"] * (max_length - len(name))) for name in names]

    input_ids = [line_to_tensor(name) for name in names] # turn each list of chars into a tensor with one hot vecs

    # Make sure each sample is max_length long
    assert (all(len(i) == max_length for i in input_ids))
    return torch.stack(input_ids), torch.tensor(seq_names_len), torch.tensor(labels)

Now, I can construct a dataloader by inserting the dataset object from above, the collate_batch_lstm() function above, and a given batch_size into the DataLoader class.
train_dat_loader = DataLoader(train_dat, batch_size = 4, collate_fn = collate_batch_lstm)

You can now iterate over train_dat_loader which returns a training batch with 4 names in each iteration.
Consider a given batch from train_dat_loader:
seq_tensor, seq_lengths, labels = iter(train_dat_loader).next()
print(seq_tensor.shape, seq_lengths.shape, labels.shape)
print(seq_tensor)
print(seq_lengths)
print(labels)
# output:
# torch.Size([4, 11, 59]) torch.Size([4]) torch.Size([4])
# tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          ...,
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.]],

#         [[0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          ...,
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.]],

#         [[0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          ...,
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.]],

#         [[0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          [0., 0., 0.,  ..., 0., 0., 0.],
#          ...,
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.],
#          [1., 0., 0.,  ..., 0., 0., 0.]]])
# tensor([11,  3,  8,  7])
# tensor([14,  1, 14,  2])

It gives us an tensor of size (4 x 11 x 59).
4 because we have specified that we want a batch size of 4.
11 is the length of the longest name in the given batch (all other names have been padded with zeros such that they are equal length).
59 is the number of characters in our vocabulary.
The next thing is to incorporate this into your training routine and use a packing routine to avoid doing redundant calculations on the zeros that you have padded your data with :)


Instruction: Input fixed length sequence of frames to CNN

Input:
I want my pytorch CNN to take as input a sequence of length SEQ_LEN of 32x32 RGB images concatenated along channels dimension. Therefore, a single input of the network has shape (32, 32, 3, SEQ_LEN). How should I define my CNN input layer?
The common way
SEQ_LEN = 10
input_conv = nn.Conv2d(in_channels=SEQ_LEN, out_channels=32, kernel_size=3)

BATCH_SIZE = 64
frames = np.random.randint(0, 255, size=(BATCH_SIZE, SEQ_LEN, 3, 32, 32))
frames_tensor = torch.tensor(frames)

input_conv(frames_tensor)

gives the error

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 10, 3, 3], but got 5-dimensional input of size [64, 10, 3, 32, 32] instead

Output: Given your comments, it sounds like your data is not fit for a 2D convolutional neural network at all, and that a 3D one (Conv3d) would be more appropriate. As you can see from its documentation, its input shape is what you would expect.


Instruction: PyTorch LSTM not learning in training

Input:
I have the following simple LSTM network:
class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.rnn = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.batch_size = None
        self.hidden = None

    def forward(self, x):
        h0, c0 = self.init_hidden(x)
        out, (hn, cn) = self.rnn(x, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        return [t for t in (h0, c0)]

I am initialising this model as"
model = LSTMClassifier(28, 10, 6, 1)

i.e. each input instance has 6 time steps and the dimension of each time step is 28, and the hidden dimension is 10. The inputs are being mapped to an output dim of 1.
The training data is being prepared in batches of size 16, meaning that the data passed in the training loop has the shape:
torch.Size([16, 6, 28])

With labels of shape:
batches[1][0].size()

An example of the input is:
tensor([[-0.3674,  0.0347, -0.2169, -0.0821, -0.3673, -0.1773,  1.1840, -0.2669,
         -0.4202, -0.1473, -0.1132, -0.4756, -0.3565,  0.5010,  0.1274, -0.1147,
          0.2783,  0.0836, -1.3251, -0.8067, -0.6447, -0.7396, -0.3241,  1.3329,
          1.3801,  0.8198,  0.6098,  0.0697],
        [-0.2710,  0.1596, -0.2524, -0.0821, -0.3673, -0.1773,  0.0302, -0.2099,
         -0.4550,  0.1451, -0.4561, -0.5207, -0.5657, -0.5287, -0.2690, -0.1147,
         -0.0346, -0.1043, -0.7515, -0.8392, -0.4745, -0.7396, -0.3924,  0.8122,
         -0.1624, -1.2198,  0.0326, -0.9306],
        [-0.1746,  0.0972, -0.2702, -0.0821, -0.3673, -0.1773, -0.0468, -1.1225,
         -0.4480, -0.4397,  0.4011, -1.1073, -1.0536, -0.1855, -0.7502, -0.1147,
         -0.0146, -0.1545, -0.1919, -0.1674,  0.0930, -0.7396,  0.8106,  1.1594,
          0.4546, -1.2198, -0.5446, -1.2640],
        [-0.2710,  0.0660, -0.2524, -0.0821, -0.4210, -0.1773,  1.8251, -0.5236,
         -0.4410, -0.7321,  0.4011, -0.6110, -0.2171,  1.1875, -0.2973, -0.1147,
         -0.1278,  0.7728, -0.9334, -0.5141, -2.1202,  1.3521, -0.9393,  0.5085,
         -0.4709,  0.8198, -1.1218,  0.0697],
        [-0.3674, -0.0277, -0.2347, -0.0821, -0.0448, -0.1773,  0.2866, -0.1386,
         -0.4271,  0.4375, -0.2847, -0.1146, -0.4262, -0.3571, -0.0425, -0.1147,
         -0.4207, -0.4552, -0.5277, -0.9584, -0.4177, -0.7396, -0.2967,  0.5085,
          0.4546, -1.2198, -0.3522, -1.2640],
        [-0.3674, -0.1447, -0.1991, -0.0821,  0.1701, -0.1773,  0.0430,  0.1324,
         -0.4271,  0.7299, -0.4561,  0.2915, -0.5657, -0.1855, -0.2123, -0.1147,
         -0.0413, -0.8311, -0.6396, -1.0451, -0.4177, -0.7396, -0.2967, -0.4028,
          0.7631, -1.2198, -0.3522, -1.2640]])

When I train the model as:
Epochs = 10
batch_size = 32
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=5e-4)
for epoch in range(Epochs):
    print(f"Epoch {epoch + 1}")
    for n, (X, y) in enumerate(batches):
        model.train()
        optimizer.zero_grad()
        y_pred = model(X)
        loss = criterion(y_pred, y)
        loss.backward()
        optimizer.step()

    model.eval()
    accurate = 0
    for X_instance, y_instance in zip(test_X, test_y):
        if y_instance == round(model(X_instance.view(-1, 6, 28)).detach().item()):
            accurate += 1
    print(f"Accuracy test set: {accurate/len(test_X)}")

The accuracy does not converge:
Epoch 1
Accuracy test set: 0.23169107856191745
Sample params:
tensor([-0.3356, -0.0105, -0.3405, -0.0049,  0.0037,  0.1707,  0.2685, -0.3893,
        -0.4707, -0.2872, -0.1544, -0.1455,  0.0393,  0.0774, -0.4194,  0.0780,
        -0.2177, -0.3829, -0.4679,  0.0370, -0.0794,  0.0455, -0.1331, -0.0169,
        -0.1551, -0.0348,  0.1746, -0.5163], grad_fn=<SelectBackward>)
tensor([ 0.2137, -0.2558,  0.1509, -0.0975,  0.5591,  0.0907, -0.1249,  0.3095,
         0.2112,  0.3134, -0.1581, -0.3051, -0.3559, -0.0177,  0.1485,  0.4397,
        -0.1441,  0.1705,  0.3230, -0.3236,  0.0692,  0.0920, -0.2691, -0.3695,
        -0.0692,  0.3747,  0.0149,  0.5216], grad_fn=<SelectBackward>)
Epoch 2
Accuracy test set: 0.23049267643142476
Sample params:
tensor([-0.3483, -0.0144, -0.3512,  0.0213, -0.0081,  0.1777,  0.2674, -0.4031,
        -0.4628, -0.3041, -0.1651, -0.1511,  0.0216,  0.0513, -0.4320,  0.0839,
        -0.2602, -0.3629, -0.4541,  0.0398, -0.0768,  0.0432, -0.1150, -0.0160,
        -0.1346, -0.0727,  0.1801, -0.5253], grad_fn=<SelectBackward>)
tensor([ 0.1879, -0.2534,  0.1461, -0.1141,  0.5735,  0.0872, -0.1286,  0.3273,
         0.2084,  0.3037, -0.1535, -0.2934, -0.3870, -0.0252,  0.1492,  0.4752,
        -0.1709,  0.1776,  0.3390, -0.3318,  0.0734,  0.1077, -0.2790, -0.3777,
        -0.0518,  0.3726,  0.0228,  0.5404], grad_fn=<SelectBackward>)
Epoch 3
Accuracy test set: 0.22982689747003995
Sample params:
tensor([-0.3725, -0.0069, -0.3623,  0.0393, -0.0167,  0.1748,  0.2577, -0.4183,
        -0.4681, -0.3196, -0.1657, -0.1613,  0.0122,  0.0268, -0.4361,  0.0838,
        -0.2962, -0.3566, -0.4344,  0.0366, -0.0822,  0.0486, -0.1150, -0.0295,
        -0.1080, -0.1094,  0.1841, -0.5336], grad_fn=<SelectBackward>)
tensor([ 0.1664, -0.2456,  0.1477, -0.1332,  0.5820,  0.0819, -0.1228,  0.3426,
         0.2066,  0.2985, -0.1464, -0.2824, -0.4199, -0.0323,  0.1530,  0.5057,
        -0.1991,  0.1856,  0.3407, -0.3347,  0.0800,  0.1203, -0.2791, -0.3863,
        -0.0426,  0.3760,  0.0327,  0.5641], grad_fn=<SelectBackward>)
Epoch 4
Accuracy test set: 0.23249001331557922
Sample params:
tensor([-0.3945,  0.0032, -0.3765,  0.0600, -0.0248,  0.1713,  0.2442, -0.4297,
        -0.4741, -0.3311, -0.1653, -0.1667,  0.0029,  0.0066, -0.4373,  0.0738,
        -0.3320, -0.3530, -0.4136,  0.0390, -0.0731,  0.0552, -0.1117, -0.0517,
        -0.0871, -0.1455,  0.1841, -0.5359], grad_fn=<SelectBackward>)
tensor([ 0.1495, -0.2292,  0.1524, -0.1473,  0.5938,  0.0661, -0.1157,  0.3626,
         0.2013,  0.2927, -0.1350, -0.2661, -0.4558, -0.0411,  0.1562,  0.5381,
        -0.2279,  0.1927,  0.3319, -0.3431,  0.0852,  0.1402, -0.2747, -0.4026,
        -0.0297,  0.3757,  0.0396,  0.5856], grad_fn=<SelectBackward>)

Have I made a mistake in the model definition?

Output:
So normally 6 layers in your LSTM are way to much. The input dimension is 28 (are you training MNIST, or are the inputs letters?) so 10 as hidden dimension is acutally way to small. Try the following parameters:
hidden_dim = 128 to 512
layer_dim = 2 to max. 4

I see your output-shape is 1 and you dont use an activation function. Are you trying to predict intergers (like 1 for class "dog", 2 for class "cat")? If so you should switch to one-hot encoding, so that your output shape is equal to the classes you want to predict. And then use softmax as activation for your last layer.


Instruction: In 2020 what is the optimal way to train a model in Pytorch on more than one GPU on one computer?

Input:
What are the best practices for training one neural net on more than one GPU on one machine?
I'm a little confused by the different options from nn.DataParallel vs putting different layers on different GPUs with .to('cuda:0') and .to('cuda:1'). I see in the Pytorch docs the latter method the date was 2017. Is there a standard or does it depend on preference or the type of model?
Method 1
class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10)
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5)

    def forward(self, x):
        x = self.relu(self.net1(x))
        return self.net2(x)

model = ToyModel().to('cuda')
model = nn.DataParallel(model)

Method 2
class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10).to('cuda:0')
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5).to('cuda:1')

    def forward(self, x):
        x = self.relu(self.net1(x.to('cuda:0')))
        return self.net2(x.to('cuda:1'))

I'm not sure there aren't more ways Pytorch provides to train on more than one GPU.
Both of these methods seem to cause my system to freeze depending on what model I use them. In Jupyter the cell stays at a [*] and if I don't restart the kernel the screen freezes and I have to do a hard reset. A few tutorials on multi-gpu cause my system to hang and freeze like this.

Output:
If you cannot fit all the layers of your model on a single GPU, then you can use model parallel (that article describes model parallel on a single machine, with layer0.to('cuda:0') and layer1.to('cuda:1') like you mentioned).
If you can, then you can try distributed data parallel - each worker will hold its own copy of the entire model (all layers), and will work on a small portion of the data in each batch. DDP is recommended instead of DP, even if you only use a single machine.
Do you have some examples that can reproduce the issues you're having?
Have you tried running your code with tiny inputs, and adding print statements to see whether progress is being made?


Instruction: Why is looping through pytorch tensors so slow (compared to Numpy)?

Input:
I've been working with image transformations recently and came to a situation where I have a large array (shape of 100,000 x 3) where each row represents a point in 3D space like:
pnt = [x y z]

All I'm trying to do is iterating through each point and matrix multiplying each point with a matrix called T (shape = 3 X 3).
Test with Numpy:
def transform(pnt_cloud, T):

    i = 0
    for pnt in pnt_cloud:
        xyz_pnt = np.dot(T, pnt)

        if xyz_pnt[0] > 0:
            arr[i] = xyz_pnt[0]

        i += 1

    return arr

Calling the following code and calculating runtime (using %time) gives the output:
Out[190]: CPU times: user 670 ms, sys: 7.91 ms, total: 678 ms
Wall time: 674 ms

Test with Pytorch Tensor:
import torch

tensor_cld = torch.tensor(pnt_cloud)
tensor_T   = torch.tensor(T)

def transform(pnt_cloud, T):
    depth_array = torch.tensor(np.zeros(pnt_cloud.shape[0]))

    i = 0
    for pnt in pnt_cloud:
        xyz_pnt = torch.matmul(T, pnt)

        if xyz_pnt[0] > 0:
            depth_array[i] = xyz_pnt[0]

        i += 1

    return depth_array

Calling the following code and calculating runtime (using %time) gives the output:
Out[199]: CPU times: user 6.15 s, sys: 28.1 ms, total: 6.18 s
Wall time: 6.09 s

NOTE: Doing the same with torch.jit only reduces 2s
I would have thought that PyTorch tensor computations would be much faster due to the way PyTorch breaks its code down in the compiling stage. What am I missing here?
Would there be any faster way to do this other than using Numba?

Output:
Why are you using a for loop??
Why do you compute a 3x3 dot product and only uses the first element of the result??
You can do all the math in a single matmul:
with torch.no_grad():
  depth_array = torch.matmul(pnt_cloud, T[:1, :].T)  # nx3 dot 3x1 -> nx1
  # since you only want non negative results
  depth_array = torch.maximum(depth_array, 0)

Since you want to compare runtime to numpy, you should disable gradient accumulation.


Instruction: How to load checkpoints across different versions of pytorch (1.3.1 and 1.6.x) using ppc64le and x86?

Input:
As I outlined here I am stuck using old versions of pytorch and torchvision due to hardware e.g. using ppc64le IBM architectures.
For this reason, I am having issues when sending and receiving checkpoints between different computers, clusters and my personal mac. I wonder if there is any way to load models in a way to avoid this issue? e.g. perhaps saving models in with a old and new format when using 1.6.x. Of course for the 1.3.1 to 1.6.x is impossible but at leat I was hoping something would work.
Any advice? Of course my ideal solution is that I don't have to worry about it and I can always load and save my checkpoints and everything I usually pickle uniformly across all my hardware.

The first error I got was a zip jit error:
RuntimeError: /home/miranda9/data/f.pt is a zip archive (did you mean to use torch.jit.load()?)

so I used that (and other pickle libraries):
# %%
import torch
from pathlib import Path

def load(path):
    import torch
    import pickle
    import dill

    path = str(path)
    try:
        db = torch.load(path)
        f = db['f']
    except Exception as e:
        db = torch.jit.load(path)
        f = db['f']
        #with open():
        # db = pickle.load(open(path, "r+"))
        # db = dill.load(open(path, "r+"))
        #raise ValueError(f'FAILED: {e}')
    return db, f

p = "~/data/f.pt"
path = Path(p).expanduser()

db, f = load(path)

Din, nb_examples = 1, 5
x = torch.distributions.Normal(loc=0.0, scale=1.0).sample(sample_shape=(nb_examples, Din))

y = f(x)

print(y)
print('Success!\a')

but I get complains of different pytorch versions which I am forced to use:
Traceback (most recent call last):
  File "hal_pg.py", line 27, in <module>
    db, f = load(path)
  File "hal_pg.py", line 16, in load
    db = torch.jit.load(path)
  File "/home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/jit/__init__.py", line 239, in load
    cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
RuntimeError: version_number <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbc (0x7fff7b527b9c in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff1d293c78 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x88 (0x7fff1d2950d8 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x64 (0x7fff1e624664 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: <unknown function> + 0x70e210 (0x7fff7c0ae210 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x28efc4 (0x7fff7bc2efc4 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #26: <unknown function> + 0x25280 (0x7fff84b35280 in /lib64/libc.so.6)
frame #27: __libc_start_main + 0xc4 (0x7fff84b35474 in /lib64/libc.so.6)

any ideas how to make everything consistent across the clusters? I can't even open the pickle files.

maybe this is just impossible with the current pytorch version I am forced to use :(
RuntimeError: version_number <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbc (0x7fff83ba7b9c in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff25993c78 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x88 (0x7fff259950d8 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x64 (0x7fff26d24664 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: <unknown function> + 0x70e210 (0x7fff8472e210 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x28efc4 (0x7fff842aefc4 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #23: <unknown function> + 0x25280 (0x7fff8d335280 in /lib64/libc.so.6)
frame #24: __libc_start_main + 0xc4 (0x7fff8d335474 in /lib64/libc.so.6)

using code:
from pathlib import Path

import torch

path = '/home/miranda9/data/dataset/'
path = Path(path).expanduser() / 'fi_db.pt'
path = str(path)

# db = torch.load(path)
# torch.jit.load(path)
db = torch.jit.load(str(path))

print(db)

related links:

How to load checkpoints across different versions of pytorch (1.3.1 and 1.6.x) using ppc64le and x86?
https://discuss.pytorch.org/t/how-to-load-checkpoints-across-different-versions-of-pytorch-1-3-1-and-1-6-x-using-ppc64le-and-x86/97829
related gitissue: https://github.com/pytorch/pytorch/issues/43766
reddit: https://www.reddit.com/r/pytorch/comments/jvza7v/how_to_load_checkpoints_across_different_versions/

Output:
I believe what the developers intend is passing a flag for saving as a pickle. Just a default behavior change.
For previously checkpointed files reload the zip file saved weights in the newer env(with pytorch>=1.6), and then checkpoint again as a pickle (no need to re-train);
update your code and add flag from next time
Deprecation from ver 1.6 :

We have switched torch.save to use a zip file-based format by default
rather than the old Pickle-based format. torch.load has retained the
ability to load the old format, but use of the new format is
recommended. The new format is:
more friendly for inspection and building tooling for manipulating the
save files fixes a long-standing issue wherein serialization
(getstate, setstate) functions on Modules that depended on
serialized Tensor values were getting the wrong data the same as the
TorchScript serialization format, making serialization more consistent
across PyTorch

Usage is as follows:
m = MyMod()
torch.save(m.state_dict(), 'mymod.pt') # Saves a zipfile to mymod.pt

To use the old format, pass the flag _use_new_zipfile_serialization=False
m = MyMod()
torch.save(m.state_dict(), 'mymod.pt', _use_new_zipfile_serialization=False) # Saves pickle


Instruction: Fine tuning of Bert word embeddings

Input:
I would like to load a pre-trained Bert model and to fine-tune it and particularly the word embeddings of the model using a custom dataset.
The task is to use the word embeddings of chosen words for further analysis.
It is important to mention that the dataset consists of tweets and there are no labels.
Therefore, I used the BertForMaskedLM model.
Is it OK for this task to use the input ids (the tokenized tweets) as the labels?
I have no labels. There are just tweets in randomized order.
From this point, I present the code I wrote:
First, I cleaned the dataset from emojis, non-ASCII characters, etc as described in the following link (2.3 Section):
https://www.kaggle.com/jaskaransingh/bert-fine-tuning-with-pytorch
Second, the code of the fine tuning process:
import torch

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM.from_pretrained('bert-base-uncased')

model.to(device)
model.train()

lr = 1e-2

optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)
max_len = 82
chunk_size = 20
epochs = 20

for epoch in range(epochs):
    epoch_losses = []
    for j, batch in enumerate(pd.read_csv(path + file_name, chunksize=chunk_size)):
        tweets = batch['content_cleaned'].tolist()

        encoded_dict = tokenizer.batch_encode_plus(
                            tweets,                      # Sentence to encode.
                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                            max_length = max_len,           # Pad & truncate all sentences.
                            pad_to_max_length = True,
                            truncation=True,
                            return_attention_mask = True,   # Construct attn. masks.
                            return_tensors = 'pt',     # Return pytorch tensors.
                       )
        input_ids = encoded_dict['input_ids'].to(device)

        # Is it correct? or should I train it in another way?
        loss, _ = model(input_ids, labels=input_ids)
        loss_score = loss.item()

        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)
        optimizer.step()
        optimizer.zero_grad()
model.save_pretrained(path + "Fine_Tuned_BertForMaskedLM")

The loss starts from 50 and reduced until 2.3.

Output:
Since the objective of the masked language model is to predict the masked token, the label and the inputs are the same. So, whatever you have written is correct.
However, I would like to add on the concept of comparing word embeddings. Since, BERT is not a word embeddings model, it is contextual, in the sense, that the same word can have different embeddings in different context. Example: the word 'talk' will have a different embeddings in the sentences "I want to talk" and "I will attend a talk". So, there is no single vector of embeddings for each word. (Which makes BERT different from word2vec or fastText). Masked Language Model (MLM) on a pre-trained BERT is usually performed when you have a small new corpus, and want your BERT model to adapt to it. However, I am not sure on the performance gain that you would get by using MLM and then fine-tuning to a specific task than directly fine-tuning the pre-trained model with task specific corpus on a downstream task.


Instruction: BERT NER Python

Input:
I am using BERT model for Named Entity Recognition task.
I have torch version - 1.2.0+cu9.2
torch vision version - 0.4.0+cu9.2
Nvidia drivers compatible with cuda 9.2
when i am trying to train my model using the command
loss, scores = model(b_input_ids.type(torch.cuda.LongTensor), token_type_ids=None,attention_mask=b_input_mask.to(device), labels=b_labels.type(torch.cuda.LongTensor))

i am getting the error below -
C:/w/1/s/windows/pytorch/aten/src/THC/THCTensorIndex.cu:361: block: [35,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

Can somebody help me with this one?

Output:
A bit of googling provided the following hint with the following suggestions:

This is due to an out of bounds index in the embedding matrix.

If you are seeing this error using an nn.Embedding layer, you might add a print statement which shows the min and max values for each input. Some batches might have an out of bounds index.
Once you find, the erroneous batch you should have a look how it was created so that you can fix this error.

Without seeing your code nobody is going to be able to help more.


Instruction: Pytorch issue: torch.load() does not correctly load a saved model from file after closing and reopening Spyder IDE

Input:
I followed the most basic code procedure for saving and loading neural network model parameters and it works perfectly fine. After training the network, it is saved to a specified file in a specified folder in the package using the standard torch.save(model.state_dict(), file) method; when I need to rerun the program to evaluate instead of train, it is loaded using the standard model.load_state_dict(torch.load(file)) method. However, as soon as I close the Spyder IDE application and reopen the IDE and project, the torch.load does not result in the desired saved model. Note that I am testing the 'correctness' of the model by feeding inputs and checking the outputs. I am not sure if this is a problem with Spyder or with Pytorch, though I'm learning towards Spyder because the loading does work.
In short, the program works if I run it consecutively while Spyder is open, but as soon as Spyder closes and reopens, it ceases to work properly. Does anyone have a clue?

Output:
It might be due to the fact that the state of your program is known while running the IDE but when closing it the state is lost resulting in the inability to know how to load the model (because the IDE doesnt know what model you are using). To solve this, try defining a new model and loading the parameters to it via load like so:
my_model = MyModelClass(parameters).to(device)
checkpoint = torch.load(path)
my_model.load_state_dict(checkpoint)

This way the IDE knows what class is your model and can treat it properly.