Instruction: PyTorch Cuda with anaconda not available

Input:
I'm using anaconda to regulate my environment,
for a project i have to use my GPU for network training.
I use pytorch for my project and i'm trying to get CUDA working.

I installed cudatoolkit, numba, cudnn

still, when i try this command:

torch.cuda.is_available()

I get "False" as output.
This is my environment:

# Name                    Version                   Build  Channel
blas                      1.0                         mkl
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.03.07                    0
cairo                     1.14.12              he6fea26_5    conda-forge
certifi                   2018.8.24                py35_1
cffi                      1.11.5           py35he75722e_1
cloudpickle               0.5.5                    py35_0
cudatoolkit               9.2                           0    anaconda
cudnn                     7.2.1                 cuda9.2_0    anaconda
cycler                    0.10.0                     py_1    conda-forge
cython                    0.28.5           py35hf484d3e_0    anaconda
dask-core                 0.19.2                   py35_0
dbus                      1.13.0               h3a4f0e9_0    conda-forge
decorator                 4.3.0                    py35_0
expat                     2.2.5                hfc679d8_2    conda-forge
ffmpeg                    4.0.2                ha0c5888_1    conda-forge
fontconfig                2.13.1               h65d0f4c_0    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
giflib                    5.1.4                h470a237_1    conda-forge
glib                      2.55.0               h464dc38_2    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
graphite2                 1.3.12               hfc679d8_1    conda-forge
gst-plugins-base          1.12.5               hde13a9d_0    conda-forge
gstreamer                 1.12.5               h61a6719_0    conda-forge
harfbuzz                  1.9.0                h08d66d9_0    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
imageio                   2.4.1                    py35_0
intel-openmp              2019.0                      118
jasper                    1.900.1              hff1ad4c_5    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
kiwisolver                1.0.1            py35h2d50403_2    conda-forge
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15                 h470a237_3    conda-forge
libopenblas               0.3.3                h5a2b251_3
libpng                    1.6.35               ha92aebf_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.9                he6b73bb_2    conda-forge
libuuid                   2.32.1               h470a237_2    conda-forge
libwebp                   0.5.2                         7    conda-forge
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
llvmlite                  0.24.0           py35hdbcaa40_0
matplotlib                3.0.0            py35h0b34cb6_1    conda-forge
mkl                       2019.0                      118
mkl_fft                   1.0.6                    py35_0    conda-forge
mkl_random                1.0.1                    py35_0    conda-forge
ncurses                   6.1                  hf484d3e_0
nettle                    3.3                           0    conda-forge
networkx                  2.1                      py35_0
ninja                     1.8.2            py35h6bb024c_1
numba                     0.39.0           py35h04863e7_0
numpy                     1.15.2           py35h1d66e8a_0
numpy-base                1.15.2           py35h81de0dd_0
olefile                   0.46                     py35_0
openblas                  0.2.20                        8    conda-forge
opencv                    3.4.1            py35h6fd60c2_1
opencv-python             3.4.3.18                  <pip>
openh264                  1.7.0                         0    conda-forge
openssl                   1.0.2p               h14c3975_0
pandas                    0.23.4           py35h04863e7_0
pcre                      8.41                 hfc679d8_3    conda-forge
pillow                    5.2.0            py35heded4f4_0
Pillow                    5.3.0                     <pip>
pip                       10.0.1                   py35_0
pixman                    0.34.0               h470a237_3    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
pycparser                 2.19                     py35_0
pyparsing                 2.2.2                      py_0    conda-forge
pyqt                      5.6.0            py35h8210e8a_7    conda-forge
python                    3.5.6                hc3d631a_0
python-dateutil           2.7.3                      py_0    conda-forge
pytorch                   0.4.1           py35_py27__9.0.176_7.1.2_2    pytorch
pytz                      2018.5                   py35_0
pywavelets                1.0.0            py35hdd07704_0
qt                        5.6.2                hf70d934_9    conda-forge
readline                  7.0                  h7b6447c_5
scikit-image              0.14.0           py35hf484d3e_1
scipy                     1.1.0            py35hfa4b5c9_1
setuptools                40.2.0                   py35_0
sip                       4.18.1           py35hfc679d8_0    conda-forge
six                       1.11.0                   py35_1    conda-forge
sqlite                    3.25.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
toolz                     0.9.0                    py35_0
torchvision               0.1.9            py35h72e4c6f_1    soumith
tornado                   5.1.1            py35h470a237_0    conda-forge
wheel                     0.31.1                   py35_0
x264                      1!152.20180717       h470a237_1    conda-forge
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.2                h8c8a85c_6    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.4                h14c3975_4
zlib                      1.2.11               ha838bed_2

My desktop has a NVIDIA GeForce GTX 970 (so it is cuda available)
Also for some reason, as you can see here:

My graphics card doesn't show, however when using the

lspci -v

command, i can see my graphics card there. Don't know if that has something to do with it. Anyone knows how i can fix this?

Output:
You need to install pytorch "in one go" using https://pytorch.org/get-started/locally/ to construct the anaconda command.
With the standard configuration in anaconda, you get:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

(Please always check on https://pytorch.org/get-started/locally/ whether this command is still up to date.)
You seem to need the right cuda version 10.2 package to be aligned with what pytorch can handle. This is what is meant with @RussellGallop's helpful message.
We can see that installing pytorch and cuda separately is not recommended, and that Anaconda installation is recommended, against your answer:

Anaconda is our recommended package manager since it installs all dependencies.

Uninstall and install better than repair
In case of problems, you better uninstall all covered packages and apply https://pytorch.org/get-started/locally/ to get the command again, instead of trying to fix it with separate installations. Thus, if you want to uninstall, you need to use the exactly same command of the installation, but with "conda uninstall" instead.
For the example above, the uninstall command would be:
conda uninstall pytorch torchvision cudatoolkit=10.2 -c pytorch

This needed uninstall "in one go" again is another hint at the sensitive installation of pytorch, and that separate installation is risky. See How can l uninstall PyTorch?)


Instruction: Error in training PyTorch classifier from the 60 minute blitz in GPU

Input:
I've started to learn pytorch with their official 60 minute blitz tutorial in a jupyter lab (using their .ipynb file, link to the tutorial), and have completed it successfully until the conversion and training of the classifier using the gpu. I think that I have managed to change the device for the net, inputs and labels according to these results:

net=net.to(device)
net.fc1.weight.type()

With output:

'torch.cuda.FloatTensor'

And:

inputs, labels = inputs.to(device), labels.to(device)
inputs.type(),labels.type()

With output:

('torch.cuda.FloatTensor', 'torch.cuda.LongTensor')

After running these cells, I ran the cell for training the model, containing this code:

for epoch in range(2):  # loop over the dataset multiple times

running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

print('Finished Training')

And received this error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-55-fe85c778b0e6> in <module>()
     10
     11         # forward + backward + optimize
---> 12         outputs = net(inputs)
     13         loss = criterion(outputs, labels)
     14         loss.backward()

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

<ipython-input-52-725d44154459> in forward(self, x)
    14
    15     def forward(self, x):
--->16         x=self.conv1(x)
    17         x = self.pool(F.relu(x))
    18         x = self.pool(F.relu(self.conv2(x)))

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self,
input)
    299     def forward(self, input):
    300         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 301                         self.padding, self.dilation, self.groups)
    302
    303

RuntimeError: Expected object of type torch.FloatTensor but found type
torch.cuda.FloatTensor for argument #2 'weight'

Why did I receive this error and how can I fix it?

Output:
You also need to move your inputs and labels to GPU inside the training loop.

for i, data in enumerate(trainloader, 0):
    # get the inputs
    inputs, labels = data

    # move to GPU
    inputs = inputs.to(device)
    labels = labels.to(device)

    ...


Instruction: Pytorch: is there a function similar to torch.argmax which can really keep the dimension of the original data?

Input:
For example,
the code is

input = torch.randn(3, 10)
result = torch.argmax(input, dim=0, keepdim=True)

input is

tensor([[ 1.5742,  0.8183, -2.3005, -1.1650, -0.2451],
       [ 1.0553,  0.6021, -0.4938, -1.5379, -1.2054],
       [-0.1728,  0.8372, -1.9181, -0.9110,  0.2422]])

and result is

tensor([[ 0,  2,  1,  2,  2]])

However, I want a result like this

tensor([[ 1,  0,  0,  0,  0],
        [ 0,  0,  1,  0,  0],
        [ 0,  1,  0,  1,  1]])

Output:
Finally, I solved it. But this solution may not be efficient.
Code as follow,

input = torch.randn(3, 10)
result = torch.argmax(input, dim=0, keepdim=True)
result_0 = result == 0
result_1 = result == 1
result_2 = result == 2
result = torch.cat((result_0, result_1, result_2), 0)


Instruction: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'weight'

Input:
Firstly I have used like 'model.cuda()' to convert model and data to cuda. But it still has such a problem. I debug every layer of the model, and weights for every module has iscuda=True. So does anyone know why there is such a problem?

I have two models, one is resnet50 and another one which contains the first one as backbone.

class FC_Resnet(nn.Module):
    def __init__(self, model, num_classes):
        super(FC_Resnet, self).__init__()

        # feature encoding
        self.features = nn.Sequential(
            model.conv1,
            model.bn1,
            model.relu,
            model.maxpool,
            model.layer1,
            model.layer2,
            model.layer3,
            model.layer4)

        # classifier
        num_features = model.layer4[1].conv1.in_channels
        self.classifier = nn.Sequential(
            nn.Conv2d(num_features, num_classes, kernel_size=1, bias=True))

    def forward(self, x):
        # children=self.features.children()
        # for child in children:
        #     if child.weight is not None:
        #         print(child.weight.device)
        x = self.features(x)
        x = self.classifier(x)
        return x

def fc_resnet50(num_classes=20, pre_trained=True):
    model = FC_Resnet(models.resnet50(pre_trained), num_classes)

    return model

And another one:

class PeakResponseMapping(nn.Sequential):
    def __init__(self, *args, **kargs):
        super(PeakResponseMapping, self).__init__(*args)
        ...

    def forward(self, input, class_threshold=0, peak_threshold=30, retrieval_cfg=None):
        assert input.dim() == 4
        if self.inferencing:
            input.requires_grad_()

        class_response_maps = super(PeakResponseMapping, self).forward(input)

        return class_response_maps

And the main is very simple:

def main():
    dataset = VOC(img_transform=image_transform())
    dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True)

    model = peak_response_mapping(fc_resnet50(), win_size=3, sub_pixel_locating_factor=8, enable_peak_stimulation=True)
    model=model.cuda()

    for step, (b_x, b_y) in enumerate(dataloader):
        b_x.cuda()
        b_y.cuda()

        result = model.forward(b_x)

Output:
Somewhere down in the stack trace, Torch is expecting a CPU tensor (torch.FloatTensor) but is getting a GPU / CUDA tensor (torch.cuda.FloatTensor).

Given a tensor tensor:

tensor.to('cpu') returns the CPU version of the tensor
tensor.to('cuda') returns the CUDA version of the tensor

To write hardware-agnostic code:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Then you can do:

tensor.to(device)

For the OP, this becomes:

result = model.forward(b_x.to(device))


Instruction: How do you convert a .onnx to tflite?

Input:
I've exported my model to ONNX via:

# Export the model
torch_out = torch.onnx._export(learn.model,             # model being run
                           x,                       # model input (or a tuple for multiple inputs)
                          EXPORT_PATH + "mnist.onnx", # where to save the model (can be a file or file-like object)
                           export_params=True)      # store the trained parameter weights inside the model file

And now I am trying to convert the model to a Tensorflow Lite file so that I can do inference on Android. Unfortunately, PyTorch/Caffe2 support is fairly lacking or too complex for Android but Tensorflow appears much simpler.

The documentation for ONNX to Tflite is pretty light on this.

I've tried exporting to a Tensorflow GraphDef proto via:

tf_rep.export_graph(EXPORT_PATH + 'mnist-test/mnist-tf-export.pb')

And then running toco:

toco \
--graph_def_file=mnist-tf-export.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--inference_type=FLOAT \
--input_type=FLOAT \
--input_arrays=0 \
--output_arrays=add_10 \
--input_shapes=1,3,28,28 \
--output_file=mnist.tflite`

When I do though I get the following error:

File "anaconda3/lib/python3.6/site-packages/tensorflow/lite/python/convert.py", line 172, in toco_convert_protos
    "TOCO failed. See console for info.\n%s\n%s\n" % (stdout, stderr))
tensorflow.lite.python.convert.ConverterError: TOCO failed. See console for info.
2018-11-06 16:28:33.864889: I tensorflow/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: PyFunc
2018-11-06 16:28:33.874130: F tensorflow/lite/toco/import_tensorflow.cc:114] Check failed: attr.value_case() == AttrValue::kType (1 vs. 6)

Further, even when I run the command I don't know what to specify for the input_arrays or output_arrays since the model was originally built in PyTorch.

Has anyone successfully converted their ONNX model to TFlite?

Here's the ONNX file I'm trying to convert: https://drive.google.com/file/d/1sM4RpeBVqPNw1WeCROpKLdzbSJPWSK79/view?usp=sharing

Extra info

Python 3.6.6 :: Anaconda custom (64-bit)
onnx.version = '1.3.0'
tf.version = '1.13.0-dev20181106'
torch.version = '1.0.0.dev20181029'

Output:
I think the ONNX file i.e. model.onnx that you have given is corrupted I don't know what is the issue but it is not doing any inference on ONNX runtime.

Now you can run PyTorch Models directly on mobile phones. check out PyTorch Mobile's documentation here

This answer is for TensorFlow version 1,
For TensorFlow version 2 or higher click link

The best way to convert the model from protobuf freezeGraph to TFlite is to use the official TensorFlow lite converter documentation
According to TensorFlow Docs, TocoConverter has been deprecated

This class (tf.compat.v1.lite.TocoConverter) has been deprecated. Please use lite.TFLiteConverter instead.

Convert from PyTorch to ONNX model
The best practice to convert the model from Pytorch to Onnx is that you should add the following parameters to specify the names of the input and output layer of your model in torch.onnx.export() function

# Export the model from PyTorch to ONNX
torch_out = torch.onnx._export(model,             # model being run
                                x,          # model input (or a tuple for multiple inputs)
                                EXPORT_PATH + "mnist.onnx",      # where to save the model (can be a file or file-like object)
                                export_params=True,       # store the trained parameter weights inside the model file
                                input_names=['main_input'],     # specify the name of input layer in onnx model
                                output_names=['main_output'])     # specify the name of input layer in onnx model

So in your case:
Now export this model to TensorFlow protobuf FreezeGraph using onnx-tf

Please note that this method is only working when tensorflow_version < 2

Convert from ONNX to TensorFlow freezGraph
To convert the model please install onnx-tf version 1.5.0 from the below command
pip install  onnx-tf==1.5.0

Now to convert .onnx model to TensorFlow freeze graph run this below command in shell
onnx-tf convert -i "mnist.onnx" -o  "mnist.pb"

Convert from TensorFlow FreezeGraph .pb to TF
Now to convert this model from .pb file to tflite model use this code
import tensorflow as tf
# make a converter object from the saved tensorflow file
converter = tf.lite.TFLiteConverter.from_frozen_graph('mnist.pb', #TensorFlow freezegraph .pb model file
                                                      input_arrays=['main_input'], # name of input arrays as defined in torch.onnx.export function before.
                                                      output_arrays=['main_output']  # name of output arrays defined in torch.onnx.export function before.
                                                      )
# tell converter which type of optimization techniques to use
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# to view the best option for optimization read documentation of tflite about optimization
# go to this link https://www.tensorflow.org/lite/guide/get_started#4_optimize_your_model_optional

# convert the model
tf_lite_model = converter.convert()
# save the converted model
open('mnist.tflite', 'wb').write(tf_lite_model)

To choose which option is best for optimization for your model use case see this official guide about TensorFlow lite optimization
https://www.tensorflow.org/lite/guide/get_started#4_optimize_your_model_optional

Note: You can try my Jupyter Notebook Convert ONNX model to Tensorflow Lite on Google Colaboratory link


Instruction: Making custom non-trivial loss function in pytorch

Input:
I'm just started with pytorch and trying to understand how to deal with custom loss functions, especially with some non trivial ones.

Problem 1. I'd like to stimulate my nn to maximize true positive rate and at the same time minimize
false discovery rate. For example increase total score on +2 for true positive, and decrease on -5 for false positive.

def tp_fp_loss(yhat, y):
    total_score = 0
    for i in range(y.size()):
        if is_tp(yhat[i],y[i]):
            total_score += 2
        if is_fp(yhat[i],y[i]):
            total_score -= 5
    return -total_score

Problem 2. In case when y is a list of positive and negative rewards (y = [10,-5, -40, 23, 11, -7]),
stimulate nn to maximize sum of rewards.

def max_reward_loss(yhat,y):
    r = torch.autograd.Variable(torch.Tensor(y[yhat >= .5]), requires_grad=True).sum()
    return -r

Maybe I'm not completely understand some autograd mechanics, functions which I implemented correctly calculate loss but
learning with them doesnt work :( What I'm doing wrong? Can anybody help me with some working solution of any of that problems?

Output:
@Shai already summed it up: Your loss function is not differentiable.

One way to think about it is that your loss function should be plottable, and the "downhill" slope should "roll" toward the desired model output.  In order to plot your loss function, fix y_true=1 then plot [loss(y_pred) for y_pred in np.linspace(0, 1, 101)] where loss is your loss function, and make sure your plotted loss function has the slope as desired.  In your case, it sounds like you want to weight the the loss more strongly when it is on the wrong side of the threshold.  As long as you can plot it, and the slope is always downhill toward your target value (no flat spots or uphill slopes on the way from a valid prediction to the target value), your model should learn from it.

Also note that if you're just trying to take into account some business objective which prioritizes precision over recall, you could accomplish this by training to convergence with cross entropy or some well-known loss function, and then by tuning your model threshold based on your use case.  A higher threshold would normally prioritize precision, and a lower threshold would normally prioritize recall.  After you've trained, you can then evaluate your model at a variety of thresholds and choose the most appropriate.


Instruction: CUDNN_STATUS_MAPPING_ERROR when training with pose2body

Input:
I'm trying to train https://github.com/NVIDIA/vid2vid. I'm...

...executing with pretty much the vanilla parametrization shown in the readme, I had to change the number of GPUs though and increased the number of threads for reading the dataset. Command:

python train.py \
--name pose2body_256p  \
--dataroot datasets/pose  \
--dataset_mode pose  \
--input_nc 6  \
--num_D 2  \
--resize_or_crop ScaleHeight_and_scaledCrop  \
--loadSize 384  \
--fineSize 256  \
--gpu_ids 0,1  \
--batchSize 1  \
--max_frames_per_gpu 3  \
--no_first_img  \
--n_frames_total 12  \
--max_t_step 4  \
--nThreads 6
...training on the supplied example datasets.
...running a docker container built with the scripts in vid2vid/docker, e. g. with CUDA 9.0 and CUDNN 7.
...using two NVIDIA V100 GPUs.

Whenever I start training the script crashes after a couple of minutes with the message RuntimeError: CUDNN_STATUS_MAPPING_ERROR. Full error message:

Traceback (most recent call last):
  File "train.py", line 329, in <module>
    train()
  File "train.py", line 104, in train
    fake_B, fake_B_raw, flow, weight, real_A, real_Bp, fake_B_last = modelG(input_A, input_B, inst_A, fake_B_last)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/vid2vid/models/vid2vid_model_G.py", line 130, in forward
    fake_B, fake_B_raw, flow, weight = self.generate_frame_train(netG, real_A_all, fake_B_prev, start_gpu, is_first_frame)
  File "/vid2vid/models/vid2vid_model_G.py", line 175, in generate_frame_train
    fake_B_feat, flow_feat, fake_B_fg_feat, use_raw_only)
  File "/vid2vid/models/networks.py", line 171, in forward
    downsample = self.model_down_seg(input) + self.model_down_img(img_prev)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR

From reading the issues in the vid2vid using two V100 should work with this setup. The error also occurs if CUDA 8/CUDNN 6 are used. I checked the flags but haven't found any indication of further necessary changes to the arguments supplied to train.py.

Any ideas on how to solve (or work around) this?

Output: In case anybody deals the same issue: Training on P100 cards worked. Seems like the V100 architecture clashes with version of pytorch used in the supplied Dockerfile at some point. Not quite a solution, but a workaround.


Instruction: Trouble using openCV to load a net from ONNX (python/pytorch)

Input:
I'm trying to load a trained .onnx model (from a neural-style-transfer algorithm) into cv2. I've seen that there is a

cv.dnn.readNetFromONNX()

function, but there is no such function in cv2. I can't seem to import or load opencv as cv, and as such can't seem to load my model in cv2. Does anyone know a solution?

I've basically trained a model with
https://github.com/pytorch/examples/blob/master/fast_neural_style/neural_style/neural_style.py#L122-L150
this script, and made an export of an onnx model by adding

torch.onnx.export(style_model, dummy_input, "chipsoft_mod.onnx", verbose=True)

Now I want to run the trained model trough the cv2 reader, but I fail spectacularly.

Output:
Update your opencv to a newer version. It should help.

pip install opencv-python==4.1.0.25


Instruction: What does flatten_parameters() do?

Input:
I saw many Pytorch examples using flatten_parameters in the forward function of the RNN

self.rnn.flatten_parameters()

I saw this RNNBase and it is written that it

  Resets parameter data pointer so that they can use faster code paths

What does that mean?

Output:
It may not be a full answer to your question. But, if you give a look at the flatten_parameters's  source code , you will notice that it calls _cudnn_rnn_flatten_weight in

...
NoGradGuard no_grad;
torch::_cudnn_rnn_flatten_weight(...)
...

is the function that does the job. You will find that what it actually does is copying the model's weights into a vector<Tensor> (check the params_arr declaration) in:

  // Slice off views into weight_buf
  std::vector<Tensor> params_arr;
  size_t params_stride0;
  std::tie(params_arr, params_stride0) = get_parameters(handle, rnn, rnn_desc, x_desc, w_desc, weight_buf);

  MatrixRef<Tensor> weight{weight_arr, static_cast<size_t>(weight_stride0)},
                    params{params_arr, params_stride0};

And the weights copying in

  // Copy weights
  _copyParams(weight, params);

Also note that they update (or Reset as they explicitly say in docs) the original pointers  of weights with the new pointers of params by doing an in-place operation .set_ (_ is their notation for the in-place operations) in orig_param.set_(new_param.view_as(orig_param));

  // Update the storage
  for (size_t i = 0; i < weight.size(0); i++) {
    for (auto orig_param_it = weight[i].begin(), new_param_it = params[i].begin();
         orig_param_it != weight[i].end() && new_param_it != params[i].end();
         orig_param_it++, new_param_it++) {
      auto orig_param = *orig_param_it, new_param = *new_param_it;
      orig_param.set_(new_param.view_as(orig_param));
    }
  }

And according to n2798 (draft of C++0x)

©ISO/IECN3092

  23.3.6  Class template vector

  A vector is a sequence container that supports random access iterators. In addition, it supports (amortized)constant time insert and erase operations at the end; insert and erase in the middle take linear time. Storage management is handled automatically, though hints can be given to improve efficiency. The elements of a vector are stored contiguously, meaning that if v is a vector <T, Allocator> where T is some type other than bool, then it obeys the identity&v[n] == &v[0] + n for all 0 <= n < v.size().

In some situations

  UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters().

They explicitly advise people in code warnings to have a contiguous chunk of memory.


Instruction: Error in Google Colaboratory - AttributeError: module 'PIL.Image' has no attribute 'register_decoder'

Input:
I am running this code on Google Colaboratory and I am getting error of register decoder

image_data = dset.ImageFolder(root="drive/SemanticDataset/train/", transform = transforms.Compose([
                                        transforms.Scale(size=img_size),
                                        transforms.CenterCrop(size=(img_size,img_size*2)),
                                        transforms.ToTensor(),
                                        ]))
enter code herelabel_data = dset.ImageFolder(root="drive/SemanticDataset/label/", transform = transforms.Compose([
                                        transforms.Scale(size=img_size),
                                        transforms.CenterCrop(size=(img_size,img_size*2)),
                                        transforms.ToTensor(),
                                        ]))

image_batch = data.DataLoader(image_data, batch_size=batch_size, shuffle=False, num_workers=2)
label_batch = data.DataLoader(label_data, batch_size=batch_size, shuffle=False, num_workers=2)
for i in range(epoch):
   for _, (image, label) in enumerate(zip(image_batch, label_batch)):
       optimizer.zero_grad()

       x = Variable(image, requires_grad=True).cuda()
       y = Variable(label).cuda()

       out = model.forward(x)
       loss = loss_func(out, y)

       loss.backward()
       optimizer.step()

       if _ % 100 == 0:
           print("Epoch: "+i+"| Loss: " , loss)

here is the screenshot of error

Output:
First, check the version of pillow you have by using:

import PIL
print(PIL.PILLOW_VERSION)

and make sure you have the newest version, the one I am using right now is 5.3.0
If you have like 4.0.0, install a new version by using:
!pip install Pillow==5.3.0 in the Colab environment.

Second, restart your Google colab environment, and check the version again, it should be updated.

I had the same problem, and I spent some time trying to solve it.

Note: I was using PyTorch 0.4.

I hope this will solve your problem.


Instruction: Pytorch Chatbot Tutorial problem: How can I solve List Index Out of Range

Input:
I’m new to pytorch and have been following the many tutorials available.
But, When I did The CHATBOT TUTORIAL is not work.

Like the figure below

What should I do and what is causing this?

Output:
A little late, but I hit this same error at this line before on this same tutorial.  I am also running on 3.6 on Windows, had no issues loading PyTorch or running anything with CUDA.  The issue for me was in the data.

It was because there was a blank line somewhere in the source data, so when it went to split the words in the list for that function, it threw this error - there were no words to split.


Instruction: Python - importing fastai results in SyntaxError

Input:
I am trying to setup an ML model using fastai and have to do the following imports:

import fastai.models
import fastai.nlp
import fastai.dataset

However, it gives me the following error by the fastai imports.

Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastai.nlp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/fastai/nlp.py", line 172
    if os.path.isdir(path): paths=glob(f'{path}/*.*')
                                                   ^
SyntaxError: invalid syntax

Apparently, the character f in glob(f'{path}/*.*') is causing the error. I fixed the error by removing f, but it seems that there are lots of these errors in the fastai library.

My current thought is that I am using an incorrect python version. Could anyone give me some pointer?

Output:
Strings in the shape of:

f'{path}/*.*'

are called f-strings and were introduced in Python3.6.
That's why you get the SyntaxError - for versions lower than Python3.6 a SyntaxError will be raised, as this syntax just doesn't exist in lower versions.

So obviously fast-ai is programmed for Python3.6 or higher.

When you take a look at the installation issues (you have to scroll down a bit),
you can see under Is My System Supported? the first point:

  Python: You need to have python 3.6 or higher

So I'm afraid updating your python is the easiest way to solve the problem!

If you like to learn more about f-strings you can take a look here: https://www.python.org/dev/peps/pep-0498/


Instruction: pytorch RuntimeError: CUDA error: device-side assert triggered

Input:
I've a notebook on google colab that fails with following error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     93         exception = e
---> 94         raise e
     95     finally: cb_handler.on_train_end(exception)

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     83                 xb, yb = cb_handler.on_batch_begin(xb, yb)
---> 84                 loss = loss_batch(model, xb, yb, loss_func, opt, cb_handler)
     85                 if cb_handler.on_batch_end(loss): break

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in loss_batch(model, xb, yb, loss_func, opt, cb_handler)
     24     if opt is not None:
---> 25         loss = cb_handler.on_backward_begin(loss)
     26         loss.backward()

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_backward_begin(self, loss)
    223         for cb in self.callbacks:
--> 224             a = cb.on_backward_begin(**self.state_dict)
    225             if a is not None: self.state_dict['last_loss'] = a

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in on_backward_begin(self, smooth_loss, **kwargs)
    266         if self.pbar is not None and hasattr(self.pbar,'child'):
--> 267             self.pbar.child.comment = f'{smooth_loss:.4f}'
    268

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __format__(self, format_spec)
    377         if self.dim() == 0:
--> 378             return self.item().__format__(format_spec)
    379         return object.__format__(self, format_spec)

RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-33-dd390b1c8108> in <module>()
----> 1 lr_find(learn)
      2 learn.recorder.plot()

/usr/local/lib/python3.6/dist-packages/fastai/train.py in lr_find(learn, start_lr, end_lr, num_it, stop_div, **kwargs)
     26     cb = LRFinder(learn, start_lr, end_lr, num_it, stop_div)
     27     a = int(np.ceil(num_it/len(learn.data.train_dl)))
---> 28     learn.fit(a, start_lr, callbacks=[cb], **kwargs)
     29
     30 def to_fp16(learn:Learner, loss_scale:float=512., flat_master:bool=False)->Learner:

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
    160         callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
    161         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 162             callbacks=self.callbacks+callbacks)
    163
    164     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     93         exception = e
     94         raise e
---> 95     finally: cb_handler.on_train_end(exception)
     96
     97 loss_func_name2activ = {'cross_entropy_loss': partial(F.softmax, dim=1), 'nll_loss': torch.exp, 'poisson_nll_loss': torch.exp,

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_train_end(self, exception)
    254     def on_train_end(self, exception:Union[bool,Exception])->None:
    255         "Handle end of training, `exception` is an `Exception` or False if no exceptions during training."
--> 256         self('train_end', exception=exception)
    257
    258 class AverageMetric(Callback):

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in __call__(self, cb_name, call_mets, **kwargs)
    185         "Call through to all of the `CallbakHandler` functions."
    186         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
--> 187         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    188
    189     def on_train_begin(self, epochs:int, pbar:PBar, metrics:MetricFuncList)->None:

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in <listcomp>(.0)
    185         "Call through to all of the `CallbakHandler` functions."
    186         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
--> 187         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    188
    189     def on_train_begin(self, epochs:int, pbar:PBar, metrics:MetricFuncList)->None:

/usr/local/lib/python3.6/dist-packages/fastai/callbacks/lr_finder.py in on_train_end(self, **kwargs)
     45         # restore the valid_dl we turned of on `__init__`
     46         self.data.valid_dl = self.valid_dl
---> 47         self.learn.load('tmp')
     48         if hasattr(self.learn.model, 'reset'): self.learn.model.reset()
     49         print('LR Finder complete, type {learner_name}.recorder.plot() to see the graph.')

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in load(self, name, device)
    202         "Load model `name` from `self.model_dir` using `device`, defaulting to `self.data.device`."
    203         if device is None: device = self.data.device
--> 204         self.model.load_state_dict(torch.load(self.path/self.model_dir/f'{name}.pth', map_location=device))
    205         return self
    206

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in load(f, map_location, pickle_module)
    356         f = open(f, 'rb')
    357     try:
--> 358         return _load(f, map_location, pickle_module)
    359     finally:
    360         if new_fd:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    527     unpickler = pickle_module.Unpickler(f)
    528     unpickler.persistent_load = persistent_load
--> 529     result = unpickler.load()
    530
    531     deserialized_storage_keys = pickle_module.load(f)

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in persistent_load(saved_id)
    493             if root_key not in deserialized_objects:
    494                 deserialized_objects[root_key] = restore_location(
--> 495                     data_type(size), location)
    496             storage = deserialized_objects[root_key]
    497             if view_metadata is not None:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in restore_location(storage, location)
    376     elif isinstance(map_location, torch.device):
    377         def restore_location(storage, location):
--> 378             return default_restore_location(storage, str(map_location))
    379     else:
    380         def restore_location(storage, location):

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in default_restore_location(storage, location)
    102 def default_restore_location(storage, location):
    103     for _, _, fn in _package_registry:
--> 104         result = fn(storage, location)
    105         if result is not None:
    106             return result

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _cuda_deserialize(obj, location)
     84                                'to an existing device.'.format(
     85                                    device, torch.cuda.device_count()))
---> 86         return obj.cuda(device)
     87
     88

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in _cuda(self, device, non_blocking, **kwargs)
     74         else:
     75             new_type = getattr(torch.cuda, self.__class__.__name__)
---> 76             return new_type(self.size()).copy_(self, non_blocking)
     77
     78

RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorCopy.cpp:20

There is no information about the real cause, I tried to get the stack trace by forcing cuda to run on one gpu (as suggested here) using a cell like this

!export CUDA_LAUNCH_BLOCKING=1

But this does not seem to work, still having the same error with.

Is there another way that works with Google Colab?

Output: Be sure that your targets values starts from zero to number of classes - 1. Ex: you have 100 classification class so your target should be from 0 to 99


Instruction: When using Pytorch-GPU in Anaconda, is it not necessary to install CUDA?

Input:
I found that after installing Pytorch 0.4 GPU version in Anaconda, you don't need to install CUDA locally to call gpu acceleration. When running code, the GPU core can be used at more than 90%.

Edit:I used it in Windows 10. Don't know if it works in Linux.

Output:
@talonmies

Thanks for your url. It seems that pytorch don't need cuda in Windows, since its dependencies are cffi, mkl, numpy, and python.

I entered this command conda search -c pytorch pytorch=0.4.0 --info in Anaconda Prompt and it says

Loading channels: done
pytorch 0.4.0 py35_cuda80_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py35_cuda80_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py35_cuda80_cudnn7he774522_1
build number: 1
size        : 528.5 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py35_cuda80_cudnn7he774522_1.tar.bz2
md5         : 7db3971bb054079d7c7ff84b6286c58e
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.5,<3.6.0a0

pytorch 0.4.0 py35_cuda90_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py35_cuda90_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py35_cuda90_cudnn7he774522_1
build number: 1
size        : 578.5 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py35_cuda90_cudnn7he774522_1.tar.bz2
md5         : 8200c9841f9cad6f2e605015812aa3f2
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.5,<3.6.0a0

pytorch 0.4.0 py35_cuda91_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py35_cuda91_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py35_cuda91_cudnn7he774522_1
build number: 1
size        : 546.1 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py35_cuda91_cudnn7he774522_1.tar.bz2
md5         : 79d99a825f66b55b1aa6f04d22d68aac
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.5,<3.6.0a0

pytorch 0.4.0 py36_cuda80_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py36_cuda80_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py36_cuda80_cudnn7he774522_1
build number: 1
size        : 529.2 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py36_cuda80_cudnn7he774522_1.tar.bz2
md5         : 27d20c9869fb57ffe0d6d014cf348855
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.6,<3.7.0a0

pytorch 0.4.0 py36_cuda90_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py36_cuda90_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py36_cuda90_cudnn7he774522_1
build number: 1
size        : 577.6 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py36_cuda90_cudnn7he774522_1.tar.bz2
md5         : 138dcca8eeff1d58a8fd9b1febf702f6
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.6,<3.7.0a0

pytorch 0.4.0 py36_cuda91_cudnn7he774522_1
------------------------------------------
file name   : pytorch-0.4.0-py36_cuda91_cudnn7he774522_1.tar.bz2
name        : pytorch
version     : 0.4.0
build string: py36_cuda91_cudnn7he774522_1
build number: 1
size        : 546.4 MB
arch        : x86_64
constrains  : ()
platform    : Platform.win
license     : BSD 3-Clause
subdir      : win-64
url         : https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py36_cuda91_cudnn7he774522_1.tar.bz2
md5         : 326265665000de6f7501160b10b089c8
dependencies:
  - cffi
  - mkl >=2018
  - numpy >=1.11
  - python >=3.6,<3.7.0a0


Instruction: Pytorch Exception in Thread: ValueError: signal number 32 out of range

Input:
I'm getting this error:

Exception in Thread: ValueError: signal number 32 out of range

The specific tutorial that raises an issue for me is the training a classifier (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html), the specific line is:

dataiter = iter(trainloader)

and the full error traceback is:

Exception in thread Thread-5:
Traceback (most recent call last):
File "/home/chenchen/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/chenchen/anaconda3/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/chenchen/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
File "/home/chenchen/anaconda3/lib/python3.6/signal.py", line 60, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

My operation system is Ubuntu 18.10 and my python env is Anaconda3 for python 3.6. I installed pytorch from the latest source. My cuda version is 10.0.

Output:
I have faced a similar issue and it got resolved when I set:

num_workers=0


Instruction: Pytorch speed comparison - GPU slower than CPU

Input:
I was trying to find out if GPU tensor operations are actually faster than CPU ones. So, I wrote this particular code below to implement a simple 2D addition of CPU tensors and GPU cuda tensors successively to see the speed difference:

import torch
import time

###CPU
start_time = time.time()
a = torch.ones(4,4)
for _ in range(1000000):
    a += a
elapsed_time = time.time() - start_time

print('CPU time = ',elapsed_time)

###GPU
start_time = time.time()
b = torch.ones(4,4).cuda()
for _ in range(1000000):
    b += b
elapsed_time = time.time() - start_time

print('GPU time = ',elapsed_time)

To my surprise, the CPU time was 0.93 sec and the GPU time was as high as 63 seconds. Am I doing the cuda tensor operation properly or is the concept of cuda tensors works faster only in very highly complex operations, like in neural networks?

Note: My GPU is NVIDIA 940MX and torch.cuda.is_available() call returns True.

Output:
GPU acceleration works by heavy parallelization of computation. On a GPU you have a huge amount of cores, each of them is not very powerful, but the huge amount of cores here matters.

Frameworks like PyTorch do their to make it possible to compute as much as possible in parallel. In general matrix operations are very well suited for parallelization, but still it isn't always possible to parallelize computation!

In your example you have a loop:

b = torch.ones(4,4).cuda()
for _ in range(1000000):
    b += b

You have 1000000 operations, but due to the structure of the code it impossible to parallelize much of these computations. If you think about it, to compute the next b you need to know the value of the previous (or current) b.

So you have 1000000 operations, but each of these has to be computed one after another. Possible parallelization is limited to the size of your tensor. This size though is not very large in your example:

torch.ones(4,4)

So you only can parallelize 16 operations (additions) per iteration.
As the CPU has few, but much more powerful cores, it is just much faster for the given example!

But things change if you change the size of the tensor, then PyTorch is able to parallelize much more of the overall computation. I changed the iterations to 1000 (because I did not want to wait so long :), but you can put in any value you like, the relation between CPU and GPU should stay the same.

Here are the results for different tensor sizes:

#torch.ones(4,4)       - the size you used
CPU time =  0.00926661491394043
GPU time =  0.0431208610534668

#torch.ones(40,40)     - CPU gets slower, but still faster than GPU
CPU time =  0.014729976654052734
GPU time =  0.04474186897277832

#torch.ones(400,400)   - CPU now much slower than GPU
CPU time =  0.9702610969543457
GPU time =  0.04415607452392578

#torch.ones(4000,4000) - GPU much faster then CPU
CPU time =  38.088677167892456
GPU time =  0.044649362564086914

So as you see, where it is possible to parallelize stuff (here the addition of the tensor elements), GPU becomes very powerful. GPU time is not changing at all for the given calculations, the GPU can handle much more! (as long as it doesn't run out of memory :)


Instruction: pytorch delete model from gpu

Input:
I want to make a cross validation in my project based on Pytorch.
And I didn't find any method that pytorch provided to delete the current model and empty the memory of GPU. Could you tell that how can I do it?

Output:
Freeing memory in PyTorch works as it does with the normal Python garbage collector. This means once all references to an Python-Object are gone it will be deleted.

You can delete references by using the del operator:

del model

You have to make sure though that there is no reference to the respective object left, otherwise the memory won't be freed.

So once you've deleted all references of your model, it should be deleted and the memory freed.

If you want to learn more about memory management you can take a look here:
https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management


Instruction: Issues loading from Drive with pytorch' datasets.DatasetFolder

Input:
The loading works great using jupyter and local files, but when I adapted to Colab, fetching data from a Drive folder, datasets.DatasetFolder always loads 9500 odd datapoints, never the full 10 000. Anyone had similar issues?

train_data = datasets.DatasetFolder('/content/drive/My Drive/4 - kaggle/data', np.load, list(('npy'))  )

print(train_data.__len__)

Returns

<bound method DatasetFolder.__len__ of Dataset DatasetFolder
    Number of datapoints: 9554
    Root Location: /content/drive/My Drive/4 - kaggle/data
    Transforms (if any): None
    Target Transforms (if any): None>

Where I would get the full 10 000 elements usually.

Output: Loading lots of files from a single folder in Drive is likely to be slow and error-prone. You'll probably end up much happier if you either stage the data on GCS or upload an archive (.zip or .tar.gz) to Drive and copy that one file to your colab VM, unarchive it there, and then run your code over the local data.


Instruction: Theoretical underpinning behind Hardmax operator

Input:
In the tensor flow Github repository, in the file attentionwrapper.py, hardmax operator has been defined. On the docs, it has been mentioned tf.contrib.seq2seq.hardmax

I want to know what's the theoretical underpinning behind providing this functionality for hardmax operator. Prima facie google searches for past few weeks haven't led me to concrete understanding of the concept.

If softmax is differentiable (soft), why would hardmax be ever used? If it can't be used in back propagation (due to non-differentiability required in gradient calculation), where else can it be used?
Reinforcement learning literature talks about Soft vs Hard attention. However I couldn't see concrete examples nor explanations of where the tf.contrib.seq2seq.hardmax can be actually used in some RL model.
By the looks of it, since it is mentioned in seq2seq, it should be obviously having some application in Natural Language Processing. But exactly where? There are tonnes of NLP tasks. Couldn't find any direct task SOTA algorithm which uses hardmax.

Output: Hardmax is used when you have no choice but to make a decision nonprobabalistically. For example, when you are using a model to generate a neural architecture as in neural module networks, you have to make a discrete choice. To make this trainable (since this would be non-differentiable as you state), you can use REINFORCE (an algorithm in RL) to train via policy gradient and estimate this loss contribution via Monte Carlo sampling. Neural module networks are an NLP construct and depend on seq2seq. I'm sure there are many examples, but this is one that immediately came to mind.


Instruction: PyTorch element-wise product of vectors / matrices / tensors

Input:
In PyTorch, how do I get the element-wise product of two vectors / matrices / tensors?

For googlers, this is product is also known as:

Hadamard product
Schur product
Entrywise product

Output:
Given two tensors A and B you can use either:

A * B
torch.mul(A, B)
A.mul(B)

Note: for matrix multiplication, you want to use A @ B which is equivalent to torch.matmul().


Instruction: PyTorch mapping operators to functions

Input:
What are all the PyTorch operators, and what are their function equivalents?

Eg, is a @ b equivalent to a.mm(b) or a.matmul(b)?

I'm after a canonical listing of operator -> function mappings.

I'd be happy to be given a PyTorch documentation link as an answer - my googlefu couldn't track it down.

Output:
The Python documentation table Mapping Operators to Functions provides canonical mappings from:

operator -> __function__()

Eg:

Matrix Multiplication        a @ b        matmul(a, b)

Elsewhere on the page, you will see the __matmul__ name as an alternate to matmul.

The definitions of the PyTorch __functions__ are found either in:

The torch.Tensor module documentation
python_variable_methods.cpp

You can look up the documentation for the named functions at:

https://pytorch.org/docs/stable/torch.html?#torch.<FUNCTION-NAME>


Instruction: Get the data type of a PyTorch tensor

Input:
I understand that PyTorch tensors are homogenous, ie, each of the elements are of the same type.

How do I find out the type of the elements in a PyTorch tensor?

Output:
There are three kinds of things:

dtype                   || CPU tensor               || GPU tensor

torch.float32              torch.FloatTensor           torch.cuda.FloatTensor

The first one you get with print(t.dtype) if t is your tensor, else you use t.type() for the other two.


Instruction: Check if PyTorch tensors are equal within epsilon

Input:
How do I check if two PyTorch tensors are semantically equal?

Given floating point errors, I want to know if the the elements differ only by a small epsilon value.

Output:
At the time of writing, this is a undocumented function in the latest stable release (0.4.1), but the documentation is in the master (unstable) branch.

torch.allclose() will return a boolean indicating whether all element-wise differences are equal allowing for a margin of error.

Additionally, there's the undocumented isclose():

>>> torch.isclose(torch.Tensor([1]), torch.Tensor([1.00000001]))
tensor([1], dtype=torch.uint8)


Instruction: How does pytorch's parallel method and distributed method work?

Input:
I'm not an expert in distributed system and CUDA. But there is one really interesting feature that PyTorch support which is nn.DataParallel and nn.DistributedDataParallel. How are they actually implemented? How do they separate common embeddings and synchronize data?
Here is a basic example of DataParallel.
import torch.nn as nn
from torch.autograd.variable import Variable
import numpy as np

class Model(nn.Module):
    def __init__(self):
        super().__init__(
            embedding=nn.Embedding(1000, 10),
            rnn=nn.Linear(10, 10),
        )

    def forward(self, x):
        x = self.embedding(x)
        x = self.rnn(x)
        return x

model = nn.DataParallel(Model())
model.forward(Variable.from_numpy(np.array([1,2,3,4,5,6], dtype=np.int64)).cuda()).cpu()

PyTorch can split the input and send them to many GPUs and merge the results back.
How does it manage embeddings and synchronization for a parallel model or a distributed model?
I wandered around PyTorch's code but it's very hard to know how the fundamentals work.

Output:
That's a great question.
PyTorch DataParallel paradigm is actually quite simple and the implementation is open-sourced here . Note that his paradigm is not recommended today as it bottlenecks at the master GPU and not efficient in data transfer.

This container parallelizes the application of the given :attr:module by
splitting the input across the specified devices by chunking in the batch
dimension (other objects will be copied once per device). In the forward
pass, the module is replicated on each device, and each replica handles a
portion of the input. During the backwards pass, gradients from each replica
are summed into the original module.

As of DistributedDataParallel, thats more tricky. This is currently the more advanced approach and it is quite efficient (see here).

This container parallelizes the application of the given module by
splitting the input across the specified devices by chunking in the batch
dimension. The module is replicated on each machine and each device, and
each such replica handles a portion of the input. During the backwards
pass, gradients from each node are averaged.

There are several approaches towards how to average the gradients from each node. I would recommend this paper to get a real sense how things work. Generally speaking, there is a trade-off between transferring the data from one GPU to another, regarding bandwidth and speed, and we want that part to be really efficient. So one possible approach is to connect each pairs of GPUs with a really fast protocol in a circle, and to pass only part of gradients from one to another, s.t. in total, we transfer less data, more efficiently, and all the nodes get all the gradients (or their average at least). There will still be a master GPU in that situation, or at least a process, but now there is no bottleneck on any GPU, they all share the same amount of data (up to...).
Now this can be further optimized if we don't wait for all the batches to finish compute and start do a time-sharing thing where each node sends his portion when he's ready. Don't take me on the details, but it turns out that if we don't wait for everything to end, and do the averaging as soon as we can, it might also speed up the gradient averaging.
Please refer to literature for more information about that area as it is still developing (as of today).
PS 1: Usually these distributed training work better on machines that are set for that task, e.g. AWS deep learning instances that implement those protocols in HW.
PS 2: Disclaimer: I really don't know what protocol PyTorch devs chose to implement and what is chosen according to what. I work with distributed training and prefer to follow PyTorch best practices without trying to outsmart them. I recommend for you to do the same unless you are really into researching this area.
References:
[1] Distributed Training of Deep Learning Models: A Taxonomic Perspective


Instruction: load test data in pytorch

Input: All is in the title, I just want to know, how can I load my own test data (image.jpg) in pytorch in order to test my CNN.

Output:
You need to feed images to net the same as in training: that is, you should apply exactly the same transformations to get similar results.

Assuming your net was trained using this code (or similar), you can see that an input image (for validation) undergoes the following transformations:

transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            normalize,
        ])),

Following torchvision.transforms docs you can see that an input image goes through:

Resizing to 256x256 pix
Cropping 224x224 rect from the center of the image
The image is converted from uint8 datatype to float in range [0, 1], and transposed to 3-by-224-by-224 array
The image is normalize by subtracting mean and dividing by std.

You can do all this manually to any image

import numpy as np
from PIL import Image

pil_img = Image.open('image.jpg').resize((256, 256), Image.BILINEAR)  # read and resize
# center crop
w, h = pil_img.size
i = int(round((h - 224) / 2.))
j = int(round((w - 224) / 2.))
pil_img = pil_img.crop((j, i, j+224, i+224))
np_img = np.array(pil_img).astype(np.float32) / 255.
np_img = np.transpose(np_img, (2, 0, 1))
# normalize
mean = [0.485, 0.456, 0.406]
std = [0.229, 0.224, 0.225]
for c in range(3):
  np_img = (np_img[c, ...] - mean[c]) / std[c]

Once you have np_img ready for your model, you can run a feed forward pass:

pred = model(np_img[None, ...])  # note that we add a singleton leading dim for batch


Instruction: Wasserstein GAN critic training ambiguity

Input:
I'm running a DCGAN-based GAN, and am experimenting with WGANs, but am a bit confused about how to train the WGAN.

In the official Wasserstein GAN PyTorch implementation, the discriminator/critic is said to be trained Diters (usually 5) times per each generator training.

Does this mean that the critic/discriminator trains on Diters batches or the whole dataset Diters times? If I'm not mistaken, the official implementation suggests the discriminator/critic is trained on the whole dataset Diters times, but other implementations of WGAN (in PyTorch and TensorFlow etc.) do the opposite.

Which is correct? The WGAN paper (to me, at least), indicates that it is Diters batches. Training on the whole dataset is obviously orders of magnitude slower.

Thanks in advance!

Output:
The correct is to consider an iteration as a batch.
In the original paper, for each iteration of the critic/discriminator they are sampling a batch of size m of the real data and  a batch of size m of prior samples p(z) to work it. After the critic is trained over Diters iterations, they train the generator which also starts by the sampling of a batch of prior samples of p(z).
Therefore, each iteration is working on a batch.

In the official implementation this is also happening. What may be confusing is that they use the variable name niter to represent the number of epochs to train the model. Although they use a different scheme to set Diters at lines 162-166:

# train the discriminator Diters times
    if gen_iterations < 25 or gen_iterations % 500 == 0:
        Diters = 100
    else:
        Diters = opt.Diters

they are, as in the paper, training the critic over Diters batches.


Instruction: Apply transformation for Tensor without including it in Backward

Input:
Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)
#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter
output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.

Output:
If you just don't want to compute gradients for your TransformationFunction, it is easiest to turn off gradient computation for all parameters involved in this computation by setting the requires_grad flag to False.

Excluding subgraphs from backward:

  If there’s a single input to an operation that requires gradient, its
  output will also require gradient. Conversely, only if all inputs
  don’t require gradient, the output also won’t require it. Backward
  computation is never performed in the subgraphs, where all Tensors
  didn’t require gradients.

  This is especially useful when you want to freeze part of your model,
  or you know in advance that you’re not going to use gradients w.r.t.
  some parameters. For example if you want to finetune a pretrained CNN,
  it’s enough to switch the requires_grad flags in the frozen base, and
  no intermediate buffers will be saved, until the computation gets to
  the last layer, where the affine transform will use weights that
  require gradient, and the output of the network will also require
  them.

Here is a small example which would do so:

import torch
import torch.nn as nn

# define layers
normal_layer = nn.Linear(5, 5)
TransformationFunction = nn.Linear(5, 5)
# disable gradient computation for parameters of TransformationFunction
# here weight and bias
TransformationFunction.weight.requires_grad = False
TransformationFunction.bias.requires_grad   = False

# input
inp = torch.rand(1, 5)

# do computation
out = normal_layer(inp)
out = TransformationFunction(out)

# loss
loss = torch.sum(out)
# backward
loss.backward()

# gradient for l1
print('Gradients for "normal_layer"', normal_layer.weight.grad, normal_layer.bias.grad)
# gradient for l2
print('Gradients for "TransformationFunction"', TransformationFunction.weight.grad, TransformationFunction.bias.grad)

Output:

Gradients for "normal_layer" tensor([[0.1607, 0.0215, 0.0192, 0.2595, 0.0811],
        [0.0788, 0.0105, 0.0094, 0.1272, 0.0398],
        [0.1552, 0.0207, 0.0186, 0.2507, 0.0784],
        [0.1541, 0.0206, 0.0184, 0.2489, 0.0778],
        [0.2945, 0.0393, 0.0352, 0.4756, 0.1486]]) tensor([0.2975, 0.1458, 0.2874, 0.2853, 0.5452])
Gradients for "TransformationFunction" None None

I hope this is what you were looking for, if not please edit your question with more detail!


Instruction: How can I change the padded input size per channel in Pytorch?

Input:
I am trying to set up an image classifier using Pytorch. My sample images have 4 channels and are 28x28 pixels in size. I am trying to use the built-in torchvision.models.inception_v3() as my model. Whenever I try to run my code, I get this error:

  RuntimeError: Calculated padded input size per channel: (1 x 1).
  Kernel size: (3 x 3). Kernel size can't greater than actual input size
  at
  /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THNN/generic/SpatialConvolutionMM.c:48

I can't find how to change the padded input size per channel or quite figure out what the error means. I figure that I must modify the padded input size per channel since I can't edit the Kernel size in the pre-made model.

I have tried padding, but it didn't help.
Here is a shortened part of my code that throws the error when I call train():

import torch
import torchvision as tv
import torch.optim as optim
from torch import nn
from torch.utils.data import DataLoader

model = tv.models.inception_v3()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001, weight_decay=0)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=4, gamma=0.9)

trn_dataset = tv.datasets.ImageFolder(
    "D:/tests/classification_test_data/trn",
    transform=tv.transforms.Compose([tv.transforms.RandomRotation((0,275)), tv.transforms.RandomHorizontalFlip(),
                                  tv.transforms.ToTensor()]))
trn_dataloader = DataLoader(trn_dataset, batch_size=32, num_workers=4, shuffle=True)

for epoch in range(0, 10):
    train(trn_dataloader, model, criterion, optimizer, lr_scheduler, 6, 32)
print("End of training")

def train(train_loader, model, criterion, optimizer, scheduler, num_classes, batch_size):
    model.train()
    scheduler.step()

    for index, data in enumerate(train_loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = model(inputs)
        outputs_flatten = flatten_outputs(outputs, num_classes)
        loss = criterion(outputs_flatten, labels)
        loss.backward()
        optimizer.step()

def flatten_outputs(predictions, number_of_classes):
    logits_permuted = predictions.permute(0, 2, 3, 1)
    logits_permuted_cont = logits_permuted.contiguous()
    outputs_flatten = logits_permuted_cont.view(-1, number_of_classes)
    return outputs_flatten

Output:
It could be due the following. Pytorch documentation for Inception_v3 model notes that the model expects input of shape Nx3x299x299. This is because the architecture contains a fully connected layer which fixed shape.

  Important: In contrast to the other models the inception_v3 expects tensors with a size of N x 3 x 299 x 299, so ensure your images are sized accordingly.

https://pytorch.org/docs/stable/torchvision/models.html#inception-v3


Instruction: How to handle Multi Label DataSet from Directory for image captioning in PyTorch

Input:
I need  a  help in PyTorch,
Regarding Dataloader, and dataset
Can someone aid/guide me

Here is my query :
I am trying for Image Captioning using https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning.

Here they have used Standard COCO Dataset.

I have dataset as images/ and captions/ directory .

Example

Directory Structure:

images/T001.jpg
images/T002.jpg
...
...
captions/T001.txt
captions/T002.txt
....
....

The above is the relation. Caption file has 'n' number of captions in each separate line.

I am able to create a custom Dataset class, in that the complete caption file content is being returned. But I want only one line alone gas to be returned.

Any guidance/suggestion on how to achieving this.

++++++++++++++++++++++++++++++++++++++++++++++++
Here is the class that i have designed:

from __future__ import print_function
import torch
from torchvision import datasets, models, transforms
from torchvision import transforms
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence
import torch.optim as optim
import torch.nn as nn
#from torch import np
import numpy as np
import utils_c
from data_loader_c import get_cust_data_loader
from models import CNN, RNN
from vocab_custom import Vocabulary, load_vocab
import os

class ImageCaptionDataSet(data.Dataset):
    def __init__(self, path, json, vocab=None, transform=None):
        self.vocab = vocab
        self.transform = transform
        self.img_dir_path = path
        self.cap_dir_path = json
        self.all_imgs_path = glob.glob(os.path.join(self.img_dir_path,'*.jpg'))
        self.all_caps_path = glob.glob(os.path.join(self.cap_dir_path,'*.txt'))
        pass

    def __getitem__(self,index):
        vocab = self.vocab

        img_path = self.all_imgs_path[index]
        img_base_name = os.path.basename(img_path)
        cap_base_name = img_base_name.replace(".jpg",".txt")
        cap_path  = os.path.join(self.cap_dir_path,cap_base_name)

        caption_all_for_a_image = open(cap_path).read().split("\n")

        image = Image.open(img_path)
        image = image.convert('RGB')

        if self.transform != None:
            # apply image preprocessing
            image = self.transform(image)

        #captions_combined = []
        #max_len = 0
        #for caption in caption_all_for_a_image:
        #    caption_str = str(caption).lower()
        #    tokens = nltk.tokenize.word_tokenize(caption_str)
        #    m = len(tokens) + 2
        #    if m>max_len:
        #        max_len = m
        #    caption = torch.Tensor([vocab(vocab.start_token())] +
        #                           [vocab(token) for token in tokens] +
        #                           [vocab(vocab.end_token())])
        #    captions_combined.append(caption)
        #    #yield image, caption
        #return image,torch.Tensor(captions_combined)

        caption_str = str(caption_all_for_a_image).lower()
        tokens = nltk.tokenize.word_tokenize(caption_str)
        caption = torch.Tensor([vocab(vocab.start_token())] +
                                   [vocab(token) for token in tokens] +
                                   [vocab(vocab.end_token())])

        return image,caption

    def __len__(self):
        return len(self.all_imgs_path)

+++++++++++++++++++++++++++++++++

Output:
First, using str() to convert the list of captions into a single string (caption_str = str(caption_all_for_a_image)) is a bad idea:

cap = ['a sentence', 'bla bla bla']
str(cap)

Returns this sting:

"['a sentence', 'bla bla bla']"

Note that [', and ', ' are part of the resulting string!

You can pick one of the captions at random:

import random
...
cap_idx = random.randi(0, len(caption_all_for_a_image)-1)  # pick one at random
caption_str = caption_all_for_a_image[cap_idx].lower()  # actual selection


Instruction: Convert PyTorch CUDA tensor to NumPy array

Input: How do I convert a torch.Tensor (on GPU) to a numpy.ndarray (on CPU)?

Output:
Use .detach() to convert from GPU / CUDA Tensor to numpy array:
tensor.detach().cpu().numpy()


Instruction: one of the variables needed for gradient computation has been modified by an inplace operation

Input:
Here is my LossFunction, when I use this function, it will make this error.
And I have tested that using the nn.L1Loss() instead of my LossFunction, and the network is ok.
what should I do? Thanks for your help!

class LossV1(nn.Module):
    def __init__(self,weight=1,pos_weight=1,scale_factor=2.5):
        super(LossV1,self).__init__()
        self.weight = weight
        self.pos_weight = pos_weight
        self.scale_factor = scale_factor

    def forward(self,pred,truth):
        objmask = torch.tensor(truth[:,:6,:,:],dtype=torch.float32,requires_grad=False)
        #没有物体的Boxes，置信度损失*0.4
        objmask[objmask<0.65] = 0.4
        #辅助Boxes，系数0.8
        objmask[(objmask>0.649)*(objmask<0.949)] = 0.8
        objLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,:6,:,:],truth[:,:6,:,:]))
        #没有物体的Boxes，只计算置信度损失
        objmask[objmask<0.41] = 0
        personLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,6:12,:,:],truth[:,6:12,:,:]))
        carLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,12:18,:,:],truth[:,12:18,:,:]))
        wLoss = torch.sum(objmask*self.myL2Loss(pred[:,18:24,:,:],truth[:,18:24,:,:]))
        hLoss = torch.sum(objmask*self.myL2Loss(pred[:,24:,:,:],truth[:,24:,:,:]))
        return objLoss+personLoss+carLoss+wLoss+hLoss

    def myBCEWithLogitsLoss(self,x,y):
        #pos_weight>1增加召回,pos_weight<1提高精度
        return -self.weight*(self.pos_weight*y*torch.log(torch.sigmoid(x))+(1-y)*torch.log(1-torch.sigmoid(x)))

    def myL2Loss(self,x,y):
        return torch.pow(self.scale_factor*torch.sigmoid(x/self.scale_factor) - y,2)

Output:
I just remove the objmask,and compute it in my generation function,then pass it on to LossFunction with truth label,and the network works.
I can't understand I have already done that making the requires_grad=False,why pytroch still compute the gradient.


Instruction: Anoconda does not allow Torchvision to be installed

Input:
I tried the following commands
conda create -n torch_env -c pytorch pytorch torchvision
conda install -c soumith/label/pytorch torchvision
conda install -c soumith torchvision

Provided by Anaconda but none of them works. Please help me I am stack !

Error message:
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

torchvision

Current channels:

https://conda.anaconda.org/pytorch/win-64
https://conda.anaconda.org/pytorch/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/win-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/win-64
https://repo.anaconda.com/pkgs/pro/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

Output:
I encountered the same problem. https://github.com/pytorch/pytorch/issues/12210
Then I found that torchvision should be installed together with pytorch in anaconda.
conda install -c pytorch torchvision


Instruction: PyTorch: get input layer size

Input:
I want to programatically find the size of my input layer.

If my first layer is called fc1, how do I find out its input?

Output:
Assuming your model is called model, this will give the number of input features of the fc1 layer:

model.fc1.in_features

This is useful inside the .forward() method:

def forward(self, x):
    x = x.view(-1, self.fc1.in_features)  # resize the input to match the input layer
    ...


Instruction: Increasingly large, positive WGAN-GP loss

Input:
I'm investigating the use of a Wasserstein GAN with gradient penalty in PyTorch, but consistently get large, positive generator losses that increase over epochs.
I'm heavily borrowing from Caogang's implementation, but am using the discriminator and generator losses used in this implementation because I get Invalid gradient at index 0 - expected shape[] but got [1] if I try to call .backward() with the one and mone args used in the Caogang implementation.

I'm training on a augmented WikiArt dataset (>400k 64x64 images) and CIFAR-10, and have gotten a normal WGAN (with weight clipping to work) [i.e. it produces passable images after 25 epochs], despite the fact that the D and G losses both hover around 3 [I calculate them using torch.mean(D_real) etc.] for all epochs. However, in the WGAN-GP version, the generator loss increases dramatically on both the WikiArt and CIFAR-10 datasets, and completely fails to generate anything other than noise on WikiArt.

Here's an example of the loss after 25 epochs on CIFAR-10:

I don't use any tricks like one-sided label smoothing, and I train with the default learning rate of 0.001, the Adam optimizer and I train the discriminator 5 times for every generator update. Why does this crazy loss behaviour happen, and why does the normal weight-clipping WGAN still 'work' on WikiArt but WGANGP completely fail?

This happens irrespective of the structure, whether both G and D are DCGANs or when using this modified DCGAN, the Creative Adversarial Network, which requires that D be able to classify images and G generate ambiguous images.

Below is the relevant part of my current trainmethod:

self.generator = Can64Generator(self.z_noise, self.channels, self.num_gen_filters).to(self.device)
self.discriminator =WCan64Discriminator(self.channels,self.y_dim, self.num_disc_filters).to(self.device)
style_criterion = nn.CrossEntropyLoss()

self.disc_optimizer = optim.Adam(self.discriminator.parameters(), lr=self.lr, betas=(self.beta1, 0.9))
self.gen_optimizer = optim.Adam(self.generator.parameters(), lr=self.lr, betas=(self.beta1, 0.9))

while i < len(dataloader):
            j = 0
            disc_loss_epoch = []
            gen_loss_epoch = []
            if self.type == "can":
                disc_class_loss_epoch = []
                gen_class_loss_epoch = []

            if self.gradient_penalty == False:
                # critic training methodology in official WGAN implementation
                if gen_iterations < 25 or (gen_iterations % 500 == 0):
                    disc_iters = 100
            else:
                disc_iters = self.disc_iterations

            while j < disc_iters and (i < len(dataloader)):
                # if using wgan with weight clipping
                if self.gradient_penalty == False:
                    # Train Discriminator
                    for param in self.discriminator.parameters():
                        param.data.clamp_(self.lower_clamp,self.upper_clamp)

                for param in self.discriminator.parameters():
                    param.requires_grad_(True)

                j+=1
                i+=1
                data = data_iterator.next()
                self.discriminator.zero_grad()
                real_images, image_labels = data
                # image labels are the the image's classes (e.g. Impressionism)
                real_images = real_images.to(self.device)
                batch_size = real_images.size(0)
                real_image_labels = torch.LongTensor(batch_size).to(self.device)
                real_image_labels.copy_(image_labels)

                labels = torch.full((batch_size,),real_label,device=self.device)

                if self.type == 'can':
                    predicted_output_real, predicted_styles_real = self.discriminator(real_images.detach())
                    predicted_styles_real = predicted_styles_real.to(self.device)
                    disc_class_loss = style_criterion(predicted_styles_real,real_image_labels)
                    disc_class_loss.backward(retain_graph=True)

                else:
                    predicted_output_real = self.discriminator(real_images.detach())

                disc_loss_real = -torch.mean(predicted_output_real)

                # fake

                noise = torch.randn(batch_size,self.z_noise,1,1,device=self.device)
                with torch.no_grad():
                    noise_g = noise.detach()
                fake_images = self.generator(noise_g)
                labels.fill_(fake_label)

                if self.type == 'can':
                    predicted_output_fake, predicted_styles_fake = self.discriminator(fake_images)

                else:
                    predicted_output_fake = self.discriminator(fake_images)

                disc_gen_z_1 = predicted_output_fake.mean().item()

                disc_loss_fake = torch.mean(predicted_output_fake)

                #via https://github.com/znxlwm/pytorch-generative-model-collections/blob/master/WGAN_GP.py
                if self.gradient_penalty:
                    # gradient penalty
                    alpha = torch.rand((real_images.size()[0], 1, 1, 1)).to(self.device)
                    x_hat = alpha * real_images.data + (1 - alpha) * fake_images.data
                    x_hat.requires_grad_(True)
                    if self.type == 'can':
                        pred_hat, _ = self.discriminator(x_hat)
                    else:
                        pred_hat = self.discriminator(x_hat)
                    gradients = grad(outputs=pred_hat, inputs=x_hat, grad_outputs=torch.ones(pred_hat.size()).to(self.device),
                                    create_graph=True, retain_graph=True, only_inputs=True)[0]

                    gradient_penalty = lambda_ * ((gradients.view(gradients.size()[0], -1).norm(2, 1) - 1) ** 2).mean()
                    disc_loss = disc_loss_fake + disc_loss_real + gradient_penalty
                else:
                    disc_loss  =  disc_loss_fake  + disc_loss_real

                if self.type == 'can':
                    disc_loss += disc_class_loss.mean()

                disc_x = disc_loss.mean().item()
                disc_loss.backward(retain_graph=True)
                self.disc_optimizer.step()

            # train generator
            for param in self.discriminator.parameters():
                param.requires_grad_(False)

            self.generator.zero_grad()
            labels.fill_(real_label)

            if self.type == 'can':
                predicted_output_fake, predicted_styles_fake = self.discriminator(fake_images)
                predicted_styles_fake = predicted_styles_fake.to(self.device)

            else:
                predicted_output_fake = self.discriminator(fake_images)

            gen_loss = -torch.mean(predicted_output_fake)
            disc_gen_z_2 = gen_loss.mean().item()

            if self.type == 'can':
                fake_batch_labels = 1.0/self.y_dim * torch.ones_like(predicted_styles_fake)
                fake_batch_labels = torch.mean(fake_batch_labels,1).long().to(self.device)
                gen_class_loss = style_criterion(predicted_styles_fake,fake_batch_labels)
                gen_class_loss.backward(retain_graph=True)
                gen_loss += gen_class_loss.mean()

            gen_loss.backward()
            gen_iterations += 1

This is the code for the (DCGAN) generator:

class Can64Generator(nn.Module):
def __init__(self, z_noise, channels, num_gen_filters):
    super(Can64Generator,self).__init__()
    self.ngpu = 1
    self.main = nn.Sequential(
    nn.ConvTranspose2d(z_noise, num_gen_filters * 16, 4, 1, 0, bias=False),
    nn.BatchNorm2d(num_gen_filters * 16),
    nn.ReLU(True),
    nn.ConvTranspose2d(num_gen_filters * 16, num_gen_filters * 4, 4, 2, 1, bias=False),
    nn.BatchNorm2d(num_gen_filters * 4),
    nn.ReLU(True),
    nn.ConvTranspose2d(num_gen_filters * 4, num_gen_filters * 2, 4, 2, 1, bias=False),
    nn.BatchNorm2d(num_gen_filters * 2),
    nn.ReLU(True),
    nn.ConvTranspose2d(num_gen_filters * 2, num_gen_filters, 4, 2, 1, bias=False),
    nn.BatchNorm2d(num_gen_filters),
    nn.ReLU(True),
    nn.ConvTranspose2d(num_gen_filters, 3, 4, 2, 1, bias=False),
    nn.Tanh()
    )
def forward(self, inp):
    output = self.main(inp)
    return output

And this is the (current) CAN discriminator, which has extra layers for
style (image class) classification):

class Can64Discriminator(nn.Module):

def __init__(self, channels,y_dim, num_disc_filters):
        super(Can64Discriminator, self).__init__()
        self.ngpu = 1
        self.conv = nn.Sequential(
                nn.Conv2d(channels, num_disc_filters // 2, 4, 2, 1, bias=False),
                nn.LeakyReLU(0.2, inplace=True),

                nn.Conv2d(num_disc_filters // 2, num_disc_filters, 4, 2, 1, bias=False),
                nn.BatchNorm2d(num_disc_filters),
                nn.LeakyReLU(0.2, inplace=True),

                nn.Conv2d(num_disc_filters, num_disc_filters * 2, 4, 2, 1, bias=False),
                nn.BatchNorm2d(num_disc_filters * 2),
                nn.LeakyReLU(0.2, inplace=True),

                nn.Conv2d(num_disc_filters * 2, num_disc_filters * 4, 4, 2, 1, bias=False),
                nn.BatchNorm2d(num_disc_filters * 4),
                nn.LeakyReLU(0.2, inplace=True),

                nn.Conv2d(num_disc_filters * 4, num_disc_filters * 8, 4, 1, 0, bias=False),
                nn.BatchNorm2d(num_disc_filters * 8),
                nn.LeakyReLU(0.2, inplace=True),

            )
        # was this
        #self.final_conv = nn.Conv2d(num_disc_filters * 8, num_disc_filters * 8, 4, 2, 1, bias=False)

        self.real_fake_head = nn.Linear(num_disc_filters * 8, 1)

        # no bn and lrelu needed
        self.sig = nn.Sigmoid()
        self.fc = nn.Sequential()
        self.fc.add_module("linear_layer{0}".format(num_disc_filters*16),nn.Linear(num_disc_filters*8,num_disc_filters*16))
        self.fc.add_module("linear_layer{0}".format(num_disc_filters*8),nn.Linear(num_disc_filters*16,num_disc_filters*8))
        self.fc.add_module("linear_layer{0}".format(num_disc_filters),nn.Linear(num_disc_filters*8,y_dim))
        self.fc.add_module('softmax',nn.Softmax(dim=1))

def forward(self, inp):
    x = self.conv(inp)
    x = x.view(x.size(0),-1)
    real_out = self.sig(self.real_fake_head(x))
    real_out = real_out.view(-1,1).squeeze(1)
    style = self.fc(x)
    #style = torch.mean(style,1) # CrossEntropyLoss requires input be (N,C)
    return real_out,style

The only differences between the WGANGP version and the WGAN version of my GAN is the WGAN version uses RMSprop with lr=0.00005 and clips the weights of the discriminator, as per the WGAN paper.

What could be causing this? I'd like to make as minimal change as possible, as I want to compare loss functions alone. The same problem is encountered even when using an unmodified DCGAN discriminator on CIFAR-10. Am I encountering this perhaps because I am only training currently for 25 epochs, or is there another reason? Interestingly, my GAN also completely fails to generate anything other than noise when using LSGAN (nn.MSELoss()).

Thanks in advance!

Output:
Batch Normalization in the discriminator breaks Wasserstein GANs with gradient penalty. The authors themselves advocate the usage of layer normalization instead, but this is clearly written in bold in their paper (https://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans.pdf). It is hard to say if there are other bugs in your code, but I urge you to thoroughly read the DCGAN and the Wasserstein GAN paper and really take notes on the hyperparameters. Getting them wrong really destroys the performance of the GAN and doing a hyperparameter search gets expensive quite quickly.

By the way transposed convolutions produce stairway artifacts in your output images. Use image resizing instead. For an indepth explanation of that phenomenon I can recommend the following resource (https://distill.pub/2016/deconv-checkerboard/).


Instruction: Pytorch RuntimeError: CUDA error: out of memory at loss.backward() , No error when using CPU

Input:
I'm training a Fully convolutional network (FCN32) for semantic segmentation on Tesla K80 with more than 11G memory.

The input image is pretty large: 352x1216. Network structure is shown below. I used batch_size=1, but still encounter the
out_of_memory error.

Criterion is nn.BCEWithLogitsLoss()

The network works fine when I run on CPU.

    Layer (type)               Output Shape       #  Param
        Conv2d-1        [-1, 64, 352, 1216]           1,792
        Conv2d-2        [-1, 64, 352, 1216]          36,928
     MaxPool2d-3         [-1, 64, 176, 608]               0
        Conv2d-4        [-1, 128, 176, 608]          73,856
        Conv2d-5        [-1, 128, 176, 608]         147,584
     MaxPool2d-6         [-1, 128, 88, 304]               0
        Conv2d-7         [-1, 256, 88, 304]         295,168
        Conv2d-8         [-1, 256, 88, 304]         590,080
        Conv2d-9         [-1, 256, 88, 304]         590,080
    MaxPool2d-10         [-1, 256, 44, 152]               0
       Conv2d-11         [-1, 512, 44, 152]       1,180,160
       Conv2d-12         [-1, 512, 44, 152]       2,359,808
       Conv2d-13         [-1, 512, 44, 152]       2,359,808
    MaxPool2d-14          [-1, 512, 22, 76]               0
       Conv2d-15          [-1, 512, 22, 76]       2,359,808
       Conv2d-16          [-1, 512, 22, 76]       2,359,808
       Conv2d-17          [-1, 512, 22, 76]       2,359,808
    MaxPool2d-18          [-1, 512, 11, 38]               0
       Conv2d-19         [-1, 4096, 11, 38]     102,764,544
       Conv2d-20         [-1, 4096, 11, 38]      16,781,312
       Conv2d-21          [-1, 1, 11, 38]           4,097  ConvTranspose2d-22         [-1, 1, 352, 1216]             4,096

Error message:

  --------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call
  last)  in ()
       36         print (loss)
       37         #torch.cuda.empty_cache()
  ---> 38         loss.backward()
       39         optimizer.step()
       40

  /anaconda/envs/py35/lib/python3.5/site-packages/torch/tensor.py in
  backward(self, gradient, retain_graph, create_graph)
       91                 products. Defaults to False.
       92         """
  ---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
       94
       95     def register_hook(self, hook):

  /anaconda/envs/py35/lib/python3.5/site-packages/torch/autograd/init.py
  in backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       88     Variable._execution_engine.run_backward(
       89         tensors, grad_tensors, retain_graph, create_graph,
  ---> 90         allow_unreachable=True)  # allow_unreachable flag
       91
       92

  RuntimeError: CUDA error: out of memory

Output:
Usually this happens because of memory on your GPU. If you have more powerful GPUs, your problem could be solved (as you mentioned in your answer).
But if you do not have, you can scale down your images into about 256*x sizes.
It is also good practice for performance's sake.


Instruction: Gradient is none in pytorch when it shouldn't

Input:
I am trying to get/trace the gradient of a variable using pytorch, where I have that variable, pass it to a first function that looks for some minimum value of some other variable, then the output of the first function is inputted to a second function, and the whole thing repeats multiple times.

Here is my code:

import torch

def myFirstFunction(parameter_current_here):
    optimalValue = 100000000000000
    Optimal = 100000000000000
    for j in range(2, 10):
        i = torch.ones(1, requires_grad=True)*j
        with torch.enable_grad():
            optimalValueNow = i*parameter_current_here.sum()
        if (optimalValueNow < optimalValue):
            optimalValue = optimalValueNow
            Optimal = i
    return optimalValue, Optimal

def mySecondFunction(Current):
    with torch.enable_grad():
        y = (20*Current)/2 + (Current**2)/10
    return y

counter = 0
while counter < 5:
    parameter_current = torch.randn(2, 2, requires_grad=True)

    outputMyFirstFunction = myFirstFunction(parameter_current)
    outputmySecondFunction = mySecondFunction(outputMyFirstFunction[1])
    outputmySecondFunction.backward()

    print("outputMyFirstFunction after backward:",
               outputMyFirstFunction)
    print("outputmySecondFunction after backward:",
               outputmySecondFunction)
    print("parameter_current Gradient after backward:",
               parameter_current.grad)

    counter = counter + 1

The parameter_current.grad is none for all iterations when it obviously shouldn't be none. What am I doing wrong? And how can I fix it?

Your help on this would be highly appreciated. Thanks a lot!

Aly

Output:
I have a similar experience with this. Reference: https://pytorch.org/docs/stable/tensors.html

For Tensors that have requires_grad which is True, they will be leaf Tensors if they were created by the user. This means that they are not the result of an operation and so grad_fn is None.
Only leaf Tensors will have their grad populated during a call to backward(). To get grad populated for non-leaf Tensors, you can use retain_grad().
Example:

    >>> a = torch.tensor([[1,1],[2,2]], dtype=torch.float, requires_grad=True)
    >>> a.is_leaf
    True
    >>> b = a * a
    >>> b.is_leaf
    False
    >>> c = b.mean()
    >>> c.backward()
    >>> print(c.grad)
    None
    >>> print(b.grad)
    None
    >>> print(a.grad)
    tensor([[0.5000, 0.5000],
            [1.0000, 1.0000]])
    >>> b = a * a
    >>> c = b.mean()
    >>> b.retain_grad()
    >>> c.retain_grad()
    >>> c.backward()
    >>> print(a.grad)
    tensor([[1., 1.],
            [2., 2.]])
    >>> print(b.grad)
    tensor([[0.2500, 0.2500],
            [0.2500, 0.2500]])
    >>> print(c.grad)
    tensor(1.)