Instruction: Accessing PyTorch GPU matrix from TensorFlow directly

Input:
I have a neural network written in PyTorch, that outputs some Tensor a on GPU. I would like to continue processing a with a highly efficient TensorFlow layer.

As far as I know, the only way to do this is to move a from GPU memory to CPU memory, convert to numpy, and then feed that into TensorFlow. A simplified example:

import torch
import tensorflow as tf

# output of some neural network written in PyTorch
a = torch.ones((10, 10), dtype=torch.float32).cuda()

# move to CPU / pinned memory
c = a.to('cpu', non_blocking=True)

# setup TensorFlow stuff (only needs to happen once)
sess = tf.Session()
c_ph = tf.placeholder(tf.float32, shape=c.shape)
c_mean = tf.reduce_mean(c_ph)

# run TensorFlow
print(sess.run(c_mean, feed_dict={c_ph: c.numpy()}))

This is a bit far fetched maybe but is there a way to make it so that either

a never leaves GPU memory, or
a goes from GPU memory to Pinned Memory to GPU memory.

I attempted 2. in the code snipped above using non_blocking=True but I am not sure if it does what I expect (i.e. move it to pinned memory).

Ideally, my TensorFlow graph would operate directly on the memory occupied by the PyTorch tensor, but I supposed that is not possible?

Output:
I am not familiar with tensorflow, but you may use pyTorch to expose the "internals" of a tensor.
You can access the underlying storage of a tensor

a.storage()

Once you have the storage, you can get a pointer to the memory (either CPU or GPU):

a.storage().data_ptr()

You can check if it is pinned or not

a.storage().is_pinned()

And you can pin it

a.storage().pin_memory()

I am not familiar with interfaces between pyTorch and tensorflow, but I came across an example of a package (FAISS) directly accessing pytorch tensors in GPU.


Instruction: conda list returning run-time error Path not Found after installing PyTortch

Input:
Recently I tried to install Pytrotch using the command
    conda install pytorch torchvision cuda100 -c pytorch

To verity the package installed correctly I ran conda list in the anaconda prompt and got the following error:

RuntimeError: Path not found: C:\Users\[name]\AppData\Local\Continuum\Anaconda3\Lib\site-packages\Sphinx-1.5.6-py3.6.egg\EGG-INFO

I'm currently running conda version 4.6.1 and python version 3.6.7 on windows 10, I'd appreciate any  help in determining what caused this error and how it can be fixed so I can properly manage my anaconda packages.

Full stack trace:

Traceback (most recent call last):
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\exceptions.py", line 1001, in __call__
        return func(*args, **kwargs)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\conda_argparse.py", line 81, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 142, in execute
        show_channel_urls=context.show_channel_urls)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 80, in print_packages
        show_channel_urls=show_channel_urls)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 45, in list_packages
        installed = sorted(PrefixData(prefix, pip_interop_enabled=True).iter_records(),
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 116, in iter_records
        return itervalues(self._prefix_records)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 145, in _prefix_records
        return self.__prefix_records or self.load() or self.__prefix_records
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 69, in load
        self._load_site_packages()
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 258, in _load_site_packages
        python_record = read_python_record(self.prefix_path, af, python_pkg_record.version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\gateways\disk\read.py", line 245, in read_python_record
        pydist = PythonDistribution.init(prefix_path, anchor_file, python_version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 90, in init
        return PythonEggInfoDistribution(anchor_full_path, python_version, sp_reference)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 400, in __init__
        super(PythonEggInfoDistribution, self).__init__(anchor_full_path, python_version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 104, in __init__
        raise RuntimeError("Path not found: %s" % anchor_full_path)
    RuntimeError: Path not found: C:\Users\[name]\AppData\Local\Continuum\Anaconda3\Lib\site-packages\Sphinx-1.5.6-py3.6.egg\EGG-INFO

any help would be appreciated.

Output:
AS @P.Antoniadis said, this is an ongoing issue. And removing 'Sphinx-1.5.6-py3.6.egg' folder is the suggested workaround.

https://github.com/conda/conda/issues/8156#issuecomment-458777849


Instruction: How to use GPUs with Ray in Pytorch? Should I specify the num_gpus for the remote class?

Input:
When I use the Ray with pytorch, I do not set any num_gpus flag for the remote class.

I get the following error:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False.

The main process is: I create a remote class and transfer a pytorch model state_dict()(created in main function) to it. In the main function, the torch.cuda.is_available() is True, but In the remote function, torch.cuda.is_available() is False. Thanks

I try to set the num_gpus=1 and got a new issue: the program just got stuck. Below is the minimal example code for reproducing this issue. Thanks.

import ray

@ray.remote(num_gpus=1)
class Worker(object):
    def __init__(self, args):
        self.args = args
        self.gen_frames = 0

    def set_gen_frames(self, value):
        self.gen_frames = value
        return self.gen_frames

    def get_gen_num(self):
        return self.gen_frames

class Parameters:
    def __init__(self):
        self.is_cuda = False;
        self.is_memory_cuda = True
        self.pop_size = 10

if __name__ == "__main__":
    ray.init()
    args = Parameters()
    workers = [Worker.remote(args) for _ in range(args.pop_size)]
    get_num_ids = [worker.get_gen_num.remote() for worker in workers]
    gen_nums = ray.get(get_num_ids)
    print(gen_nums)

Output: If you also want to deploy the model on a gpu, you need to make sure that your actor or task indeed has access to a gpu (with @ray.remote(num_gpus=1), this will make sure that torch.cuda.is_available() will be true in that remote function). If you want to deploy your model on a CPU, you need to specify that when loading the model, see for example https://github.com/pytorch/pytorch/issues/9139.


Instruction: AttributeError: module 'torch' has no attribute 'device'

Input:
In following the Pytorch tutorial at https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

I received the following error:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ python pytorch-1.py
Traceback (most recent call last):
  File "pytorch-1.py", line 39, in <module>
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
AttributeError: module 'torch' has no attribute 'device'

In my code below, I added this statement:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    net.to(device)

But this seems not right or enough. This is the first time for me to run Pytorch with GPU on a linux machine. What else should I do to get right running?

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x

net = Net()

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
print(transform)

trainSet = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainLoader = torch.utils.data.DataLoader(trainSet, batch_size=4, shuffle=True, num_workers=2)

testSet = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testLoader = torch.utils.data.DataLoader(testSet, batch_size=4, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):
    running_loss = 0.0
    for i, data in enumerate(trainLoader, 0):
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)

        optimizer.zero_grad()

        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss %.3f' % (epoch + 1, i + 1, running_loss / 2000))

print('Finished traning!')

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(numpy.transpose(npimg, (1, 2, 0)))
    plt.show()

dataIter = iter(trainLoader)
images, labels = dataIter.next()
# imshow(torchvision.utils.make_grid(images))

print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

outputs = net(images)

_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join('%5s' % classes[predicted[j]] for j in range(4)))
dataIter = iter(testLoader)
images, labels = dataIter.next()
# imshow(torchvision.utils.make_grid(images))

correct = 0
total = 0

with torch.no_grad():
    for data in testLoader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)

        correct += (predicted == labels).sum().item()

print("accuracy: %d %%", 100 * correct / total)

EDIT:

My conda version is the latest:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda -V
conda 4.6.2

Then I installed pytorch-gpu with:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda install -c anaconda pytorch-gpu

As you can see, the version 0.1.12 is installed:

Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /home/martin/anaconda3/envs/pt_gpu

  added / updated specs:
    - pytorch-gpu

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2018.12.5  |                0         123 KB  anaconda
    certifi-2018.11.29         |           py36_0         146 KB  anaconda
    pytorch-gpu-0.1.12         |           py36_0        16.8 MB  anaconda
    ------------------------------------------------------------
                                           Total:        17.0 MB

The following packages will be UPDATED:

  openssl              pkgs/main::openssl-1.1.1a-h7b6447c_0 --> anaconda::openssl-1.1.1-h7b6447c_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates                                 pkgs/main --> anaconda
  certifi                                         pkgs/main --> anaconda
  mkl                    pkgs/main::mkl-2017.0.4-h4c4d0af_0 --> anaconda::mkl-2017.0.1-0
  pytorch-gpu                                     pkgs/free --> anaconda

Proceed ([y]/n)? y

Downloading and Extracting Packages
certifi-2018.11.29   | 146 KB    | ########################################################################################################################## | 100%
ca-certificates-2018 | 123 KB    | ########################################################################################################################## | 100%
pytorch-gpu-0.1.12   | 16.8 MB   | ########################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To verify the version, I do:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ python -c "import torch; print(torch.__version__)"
0.1.12

Why does it install such a low version?

Output:
Although this question is very old, I would recommend those who are facing this problem to visit pytorch.org and check the command to install pytorch from there, there is a section dedicated to this:

or in your case:

As you can see, the command you used to install pytorch is different from the one here. I have not tested it on Linux, but I used the command for Windows and it worked great for me on Anaconda. (Initially, I also got the same error, that was before following this)


Instruction: GANs on color images

Input:
Most (PyTorch) open source GANs work on  MNIST dataset, i.e. gray level image.

Can I use a GAN on each channel of a color image, then combine the result?

Output:
You can just have your generator and discriminator generate and classify 3-channel images - speaking in terms of implementation, make them work on B x 3 x H x W tensors instead of B x 1 x H x W, as they do for MNIST.

You can't just use your GAN on each channel separately and concatenate at the end, because you would have no way to ensure that each channel corresponds to the same image. Say you're generating celebrity faces by first generating red channel, then green and finally blue. How would you make sure that you don't get a female sample for the red channel and a male for the green?


Instruction: How to return back to cpu from gpu in pytorch?

Input:
I have a text classifier in pytorch and I want to use GPUs to increase running speed.
I have used this part of code to check CUDA and use it:

if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    my_rnn_model = nn.DataParallel(my_rnn_model)
if torch.cuda.is_available():
    my_rnn_model.cuda()

Now I want to return back to use cpu (instead of gpu). So I cleared this part of code. But it does’nt work and I receive this error:

RuntimeError: cuda runtime error (8) : invalid device function at /opt/conda/conda-bld/pytorch_1503963423183/work/torch/lib/THC/THCTensorCopy.cu:204

Would you please guide me how can I return back to cpu running?

Output:
You can set the GPU device that you want to use using:

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

And in your case just you can return to CPU using:

torch.device('cpu')


Instruction: Problem with building PyTorch from source on Linux

Input:
❓ Problem with building PyTorch from source

Hello everyone,

I have problem with building PyTorch from source. I followed the official build instructions. I use Anaconda Python 3.7.1 (version 2018.12, build py37_0). I installed all neccessary dependencies using conda and issued  python setup.py install command to build it. It builds all files successfully but then it fails at the installation step saying:

build/temp.linux-x86_64-3.7/torch/csrc/stub.o: file not recognized: file format not recognized

I tried building with gcc/g++ versions 5,6,7 but it did not help.  Can you help me resolve this problem?

Build output in terminal

This is the output of python setup.py install with the error I am getting:

[manjaro-pc pytorch]# python setup.py install
Building wheel torch-1.1.0a0+44809fd
-- Building version 1.1.0a0+44809fd
[0/1] Install the project...
-- Install configuration: "Release"
running install
running build
running build_py
copying torch/version.py -> build/lib.linux-x86_64-3.7/torch
copying caffe2/proto/caffe2_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/caffe2_legacy_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/predictor_consts_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/metanet_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/torch_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/prof_dag_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/hsm_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
running build_ext
-- Building with NumPy bindings
-- Not using cuDNN
-- Not using MIOpen
-- Not using CUDA
-- Not using MKLDNN
-- Not using NCCL
-- Building without distributed package

Copying extension caffe2.python.caffe2_pybind11_state
Copying caffe2.python.caffe2_pybind11_state from torch/lib/python3.7/site-packages/caffe2/python/caffe2_pybind11_state.cpython-37m-x86_64-linux-gnu.so to /home/manjaro/Downloads/pytorch/build/lib.linux-x86_64-3.7/caffe2/python/caffe2_pybind11_state.cpython-37m-x86_64-linux-gnu.so
building 'torch._C' extension
gcc -pthread -B /opt/anaconda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/anaconda/include/python3.7m -c torch/csrc/stub.cpp -o build/temp.linux-x86_64-3.7/torch/csrc/stub.o -std=c++11 -Wall -Wextra -Wno-strict-overflow -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-deprecated-declarations -fno-strict-aliasing -Wno-missing-braces
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /opt/anaconda/compiler_compat -L/opt/anaconda/lib -Wl,-rpath=/opt/anaconda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/torch/csrc/stub.o -L/home/manjaro/Downloads/pytorch/torch/lib -lshm -ltorch_python -o build/lib.linux-x86_64-3.7/torch/_C.cpython-37m-x86_64-linux-gnu.so -Wl,-rpath,$ORIGIN/lib
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/torch/csrc/stub.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

From this output, I assume the error was caused by this line:

g++ -pthread -shared -B /opt/anaconda/compiler_compat -L/opt/anaconda/lib -Wl,-rpath=/opt/anaconda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/torch/csrc/stub.o -L/home/manjaro/Downloads/pytorch/torch/lib -lshm -ltorch_python -o build/lib.linux-x86_64-3.7/torch/_C.cpython-37m-x86_64-linux-gnu.so -Wl,-rpath,$ORIGIN/lib

My environment

PyTorch version: 1.0
Is debug build: N/A
CUDA used to build PyTorch: 10

OS: Manjaro Linux
GCC version: (GCC) 6.4.1 20171002
CMake version: version 3.12.2

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 650
Nvidia driver version: 415.27
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.7.4.1
/usr/lib/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl
[conda] magma-cuda100             2.4.0                         1    pytorch
[conda] mkl                       2019.1                      144
[conda] mkl-include               2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] mkldnn                    0.16.1                        0    mingfeima

My CMake configuration printed before building PyTorch

Building wheel torch-1.1.0a0+44809fd
-- Building version 1.1.0a0+44809fd
['cmake',
 '/home/manjaro/Downloads/pytorch',
 '-GNinja',
 '-DBLAS=MKL',
 '-DBUILDING_WITH_TORCH_LIBS=ON',
 '-DBUILD_BINARY=False',
 '-DBUILD_CAFFE2_OPS=False',
 '-DBUILD_PYTHON=True',
 '-DBUILD_SHARED_LIBS=ON',
 '-DBUILD_TEST=False',
 '-DBUILD_TORCH=ON',
 '-DCAFFE2_STATIC_LINK_CUDA=False',
 '-DCMAKE_BUILD_TYPE=Release',
 '-DCMAKE_CXX_FLAGS= ',
 '-DCMAKE_C_FLAGS= ',
 '-DCMAKE_EXE_LINKER_FLAGS=',
 '-DCMAKE_INSTALL_PREFIX=/home/manjaro/Downloads/pytorch/torch',
 '-DCMAKE_PREFIX_PATH=/opt/anaconda/bin/../',
 '-DCMAKE_SHARED_LINKER_FLAGS=',
 '-DINSTALL_TEST=False',
 '-DNCCL_EXTERNAL=False',
 '-DNUMPY_INCLUDE_DIR=/opt/anaconda/lib/python3.7/site-packages/numpy/core/include',
 '-DONNX_NAMESPACE=onnx_torch',
 '-DPYTHON_EXECUTABLE=/opt/anaconda/bin/python',
 '-DPYTHON_INCLUDE_DIR=/opt/anaconda/include/python3.7m',
 '-DPYTHON_LIBRARY=/opt/anaconda/lib/libpython3.7m.so.1.0',
 '-DTHD_SO_VERSION=1',
 '-DTORCH_BUILD_VERSION=1.1.0a0+44809fd',
 '-DUSE_CUDA=False',
 '-DUSE_DISTRIBUTED=False',
 '-DUSE_FBGEMM=False',
 '-DUSE_FFMPEG=False',
 '-DUSE_LEVELDB=False',
 '-DUSE_LMDB=False',
 '-DUSE_MKLDNN=False',
 '-DUSE_NNPACK=False',
 '-DUSE_NUMPY=True',
 '-DUSE_OPENCV=False',
 '-DUSE_QNNPACK=False',
 '-DUSE_ROCM=False',
 '-DUSE_SYSTEM_EIGEN_INSTALL=OFF',
 '-DUSE_SYSTEM_NCCL=False',
 '-DUSE_TENSORRT=False']
{'BLAS': 'MKL',
 'BUILD_CAFFE2_OPS': '0',
 'BUILD_TEST': '0',
 'CMAKE_PREFIX_PATH': '/opt/anaconda/bin/../',
 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',
 'HG': '/usr/bin/hg',
 'HOME': '/root',
 'LANG': 'en_US.UTF-8',
 'LC_ADDRESS': 'sk_SK.UTF-8',
 'LC_IDENTIFICATION': 'sk_SK.UTF-8',
 'LC_MEASUREMENT': 'sk_SK.UTF-8',
 'LC_MONETARY': 'sk_SK.UTF-8',
 'LC_NAME': 'sk_SK.UTF-8',
 'LC_NUMERIC': 'sk_SK.UTF-8',
 'LC_PAPER': 'sk_SK.UTF-8',
 'LC_TELEPHONE': 'sk_SK.UTF-8',
 'LC_TIME': 'sk_SK.UTF-8',
 'LD_LIBRARY_PATH': ':/usr/local/lib64:/usr/local/lib:/opt/cuda/lib64',
 'LOGNAME': 'manjaro',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:',
 'MAIL': '/var/spool/mail/manjaro',
 'MAX_JOBS': '4',
 'MOZ_PLUGIN_PATH': '/usr/lib/mozilla/plugins',
 'NO_CAFFE2_OPS': '1',
 'NO_CUDA': '1',
 'NO_CUDNN': '1',
 'NO_DISTRIBUTED': '1',
 'NO_FBGEMM': '1',
 'NO_MIOPEN': '1',
 'NO_MKLDNN': '1',
 'NO_NNPACK': '1',
 'NO_QNNPACK': '1',
 'NO_TEST': '1',
 'OLDPWD': '/home/manjaro/Downloads',
 'PATH': '/opt/anaconda/bin:/opt/anaconda/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/local/bin:/opt/cuda/bin:/usr/local/bin:/opt/cuda/bin',
 'PWD': '/home/manjaro/Downloads/pytorch',
 'PYTHONPATH': ':/usr/local/lib/python3.7/site-packages/:/usr/local/lib/python3.7/site-packages/',
 'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1',
 'SHELL': '/bin/bash',
 'SHLVL': '2',
 'SSH_CLIENT': '62.197.243.30 45360 44444',
 'SSH_CONNECTION': '62.197.243.30 45360 192.168.1.147 44444',
 'SSH_TTY': '/dev/pts/4',
 'TERM': 'xterm-256color',
 'TORCH_CUDA_ARCH_LIST': '3.0',
 'USER': 'manjaro',
 'USE_CUDA': '0',
 'USE_CUDNN': '0',
 'USE_DISTRIBUTED': '0',
 'USE_FBGEMM': '0',
 'USE_MIOPEN': '0',
 'USE_MKLDNN': '0',
 'USE_NNPACK': '0',
 'USE_QNNPACK': '0',
 'XDG_DATA_DIRS': '/home/manjaro/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share',
 'XDG_RUNTIME_DIR': '/run/user/1000',
 'XDG_SESSION_ID': 'c8',
 'YAOURT_COLORS': 'nb=1:pkg=1:ver=1;32:lver=1;45:installed=1;42:grp=1;34:od=1;41;5:votes=1;44:dsc=0:other=1;35',
 '_': '/opt/anaconda/bin/python'}
-- The CXX compiler identification is GNU 5.4.1
-- The C compiler identification is GNU 5.4.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE - Success
-- NUMA is available
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Success
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/manjaro/Downloads/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: /opt/anaconda/lib/libmkl_intel_lp64.so
--   Library mkl_gnu_thread: /opt/anaconda/lib/libmkl_gnu_thread.so
--   Library mkl_core: /opt/anaconda/lib/libmkl_core.so
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Found OpenMP: TRUE (found version "4.0")
--   Library gomp: -fopenmp
--   Library pthread: /usr/lib/libpthread.so
--   Library m: /usr/lib/libm.so
--   Library dl: /usr/lib/libdl.so
-- Looking for cblas_sgemm
-- Looking for cblas_sgemm - found
-- MKL library found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Caffe2: Found gflags with new-style gflags target.
-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Found glog: /usr/include
-- Caffe2: Found glog (include: /usr/include, library: /usr/lib/libglog.so)
-- Found Numa: /usr/include
-- Found Numa  (include: /usr/include, library: /usr/lib/libnuma.so)
-- Downloading PSimd to /home/manjaro/Downloads/pytorch/build/confu-srcs/psimd (define PSIMD_SOURCE_DIR to avoid it)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/manjaro/Downloads/pytorch/build/confu-deps/psimd-download
[1/9] Creating directories for 'psimd'
[2/9] Performing download step (git clone) for 'psimd'
Cloning into 'psimd'...
Already on 'master'
Your branch is up to date with 'origin/master'.
[3/9] No patch step for 'psimd'
[4/9] Performing update step for 'psimd'
Current branch master is up to date.
[5/9] No configure step for 'psimd'
[6/9] No build step for 'psimd'
[7/9] No install step for 'psimd'
[8/9] No test step for 'psimd'
[9/9] Completed 'psimd'
-- Using third party subdirectory Eigen.
Python 3.7.1
-- Found PythonInterp: /opt/anaconda/bin/python (found suitable version "3.7.1", minimum required is "2.7")
-- Found PythonLibs: /opt/anaconda/lib/libpython3.7m.so.1.0 (found suitable version "3.7.1", minimum required is "2.7")
-- Found PythonInterp: /opt/anaconda/bin/python (found version "3.7.1")
-- Found PythonLibs: /opt/anaconda/lib/libpython3.7m.so.1.0
-- System pybind11 found
-- pybind11 include dirs: /usr/include;/opt/anaconda/include/python3.7m
CMake Warning at cmake/Dependencies.cmake:805 (message):
  Not using CUDA, so disabling NCCL.  Suppress this warning with
  -DUSE_NCCL=OFF.
Call Stack (most recent call first):
  CMakeLists.txt:219 (include)

CMake Warning at cmake/Dependencies.cmake:950 (message):
  Metal is only used in ios builds.
Call Stack (most recent call first):
  CMakeLists.txt:219 (include)

--
-- ******** Summary ********
--   CMake version         : 3.12.2
--   CMake command         : /opt/anaconda/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.1
--   CXX flags             :   -fvisibility-inlines-hidden -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL
--   CMAKE_PREFIX_PATH     : /opt/anaconda/bin/../
--   CMAKE_INSTALL_PREFIX  : /home/manjaro/Downloads/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/manjaro/Downloads/pytorch/cmake/Modules;/usr/share/cmake/pybind11
--
--   ONNX version          : 1.4.1
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
--
--   Protobuf compiler     :
--   Protobuf includes     :
--   Protobuf libraries    :
--   BUILD_ONNX_PYTHON     : OFF
-- Found gcc >=5 and CUDA <= 7.5, adding workaround C++ flags
-- Could not find CUDA with FP16 support, compiling without torch.CudaHalfTensor
-- Removing -DNDEBUG from compile flags
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Compiling with OpenMP support
-- MAGMA not found. Compiling without MAGMA support
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Performing Test HAVE_GCC_GET_CPUID
-- Performing Test HAVE_GCC_GET_CPUID - Success
-- Performing Test NO_GCC_EBX_FPIC_BUG
-- Performing Test NO_GCC_EBX_FPIC_BUG - Success
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- AVX compiler support found
-- AVX2 compiler support found
-- Performing Test HAS_C11_ATOMICS
-- Performing Test HAS_C11_ATOMICS - Failed
-- Performing Test HAS_MSC_ATOMICS
-- Performing Test HAS_MSC_ATOMICS - Failed
-- Performing Test HAS_GCC_ATOMICS
-- Performing Test HAS_GCC_ATOMICS - Success
-- Atomics: using GCC intrinsics
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API (mkl).
disabling CUDA because NOT USE_CUDA is set
-- CuDNN not found. Compiling without CuDNN support
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
disabling MKLDNN because USE_MKLDNN is not set
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for shm_open
-- Looking for shm_open - found
-- Looking for shm_unlink
-- Looking for shm_unlink - found
-- Looking for malloc_usable_size
-- Looking for malloc_usable_size - found
-- Performing Test C_HAS_THREAD
-- Performing Test C_HAS_THREAD - Success
-- GCC 5.4.1: Adding gcc and gcc_s libs to link line
-- NUMA paths:
-- /usr/include
-- /usr/lib/libnuma.so
disabling CUDA because USE_CUDA is set false
-- Check size of long double
-- Check size of long double - done
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE - Success
-- Performing Test COMPILER_SUPPORTS_FLOAT128
-- Performing Test COMPILER_SUPPORTS_FLOAT128 - Success
-- Performing Test COMPILER_SUPPORTS_SSE2
-- Performing Test COMPILER_SUPPORTS_SSE2 - Success
-- Performing Test COMPILER_SUPPORTS_SSE4
-- Performing Test COMPILER_SUPPORTS_SSE4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX
-- Performing Test COMPILER_SUPPORTS_AVX - Success
-- Performing Test COMPILER_SUPPORTS_FMA4
-- Performing Test COMPILER_SUPPORTS_FMA4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX2
-- Performing Test COMPILER_SUPPORTS_AVX2 - Success
-- Performing Test COMPILER_SUPPORTS_SVE
-- Performing Test COMPILER_SUPPORTS_SVE - Failed
-- Performing Test COMPILER_SUPPORTS_AVX512F
-- Performing Test COMPILER_SUPPORTS_AVX512F - Failed
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Performing Test COMPILER_SUPPORTS_OPENMP
-- Performing Test COMPILER_SUPPORTS_OPENMP - Success
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES - Success
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH - Success
-- Configuring build for SLEEF-v3.2
   Target system: Linux-4.14.94-1-MANJARO
   Target processor: x86_64
   Host system: Linux-4.14.94-1-MANJARO
   Host processor: x86_64
   Detected C compiler: GNU @ /usr/bin/cc
-- Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
-- Building shared libs : OFF
-- MPFR : /opt/anaconda/lib/libmpfr.so
-- MPFR header file in /opt/anaconda/include
-- GMP : /opt/anaconda/lib/libgmp.so
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
-- /usr/bin/c++ /home/manjaro/Downloads/pytorch/torch/abi-check.cpp -o /home/manjaro/Downloads/pytorch/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
-- NCCL operators skipped due to no CUDA support
-- Excluding ideep operators as we are not using ideep
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- MPI operators skipped due to no MPI support
-- Include Observer library
-- Using lib/python3.7/site-packages as python relative installation path
CMake Warning at CMakeLists.txt:416 (message):
  Generated cmake files are only fully tested if one builds with system glog,
  gflags, and protobuf.  Other settings may generate files that are not well
  tested.

--
-- ******** Summary ********
-- General:
--   CMake version         : 3.12.2
--   CMake command         : /opt/anaconda/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.1
--   BLAS                  : MKL
--   CXX flags             :   -fvisibility-inlines-hidden -D_FORCE_INLINES -D_MWAITXINTRIN_H_INCLUDED -D__STRICT_ANSI__ -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_NAMESPACE=onnx_torch;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /opt/anaconda/bin/../
--   CMAKE_INSTALL_PREFIX  : /home/manjaro/Downloads/pytorch/torch
--
--   TORCH_VERSION         : 1.1.0
--   CAFFE2_VERSION        : 1.1.0
--   BUILD_ATEN_MOBILE     : OFF
--   BUILD_ATEN_ONLY       : OFF
--   BUILD_BINARY          : False
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.7.1
--     Python executable   : /opt/anaconda/bin/python
--     Pythonlibs version  : 3.7.1
--     Python library      : /opt/anaconda/lib/libpython3.7m.so.1.0
--     Python includes     : /opt/anaconda/include/python3.7m
--     Python site-packages: lib/python3.7/site-packages
--   BUILD_CAFFE2_OPS      : False
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : False
--   USE_ASAN              : OFF
--   USE_CUDA              : False
--   USE_ROCM              : False
--   USE_EIGEN_FOR_BLAS    :
--   USE_FBGEMM            : OFF
--   USE_FFMPEG            : False
--   USE_GFLAGS            : ON
--   USE_GLOG              : ON
--   USE_LEVELDB           : False
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : False
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : OFF
--   USE_NCCL              : OFF
--   USE_NNPACK            : False
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : False
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : False
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : False
--   Public Dependencies  : Threads::Threads;caffe2::mkl;glog::glog
--   Private Dependencies : cpuinfo;/usr/lib/libnuma.so;fp16;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    NCCL_EXTERNAL
    THD_SO_VERSION

Output:
Problem solved.

I found what was wrong and I fixed it. The whole problem lies in the fact that Anaconda distribution comes with its own ld linker that is located in /opt/anaconda/compiler_compat/ and it overshadows system ld residing at /usr/bin.

To fix my error I ran python setup.py clean and then I temporarily renamed Anaconda's ld linker to ld-old to make it invisible during PyTorch installation.


Instruction: Running LSTM with multiple GPUs gets "Input and hidden tensors are not at the same device"

Input:
I am trying to train a LSTM layer in pytorch. I am using 4 GPUs. When initializing, I added the .cuda() function move the hidden layer to GPU. But when I run the code with multiple GPUs I am getting this runtime error :

RuntimeError: Input and hidden tensors are not at the same device

I have tried to solve the problem by using .cuda() function in the forward function like below :

self.hidden = (self.hidden[0].type(torch.FloatTensor).cuda(), self.hidden[1].type(torch.FloatTensor).cuda())

This line seems to solve the problem, but it raises my concern that if the updated hidden layer is seen in different GPUs. Should I move the vector back to cpu at the end of the forward function for a batch or is there any other way to solve the problem.

Output:
When you call .cuda() on the tensor, Pytorch moves it to the current GPU device by default (GPU-0). So, due to data parallelism, your data lives in a different GPU while your model goes to another, this results in the runtime error you are facing.

The correct way to implement data parallelism for recurrent neural networks is as follows:

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class MyModule(nn.Module):
    # ... __init__, other methods, etc.

    # padded_input is of shape [B x T x *] (batch_first mode) and contains
    # the sequences sorted by lengths
    #   B is the batch size
    #   T is max sequence length
    def forward(self, padded_input, input_lengths):
        total_length = padded_input.size(1)  # get the max sequence length
        packed_input = pack_padded_sequence(padded_input, input_lengths,
                                            batch_first=True)
        packed_output, _ = self.my_lstm(packed_input)
        output, _ = pad_packed_sequence(packed_output, batch_first=True,
                                        total_length=total_length)
        return output

m = MyModule().cuda()
dp_m = nn.DataParallel(m)

You also need to set the CUDA_VISIBLE_DEVICES environment variable accordingly for a multi GPU setup.

References:

Data Parallelism
Fast.ai Forums
RNNs and Data Parallelism


Instruction: How does PyTorch handle labels when loading image/mask files for image segmentation?

Input:
I am starting an image segmentation project using PyTorch. I have a reduced dataset in a folder and 2 subfolders - "image" to store the images and "mask" for the masked images. Images and masks are .png files with 3 channels and 256x256 pixels. Because it is image segmentation, the labelling has to be performed a pixel by pixel. I am working only with 2 classes at the moment for simplicity. So far, I achieved the following:

I was able to load my files into classes "images" or "masks" by

root_dir="./images_masks"
train_ds_untransf = torchvision.datasets.ImageFolder(root=root_dir)
train_ds_untransf.classes
Out[621]:
['images', 'masks']

and transform the data into tensors

from torchvision import transforms
train_trans = transforms.Compose([transforms.ToTensor()])
train_dataset = torchvision.datasets.ImageFolder(root=root_dir,transform=train_trans)

Each tensor in this "train_dataset" has the following shape:

train_dataset[1][0].shape
torch.Size([3, 256, 256])

Now I need to feed the loaded data into the CNN model, and have explored the PyTorch DataLoader for this

train_dataloaded = DataLoader(train_dataset, batch_size=2, shuffle=False, num_workers=4)

I use the following code to check the resulting tensor's shape

for x, y in train_dl:
    print (x.shape)
    print (y.shape)
    print(y)

and get

torch.Size([2, 3, 256, 256])
torch.Size([2])
tensor([0, 0])
torch.Size([2, 3, 256, 256])
torch.Size([2])
tensor([0, 1])
.
.
.

Shapes seem correct. However, the first problem is that I got tensors of the same folder, indicated by some "y" tensors with the same value [0, 0]. I would expect that they all are [1, 0]: 1 representing image, 0 representing masks.

The second problem is that, although the documentation is clear when labels are entire images, it is not clear as to how to apply it for labeling at the pixel level, and I am certain the labels are not correct.

What would be an alternative to correctly label this dataset?

thank you

Output:
The class torchvision.datasets.ImageFolder is designed for image classification problems, and not for segmentation; therefore, it expects a single integer label per image and the label is determined by the subfolder in which the images are stored. So, as far as your dataloader concern you have two classes of images "images" and "masks" and your net tries to distinguish between them.

What you actually need is a different implementation of dataset that for each __getitem__ return an image and the corresponding mask. You can see examples of such classes here.

Additionally, it is a bit weird that your binary pixel-wise labels are stored as 3 channel image. Segmentation masks are usually stored as a single channel image.


Instruction: Pytorch - Efficient Elementwise Multiply?

Input:
I have a tensor of 3D Points of [100x3]

I have a vector of weights of [100x1], which needs to be element wise multiplied into the X,Y,Z coordinates.

Currently, I am creating a new vector W where I stack the [100x3] element with repetition into a [100x3] tensor, before i do an element wise multiply.

I need to do this many times, and this is way too slow and memory intensive. Is there a better way?

Output:
Standard multiplication (*) in PyTorch already is elementwise. Additionally, it broadcasts. So

import torch
xyz = torch.randn(100, 3)
w = torch.randn(100, 1)
multiplied = xyz * w

will just do the trick.


Instruction: How to Save pytorch tensor in append mode

Input:
How to save several tensor appending using torch.save()?

For example

for i in range(20):
       ......
    loss = criterion(scores, labels)
    torch.save(loss,'loss.pt')

How to save these all 20 losses?

Output:
It's probably not possible to directly append to the file, at least, I could not find documentation for this. In your example, however, a better approach is to append to a list, and save at the end.

import torch

losses = []
for i in range(20):
    #   ......
    loss = criterion(scores, labels)
    losses.append(loss.item())

torch.save(losses, 'loss.pt')


Instruction: How to create a custom PyTorch dataset when the order and the total number of training samples is not known in advance?

Input:
I have a 42 GB jsonl file. Every element of this file is a json object. I create training samples from every json object. But the number of training samples from every json object that I extract can vary between 0 to 5 samples. What is the best way to create a custom PyTorch dataset without reading the entire jsonl file in memory?

This is the dataset I am talking about - Google Natural Questions.

Output:
You have a couple of options.

The simplest option, if having lots of small files is not a problem, is to preprocess each json object into a single file. Then you can just read each one depending on the index requested. E.g

       class SingleFileDataset(Dataset):
        def __init__(self, list_of_file_paths):
            self.list_of_file_paths = list_of_file_paths

        def __getitem__(self, index):
            return np.load(self.list_of_file_paths[index]) # Or equivalent reading code for single file

You can also split the data into a constant number of files, and then calculate, given the index, which file the sample resides in. Then you need to open that file into memory and read the appropriate index. This gives a trade-off between disk access and memory usage. Assume you have n samples, and we split the samples into c files evenly during preprocessing. Now, to read the sample at index i we would do

       class SplitIntoFilesDataset(Dataset):
        def __init__(self, list_of_file_paths, n_splits):
            self.list_of_file_paths = list_of_file_paths
            self.n_splits = n_splits

        def __getitem__(self, index):
            # index // n_splits is the relevant file, and
            # index % len(self) is the index in in that file
            file_to_load = self.list_of_file_paths[index // self.n_splits]
            # Load file
            file = np.load(file)
            datapoint = file[index % len(self)]

Finally, you could use a HDF5 file that allows access to rows on disk. This is possibly the best solution if you have a lot of data, since the data will be close on disk. There's an implementation here which I have copy pasted below:

import h5py
import torch
import torch.utils.data as data
class H5Dataset(data.Dataset):

    def __init__(self, file_path):
        super(H5Dataset, self).__init__()
        h5_file = h5py.File(file_path)
        self.data = h5_file.get('data')
        self.target = h5_file.get('label')

    def __getitem__(self, index):
        return (torch.from_numpy(self.data[index,:,:,:]).float(),
                torch.from_numpy(self.target[index,:,:,:]).float())

    def __len__(self):
        return self.data.shape[0]


Instruction: What does log_prob do?

Input:
In some (e.g. machine learning) libraries, we can find log_prob function. What does it do and how is it different from taking just regular log?

For example, what is the purpose of this code:

dist = Normal(mean, std)
sample = dist.sample()
logprob = dist.log_prob(sample)

And subsequently, why would we first take a log and then exponentiate the resulting value instead of just evaluating it directly:

prob = torch.exp(dist.log_prob(sample))

Output:
As your own answer mentions, log_prob returns the logarithm of the density or probability. Here I will address the remaining points in your question:

How is that different from log? Distributions do not have a method log. If they did, the closest possible interpretation would indeed be something like log_prob but it would not be a very precise name since if begs the question "log of what"? A distribution has multiple numeric properties (for example its mean, variance, etc) and the probability or density is just one of them, so the name would be ambiguous.

The same does not apply to the Tensor.log() method (which may be what you had in mind) because Tensor is itself a mathematical quantity we can take the log of.

Why take the log of a probability only to exponentiate it later? You may not need to exponentiate it later. For example, if you have the logs of probabilities p and q, then you can directly compute log(p * q) as log(p) + log(q), avoiding intermediate exponentiations. This is more numerically stable (avoiding underflow) because probabilities may become very close to zero while their logs do not. Addition is also more efficient than multiplication in general, and its derivative is simpler. There is a good article about those topics at https://en.wikipedia.org/wiki/Log_probability.


Instruction: How to load a checkpoint file in a pytorch model?

Input:
In my pytorch model, I'm initializing my model and optimizer like this.

model = MyModelClass(config, shape, x_tr_mean, x_tr,std)
optimizer = optim.SGD(model.parameters(), lr=config.learning_rate)

And here is the path to my checkpoint file.

checkpoint_file = os.path.join(config.save_dir, "checkpoint.pth")

To load this checkpoint file, I check and see if the checkpoint file exists and then I load it as well as the model and optimizer.

if os.path.exists(checkpoint_file):
    if config.resume:
        torch.load(checkpoint_file)
        model.load_state_dict(torch.load(checkpoint_file))
        optimizer.load_state_dict(torch.load(checkpoint_file))

Also, here's how I'm saving my model and optimizer.

 torch.save({'model': model.state_dict(), 'optimizer': optimizer.state_dict(), 'iter_idx': iter_idx, 'best_va_acc': best_va_acc}, checkpoint_file)

For some reason I keep getting a strange error whenever I run this code.

model.load_state_dict(torch.load(checkpoint_file))
File "/home/Josh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for MyModelClass:
        Missing key(s) in state_dict: "mean", "std", "attribute.weight", "attribute.bias".
        Unexpected key(s) in state_dict: "model", "optimizer", "iter_idx", "best_va_acc"

Does anyone know why I'm getting this error?

Output:
You saved the model parameters in a dictionary. You're supposed to use the keys, that you used while saving earlier, to load the model checkpoint and state_dicts like this:

if os.path.exists(checkpoint_file):
    if config.resume:
        checkpoint = torch.load(checkpoint_file)
        model.load_state_dict(checkpoint['model'])
        optimizer.load_state_dict(checkpoint['optimizer'])

You can check the official tutorial on PyTorch website for more info.


Instruction: Apply a PyTorch CrossEntropy method for multiclass segmentation

Input:
I am trying to implement a simple example of how to apply cross-entropy to what is supposed to be the output of my semantic segmentation CNN.

Using the pytorch format I would have something like this:

out = np.array([[
    [
        [1.,1, 1],
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [1, 1, 1],
        [0, 0.,0],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [0, 0, 0],
        [1, 1, 1],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0],
        [1, 1, 1]
    ]
]])

out = torch.tensor(out)

So, my output here has dimensions (1, 4, 4, 3), being 1 element batch, 4 channels representing the 4 possible classes, and 4 by 3 data in each, storing the probability of that cell being from it's class.

Now my target is like this:

target=[
        [0, 0, 0],
        [1, 1, 1],
        [2, 2, 2],
        [3, 3, 3]
    ]

Please notice how in the 'out' tensor each row has a 1.0 probability of being from that class resulting in a perfect match with the target.

For example, the third channel (channel 2) has its whole 3rd row (row 2) with 1.0 probabilities of being from that channel, and zero's in any other place; so it matches the 2's on the target in third row as well.

With this example I expect a minimal loss value between the two tensors.

My question are:

What's the best way to use a cross-entropy loss method in PyTorch in order to reflect that this case has no difference between the target and its prediction?
What loss value should I expect from this?

This is what I got so far:

import torch
from torch.nn import CrossEntropyLoss
import numpy as np

out = torch.Tensor(np.array([[
    [
        [1.,1, 1],
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [1, 1, 1],
        [0, 0.,0],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [0, 0, 0],
        [1, 1, 1],
        [0, 0, 0]
    ],
    [
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0],
        [1, 1, 1]
    ]
]]))

target = torch.Tensor([[
    [0, 0, 0],
    [1, 1, 1],
    [2, 2, 2],
    [3, 3, 3]
]]).type('torch.LongTensor')

criterion = CrossEntropyLoss()
print(criterion(out, target))

And outputs: tensor(0.7437)

Shouldn't I expect a value closer to cero?

Thank you in advance

Output: Look at the description of nn.CrossEntropyLoss function, the prediction out you provide to nn.CrossEntropyLoss are not treated as class probabilities, but rather as logits; The loss function derive the class probabilities from out using soft max therefore nn.CrossEntropyLoss will never output exactly zero loss.


Instruction: Optional tensors in PyTorch c++ extension

Input:
I'm writing  a C++ extension for pytorch, and using the c++ api to do so. To my forward function, I need to pass an optional tensor. Inside the function, I want to do different things based on whether this optional parameter was passed or not. In general, we use NULL for optional pointer arguments in C++ and check inside the function if the pointer is NULL or not. I don't know how to do this for the at::Tensor type of Torch's c++ api.

void xyz_forward(
    const at::Tensor xyz1,
    const at::Tensor xyz2,
    const at::Tensor optional_constraints = something)
{
     if(optional_constraints){
        //do something
     }else{
        //do something else
     }
}

Note that, I can't do const at::Tensor optional_constraints = at::ones or something, because that parameter can take any real value and can be of varying size/shape. I can't assign it a numerical value as an optional argument. Is there a NULL equivalent for this?

Output: One possibility could be to use std::optional as std::optional<at::Tensor> optional_constraints = std::nullopt. It is contextually convertible to bool, so you can check it with if (optional_constraints). Use the .value() method to get the tensor if you pass one, otherwise the default value will be std::nullopt.


Instruction: Can anyone solve this?

Input:
torch.tensor(2,require_grad=True)

      Traceback (most recent call last):

      File "", line 1, in
       torch.tensor(2,require_grad=True)

      TypeError: tensor() got an unexpected keyword argument 'require_grad'

Output:
It's actually:

  requires_grad


Instruction: How does one determine when the CartPole environment has been solved?

Input:
I was going through this tutorial and saw the following piece of code:

        # Calculate score to determine when the environment has been solved
        scores.append(time)
        mean_score = np.mean(scores[-100:])

        if episode % 50 == 0:
            print('Episode {}\tAverage length (last 100 episodes): {:.2f}'.format(
                episode, mean_score))

        if mean_score > env.spec.reward_threshold:
            print("Solved after {} episodes! Running average is now {}. Last episode ran to {} time steps."
                  .format(episode, mean_score, time))
            break

however, it didn't really made sense to me. How does one define when a "RL environment has been solved"? Not sure what that even means. I guess in classification it would make sense to define it to be when loss is zero. In regression maybe when the total l2 loss is less than some value? Perhaps it would have made sense to define it when the expected returns (discounted rewards) is greater than some value.

But here it seems they are counting the # of time steps? This doesn't make any sense to me.

Note the original tutorial had this:

def main(episodes):
    running_reward = 10
    for episode in range(episodes):
        state = env.reset() # Reset environment and record the starting state
        done = False

        for time in range(1000):
            action = select_action(state)
            # Step through environment using chosen action
            state, reward, done, _ = env.step(action.data[0])
# Save reward
            policy.reward_episode.append(reward)
            if done:
                break

        # Used to determine when the environment is solved.
        running_reward = (running_reward * 0.99) + (time * 0.01)
update_policy()
if episode % 50 == 0:
            print('Episode {}\tLast length: {:5d}\tAverage length: {:.2f}'.format(episode, time, running_reward))
if running_reward > env.spec.reward_threshold:
            print("Solved! Running reward is now {} and the last episode runs to {} time steps!".format(running_reward, time))
            break

not sure if this makes much more sense...

is this only a particular quirk of this environment/task? How does the task end in general?

Output:
The time used in case of cartpole equals the reward of the episode. The longer you balance the pole the higher the score, stopping at some maximum time value.

So the episode would be considered solved if the running average of the last episodes is near enough that maximum time.


Instruction: Calling super's forward() method

Input:
What is the most appropriate way to call the forward() method of a parent Module? For example, if I subclass the nn.Linear module, I might do the following

class LinearWithOtherStuff(nn.Linear):
    def forward(self, x):
        y = super(Linear, self).forward(x)
        z = do_other_stuff(y)
        return z

However, the docs say not to call the forward() method directly:

  Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

which makes me think super(Linear, self).forward(x) could result in some unexpected errors. Is this true or am I misunderstanding inheritance?

Output:
TLDR;
You can use super().forward(...) freely even with hooks and even with hooks registered in super() instance.
Explanation
As stated by this answer __call__ is here so the registered hooks (e.g. register_forward_hook) will be run.
If you inherit and want to reuse base class's forward, e.g. this:
import torch

class Parent(torch.nn.Module):
    def forward(self, tensor):
        return tensor + 1

class Child(Parent):
    def forward(self, tensor):
        return super(Child, self).forward(tensor) + 1

module = Child()
# Increment output by 1 so we should get `4`
module.register_forward_hook(lambda module, input, output: output + 1)
print(module(torch.tensor(1))) # and it is 4 indeed
print(module.forward(torch.tensor(1))) # here it is 3 still

You are perfectly fine if you call __call__ method, forward won't run the hook (so you get 3 as above).
It is unlikely you would like to register_hook on the instance of super , but let's consider such example:
def increment_by_one(module, input, output):
    return output + 1

class Parent(torch.nn.Module):
    def forward(self, tensor):
        return tensor + 1

class Child(Parent):
    def forward(self, tensor):
        # Increment by `1` from Parent
        super().register_forward_hook(increment_by_one)
        return super().forward(tensor) + 1

module = Child()
# Increment output by 1 so we should get `5` in total
module.register_forward_hook(increment_by_one)
print(module(torch.tensor(1)))  # and it is 5 indeed
print(module.forward(torch.tensor(1)))  # here is 3

You are perfectly fine using super().forward(...) and even hooks will work correctly (and that is the main idea of using __call__ instead of forward).
BTW.  Calling super().__call__(...) would raise InifiniteRecursion error.


Instruction: How to work with large dataset in pytorch

Input:
I have a huge dataset that does not fit in memory (150G) and I'm looking for the best way to work with it in pytorch. The dataset is composed of several .npz files of 10k samples each. I tried to build a Dataset class

class MyDataset(Dataset):
    def __init__(self, path):
        self.path = path
        self.files = os.listdir(self.path)
        self.file_length = {}
        for f in self.files:
            # Load file in as a nmap
            d = np.load(os.path.join(self.path, f), mmap_mode='r')
            self.file_length[f] = len(d['y'])

    def __len__(self):
        raise NotImplementedException()

    def __getitem__(self, idx):
        # Find the file where idx belongs to
        count = 0
        f_key = ''
        local_idx = 0
        for k in self.file_length:
            if count < idx < count + self.file_length[k]:
                f_key = k
                local_idx = idx - count
                break
            else:
                count += self.file_length[k]
        # Open file as numpy.memmap
        d = np.load(os.path.join(self.path, f_key), mmap_mode='r')
        # Actually fetch the data
        X = np.expand_dims(d['X'][local_idx], axis=1)
        y = np.expand_dims((d['y'][local_idx] == 2).astype(np.float32), axis=1)
        return X, y

but when a sample is actually fetched, it takes more than 30s. It looks like the entire .npz is opened, stocked in RAM and it accessed the right index.
How to be more efficient ?

EDIT

It appears to be a misunderstading of .npz files see post, but is there a better approach ?

SOLUTION PROPOSAL

As proposed by @covariantmonkey, lmdb can be a good choice. For now, as the problem comes from .npz files and not memmap, I remodelled my dataset by splitting .npz packages files into several .npy files. I can now use the same logic where memmap makes all sense and is really fast (several ms to load a sample).

Output:
How large are the individual .npz files? I was in similar predicament a month ago. Various forum posts, google searches later I went the lmdb route. Here is what I did

Chunk the large dataset into small enough files that I can fit in gpu — each of them is essentially my minibatch. I did not optimize for load time at this stage just memory.
create an lmdb index with key = filename and data = np.savez_compressed(stff)

lmdb takes care of the mmap for you and insanely fast to load.

Regards,
A

PS: savez_compessed requires a byte object so you can do something like

output = io.BytesIO()
np.savez_compressed(output, x=your_np_data)
#cache output in lmdb


Instruction: Espresso ANERuntimeEngine Program Inference overflow

Input:
I have two CoreML models. One works fine, and the other generates this error message:

[espresso] [Espresso::ANERuntimeEngine::__forward_segment 0] evaluate[RealTime]WithModel returned 0; code=5 err=Error Domain=com.apple.appleneuralengine Code=5 "processRequest:qos:qIndex:error:: 0x3: Program Inference overflow" UserInfo={NSLocalizedDescription=processRequest:qos:qIndex:error:: 0x3: Program Inference overflow}
[espresso] [Espresso::overflow_error] /var/containers/Bundle/Application/E0DE5E08-D2C6-48AF-91B2-B42BA7877E7E/xxx demoapp.app/mpii-hg128.mlmodelc/model.espresso.net:0

Both models are very similar, (Conv2D models). There are generated with the same scripts and versions of PyTorch, ONNX, and onnx-coreml. The model that works has 1036 layers, and the model that generates the error has 599 layers. They both use standard layers - Conv2D, BatchNorm, ReLU, MaxPool, and Upsample (no custom layers and no Functional or Numpy stuff). They both use relatively the same number of features per layer. They follow essentially the same structure, except the erroring model skips a maxpool layer at the start (hence the higher output resolution).

They both take a 256x256 color image as input, and output 16 channels at (working) 64x64 and (erroring) 128x128 pixels.

The app does not crash, but gives garbage results for the erroring model.

Both models train, evaluate, etc. fine in their native formats (PyTorch).

I have no idea what a Code=5 "processRequest:qos:qIndex:error:: 0x3: Program Inference overflow" error is, and google searches are not yielding anything productive, as I gather "Espresso" and "ANERuntimeEngine" are both private Apple Libraries.

What is this error message telling me? How can I fix it?

Can I avoid this error message by not running the model on the bionic chip but on the CPU/GPU?

Any help is appreciated, thanks.

Output:
That's a LOT of layers!

Espresso is the C++ library that runs the Core ML models. ANERuntimeEngine is used with the Apple Neural Engine chip.

By passing in an MLModelConfiguration with computeUnits set to .cpuAndGPU when you load the Core ML model, you can tell Core ML to not use the Neural Engine.


Instruction: Correctly converting a NumPy array to a PyTorch tensor running on the gpu

Input:
I have created a DataLoader that looks like this

class ToTensor(object):
    def __call__(self, sample):
        return torch.from_numpy(sample).to(device)

class MyDataset(Dataset):
    def __init__(self, data, transform=None):
        self.data = data
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        sample = self.data[idx, :]

        if self.transform:
            sample = self.transform(sample)

        return sample

I am using this data loader like so

dataset = MLBDataset(
        data=data,
        transform=transforms.Compose([
            ToTensor()
        ]))
dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
dataiter = iter(dataloader)
x = dataiter.next()

This fails with the message

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
...
    torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp:55

For the return command inside ToTensor(), in fact any attempt to move the tensor te the GPU will fail inside that class. I have tried:

a = np.array([[[1, 2, 3, 4], [5, 6, 7, 8], [25, 26, 27, 28]],
             [[11, 12, np.nan, 14], [15, 16, 17, 18], [35, 36, 37, 38]]])
print(torch.from_numpy(a).to(device))

inside the body of __call__ in ToTensor() and it fails with the same message, whereas it succeeds everywhere else.

Why is this error generated and how can I resolve this?

Output:
Try this one:

Code:

import numpy as np
import torch
import torch.nn as nn

torch.cuda.set_device(0)

X = np.ones((1, 10), dtype=np.float32)
print(type(X), X)
X = torch.from_numpy(X).cuda(0)
print(type(X), X)

model = nn.Linear(10, 10).cuda(0)
Y = model(X)
print(type(Y), Y)

Output:

<class 'numpy.ndarray'> [[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]]
<class 'torch.Tensor'> tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], device='cuda:0')
<class 'torch.Tensor'> tensor([[ 0.4867, -1.0050,  0.4872, -0.0260, -0.0788,  0.0161,  1.2210, -0.3957,
          0.2097,  0.2296]], device='cuda:0', grad_fn=<AddmmBackward>)


Instruction: The strange loss fluctuation when loading previous trained model

Input:
I am using PyTorch for deep learning now.

I trained a model before and had the parameters saved. Loss values before the end of the training were about 0.003~0.006.

However, when I load the same model with the same training data, loss values at first fluctuate to around 0.5.

The loss values then decrease very quickly to around 0.01 in ~10 iterations and now decreasing slowly.

Does anyone know why this situation keep happens? Since I am loading the same model/ training data. I was expecting the loss values would start at a similar level as the end of last training.

Output:
When resuming a training, you should not only load the network's weights but also the optimizer state. For that, you can use torch.save:

torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)

Then, for resuming the training:

model = TheModelClass(*args, **kwargs)
model.train()
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

If you don't save the optimizer state, you loss important information such as the current learning rate, momentum etc. This is probably the cause of your problem.

Reference:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training


Instruction: PyTorch why does the forward function run multiple times and can I change the input shape?

Input:
import torch
import torch.nn as nn
import torchvision.datasets as dsets
from skimage import transform
import torchvision.transforms as transforms
from torch.autograd import Variable
import pandas as pd;
import numpy as np;
from torch.utils.data import Dataset, DataLoader
import statistics

import random
import math

class FashionMNISTDataset(Dataset):
  '''Fashion MNIST Dataset'''
  def __init__(self, csv_file, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file
        transform (callable): Optional transform to apply to sample
    """

    data = pd.read_csv(csv_file)
    self.X = np.array(data.iloc[:, 1:]).reshape(-1, 1, 28, 28)
    self.Y = np.array(data.iloc[:, 0])

    del data
    self.transform = transform

  def __len__(self):
    return len(self.X)

  def __getitem__(self, idx):
    item = self.X[idx]
    label = self.Y[idx]

    if self.transform:
        item = self.transform(item)

    return (item, label)

class CNN(nn.Module):
  def __init__(self):
    super(CNN, self).__init__()
    self.layer1 = nn.Sequential(
      nn.Linear(616,300),
      nn.ReLU())
    self.layer2 = nn.Sequential(
      nn.Linear(300,100),
      nn.ReLU())
    self.fc = nn.Linear(100, 10)

  def forward(self, x):
    print("x shape",x.shape)
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.fc(out)
    return out

def run():

  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  num_epochs = 15
  batch_size = 100
  learning_rate = 0.0001

  train_dataset = FashionMNISTDataset(csv_file='fashion-mnist_train.csv')
  test_dataset = FashionMNISTDataset(csv_file='fashion-mnist_test.csv')

  train_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=batch_size,shuffle=True)
  test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=True)

  #instance of the Conv Net
  cnn = CNN()
  cnn.to(device)

  #loss function and optimizer
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)

  losses = []
  for epoch in range(num_epochs):
    l = 0
    for i, (images, labels) in enumerate(train_loader):

      images = Variable(images.float())
      labels = Variable(labels)
      #print(images[0])
      images = images.to(device)
      labels = labels.to(device)
      print("img shape=",images.shape, "label shape=",labels.shape)
      images = images.resize_((100,616))

      print("img shape=",images.shape, "label shape=",labels.shape)
      # Forward + Backward + Optimize
      optimizer.zero_grad()
      outputs = cnn(images)

      loss = criterion(outputs, labels)
      #print(loss)
      loss.backward()
      optimizer.step()
      #print(loss.item())
      losses.append(loss.item())
      l = loss.item()

      cnn.eval()
      with torch.no_grad():
        val_loss = []
        for images, labels in test_loader:
          images = Variable(images.float()).to(device)
          labels = labels.to(device)
          outputs = cnn.forward(images)
          batch_loss = criterion(outputs, labels)
          val_loss.append(batch_loss.item())
        avgloss = statistics.mean(val_loss)
        if avgloss < min(losses):
          torch.save(cnn.state_dict(), 'model')
        cnn.train()
      if (i+1) % 100 == 0:
        print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'
              %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.item()))
    print(l)

  final_model = CNN()
  final_model.load_state_dict(torch.load('model'))
  final_model.eval()
  correct = 0
  total = 0
  for images, labels in test_loader:
      images = Variable(images.float()).to(device)
      outputs = final_model(images).to(device)
      labels.to(device)
      _, predicted = torch.max(outputs.data, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum()
  print('Test Accuracy of the model on the 10000 test images: %.4f %%' % (100 * correct / total))

if __name__ == '__main__':
  run()

I have enclosed all the code for testing purposes. But Here is the error I get

  img shape= torch.Size([100, 1, 28, 28]) label shape= torch.Size([100])
  img shape= torch.Size([100, 616]) label shape= torch.Size([100]) x
  shape torch.Size([100, 616]) x shape torch.Size([100, 1, 28, 28])
  Traceback (most recent call last):   File "test.py", line 145, in

      run()   File "test.py", line 115, in run
      outputs = cnn.forward(images)   File "test.py", line 56, in forward
      out = self.layer1(x)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 489, in call
      result = self.forward(*input, **kwargs)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/container.py",
  line 92, in forward
      input = module(input)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 489, in call
      result = self.forward(*input, **kwargs)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/linear.py",
  line 67, in forward
      return F.linear(input, self.weight, self.bias)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/functional.py",
  line 1354, in linear
      output = input.matmul(weight.t()) RuntimeError: size mismatch, m1: [2800 x 28], m2: [616 x 300] at
  /opt/conda/conda-bld/pytorch_1549630534704/work/aten/src/THC/generic/THCTensorMathBlas.cu:266

The problem here is that I want all 616 pixels to feed as input into the neural network but I dont know how to do so. I tried to reshape the input to solve the problem but it ran model.forward twice, once with the correct shape and then the wrong shape.

Output:
You are calling forward twice in run:

Once for the training data
Once for the validation data

However, you do not appear to have applied the following transformation to your validation data:

images = images.resize_((100,616))

Maybe consider doing the resize in the forward function.


Instruction: Load Pre-trained model in Pytorch

Input:
I am currently working on GANS, I have downloaded the code and models from http://www.cs.columbia.edu/~vondrick/tinyvideo/, where I just need to run to get output. I have given paths and all correctly, and the code where i am receiving an error is shown below. The have wrote the lines correctly but still getting syntax error. Please help me!

import torch

import torch.legacy.nn

import torchfile

from skimage import io, transform

import torch.nn as nn

import torch.backends.cudnn as cudnn

cudnn.enabled = True

opt = {
  model = 'models/beach/iter63000_net.t7',
  batchSize = 128,
  gpu = 1,
  cudnn = 1,
}

Output:
you are trying to init a python dictionary. this is how you do it:

opt = {
  'model': 'models/beach/iter63000_net.t7',
  'batchSize': 128,
  'gpu':  1,
  'cudnn': 1,
}


Instruction: Pytorch: Set Block-Diagonal Matrix Efficiently?

Input:
I have a Tensor A of size [N x 3 x 3], and a Matrix B of size [N*3 x N*3]
I want to copy the contents of A -> B, so that the diagonal elements are filled up basically, and I want to do this efficiently:
It should kind of fill up B to look something filled like this:

So each [i,3,3] fills into each [3x3] part in B diagonally down the line.
How do I do this? As efficiently as possible as this is for a real time application. I could write a CUDA kernel to do this, but I would prefer to do it with some special Pytorch function

Output:
Use torch.block_diag():
# Setup
A = torch.ones(3,3,3, dtype=int)

# Unpack blocks and apply
B = torch.block_diag(*A)

>>> B
tensor([[1, 1, 1, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 1, 1, 1, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 1, 1, 1],
        [0, 0, 0, 0, 0, 0, 1, 1, 1],
        [0, 0, 0, 0, 0, 0, 1, 1, 1]])


Instruction: Input labels for semantic segmentation with U-Net as single layer

Input:
When doing semantic segmentation with the U-Net for example, it seems to be common-practice to provide the label data as one-hot-encoded tensors. In another SO question, a user pointed out that this is due to the labels usually representing categorical values. Feeding them to the network as class labels within only one layer (as greyscale intensity values) would introduce difficulties.

In another blog post however, the author explains that the labels

  "[...] sometimes [get] packaged as greyscale images, where the pixel intensity represents the class id [...]. This method can be the easiest to work with. It allows for a small file size for distribution and [...] One Hot Vector representations [use] up more memory than [the greyscale encoding format]."

My hardware is only very limited, and I am hoping that encoding the labels as 1-layered greyscale tensors, rather than n-layered (n being the number of classes to segment), will lead to lower memory usage. However, the author of the blog then also states:

  "Even if the deep learning framework you use accepts the labels data as class ids, as in [the greyscale format], it will convert that data to one-hot encoding behind the scenes."

Does this mean, there wouldn't be any savings memory-wise after all?

If it is worthwhile, how would I go on to implementing this in the dataset-reader? I also haven't encountered any implementation, where the greyscale labeling has in fact been practiced. I'd therefore also be thankful for any links to implementations that have been using greyscale labels for semantic segmentation!

I am working with PyTorch and my code is based on this implementation, with the difference that I have 3 classes to segment.

Any suggestions/links are greatly appreciated!

Output:
This can help you save disk memory as you will be able to store the labels, the ground truth, as a greyscale image (width, heigh, 1) and not as a bigger 3D tensor of shape (width, height, n). But, during the training process, you will have to convert the greyscale ground truth images to 3D tensor to be able to train your network. So It won't help you to reduce the RAM cost of the process.

If you really need to reduce the RAM usage, you can decrease the training batch size, or the image sizes.


Instruction: RuntimeError: _thnn_mse_loss_forward is not implemented for type torch.cuda.LongTensor

Input:
I'm using PyTorch,but I get a error!
My error code as following:

for train_data in trainloader:
    example_count += 1
    if example_count == 100:
        break
    optimer.zero_grad()
    image, label = train_data
    image = image.cuda()
    label = label.cuda()
    out = model(image)
    _, out = torch.max(out, 1)
    # print(out.cpu().data.numpy())
    # print(label.cpu().data.numpy())
    # out = torch.zeros(4, 10).scatter_(1, out.cpu(), 1).cuda()
    # label= torch.zeros(4, 10).scatter_(1, label.cpu(), 1).cuda()
    l = loss(out, label)
    l.bakeward()
    optimer.setp()
    j += 1
    count += label.size(0)
    acc += (out == label).sum().item()
    if j % 1000 == 0:
        print(j + ' step:curent accurity is %f' % (acc / count))

the traceback:

    Traceback (most recent call last):
  File "VGG实现.py", line 178, in <module>
    utils.train(testloader,model)
  File "VGG实现.py", line 153, in train
    l=loss(out,label)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 435, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2156, in mse_loss
    ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: _thnn_mse_loss_forward is not implemented for type torch.cuda.LongTensor

I get a answer whose here
Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

But I don't know how to solve this question.

Output:
Look at the documentation of torch.max():

torch.max(input, dim, keepdim=False, out=None) -> (Tensor, LongTensor)

  Returns the maximum value of each row of the input tensor in the given
  dimension dim. The second return value is the index location of each
  maximum value found (argmax).

Your line of code

_, out = torch.max(out, 1)

Takes the float prediction of the model, out, and uses torch.max() to return the argmax = type long int index of the maximal prediction.
The error message you get is that your loss function (you are using cross entropy with softmax I guess) does not support a long type first argument.
Moreover, you cannot take derivative through argmax - so I don't think converting out to float using .to(torch.float) is going to do you any good either.
The softmax function inside the loss function you are using is taking care of the argmax for you.


Instruction: RuntimeError: Expected hidden[0] size (2, 20, 256), got (2, 50, 256)

Input:
I get this error while trying to build a multiclass text classification network using LSTM (RNN). The code seems to run fine for the training part of the code whereas it throws the error for the validation part. Below is the network architecture and training code. Appreciate any help here.

I tried taking an existing code that predicts sentiment using RNN and replaced sigmoid with softmax function in the end and loss function from BCE Loss to NLLLoss()

  def forward(self, x, hidden):
    """
    Perform a forward pass of our model on some input and hidden state.
    """
    batch_size = x.size(0)
    embeds = self.embedding(x)
    lstm_out,hidden= self.lstm(embeds,hidden)

     # stack up lstm outputs
    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

    # dropout and fully-connected layer
    out = self.dropout(lstm_out)
    out = self.fc(out)

    # softmax function
    soft_out = self.sof(out)

    # reshape to be batch_size first
    soft_out = soft_out.view(batch_size, -1)
#         soft_out = soft_out[:, -1] # get last batch of labels

    # return last sigmoid output and hidden state
    return soft_out, hidden

def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data

    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

    return hidden

# Instantiate the model w/ hyperparams
vocab_size = len(vocab_to_int)+1
output_size = 44
embedding_dim = 100
hidden_dim = 256
n_layers = 2

net = ClassificationRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)

print(net)

# loss and optimization functions
lr=0.001

criterion = nn.NLLLoss()

optimizer = torch.optim.Adam(net.parameters(), lr=lr)

# training params

epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing

counter = 0
print_every = 100
clip=5 # gradient clipping

# move model to GPU, if available
if(train_on_gpu):
    net.cuda()

net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        if(train_on_gpu):
            inputs, labels = inputs.cuda(), labels.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

#         print('output:',output.squeeze())
#         print('labels:',labels.float())

        # calculate the loss and perform backprop
        loss = criterion(output, labels)
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])

                if(train_on_gpu):
                    inputs, labels = inputs.cuda(), labels.cuda()

                output, val_h = net(inputs, val_h)

                val_loss = criterion(output, labels)

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-41-805ed880b453> in <module>()
     58                     inputs, labels = inputs.cuda(), labels.cuda()
     59
---> 60                 output, val_h = net(inputs, val_h)
     61
     62                 val_loss = criterion(output, labels)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-38-dbfb8d384231> in forward(self, x, hidden)
     34         batch_size = x.size(0)
     35         embeds = self.embedding(x)
---> 36         lstm_out,hidden= self.lstm(embeds,hidden)
     37
     38          # stack up lstm outputs

Output:
Try adding drop_last=True in your line of codes that loads data using DataLoader,
for example for loading training data from data set train_data:
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)

Explanation:
The error may caused by your training data not being divisible by batch size. Suppose your training data has 130 items and batch size is 8, the last batch will have only 2 (remainder of 130/8) items. Therefore by setting drop_last to True, the 2 items will be ignored.


Instruction: Choosing a margin for contrastive loss in a siamese network

Input:
I'm building a siamese network for a metric-learning task, using a contrastive loss function, and I'm uncertain on how to set the 'margin' hyperparameter for the loss.

My inputs to the loss function are currently 1024-dimension dense embeddings from an RNN layer - Does the dimensionality of that input affect how I pick a margin? Should I use a dense layer to project it to a lower-dimensional space first? Any pointers on how to pick a specific margin value (or any relevant research) would be really appreciated! In case it matters, I'm using PyTorch.

Output:
You don't need to project it to a lower dimensional space.
The dependence of the margin with the dimensionality of the space depends on how the loss is formulated: If you don't normalize the embedding values and compute a global difference between vectors, the right margin will depend on the dimensionality. But if you compute a normalize difference, such as cosine distance, the margin values won't depend on the dimensionality of the embedding space.
Here ranking (or contrastive) losses are explained, it might be useful https://gombru.github.io/2019/04/03/ranking_loss/


Instruction: Mismatching Conda and Pycharm

Input:
I'm new in python and I'm confused about  mismatching Conda packages list and Pycharm. In a project I need to install pytorch. Installing with pycharm lead to some error and when I install it through conda, It does not appear in pycharm. Both list is the same env.

Thanks in advance.

pycharm list

Anaconda list

Output: PyCharm shows you the list of installed packages with pip, while conda list shows both pip and conda. Meanwhile, you can switch between pip and conda with a dedicated button in PyCharm:


Instruction: Fastai learner not loading

Input:
So I'm trying to load a model using:

learn = create_cnn(data, models.resnet50, lin_ftrs=[2048], metrics=accuracy)
learn.clip_grad();
learn.load(f'{name}-stage-2.1')

But I get the following error

RuntimeError: Error(s) in loading state_dict for Sequential:
size mismatch for 1.8.weight: copying a param with shape torch.Size([5004, 2048]) from checkpoint, the shape in current model is torch.Size([4542, 2048]).
size mismatch for 1.8.bias: copying a param with shape torch.Size([5004]) from checkpoint, the shape in current model is torch.Size([4542]).

The only thing that is different thing is that I added a random validation split that wasn't there in the stage-2.1 model, when I remove the split and have no validation set as the stage-2.1 was trained all goes well.

Whats happening?

Output:
Use cnn_learner method and latest Pytorch with latest FastAI. There was a breaking change and discontinuity so you suffer now.

The fastai website has many examples such as this one.

learn = cnn_learner(data, models.resnet50, metrics=accuracy)


Instruction: How to convert a Label matrix to colour matrix for image segmentation?

Input:
I have a label matrix of 256*256 for example. And the classes are 0-11 so 12 classes. I want to convert the label matrix to colour matrix. I tried do it in a code like this

`for i in range(256):
    for j in range(256):
        if x[i][j] == 11:
            dummy[i][j] = [255,255,255]
        if x[i][j] == 1:
            dummy[i][j] = [144,0,0]
        if x[i][j] == 2:
            dummy[i][j] = [0,255,0]
        if x[i][j] == 3:
            dummy[i][j] = [0,0,255]
        if x[i][j] == 4:
            dummy[i][j] = [144,255,0]
        if x[i][j] == 5:
            dummy[i][j] = [144,0,255]
        if x[i][j] == 6:
            dummy[i][j] = [0,255,255]
        if x[i][j] == 7:
            dummy[i][j] = [122,0,0]
        if x[i][j] == 8:
            dummy[i][j] = [0,122,0]
        if x[i][j] == 9:
            dummy[i][j] = [0,0,122]
        if x[i][j] == 10:
            dummy[i][j] = [122,0,122]
        if x[i][j] == 11:
            dummy[i][j] = [122,122,0]
            `

It is highly inefficient. PS: the shape of x is [256 256] and dummy is [256 256 3]. Is there any better way to do it ?

Output:
You are looking into indexed RGB images - an RGB image where you have a fixed "pallet" of colors, each pixel indexes to one of the colors of the pallet. See this page for more information.

from PIL import Image

img = Image.fromarray(x, mode="P")
img.putpalette([
    255, 255, 255,   # index 0
    144, 0, 0, # index 1
    0, 255, 0, # index 2
    0, 0, 255, # index 3
    # ... and so on, you can take it from here.
])
img.show()


Instruction: Efficient way to implement matrix multiplication when one matrix is extremely wide?

Input:
I need to multiply 3 matrices, A: 3000x100, B: 100x100, C: 100x3.6MM.  I currently am just using normal matrix multiplication in PyTorch

A_gpu = torch.from_numpy(A)
B_gpu = torch.from_numpy(B)
C_gpu = torch.from_numpy(C)
D_gpu = (A_gpu @ B_gpu @ C_gpu.t()).t()

C is very wide so the data reuse on gpu is limited but are there other ways to speed this up? I have a machine with 4x GPUs.

Output:
Since you have four GPUs, you can harness them to perform efficient matrix multiplication. Notice however that the results of the multiplication has size 3000x3600000, which takes up 40GB in single precision floating point (fp32). Unless you have a large enough RAM for the CPU, you cannot store the results of this computation on the RAM.

A possible solution for this is to break up the large matrix C into four smaller chunks, perform the matrix multiplication of each chunk on a different GPU, and keep the result on the GPU. Provided that each GPU has at least 10GB of memory, you will have enough memory for this.

If you do have also enough CPU memory, you can then move the results of all four GPUs onto the CPU and concatenate them (in fact, in this case you could have used only a single GPU and transfer the results from GPU to CPU each time). Otherwise, you can keep the results chunked on the GPUs, and you need to remember and keep track that the four chunks are actually part of one matrix.

import numpy as np
import torch.nn as nn
import torch

number_of_gpus = 4

# create four matrics
A = np.random.normal(size=(3000,100))
B = np.random.normal(size=(100,100))
C = np.random.normal(size=(100,3600000))

# convert them to pytorch fp32 tensors
A = torch.from_numpy(A).float()
B = torch.from_numpy(B).float()
C = torch.from_numpy(C).float()

# calcualte `A@B`, which is easy
AB = A@B

# split the large matrix `C` into 4 smaller chunks along the second dimension.
# we assume here that the size of the second dimension of `C` is divisible by 4.
C_split = torch.split(C,C.shape[1]//number_of_gpus,dim=1)

# loop over the four GPUs, and perform the calculation on each using the corresponding chunk of `C`
D_split = []
for i in range(number_of_gpus):
    device = 'cuda:{:d}'.format(i)
    D_split.append( AB.to(device) @ C_split[i].to(device))

# DO THIS ONLY IF YOU HAVE ENOUGH CPU MEMORY!! :
D = torch.cat([d.cpu() for d in D_split],dim=1)