Instruction: Concatenating branches(of different dimensions) of googlenet on Pytorch

Input:
is there any way I can concatenate branches of different dimensions of a small inception(googlenet)?
For example, at a 32 x 32 x 3 image (torch.Size([1, 3, 32, 32])), it will pass through an inception module with the following branches:

a convolution with (32 channels, 1 x 1 filters)
another convolution with (32 channels, 3 x3 filters)
merge (concatenate along the channel dimension)

Inception Module
The issue however is that the torch sizes as a result of the two convolutions are different.
(32 channels, 1 x 1 filters) -> [1, 32, 30, 30]
(32 channels, 3 x3 filters) -> [1, 32, 28, 28]
How can I concatenate the two branches?
Should I add padding? I tried ZeroPad2d. It’s not working.
For your reference:
class Inception(nn.Module):
def __init__(self, in_channel, ch1, ch3):
    super(Inception, self).__init__()
    self.branch1 = nn.Sequential(
        ConvBlock(in_channel, ch1, kernel_size = 1,stride=1, padding=0)
    )

    self.branch2 = nn.Sequential(
        ConvBlock(in_channel, ch3, kernel_size = 3,stride=1, padding=0)
    )

def forward(self, x):
    branch1 = self.branch1(x)
    branch2 = self.branch2(x)
    return torch.cat([branch1, branch2], 1) *(error here)*

Output:
You need to match your kernel size and padding.
For kernel_size=1 no padding is needed, but for kernel_size=3 you need padding to be 1:
self.branch2 = nn.Sequential(
        ConvBlock(in_channel, ch3, kernel_size=3,stride=1, padding=1)
    )

See this nice tutorial for more details.


Instruction: How to make PyTorch DataLoader from scratch?

Input:
Is it possible to recreate a simple version of PyTorch DataLoader from scratch?
The class should be able to return current batch based on the batch size.
For example, the code bellow only allows me to return one example at the time
X = np.array([[1,2],[3,4],[5,6],[6,7]])

class DataLoader:
    def __init__(self, X, b_size):
        self.X = X
        self.b_size = b_size

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        return self.X[index]

But what I want to achieve is that if I specify b_size=2, it would return:
Iteration 0: [[1,2],[3,4]]
Iteration 1: [[5,6],[7,8]]

Is it possible to do something like that in Python? I can't use the DataLoader class.

Output:
X = np.array([[1,2],[3,4],[5,6],[6,7]])

class DataLoader:
    def __init__(self, X, b_size):
        self.X = X
        self.b_size = b_size

    def __len__(self):
        return len(self.X)//self.b_size

    def __getitem__(self, index):
        return self.X[index*self.b_size:index*self.b_size+self.b_size]

d = DataLoader(X, 2)
for i in range(len(d)):
  print (f"Iteration {i}: {d[i]}")

Output:
Iteration 0: [[1 2]
 [3 4]]
Iteration 1: [[5 6]
 [6 7]]


Instruction: Error Compiling C++/Cuda extension with Pytorch Cuda c++ in MSVC using CMake

Input:
I am trying to build a c++/cuda extension with Pytorch following the tutorial here, (with instructions how to use pytorch with c++ here). My environment details are:

Using Microsoft Visual Studio 2019 version 16.6.5
Windows 10
libtorch c++ debug 1.70 with cuda 11.0 installed from the pytorch website

I am using this cmake code where I set the include directory for python 3.6 and the library for python36.lib
cmake_minimum_required (VERSION 3.8)

project ("DAConvolution")

find_package(Torch REQUIRED)

# Add source to this project's executable.
add_executable (DAConvolution "DAConvolution.cpp" "DAConvolution.h")

include_directories("C:/Users/James/Anaconda3/envs/masters/include")

target_link_libraries(DAConvolution "${TORCH_LIBRARIES}" "C:/Users/James/Anaconda3/envs/masters/libs/python36.lib")

if (MSVC)
  file(GLOB TORCH_DLLS "${TORCH_INSTALL_PREFIX}/lib/*.dll")
  add_custom_command(TARGET DAConvolution
                     POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy_if_different
                     ${TORCH_DLLS}
                     $<TARGET_FILE_DIR:DAConvolution>)
endif (MSVC)

I set the CMake command arguments to be -DCMAKE_PREFIX_PATH=C:\libtorch  (my path to libtorch debug mentioned above). I am building with the x64-debug option in MSVC version (as building with the x-64 Release option gives me a torch-NOTFOUND error).
The example DAConvolution.cpp file is:

#ifdef _DEBUG
#undef _DEBUG
#include <python.h>
#define _DEBUG
#else
#include <python.h>
#endif

#include <torch/extension.h>

Where I have undefined the _DEBUG flag so that the linker does not look for the python36_d.lib file (which I do not have).
I am getting a linking error:

Simply including torch.h works fine, but when I want to include the extension header thats when I get these problems, as it uses Pybind 11 I believe. Any insights much appreciated. I have tried to include all the info I can, but would be happy to give more information.

Output:
For Windows and with Visual studio, you are better to work with the Visual Studio rather than the CMake.
Just create a simple Console Application, go to the project's Properties, change the Configuration type to Dynamic Library (dll), Configure the include and Library directories, add the required enteries to your linker in Linker>Input (such as torch.lib, torch_cpu.lib, etc) and you are good to go click build, and if you have done everything correctly you'll get yourself a dll that you can use (e.g loading it using torch.classes.load_library from Python and use it.
The Python debug version is not shipped with Anaconda/ normal python distribution, but if you install the Microsoft Python distribution which I believe can be downloaded/installed from Visual Studio installer, its available.
Also starting from Python 3.8 I guess the debug binaries are also shipped.
In case they are not, see this.
For the cmake part you can follow something like the following. This is a butchered version taken from my own cmake that I made for my python extension some time ago.
Read it and change it based on your own requirements it should be straight forward :
# NOTE:
# TORCH_LIB_DIRS needs to be set. When calling cmake you can specify them like this:
# cmake -DCMAKE_PREFIX_PATH="somewhere/libtorch/share/cmake" -DTORCH_LIB_DIRS="/somewhere/lib" ..

cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
project(DAConvolution)

find_package(Torch REQUIRED)
# we are using the C++17, if you are not change this or remove it altogether
set(CMAKE_CXX_STANDARD 17)

#define where your headers and libs are, specify for example where your DaConvolution.h resides!
include_directories( somewhere/Yourinclude_dir ${TORCH_INCLUDE_DIRS})

set(DAConvolution_SRC ./DAConvolution.cpp )

LINK_DIRECTORIES(${TORCH_LIB_DIRS})

add_library(
    DAConvolution
    SHARED
    ${DAConvolution_SRC}
  )

# if you use some custom libs, you previously built, specify its location here
# target_link_directories(DAConvolution PRIVATE somewhere/your_previously_built_stuff/libs)
target_link_libraries(DAConvolution ${TORCH_LIB_DIRS}/libc10.so)
target_link_libraries(DAConvolution ${TORCH_LIB_DIRS}/libtorch_cpu.so)

install(TARGETS DAConvolution LIBRARY DESTINATION lib )

Side note:
I made the cmake for Linux only, so under Windows, I always use Visual Studio (2019 to be exact), in the same way I explained earlier. its by far the best /easiest approach imho.  Suit yourself and choose either of them that best fits your problem.


Instruction: PyTorch Tutorial freeze_support() issue

Input:
I tried following the tutorial from PyTorch here: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py.
Full code is here:
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# Loading and normalizing CIFAR10
transform = transforms.Compose(
    [transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# Shows training images, DOESN'T WORK

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

# define a convolutional neural network
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

# Define a loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

# Train the network
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0

    # DOESN'T WORK
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

# save trained model
PATH = './cifar_net.pth'
torch.save(net.state_dict(), PATH)

# test the network on the test data
dataiter = iter(testloader)
images, labels = dataiter.next()

# print images
dataiter = iter(testloader)
images, labels = dataiter.next()
imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

# load back saved model
net = Net()
net.load_state_dict(torch.load(PATH))

# see what the nueral network thinks these examples above are:
ouputs = net(images)

# index of the highest energy
_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                              for j in range(4)))

# accuracy on the whole dataset
correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))

# classes that perfomed well vs classes that didn't perform well
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1

for i in range(10):
    print('Accuracy of %5s : %2d %%' % (
        classes[i], 100 * class_correct[i] / class_total[i]))

if __name__ == '__main__':
    torch.multiprocessing.freeze_support()

However I got this issue:
An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I'm just trying to run this in a regular python file. When I added
if __name__ == '__main__':
                freeze_support()

to the end of my file, I still get the error.

Output:
To anyone else with this issue, I believe you need to define a main function and run the training there. Then add:
if __name__ == '__main__':
    main()

at the end of the python file.
This fixed the freeze_support() issue for me on a different PyTorch training program.


Instruction: in Torchaudio=0.7.0, where is torchaudio.functional.istft?

Input:
I used torchaudio.functional.isftf when the version torchaudio=0.4.0
But now, the version of torchaudio has been upgraded to 0.7.0, so I installed it, but torchaudio.functional.istft is gone.
I need to learn a deep learning model using torchaudio.functional.istft.  How do I do it??

Output:
It's deprecated now in favor of torch.istft.
Check release notes and PR#523.


Instruction: Formula to compute the padding in convolutions pytorch (googlenet)

Input:
I am implementing googlenet (smaller version) from scratch in pytorch.  The architecture is below:

For the Downsample Module I have the following code:
   class DownSampleModule(nn.Module):
   def __init__(self, in_channel, ch3, w):
       super(DownSampleModule, self).__init__()
       kernel_size = 3
       padding = (kernel_size-1)/2

       self.branch1 = nn.Sequential(
           ConvBlock(in_channel, ch3, kernel_size = 3,stride=2, padding=int(padding))
       )
       self.branch2 = nn.Sequential(
           nn.MaxPool2d(3, stride=2, padding=0, ceil_mode=True)
       )
   def forward(self, x):
       branch1 = self.branch1(x)
       branch2 = self.branch2(x)

       return torch.cat([padded_tensor, branch2], 1)

The ConvBlock is from this module
class ConvBlock(nn.Module):
   def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
       super(ConvBlock, self).__init__()
       #padding = (kernel_size -1 )/2
       #print(padding)
       self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
       self.bn = nn.BatchNorm2d(out_channels)
       self.act = nn.ReLU()

   def forward(self, x):
       x = self.conv(x)
       x = self.bn(x)
       x = self.act(x)
       return x

Basically, we are creating two branches: a convolution module and a max pool.  The output of these two branches are then concatenated on the channel dimension.
However, I have the following problem:

Firstly, we call self.pool1 = DownSampleModule(in_channel=80, ch3 = 80, w=30).  The dimensions of the two branches are similar.  These are:

Downsample Convolution:torch.Size([1, 80, 15, 15])
Maxpool Convolution:torch.Size([1, 80, 15, 15])

However, when we call self.pool2 = DownSampleModule(in_channel = 144, ch3 = 96, w=15).  The dimensions are different which prevents it from being concatenated.

Downsample Convolution:torch.Size([1, 96, 8, 8])
Maxpool Convolution:torch.Size([1, 144, 7, 7])

Does anyone know the formula to compute the correct padding?  Thank you.
In Keras, you can just set the padding="same" or "valid" but it's not supported on pytorch.

Output:
Your maxpool and conv branches have the same input, and will produce identically-shaped output if you give them the same parameters for kernel size, stride and padding. So just replacing your padding = 0 with padding = int(padding) should be enough to make both branches be concat-compatible.
ceil_modeshould also be set to False. When the resulting dimension is not an integer, the rounding behavior of conv2d is to use floor, so you want your maxpool to do that as well.
By the way, you can remove your nn.Sequential. Your "sequences" of layers are made of only one layer, so... not really sequential :)


Instruction: How to build pytorch from source using numpy

Input:
I am trying to build pytorch v1.4.0 from source because I need it for another module.
I have succeeded in building pytorch from source but when I try to run the intended python script I get this error:
RuntimeError: PyTorch was compiled without NumPy support

So I looked up what I did wrong and it turns out I needed to install numpy before i build pytorch from source, so thats what I did.
I used the command:
pip3 install numpy==1.19.4

When installing pytorch the console gives me a lot of information, including the build settings. The build settings looked like this:
--   Compile definitions   : ONNX_ML=1;ONNX_NAMESPACE=onnx_torch;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /home/elvygcp/venv/lib/python3.6/site-packages;/usr/local/cuda
--   CMAKE_INSTALL_PREFIX  : /home/elvygcp/venv/pytorch-1.4.0/torch
--
--   TORCH_VERSION         : 1.4.0
--   CAFFE2_VERSION        : 1.4.0
--   BUILD_CAFFE2_MOBILE   : ON
--   USE_STATIC_DISPATCH   : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.6.9
--     Python executable   : /home/elvygcp/venv/bin/python3
--     Pythonlibs version  : 3.6.9
--     Python library      : /usr/lib/libpython3.6m.so.1.0
--     Python includes     : /usr/include/python3.6m
--     Python site-packages: lib/python3.6/site-packages
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   INTERN_BUILD_MOBILE   :
--   USE_ASAN              : OFF
--   USE_CUDA              : ON
--     CUDA static link    : OFF
--     USE_CUDNN           : OFF
--     CUDA version        : 10.2
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
--     cudart library      : /usr/local/cuda/lib64/libcudart.so
--     cublas library      : /usr/lib/x86_64-linux-gnu/libcublas.so
--     cufft library       : /usr/local/cuda/lib64/libcufft.so
--     curand library      : /usr/local/cuda/lib64/libcurand.so
--     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     CUDA host compiler  : /usr/bin/cc
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : ON
--   USE_FBGEMM            : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_MKL               : OFF
--   USE_MKLDNN            : ON
--   USE_MKLDNN_CBLAS      : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : OFF
--   USE_NNPACK            : ON
--   USE_NUMPY             : OFF
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : OFF
--     USE_GLOO            : ON
--   BUILD_NAMEDTENSOR   : OFF

There are 2 things I do not understand:

The line: USE_CUDNN           : OFF, which I think is strange since i followed the pytorch build instructions from their github page which mentions I need CuDNN to build pytorch from source, and CuDNN is installed on my system.

The line: USE_NUMPY             : OFF, I dont know why but pytorch does not recognize numpy.

My system:
OS: Ubuntu 18.04 LTS
Cuda version: 10.2
CuDNN version: 8
python venv in Google Cloud Compute Engine VM

How I build pytorch 1.4.0 from source:
git clone --branch v1.4.0 https://github.com/pytorch/pytorch.git pytorch-1.4.0
cd pytorch-1.4.0/
git submodule update --init --recursive
sudo apt install cmake -y
sudo apt-get update
cd ../
sudo apt install python3-venv -y
python3 -m venv venv/
cd venv
source bin/activate
cd pytorch-1.4.0/
pip install pyyaml
python3 setup.py install
cd ../
git clone --branch v0.5.0 https://github.com/pytorch/vision.git torchvision-0.5.0
cd torchvision-0.5.0/
python3 setup.py install
cd ../

If anyone could tell how I make pytorch recognize and build with numpy I'd be very grateful.
If I need to supply more information I'd be happy to.

Output:
Okay so I don't exactly know what the solution was since I did two things:

I installed a lower version of numpy since my torch version is also a bit older.
I cleared all the pytorch install cache with the command: sudo USE_ROCM=1 USE_LMDB=1 USE_OPENCV=1 MAX_JOBS=15 python3 setup.py clean

Downgrading numpy might have been unnecessary, since I can't remember if I cleared all the pytorch installation cache after installing numpy and trying again.
Now my installation log contains USE_NUMPY             : ON.
Dont know if this fixed all my problems, building pytorch takes forever so I'll just have to wait and see, but at least it fixed this one :)


Instruction: Split DataLoader PyTorch

Input:
Is it possible to split a dataloader object of training dataset into training and validation dataloader?
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

train_dataset = datasets.ImageFolder(train_data_directory, transform=transforms.ToTensor())
# Data loader
train_loader = DataLoader(train_dataset, batch_size=100, shuffle=True)

Now I would like to split a train_loader to train and validation dataloader.

Output: Look at random_split in torch.utils.data. It will handle a random Dataset split (you have to split before creating the DataLoader, not after).


Instruction: Embedding layer in neural machine translation with attention

Input:
I am trying to understanding how to implement a seq-to-seq model with attention from this website.
My question: Is nn.embedding just returns some IDs for each word, so the embedding for each word would be the same during whole training? Or are they getting changed during the procedure of training?
My second question is because I am confused whether after training, the output of nn.embedding is something such as word2vec word embeddings or not.
Thanks in advance

Output:
According to the PyTorch docs:

A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

In short, nn.Embedding embeds a sequence of vocabulary indices into a new embedding space. You can indeed roughly understand this as a word2vec style mechanism.
As a dummy example, let's create an embedding layer that takes as input a total of 10 vocabularies (i.e. the input data only contains a total of 10 unique tokens), and returns embedded word vectors living in 5-dimensional space. In other words, each word is represented as 5-dimensional vectors. The dummy data is a sequence of 3 words with indices 1, 2, and 3, in that order.
>>> embedding = nn.Embedding(10, 5)
>>> embedding(torch.tensor([1, 2, 3]))
tensor([[-0.7077, -1.0708, -0.9729,  0.5726,  1.0309],
        [ 0.2056, -1.3278,  0.6368, -1.9261,  1.0972],
        [ 0.8409, -0.5524, -0.1357,  0.6838,  3.0991]],
       grad_fn=<EmbeddingBackward>)

You can see that each of the three words are now represented as 5-dimensional vectors. We also see that there is a grad_fn function, which means that the weights of this layer will be adjusted through backprop. This answers your question of whether embedding layers are trainable: the answer is yes. And indeed this is the whole point of embedding: we expect the embedding layer to learn meaningful representations, the famous example of king - man = queen being the classic example of what these embedding layers can learn.

Edit
The embedding layer is, as the documentation states, a simple lookup table from a matrix. You can see this by doing
>>> embedding.weight
Parameter containing:
tensor([[-1.1728, -0.1023,  0.2489, -1.6098,  1.0426],
        [-0.7077, -1.0708, -0.9729,  0.5726,  1.0309],
        [ 0.2056, -1.3278,  0.6368, -1.9261,  1.0972],
        [ 0.8409, -0.5524, -0.1357,  0.6838,  3.0991],
        [-0.4569, -1.9014, -0.0758, -0.6069, -1.2985],
        [ 0.4545,  0.3246, -0.7277,  0.7236, -0.8096],
        [ 1.2569,  1.2437, -1.0229, -0.2101, -0.2963],
        [-0.3394, -0.8099,  1.4016, -0.8018,  0.0156],
        [ 0.3253, -0.1863,  0.5746, -0.0672,  0.7865],
        [ 0.0176,  0.7090, -0.7630, -0.6564,  1.5690]], requires_grad=True)

You will see that the first, second, and third rows of this matrix corresponds to the result that was returned in the example above. In other words, for a vocabulary whose index is n, the embedding layer will simply "lookup" the nth row in its weights matrix and return that row vector; hence the lookup table.


Instruction: BERT always predicts same class (Fine-Tuning)

Input:
I am fine-tuning BERT on a financial news dataset.
Unfortunately BERT seems to be trapped in a local minimum. It is content with learning to always predict the same class.

balancing the dataset didnt work
tuning parameters didnt work as well

I am honestly not sure what is causing this problem. With the simpletransformers library I am getting very good results. I would really appreciate if somebody could help me. thanks a lot!
Full code on github:
https://github.com/Bene939/BERT_News_Sentiment_Classifier
Code:
from transformers import BertForSequenceClassification, AdamW, BertTokenizer, get_linear_schedule_with_warmup, Trainer, TrainingArguments
import torch
from torch.utils.data import DataLoader, RandomSampler, SequentialSampler, TensorDataset
import pandas as pd
from pathlib import Path
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
import numpy as np
from torch.nn import functional as F
from collections import defaultdict
import random

#defining tokenizer, model and optimizer
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = BertForSequenceClassification.from_pretrained('bert-base-cased', num_labels=3)

if torch.cuda.is_available():
  print("\nUsing: ", torch.cuda.get_device_name(0))
  device = torch.device('cuda')
else:
  print("\nUsing: CPU")
  device = torch.device('cpu')
model = model.to(device)

#loading dataset
labeled_dataset = "news_headlines_sentiment.csv"
labeled_dataset_file = Path(labeled_dataset)
file_loaded = False
while not file_loaded:
  if labeled_dataset_file.exists():
    labeled_dataset = pd.read_csv(labeled_dataset_file)
    file_loaded = True
    print("Dataset Loaded")
  else:
    print("File not Found")
print(labeled_dataset)

#counting sentiments
negative = 0
neutral = 0
positive = 0
for idx, row in labeled_dataset.iterrows():
  if row["sentiment"] == 0:
    negative += 1
  elif row["sentiment"] == 1:
    neutral += 1
  else:
    positive += 1
print("Unbalanced Dataset")
print("negative: ", negative)
print("neutral: ", neutral)
print("positive: ", positive)

#balancing dataset to 1/3 per sentiment
for idx, row in labeled_dataset.iterrows():
  if row["sentiment"] == 0:
    if negative - neutral != 0:
      index_name = labeled_dataset[labeled_dataset["news"] == row["news"]].index
      labeled_dataset.drop(index_name, inplace=True)
      negative -= 1
  elif row["sentiment"] == 2:
    if positive - neutral != 0:
      index_name = labeled_dataset[labeled_dataset["news"] == row["news"]].index
      labeled_dataset.drop(index_name, inplace=True)
      positive -= 1

#custom dataset class
class NewsSentimentDataset(torch.utils.data.Dataset):
  def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

  def __getitem__(self, idx):
      item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
      item['labels'] = torch.tensor(self.labels[idx])
      return item

  def __len__(self):
      return len(self.labels)

#method for tokenizing dataset list
def tokenize_headlines(headlines, labels, tokenizer):

  encodings = tokenizer.batch_encode_plus(
      headlines,
      add_special_tokens = True,
      truncation = True,
      padding = 'max_length',
      return_attention_mask = True,
      return_token_type_ids = True
  )

  dataset = NewsSentimentDataset(encodings, labels)
  return dataset

#splitting dataset into training and validation set
#load news sentiment dataset
all_headlines = labeled_dataset['news'].tolist()
all_labels = labeled_dataset['sentiment'].tolist()

train_headlines, val_headlines, train_labels, val_labels = train_test_split(all_headlines, all_labels, test_size=.2)

val_dataset = tokenize_headlines(val_headlines, val_labels, tokenizer)
train_dataset = tokenize_headlines(train_headlines, val_labels, tokenizer)

#data loader
train_batch_size = 8
val_batch_size = 8

train_data_loader = DataLoader(train_dataset, batch_size = train_batch_size, shuffle=True)
val_data_loader = DataLoader(val_dataset, batch_size = val_batch_size, sampler=SequentialSampler(val_dataset))

#optimizer and scheduler
num_epochs = 1
num_steps = len(train_data_loader) * num_epochs
optimizer = AdamW(model.parameters(), lr=5e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_steps*0.06, num_training_steps=num_steps)

#training and evaluation
seed_val = 64

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

for epoch in range(num_epochs):

  print("\n###################################################")
  print("Epoch: {}/{}".format(epoch+1, num_epochs))
  print("###################################################\n")

  #training phase

  average_train_loss = 0
  average_train_acc = 0
  model.train()
  for step, batch in enumerate(train_data_loader):

      input_ids = batch['input_ids'].to(device)
      attention_mask = batch['attention_mask'].to(device)
      labels = batch['labels'].to(device)
      token_type_ids = batch['token_type_ids'].to(device)

      outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids = token_type_ids)

      loss = F.cross_entropy(outputs[0], labels)
      average_train_loss += loss

      if step % 40 == 0:
        print("Training Loss: ", loss)

      logits = outputs[0].detach().cpu().numpy()
      label_ids = labels.to('cpu').numpy()

      average_train_acc += sklearn.metrics.accuracy_score(label_ids, np.argmax(logits, axis=1))
      print("predictions: ",np.argmax(logits, axis=1))
      print("labels:      ",label_ids)
      print("#############")
      optimizer.zero_grad()
      loss.backward()
      #maximum gradient clipping
      torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

      optimizer.step()
      scheduler.step()
      model.zero_grad()

  average_train_loss = average_train_loss / len(train_data_loader)
  average_train_acc = average_train_acc / len(train_data_loader)
  print("======Average Training Loss: {:.5f}======".format(average_train_loss))
  print("======Average Training Accuracy: {:.2f}%======".format(average_train_acc*100))

  #validation phase
  average_val_loss = 0
  average_val_acc = 0
  model.eval()
  for step,batch in enumerate(val_data_loader):
    input_ids = batch['input_ids'].to(device)
    attention_mask = batch['attention_mask'].to(device)
    labels = batch['labels'].to(device)
    token_type_ids = batch['token_type_ids'].to(device)

    pred = []
    with torch.no_grad():

      outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)

      loss = F.cross_entropy(outputs[0], labels)
      average_val_loss += loss

      logits = outputs[0].detach().cpu().numpy()
      label_ids = labels.to('cpu').numpy()
      print("predictions: ",np.argmax(logits, axis=1))
      print("labels:      ",label_ids)
      print("#############")

      average_val_acc += sklearn.metrics.accuracy_score(label_ids, np.argmax(logits, axis=1))

  average_val_loss = average_val_loss / len(val_data_loader)
  average_val_acc = average_val_acc / len(val_data_loader)

  print("======Average Validation Loss: {:.5f}======".format(average_val_loss))
  print("======Average Validation Accuracy: {:.2f}%======".format(average_val_acc*100))
###################################################
Epoch: 1/1
###################################################

Training Loss:  tensor(1.1006, device='cuda:0', grad_fn=<NllLossBackward>)
predictions:  [1 0 2 0 0 0 2 0]
labels:       [2 0 1 1 0 1 0 1]
#############
predictions:  [2 2 0 0 0 2 0 0]
labels:       [1 2 1 0 2 0 1 2]
#############
predictions:  [0 0 0 0 1 0 0 1]
labels:       [0 1 1 0 1 1 2 0]
#############
predictions:  [0 0 0 2 0 1 0 0]
labels:       [0 0 0 2 0 0 2 1]
#############
predictions:  [1 0 0 0 0 0 2 0]
labels:       [0 2 2 1 0 0 0 0]
#############
predictions:  [0 0 0 0 0 1 0 0]
labels:       [1 0 2 2 2 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 2 2 2 0 2 0]
#############
predictions:  [0 1 0 0 0 0 0 0]
labels:       [2 2 0 2 0 0 0 1]
#############
predictions:  [0 0 0 0 0 2 0 1]
labels:       [0 1 0 2 2 0 1 2]
#############
predictions:  [0 0 2 0 0 0 1 0]
labels:       [0 0 0 1 2 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 1 0 1 0 1 1]
#############
predictions:  [0 2 0 0 0 0 0 0]
labels:       [2 2 0 1 0 1 2 1]
#############
predictions:  [0 1 0 0 0 0 1 2]
labels:       [2 2 1 0 2 0 0 2]
#############
predictions:  [0 0 1 1 1 1 0 1]
labels:       [1 2 1 1 1 1 2 2]
#############
predictions:  [1 0 0 0 0 1 2 1]
labels:       [1 0 1 1 0 0 0 2]
#############
predictions:  [0 1 1 1 1 0 2 1]
labels:       [2 2 1 2 2 1 1 2]
#############
predictions:  [0 0 1 0 1 1 0 0]
labels:       [1 0 0 1 0 1 0 2]
#############
predictions:  [1 2 0 0 1 2 0 0]
labels:       [0 2 2 1 2 0 1 0]
#############
predictions:  [0 2 1 1 0 1 1 0]
labels:       [2 2 0 1 1 0 1 2]
#############
predictions:  [1 0 1 1 1 1 1 0]
labels:       [0 2 0 1 0 1 2 2]
#############
predictions:  [0 2 1 2 0 0 1 1]
labels:       [2 1 1 1 1 2 2 0]
#############
predictions:  [0 1 2 2 2 1 1 2]
labels:       [2 2 1 1 2 1 0 1]
#############
predictions:  [2 2 2 1 2 1 1 1]
labels:       [0 1 1 0 0 2 2 1]
#############
predictions:  [1 2 2 2 1 2 1 2]
labels:       [0 0 0 0 2 0 1 2]
#############
predictions:  [2 1 1 1 2 2 2 2]
labels:       [1 0 2 2 1 0 0 0]
#############
predictions:  [2 1 2 2 2 1 2 2]
labels:       [2 1 1 1 1 1 2 2]
#############
predictions:  [1 1 0 2 1 2 1 2]
labels:       [2 2 0 2 0 1 2 0]
#############
predictions:  [0 1 1 2 0 1 2 1]
labels:       [2 2 2 1 2 2 0 1]
#############
predictions:  [2 1 1 1 1 2 1 1]
labels:       [0 1 1 2 1 0 0 2]
#############
predictions:  [1 2 2 0 1 1 1 2]
labels:       [0 1 2 1 2 1 0 1]
#############
predictions:  [0 1 1 1 1 1 1 0]
labels:       [0 2 0 1 1 2 2 2]
#############
predictions:  [1 2 1 1 2 1 1 0]
labels:       [0 2 2 2 0 0 1 0]
#############
predictions:  [2 2 2 1 2 1 1 2]
labels:       [2 2 1 2 1 0 0 0]
#############
predictions:  [2 2 1 2 2 2 1 2]
labels:       [1 1 2 2 2 0 2 1]
#############
predictions:  [2 2 2 2 2 0 2 2]
labels:       [2 2 1 2 0 1 1 2]
#############
predictions:  [1 1 2 1 2 2 0 1]
labels:       [2 1 1 1 0 0 2 2]
#############
predictions:  [2 1 2 2 2 2 1 0]
labels:       [0 2 0 2 0 0 0 0]
#############
predictions:  [2 2 2 2 2 2 2 2]
labels:       [1 1 0 2 0 1 2 1]
#############
predictions:  [2 2 2 2 1 2 2 2]
labels:       [1 0 0 1 1 0 0 0]
#############
predictions:  [2 2 2 1 2 2 2 2]
labels:       [1 0 1 1 0 2 2 0]
#############
Training Loss:  tensor(1.1104, device='cuda:0', grad_fn=<NllLossBackward>)
predictions:  [2 0 1 2 1 2 2 0]
labels:       [2 2 0 0 1 0 0 2]
#############
predictions:  [0 2 2 0 2 1 1 1]
labels:       [0 0 0 1 0 0 1 0]
#############
predictions:  [0 2 2 0 1 1 1 2]
labels:       [2 1 1 1 2 2 1 0]
#############
predictions:  [2 1 1 2 2 0 2 0]
labels:       [1 2 1 2 1 0 2 1]
#############
predictions:  [0 2 2 0 0 2 1 2]
labels:       [0 0 2 2 0 0 2 0]
#############
predictions:  [0 0 1 2 2 0 2 2]
labels:       [0 0 0 0 0 0 0 0]
#############
predictions:  [1 1 2 1 2 0 1 2]
labels:       [0 0 2 0 0 0 1 1]
#############
predictions:  [0 0 2 1 0 2 0 1]
labels:       [1 1 2 1 1 0 2 0]
#############
predictions:  [0 0 0 0 1 0 0 0]
labels:       [2 2 1 1 2 1 1 1]
#############
predictions:  [0 0 0 0 1 0 0 0]
labels:       [1 1 2 2 1 1 2 0]
#############
predictions:  [0 0 0 0 0 1 1 1]
labels:       [2 0 1 1 0 1 2 2]
#############
predictions:  [0 0 1 0 0 1 2 1]
labels:       [1 2 0 2 2 0 2 1]
#############
predictions:  [1 1 1 1 0 1 0 1]
labels:       [2 0 1 0 1 0 1 2]
#############
predictions:  [1 2 2 0 0 0 1 1]
labels:       [2 0 0 2 1 2 2 2]
#############
predictions:  [1 0 2 1 0 2 2 0]
labels:       [0 0 2 1 2 1 1 1]
#############
predictions:  [0 0 0 1 1 1 1 1]
labels:       [1 2 1 0 0 0 1 0]
#############
predictions:  [1 1 1 0 1 1 0 1]
labels:       [0 2 1 2 1 2 2 0]
#############
predictions:  [2 1 0 1 1 2 0 0]
labels:       [0 1 0 0 1 2 0 2]
#############
predictions:  [0 1 1 0 0 1 0 1]
labels:       [1 0 0 2 2 1 1 2]
#############
predictions:  [1 1 1 1 1 1 1 1]
labels:       [2 0 1 0 2 0 0 2]
#############
predictions:  [1 0 0 1 0 1 0 2]
labels:       [1 0 0 1 1 2 2 1]
#############
predictions:  [1 1 1 1 1 1 0 0]
labels:       [1 1 0 2 1 0 2 0]
#############
predictions:  [1 1 2 1 0 1 0 0]
labels:       [0 2 1 2 1 1 0 2]
#############
predictions:  [1 1 0 0 1 2 1 1]
labels:       [0 2 1 0 2 2 0 1]
#############
predictions:  [0 1 1 0 0 1 0 1]
labels:       [0 0 1 2 2 0 1 2]
#############
predictions:  [1 0 2 2 2 1 1 0]
labels:       [2 2 1 0 0 1 1 2]
#############
predictions:  [1 2 2 1 1 2 1 1]
labels:       [1 0 0 1 0 0 0 0]
#############
predictions:  [0 2 0 2 2 0 2 2]
labels:       [2 0 0 0 2 1 1 2]
#############
predictions:  [0 0 1 0 1 0 2 2]
labels:       [0 0 1 0 1 0 2 0]
#############
predictions:  [0 2 0 1 1 2 2 0]
labels:       [0 2 0 2 0 2 0 0]
#############
predictions:  [2 2 2 2 2 2 2 1]
labels:       [2 2 1 1 0 0 2 2]
#############
predictions:  [2 0 0 2 2 1 1 0]
labels:       [1 0 0 1 0 2 1 2]
#############
predictions:  [2 0 0 2 0 2 2 0]
labels:       [2 2 2 2 0 1 1 1]
#############
predictions:  [0 2 2 0 2 2 0 0]
labels:       [1 0 1 2 0 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 2]
labels:       [2 1 1 0 0 0 1 2]
#############
predictions:  [2 0 2 0 2 1 0 2]
labels:       [2 1 1 2 1 1 0 0]
#############
predictions:  [1 1 2 0 2 0 2 2]
labels:       [0 2 1 2 1 2 1 0]
#############
predictions:  [2 0 1 1 0 2 0 0]
labels:       [2 1 0 1 1 0 2 0]
#############
predictions:  [2 0 0 2 0 2 1 0]
labels:       [0 0 0 0 2 1 0 1]
#############
predictions:  [1 2 1 0 0 2 0 2]
labels:       [2 0 2 1 0 0 1 1]
#############
Training Loss:  tensor(1.1162, device='cuda:0', grad_fn=<NllLossBackward>)
predictions:  [2 0 0 1 1 1 0 1]
labels:       [0 1 1 1 1 2 2 1]
#############
predictions:  [0 2 0 1 2 0 0 1]
labels:       [2 2 1 0 1 0 0 0]
#############
predictions:  [0 0 1 0 0 0 0 1]
labels:       [1 0 2 0 0 2 2 0]
#############
predictions:  [2 1 2 2 0 1 2 0]
labels:       [2 0 1 0 2 1 0 1]
#############
predictions:  [1 0 0 2 0 0 1 1]
labels:       [2 2 0 2 0 2 0 0]
#############
predictions:  [0 0 1 0 0 0 0 0]
labels:       [2 2 2 1 2 2 2 2]
#############
predictions:  [0 0 1 1 0 1 1 0]
labels:       [2 1 1 1 0 2 1 0]
#############
predictions:  [0 0 0 1 0 0 1 0]
labels:       [2 0 2 2 0 0 1 2]
#############
predictions:  [1 0 1 0 0 2 0 0]
labels:       [1 1 2 0 0 1 0 0]
#############
predictions:  [2 1 0 0 0 1 0 0]
labels:       [1 2 0 0 0 0 0 0]
#############
predictions:  [0 2 0 0 0 0 0 0]
labels:       [2 0 1 1 2 2 1 1]
#############
predictions:  [0 1 0 0 0 1 0 2]
labels:       [0 2 1 1 0 0 1 2]
#############
predictions:  [0 2 1 0 0 1 1 1]
labels:       [1 1 0 2 0 1 1 0]
#############
predictions:  [0 1 1 0 0 0 1 0]
labels:       [0 0 1 0 1 2 1 1]
#############
predictions:  [0 1 1 0 1 0 0 0]
labels:       [0 1 1 1 2 2 2 0]
#############
predictions:  [0 0 0 0 1 1 0 0]
labels:       [2 0 2 2 1 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 0 2 2 0 1 1]
#############
predictions:  [0 1 0 0 0 0 0 0]
labels:       [0 2 0 1 1 2 0 2]
#############
predictions:  [1 1 0 1 0 1 0 2]
labels:       [1 2 0 0 2 2 2 1]
#############
predictions:  [1 1 0 0 0 1 2 1]
labels:       [0 0 1 2 2 1 2 2]
#############
predictions:  [1 1 1 0 1 1 2 0]
labels:       [0 0 0 2 0 1 0 2]
#############
predictions:  [0 1 0 0 1 1 2 1]
labels:       [2 0 0 1 2 2 1 2]
#############
predictions:  [1 0 0 0 1 0 0 1]
labels:       [1 2 2 2 2 1 0 1]
#############
predictions:  [2 0 0 0 0 0 0 0]
labels:       [1 2 0 2 2 1 1 1]
#############
predictions:  [2 0 1 1 0 0 1 0]
labels:       [0 0 0 0 2 2 1 1]
#############
predictions:  [2 0 0 1 0 0 1 1]
labels:       [2 2 1 1 0 0 1 0]
#############
predictions:  [1 1 1 1 1 2 0 0]
labels:       [0 0 2 1 0 0 0 0]
#############
predictions:  [1 1 2 0 1 2 0 1]
labels:       [0 2 1 0 2 0 0 1]
#############
predictions:  [0 0 2 1 0 2 0 1]
labels:       [1 2 0 2 2 1 0 0]
#############
predictions:  [0 0 2 0 2 1 1 2]
labels:       [2 2 1 2 2 2 0 0]
#############
predictions:  [0 1 0 0 0 0 2 1]
labels:       [1 1 0 1 1 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 0 0 2 0 0 2]
#############
predictions:  [2 2 2 0 1 1 1 0]
labels:       [1 0 2 1 1 2 0 0]
#############
predictions:  [0 0 1 0 0 0 2 0]
labels:       [0 1 2 1 1 0 0 0]
#############
predictions:  [0 2 0 1 0 2 0 0]
labels:       [0 0 2 1 1 0 2 2]
#############
predictions:  [0 0 1 2 0 2 0 1]
labels:       [2 2 0 0 0 2 2 2]
#############
predictions:  [1 0 0 0 2 0 0 1]
labels:       [2 0 1 1 1 0 0 1]
#############
predictions:  [0 1 0 0 0 0 0 2]
labels:       [1 1 1 0 0 0 2 2]
#############
predictions:  [0 2 0 1 0 2 0 0]
labels:       [1 1 1 1 2 2 1 0]
#############
predictions:  [1 2 0 0 0 0 0 0]
labels:       [2 0 2 1 0 1 1 1]
#############
Training Loss:  tensor(1.2082, device='cuda:0', grad_fn=<NllLossBackward>)
predictions:  [0 2 0 0 0 0 2 0]
labels:       [1 0 2 1 2 2 1 1]
#############
predictions:  [2 0 0 0 0 0 1 0]
labels:       [1 0 0 0 0 2 1 0]
#############
predictions:  [0 0 0 0 2 1 1 1]
labels:       [0 2 2 0 1 2 1 1]
#############
predictions:  [2 1 0 1 0 0 2 0]
labels:       [1 0 2 1 0 2 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 0 0 0 0 1 0]
#############
predictions:  [0 2 1 0 0 0 1 1]
labels:       [0 2 2 2 2 1 1 0]
#############
predictions:  [0 0 0 1 1 0 0 1]
labels:       [0 1 0 1 2 2 2 2]
#############
predictions:  [0 0 0 1 1 1 1 2]
labels:       [2 2 1 2 0 1 1 1]
#############
predictions:  [0 1 2 0 0 1 0 0]
labels:       [0 2 1 0 0 1 0 0]
#############
predictions:  [1 1 1 1 0 0 0 0]
labels:       [2 1 2 1 0 2 2 1]
#############
predictions:  [0 1 2 0 0 1 1 0]
labels:       [2 0 2 1 1 1 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 0 1 1 0 0]
#############
predictions:  [0 0 0 0 0 1 2 2]
labels:       [2 2 1 1 0 2 1 2]
#############
predictions:  [0 1 0 0 1 1 0 1]
labels:       [0 1 0 2 1 0 0 1]
#############
predictions:  [0 2 2 0 0 0 0 2]
labels:       [0 0 2 1 2 2 0 1]
#############
predictions:  [2 0 0 2 2 0 2 0]
labels:       [2 1 0 2 2 0 1 0]
#############
predictions:  [0 2 2 0 2 1 1 2]
labels:       [1 1 0 0 2 1 0 0]
#############
predictions:  [1 1 2 2 0 0 1 2]
labels:       [2 0 2 0 1 1 1 1]
#############
predictions:  [0 1 1 0 0 1 1 0]
labels:       [0 2 1 0 0 2 2 0]
#############
predictions:  [2 1 0 0 0 0 1 1]
labels:       [0 2 0 2 0 0 1 1]
#############
predictions:  [1 2 0 1 2 0 0 0]
labels:       [1 0 1 1 0 2 2 2]
#############
predictions:  [0 0 0 0 2 2 1 2]
labels:       [2 2 2 1 1 1 1 0]
#############
predictions:  [1 2 0 1 0 0 2 0]
labels:       [2 2 1 1 1 0 2 0]
#############
predictions:  [2 0 0 0 0 2 1]
labels:       [0 1 1 2 2 0 2]
#############
======Average Training Loss: 1.11279======
======Average Training Accuracy: 33.77%======
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 1 1 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 0 2 1 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 2 2 1 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 1 2 0 1 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 0 0 1 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 1 2 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 1 2 0 2 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 1 2 2 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 2 2 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 0 2 0 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 1 1 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 2 2 0 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 0 1 2 2 1]
#############
predictions:  [0 0 0 1 0 0 0 0]
labels:       [0 0 1 1 0 2 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 2 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 1 2 2 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 2 1 2 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 0 2 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 0 1 0 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 2 2 2 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 1 1 0 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 1 1 2 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 2 1 2 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 1 0 2 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 1 1 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 0 1 2 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 1 1 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 1 0 0 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 0 0 0 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 1 1 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 1 2 1 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 0 1 1 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 1 0 1 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 2 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 0 2 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 1 1 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 1 2 2 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 0 0 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 0 1 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 2 1 1 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 2 2 2 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 2 2 1 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 2 2 2 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 0 0 1 0 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 0 0 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 1 2 0 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 2 0 1 2 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 2 0 0 0 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 1 0 0 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 0 1 1 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 0 0 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 2 1 1 1 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 0 2 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 2 1 0 2 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 1 2 2 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 0 0 2 1 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 2 0 2 1 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 2 0 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 0 0 1 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 2 2 0 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 1 1 1 0 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 2 2 1 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 0 2 0 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 1 1 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 1 1 1 1 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 2 1 0 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 2 1 0 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 2 0 0 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 2 0 0 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 0 2 2 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 0 1 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 1 2 0 1 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 0 0 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 1 2 0 2 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 1 0 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 0 1 0 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 0 0 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 1 2 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 2 0 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 0 1 1 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 0 1 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 1 2 0 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 1 1 1 0 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 1 2 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 0 1 1 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 0 2 1 0 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 0 0 0 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 1 2 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 0 1 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 1 0 2 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 1 2 0 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 2 2 2 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 1 2 0 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 1 1 0 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 2 2 2 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 0 0 1 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 2 1 2 1 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 0 0 0 2 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 1 1 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 1 0 2 2 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 1 1 1 2 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 0 1 0 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 2 2 0 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 1 2 2 2 1 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 0 1 0 2 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 2 1 0 2 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 2 0 2 2 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 2 0 0 1 0 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 1 0 0 0 2 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 2 0 1 2 1 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 2 2 2 2 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 0 1 2 0 2 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 1 1 1 1 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 0 0 0 1 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 1 2 0 1 2 2 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 1 1 1 2 1 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [1 0 1 1 1 0 0 2]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 2 0 0 0 0 1 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [0 0 1 1 2 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 1 1 1 0 1 0 0]
#############
predictions:  [0 0 0 0 0 0 0 0]
labels:       [2 0 2 2 2 0 0 1]
#############
predictions:  [0 0 0 0 0 0 0]
labels:       [2 2 1 1 0 0 1]
#############
======Average Validation Loss: 1.09527======
======Average Validation Accuracy: 35.53%======

Output: For multi-class classification/sentiment analysis using BERT the 'neutral' class HAS TO BE 2!! It CANNOT be between 'negative' = 0 and 'positive' = 2


Instruction: Error while installing PyTorch using pip - cannot build wheel

Input:
I get the following output when I try to run pip3 install pytorch or pip install pytorch
Collecting pytorch
  Using cached pytorch-1.0.2.tar.gz (689 bytes)
Building wheels for collected packages: pytorch
  Building wheel for pytorch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-8rsdyb8e
       cwd: /tmp/pip-install-3v4wd97t/pytorch/
  Complete output (5 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-3v4wd97t/pytorch/setup.py", line 15, in <module>
      raise Exception(message)
  Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
  ----------------------------------------
  ERROR: Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
    Running setup.py install for pytorch ... error
    ERROR: Command errored out with exit status 1:
     command: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eld9j0g4/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/chaitanya/.local/include/python3.8/pytorch
         cwd: /tmp/pip-install-3v4wd97t/pytorch/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-3v4wd97t/pytorch/setup.py", line 11, in <module>
        raise Exception(message)
    Exception: You tried to install "pytorch". The package named for PyTorch is "torch"
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/chaitanya/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3v4wd97t/pytorch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-eld9j0g4/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/chaitanya/.local/include/python3.8/pytorch Check the logs for full command output.

I downloaded the matching wheel from here, but am couldn't figure out what to do with it. My Python installation is using anaconda3, if that's needed. What should I do from here? Tips on how I could have resolved this on my own would also be appreciated.

Output:
From your error:

Exception: You tried to install "pytorch". The package named for PyTorch is "torch"

which tells you what you need to know, instead of
pip install pytorch

it should be
pip install torch

I downloaded the matching wheel from here, but am couldn't figure out what to do with it

Installing .whl files is as easy as
pip install <path to .whl file>

My Python installation is using anaconda3

That is very relevant. You should generally avoid as much as possible to use pip in your conda environment. Instead, you can find the correct conda install command for your setup(cuda version etc.) from pytroch.org, e.g. for cuda 11 it would be
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch


Instruction: Transformers get named entity prediction for words instead of tokens

Input:
This is very basic question, but I spend hours struggling to find the answer. I built NER using Hugginface transformers.
Say I have input sentence
input = "Damien Hirst oil in canvas"

I tokenize it to get
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
tokenized = tokenizer.encode(input) #[101, 12587, 7632, 12096, 3514, 1999, 10683, 102]

Feed tokenized sentence to the model to get predicted tags for the tokens
['B-ARTIST' 'B-ARTIST' 'I-ARTIST' 'I-ARTIST' 'B-MEDIUM' 'I-MEDIUM'
 'I-MEDIUM' 'B-ARTIST']

prediction comes as output from the model. It assigns tags to different tokens.
How can I recombine this data to obtain tags for words instead of tokens? So I would know that
"Damien Hirst" = ARTIST
"Oil in canvas" = MEDIUM

Output:
There are two questions here.
Annotating Token Classification
A common sequential tagging, especially in Named Entity Recognition, follows the scheme that a sequence to tokens with tag X at the beginning gets B-X and on reset of the labels it gets I-X.
The problem is that most annotated datasets are tokenized with space! For example:
[CSL]  O
Damien  B-ARTIST
Hirst  I-ARTIST
oil  B-MEDIUM
in  I-MEDIUM
canvas  I-MEDIUM
[SEP]  O

where O indicates that it is not a named-entity, B-ARTIST is the beginning of the sequence of tokens labelled as ARTIST and I-ARTIST is inside the sequence - similar pattern for MEDIUM.
At the moment I posted this answer, there is an example of NER in huggingface documentation here:
https://huggingface.co/transformers/usage.html#named-entity-recognition
The example doesn't exactly answer the question here, but it can add some clarification. The similar style of named entity labels in that example could be as follows:
label_list = [
    "O", # not a named entity
    "B-ARTIST", # beginning of an artist name
    "I-ARTIST", # an artist name
    "B-MEDIUM", # beginning of a medium name
    "I-MEDIUM", # a medium name
]

Adapt Tokenizations
With all that said about annotation schema, BERT and several other models have different tokenization model. So, we have to adapt these two tokenizations.
In this case with bert-base-uncased, the expected outcome is like this:
damien  B-ARTIST
hi  I-ARTIST
##rst  I-ARTIST
oil  B-MEDIUM
in  I-MEDIUM
canvas  I-MEDIUM

In order to get this done, you can go through each token in original annotation, then tokenize it and add its label again:
tokens_old = ['Damien', 'Hirst', 'oil', 'in', 'canvas']
labels_old = ["B-ARTIST", "I-ARTIST", "B-MEDIUM", "I-MEDIUM", "I-MEDIUM"]
label2id = {label: idx for idx, label in enumerate(label_list)}

tokens, labels = zip(*[
   (token, label)
   for token_old, label in zip(tokens_old, labels_old)
   for token in tokenizer.tokenize(token_old)
])

When you add [CLS] and [SEP] in the tokens, their labels "O" must be added to labels.
With the code above, it is possible to get into a situation that a beginning tag like B-ARTIST get repeated when the beginning word splits into pieces. According to the description in huggingface documentation, you can encode these labels with -100 to be ignored:
https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities
Something like this should work:
tokens, labels = zip(*[
   (token, label2id[label] if (label[:2] != "B-" or i == 0) else -100)
   for token_old, label in zip(tokens_old, labels_old)
   for i, token in enumerate(tokenizer.tokenize(token_old))
])


Instruction: Specifying cpu-only for pytorch in conda YAML file

Input:
I can set up a conda environment successfully as follows:
conda create --name temp python=3.8.5
conda install pytorch==1.6.0 torchvision==0.7.0 cpuonly -c pytorch

I then save the environment to a YAML config file. The looks like this:
name: temp
channels:
  - pytorch
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - blas=1.0=mkl
  - ca-certificates=2020.10.14=0
  - certifi=2020.6.20=pyhd3eb1b0_3
  - cpuonly=1.0=0
  - freetype=2.10.4=h5ab3b9f_0
  - intel-openmp=2020.2=254
  - jpeg=9b=h024ee3a_2
  - lcms2=2.11=h396b838_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libedit=3.1.20191231=h14c3975_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libpng=1.6.37=hbc83047_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - libtiff=4.1.0=h2733197_1
  - lz4-c=1.9.2=heb0550a_3
  - mkl=2020.2=256
  - mkl-service=2.3.0=py38he904b0f_0
  - mkl_fft=1.2.0=py38h23d657b_0
  - mkl_random=1.1.1=py38h0573a6f_0
  - ncurses=6.2=he6710b0_1
  - ninja=1.10.1=py38hfd86e86_0
  - numpy=1.19.2=py38h54aff64_0
  - numpy-base=1.19.2=py38hfa32c7d_0
  - olefile=0.46=py_0
  - openssl=1.1.1h=h7b6447c_0
  - pillow=8.0.1=py38he98fc37_0
  - pip=20.2.4=py38h06a4308_0
  - python=3.8.5=h7579374_1
  - pytorch=1.6.0=py3.8_cpu_0
  - readline=8.0=h7b6447c_0
  - setuptools=50.3.0=py38h06a4308_1
  - six=1.15.0=py_0
  - sqlite=3.33.0=h62c20be_0
  - tk=8.6.10=hbc83047_0
  - torchvision=0.7.0=py38_cpu
  - wheel=0.35.1=py_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.4.5=h9ceee32_0
prefix: /data/anaconda/envs/temp

But if I try making a conda environment from the following file:
name: temp
channels:
  - defaults
  - pytorch
dependencies:
  - python==3.8.5
  - pytorch==1.6.0=py3.8_cpu_0

it fails, with the following incompatibilities:
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package ld_impl_linux-64 conflicts for:
python==3.8.5 -> ld_impl_linux-64
Package sqlite conflicts for:
python==3.8.5 -> sqlite[version='>=3.32.3,<4.0a0|>=3.33.0,<4.0a0']
Package * conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> *[track_features=cpuonly]
Package ncurses conflicts for:
python==3.8.5 -> ncurses[version='>=6.2,<7.0a0']
Package mkl conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> mkl[version='>=2018']
Package blas conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> blas=[build=mkl]
Package zlib conflicts for:
python==3.8.5 -> zlib[version='>=1.2.11,<1.3.0a0']
Package openssl conflicts for:
python==3.8.5 -> openssl[version='>=1.1.1g,<1.1.2a']
Package python conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> python[version='>=3.8,<3.9.0a0']
Package xz conflicts for:
python==3.8.5 -> xz[version='>=5.2.5,<6.0a0']
Package libffi conflicts for:
python==3.8.5 -> libffi[version='>=3.3,<3.4.0a0']
Package libgcc-ng conflicts for:
python==3.8.5 -> libgcc-ng[version='>=7.3.0']
Package tk conflicts for:
python==3.8.5 -> tk[version='>=8.6.10,<8.7.0a0']
Package pip conflicts for:
python==3.8.5 -> pip
Package ninja conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> ninja
Package readline conflicts for:
python==3.8.5 -> readline[version='>=8.0,<9.0a0']
Package numpy conflicts for:
pytorch==1.6.0=py3.8_cpu_0 -> numpy[version='>=1.11']

Why? How could such a simple configuration, which was cut down from a successful environment, fail? How should I specify the CPU-only version of pytorch 1.6.0 in a YAML config file for a conda environment?

Output:
For systems that have optional CUDA support (Linux and Windows) PyTorch provides a mutex metapackage cpuonly that when installed constrains the pytorch package solve to only non-CUDA builds. Going through the PyTorch installation widget will suggest including the cpuonly package when selecting "NONE" of the CUDA option

I don't know the internals of how to build packages that use such mutex metapackages, but mutex metapackages are documented with metapackages in general, and the docs include links to MKL vs OpenBLAS examples.
Exactly why the simple YAML you started with fails is still unclear to me, but my guess is that cpuonly constrains more than just the pytorch build and having the specific pytorch build alone is not sufficient to constrain its dependencies.


Instruction: Inconsistency in loss on SAME data for train and validation modes tensorflow

Input:
I'm implementing a semantic segmentation model with images. As a good practice I tested my training pipeline with just one image and tried to over-fit that image. To my surprise, when training with the exactly the same images, the loss goes to near 0 as expected but when evaluating THE SAME IMAGES, the loss is much much higher, and it keeps going up as the training continues. So the segmentation output is garbage when training=False, but when run with training=True is works perfectly.
To be able to anyone to reproduce this I took the official segmentation tutorial and modified it a little for training a convnet from scratch and just 1 image. The model is very simple, just a sequence of Conv2D with batch normalization and Relu. The results are the following

As you see the loss and eval_loss are really different, and making inference to the image gives perfect result in training mode and in eval mode is garbage.
I know Batchnormalization behaves differently in inference time since it uses the averaged statistics calculated whilst training. Nonetheless, since we are training with just 1 same image and evaluating in the same image, this shouldn't happen right? Moreover I implemented the same architecture with the same optimizer in Pytorch and this does not happen there. With pytorch it trains and eval_loss converges to train loss
Here you can find the above mentioned https://colab.research.google.com/drive/18LipgAmKVDA86n3ljFW8X0JThVEeFf0a#scrollTo=TWDATghoRczu
and at the end also the Pytorch implementation

Output:
It had to do more with the defaults values that tensorflow uses. Batchnormalization has a parameter momentum which controls the averaging of batch statistics. The formula is: moving_mean = moving_mean * momentum + mean(batch) * (1 - momentum)
If you set momentum=0.0 in the BatchNorm layer, the averaged statistics should match perfectly with the statistics from the current batch (which is just 1 image). If you do so, you see that the validation loss almost immediately matches the training loss. Also if you try with momentum=0.9 (which is the equivalent default value in pytorch) and it works and converges faster (as in pytorch).


Instruction: Pytorch DQN, DDQN using .detach() caused very wield loss (increases exponentially) and do not learn at all

Input:
Here is my implementation of DQN and DDQN for CartPole-v0 which I think is correct.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import gym
import torch.optim as optim
import random
import os
import time

class NETWORK(torch.nn.Module):
    def __init__(self, input_dim: int, output_dim: int, hidden_dim: int) -> None:

        super(NETWORK, self).__init__()

        self.layer1 = torch.nn.Sequential(
            torch.nn.Linear(input_dim, hidden_dim),
            torch.nn.ReLU()
        )

        self.layer2 = torch.nn.Sequential(
            torch.nn.Linear(hidden_dim, hidden_dim),
            torch.nn.ReLU()
        )

        self.final = torch.nn.Linear(hidden_dim, output_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.final(x)

        return x

class ReplayBuffer(object):
    def __init__(self, capacity=50000):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, s0, a0, r, s1):
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = (s0, a0, r, s1)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size=64):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

class DQN(object):
    def __init__(self):
      self.state_dim = 4
      self.action_dim = 2
      self.lr = 0.001
      self.discount_factor = 0.99
      self.epsilon = 1
      self.epsilon_decay = 0.95
      self.num_train = 0
      self.num_train_episodes = 0
      self.batch_size = 64

      self.predict_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()

      self.memory = ReplayBuffer(capacity=50000)
      self.optimizer = torch.optim.Adam(self.predict_network.parameters(), lr=self.lr)
      self.loss = 0

    def select_action(self, states: np.ndarray) -> int:
      if np.random.uniform(0, 1) < self.epsilon:
        return np.random.choice(self.action_dim)
      else:
        states = torch.from_numpy(states).unsqueeze_(dim=0)
        with torch.no_grad():
          Q_values = self.predict_network(states)
          action = torch.argmax(Q_values).item()
        return action

    def policy(self, states: np.ndarray) -> int:
      states = torch.from_numpy(states).unsqueeze_(dim=0)
      with torch.no_grad():
        Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
      return action

    def train(self, s0, a0, r, s1, sign):
      if sign == 1:
        self.num_train_episodes += 1
        if self.epsilon > 0.01:
          self.epsilon = max(self.epsilon * self.epsilon_decay, 0.01)
        return

      self.num_train += 1
      self.memory.push(s0, a0, r, s1)
      if len(self.memory) < self.batch_size:
        return

      batch = self.memory.sample(self.batch_size)
      state_batch = torch.from_numpy(np.stack([b[0] for b in batch]))
      action_batch = torch.from_numpy(np.stack([b[1] for b in batch]))
      reward_batch = torch.from_numpy(np.stack([b[2] for b in batch]))
      next_state_batch = torch.from_numpy(np.stack([b[3] for b in batch]))

      Q_values = self.predict_network(state_batch)[torch.arange(self.batch_size), action_batch]

      next_state_Q_values = self.predict_network(next_state_batch).max(dim=1)[0]

      Q_targets = self.discount_factor * next_state_Q_values + reward_batch

      loss = F.mse_loss(Q_values, Q_targets.detach())

      self.optimizer.zero_grad()
      loss.backward()
      self.optimizer.step()

      self.loss = loss.data.item()

class DDQN(object):
    def __init__(self):
      self.state_dim = 4
      self.action_dim = 2
      self.lr = 0.001
      self.discount_factor = 0.9
      self.epsilon = 1
      self.epsilon_decay = 0.95
      self.num_train = 0
      self.num_train_episodes = 0
      self.batch_size = 64

      self.predict_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()
      self.target_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()
      self.target_network.load_state_dict(self.predict_network.state_dict())
      self.target_network.eval()

      self.memory = ReplayBuffer(capacity=50000)
      self.optimizer = torch.optim.Adam(self.predict_network.parameters(), lr=self.lr)

      self.loss = 0

    def select_action(self, states: np.ndarray) -> int:
      if np.random.uniform(0, 1) < self.epsilon:
        return np.random.choice(self.action_dim)
      else:
        states = torch.from_numpy(states).unsqueeze_(dim=0)
        with torch.no_grad():
          Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
        return action

    def policy(self, states: np.ndarray) -> int:
      states = torch.from_numpy(states).unsqueeze_(dim=0)
      with torch.no_grad():
        Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
      return action

    def train(self, s0, a0, r, s1, sign):
      if sign == 1:
        self.num_train_episodes += 1
        if self.num_train_episodes % 2 == 0:
          self.target_network.load_state_dict(self.predict_network.state_dict())
          self.target_network.eval()

          if self.epsilon > 0.01:
            self.epsilon = max(self.epsilon * self.epsilon_decay, 0.01)
        return

      self.num_train += 1
      self.memory.push(s0, a0, r, s1)
      if len(self.memory) < self.batch_size:
        return
      batch = self.memory.sample(self.batch_size)
      state_batch = torch.from_numpy(np.stack([b[0] for b in batch]))
      action_batch = torch.from_numpy(np.stack([b[1] for b in batch]))
      reward_batch = torch.from_numpy(np.stack([b[2] for b in batch]))
      next_state_batch = torch.from_numpy(np.stack([b[3] for b in batch]))

      Q_values = self.predict_network(state_batch)[torch.arange(self.batch_size), action_batch]

      next_state_action_batch = torch.argmax(self.predict_network(next_state_batch), dim=1)

      next_state_Q_values = self.target_network(next_state_batch)[torch.arange(self.batch_size), next_state_action_batch]

      Q_targets = self.discount_factor * next_state_Q_values + reward_batch

      loss = F.smooth_l1_loss(Q_values, Q_targets.detach())
      self.optimizer.zero_grad()
      loss.backward()
      self.optimizer.step()

      self.loss = loss.data.item()

I used following to evaluate and train my DQN and DDQN.
def eval_policy(agent, env_name, eval_episodes=10):
    eval_env = gym.make(env_name)
    avg_reward = 0.
    for _ in range(eval_episodes):
        state, done = eval_env.reset(), False
        while not done:
            action = agent.policy(state)
            state, reward, done, _ = eval_env.step(action)
            avg_reward += reward
    avg_reward /= eval_episodes
    print("---------------------------------------")
    print(f"Evaluation over {eval_episodes} episodes: {avg_reward:.3f}")
    print("---------------------------------------")
    return avg_reward

env_name = 'CartPole-v0'
env = gym.make(env_name)

agent = DQN() # agent = DDQN()

for i in range(1000):
    state, done = env.reset(), False
    episodic_reward = 0
    while not done:
        action = agent.select_action(np.squeeze(state))
        next_state, reward, done, info = env.step(action)
        episodic_reward += reward
        sign = 1 if done else 0
        agent.train(state, action, reward, next_state, sign)
        state = next_state
    print(f'episode: {i}, reward: {episodic_reward}')
    if i % 20 == 0:
        eval_reward = eval_policy(agent, env_name, eval_episodes=50)
        if eval_reward >= 195:
            print("Problem solved in {} episodes".format(i + 1))
            break

The thing is that my DQN networks do not train and the loss grow exponentially using target.detach() in loss calculation. If I do not use .detach(), the DQN object would train but I believe that is not the correct way. For DDQN, my networks always do not train. Can anyone give some advice on where might be wrong?

Output:
so one mistake in your implementation is that you never add the end of an episode to your replay buffer. In your train function you return if sign==1 (end of the episode). Remove that return and adjust the target calculation via (1-dones)*... in case you sample a transition of the end of an episode. The reason why the end of the episode is important is that it is the only experience is where the target is not approximated via bootstrapping. Then DQN trains. For reproducibility I used a discount rate of 0.99 and the seed 2020 (for torch, numpy and the gym environment). I achieved a reward of 199.100 after 241 episodes of training.
Hope that helps, code is very readable btw.


Instruction: How PyTorch implements Convolution Backward?

Input:
I read about the Pytorch's source code, and I find it's weird that it doesn't implement the convolution_backward function, The only convolution_backward_overrideable function is directly raises an error and supposed not to fall here.
So I referred to CuDNN / MKLDNN implementation, they both implements functions like cudnn_convolution_backward.
I got the following question:

What are the native implementation of CUDA/ CPU? I can find something like thnn_conv2d_backward_out, but I could not find where are this is called.

Why PyTorch didn't put the convolution_backward function in Convolution.cpp? It offers an _convolution_double_backward() function. But this is the double backward, it's the gradient of gradient. Why don't they offer a single backward function?

If I want to call the native convolution/ convolution_backward function for my pure cpu/cuda tensor, how should I write code? Or where could I refer to? I couldn't find example for this.

Thanks !

Output:
1- Implementation may differ depending on which backend you use, it may use CUDA convolution implementation from some library, CPU convolution implementation from some other library, or custom implementation, see here: pytorch - Where is “conv1d” implemented?.
2- I am not sure about the current version, but single backward was calculated via autograd, that is why there was not an explicit different function for it. I don't know the underlying details of autograd but you can check https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/autograd.cpp. That double_backward function is only there if you need higher order derivatives.
3- If you want to do this in C, the file you linked (convolution.cpp) shows you how to do this (function at::Tensor _convolution...). If you inspect the function you see it just checks which implementation to use (params.use_something...) and use it. If you want to do this in python you should start tracing from conv until where this file convolution.cpp is called.


Instruction: Exporting forecast data from darts

Input:
I am trying to export the forecasts I made with Pytorch models in darts library to some exchangeable format like XLS or CSV.
Is it somehow possible to export the predictions from the Timeseries class? How can I specify output format?

Output:
Forecasts in Darts are nothing but regular TimeSeries objects and a TimeSeries is internally represented as a pandas.DataFrame.
You can access this DataFrame using series.pd_dataframe() (to get a copy of the DataFrame) or series._df directly if you want to avoid copying the DataFrame to save it. Be careful in the latter case however, as modifying the DataFrame in place will break the TimeSeries immutability.
You can then use any method you'd like to save pandas dataframes, e.g. pandas.DataFrame.to_csv() or pandas.DataFrame.to_pickle(). You can have a look at this article on Medium to see a comparison of a couple different formats' performances for saving and loading dataframes.


Instruction: Teacher force training PyTorch

Input:
I am trying to do a seq2seq prediction. For this, I have a LSTM layer followed by a fully connected layer. I employ Teacher training during the training phase and would like to skip this (I maybe wrong here) during testing phase. I have not found a direct way of doing this so I have taken the approach shown below.
def forward(self, inputs, future=0, teacher_force_ratio=0.2, target=None):
    outputs = []
    for idx in range(future):
        rnn_out, _ = self.rnn(inputs)
        output = self.fc1(rnn_out)
        if self.teacher_training:
            new_input = output if np.random.random() >= teacher_force_ratio else target[idx]
        else:
            new_input = output
        inputs = new_input

I use a bool variable teacher_training to check if Teacher training is needed or not. Is this correct? If yes, is there a better way to do this? Thanks.

Output: In PyTorch all classes that extend nn.Module have a kwarg boolean param called training . So instead of teacher_training we should simply use training param. This param is automatically set depending on your model training mode (model.train() and model.eval()).


Instruction: Find module corresponding to the method in python

Input:
I am learning to work with fastai. And I come across this:
xt = tensor(3.).requires_grad_()
xt
[out1]: tensor(3., requires_grad=True)

I want to know where requires_grad_() is from i.e., is it from fastai or is it pytorch related. So I do:
requires_grad_.__module__
[out2]: NameError: name 'require_grad_' is not defined

What am I doing wrong? What am I missing?

Output:
Err
The key is you cannot access requires_grad_ directly. requires_grad_ is not a global object, it is a member method of xt, so you cannot access it directly (when you do this, you'll get  the not define error),  the way to access it is through the class instance it belongs to (xt), like print(xt.requires_grad_.__module__).
Test
According to doc on python.org, all of function, member method, class and class instance have __module__ attr. So the code print(xt.requires_grad_.__module__) should work, but when I tested the code
print(xt.requires_grad_.__module__)

and got output None rather than torch I expect, which is a litte bit wierd. I guess for torch Tensor's member method, this __module__ varible is unavailable. So then I tested another case (print the __module__ attr of a member method for another class DataFrame):
from pandas import DataFrame
a = DataFrame()
a.min.__module__

and I got output pandas.core.frame which means we do can get __module__ attr from a member method object.
However, in you case, the fastest way is to get the module of xt, since a class method must belongs to the module of its class instance, so you can just use
print(xt.__module__)

and you will get correct module name: torch.
BTW, requires_grad_ belongs to torch.Tensor which belongs to torch module, at backprop phase, this func set a bool flag that tells torch whether gradient of tensor `xt


Instruction: Difference between loading my trained model in a pretrained model and loading it in not pretrained one?

Input:
I trained an Inception_v3 for my task. I have 3 classes.
After training I try to test my trained model and I use following code to load my model:
model = models.inception_v3(pretrained=True, aux_logits=False)
model.fc = nn.Linear(model.fc.in_features, 3)
model.load_state_dict(torch.load(My Model Path.pth))

Download a pretrained Inception_v3, change output features and load my weight in this model.
I obtained very good results as I expect from validation phase.
If I use the same code but with pretrained=False the test go very bad.
model = models.inception_v3(pretrained=False, aux_logits=False)
model.fc = nn.Linear(model.fc.in_features, 3)
model.load_state_dict(torch.load(My Model Path.pth))

Since in the model I download I load my weights, there should be no difference between pretrained True or False.
Does anyone know what changes?

Output:
The pretrained=True has an additional effect on the inception_v3 model: it controls whether or not the input will be

preprocessed according to the method with which it was trained on
ImageNet

(source-code here).
When you set pretrained=False, if you want to make things comparable at test time, you should also set transform_input=True:
model = models.inception_v3(pretrained=False, aux_logits=False, transform_input=True)
model.fc = nn.Linear(model.fc.in_features, 3)
model.load_state_dict(torch.load(My Model Path.pth))

In case you're wondering, this is the preprocessing:
def _transform_input(self, x: Tensor) -> Tensor:
    if self.transform_input:
        x_ch0 = torch.unsqueeze(x[:, 0], 1) * (0.229 / 0.5) + (0.485 - 0.5) / 0.5
        x_ch1 = torch.unsqueeze(x[:, 1], 1) * (0.224 / 0.5) + (0.456 - 0.5) / 0.5
        x_ch2 = torch.unsqueeze(x[:, 2], 1) * (0.225 / 0.5) + (0.406 - 0.5) / 0.5
        x = torch.cat((x_ch0, x_ch1, x_ch2), 1)
    return x


Instruction: Save Pytorch 4D tensor as image

Input:
I have a 4-d Pytorch tensor that I would like to save to disk as a .jpg
My tensor is the following size:
print(image_tensor.size())

>>>torch.Size([1, 3, 400, 711])

I can view the entire tensor as one image within my IDE:
ax1.imshow(im_convert(image_tensor))

Since I am able to view the entire tensor as one image, I am assuming there is a way to also save it as such. However, when I try to save the image, it looks like it only saves the blue color channel. I would like to save the entire tensor as a single image.
img1 = image_tensor[0]
save_image(img1, 'img1.jpg')

Output:
In PyTorch this snippet is working and saving the image:
from torchvision.utils import save_image
import torch
import torchvision

tensor= torch.rand(2, 3, 400, 711)

img1 = tensor[0]
save_image(img1, 'img1.png')

Before saving the image can you check the shape of the img1 in any case something happened.


Instruction: Pytorch: How to create a random int tensor where a certain percent are of a certain value? For example, 25% are 1s, and rest 0s

Input:
In pytorch I can create a random zero and one tensor with around %50 distribution of each
import torch
torch.randint(low=0, high=2, size=(2, 5))

I am wondering how I can make a tensor where only 25% of the values are 1s, and the rest are zeros?

Output:
Following my answer here: How to randomly set a fixed number of elements in each row of a tensor in PyTorch
Say you want a matrix with dimensions n X d where exactly 25% of the values in each row are 1 and the rest 0, desired_tensor will have the result you want:
n = 2
d = 5
rand_mat = torch.rand(n, d)
k = round(0.25 * d) # For the general case change 0.25 to the percentage you need
k_th_quant = torch.topk(rand_mat, k, largest = False)[0][:,-1:]
bool_tensor = rand_mat <= k_th_quant
desired_tensor = torch.where(bool_tensor,torch.tensor(1),torch.tensor(0))


Instruction: How to save information about the result of instance segmentation by YOLACT?

Input:
Is there any way to save the detected categories, their number, MASK area, etc. to a TXT file or CSV file when performing instance segmentation using YOLACT?
I’m using YOLACT (https://github.com/dbolya/yolact) to challenge instance segmentation.
I was able to use eval.py to do an instance segmentation of my own data and save that image or video.
However, what I really need is the class names and their numbers detected and classified by YOLACT's AI, and the area of MASK.
If we can output this information to a txt file or csv file, we can use YOLACT even more advanced.
If I can achieve that by adding an option in eval.py or modifying the code, please teach me.
Thank you.

Output:
You already have that information from the eval.py.
This line in the eval.py gives you the information.
# line 160
classes, scores, boxes = [x[:args.top_k].cpu().numpy() for x in t[:3]]
masks = t[3][:args.top_k]

areas = []

for mask in masks:
    # Use OpenCV's findContours() to detect and obtain the contour of the variable mask
    # Use OpenCV's contourArea() to calculate the area
    areas.append(area)

Getting the no. of detections:
# line 162
num_dets_to_consider = min(args.top_k, classes.shape[0])
    for j in range(num_dets_to_consider):
        if scores[j] < args.score_threshold:
            num_dets_to_consider = j
            break

To save the classes and scores as a csv file:
import pandas as pd

df = pd.DataFrame(columns=['class', 'score'])
c = 0
for i in reversed(range(num_dets_to_consider)):
    classname = cfg.dataset.class_names[classes[j]]
    score = scores[i]
    df.loc[c] = [classname, score]
    c += 1
df.to_csv('info.csv', index=None)

EDIT1:
You can return the values (classes, scores, areas) from the prep_display() and then retrieve them in evalimage(). Note that evalimage() calls prep_display(). You can do something like this:
classes, scores, areas = prep_display()
# Here you can include the pandas code. Note that it stores the information only for one image. You can use a loop and then store the information of all the images.

EDIT2:
# This is the default line in eval.py at line ~600
# This is found at evalimage()
img_numpy = prep_display(preds, frame, None, None, undo_transform=False)

# Change the above line to this:
classes, scores, areas, img_numpy = prep_display(preds, frame, None, None, undo_transform=False)

# Also, remember to return these 4 values from the prep_display().
# Otherwise, it will lead to an error.
# img_numpy is the default one that will be returned from prep_display().
# You're now simply returning 4 values instead of 1.

# NOTE: I have included img_numpy because I found that prep_display() is being used in many places.
# You need to modify the returns wherever prep_display() is being called in the eval.py. I think it's being called 3 times from different regions in eval.py.


Instruction: How to implent tf.nn.in_top_k in pytorch

Input:
I want to implent tf.nn.in_top_k in pytorch. Here is the link of tf.nn.in_top_k,
tf.math.in_top_k(
    targets, predictions, k, name=None
)

It computed precision at k as a bool Tensor and will return a Tensor of type bool.
tf.nn.in_top_k
I wonder whether there are similar api in pytorch?

Output:
AFAIK there is no equivalent in_top_k function built into pytorch. It's relatively straightforward to write one. For example
def in_top_k(targets, preds, k):
    topk = preds.topk(k)[1]
    return (targets.unsqueeze(1) == topk).any(dim=1)


Instruction: Initiaized class member in pytorch module

Input:
When declaring a model in pytorch, having a model class member variable declared and initialized mysteriously prevents it from being populated in the constructor. Is this expected? If so, why?
Testing code below, with example models with a component member variable.
The initialization value of the component (e.g. None, a number or a Tensor) does not change the behaviour.
import torch

class Lin1(torch.nn.Module):
    def __init__(self):
        super(Lin1, self).__init__()
        self.component = torch.nn.Linear(100,200)

class Lin2(torch.nn.Module):
    component = None
    def __init__(self):
        super(Lin2, self).__init__()
        self.component = torch.nn.Linear(100,200)

# instantiate and check member component
for cl in [Lin1, Lin2]:
    model = cl()
    print("\nModel:", cl)
    print("Component:", model.component)
    print("It's None?: ", model.component is None)

Model: <class '__main__.Lin1'>
Component: Linear(in_features=100, out_features=200, bias=True)
It's None?:  False

Model: <class '__main__.Lin2'>
Component: None
It's None?:  True

Output:
This happens because nn.Module overwrites __getattr__, and it would only work as you expect if component was not in Lin2.__dict__ (nor in Lin2().__dict__). Since component is a class attribute, it is in Lin2.__dict__ and will be returned as it should.
When you write self.x = nn.Linear(...) or any other nn.Module (or even nn.Buffer or nn.Parameter), x is actually registered in a dictionary called _modules (or _buffers, etc.) In this way, when you ask for self.component, if component is already in the __dict__ of the class or the instance, Python will not call the custom nn.Module's __getattr__().
You can check the source-code of __getattr__ from nn.Module here. A similar discussion was done here. There was also a discussion about changing from __getattr__ to __getattribute__ in PyTorch, but as of now, this is a wontfix issue.


Instruction: Gradient Exploding Using CIFAR-10 Dataset from the Official Website

Input:
I want to train a VGG16 model with Horovod PyTorch on 4 GPUs. Instead of using the CIFAR10 dataset of torch vision.datasets.CIFAR10, I would like to split the dataset on my own. So I downloaded the dataset from the official website and split the dataset. This is how I split the data:
if __name__ == '__main__':
    import pickle

    train_data, train_label = [], []
    test_data, test_label = [], []
    for i in range(1, 6):
        with open('/Users/wangqipeng/Downloads/cifar-10-batches-py/data_batch_{}'.format(i), 'rb') as f:
            b = pickle.load(f, encoding='bytes')
        train_data.extend(b[b'data'].tolist()[:8000])
        train_label.extend(b[b'labels'][:8000])
        test_data.extend(b[b'data'].tolist()[8000:])
        test_label.extend(b[b'labels'][8000:])
    num_train = len(train_data)
    num_test = len(test_data)
    print(num_train, num_test)
    train_data = np.array(train_data)
    test_data = np.array(test_data)
    for i in range(4):
        with open('/Users/wangqipeng/Downloads/train_{}'.format(i), 'wb') as f:
            d = {b'data': train_data[int(0.25 * i * num_train): int(0.25 * (i + 1) * num_train)],
                 b'labels': train_label[int(0.25 * i * num_train): int(0.25 * (i + 1) * num_train)]}
            pickle.dump(d, f)
    with open('/Users/wangqipeng/Downloads/test'.format(i), 'wb') as f:
        d = {b'data': test_data,
             b'labels': test_label}
        pickle.dump(d, f)

However, I found that if I use the dataset that I downloaded from the official website, there will be an exploding gradient problem. I found that the loss will increase and be "nan" after several iterations. This is how I read the dataset:
class DataSet(torch.utils.data.Dataset):
    def __init__(self, path):
        self.dataset = unpickle(path)

    def __getitem__(self, index):
        data = torch.tensor(
            self.dataset[b'data'][index], dtype=torch.float32).resize(3, 32, 32)
        return data, torch.tensor(self.dataset[b'labels'][index])

    def __len__(self):
        return len(self.dataset[b'data'])

train_dataset = DataSet("./cifar10/train_" + str(hvd.rank()))
train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=args.batch_size, sampler=None, **kwargs)

If I print the loss of every iteration, I see something like this:
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 0 / 313 ] loss 7.725658416748047 accuracy 5.46875
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 1 / 313 ] loss 15.312677383422852 accuracy 8.59375
Mon Nov  9 11:28:29 2020[0]<stdout>:epoch 0 iter[ 2 / 313 ] loss 16.333066940307617 accuracy 9.375
Mon Nov  9 11:28:30 2020[0]<stdout>:epoch 0 iter[ 3 / 313 ] loss 15.549728393554688 accuracy 9.9609375
Mon Nov  9 11:28:30 2020[0]<stdout>:epoch 0 iter[ 4 / 313 ] loss 14.090616226196289 accuracy 9.843750298023224
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 5 / 313 ] loss 12.310989379882812 accuracy 9.63541641831398
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 6 / 313 ] loss 11.578919410705566 accuracy 9.15178582072258
Mon Nov  9 11:28:31 2020[0]<stdout>:epoch 0 iter[ 7 / 313 ] loss 13.210229873657227 accuracy 8.7890625
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 8 / 313 ] loss 764.713623046875 accuracy 9.28819477558136
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 9 / 313 ] loss 4.590414250749922e+20 accuracy 8.984375
Mon Nov  9 11:28:32 2020[0]<stdout>:epoch 0 iter[ 10 / 313 ] loss nan accuracy 9.446022659540176
Mon Nov  9 11:28:33 2020[0]<stdout>:epoch 0 iter[ 11 / 313 ] loss nan accuracy 10.09114608168602
Mon Nov  9 11:28:33 2020[0]<stdout>:epoch 0 iter[ 12 / 313 ] loss nan accuracy 10.39663478732109

However, If I use the dataset from torchvision, everything will be fine:
train_dataset = \
    datasets.CIFAR10(args.train_dir, download=True,
                         transform=transforms.Compose([transforms.ToTensor(),
                           transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]))
train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset, num_replicas=hvd.size(), rank=hvd.rank())
train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=args.batch_size, sampler=train_sampler, **kwargs)

There can also be something wrong with the DistributedSampler. But I think the DistributedSampler only serves to split the data. I don't know whether the DistributedSampler can be a reason to this problem.
Is there something wrong with the way I read the CIFAR10 dataset? Or is there something wrong with the way I "reshape" it? Thanks for your help!

Output: Maybe it is because I did not normalize the dataset. Thanks for everyone's help!