Instruction: The results of tf.pow and torch.pow are different

Input:
I am trying to convert PyTorch scripts to Tensorflow.
This is the situation.
>>> a=np.random.rand(500,500)
>>> b=tf.pow(tf.constant(a, tf.float32),2)
>>> c=torch.pow(torch.tensor(a, dtype=torch.float32),2)
>>> np.sum(b.numpy() - c.numpy())
3.2455164e-06

I guess this difference is caused by floating-point representations between the two platform. (I am not sure)
Questions
Do I need to make them exactly the same with each other?
If so how do solve this problem?

Output:
As the comments already say, this is a common issue.
float32 only provides 6 digits of precision, which is what you see, since the difference is smaller than e-06. So as the comments already mentioned this is usual beheviour, if you need more precision you might want to use float64.
There was already a github issue regarding this "problem", which was explained by albanD, who I took this information from.


Instruction: In the PyTorch C++ extension, how can I access a single element in a tensor and convert it to a standard c++ datatype?

Input:
I am writing a c++ extension for pytorch, in which I need to access the elements of a tensor by index, and I also need to convert the element to a standard c++ type. Here is a short example. Suppose I have a 2d tensor a and I need to access a[i][j] and convert it to float.
#include <torch/extension.h>

float get(torch::Tensor a, int i, int j) {
    return a[i][j];
}

The above is put into a file called tensortest.cpp. In another file setup.py I write
from setuptools import setup, Extension
from torch.utils import cpp_extension

setup(name='tensortest',
      ext_modules=[cpp_extension.CppExtension('tensortest_cpp', ['tensortest.cpp'])],
      cmdclass={'build_ext': cpp_extension.BuildExtension})

When I run python setup.py install the compiler reports the following error
running install
running bdist_egg
running egg_info
creating tensortest.egg-info
writing tensortest.egg-info/PKG-INFO
writing dependency_links to tensortest.egg-info/dependency_links.txt
writing top-level names to tensortest.egg-info/top_level.txt
writing manifest file 'tensortest.egg-info/SOURCES.txt'
/home/trisst/.local/lib/python3.8/site-packages/torch/utils/cpp_extension.py:335: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'tensortest.egg-info/SOURCES.txt'
writing manifest file 'tensortest.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'tensortest_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/user/.local/lib/python3.8/site-packages/torch/include -I/home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/user/.local/lib/python3.8/site-packages/torch/include/TH -I/home/user/.local/lib/python3.8/site-packages/torch/include/THC -I/usr/include/python3.8 -c tensortest.cpp -o build/temp.linux-x86_64-3.8/tensortest.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=tensortest_cpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
In file included from /home/user/.local/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:149,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /home/user/.local/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
                 from tensortest.cpp:1:
/home/user/.local/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   84 | #pragma omp parallel for if ((end - begin) >= grain_size)
      |
tensortest.cpp: In function ‘float get(at::Tensor, int, int)’:
tensortest.cpp:4:15: error: cannot convert ‘at::Tensor’ to ‘float’ in return
    4 |  return a[i][j];
      |               ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

What can I do?

Output:
Edited
#include <torch/extension.h>

float get(torch::Tensor a, int i, int j)
{
    return a[i][j].item<float>();
}


Instruction: Why the CUDA memory is not release with torch.cuda.empty_cache()

Input:
On my Windows 10, if I directly create a GPU tensor, I can successfully release its memory.
import torch
a = torch.zeros(300000000, dtype=torch.int8, device='cuda')
del a
torch.cuda.empty_cache()

But if I create a normal tensor and convert it to GPU tensor, I can no longer release its memory.
import torch
a = torch.zeros(300000000, dtype=torch.int8)
a.cuda()
del a
torch.cuda.empty_cache()

Why this is happening.

Output:
At least in Ubuntu, your script does not release memory when it is run in the interactive shell and works as expected when running as a script. I think there are some reference issues in the in-place call. The following will work in both the interactive shell and as a script.
import torch
a = torch.zeros(300000000, dtype=torch.int8)
a = a.cuda()
del a
torch.cuda.empty_cache()


Instruction: pytorch does not save pre-trained model weights loaded and the parts of it in the final model

Input:
I am currently working on pre-trained model on CIFAR-10 on my data, have removed the final fc layer of the model and have appended my own fc layer and  softmax. There are seven networks which each of them are same as pre-trained part and are combined using appended fc layer. The following is pre-trained Network code :
class Bottleneck(nn.Module):
    def __init__(self, inplanes, expansion=4, growthRate=12, dropRate=0):
        super(Bottleneck, self).__init__()
        planes = expansion * growthRate
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.conv2 = nn.Conv2d(planes, growthRate, kernel_size=3,
                               padding=1, bias=False)
        self.relu = nn.ReLU(inplace=True)
        self.dropRate = dropRate

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        out = self.bn2(out)
        out = self.relu(out)
        out = self.conv2(out)
        if self.dropRate > 0:
            out = F.dropout(out, p=self.dropRate, training=self.training)

        out = torch.cat((x, out), 1)

        return out

class BasicBlock(nn.Module):
    def __init__(self, inplanes, expansion=1, growthRate=12, dropRate=0):
        super(BasicBlock, self).__init__()
        planes = expansion * growthRate
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, growthRate, kernel_size=3,
                               padding=1, bias=False)
        self.relu = nn.ReLU(inplace=True)
        self.dropRate = dropRate

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        if self.dropRate > 0:
            out = F.dropout(out, p=self.dropRate, training=self.training)

        out = torch.cat((x, out), 1)

        return out

class Transition(nn.Module):
    def __init__(self, inplanes, outplanes):
        super(Transition, self).__init__()
        self.bn1 = nn.BatchNorm2d(inplanes)
        self.conv1 = nn.Conv2d(inplanes, outplanes, kernel_size=1,
                               bias=False)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        out = self.bn1(x)
        out = self.relu(out)
        out = self.conv1(out)
        out = F.avg_pool2d(out, 2)
        return out

class DenseNet(nn.Module):

    def __init__(self, depth = 22, block = Bottleneck,
        dropRate = 0, num_classes = 10, growthRate = 12, compressionRate = 2):
        super(DenseNet, self).__init__()

        assert (depth - 4) % 3 == 0, 'depth should be 3n+4'
        n = (depth - 4) / 3 if block == BasicBlock else (depth - 4) // 6

        self.growthRate = growthRate
        self.dropRate = dropRate

        # self.inplanes is a global variable used across multiple
        # helper functions
        self.inplanes = growthRate * 2
        self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size = 3, padding = 1,
                               bias = False)
        self.dense1 = self._make_denseblock(block, n)
        self.trans1 = self._make_transition(compressionRate)
        self.dense2 = self._make_denseblock(block, n)
        self.trans2 = self._make_transition(compressionRate)
        self.dense3 = self._make_denseblock(block, n)
        self.bn = nn.BatchNorm2d(self.inplanes)
        self.relu = nn.ReLU(inplace=True)
        self.avgpool = nn.AvgPool2d(8)
        #self.fc = nn.Linear(self.inplanes, num_classes)

        # Weight initialization
#         for m in self.modules():
#             if isinstance(m, nn.Conv2d):
#                 n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
#                 m.weight.data.normal_(0, math.sqrt(2. / n))
#             elif isinstance(m, nn.BatchNorm2d):
#                 m.weight.data.fill_(1)
#                 m.bias.data.zero_()

    def _make_denseblock(self, block, blocks):
        layers = []
        for i in range(blocks):
            # Currently we fix the expansion ratio as the default value
            layers.append(block(self.inplanes, growthRate = self.growthRate, dropRate=self.dropRate))
            self.inplanes += self.growthRate

        return nn.Sequential(*layers)

    def _make_transition(self, compressionRate):
        inplanes = self.inplanes
        outplanes = int(math.floor(self.inplanes // compressionRate))
        self.inplanes = outplanes
        return Transition(inplanes, outplanes)

    def forward(self, x):
        x = self.conv1(x)

        x = self.trans1(self.dense1(x))
        x = self.trans2(self.dense2(x))
        x = self.dense3(x)
        x = self.bn(x)
        x = self.relu(x)

        x = self.avgpool(x)
        #x = x.view(x.size(0), -1)
        #x = self.fc(x)

        return x

    def getParams(self, paramName):
        if paramName == 'inplanes':
            return self.inplanes
        elif paramName == 'growthRate':
            return self.growthRate
        elif paramName == 'dropRate':
            return self.dropRate

def densenet(**kwargs):
    """
    Constructs a DenseNet model.
    """
    return DenseNet(**kwargs)

and the next following is my code:
class Network(nn.Module):

    def __init__(self,pretrained_dict, num_classes = 6, num_channels = 7,
                 expansion = 4, depth = 100, growthRate = 12, dropRate = 0):

        super(Network, self).__init__()

        self.num_channels = num_channels

        # creating 7 channels networks
        self.channels_dnsnets = []

        for ch in range(self.num_channels):
#             print(ch)

            d = densenet(depth = depth)
            d_dict = d.state_dict()

            # 1. filter out unnecessary keys
            pretrained_dict2 = {k[7:]: v for k, v in pretrained_dict.items() if k[7:] in d_dict}
#             print('d_dict_keys :')
#             print(d_dict.keys())
#             print('*'*50)
#             print('pretrained_dict2.keys:')
#             print(pretrained_dict2.keys())
#             print('*'*50)

            # 2. overwrite entries in the existing state dict
            d_dict.update(pretrained_dict2)

            # 3. load the new state dict
            d.load_state_dict(pretrained_dict2)

            # freeze the layers of densenet
            for param in d.parameters():
                param.requires_grad = False

            self.channels_dnsnets.append(d)

        self.inplanes = self.channels_dnsnets[0].getParams(paramName = 'inplanes')
        self.fc = nn.Linear(self.inplanes * self.num_channels, num_classes)
        self.softmax = nn.Softmax(dim = 1)

    def forward(self, x):

        batch_size, channels, ht, wd, in_channels = x.shape
        x = np.reshape(x,(batch_size,channels,in_channels,ht,wd))

        out = []

        for num in range(self.num_channels):
            temp_out = self.channels_dnsnets[0](x[:,num,:])
            temp_out = temp_out.view(temp_out.size(0),-1)
#             print(temp_out.shape)
#             print('*' * 50)
            out.append(temp_out)

        out = torch.stack(out,dim = 1)
#         print(out.shape)
        out = out.view(out.size(0),-1)
        out = self.fc(out)
        out = self.softmax(out)
        return out

I am setting the optimizer as :
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr = lr,
                                 betas = (0.9, 0.999), eps = 1e-08, weight_decay = wd, amsgrad = False)

However, whenever I save the model the list of densenets and their weights are not saved and only fc layer and softmax layer weights are saved. Is there anything problematic with code? I am new to pytorch.

Output:
The problem is that self.channels_dnsnets is just a list and will not be part of the state_dict. Only self.fc and self.softmax will be registered into the Module. The simplest change would be to define it like this:
self.channels_dnsnets = nn.ModuleList()


Instruction: Pytorch Softmax giving nans and negative values as output

Input:
I am using softmax at the end of my model.
However after some training softmax is giving negative probability.In some situations I have encountered nans as probability as well.
one solution i found on searching is to use normalized softmax…however I can not find any pytorch imlpementaion for this.
Can someone please help to let know if there is a normalized softmax available or how to achieve this so that forward and backward propagations are smooth.
Please note that I am already using torch.nn.utils.clip_grad_norm_(model.parameters(), 40) to avoid exploding gradients
I am using pytorch 1.6.0

Output:
Softmax will always return positive results, but it will keep track of other results:
m = nn.Softmax(dim=1)
input = torch.randn(2, 3)
print(input)
output = m(input)
output

Out:
tensor([[ 0.0983,  0.4150, -1.1342],
        [ 0.3411,  0.5553,  0.0182]])

tensor([[0.3754, 0.5152, 0.1094],
        [0.3375, 0.4181, 0.2444]])

You are tracking the rows.
Note how for
0.0983,  0.4150, -1.1342
You will get
0.3411,  0.5553,  0.0182
Saying that 0.4150 is the biggest value.
The hard max (as we know this is max()) will just return the maximum value.
So, if you have negative results for softmax this is not possible, you may have hit some implementation failure.


Instruction: How to get columns from 2D tensor list in Pytorch

Input:
I have a one 2D list that contains tensors like:
[
 [tensor([-0.0705,  1.2019]), tensor([[0]]), tensor([-1.3865], dtype=torch.float64), tensor([-0.0744,  1.1880]), tensor([False])],
 [tensor([-0.0187,  1.3574]), tensor([[2]]), tensor([0.3373], dtype=torch.float64), tensor([-0.0221,  1.3473]), tensor([False])],
 [....] ]

The outer list contains 64 little list. One little list contains 5 different tensor elements.
And I want to get first elements of inner lists like tensor([-0.0705,  1.2019]) and tensor([-0.0187,  1.3574]) and create tensor like 64x2 to feed my neural  net.
How can I do this in the fastest way?
Thanks

Output:
Use list comprehension
[item[0] for item in your_list]

Example:
li = [[tensor([-0.0705,  1.2019]), tensor([[0]]), tensor([-1.3865], dtype=torch.float64), tensor([-0.0744,  1.1880]), tensor([False])],
     [tensor([-0.0187,  1.3574]), tensor([[2]]), tensor([0.3373], dtype=torch.float64), tensor([-0.0221,  1.3473]), tensor([False])]]

[item[0] for item in li]
[tensor([-0.0705,  1.2019]), tensor([-0.0187,  1.3574])]


Instruction: Getting Unknown type annotation error when JIT saving pytorch model

Input:
When JIT saving “model.pt” of a complex pytorch model with many custom classes, I am encountering the error that pytorch doesn’t know the type annotation of one of those custom classes. In other words, the following code (drastically summarized from original) fails on the seventh line:
import torch
from gan import Generator
from gan.blocks import SpadeBlock

generator = Generator()
generator.load_weights("path/to/weigts")
jitted = torch.jit.script(generator)
torch.jit.save(jitted, "model.pt")

Error:
Traceback (most recent call last):
  File "pth2onnx.py", line 72, in <module>
    to_torch_jit(generator)
  File "pth2onnx.py", line 24, in to_torch_jit
    jitted = torch.jit.script(generator)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/__init__.py", line 1516, in script
    return torch.jit._recursive.create_script_module(obj, torch.jit._recursive.infer_methods_to_compile)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 310, in create_script_module
    concrete_type = concrete_type_store.get_or_create_concrete_type(nn_module)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 269, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 138, in infer_concrete_type_builder
    sub_concrete_type = concrete_type_store.get_or_create_concrete_type(item)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 269, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 138, in infer_concrete_type_builder
    sub_concrete_type = concrete_type_store.get_or_create_concrete_type(item)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 269, in get_or_create_concrete_type
    concrete_type_builder = infer_concrete_type_builder(nn_module)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 126, in infer_concrete_type_builder
    attr_type = infer_type(name, item)
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/_recursive.py", line 99, in infer_type
    attr_type = torch.jit.annotations.ann_to_type(class_annotations[name], _jit_internal.fake_range())
  File "/home/a.nieuwland/.conda/envs/python3.6/lib/python3.6/site-packages/torch/jit/annotations.py", line 303, in ann_to_type
    raise ValueError("Unknown type annotation: '{}'".format(ann))
ValueError: Unknown type annotation: '<class 'gan.blocks.SpadeBlock'>'

The type it complains about is indeed a class we ourselves have programmed and used in the loaded Generator. I would appreciate pointers on what could cause this or how to investigate this!
I tried the following:

explicitly importing SpadeBlock in the script that calls torch.jit.script
ensured it inherits from nn.Module (as does Generator)
ensured the gan package is installed, using pip install --user -e

Any ideas? Thanks in advance!

Output:
The issue turned out to be the that I was using class variable names which got mangled. Example:
class Generator(nn.Module):
    __main: nn.Module

The two leading underscores are the cause. Changing them to a single or no underscore. resolves the issue.
class Generator(nn.Module):
    main: nn.Module


Instruction: PyTorch: Dropout (?) causes different model convergence for training+validation V. training-only

Input:
We are facing a very strange issue. We tested the exact same model into two different “execution” settings. In the first case, given a certain amount of epochs, we train using mini-batches for one epoch, and thereafter we test on the validation set following the same criteria. Then, we go for the next epoch. Clearly, before each training epoch, we use model.train(), and before validation we turn on model.eval().
Then we take the exact same model (same init, same dataset, same epochs, etc.) and we just train it without validation after each epoch.
Just looking at performance on training set, we observed that, even if we fixed all seeds, the two training procedures evolve differently and produce quite different metrics results (losses, accuracy, and so on). Specifically, the training-only procedure is less performing.
We also observe the following things:

It is not a reproducibility issue, because multiple executions of the
same procedure produce exactly the same results (and this is
intended);
Removing the dropout, it appears that the problem vanishes;
Batchnorm1d layer, that still has different behaviours between
training and evaluation, seems to work properly;
The issue still happens if we move from training onto TPUs to CPUs.
We are working and tried Pythorch 1.6, Pythorch nightly, XLA 1.6.

We quite lost one full day in trying to tackle this issue (and no, we cannot avoid using dropout). Does anyone have any idea about how to solve this fact?
Thank you very much!
p.s. Here the code employed for the training (on CPU).
def sigmoid(x):
    return 1 / (1 + torch.exp(-x))

def _run(model, EPOCHS, training_data_in, validation_data_in=None):

    def train_fn(train_dataloader, model, optimizer, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.

        model.train()

        for batch_idx, (ecg, spo2, labels) in enumerate(train_dataloader, 1):

            optimizer.zero_grad()

            outputs = model(ecg)

            loss = criterion(outputs, labels)

            loss.backward() # calculate the gradients
            torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step() # update the network weights

            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model

            running_accuracy += (predicted == labels).sum().item() / labels.size(0)

            fp = ((predicted - labels) == 1.).sum().item()
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn

        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }

        return retval

    def valid_fn(valid_dataloader, model, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.

        model.eval()

        for batch_idx, (ecg, spo2, labels) in enumerate(valid_dataloader, 1):

            outputs = model(ecg)

            loss = criterion(outputs, labels)

            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model

            running_accuracy += (predicted == labels).sum().item() / labels.size(0)

            fp = ((predicted - labels) == 1.).sum().item()
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn

        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }

        return retval

    # Defining data loaders

    train_dataloader = torch.utils.data.DataLoader(training_data_in, batch_size=BATCH_SIZE, shuffle=True, num_workers=1)

    if validation_data_in != None:
        validation_dataloader = torch.utils.data.DataLoader(validation_data_in, batch_size=BATCH_SIZE, shuffle=False, num_workers=1)

    # Defining the loss function
    criterion = nn.BCEWithLogitsLoss()

    # Defining the optimizer
    import torch.optim as optim
    optimizer = optim.AdamW(model.parameters(), lr=3e-4, amsgrad=False, eps=1e-07)

    # Training code

    metrics_history = {"loss":[], "accuracy":[], "precision":[], "recall":[], "f1":[], "specificity":[], "accuracy_bis":[], "tp":[], "tn":[], "fp":[], "fn":[],
                    "val_loss":[], "val_accuracy":[], "val_precision":[], "val_recall":[], "val_f1":[], "val_specificity":[], "val_accuracy_bis":[], "val_tp":[], "val_tn":[], "val_fp":[], "val_fn":[],}

    train_begin = time.time()
    for epoch in range(EPOCHS):
        start = time.time()

        print("EPOCH:", epoch+1)

        train_metrics = train_fn(train_dataloader=train_dataloader,
                                model=model,
                                optimizer=optimizer,
                                criterion=criterion)

        metrics_history["loss"].append(train_metrics["loss"])
        metrics_history["accuracy"].append(train_metrics["accuracy"])
        metrics_history["tp"].append(train_metrics["tp"])
        metrics_history["tn"].append(train_metrics["tn"])
        metrics_history["fp"].append(train_metrics["fp"])
        metrics_history["fn"].append(train_metrics["fn"])

        precision = train_metrics["tp"] / (train_metrics["tp"] + train_metrics["fp"]) if train_metrics["tp"] > 0 else 0
        recall = train_metrics["tp"] / (train_metrics["tp"] + train_metrics["fn"]) if train_metrics["tp"] > 0 else 0
        specificity = train_metrics["tn"] / (train_metrics["tn"] + train_metrics["fp"]) if train_metrics["tn"] > 0 else 0
        f1 = 2*precision*recall / (precision + recall) if precision*recall > 0 else 0
        metrics_history["precision"].append(precision)
        metrics_history["recall"].append(recall)
        metrics_history["f1"].append(f1)
        metrics_history["specificity"].append(specificity)

        if validation_data_in != None:
            # Calculate the metrics on the validation data, in the same way as done for training
            with torch.no_grad(): # don't keep track of the info necessary to calculate the gradients

                val_metrics = valid_fn(valid_dataloader=validation_dataloader,
                                    model=model,
                                    criterion=criterion)

                metrics_history["val_loss"].append(val_metrics["loss"])
                metrics_history["val_accuracy"].append(val_metrics["accuracy"])
                metrics_history["val_tp"].append(val_metrics["tp"])
                metrics_history["val_tn"].append(val_metrics["tn"])
                metrics_history["val_fp"].append(val_metrics["fp"])
                metrics_history["val_fn"].append(val_metrics["fn"])

                val_precision = val_metrics["tp"] / (val_metrics["tp"] + val_metrics["fp"]) if val_metrics["tp"] > 0 else 0
                val_recall = val_metrics["tp"] / (val_metrics["tp"] + val_metrics["fn"]) if val_metrics["tp"] > 0 else 0
                val_specificity = val_metrics["tn"] / (val_metrics["tn"] + val_metrics["fp"]) if val_metrics["tn"] > 0 else 0
                val_f1 = 2*val_precision*val_recall / (val_precision + val_recall) if val_precision*val_recall > 0 else 0
                metrics_history["val_precision"].append(val_precision)
                metrics_history["val_recall"].append(val_recall)
                metrics_history["val_f1"].append(val_f1)
                metrics_history["val_specificity"].append(val_specificity)

            print("  > Training/validation loss:", round(train_metrics['loss'], 4), round(val_metrics['loss'], 4))
            print("  > Training/validation accuracy:", round(train_metrics['accuracy'], 4), round(val_metrics['accuracy'], 4))
            print("  > Training/validation precision:", round(precision, 4), round(val_precision, 4))
            print("  > Training/validation recall:", round(recall, 4), round(val_recall, 4))
            print("  > Training/validation f1:", round(f1, 4), round(val_f1, 4))
            print("  > Training/validation specificity:", round(specificity, 4), round(val_specificity, 4))
        else:
            print("  > Training loss:", round(train_metrics['loss'], 4))
            print("  > Training accuracy:", round(train_metrics['accuracy'], 4))
            print("  > Training precision:", round(precision, 4))
            print("  > Training recall:", round(recall, 4))
            print("  > Training f1:", round(f1, 4))
            print("  > Training specificity:", round(specificity, 4))

        print("Completed in:", round(time.time() - start, 1), "seconds \n")

    print("Training completed in:", round((time.time()- train_begin)/60, 1), "minutes")

    # Save the model weights
    torch.save(model.state_dict(), './nnet_model.pt')

    # Save the metrics history
    torch.save(metrics_history, 'training_history')

And here is the function that initializes the model and set the seeds, called before each execution of the code of "_run":
def reinit_model():
    torch.manual_seed(42)
    np.random.seed(42)
    random.seed(42)
    net = Net() # the model
    return net

Output:
Ok, I found the issue.
The problem is determined by the fact that, apparently, running the evaluation some random seeds are changed, and this affects the training phase.
The solution is thus as follows:

at the beginning of function "_run()", set all seeds states to the desired value, e.g., 42. Then, save those seeds to disk.
at the beginning of function "train_fn()", read the seeds states from disk, and set them
at the end of function "train_fn()", save the seeds states to disk

For instance, running on TPU with XLA, the following instructions have to be used:

at the beginning of function "_run()": xm.set_rng_state(42), xm.save(xm.get_rng_state(), 'xm_seed')
at the beginning of function "train_fn()": xm.set_rng_state(torch.load('xm_seed'), device=device)  (you can also print here the seed for verification purposes with xm.master_print(xm.get_rng_state())
at the end of function "train_fn_()": xm.save(xm.get_rng_state(), 'xm_seed')


Instruction: Why Pytorch autograd need another vector to backward instead of computing Jacobian?

Input:
To perform backward in Pytorch, we can use an optional parameter y.backward(v) to compute the Jacobian matrix multiplied by v:
x = torch.randn(3, requires_grad=True)
y = x * 2

v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)

print(x.grad)

I think that costs the same to compute the Jacobian matrix, because each node in the AD graph which is necessary to compute the Jacobian matrix is still computed. So why not Pytorch doesn't want to give us the Jacobian matrix?

Output:
When you call backward() PyTorch udpdates the grad of each learnable parameter with the gradient of some loss function L w.r.t to that parameter. It has been designed with Gradient Descent [GD] (and its variants) in mind. Once the gradient has been computed you can update each parameter with x = x - learning_rate * x.grad. Indeed in the background the Jacobian has to be computed but it is not what one needs (generally) when applying GD optimization. The vector [0.1, 1.0, 0.0001] lets you reduce the output to a scalar so that x.grad will be a vector (and not a matrix, in case you do not reduce), and hence GD is well defined. You could, however, obtain the Jacobian using backward with one-hot vectors. For example, in this case:
x = torch.randn(3, requires_grad=True)
y = x * 2
J = torch.zeros(x.shape[0],x.shape[0])
for i in range(x.shape[0]):
    v = torch.tensor([1 if j==i else 0 for j in range(x.shape[0])], dtype=torch.float)
    y.backward(v, retain_graph=True)
    J[:,i] = x.grad
    x.grad.zero_()
print(J)


Instruction: How to make a 3-d like visualization of a CNN model developed using pytorch?

Input:
Currently, I have a CNN model that I developed in Pytorch. I have used hiddenlayer package to create an image like shown in image 1. But, I want to create an image of the model that should look like something in image 2. Is there any package that I can use to achieve this?
image 1:

image 2:

Output: I personally use figma.com and "draw" it myself, but if you want to create it automatically you should check out this github repository, you might find a nice tool.


Instruction: Converting Tensorflow code to Pytorch - performance metrics very different

Input:
I have converted a tensorflow code for timeseries analysis to pytorch and performance difference is very high, in fact pytorch layers cannot account for seasonality at all. It feels like I must be missing something important.
Please help find where the pytorch code is lacking  that the learning is not up to the par. I noticed that loss values has high jumps when it encounters the season change and is not learning that. With the same layers, nodes and every other thing, I imagined the performance to be close.
# tensorflow code
window_size = 20
batch_size = 32
shuffle_buffer_size = 1000

def windowed_dataset(series, window_size, batch_size, shuffle_buffer):
  dataset = tf.data.Dataset.from_tensor_slices(series)
  dataset = dataset.window(window_size + 1, shift=1, drop_remainder=True)
  dataset = dataset.flat_map(lambda window: window.batch(window_size + 1))
  dataset = dataset.shuffle(shuffle_buffer).map(lambda window: (window[:-1], window[-1]))
  dataset = dataset.batch(batch_size).prefetch(1)
  return dataset

dataset = windowed_dataset(x_train, window_size, batch_size, shuffle_buffer_size)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, input_shape=[window_size], activation="relu"),
    tf.keras.layers.Dense(10, activation="relu"),
    tf.keras.layers.Dense(1)
])

model.compile(loss="mse", optimizer=tf.keras.optimizers.SGD(lr=1e-6, momentum=0.9))
model.fit(dataset,epochs=100,verbose=0)
forecast = []
for time in range(len(series) - window_size):
  forecast.append(model.predict(series[time:time + window_size][np.newaxis]))

forecast = forecast[split_time-window_size:]
results = np.array(forecast)[:, 0, 0]

plt.figure(figsize=(10, 6))

plot_series(time_valid, x_valid)
plot_series(time_valid, results)

tf.keras.metrics.mean_absolute_error(x_valid, results).numpy()

# pytorch code
window_size = 20
batch_size = 32
shuffle_buffer_size = 1000

class tsdataset(Dataset):
  def __init__(self, series, window_size):
    self.series = series
    self.window_size = window_size
    self.dataset, self.labels = self.preprocess()

  def preprocess(self):
    series = self.series
    final, labels = [], []
    for i in range(len(series)-self.window_size):
      final.append(np.array(series[i:i+window_size]))
      labels.append(np.array(series[i+window_size]))
    return torch.from_numpy(np.array(final)), torch.from_numpy(np.array(labels))

  def __getitem__(self,index):
    # print(self.dataset[index], self.labels[index], index)
    return self.dataset[index], self.labels[index]

  def __len__(self):
    return len(self.dataset)

train_dataset = tsdataset(x_train, window_size)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

class tspredictor(nn.Module):
  def __init__(self, window_size, out1, out2, out3):
    super(tspredictor, self).__init__()
    self.l1 = nn.Linear(window_size, out1)
    self.l2 = nn.Linear(out1, out2)
    self.l3 = nn.Linear(out2, out3)

  def forward(self,seq):
    l1 = F.relu(self.l1(seq))
    l2 = F.relu(self.l2(l1))
    l3 = self.l3(l2)
    return l3

model = tspredictor(20, 100,10,1)
loss_function = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=1e-6, momentum=0.9)

for epoch in range(100):
    for t,l in train_dataloader:
      model.zero_grad()
      tag_scores = model(t)
      loss = loss_function(tag_scores, l)
      loss.backward()
      optimizer.step()
    # print("Epoch is {}, loss is {}".format(epoch, loss.data))

forecast = []
for time in range(len(series) - window_size):
    prediction = model(torch.from_numpy(series[time:time + window_size][np.newaxis]))
    forecast.append(prediction)

forecast = forecast[split_time-window_size:]

results = np.array(forecast)

plt.figure(figsize=(10, 6))
plot_series(time_valid, x_valid)
plot_series(time_valid, results)

To generate data, you can use:
def plot_series(time, series, format="-", start=0, end=None):
    plt.plot(time[start:end], series[start:end], format)
    plt.xlabel("Time")
    plt.ylabel("Value")
    plt.grid(False)

def trend(time, slope=0):
    return slope * time

def seasonal_pattern(season_time):
    """Just an arbitrary pattern, you can change it if you wish"""
    return np.where(season_time < 0.1,
                    np.cos(season_time * 6 * np.pi),
                    2 / np.exp(9 * season_time))

def seasonality(time, period, amplitude=1, phase=0):
    """Repeats the same pattern at each period"""
    season_time = ((time + phase) % period) / period
    return amplitude * seasonal_pattern(season_time)

def noise(time, noise_level=1, seed=None):
    rnd = np.random.RandomState(seed)
    return rnd.randn(len(time)) * noise_level

time = np.arange(10 * 365 + 1, dtype="float32")
baseline = 10
series = trend(time, 0.1)
baseline = 10
amplitude = 40
slope = 0.005
noise_level = 3

# Create the series
series = baseline + trend(time, slope) + seasonality(time, period=365, amplitude=amplitude)
# Update with noise
series += noise(time, noise_level, seed=51)

split_time = 3000
time_train = time[:split_time]
x_train = series[:split_time]
time_valid = time[split_time:]
x_valid = series[split_time:]

Output:
There was a broadcasting issue in the loss function. Changing to the loss to one below fixes it:
loss = loss_function(tag_scores, l.view(-1,1))


Instruction: PyTorch RNN is more efficient with `batch_first=False`?

Input:
In machine translation, we always need to slice out the first timestep (the SOS token) in the annotation and prediction.
When using batch_first=False, slicing out the first timestep still keeps the tensor contiguous.
import torch
batch_size = 128
seq_len = 12
embedding = 50

# Making a dummy output that is `batch_first=False`
batch_not_first = torch.randn((seq_len,batch_size,embedding))
batch_not_first = batch_first[1:].view(-1, embedding) # slicing out the first time step

However, if we use batch_first=True, after slicing, the tensor is no longer contiguous. We need to make it contiguous before we can do different operations such as view.
batch_first = torch.randn((batch_size,seq_len,embedding))
batch_first[:,1:].view(-1, embedding) # slicing out the first time step

output>>>
"""
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-a9bd590a1679> in <module>
----> 1 batch_first[:,1:].view(-1, embedding) # slicing out the first time step

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
"""

Does that mean batch_first=False is better, at least, in the context of machine translation? Since it saves us from doing the contiguous() step. Is there any cases that batch_first=True works better?

Output:
Performance
There doesn't seem to be a considerable difference between batch_first=True and batch_first=False. Please see the script below:
import time

import torch

def time_measure(batch_first: bool):
    torch.cuda.synchronize()
    layer = torch.nn.RNN(10, 20, batch_first=batch_first).cuda()
    if batch_first:
        inputs = torch.randn(100000, 7, 10).cuda()
    else:
        inputs = torch.randn(7, 100000, 10).cuda()

    start = time.perf_counter()

    for chunk in torch.chunk(inputs, 100000 // 64, dim=0 if batch_first else 1):
        _, last = layer(chunk)

    return time.perf_counter() - start

print(f"Time taken for batch_first=False: {time_measure(False)}")
print(f"Time taken for batch_first=True: {time_measure(True)}")

On my device (GTX 1050 Ti), PyTorch 1.6.0 and CUDA 11.0 here are the results:
Time taken for batch_first=False: 0.3275816479999776
Time taken for batch_first=True: 0.3159054920001836

(and it varies either way so nothing conclusive).
Code readability
batch_first=True is simpler when you want to use other PyTorch layers which require batch as 0th dimension (which is the case for almost all torch.nn layers like torch.nn.Linear).
In this case you would have to permute returned tensor anyway if batch_first=False was specified.
Machine translation
It should be better as the tensor is contiguous all the time and no copy of data has to be done. It also looks cleaner to slice using [1:] instead of [:,1:].


Instruction: Why do we need the custom dataset class and use of _getitem_ method in NLP, BERT fine tuning etc

Input:
I am a newbie in NLP and has been studying the usage of BERT for NLP tasks. In many notebooks, I See that a custom dataset class is defined and getitem method is defined (along with len).
Tweetdataset class in this notebook - https://www.kaggle.com/abhishek/roberta-inference-5-folds
and text_Dataset class in this notebook - https://engineering.wootric.com/when-bert-meets-pytorch
Can some one please explain the reason, need for defining the custom dataset class and the getitem (and len) method. thank you

Output:
It is a recommended abstraction in pytorch to define datasets by inheriting torch.utils.data.Dataset. Those objects define how many elements are there (__len__ method) and how to get a single item via specified index (__getitem__(index)).
Its source code:
class Dataset(object):
    def __getitem__(self, index):
        raise NotImplementedError

    def __add__(self, other):
        return ConcatDataset([self, other])

So it's basically a thin wrapper which adds possibility to concatenate two Dataset objects. For readability and API compatibility you should inherit from it (unlike the one provided in kaggle).
You can read more about PyTorch's data functionality here


Instruction: AttributeError : 'tuple' has no attribute 'to'

Input:
I am writing this Image Classifier and I have defined the loaders but getting this mistake and I have no clue about it.
I have defined the train loader, for a better explanation I tried this
for ina,lab in train_loader:
    print(type(ina))
    print(type(lab))

and I got
<class 'torch.Tensor'>
<class 'tuple'>

Now, For training of the model, I did
def train_model(model,optimizer,n_epochs,criterion):
    start_time = time.time()
    for epoch in range(1,n_epochs-1):
        epoch_time = time.time()
        epoch_loss = 0
        correct = 0
        total = 0
        print( "Epoch {}/{}".format(epoch,n_epochs))

        model.train()

        for inputs,labels in train_loader:
            inputs = inputs.to(device)
            labels  = labels.to(device)
            optimizer.zero_grad()
            output = model(inputs)
            loss = criterion(output,labels)
            loss.backward()
            optimizer.step()
            epoch_loss +=loss.item()
            _,pred =torch.max(output,1)
            correct += (pred.cpu()==label.cpu()).sum().item()
            total +=labels.shape[0]

        acc = correct/total

and I got the error:
Epoch 1/15
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-36-fea243b3636a> in <module>
----> 1 train_model(model=arch, optimizer=optim, n_epochs=15, criterion=criterion)

<ipython-input-34-b53149a4bac0> in train_model(model, optimizer, n_epochs, criterion)
     12         for inputs,labels in train_loader:
     13             inputs = inputs.to(device)
---> 14             labels  = labels.to(device)
     15             optimizer.zero_grad()
     16             output = model(inputs)

AttributeError: 'tuple' object has no attribute 'to'

If you want anything more, please tell me!
Thanks
Edit: The label looks like this.
This was an Image Classification between Bee and Wasp. It also contains insects and non insects
('wasp', 'wasp', 'insect', 'insect', 'wasp', 'insect', 'insect', 'wasp', 'wasp', 'bee', 'insect', 'insect', 'other', 'bee', 'other', 'wasp', 'other', 'wasp', 'bee', 'bee', 'wasp', 'wasp', 'wasp', 'wasp', 'bee', 'wasp', 'wasp', 'other', 'bee', 'wasp', 'bee', 'bee')
('wasp', 'wasp', 'insect', 'bee', 'other', 'wasp', 'insect', 'wasp', 'insect', 'insect', 'insect', 'wasp', 'wasp', 'insect', 'wasp', 'wasp', 'wasp', 'bee', 'wasp', 'wasp', 'insect', 'insect', 'wasp', 'wasp', 'bee', 'wasp', 'insect', 'bee', 'bee', 'insect', 'insect', 'other')

Output:
It literally means that the the tuple class in Python doesn't have a method called to. Since you're trying to put your labels onto your device, just do labels = torch.tensor(labels).to(device).
If you don't want to do this, you can change the way the DataLoader works by making it return your labels as a PyTorch tensor rather than a tuple.
Edit
Since the labels seem to be strings, I would convert them to one-hot encoded vectors first:
>>> import torch
>>> labels_unique = set(labels)
>>> keys = {key: value for key, value in zip(labels_unique, range(len(labels_unique)))}
>>> labels_onehot = torch.zeros(size=(len(labels), len(keys)))
>>> for idx, label in enumerate(labels_onehot):
...     labels_onehot[idx][keys[label]] = 1
...
>>> labels_onehot = labels.to(device)

I'm shooting a bit in the dark here because I don't know the details exactly, but yeah strings won't work with tensors.


Instruction: torch.nn.functional vs torch.nn - Pytorch

Input:
While adding loss in Pytorch, I have the same function in torch.nn.Functional as well as in torch.nn. what is the difference ?
 torch.nn.CrossEntropyLoss() and torch.nn.functional.cross_entropy

Output:
Putting same text from PyTorch discussion forum @Alban D has given answer to similar question.
F.cross entropy vs torch.nn.Cross_Entropy_Loss

There isn’t much difference for losses.
The main difference between the nn.functional.xxx and the nn.Xxx is that one has a state and one does not.
This means that for a linear layer for example, if you use the functional version, you will need to handle the weights yourself (including passing them to the optimizer or moving them to the gpu) while the nn.Xxx version will do all of that for you with .parameters() or .to(device).
For loss functions, as no parameters are needed (in general), you won’t find much difference. Except for example, if you use cross entropy with some weighting between your classes, using the nn.CrossEntropyLoss() module, you will give your weights only once while creating the module and then use it. If you were using the functional version, you will need to pass the weights every single time you will use it.


Instruction: what is the pytorch equivalent of a tensorflow linear regression?

Input:
I am learning pytorch, that to do a basic linear regression on this data created this way here:
from sklearn.datasets import make_regression

x, y = make_regression(n_samples=100, n_features=1, noise=15, random_state=42)
y = y.reshape(-1, 1)
print(x.shape, y.shape)

plt.scatter(x, y)

I know that using tensorflow this code can solve:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=1, activation='linear', input_shape=(x.shape[1], )))

model.compile(optimizer=tf.keras.optimizers.SGD(lr=0.05), loss='mse')

hist = model.fit(x, y, epochs=15, verbose=0)

but I need to know what the pytorch equivalent would be like, what I tried to do was this:
# Model Class
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear = nn.Linear(1,1)

    def forward(self, x):
        x = self.linear(x)
        return x

    def predict(self, x):
        return self.forward(x)

model = Net()

loss_fn = F.mse_loss
opt = torch.optim.SGD(modelo.parameters(), lr=0.05)

# Funcao para treinar
def fit(num_epochs, model, loss_fn, opt, train_dl):

    # Repeat for given number of epochs
    for epoch in range(num_epochs):

        # Train with batches of data
        for xb, yb in train_dl:

            # 1. Generate predictions
            pred = model(xb)

            # 2. Calculate Loss
            loss = loss_fn(pred, yb)

            # 3. Campute gradients
            loss.backward()

            # 4. Update parameters using gradients
            opt.step()

            # 5. Reset the gradients to zero
            opt.zero_grad()

        # Print the progress
        if (epoch+1) % 10 == 0:
            print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

# Training
fit(200, model, loss_fn, opt, data_loader)

But the model doesn't learn anything, I don't know what I can do anymore.
The input/output dimensions is (1/1)

Output:
Dataset
First of all, you should define torch.utils.data.Dataset
import torch
from sklearn.datasets import make_regression

class RegressionDataset(torch.utils.data.Dataset):
    def __init__(self):
        data = make_regression(n_samples=100, n_features=1, noise=0.1, random_state=42)
        self.x = torch.from_numpy(data[0]).float()
        self.y = torch.from_numpy(data[1]).float()

    def __len__(self):
        return len(self.x)

    def __getitem__(self, index):
        return self.x[index], self.y[index]

It converts numpy data to PyTorch's tensor inside __init__ and converts data to float (numpy has double by default while PyTorch's default is float in order to use less memory).
Apart from that it will simply return tuple of features and respective regression targets.
Fit
Almost there, but you have to flatten output from the model (described below). torch.nn.Linear will return tensors of shape (batch, 1) while your targets are of shape (batch,). flatten() will remove unnecessary 1 dimension.
# 2. Calculate Loss
loss = criterion(pred.flatten(), yb)

Model
That is all you need actually:
model = torch.nn.Linear(1, 1)

Any layer can be called directly, no need for forward and inheritance for simple models.
Calling
The rest is almost okay, you just have to create torch.utils.data.DataLoader and pass instance of our dataset. What DataLoader does is it issues __getitem__ of dataset multiple times and creates a batch of specified size (there is some other funny business, but that's the idea):
dataset = RegressionDataset()
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
model = torch.nn.Linear(1, 1)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=3e-4)

fit(5000, model, criterion, optimizer, dataloader)

Also notice I've used torch.nn.MSELoss(), as we are passing object it looks better than function in this case.
Whole code
To make it easier:
import torch
from sklearn.datasets import make_regression

class RegressionDataset(torch.utils.data.Dataset):
    def __init__(self):
        data = make_regression(n_samples=100, n_features=1, noise=0.1, random_state=42)
        self.x = torch.from_numpy(data[0]).float()
        self.y = torch.from_numpy(data[1]).float()

    def __len__(self):
        return len(self.x)

    def __getitem__(self, index):
        return self.x[index], self.y[index]

# Funcao para treinar
def fit(num_epochs, model, criterion, optimizer, train_dl):
    # Repeat for given number of epochs
    for epoch in range(num_epochs):

        # Train with batches of data
        for xb, yb in train_dl:

            # 1. Generate predictions
            pred = model(xb)

            # 2. Calculate Loss
            loss = criterion(pred.flatten(), yb)

            # 3. Compute gradients
            loss.backward()

            # 4. Update parameters using gradients
            optimizer.step()

            # 5. Reset the gradients to zero
            optimizer.zero_grad()

        # Print the progress
        if (epoch + 1) % 10 == 0:
            print(
                "Epoch [{}/{}], Loss: {:.4f}".format(epoch + 1, num_epochs, loss.item())
            )

dataset = RegressionDataset()
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
model = torch.nn.Linear(1, 1)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=3e-4)

fit(5000, model, criterion, optimizer, dataloader)

You should get around 0.053 loss or so, vary noise or other params for harder/easier regression task.


Instruction: Low Validation Score on Pretrained Alexnet from Pytorch models for ImageNet 2012 dataset

Input:
I am using pre-trained AlexNet network to validate some prior work.
The code is as follows:
import os
import torch

import torchvision
import torchvision.datasets as datasets
import torchvision.models as models
import torchvision.transforms as transforms

model = torch.hub.load('pytorch/vision:v0.6.0', 'alexnet', pretrained=True)
model.eval()

batchsize = 50000
workers = 1
dataset_path = 'data/imagenet_2012/'
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_data = datasets.ImageFolder(root=os.path.join(dataset_path, 'val'), transform=transforms.Compose( [transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), normalize,]))
val_loader = torch.utils.data.DataLoader(val_data, batch_size=batchsize, num_workers=workers)

batch = next(iter(val_loader))
images, labels = batch

with torch.no_grad():
    output = model(images)

for i in output:
    out_soft = torch.nn.functional.softmax(i, dim=0)
    print(int(torch.argmax(out_soft)))

When I execute this and compare with ILSVRC2012_validation_ground_truth.txt, I get top-1 accuracy of 5% only.
What am I doing wrong here?
Thank you.

Output:
So, Pytorch/Caffe have their own "ground truth" files, which can be obtained from here:
https://gist.github.com/ksimonyan/fd8800eeb36e276cd6f9#note
I manually tested the images in the validation folder of the Imagenet dataset against the val.txt file in the tar file provided at the link above to verify the order.
Update:
New validation accuracy based on the groundtruth in the zip file in the link:
Top_1 = 56.522%
Top_5 = 79.066%


Instruction: Tensorboard resume training plot

Input: I ran a reinforcement learning training script which used Pytorch and logged data to tensorboardX and saved checkpoints. Now I want to continue training. How do I tell tensorboardX to continue from where I left off? Thank you!

Output:
I figured out how to continue the training plot. While creating the summarywriter, we need to provide the same log_dir that we used while training the first time.
from tensorboardX import SummaryWriter
writer = SummaryWriter('log_dir')

Then inside the training loop step needs to start from where it left (not from 0):
writer.add_scalar('average reward',rewards.mean(),step)


Instruction: Anaconda reading wrong CUDA version

Input:
I have a conda environment with PyTorch and Tensorflow, which both require CUDA 9.0 (~cudatoolkit 9.0 from conda). After installing pytorch with torchvision and the cudatoolkit (like they provided on their website) I wanted to install Tensorflow, the problem here is that I get this error:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - tensorflow==1.12.0 -> python[version='2.7.*|3.6.*']
  - tensorflow==1.12.0 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0']

Your python: python=3.5

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__cuda==10.2=0
  - feature:|@/linux-64::__cuda==10.2=0

Your installed version is: 10.2

If I run nvcc or nvidia-smi on my host or the activated conda environment, I get that I have installed CUDA 10.2, even though conda list shows me that cudatoolkit 9.0 is installed. Any solution to this?
EDIT:
When running this code sample:
# setting device on GPU if available, else CPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
print()

#Additional Info when using cuda
if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')

print(torch.version.cuda)

I get this output:
GeForce GTX 1050
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB
9.0.176

So PyTorch does get the correct CUDA version, I just cant get tensorflow-gpu installed.

Output:
If I run nvcc or nvidia-smi on my host or the activated conda environment, I get that I have installed CUDA 10.2, even though conda list shows me that cudatoolkit 9.0 is installed. Any solution to this?

cudatoolkit doesn't ship with compiler (nvcc), thus when you run nvcc you start compiler from system wide installation. That's why it prints 10.2 istead of 9.0, and pytorch sees the local cudatoolkit.
anaconda / packages / cudatoolkit :

This CUDA Toolkit includes GPU-accelerated libraries, and the CUDA runtime for the Conda ecosystem. For the full CUDA Toolkit with a compiler and development tools visit https://developer.nvidia.com/cuda-downloads

From your comment above I understood that you are using python=3.5.6. So, first of all you should search for available tensorflow py35 builds using:
conda search tensorflow | grep py35

I have the following output:
tensorflow                     1.9.0 eigen_py35h8c89287_1  pkgs/main
tensorflow                     1.9.0 gpu_py35h42d5ad8_1  pkgs/main
tensorflow                     1.9.0 gpu_py35h60c0932_1  pkgs/main
tensorflow                     1.9.0 gpu_py35hb39db67_1  pkgs/main
tensorflow                     1.9.0 mkl_py35h5be851a_1  pkgs/main
tensorflow                    1.10.0 eigen_py35h5ed898b_0  pkgs/main
tensorflow                    1.10.0 gpu_py35h566a776_0  pkgs/main
tensorflow                    1.10.0 gpu_py35ha6119f3_0  pkgs/main
tensorflow                    1.10.0 gpu_py35hd9c640d_0  pkgs/main
tensorflow                    1.10.0 mkl_py35heddcb22_0  pkgs/main

As you can see there is no tensorflow 1.12.0 builds for py35, and that's why you are getting that error. You can try to inspect other conda channels, for example, conda-forge:
conda search tensorflow -c conda-forge | grep py35

But that wasn't helpful:
tensorflow                     0.9.0          py35_0  conda-forge
tensorflow                    0.10.0          py35_0  conda-forge
tensorflow                 0.11.0rc0          py35_0  conda-forge
tensorflow                 0.11.0rc2          py35_0  conda-forge
tensorflow                    0.11.0          py35_0  conda-forge
tensorflow                    0.12.1          py35_0  conda-forge
tensorflow                    0.12.1          py35_1  conda-forge
tensorflow                    0.12.1          py35_2  conda-forge
tensorflow                     1.0.0          py35_0  conda-forge
tensorflow                     1.1.0          py35_0  conda-forge
tensorflow                     1.2.0          py35_0  conda-forge
tensorflow                     1.2.1          py35_0  conda-forge
tensorflow                     1.3.0          py35_0  conda-forge
tensorflow                     1.4.0          py35_0  conda-forge
tensorflow                     1.5.0          py35_0  conda-forge
tensorflow                     1.5.1          py35_0  conda-forge
tensorflow                     1.6.0          py35_0  conda-forge
tensorflow                     1.8.0          py35_0  conda-forge
tensorflow                     1.8.0          py35_1  conda-forge
tensorflow                     1.9.0 eigen_py35h8c89287_1  pkgs/main
tensorflow                     1.9.0 gpu_py35h42d5ad8_1  pkgs/main
tensorflow                     1.9.0 gpu_py35h60c0932_1  pkgs/main
tensorflow                     1.9.0 gpu_py35hb39db67_1  pkgs/main
tensorflow                     1.9.0 mkl_py35h5be851a_1  pkgs/main
tensorflow                     1.9.0          py35_0  conda-forge
tensorflow                    1.10.0 eigen_py35h5ed898b_0  pkgs/main
tensorflow                    1.10.0 gpu_py35h566a776_0  pkgs/main
tensorflow                    1.10.0 gpu_py35ha6119f3_0  pkgs/main
tensorflow                    1.10.0 gpu_py35hd9c640d_0  pkgs/main
tensorflow                    1.10.0 mkl_py35heddcb22_0  pkgs/main
tensorflow                    1.10.0          py35_0  conda-forge

So, the possible solutions are:

Install one of the older available tensorflow 1.10.0 gpu_py35 builds.
Switch to python 3.6.

conda search tensorflow | grep py36

...
tensorflow                    1.11.0 gpu_py36h4459f94_0  pkgs/main
tensorflow                    1.11.0 gpu_py36h9c9050a_0  pkgs/main
...
tensorflow                    1.12.0 gpu_py36he68c306_0  pkgs/main
tensorflow                    1.12.0 gpu_py36he74679b_0  pkgs/main
...

Note that versions >=1.13.1 doesn't support CUDA 9.

Use pip install inside conda env to install missing tensorflow build, because pip hosts more build combinations: Tested build configurations

Here is some best practices from Anaconda how to use pip w/ conda: Using Pip in a Conda Environment

The last option is to build your own missing conda package with conda-build


Instruction: Cannot get the value of hidden weights of RNN

Input:
I declare my RNN as
self.rnn = torch.nn.RNN(input_size=encoding_dim, hidden_size=1, num_layers=1, nonlinearity='relu')

Later
self.rnn.all_weights
# [[Parameter containing:
tensor([[-0.8099, -0.9543,  0.1117,  0.6221,  0.5034, -0.6766, -0.3360, -0.1700,
         -0.9361, -0.3428]], requires_grad=True), Parameter containing:
tensor([[-0.1929]], requires_grad=True), Parameter containing:
tensor([0.7881], requires_grad=True), Parameter containing:
tensor([0.4320], requires_grad=True)]]

self.rnn.all_weights[0][0][0].values
# {RuntimeError}Could not run 'aten::values' with arguments from the 'CPU' backend. 'aten::values' is only available for these backends: [SparseCPU, Autograd, Profiler, Tracer].

Clearly I see the value of the weights, but cannot access to it. Documentation says I need to specify requires_grad=True, but that does not work.
Is there a more elegant and usable way than self.rnn.all_weights[0][0][0]?

Output:
Use torch.nn.Module.named_parameters or torch.nn.Module.parameters.
>>> import torch.nn as nn
>>> model = nn.RNN(input_size=encoding_dim, hidden_size=1, num_layers=1, nonlinearity='relu')
>>> weights = []
>>> for name, parameter in model.named_parameters():
...     weights.append({name: parameter[0]})
...
>>> just_weights = []
>>> for parameter in model.parameters():
...     just_weights.append(parameter[0])
...


Instruction: How can I get a view of input as a complex tensor? RuntimeError: Tensor must have a last dimension with stride 1

Input:
I have a tensor with 64 elements in pytorch and I want to convert it to a complex tensor with 32 elements. Order is important for me and everything should be in PyTorch so I can use it in my customized loss function:
the first half in my primary tensor (W) are my real numbers and the second half are my imaginary ones. so my final tensor should be like:
W_final = tensor(W[0]+jW[32], W[1]+jW[33], W[2]+jW[34], W[3]+jW[35], ... , W[31]+jW[63])
I tried this approach:
import torch
W_1 = = torch.reshape(W,(2,32)) #reshape W with shape (64) to W_1 with shape (2,32)
W_2 = torch.transpose(W_1,0,1) #transpose W_1 to W_2 with shape (32,2), so I can use view_as_complex
W_final = torch.view_as_complex(W_2)

The problem is that with transpose, the stride also changes and I get this error:
RuntimeError: Tensor must have a last dimension with stride 1
Do know how can I deal with stride? or is there any way to reshape with different orders same as numpy?
or any other way to convert to complex?

Output:
It has to do with the non contiguous memory allocation for W_2 after you do reshape.
To handle this error you should call .contiguous() on W_2.
From Pytorch Docs:
" Strides are a list of integers: the k-th stride represents the jump in the memory necessary to go from one element to the next one in the k-th dimension of the Tensor. This concept makes it possible to perform many tensor operations efficiently."
Once you call contiguous all dimensions of returned tensor will have stride 1.
Here is a working sample code:
import torch
W = torch.randn(64)
W_2 = W.view(-1,32).permute(1,0).contiguous()
W_final = torch.view_as_complex(W_2)

First call view to reshape tensor to shape (2,32), then permute dimensions to transpose the result and call contiguous.


Instruction: Is max operation differentiable in Pytorch?

Input:
I am using Pytorch to training some neural networks. The part I am confused about is:
prediction = myNetwork(img_batch)
max_act = prediction.max(1)[0].sum()
loss = softcrossentropy_loss - alpha * max_act

In the above codes, "prediction" is the output tensor of "myNetwork".
I hope to maximize the larget output of "prediction" over a batch.
For example:
[[-1.2, 2.0, 5.0, 0.1, -1.5] [9.6, -1.1, 0.7, 4,3, 3.3]]
For the first prediction vector, the 3rd element is the larget, while for the second vector, the 1st element is the largets. And I want to maximize "5.0+9.6", although we cannot know what index is the larget output for a new input data.
In fact, my training seems to be successful, because the "max_act" part was really increased, which is the desired behavior to me. However, I heard some discussion about whether max() operation is differentiable or not:
Some says, mathmatically, max() is not differentiable.
Some says, max() is just an identity function to select the largest element, and this largest element is differentiable.

So I got confused now, and I am worried if my idea of maximizing "max_act" is wrong from the beginning.
Could someone provide some guidance if max() operation is differentiable in Pytorch?

Output:
max is differentiable with respect to the values, not the indices. It is perfectly valid in your application.
From the gradient point of view, d(max_value)/d(v) is 1 if max_value==v and 0 otherwise. You can consider it as a selector.
d(max_index)/d(v) is not really meaningful as it is a discontinuous function, with only 0 and undefined as possible gradients.


Instruction: matrix multiplication for complex numbers in PyTorch

Input:
I am trying to multiply two complex matrices in PyTorch and it seems the torch.matmul functions is not added yet to PyTorch library for complex numbers.
Do you have any recommendation or is there another method to multiply complex matrices in PyTorch?

Output:
Currently torch.matmul is not supported for complex tensors such as ComplexFloatTensor but you could do something as compact as the following code:
def matmul_complex(t1,t2):
    return torch.view_as_complex(torch.stack((t1.real @ t2.real - t1.imag @ t2.imag, t1.real @ t2.imag + t1.imag @ t2.real),dim=2))

When possible avoid using for loops as these will result in much slower implementations.
Vectorization is achieved by using built-in methods as demonstrated in the code I have attached.
For example, your code takes roughly 6.1s on CPU while the vectorized version takes only 101ms (~60 times faster) for 2 random complex matrices with dimensions 1000 X 1000.
Update:
Since PyTorch 1.7.0 (as @EduardoReis mentioned) you can do matrix multiplication between complex matrices similarly to real-valued matrices as follows:
t1 @ t2
(for t1, t2 complex matrices).


Instruction: Pytorch - Should 'CenterCrop' be used to test set? Does this count as cheating？

Input:
I'm learning image  classification with Pytorch. I found some papers code use 'CenterCrop' to both train set and test set，e.g. Resize to larger size，then apply CenterCrop to obtain a smaller size. The smaller size is a general size in this research direction.
In my experience, I found apply CenterCrop can get a significant improvement(e.g. 1% or 2%) on test,  compare to without CenterCrop on test set.
Because it is used in the top conference papers, confused me. So, Should CenterCrop be used to test set this count as cheating？ In addition, should I use any data augmentation to test set except 'Resize' and 'Normalization'?
Thank you for your answer.

Output:
That is not cheating. You can apply any augmentation as long as the label is not used.
In image classification, sometimes people use a FiveCrop+Reflection technique, which is to take five crops (Center, TopLeft, TopRight, BottomLeft, BottomRight) and their reflections as augmentations. They would then predict class probabilities for each crop and average the results, typically giving some performance boost with 10X running time.
In segmentation, people also use similar test-time augmentation "multi-scale testing" which is to resize the input image to different scales before feeding it to the network. The predictions are also averaged.
If you do use such kind of augmentation, do report them when you compare with other methods for a fair comparison.


Instruction: _th_addr_out not supported on CPUType for ComplexFloat

Input:
I am trying to use a customized loss function for my NN. I've implemented all operations in torch and I have complex numbers among my data.
I get the error while training a NN:
RuntimeError: _th_addr_out not supported on CPUType for ComplexFloat

Do you know any possible solution to deal with it?

Output:
Well it seems Complex Autograd in PyTorch is currently in a prototype state, and the backward functionality for some of function is not included.
For example:  torch.sign, which is used in the backward computation of torch.abs, is not defined for complex tensors. same for torch.mv. So I debugged my code line by line to find the functions which are not included, and replaced them with a customized function :)
Hope for a lot more functions to be included in the next release of PyTorch.


Instruction: How do I mutate the input using gradient descent in PyTorch?

Input:
I'm new to PyTorch. I learned it uses autograd to automatically calculate the gradients for the gradient descent function.
Instead of adjusting the weights, I would like to mutate the input to achieve a desired output, using gradient descent. So, instead of the weights of neurons changing, I want to keep all of the weights the same and just change the input to minimize the loss.
For example. The network is a trained image classifier with the numbers 0-9. I input random noise, and I want to morph it so that the network considers it a 3 with 60% confidence. I would like to utilize gradient descent to adjust the values of the input (originally noise) until the network considers the input to be a 3, with 60% confidence.
Is there a way to do this?

Output:
I assume you know how to do regular training with gradient descent. You only need to change the parameters to be optimized by the optimizer. Something like
# ... Setup your network, load the input
# ...

# Set proper requires_grad -> We train the input, not the parameters
input.requires_grad = True
for p in network.parameters():
    p.requires_grad = False

# Setup the optimizer
# Previously we should have SomeOptimizer(net.parameters())
optim = SomeOptimizer([input])

output_that_you_want = ...
actual_output = net(input)
some_loss = SomeLossFunction(output_that_you_want, actual_output)
# ...
# Back-prop and optim.step() as usual


Instruction: Pytorch NLP sequence length of target in Transformer

Input:
I'm trying to understand the code of Transformer (https://github.com/SamLynnEvans/Transformer).
If seeing the train_model function in "train" script, I wonder why need to use the different sequence length of trg_input from trg:
trg_input = trg[:, :-1]

In this case, the sequence length of trg_input is "seq_len(trg) - 1".
It means that trg is like:
<sos> tok1 tok2 tokn <eos>

and trg_input is like:
<sos> tok1 tok2 tokn    (no eos token)

Please let me know the reason.
Thank you.
The related code is like below:
    for i, batch in enumerate(opt.train):
        src = batch.src.transpose(0, 1).to('cuda')
        trg = batch.trg.transpose(0, 1).to('cuda')

        trg_input = trg[:, :-1]
        src_mask, trg_mask = create_masks(src, trg_input, opt)
        preds = model(src, trg_input, src_mask, trg_mask)
        ys = trg[:, 1:].contiguous().view(-1)
        opt.optimizer.zero_grad()
        loss = F.cross_entropy(preds.view(-1, preds.size(-1)), ys, ignore_index=opt.trg_pad)
        loss.backward()
        opt.optimizer.step()

def create_masks(src, trg, opt):

    src_mask = (src != opt.src_pad).unsqueeze(-2)

    if trg is not None:
        trg_mask = (trg != opt.trg_pad).unsqueeze(-2)
        size = trg.size(1) # get seq_len for matrix
        np_mask = nopeak_mask(size, opt)
        if trg.is_cuda:
            np_mask.cuda()
        trg_mask = trg_mask & np_mask

    else:
        trg_mask = None
    return src_mask, trg_mask

Output:
That's because the entire aim is to generate the next token based on the tokens we've seen so far. Take a look at the input into the model when we get our predictions. We're not just feeding the source sequence, but also the target sequence up until our current step. The model inside Models.py looks like:
class Transformer(nn.Module):
    def __init__(self, src_vocab, trg_vocab, d_model, N, heads, dropout):
        super().__init__()
        self.encoder = Encoder(src_vocab, d_model, N, heads, dropout)
        self.decoder = Decoder(trg_vocab, d_model, N, heads, dropout)
        self.out = nn.Linear(d_model, trg_vocab)
    def forward(self, src, trg, src_mask, trg_mask):
        e_outputs = self.encoder(src, src_mask)
        #print("DECODER")
        d_output = self.decoder(trg, e_outputs, src_mask, trg_mask)
        output = self.out(d_output)
        return output

So you can see that the forward method receives src and trg, which are each fed into the encoder and decoder. This is a bit easier to grasp if you take a look at the model architecture from the original paper:

The "Outputs (shifted right)" corresponds to trg[:, :-1] in the code.


Instruction: Training Loss and Accuracy both decreasing for my transformer model for Time Series Prediction

Input:
I am using a transformer model for predicting the forex market. I transformed the open price data and calculated the difference between each 30 min interval. And converted the difference into tokens. The tokens are obtained by applying log1.5 to the difference. I obtained 28 types of tokens for 6 years. 14-27 represents a bull market and 0-13 tokens represent bear market.
I created a transformer model in PyTorch and applied the data.
import torch
import math
import numpy as np
import copy
from torch import nn
from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader
import ast
from numpy import load
import torch.nn as nn
import random
import time
import matplotlib.pyplot as plt

class Embedder(nn.Module):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        # print(vocab_size,d_model)
        self.embed = nn.Embedding(vocab_size+1, d_model,padding_idx=0)
    def forward(self, x):
        # print(x.shape)
        # print("Embed",self.embed(x).shape)
        return self.embed(x)

class PositionalEncoder(nn.Module):
    def __init__(self, d_model, max_seq_len = 500):
        super().__init__()
        self.d_model = d_model

        # create constant 'pe' matrix with values dependant on
        # pos and i
        pe = torch.zeros(max_seq_len, d_model)
        for pos in range(max_seq_len):
            for i in range(0, d_model, 2):
                pe[pos, i] = \
                math.sin(pos / (10000 ** ((2 * i)/d_model)))
                pe[pos, i + 1] = \
                math.cos(pos / (10000 ** ((2 * (i + 1))/d_model)))

        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x * math.sqrt(self.d_model)
        seq_len = x.size(1)
        x = x + torch.autograd.Variable(self.pe[:,:seq_len],requires_grad=False)
        return x

def attention(q, k, v, d_k, mask=None, dropout=None):

    scores = torch.matmul(q, k.transpose(-2, -1)) /  math.sqrt(d_k)
    if mask is not None:
        mask = mask.unsqueeze(1)
        scores = scores.masked_fill(mask == 0, -1e9)
        scores = torch.nn.functional.softmax(scores, dim=-1)

    if dropout is not None:
        scores = dropout(scores)

    output = torch.matmul(scores, v)
    return output

class MultiHeadAttention(nn.Module):
    def __init__(self, heads, d_model, dropout = 0.1):
        super().__init__()

        self.d_model = d_model
        self.d_k = d_model // heads
        self.h = heads

        self.q_linear = nn.Linear(d_model, d_model)
        self.v_linear = nn.Linear(d_model, d_model)
        self.k_linear = nn.Linear(d_model, d_model)
        self.dropout = nn.Dropout(dropout)
        self.out = nn.Linear(d_model, d_model)

    def forward(self, q, k, v, mask=None):

        bs = q.size(0)

        # perform linear operation and split into h heads

        k = self.k_linear(k).view(bs, -1, self.h, self.d_k)
        q = self.q_linear(q).view(bs, -1, self.h, self.d_k)
        v = self.v_linear(v).view(bs, -1, self.h, self.d_k)

        # transpose to get dimensions bs * h * sl * d_model

        k = k.transpose(1,2)
        q = q.transpose(1,2)
        v = v.transpose(1,2)
        # calculate attention using function we will define next
        scores = attention(q, k, v, self.d_k, mask, self.dropout)

        # concatenate heads and put through final linear layer
        concat = scores.transpose(1,2).contiguous()\
        .view(bs, -1, self.d_model)

        output = self.out(concat)

        return output

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff=512, dropout = 0.1):
        super().__init__()
        # We set d_ff as a default to 2048
        self.linear_1 = nn.Linear(d_model, d_ff)
        self.dropout = nn.Dropout(dropout)
        self.linear_2 = nn.Linear(d_ff, d_model)
    def forward(self, x):
        x = self.dropout(torch.nn.functional.relu(self.linear_1(x)))
        x = self.linear_2(x)
        return x

class Norm(nn.Module):
    def __init__(self, d_model, eps = 1e-6):
        super().__init__()

        self.size = d_model
        # create two learnable parameters to calibrate normalisation
        self.alpha = nn.Parameter(torch.ones(self.size))
        self.bias = nn.Parameter(torch.zeros(self.size))
        self.eps = eps
    def forward(self, x):
        norm = self.alpha * (x - x.mean(dim=-1, keepdim=True)) \
        / (x.std(dim=-1, keepdim=True) + self.eps) + self.bias
        return norm

class EncoderLayer(nn.Module):
    def __init__(self, d_model, heads, dropout = 0.1):
        super().__init__()
        self.norm_1 = Norm(d_model)
        self.norm_2 = Norm(d_model)
        self.attn = MultiHeadAttention(heads, d_model)
        self.ff = FeedForward(d_model)
        self.dropout_1 = nn.Dropout(dropout)
        self.dropout_2 = nn.Dropout(dropout)

    def forward(self, x, mask):
        x2 = self.norm_1(x)
        x = x + self.dropout_1(self.attn(x2,x2,x2,mask))
        x2 = self.norm_2(x)
        x = x + self.dropout_2(self.ff(x2))
        return x

class DecoderLayer(nn.Module):
    def __init__(self, d_model, heads, dropout=0.1):
        super().__init__()
        self.norm_1 = Norm(d_model)
        self.norm_2 = Norm(d_model)
        self.norm_3 = Norm(d_model)

        self.dropout_1 = nn.Dropout(dropout)
        self.dropout_2 = nn.Dropout(dropout)
        self.dropout_3 = nn.Dropout(dropout)

        self.attn_1 = MultiHeadAttention(heads, d_model)
        self.attn_2 = MultiHeadAttention(heads, d_model)
        self.ff = FeedForward(d_model).cuda()
        # self.ff = FeedForward(d_model)

    def forward(self, x, e_outputs, src_mask, trg_mask):
        x2 = self.norm_1(x)
        x = x + self.dropout_1(self.attn_1(x2, x2, x2, trg_mask))
        x2 = self.norm_2(x)
        x = x + self.dropout_2(self.attn_2(x2, e_outputs, e_outputs,
        src_mask))
        x2 = self.norm_3(x)
        x = x + self.dropout_3(self.ff(x2))
        return x
# We can then build a convenient cloning function that can generate multiple layers:
def get_clones(module, N):
    return nn.ModuleList([copy.deepcopy(module) for i in range(N)])

class Encoder(nn.Module):
    def __init__(self, vocab_size, d_model, N, heads):
        super().__init__()
        self.N = N
        self.embed = Embedder(vocab_size, d_model)
        self.pe = PositionalEncoder(d_model)
        self.layers = get_clones(EncoderLayer(d_model, heads), N)
        self.norm = Norm(d_model)

    def forward(self, src, mask):
        x = self.embed(src)
        x = self.pe(x)
        for i in range(self.N):
            x = self.layers[i](x, mask)
        return self.norm(x)

class Decoder(nn.Module):
    def __init__(self, vocab_size, d_model, N, heads):
        super().__init__()
        self.N = N
        self.embed = Embedder(vocab_size, d_model)
        self.pe = PositionalEncoder(d_model)
        self.layers = get_clones(DecoderLayer(d_model, heads), N)
        self.norm = Norm(d_model)
    def forward(self, trg, e_outputs, src_mask, trg_mask):
        x = self.embed(trg)
        x = self.pe(x)
        for i in range(self.N):
            x = self.layers[i](x, e_outputs, src_mask, trg_mask)
        return self.norm(x)

class Transformer(nn.Module):
    def __init__(self, src_vocab, trg_vocab, d_model, N, heads):
        super().__init__()
        self.encoder = Encoder(src_vocab, d_model, N, heads)
        self.decoder = Decoder(trg_vocab, d_model, N, heads)
        self.out = nn.Linear(d_model, trg_vocab)

    def forward(self, src, trg, src_mask, trg_mask):
        e_outputs = self.encoder(src, src_mask)
        d_output = self.decoder(trg, e_outputs, src_mask, trg_mask)
        output = self.out(d_output)
        return output

def batchify(data, bsz):
    nbatch = data.size(0) // bsz
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1).t().contiguous()
    return data

bptt = 128
class CustomDataLoader:
    def __init__(self,source):
        print("Source",source.shape)
        self.batches = list(range(0, source.size(0) - 2*bptt))
        # random.shuffle(self.batches)
        # print(self.batches)
        self.data = source
        self.sample = random.sample(self.batches,120)

    def batchcount(self):
        return len(self.batches)

    def shuffle_batches(self):
        random.shuffle(self.batches)

    def get_batch_from_batches(self,i):
        if i==0:
            random.shuffle(self.batches)
        ind = self.batches[i]
        seq_len = min(bptt,len(self.data)-1-ind)
        src = self.data[ind:ind+seq_len]
        tar = self.data[ind+seq_len-3:ind+seq_len-3+seq_len+1]
        return src,tar

    def get_batch(self,i):
        # print(i,len(self.batches))
        ind = self.sample[i]
        seq_len = min(bptt,len(self.data)-1-ind)
        src = self.data[ind:ind+seq_len]
        tar = self.data[ind+seq_len-3:ind+seq_len-3+seq_len+1]
        # tar = tar.view(-1)
        if(i==len(self.sample)-1):
            random.sample(self.batches,60)
            # print("Data shuffled",self.batches[:10])
        return src,tar

def get_batch(source, i):
    seq_len = min(bptt, len(source) - 1 - i)
    data = source[i:i+seq_len]
    target = source[i+seq_len-3:i+seq_len-3+seq_len]
    return data, target

def plot_multiple(data,legend):
    fig,ax = plt.subplots()
    for line in data:
        plt.plot(list(range(len(line))),line)
    plt.legend(legend)
    plt.show()

def plot_subplots(data,legends,name):
    names = ['Accuracy', 'Loss']
    plt.figure(figsize=(10, 5))
    for i in range(len(data)):
        plt.subplot(121+i)
        plt.plot(list(range(0,len(data[i])*3,3)),data[i])
        plt.title(legends[i])
        plt.xlabel("Epochs")
    plt.savefig(name)

def evaluate(eval_model, data_source):
    eval_model.eval() # Turn on the evaluation mode
    total_loss = 0.
    ntokens = 28
    count = 0
    with torch.no_grad():
        cum_loss = 0
        acc_count = 0
        accs = 0
        print(data_source.shape)
        for batch, i in enumerate(range(0, data_source.size(0) - bptt*2, bptt)):
            data, targets = get_batch(data_source, i)
            # data,targets = dataLoader.get_batch(i)
            data = data.transpose(0,1).contiguous()
            targets= targets.transpose(0,1).contiguous()
            trg_input = targets[:,:-1]
            trg_output = targets[:,1:].contiguous().view(-1)
            src_mask , trg_mask = create_masks(data,trg_input)
            output = model(data,trg_input,src_mask,trg_mask)
            output = output.view(-1,output.size(-1))
            loss = torch.nn.functional.cross_entropy(output,trg_output-1)
            accs += ((torch.argmax(output,dim=1)==trg_output).sum().item()/output.size(0))
            # accs += ((torch.argmax(output,dim=1)==targets).sum().item()/output.size(0))
            cum_loss += loss
            count+=1
        # print(epoch,"Loss: ",(cum_loss/count),"Accuracy ",accs/count)

    return cum_loss/ (count), accs/count

def nopeak_mask(size,cuda_enabled):
    np_mask = np.triu(np.ones((1, size, size)),
    k=1).astype('uint8')
    np_mask =  torch.autograd.Variable(torch.from_numpy(np_mask) == 0)

    if cuda_enabled:
      np_mask = np_mask.cuda()
    return np_mask

def create_masks(src, trg):
    src_mask = (src != 0).unsqueeze(-2)
    if trg is not None:
        trg_mask = (trg != 0).unsqueeze(-2)
        size = trg.size(1) # get seq_len for matrix
        # print("Sequence lenght in mask ",size)
        np_mask = nopeak_mask(size,True)
        # print(np_mask.shape,trg_mask.shape)
        if trg.is_cuda:
            np_mask.cuda()
        trg_mask = trg_mask & np_mask
    else:
        trg_mask = None
    return src_mask, trg_mask

def create_padding_mask(seq):
  seq = tf.cast(tf.math.equal(seq, 0), tf.float32)

  # add extra dimensions to add the padding
  # to the attention logits.
  return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)

if __name__ == '__main__':
    data = []
    dev = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    procsd_data = load("Eavg_open.npy")
    print(set(procsd_data[:,0]))
    train_data =torch.tensor(procsd_data)[:30000*2]
    print(train_data.shape)
    val_data = torch.tensor(procsd_data)[30000*2:35000*2]
    test_data = torch.tensor(procsd_data)[35000*2:]
    train_data = train_data.to(dev)
    val_data = val_data.to(dev)
    test_data = test_data.to(dev)

    # train_data = train_data.transpose(1,0).contiguous()
    # val_data = val_data.transpose(1,0).contiguous()

    batch_size = 32
    ntokens = 28
    train_data = batchify(train_data,batch_size)
    # print(train_data.shape)
    val_data = batchify(val_data,batch_size)
    test_data = batchify(train_data,batch_size)
    # model = Transformer(n_blocks=3,d_model=256,n_heads=8,d_ff=256,dropout=0.5)
    model = Transformer(28,28,64,3,4)
    # model = torch.load("modela")
    for p in model.parameters():
        if p.dim() > 1:
            nn.init.xavier_uniform_(p)

    model.to(dev)
    criterion = nn.CrossEntropyLoss()
    lr = 0.00001 # learning rate

    optim = torch.optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)
    #########training starts###########

    accuracies = []
    lossies = []
    val_loss = []
    val_accuracy = []
    dataLoader = CustomDataLoader(train_data)
    _onehot = torch.eye(29)
    for epoch in range(500):
        count = 0
        cum_loss = 0
        acc_count = 0
        accs = 0
        s = time.time()
        # for i in range(len(range(0, train_data.size(0) - bptt))):
        model.train()
        # dataLoader.shuffle_batches()
        for i in range(300):
            # data, targets = get_batch(train_data, i)
            # d = time.time()
            hh = time.time()
            data,targets = dataLoader.get_batch_from_batches(i)
            data = data.transpose(0,1).contiguous()
            targets= targets.transpose(0,1).contiguous()
            # print(data.shape,targets.shape)
            trg_input = targets[:,:-1]
            trg_output = targets[:,1:].contiguous().view(-1)
            # print(data.shape,trg_input.shape)
            src_mask , trg_mask = create_masks(data,trg_input)
            # print("Source Mask",src_mask)
            # print("Target Mask",trg_mask)
            output = model(data,trg_input,src_mask,trg_mask)
            # output = output.view(-1,28)
            output = output.view(-1,output.size(-1))

            loss = torch.nn.functional.cross_entropy(output,trg_output-1)
            accuracy = ((torch.argmax(output,dim=1)==trg_output).sum().item()/output.size(0))
            accs += accuracy
            cum_loss += loss.item();
            loss.backward()
            optim.step()
            model.zero_grad()
            optim.zero_grad()
            print(i," Batch Loss", loss.item()," Batch Accuracy ",accuracy," Time taken ",time.time()-hh)

            count+=1

        data,targets = None,None
        print(epoch,"Loss: ",(cum_loss/count),"Accuracy ",accs/count," Time Taken: ",time.time()-s)
        if(epoch%3==0):
            lossies.append(cum_loss/count)
            accuracies.append(accs/count)
            legend = ["accuracy","Loss"]
            plot_subplots([accuracies,lossies],legend,"A&L_v1")
            print("Valdata",val_data.shape)
            eval_loss,eval_acc = evaluate(model,val_data)
            val_accuracy.append(eval_acc)
            val_loss.append(eval_loss)
            plot_subplots([val_accuracy,val_loss],legend,"Val A&L_v1")
            print(epoch,"Loss: ",(cum_loss/count),"Accuracy ",accs/count," Valid_loss: ",eval_loss," Valid_accuracy: ",eval_acc)
            if len(val_loss)>0 and eval_loss < val_loss[-1]:
                val_loss.append(eval_loss)
                torch.save(model,"evalModel")
            else:
                val_loss.append(eval_loss)
                torch.save(model,"evalModel")
        if(epoch%5==0):
            torch.save(model,"modela")

I got the following loss and accuracy while training:

What is causing this behaviour?
Am I wrong in my tokenization method?
Is it necessary to add any time embedding to the data?

Output:
Actually I made a small mistake in calculating accuracy.
accuracy = ((torch.argmax(output,dim=1)==trg_output).sum().item()/output.size(0))

Here the trg_output has tokens numbered from 1 to n but the argmax function for output returns a range from 0 to n-1. So this is causing this problem.
So I modified the above line to
accuracy = ((torch.argmax(output,dim=1)==(trg_output-1) ).sum().item()/output.size(0))

Same should be applied in the evaluation function also.


Instruction: Decoder always predicts the same token

Input:
I have the following decoder for machine translation that after a few steps only predicts the EOS token. Overfitting on a dummy, tiny dataset is impossible because of this so it seems that there is a big error in the code.
Decoder(
  (embedding): Embeddings(
    (word_embeddings): Embedding(30002, 768, padding_idx=3)
    (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
  )
  (ffn1): FFN(
    (dense): Linear(in_features=768, out_features=512, bias=False)
    (layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
    (activation): GELU()
  )
  (rnn): GRU(512, 512, batch_first=True, bidirectional=True)
  (ffn2): FFN(
    (dense): Linear(in_features=1024, out_features=512, bias=False)
    (layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
    (activation): GELU()
  )
  (selector): Sequential(
    (0): Linear(in_features=512, out_features=30002, bias=True)
    (1): LogSoftmax(dim=-1)
  )
)

The forward is relatively straightforward (see what I did there?): pass the input_ids to the embedding and a FFN, then use that representation in the RNN with the given sembedding as initial hidden state. Pass the output through another FFN and do softmax. Return logits and last hidden states of the RNN. In the next step, use those hidden states as the new hidden states, and the highest predicted token as the new input.
def forward(self, input_ids, sembedding):
    embedded = self.embedding(input_ids)
    output = self.ffn1(embedded)
    output, hidden = self.rnn(output, sembedding)
    output = self.ffn2(output)
    logits = self.selector(output)

    return logits, hidden

sembedding is the initial hidden_state for the RNN. This is similar to an encoder-deocder architecture only here we do not train the encoder but we do have access to pretrained encoder representations.
In my training loop I start off each batch with a SOS token and feed every top predicted token to next step until target_len is reached. I also swap randomly between teacher forced training.
def step(self, batch, teacher_forcing_ratio=0.5):
    batch_size, target_len = batch["input_ids"].size()[:2]
    # Init first decoder input woth SOS (BOS) token
    decoder_input = torch.tensor([[self.tokenizer.bos_token_id]] * batch_size).to(self.device)
    batch["input_ids"] = batch["input_ids"].to(self.device)

    # Init first decoder hidden_state: one zero'd second embedding in case the RNN is bidirectional
    decoder_hidden = torch.stack((batch["sembedding"],
                                  torch.zeros(*batch["sembedding"].size()))
                                 ).to(self.device) if self.model.num_directions == 2 \
        else batch["sembedding"].unsqueeze(0).to(self.device)

    loss = torch.tensor([0.]).to(self.device)

    use_teacher_forcing = random.random() < teacher_forcing_ratio
    # contains tuples of predicted and correct words
    tokens = []
    for i in range(target_len):
        # overwrite previous decoder_hidden
        output, decoder_hidden = self.model(decoder_input, decoder_hidden)
        batch_correct_ids = batch["input_ids"][:, i]

        # NLLLoss compute loss between predicted classes (bs x classes) and correct classes for _this word_
        # set to ignore the padding index
        loss += self.criterion(output[:, 0, :], batch_correct_ids)

        batch_predicted_ids = output.topk(1).indices.squeeze(1).detach()

        # if use teacher training: use current correct word for next prediction
        # else do NOT use teacher training: us current predction for next prediction
        decoder_input = batch_correct_ids.unsqueeze(1) if use_teacher_forcing else batch_predicted_ids

    return loss, loss.item() / target_len

I also clip the gradients after each step:
clip_grad_norm_(self.model.parameters(), 1.0)

At first subsequent predictions are already relatively identical, but after a few iterations there's a bit more variation. But relatively quickly ALL predictions turn into other words (but always the same ones), eventually turning into EOS tokens (edit: after changing the activation to ReLU, another token is always predicted - it seems like a random token that always gets repeated). Note that this already happens after 80 steps (batch_size 128).
I found that the returned hidden state of the RNN contains a lot of zeros. I am not sure if that is the problem but it seems like it could be related.
tensor([[[  3.9874e-02,  -6.7757e-06,   2.6094e-04,  ...,  -1.2708e-17,
            4.1839e-02,   7.8125e-03],
         [ -7.8125e-03,  -2.5341e-02,   7.8125e-03,  ...,  -7.8125e-03,
           -7.8125e-03,  -7.8125e-03],
         [ -0.0000e+00, -1.0610e-314,   0.0000e+00,  ...,   0.0000e+00,
            0.0000e+00,   0.0000e+00],
         [  0.0000e+00,   0.0000e+00,   0.0000e+00,  ...,   0.0000e+00,
           -0.0000e+00,  1.0610e-314]]], device='cuda:0', dtype=torch.float64,
       grad_fn=<CudnnRnnBackward>)

I have no idea what might be going wrong although I suspect that the issue is rather with my step than with the model. I already tried playing with the learning rate, disabling some layers (LayerNorm, dropout, ffn2), using pretrained embeddings and freezing or unfreezing them, and disabling teacher forcing, using bidrectional vs unidirectional GRU. The end result is always the same.
If you have any pointers, that would be very helpful. I have googled many things concerning neural networks always predicting the same item and I have tried all the suggestions that I could find. Any new ones, no matter how crazy, are welcome!

Output:
In my case the issue appeared to be that the dtype of the initial hidden state was a double and the input was a float. I don't quite understand why that is an issue, but casting the hidden state to a float solved the issue. If you have any intuition about why this might be a problem for PyTorch, do let me know in the comments or, better yet, on the official PyTorch forums.
EDIT: as that topic shows, this is a bug in PyTorch 1.6 that is solved in 1.7, In 1.7, you will get an error message which will hopefully save you the trouble of debugging all your code and not finding what causes strange behaviour.