Instruction: Lack of gradient when creating tensor from numpy

Input:
Can someone please explain to me the following behavior?
import torch
import numpy as np

z = torch.tensor(np.array([1., 1.]), requires_grad=True).float()

def pre_main(z):
  return z * 3.0

x = pre_main(z)
x.backward(torch.tensor([1., 1.]))
print(z.grad)

prints:
None

Meanwhile:
import torch
import numpy as np

z = torch.tensor([1., 1.], requires_grad=True).float()

def pre_main(z):
  return z * 3.0

x = pre_main(z)
x.backward(torch.tensor([1., 1.]))
print(z.grad)

prints:
tensor([3., 3.])

Why are my gradients being destroyed when constructing from a numpy array?  How do I fix this?

Output:
Your gradient is not destroyed: grad returns None because it has never been saved on the grad attribute. This is because non-leaf tensors don't have their gradients stored during backpropagation. Hence the warning message you received when running your first snippet:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward().

This is the case for your z tensor when it is defined as:
>>> z = torch.tensor(np.array([1., 1.]), requires_grad=True).float()
>>> z.is_leaf
False

Compared to:
>>> z = torch.tensor([1., 1.], requires_grad=True).float()
>>> z.is_leaf
True

which means the latter will have its gradient value in z.grad.
But notice that:
>>> z = torch.tensor(np.array([1., 1.]), requires_grad=True)
>>> z.is_leaf
True

To further explain this: when a tensor is first initialized it is a leaf node (.is_leaf returns True). As soon as you apply a function on it (here .float() is an in-place operator) it is not a leaf anymore since it has parents in the computational graph.
So really, there's nothing to fix... What you can do though is make sure the gradient is saved on z.grad when the backward pass is called. So, the second question comes down to how to store/access the gradient on a non-leaf node?.

Now, if you would like to store the gradient on .backward() call. You could use retain_grad() as explained in the warning message:
z = torch.tensor(np.array([1., 1.]), requires_grad=True).float()
z.retain_grad()

Or, since we expected it to be a leaf node, solve it by using FloatTensor to convert the numpy.array to a torch.Tensor:
z = torch.FloatTensor(np.array([1., 1.]))
z.requires_grad=True

Alternatively, you could stick with torch.tensor and supply a dtype:
z = torch.tensor(np.array([1., 1.]), dtype=torch.float64, requires_grad=True)


Instruction: Is there a depthwise constant convolutional layer option in PyTorch?

Input:
I'm interested in applying a convolutional kernel that's only got HxW parameters where (H, W) is kernel size. The kernel would still have dimensions CxHxW like a normal convolution, but the parameters are constant in the channel dimension.
Is there an inbuilt option for this in PyTorch?

Output:
That would be equivalent to a convolution kernel with a 1-dimensional (summed) input. You can verify that mathematically (just factor out the weight). We can also verify it with code, so you can use this if you really wanted to do that.
import torch
import torch.nn as nn

# Normal conv
normal_conv = nn.Conv2d(1, 2, kernel_size=1)

# We can artificially repeat the weight along the channel dimension -> constant depthwise
repeated_conv = nn.Conv2d(6, 2, kernel_size=1)
repeated_conv.weight.data = normal_conv.weight.data.expand(-1, 6, -1, -1)
repeated_conv.bias.data = normal_conv.bias.data

data = torch.randn(1, 6, 3, 3)

# same result
print(repeated_conv(data))
print(normal_conv(data.sum(1, keepdim=True)))

So, you don't need a custom layer. Just create a convolution with the number of input channels = 1, and sum the input in the channel dimension before you feed it into the layer.
UPDATE: Backward pass testing:
data1 = torch.randn(1, 6, 3, 3)
data2 = data1.clone()
data1.requires_grad = True
data2.requires_grad = True

repeated_conv(data1).mean().backward()
normal_conv(data2.sum(1, keepdim=True)).mean().backward()

print(data1.grad, repeated_conv.weight.grad.sum(1))
print(data2.grad, normal_conv.weight.grad)


Instruction: ValueError: Target size (torch.Size([1000])) must be the same as input size (torch.Size([1000, 1]))

Input:
I am trying to train my first neural net in pyTorch (I'm not a programmer, just a confused chemist).
The net itself is supposed to take 1064 element vectors and rate them with a float number.
So far I have encountered a variety of errors, ranging from 'float instead of long' to 'Target 1 is out of bounds'. Thus I have redefined the dtypes, corrected the dimensions of the input vector, changed the loss function and now I am stuck in the situation when correcting the current error sets me back to the previous ones.
Which is:
ValueError: Target size (torch.Size([1000])) must be the same as input size (torch.Size([1000, 1]))

at the 'loss=loss_calc(outputs, target)' line.
I tried unsqueezing the label during the DataSet class definition, but this solution sets me back. When I tried to label = label.view(1,1), the resulting error changed to

Target size (torch.Size([1000, 1, 1])) must be the same as input size (torch.Size([1000, 1]))

Could anyone please help me figure this out?
import pandas as pd
import numpy as np
import rdkit
from rdkit import Chem
from rdkit.Chem import AllChem
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader

class dataset(Dataset):
    def __init__(self, path, transform=None):
        self.data = pd.read_excel(path)
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        smiles=self.data.at[index, 'smiles']
        mol=Chem.MolFromSmiles(smiles)
        morgan = torch.tensor(list(AllChem.GetMorganFingerprintAsBitVect(mol, 1, nBits=1064)), dtype=torch.float)

        label=torch.tensor(self.data.at[index, 'score'], dtype=torch.long)

        if self.transform is not None:
            morgan=self.transform(morgan)

        return morgan, label

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1064, 1064)
        self.fc2 = nn.Linear(1064, 1)
        self.act = nn.Tanh()

    def forward(self, x):
        x = self.act(self.fc1(x))
        x = self.act(self.fc2(x))
        x = self.fc2(x)
        return x

trainSet=dataset(r'C:\Users\BajMic\Peptides\trainingSet.xlsx')
testSet=dataset(r'C:\Users\BajMic\Peptides\testSet.xlsx')

net = Net()
loss_calc = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):
    running_loss=0.0
    for data in DataLoader(trainSet, batch_size=1000, shuffle=True):
        inputs, target = data
        optimizer.zero_grad()
        outputs = net(inputs)
        print(outputs)
        loss = loss_calc(outputs, target)
        loss.backward()
        optimizer.step()        # print statistics
        running_loss += loss.item()
        print('[%d, %5d] loss: %.3f' %(epoch + 1, i + 1, running_loss))
        running_loss = 0.0
print('Finished Training')

Output:
When working with a loss function and having problems with shapes you will usually have an error message of this type:

Target size (torch.Size([...])) must be the same as input size (torch.Size([...]))

'Target' refers to the label, the ground-truth, while 'input' refers to your model's output. In this case, the output is 1000 predictions (you set batch_size=1000) of 1 scalar value. Therefore the shape is (1000, 1). This last axis is the one bothering you, since the prediction vector is just a 1D tensor containing 1000 scalars, i.e. (1000).
To solve this you can expand your target tensor with an extra dimension. With torch.unsqueeze():
target = target.unsqueeze(-1) # -1 stands for last here equivalent to 1


Instruction: How to reduce the inference time of Helsinki-NLP/opus-mt-es-en (translation model) from transformer

Input:
Currently Helsinki-NLP/opus-mt-es-en model takes around 1.5sec for inference from transformer. How can that be reduced?
Also when trying to convert it to onxx runtime getting this error:

ValueError: Unrecognized configuration class <class 'transformers.models.marian.configuration_marian.MarianConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of RetriBertConfig, MT5Config, T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, LongformerConfig, RobertaConfig, LayoutLMConfig, SqueezeBertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, FSMTConfig, XLMConfig, CTRLConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig, BertGenerationConfig, DebertaConfig, DPRConfig, XLMProphetNetConfig, ProphetNetConfig, MPNetConfig, TapasConfig.

Is it possible to convert this to onxx runtime?

Output:
The OPUS models are originally trained with Marian which is a highly optimized toolkit for machine translation written fully in C++. Unlike PyTorch, it does have the ambition to be a general deep learning toolkit, so it can focus on MT efficiency. The Marian configurations and instructions on how to download the models are at https://github.com/Helsinki-NLP/OPUS-MT.
The OPUS-MT models for Huggingface's Transformers are converted from the original Marian models are meant more for prototyping and analyzing the models rather than for using them for translation in a production-like setup.
Running the models in Marian will certainly much faster than in Python and it is certainly much easier than hacking Transformers to run with onxx runtime. Marian also offers further tricks to speed up the translation, e.g., by model quantization, which is however at the expense of the translation quality.
With both Marian and Tranformers, you can speed things up if you use GPU or if you narrow the beam width during decoding (attribute num_beams in the generate method in Transformers).


Instruction: How to speed up the 'Adding visible gpu devices' process in tensorflow with a 30 series card?

Input:
I get stuck with that for ~2 minute every time I run the code. Many people on the Internet said that it would only take a long time in the first run, but that's not my case. Although it doesn't make anything go wrong, it's pretty annoying. When I'm stuck, the system is under pretty low usage, including the CPU, system RAM, GPU, video memory.
I'm using Nvidia Geforce RTX 3070, Windows 10 x64 20H2.Here's my environment:
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
boto3                     1.16.47                  pypi_0    pypi
botocore                  1.19.47                  pypi_0    pypi
ca-certificates           2020.12.8            haa95532_0    defaults
certifi                   2020.12.5        py38haa95532_0    defaults
click                     7.1.2                    pypi_0    pypi
cudatoolkit               11.0.221             h74a9793_0    defaults
freetype                  2.10.4               hd328e21_0    defaults
intel-openmp              2020.2                      254    defaults
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.0.0                    pypi_0    pypi
jpeg                      9b                   hb83a4c4_2    defaults
keras                     2.4.3                    pypi_0    pypi
libpng                    1.6.37               h2a8f88b_0    defaults
libtiff                   4.1.0                h56a325e_1    defaults
libuv                     1.40.0               he774522_0    defaults
lz4-c                     1.9.2                hf4a77e7_3    defaults
mkl                       2020.2                      256    defaults
mkl-service               2.3.0            py38h196d8e1_0    defaults
mkl_fft                   1.2.0            py38h45dec08_0    defaults
mkl_random                1.1.1            py38h47e9c7a_0    defaults
ninja                     1.10.2           py38h6d14046_0    defaults
numpy                     1.19.2           py38hadc3359_0    defaults
numpy-base                1.19.2           py38ha3acd2a_0    defaults
olefile                   0.46                       py_0    defaults
openssl                   1.1.1i               h2bbff1b_0    defaults
pillow                    8.0.1            py38h4fa10fc_0    defaults
pip                       20.3.3           py38haa95532_0    defaults
python                    3.8.5                h5fd99cc_1    defaults
pytorch                   1.7.1           py3.8_cuda110_cudnn8_0    pytorch
regex                     2020.11.13               pypi_0    pypi
s3transfer                0.3.3                    pypi_0    pypi
sacremoses                0.0.43                   pypi_0    pypi
scikit-learn              0.24.0                   pypi_0    pypi
scipy                     1.6.0                    pypi_0    pypi
sentencepiece             0.1.94                   pypi_0    pypi
setuptools                51.0.0           py38haa95532_2    defaults
six                       1.15.0           py38haa95532_0    defaults
sklearn                   0.0                      pypi_0    pypi
sqlite                    3.33.0               h2a8f88b_0    defaults
tb-nightly                2.5.0a20210101           pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
thulac                    0.2.1                    pypi_0    pypi
tk                        8.6.10               he774522_0    defaults
torchaudio                0.7.2                      py38    pytorch
torchvision               0.8.2                py38_cu110    pytorch
transformers              2.1.1                    pypi_0    pypi
typing_extensions         3.7.4.3                    py_0    defaults
vc                        14.2                 h21ff451_1    defaults
vs2015_runtime            14.27.29016          h5e58377_2    defaults
wheel                     0.36.2             pyhd3eb1b0_0    defaults
wincertstore              0.2                      py38_0    defaults
xz                        5.2.5                h62dcd97_0    defaults
zlib                      1.2.11               h62dcd97_4    defaults
zstd                      1.4.5                h04227a9_0    defaults

Although I'm using PyTorch, it's tensorflow rather than PyTorch to blame(according to the logs). I got the same issue with pure tensorflow 2.3
2021-01-03 01:17:50.516100: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.622054: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-01-03 01:17:52.645796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-01-03 01:17:52.645998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.649575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-01-03 01:17:52.649707: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-01-03 01:17:52.649827: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-01-03 01:17:52.649928: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-01-03 01:17:52.651954: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-01-03 01:17:52.660165: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-01-03 01:17:52.660416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 01:17:52.660971: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-03 01:17:52.668967: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x19659fe67d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-03 01:17:52.669132: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-03 01:17:52.669395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-01-03 01:17:52.669576: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.669683: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-01-03 01:17:52.669790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-01-03 01:17:52.669896: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-01-03 01:17:52.670072: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-01-03 01:17:52.670201: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-01-03 01:17:52.670365: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-01-03 01:17:52.670542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 01:18:37.097681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-03 01:18:37.097876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2021-01-03 01:18:37.098025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2021-01-03 01:18:37.098301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:0a:00.0, compute capability: 8.6)
2021-01-03 01:18:37.101296: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1960330d0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-03 01:18:37.101474: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6
args:
Namespace(articles_per_title=10, device='0,1,2,3', length=1000, model_config='config/model_config_small.json', model_path='model/final_model', no_wordpiece=False, repetition_penalty=1.0, save_path='generated/', segment=False, temperature=2.0, titles='我', titles_file='', tokenizer_path='cache/vocab_small.txt', topk=10, topp=0)

I noticed that the tensorflow installation guide for GPU users said that GPUs with Ampere architecture may encounter this issue and can solve that by using export CUDA_CACHE_MAXSIZE=2147483648 to expand the default JIT cache. It doesn't work with Windows. I searched my environment variables and none of them namedCUDA_CACHE_MAXSIZE.
I tried adding that on my own, but it still takes a long time to pass Adding Visible Devices 0.
What should I do?

Output:
Just go to Windows Environment Variables and set CUDA_CACHE_MAXSIZE=2147483648 under system variables.
And you need a REBOOT,then everything will be fine.
You are lucky enough to get an Ampere card, since they're out of stock everywhere.


Instruction: Fairseq Transform model not working (Float can't be cast to long)

Input:
I've installed python 3.8, pytorch 1.7, and fairseq 0.10.1, on a new machine, then copied in a script and model from a machine with python 3.6, pytorch 1.4 and fairseq 0.9.0, where it is working.
The model is loaded and prepared with:
model = TransformerModel.from_pretrained(...)
model.eval()
model.cuda()

Then used with:
inputs = [model.binarize(encode(src, str)) for str in texts]
batched_hypos = model.generate(inputs, beam)

inputs looks like e.g. [tensor([ 116, 1864,  181,    6,    2]), tensor([    5,   432,     7,   2])]
It asserts, with the last bit of the call stack being:
  ...
    batched_hypos = model.generate(inputs, beam)
  File "/path/to/fairseq/hub_utils.py", line 125, in generate
    sample = self._build_sample(tokens)
  File "/path/to/fairseq/hub_utils.py", line 196, in _build_sample
    assert torch.is_tensor(src_tokens)

If instead I use fairseq-interactive from the commandline it fails with RuntimeError: result type Float can't be cast to the desired output type Long. (Full stack trace below.)
As using the cli also fails, my hunch is that my model built with fairseq 0.9.x cannot be used with fairseq 0.10.x. If so, is there a way to update the model (i.e. without having to retrain it). And if not, what could the problem be, and how do I fix it?
BTW, exactly the same error if I add --cpu to the commandline args, so the GPU or cuda version can be eliminated as a possible cause.
$ fairseq-interactive path/to/dicts --path models/big.pt --source-lang ja --target-lang en  --remove-bpe sentencepiece

  File "/path/to/bin/fairseq-interactive", line 11, in <module>
    sys.exit(cli_main())
  File "/path/to/lib/python3.8/site-packages/fairseq_cli/interactive.py", line 190, in cli_main
    main(args)
  File "/path/to/lib/python3.8/site-packages/fairseq_cli/interactive.py", line 149, in main
    translations = task.inference_step(generator, models, sample)
  File "/path/to/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 265, in inference_step
    return generator.generate(models, sample, prefix_tokens=prefix_tokens)
  File "/path/to/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/lib/python3.8/site-packages/fairseq/sequence_generator.py", line 113, in generate
    return self._generate(model, sample, **kwargs)
  File "/path/to/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/lib/python3.8/site-packages/fairseq/sequence_generator.py", line 376, in _generate
    cand_scores, cand_indices, cand_beams = self.search.step(
  File "/path/to/lib/python3.8/site-packages/fairseq/search.py", line 81, in step
    torch.div(self.indices_buf, vocab_size, out=self.beams_buf)
RuntimeError: result type Float can't be cast to the desired output type Long

Output:
(UPDATE: the below instructions install pytorch without GPU support. Going back to using the pytorch channel gets GPU support but fairseq breaks again. I've not yet cracked the secret code to get everything working together.)
Solved this by wiping conda and starting again; I've decided to self-answer, rather than delete the question, as those error messages turned out to be useless (to put it politely) so maybe it will help someone else when they google.
First: I actually had fairseq 0.9.0 installed. Even though 0.10.1 was listed  first on conda-forge. This obviously meant my hunch was wrong, and something more obscure was at work. I then couldn't get uninstall or upgrade to work. Hence my decision to wipe anaconda completely and start again.
Second, I noticed something deep in the conda documentation saying to install everything in one go, to avoid conflicts. Not my definition of how a package manager should work, but anyway.
Third, I created a "test" conda environment, rather than using the "base" default. I suspect this had nothing to do with getting it to work, but I mention it just in case.
So, my successful install command was:
conda install -c conda-forge pytorch cudatoolkit=11.0 nvidia-apex fairseq==0.10.1 sentencepiece

This gives me python 3.7.9 (not the 3.8.5 the OS has installed), pytorch 1.7.1, fairseq 0.10.1, and sentencepiece 0.1.92.


Instruction: AttributeError: 'tuple' object has no attribute 'size'

Input:
UPDATE: after looking back on this question, most of the code was unnecessary. In summary, the hidden layer of a Pytorch RNN needs to be a torch tensor. When I posted the question, the hidden layer was a tuple.
Below is my data loader.
from torch.utils.data import TensorDataset, DataLoader

def batch_data(log_returns, sequence_length, batch_size):
    """
    Batch the neural network data using DataLoader
    :param log_returns: asset's daily log returns
    :param sequence_length: The sequence length of each batch
    :param batch_size: The size of each batch; the number of sequences in a batch
    :return: DataLoader with batched data
    """

    # total number of batches we can make
    n_batches = len(log_returns)//batch_size

    # Keep only enough characters to make full batches
    log_returns = log_returns[:n_batches * batch_size]

    y_len = len(log_returns) - sequence_length

    x, y = [], []
    for idx in range(0, y_len):
        idx_end = sequence_length + idx
        x_batch = log_returns[idx:idx_end]
        x.append(x_batch)
        # only making predictions after the last word in the batch
        batch_y = log_returns[idx_end]
        y.append(batch_y)

    # create tensor datasets
    x_tensor = torch.from_numpy(np.asarray(x))
    y_tensor = torch.from_numpy(np.asarray(y))

    # make x_tensor 3-d instead of 2-d
    x_tensor = x_tensor.unsqueeze(-1)

    data = TensorDataset(x_tensor, y_tensor)

    data_loader = DataLoader(data, shuffle=False, batch_size=batch_size)

    # return a dataloader
    return data_loader

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

        return hidden

I don't know what is wrong. When I try to start training the model, I am getting the error message:
AttributeError: 'tuple' object has no attribute 'size'

Output:
The issue comes from the fact that hidden (in the forward definition) isn't a Torch.Tensor. Therefore, r_output, hidden = self.gru(nn_input, hidden) raises a rather confusing error without specifying exaclty what's wrong in the arguments. Altough you can see it's raised inside a nn.RNN function named check_hidden_size()...
I was confused at first, thinking that the second argument of nn.RNN: h0 was a tuple containing (hidden_state, cell_state). Same can be said ofthe second element returned by that call: hn. That's not the case h0 and hn are both Torch.Tensors. Interestingly enough though, you are able to unpack stacked tensors:
>>> z = torch.stack([torch.Tensor([1,2,3]), torch.Tensor([4,5,6])])
>>> a, b = z
>>> a, b
(tensor([1., 2., 3.]), tensor([4., 5., 6.]))

You are supposed to provide a tensor as the second argument of a nn.GRU __call__.

Edit - After further inspection of your code I found out that you are converting hidden back again to a tuple... In cell [14] you have hidden = tuple([each.data for each in hidden]). Which basically overwrites the modification you did in init_hidden with torch.stack.
Take a step back and look at the source code for RNNBase the base class for RNN modules. If the hidden state is not given to the forward it will default to:
if hx is None:
    num_directions = 2 if self.bidirectional else 1
    hx = torch.zeros(self.num_layers * num_directions,
                     max_batch_size, self.hidden_size,
                     dtype=input.dtype, device=input.device)

This is essentially the exact init as the one you are trying to implement. Granted you only want to reset the hidden states on every epoch, (I don't see why...). Anyhow, a basic alternative would be to set hidden to None at the start of an epoch, passed as it is to self.forward_back_prop then to rnn, then to self.rnn which will in turn default initialize it for you. Then overwrite hidden with the hidden state returned by that RNN forward call.
To summarize, I've only kept the relevant parts of the code. Remove the init_hidden function from AssetGRU and make those modifications:
def forward_back_prop(rnn, optimizer, criterion, inp, target, hidden):
    ...
    if hidden is not None:
        hidden = hidden.detach()
    ...
    output, hidden = rnn(inp, hidden)
    ...
    return loss.item(), hidden

def train_rnn(rnn, batch_size, optimizer, criterion, n_epochs, show_every_n_batches):
    ...
    for epoch_i in range(1, n_epochs + 1):

        hidden = None

        for batch_i, (inputs, labels) in enumerate(train_loader, 1):
            loss, hidden = forward_back_prop(rnn, optimizer, criterion,
                                             inputs, labels, hidden)
            ...

    ...


Instruction: Dimension mismatch in loading data

Input:
I have this codes as follows.
import torch.nn.functional as f
train_on_gpu=True
class CnnLstm(nn.Module):
    def __init__(self):
        super(CnnLstm, self).__init__()
        self.cnn = CNN()
        self.rnn = nn.LSTM(
            input_size=180000,
            hidden_size=256,
            num_layers=2,
            batch_first=True)
        self.linear = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        print('before forward ')
        print(x.shape)
        batch_size, time_steps, channels, height, width = x.size()
        c_in = x.view(batch_size * time_steps, channels, height, width)
        _, c_out = self.cnn(c_in)
        r_in = c_out.view(batch_size, time_steps, -1)
        r_out, (_, _) = self.rnn(r_in)
        r_out2 = self.linear(r_out[:, -1, :])
        return f.log_softmax(r_out2, dim=1)

cnnlstm_model = CnnLstm().to(device)
optimizer = torch.optim.Adam(cnnlstm_model.parameters(), lr=learning_rate)
#optimizer = torch.optim.SGD(cnnlstm_model.parameters(), lr=learning_rate)
#criterion = nn.functional.nll_loss()
criterion = nn.CrossEntropyLoss()
# Train the model
n_total_steps = len(train_dl)
num_epochs = 20
for epoch in range(num_epochs):
    t_losses=[]
    for i, (images, labels) in enumerate(train_dl):
        # origin shape: [5, 3, 300, 300]
        # resized: [5, 300, 300]
        print('load data '+str(images.shape))
        images = np.expand_dims(images, axis=1)
        print('after expand ')
        print(images.shape)
        images = torch.FloatTensor(images)
        images, labels = images.cuda(), labels.cuda()
        images, labels = Variable(images), Variable(labels)
        optimizer.zero_grad()
        outputs = cnnlstm_model(images)
        loss = criterion(outputs, labels)
        t_losses.append(loss)
        loss.backward()
        optimizer.step()

Three places are printing out.
(1)
print('load data '+str(images.shape))

(2)
print('after expand ')
print(images.shape)

(3)
print('before forward ')
print(x.shape)

I have batch size of 5 images.
Loading 2629 batches and only last batch has issues.
Earlier batches of loading images has no issues and loaded as
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
.
.
.
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])

At the last batch loading,
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
before forward
torch.Size([15, 300, 300])

Why do I have the 'before forward' log printed twice? Futhermore, it's not same shape.
What could be wrong?
EDIT:
This is code for loading data.
inputH = input_size
inputW = input_size
#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_resize_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
train_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.RandomHorizontalFlip(),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
valid_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
test_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                        tt.ToTensor(),
                        tt.Normalize(*stats)])

#Create dataset
train_ds = ImageFolder('./data/train', train_tfms)
valid_ds = ImageFolder('./data/valid', valid_tfms)
test_ds = ImageFolder('./data/test', test_tfms)

from torch.utils.data.dataloader import DataLoader
batch_size = 5

#Training data loader
train_dl = DataLoader(train_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Validation data loader
valid_dl = DataLoader(valid_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Test data loader
test_dl = DataLoader(test_ds, 1, shuffle = False, num_workers = 1, pin_memory=True)

Output:
I made some changes to data loader and finally it worked.
    class DataLoader:
        stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
        @staticmethod
        def get_train_data(batch_size):
            train_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                                 tt.RandomHorizontalFlip(),
                                 tt.ToTensor(),
                                 tt.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])
            train_ds = ImageFolder('./data/train', train_tfms)
            return torch.utils.data.DataLoader(
                train_ds,
                batch_size=batch_size,
                shuffle=True,
                num_workers = 8,
                pin_memory=True)

        @staticmethod
        def get_validate_data(valid_batch_size):
            valid_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                                 tt.ToTensor(),
                                 tt.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])
            valid_ds = ImageFolder('./data/valid', valid_tfms)
            return torch.utils.data.DataLoader(
                valid_ds,
                batch_size=valid_batch_size,
                shuffle=True,
                num_workers = 8,
                pin_memory=True)

        @staticmethod
        def get_test_data(test_batch_size):
            test_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                                tt.ToTensor(),
                                tt.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])
            test_ds = ImageFolder('./data/test', test_tfms)
            return torch.utils.data.DataLoader(
                test_ds,
                batch_size=test_batch_size,
                shuffle=False,
                num_workers = 1,
                pin_memory=True)


Instruction: Using PyTorch nn.Sequential() to define a network in a flexible way but with results beyond expectation

Input:
I tried to define a network in a more flexible way using nn.Sequential so that I can define its number of layers according to layernum:
seed = 0
torch.manual_seed(seed)
# ====== net_a =====
layers = [ nn.Linear(7, 64), nn.Tanh()]
for i in range(layernum-1): # layernum = 3
    layers.append(nn.Linear(64, 64))
    layers.append(nn.Tanh())
layers.append(nn.Linear(64, 8))
net_x = nn.Sequential(*layers)
net_y = nn.Sequential(*layers)
net_z = nn.Sequential(*layers)

# ====== net_b =====
net_x = nn.Sequential(
    nn.Linear(7, 64),
    nn.Tanh(),
    nn.Linear(64, 64),
    nn.Tanh(),
    nn.Linear(64, 64),
    nn.Tanh(),
    nn.Linear(64, 8),
)
net_y = nn.Sequential(
    #... same as net_x
)
net_z = nn.Sequential(
    #... same as net_x
)

# print(net_x)
# print(net_x[0].weight)

I use both of them individually, i.e. they are in the same .py file, but I use either of them and make the other as comments. Both of them consist of 3 networks with respect to 3 dimentions (x,y, and z).
I expected them to be the same network with same training performance.
The structure seems to be the same according to print(net_x):
    # Sequential(
    #   (0): Linear(in_features=7, out_features=64, bias=True)
    #   (1): Tanh()
    #   (2): Linear(in_features=64, out_features=64, bias=True)
    #   (3): Tanh()
    #   (4): Linear(in_features=64, out_features=64, bias=True)
    #   (5): Tanh()
    #   (6): Linear(in_features=64, out_features=8, bias=True)
    # )

But their initial weights are different according to print(net_x[0].weight):
print(net_x[0].weight) # net_a
    # tensor([[-0.0028,  0.2028, -0.3111, -0.2782, -0.1456,  0.1014, -0.0075],
    #         [ 0.2997, -0.0335,  0.1000, -0.1142, -0.0743, -0.3611, -0.2503],
    #         ......

print(net_x[0].weight) # net_b
    # tensor([[ 0.2813,  0.2968,  0.0078,  0.1518,  0.3776, -0.3247,  0.0071],
    #         [ 0.3448, -0.0988, -0.2798,  0.3347,  0.3581,  0.2229,  0.2841],
    #         ......

======ADDED=====
I trained the network like this:
def train_on_batch(x, y, net, stepsize=innerstepsize):
    x = totorch(x)
    y = totorch(y)
    if(use_cuda):
        x,y = x.cuda(),y.cuda()
    net.zero_grad()
    ypred = net(x)
    loss = (ypred - y).pow(2).mean()
    loss.backward()
    for param in net.parameters():
        param.data -= stepsize * param.grad.data

iteration = 100
for iter in range(iteration):

    # TRAIN
    PrepareSample() # get in_support
    for i in range(tnum_support):
        out_x = trajectory_support_x[i,1:9]
        out_y = trajectory_support_y[i,1:9]
        out_z = trajectory_support_z[i,1:9]
        # Do SGD on this task
        for _ in range(innerepochs): # SGD 1 times
            train_on_batch(in_support[i], out_x, net_x)
            train_on_batch(in_support[i], out_y, net_y)
            train_on_batch(in_support[i], out_z, net_z)

    # TEST
    if iter==0 or (iter+1) % 10 == 0:
        ind = [0,1,2,3,4,5,6,7,8,9]
        loss = [0,0,0,0,0,0]
        for i in range(tnum_test):
            inputs = in_test[i]
            outputs_x = trajectory_test_x[i].tolist()
            x_test = trajectory_test_x[i,[0,9]]
            y_test = trajectory_test_x[i,1:9]
            pred_x = np.hstack((x_test[0],predict(inputs, net_x),x_test[1]))
            loss[i] = np.square(predict(inputs, net_x) - y_test).mean() # mse

            inputs = in_test[i]
            outputs_y = trajectory_test_y[i].tolist()
            x_test = trajectory_test_y[i,[0,9]]
            y_test = trajectory_test_y[i,1:9]
            pred_y = np.hstack((x_test[0],predict(inputs, net_y),x_test[1]))
            loss[i+2] = np.square(predict(inputs, net_y) - y_test).mean() # mse

            inputs = in_test[i]
            outputs_z = trajectory_test_z[i].tolist()
            x_test = trajectory_test_z[i,[0,9]]
            y_test = trajectory_test_z[i,1:9]
            pred_z = np.hstack((x_test[0],predict(inputs, net_z),x_test[1]))
            loss[i+4] = np.square(predict(inputs, net_z) - y_test).mean() # mse

        iterNum.append(iter+1)
        avgloss.append(np.mean(loss))

both of them are trained with exactly the same data (they are in the same .py file and of course use the same data).
=====This is avgloss of net_a:

=====This is avgloss of net_a with torch.manual_seed(seed) before every network definition:

=====This is avgloss of net_b:

=====This is avgloss of net_b with torch.manual_seed(seed) before every network definition:

The training of net_a is weird, the MSE is high at initial time, and didn't reduce. On the contrary, the training of net_b seems common, the MSE is relatively low at first, and reduce to a smaller value after 100 iterations.
Is anyone know how to fix this?  I would like to go through different layer number, layer size, activation functions of the network. I don't want to write every network of specified hyper-parameters.

Output:
The random state is different after torch initialized the weights in the first network. You need to reset the random state to keep the same initialization by calling torch.manual_seed(seed) after the definition of the first network and before the second one.

The problem lies in net_x/y/z -- it will be perfectly fine if it were just net_x. When you use nn.Sequential, it does not create new modules but instead stores a reference to the given module. So, in your first definition, you only have one copy of the layers, meaning that all net_x/y/z are shared-weight. They have independent weights in your second definition, which is naturally what we are after.

You might definite it like this instead:
def get_net():
    layers = [ nn.Linear(7, 64), nn.Tanh()]
    for i in range(layernum-1): # layernum = 3
        layers.append(nn.Linear(64, 64))
        layers.append(nn.Tanh())
    layers.append(nn.Linear(64, 8))
    return layers

net_x = nn.Sequential(*get_net())
net_y = nn.Sequential(*get_net())
net_z = nn.Sequential(*get_net())

Each time get_net is called, it creates a new copy.


Instruction: output.grad None even after loss.backward()

Input:
I'm confused...
My model output:
tensor([[0.0000,0.1537],...],grad_fn=<ReluBackward0>)
If I use print(output.grad) it gives me None but even after gradient computation with
loss.backward() I get the same result, which again is None...
Even with
with torch.set_grad_enabled(True):
added, still the same.
I've tried now with multiple model variants, always the same.
I was achieving good result with my model there seemed no problem, now I see this and I'm not sure any more if there not might be a major flaw I didn't recognized so far.
But my model is learning, it improves so I guess it has to?
Why do I get None instead of an actual value?

Output:
You are getting None because the gradient is only stored on the .grad property for leaf tensors. Those are tensors that don't have parents in the computational graph.
You can check whether a tensor is a leaf or not with is_leaf:
>>> x = torch.FloatTensor([1,2,3])
>>> x.requires_grad = True
>>> x.sum().backward() # backward pass

>>> x.is_leaf
True
>>> x.grad
tensor([1., 1., 1.])

The tensor you printed shows grad_fn=<ReluBackward0> indicating it's the result of a ReLU layer, and therefore not a leaf tensor.
Here is an example of a non-leaf tensor:
>>> x = torch.FloatTensor([1,2,3])
>>> x.requires_grad=True
>>> z = x.sum()
>>> z.backward()

>>> z.is_leaf
False
>>> z.grad
None

Notice that z will show as tensor(6., grad_fn=<SumBackward0>).

Actually accessing .grad will give a warning:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.

And if you want to access the gradient on non-leaf tensors, following the warning message:
>>> z.retain_grad()
>>> z = x.sum()
>>> z.retain_grad()
>>> z.backward()

>>> z.is_leaf
False
>>> z.grad
tensor(1.)


Instruction: Understanding convolutional layers shapes

Input: I've been reading about convolutional nets and I've programmed a few models myself. When I see visual diagrams of other models it shows each layer being smaller and deeper than the last ones. Layers have three dimensions like 256x256x32. What is this third number? I assume the first two numbers are the number of nodes but I don't know what the depth is.

Output:
TLDR; 256x256x32 refers to the layer's output shape rather than the layer itself.

There are many articles and posts out there explaining how convolution layers work. I'll try to answer your question without going into too many details, just focusing on shapes.
Assuming you are working with 2D convolution layers, your input and output will both be three-dimensional. That is, without considering the batch which would correspond to a 4th axis... Therefore, the shape of a convolution layer input will be (c, h, w) (or (h, w, c) depending on the framework) where c is the number of channels, h is the width of the input and w the width. You can see it as a c-channel hxw image.
The most intuitive example of such input is the input of the first convolution layer of your convolutional neural network: most likely an image of size hxw with c channels for example c=1 for greyscale or c=3 for RGB...
What's important is that for all pixels of that input, the values on each channel gives additional information on that pixel. Having three channels will give each pixel ('pixel' as in position in the 2D input space) a richer content than having a single. Since each pixel will be encoded with three values (three channels) vs. a single one (one channel). This kind of intuition about what channels represent can be extrapolated to a higher number of channels. As we said an input can have c channels.
Now going back to convolution layers, here is a good visualization. Imagine having a 5x5 1-channel input. And a convolution layer consisting  of a single 3x3 filter (i.e. kernel_size=3)

input
filter
convolution
output

shape
(1, 5, 5)
(3, 3)

(3,3)

representation

Now keep in mind the dimension of the output will depend on the stride and padding of the convolution layer. Here the shape of the output is the same as the shape of the filter, it does not necessarily have to be! Take an input shape of (1, 5, 5), with the same convolution settings, you would end up with a shape of (4, 4) (which is different from the filter shape (3, 3).
Also, something to note is that if the input had more than one channel: shape (c, h, w), the filter would have to have the same number of channels. Each channel of the input would convolve with each channel of the filter and the results would be averaged into a single 2D feature map. So you would have an intermediate output of (c, 3, 3), which after averaging over the channels, would leave us with (1, 3, 3)=(3, 3). As a result, considering a convolution with a single filter, however many input channels there are, the output will always have a single channel.
From there what you can do is assemble multiple filters on the same layer. This means you define your layer as having k 3x3 filters. So a layer consists k filters. For the computation of the output, the idea is simple: one filter gives a (3, 3) feature map, so k filters will give k (3, 3) feature maps. These maps are then stacked into what will be the channel dimension. Ultimately, you're left with an output shape of... (k, 3, 3).
Let k_h and k_w, be the kernel height and kernel width respectively. And h', w' the height and width of one outputted feature map:

input
layer
output

shape
(c, h, w)
(k, c, k_h, k_w)
(k, h', w')

description
c-channel hxw feature map
k filters of shape (c, k_h, k_w)
k-channel h'xw' feature map

Back to your question:

Layers have 3 dimensions like 256x256x32. What is this third number? I assume the first two numbers are the number of nodes but I don't know what the depth is.

Convolution layers have four dimensions, but one of them is imposed by your input channel count. You can choose the size of your convolution kernel, and the number of filters. This number will determine is the number of channels of the output.
256x256 seems extremely high and you most likely correspond to the output shape of the feature map. On the other hand, 32 would be the number of channels of the output, which... as I tried to explain is the number of filters in that layer. Usually speaking the dimensions represented in visual diagrams for convolution networks correspond to the intermediate output shapes, not the layer shapes.
As an example, take the VGG neural network:

Very Deep Convolutional Networks for Large-Scale Image Recognition
Input shape for VGG is (3, 224, 224), knowing that the result of the first convolution has shape (64, 224, 224) you can determine there is a total of 64 filters in that layer.
As it turns out the kernel size in VGG is 3x3. So, here is a question for you: knowing there is a single bias parameter per filter, how many total parameters are in VGG's first convolution layer?


Instruction: How to shuffle the batches themselves in pytorch?

Input:
How to keep the sequences in each batch unshuffled, while shuffling the batches?
Inspired by the question asked here.

Output:
While this is not a direct answer to your question. I want to address an issue with the answer you posted yourself. In my opinion, doing the following, is a very bad idea:
dataloader = random.sample(list(dataloader), len(dataloader))

This defeats the whole purpose of creating a dataset and a data loader in the first place. Because as soon as you call list(dataloader) you end up compiling down your dataset into a single list of tensors. In other words, it will call __getitem__ for each index in the dataset. A data loader is designer to load data batch by batch (or more depending on the number of workers) avoiding to load the entire dataset in memory at once.
This is even more important when working with images which requires image loading from the file system. This is critical and I believe you shouldn't be doing this at all.
Take a look here, with a dummy dataset:
class DS(Dataset):
    def __getitem__(self, _):
        return torch.rand(100)

    def __len__(self):
        return 10000

dl = DataLoader(DS(), batch_size=16)
x = list(dl)

Here x will contain 10,000 tensors of size 100, which your computer can handle. Now imagine having a dataset made up of 10,000 512x512 RGB images, you just can't hold that much in memory!
Futhermore, something I haven't even mentioned is data augmentation. Which is only possible when retaining a data loader (i.e. a generator). So the transformation are computed on the input data at runtime vs at compile-time (if you will) when using list(dataloader).

I would instead suggest you make your Dataset generate that unshuffled sequence for each item, then make a DataLoader out of it with shuffle=True. This feels a lot more natural than generating a DataLoader only to compile it down. Use your dataset class as it's supposed to. It should be the one constructing each sequence (i.e. datapoint), or as @Prune puts it a "single observation object".


Instruction: Multi Layer Perceptron Deep Learning in Python using Pytorch

Input:
I am having errors in executing the train function of my code in MLP.
This is the error:
mat1 and mat2 shapes cannot be multiplied (128x10 and 48x10)
My code for the train function is this:
class net(nn.Module):
def __init__(self, input_dim2, hidden_dim2, output_dim2):
    super(net, self).__init__()
    self.input_dim2 = input_dim2
    self.fc1 = nn.Linear(input_dim2, hidden_dim2)
    self.relu = nn.ReLU()
    self.fc2 = nn.Linear(hidden_dim2, hidden_dim2)
    self.fc3 = nn.Linear(hidden_dim2, output_dim2)
def forward(self, x):
  x = self.fc1(x)
  x = self.relu(x)
  x = self.fc2(x)
  x = self.relu(x)
  x = self.fc3(x)
  x = F.softmax(self.fc3(x))

  return x

model = net(input_dim2, hidden_dim2, output_dim2) #create the network
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.RMSprop(model.parameters(), lr = learning_rate2)

def train(num_epochs2):
for i in range(num_epochs2):
  tmp_loss = []
  for (x,y) in train_loader:
    print(y.shape)
    print(x.shape)
    outputs = model(x) #forward pass
    print(outputs.shape)
    loss = criterion(outputs, y) #loss computation
    tmp_loss.append(loss.item()) #recording the loss
    optimizer.zero_grad() #all the accumulated gradient
    loss.backward()  #auto-differentiaton - accumulation of gradient
    optimizer.step() # a gradient step

  print("Loss at {}th epoch: {}".format(i, np.mean(tmp_loss)))

I don't know where I'm wrong. My code seems to work okay.

Output:
From the limited message, I guess the place you are wrong are the following snippets:
x = self.fc3(x)
x = F.softmax(self.fc3(x))

Try to replace with:
x = self.fc3(x)
x = F.softmax(x)

A good question should include: error backtrace information and complete toy example which could repeat the errors!


Instruction: Backtransforming a PyTorch Tensor

Input:
I have trained a WGAN on the CelebA dataset in PyTorch following this youtube video. Since I do this on Google Cloud Platform where TensorBoard is not availabe, I save one figure of generated images by the GAN every epoch to see how the GAN is actually doing.
Now, the saved pdf files look sth like this: generated images. Unfortunately, this is not really readable, and I suspect this has to do with the preprocessing I do:
trafo = transforms.Compose(
                    [transforms.Resize(size = (64, 64)),
                    transforms.ToTensor(),
                    transforms.Normalize( mean = (0.5,), std = (0.5,))])

Is there any way to kind of undo this transformation when I save the image?
Currently, I save the image every epoch as follows:
visualization = torchvision.utils.make_grid(
                           tensor = gen(fixed_noise),
                           nrow = 8,
                           normalize = False)
plt.savefig("generated_WGAN_" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".pdf")

Also, I should probably mention that in the Jupyter notebook, I get the following warning:
"Clipping input data to the valid range for imshow with RGB data ([0..1]) for floats or [0..255] for integers)."

Output:
It seems like your output pixel values are in range [-1, 1] (please verify this).
Therefore, when you save the images, the negative part is being clipped (as the error message you got suggests).
Try:
visualization = torchvision.utils.make_grid(
                           tensor = torch.clamp(gen(fixed_noise), -1, 1) * 0.5 + 0.5,  # from [-1, 1] -> [0, 1]
                           nrow = 8,
                           normalize = False)
plt.savefig("generated_WGAN_" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".pdf")


Instruction: How to map element in pytorch tensor to id?

Input:
Given a tensor:
A = torch.tensor([2., 3., 4., 5., 6., 7.])

Then, give each element in A an id:
id = torch.arange(A.shape[0], dtype = torch.int)   # tensor([0,1,2,3,4,5])

In other words, id of 2. in A is 0 and  id of 3. in A is 1:
2. -> 0
3. -> 1
4. -> 2
5. -> 3
6. -> 4
7. -> 5

Then, I have a new tensor:
B = torch.tensor([3., 6., 6., 5., 4., 4., 4.])

In pytorch, is there any way in Pytorch to map each element in B to id?
In other words, I want to obtain tensor([1, 4, 4, 3, 2, 2, 2]), in which each element is id of the element in B.

Output:
I don't think there is such a function in PyTorch to map a tensor.
It seems quite unreasonable to solve this by comparing each value from B to values from B.
Here are two possible solutions to solve this problem.

Using a dictionary as a map
You can use a dictionary. Not so not much of a pure-PyTorch solution but will most probably be the fastest and safest way...
Just create a dict to map each element to an id, then use it to map B:
>>> map = {x.item(): i for i, x in enumerate(A)}

>>> torch.tensor([map[x.item()] for x in B])
tensor([1, 4, 4, 3, 2, 2, 2])

Change of basis approach
An alternative only using torch.Tensors. This will require the values you want to map - the content of A - to be integers because they will be used to index a tensor.

Encode the content of A into one-hot encodings:
>>> A_enc = torch.zeros((int(A.max())+1,)*2)
>>> A_enc[A, torch.arange(A.shape[0])] = 1

>>> A_enc
tensor([[0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0., 0., 0.],
        [0., 0., 0., 1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0.]])

We'll use A_enc as our basis to map integers:
>>> v = torch.argmax(A_enc, dim=0)
tensor([0, 0, 0, 1, 2, 3, 4, 5])

Now, given an integer for instance x=3, we can encode it into a one-hot-encoding: x_enc = [0, 0, 0, 1, 0, 0, 0, 0]. Then, use v to map it. With a simple dot product you can get the mapping of x_enc: here <v/x_enc> gives 1 which is the desired result (first element of mapped-B). But instead of giving x_enc, we will compute the matrix multiplication between v and encoded-B. First encode B then compute the matrix multiplcition vxB_enc:
>>> B_enc = torch.zeros(A_enc.shape[0], B.shape[0])
>>> B_enc[B, torch.arange(B.shape[0])] = 1

>>> B_enc
tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 1., 1.],
        [0., 0., 0., 1., 0., 0., 0.],
        [0., 1., 1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]])

>>> v@B_enc.long()
tensor([1, 4, 4, 3, 2, 2, 2])

Note - you will have to define your tensors with Long type.


Instruction: How to access Spark DataFrame data in GPU from ML Libraries such as PyTorch or Tensorflow

Input:
Currently I am studying the usage of Apache Spark 3.0 with Rapids GPU Acceleration. In the official spark-rapids docs I came across this page which states:

There are cases where you may want to get access to the raw data on the GPU, preferably without copying it. One use case for this is exporting the data to an ML framework after doing feature extraction.

To me this sounds as if one could make data that is already available on the GPU from some upstream Spark ETL process directly available to a framework such as Tensorflow or PyTorch. If this is the case how can I access the data from within any of these frameworks? If I am misunderstanding something here, what is the quote exactly referring to?

Output:
The link you references really only allows you to get access to the data still sitting on the GPU, but using that data in another framework, like Tensorflow or PyTorch is not that simple.
TL;DR; Unless you have a library explicitly setup to work with the RAPIDS accelerator you probably want to run your ETL with RAPIDS, then save it, and launch a new job to train your models using that data.
There are still a number of issues that you would need to solve.  We have worked on these in the case of XGBoost, but it has not been something that we have tried to tackle for Tensorflow or PyTorch yet.
The big issues are

Getting the data to the correct process. Even if the data is on the GPU, because of security, it is tied to a given user process.  PyTorch and Tensorflow generally run as python processes and not in the same JVM that Spark is running in. This means that the data has to be sent to the other process.  There are several ways to do this, but it is non-trivial to try and do it as a zero-copy operation.
The format of the data is not what Tensorflow or PyTorch want. The data for RAPIDs is in an arrow compatible format. Tensorflow and PyTorch have APIs for importing data in standard formats from the CPU, but it might take a bit of work to get the data into a format that the frameworks want and to find an API to let you pull it in directly from the GPU.
Sharing GPU resources. Spark only recently added in support for scheduling GPUs. Prior to that people would just launch a single spark task per executor and a single python process so that the python process would own the entire GPU when doing training or inference. With the RAPIDS accelerator the GPU is not free any more and you need a way to share the resources.  RMM provides some of this if both libraries are updated to use it and they are in the same process, but in the case of Pytorch and and Tensoflow they are typically in python processes so figuring out how to share the GPU is hard.


Instruction: How does PyTorch Tensor.index_select() evaluates tensor output?

Input:
I am not able to understand how complex indexing - non contiguous indexing of a tensor works. Here is a sample code and its output
import torch

def describe(x):
  print("Type: {}".format(x.type()))
  print("Shape/size: {}".format(x.shape))
  print("Values: \n{}".format(x))

indices = torch.LongTensor([0,2])
x = torch.arange(6).view(2,3)
describe(torch.index_select(x, dim=1, index=indices))

Returns output as

Type: torch.LongTensor Shape/size: torch.Size([2, 2]) Values:
tensor([[0, 2],
[3, 5]])

Can someone explain how did it arrive to this output tensor?
Thanks!

Output:
You are selecting the first (indices[0] is 0) and third (indices[1] is 2) tensors from x on the first axis (dim=0). Essentially, torch.index_select with dim=1 works the same as doing a direct indexing on the second axis with x[:, indices].
>>> x
tensor([[0, 1, 2],
        [3, 4, 5]])

So selecting columns (since you're looking at dim=1 and not dim=0) which indices are in indices. Imagine having a simple list [0, 2] as indices:
>>> indices = [0, 2]

>>> x[:, indices[0]] # same as x[:, 0]
tensor([0, 3])

>>> x[:, indices[1]] # same as x[:, 2]
tensor([2, 5])

So passing the indices as a torch.Tensor allows you to index on all elements of indices directly, i.e. columns 0 and 2. Similar to how NumPy's indexing works.
>>> x[:, indices]
tensor([[0, 2],
        [3, 5]])

Here's another example to help you see how it works. With x defined as x = torch.arange(9).view(3, 3) so we have 3 rows (a.k.a. dim=0) and 3 columns (a.k.a. dim=1).
>>> indices
tensor([0, 2]) # namely 'first' and 'third'

>>> x = torch.arange(9).view(3, 3)
tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])

>>> x.index_select(0, indices) # select first and third rows
tensor([[0, 1, 2],
        [6, 7, 8]])

>>> x.index_select(1, indices) # select first and third columns
tensor([[0, 2],
        [3, 5],
        [6, 8]])

Note: torch.index_select(x, dim, indices) is equivalent to x.index_select(dim, indices)


Instruction: Add blocks of values to a tensor at specific locations in PyTorch

Input:
I have a list of indices:
indx = torch.LongTensor([
    [ 0,  2,  0],
    [ 0,  2,  4],
    [ 0,  4,  0],
    [ 0, 10, 14],
    [ 1,  4,  0],
    [ 1,  8,  2],
    [ 1, 12,  0]
])

And I have a tensor of 2x2 blocks:
blocks = torch.FloatTensor([
    [[1.5818, 2.3108],
     [2.6742, 3.0024]],

    [[2.0472, 1.6651],
     [3.2807, 2.7413]],

    [[1.5587, 2.1905],
     [1.9231, 3.5083]],

    [[1.6007, 2.1426],
     [2.4802, 3.0610]],

    [[1.9087, 2.1021],
     [2.7781, 3.2282]],

    [[1.5127, 2.6322],
     [2.4233, 3.6836]],

    [[1.9645, 2.3831],
     [2.8675, 3.3770]]
])

What I want to do is to add each block at an index position to another tensor (i.e. so that it starts at that index). Let's assume that I want to add it to the following tensor:
a = torch.ones([2,18,18])

Is there any efficient way to do so? So far I came up only with:
i = 0
for b, x, y in indx:
   a[b, x:x+2, y:y+2] += blocks[i]
   i += 1

It is quite inefficient, I also tried to use index_add, but it did not work properly.

Output:
You are looking to index on three different dimensions at the same time. I had a look around in the documentation, torch.index_add will only receive a vector as index. My hopes were on torch.scatter but it doesn't to fit well to this problem. As it turns out you can achieve this pretty easily with a little work, the most difficult parts are the setup and teardown. Please hang on tight.
I'll use a simplified example here, but the same can be applied with larger tensors.
>>> indx
tensor([[ 0,  2,  0],
        [ 0,  2,  4],
        [ 0,  4,  0]]))

>>> blocks
tensor([[[1.5818, 2.3108],
         [2.6742, 3.0024]],

        [[2.0472, 1.6651],
         [3.2807, 2.7413]],

        [[1.5587, 2.1905],
         [1.9231, 3.5083]]])

>>> a
tensor([[[0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0.]]])

The main issue here is that you are looking index with slicing. That not possible in a vectorize form. To counter that though you can convert your a tensor into 2x2 chunks. This will be particulary handy since we will be able to access sub-tensors such as a[0, 2:4, 4:6] with just a[0, 1, 2]. Since the 2:4 slice on dim=1 will be grouped together on index=1 while the 4:6 slice on dim=0 will be grouped on index=2.
First we will convert a to tensor made up of 2x2 chunks. Then we will update with blocks. Finally, we will stitch back the resulting tensor into the original shape.

1. Converting a to a 2x2-chunks tensor
You can use a combination of torch.chunk and torch.cat (not torch.dog) twice: on dim=1 and dim=2. The shape of a is (1, h, w) so we're looking for a result of shape (1, h//2, w//2, 2, 2).
To do so we will unsqueeze two axes on a:
>>> a_ = a[:, None, :, None, :]
>>> a_.shape
torch.Size([1, 1, 6, 1, 6])

Then make 3 chunks on dim=2, then concatenate on dim=1:
>>> a_row_chunks = torch.cat(torch.chunk(a_, 3, dim=2), dim=1)
>>> a_row_chunks.shape
torch.Size([1, 3, 2, 1, 6])

And make 3 chunks on dim=4, then concatenate on dim=3:
>>> a_col_chunks  = torch.cat(torch.chunk(a_row_chunks, 3, dim=4), dim=3)
>>> a_col_chunks.shape
torch.Size([1, 3, 2, 3, 2])

Finally reshape all.
>>> a_chunks = a_col_chunks.reshape(1, 3, 3, 2, 2)

Create a new index with adjusted values for our new tensor with. Essentially we divide all values by 2 except for the first column which is the index of dim=0 in a which was unchanged. There's some fiddling around with the types (in short: it has to be a float in order to divide by 2 but needs to be cast back to a long in order for the indexing to work):
>>> indx_ = indx.clone().float()
>>> indx_[:, 1:] /= 2
>>> indx_ = indx_.long()
tensor([[0, 1, 0],
        [0, 1, 2],
        [0, 2, 0]])

2. Updating with blocks
We will simply index and accumulate with:
>>> a_chunks[indx_[:, 0], indx_[:, 1], indx_[:, 2]] += blocks

3. Putting it back together
I thought that was it, but actually converting a_chunk back to a 6x6 tensor is way trickier than it seems. Apparently torch.cat can only receive a tuple. I won't go into to much detail: tuple() will only consider the first axis, as a workaround you can use torch.permute to switch the axes. This combined with two torch.cat will do:
>>> a_row_cat = torch.cat(tuple(a_chunks.permute(1, 0, 2, 3, 4)), dim=2)
>>> a_row_cat.shape
torch.Size([1, 3, 6, 2])

>>> A = torch.cat(tuple(a_row_cat.permute(1, 0, 2, 3)), dim=2)
>>> A.shape
torch.Size([1, 6, 6])

>>> A
tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],
         [1.5818, 2.3108, 0.0000, 0.0000, 2.0472, 1.6651],
         [2.6742, 3.0024, 0.0000, 0.0000, 3.2807, 2.7413],
         [1.5587, 2.1905, 0.0000, 0.0000, 0.0000, 0.0000],
         [1.9231, 3.5083, 0.0000, 0.0000, 0.0000, 0.0000]]])

Et voilà.

If you didn't quite get how the chunks worked. Run this:
for x in range(0, 6, 2):
    for y in range(0, 6, 2):
        a *= 0
        a[:, x:x+2, y:y+2] = 1
        print(a)

And see for yourself: each 2x2 block of 1s corresponds to a chunk in a_chunks.
So you can do the same with:
for x in range(3):
    for y in range(3):
        a_chunks *= 0
        a_chunks[:, x, y] = 1
        print(a_chunks)


Instruction: Use of PyTorch permute in RCNN

Input:
I am looking at an implementation of RCNN for text classification using PyTorch. Full Code. There are two points where the dimensions of tensors are permuted using the permute function. The first is after the LSTM layer and before tanh. The second is after a linear layer and before a max pooling layer.
Could you please explain why the permutation is necessary or useful?
Relevant Code
    def forward(self, x):
        # x.shape = (seq_len, batch_size)
        embedded_sent = self.embeddings(x)
        # embedded_sent.shape = (seq_len, batch_size, embed_size)

        lstm_out, (h_n,c_n) = self.lstm(embedded_sent)
        # lstm_out.shape = (seq_len, batch_size, 2 * hidden_size)

        input_features = torch.cat([lstm_out,embedded_sent], 2).permute(1,0,2)
        # final_features.shape = (batch_size, seq_len, embed_size + 2*hidden_size)

        linear_output = self.tanh(
            self.W(input_features)
        )
        # linear_output.shape = (batch_size, seq_len, hidden_size_linear)

        linear_output = linear_output.permute(0,2,1) # Reshaping fot max_pool

        max_out_features = F.max_pool1d(linear_output, linear_output.shape[2]).squeeze(2)
        # max_out_features.shape = (batch_size, hidden_size_linear)

        max_out_features = self.dropout(max_out_features)
        final_out = self.fc(max_out_features)
        return self.softmax(final_out)

Similar Code in other Repositories
Similar implementations of RCNN use permute or transpose. Here are examples:

https://github.com/prakashpandey9/Text-Classification-Pytorch/blob/master/models/RCNN.py
https://github.com/jungwhank/rcnn-text-classification-pytorch/blob/master/model.py

Output:
What permute function does is rearranges the original tensor according to the desired ordering, note permute is different from reshape function, because when apply permute, the elements in tensor follow the index you provide where in reshape it's not.
Example code:
import torch
var = torch.randn(2, 4)
pe_var = var.permute(1, 0)
re_var = torch.reshape(var, (4, 2))
print("Original size:\n{}\nOriginal var:\n{}\n".format(var.size(), var) +
      "Permute size:\n{}\nPermute var:\n{}\n".format(pe_var.size(), pe_var) +
      "Reshape size:\n{}\nReshape var:\n{}\n".format(re_var.size(), re_var))

Outputs:
Original size:
torch.Size([2, 4])
Original var:
tensor([[ 0.8250, -0.1984,  0.5567, -0.7123],
        [-1.0503,  0.0470, -1.9473,  0.9925]])
Permute size:
torch.Size([4, 2])
Permute var:
tensor([[ 0.8250, -1.0503],
        [-0.1984,  0.0470],
        [ 0.5567, -1.9473],
        [-0.7123,  0.9925]])
Reshape size:
torch.Size([4, 2])
Reshape var:
tensor([[ 0.8250, -0.1984],
        [ 0.5567, -0.7123],
        [-1.0503,  0.0470],
        [-1.9473,  0.9925]])

With the role of permute in mind we could see what first permute does is reordering the concatenate tensor for it to fit the inputs format of self.W, i.e with batch as first dimension; and the second permute does similar thing because we want to max pool the linear_output along the sequence and F.max_pool1d will pool along the last dimension.


Instruction: logistic regression model with L1 regularisations

Input:
I am trying to apply L1 regularization on a logistic model
class LogisticRegression(nn.Module):

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size, num_classes)
    def forward(self, x):
        x = x.reshape(-1, 784)
        output = self.linear(x)
        return output

    def training_step(self, batch):
        images, labels = batch
        output = self(images)
        loss = F.cross_entropy(output, labels)

        acc = accuracy(output, labels)
        return {'Training_loss': loss, 'Training_acc': acc}

    def training_epoch_end(self, outputs):
        batch_losses = [x['Training_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()
        batch_accs = [x['Training_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()
        return {'Training_loss': epoch_loss.item(), 'Training_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], Training_loss: {:.4f}, Training_acc: {:.4f}".format(epoch, result['Training_loss'], result['Training_acc']))
model = LogisticRegression()

But I think I am doing it wrong the accuracy did not change.
L1=0.2
def evaluate(model_b, trainloader):
    outputs = [model_b.training_step(batch) for batch in trainloader]
    return model_b.training_epoch_end(outputs)

def fit(epochs, lr, model_b, trainloader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model_b.parameters(), lr)
    for epoch in range(epochs):
        ##### Training Phase
        for batch in trainloader:
            loss = model_b.training_step(batch)['Training_loss']

            loss_Lasso = loss  + 0.5 * L1 #  L1 reg

            loss_Lasso.backward()
            optimizer.step()
            optimizer.zero_grad()
        result = evaluate_b(model_b, trainloader)
        model_b.epoch_end(epoch, result)
        history.append(result)
    return history

Can anyone help me with what I am missing and how I can really apply  L1 regularization?
Also, is  L1 regularization called lasso?

Output:
I believe the l1-norm is a type of Lasso regularization, yes, but there are others.
In your snippet L1 is set as a constant, instead you should measure the l1-norm of your model's parameters. Then sum it with your network's loss, as you did. In your example there is a single layer, so you will only need self.linear's parameters. First gather all parameters then measure the total norm with torch.norm. You could also use nn.L1Loss.
params = torch.cat([x.view(-1) for x in model.linear.parameters()])
L1 = lamb*torch.norm(params, p=1)

Where lamb is your lambda regularization parameter and model is initialized from the LogisticRegression class.


Instruction: Conversion pytorch to coreml for element-wise maximum operation

Input:
I tried to convert a PyTorch model to coreml with the element-wise maximum operation based on coremltools.
With torch.max operation, I got
ValueError: node input.2 (max) got 2 input(s), expected [3]

With torch.maximum operation
RuntimeError: PyTorch convert function for op 'maximum' not implemented.

Any idea to solve this issue?

Output:
I encountered the same issue in conversion with pytorch element-wise operation to coreml model, but solved it by adding support for torch.maximum and torch.minimum for the MIL converter with the torch frontend.
@register_torch_op
def maximum(context, node):
    inputs = _get_inputs(context, node)
    x = inputs[0]
    y = inputs[1]
    out = mb.maximum(x=x, y=y, name=node.name)
    context.add(out)


Instruction: torch.nn.CrossEntropyLoss().ignore_index is crashing when importing transfomers library

Input:
I am using layoutlm github which require python 3.6, transformer 2.9.0. I created an conda env:
name: env_test

    channels:
    - defaults
    - conda-forge
    dependencies:
    - python=3.6
    - pip=20.3.3
    - pytorch=1.4.0
    - cudatoolkit=10.1
    - pip:
      - transformers==2.9.0

I have the following test.py code to reproduce the issue:
import sys

import torch
from torch.nn import CrossEntropyLoss

from transformers import (
    BertConfig,
    __version__
)

print (sys.version)
print(torch.__version__)
print(__version__)
CrossEntropyLoss().ignore_index

print("success!")

Importing transformers library results in segmentation fault (core dumped) a when calling CrossEntropyLoss().ignore_index:
$python test.py
3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56)
[GCC 7.3.0]
1.4.0
2.9.0
Segmentation fault (core dumped)

I tried to investigate a bit but I don't really see from where the problem is coming from:
gdb python
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.
(gdb) r test.py
Starting program: /home/jupyter/.conda-env/env_test/bin/python test.py
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 23:10:56)
[GCC 7.3.0]
1.4.0
2.9.0

Program received signal SIGSEGV, Segmentation fault.
0x00007f97000055fb in ?? ()
(gdb) where
#0  0x00007f97000055fb in ?? ()
#1  0x00007f97f4755729 in void pybind11::cpp_function::initialize<void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), void, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
   from /home/jupyter/.conda-env/env_test/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#2  0x00007f97f436bca6 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/jupyter/.conda-env/env_test/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#3  0x000055fbadd73a14 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/methodobject.c:231
#4  0x000055fbaddfba5c in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4851
#5  0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#6  0x000055fbaddf5c1b in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4933
#7  fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4968
#8  0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#9  0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#10 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#11 0x000055fbaddf5e51 in fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4992
#12 0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#13 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#14 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#15 0x000055fbaddf5e51 in fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4992
#16 0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#17 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#18 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#19 0x000055fbaddf632c in _PyFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:5084
#20 0x000055fbadd73ddf in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2310
#21 0x000055fbadd78873 in _PyObject_Call_Prepend () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2373
#22 0x000055fbadd7381e in PyObject_Call () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2261
#23 0x000055fbaddcc88b in slot_tp_init () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:6420
#24 0x000055fbaddfbd97 in type_call () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:915
#25 0x000055fbadd73bfb in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2331
#26 0x000055fbaddfbbae in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4875
#27 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#28 0x000055fbaddf6969 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=0x0, argcount=0, args=0x0,
    locals=0x7f98035bf1f8, globals=0x7f98035bf1f8, _co=0x7f980357aae0) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#29 PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4187
#30 0x000055fbaddf770c in PyEval_EvalCode (co=co@entry=0x7f980357aae0, globals=globals@entry=0x7f98035bf1f8, locals=locals@entry=0x7f98035bf1f8) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:731
#31 0x000055fbade77574 in run_mod () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:1025
#32 0x000055fbade77971 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:978
#33 0x000055fbade77b73 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:419
#34 0x000055fbade77c7d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:81
#35 0x000055fbade7b663 in run_file (p_cf=0x7fff210dc16c, filename=0x55fbaefa6dc0 L"test.py", fp=0x55fbaefda800) at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:340
#36 Py_Main () at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:811
#37 0x000055fbadd4543e in main () at /tmp/build/80754af9/python_1599604603603/work/Programs/python.c:69
#38 0x00007f9803fd6bf7 in __libc_start_main (main=0x55fbadd45350 <main>, argc=2, argv=0x7fff210dc378, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff210dc368) at ../csu/libc-start.c:310
#39 0x000055fbade24d0b in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb

I am the following list of packages:
_libgcc_mutex             0.1                        main    defaults
_pytorch_select           0.2                       gpu_0    defaults
blas                      1.0                         mkl    defaults
ca-certificates           2020.12.8            h06a4308_0    defaults
certifi                   2020.12.5        py36h06a4308_0    defaults
cffi                      1.14.4           py36h261ae71_0    defaults
chardet                   4.0.0                    pypi_0    pypi
click                     7.1.2                    pypi_0    pypi
cudatoolkit               10.1.243             h6bb024c_0    defaults
cudnn                     7.6.5                cuda10.1_0    defaults
dataclasses               0.8                      pypi_0    pypi
filelock                  3.0.12                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
intel-openmp              2020.2                      254    defaults
joblib                    1.0.0                    pypi_0    pypi
ld_impl_linux-64          2.33.1               h53a641e_7    defaults
libedit                   3.1.20191231         h14c3975_1    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 9.1.0                hdf63c60_0    defaults
libstdcxx-ng              9.1.0                hdf63c60_0    defaults
mkl                       2020.2                      256    defaults
mkl-service               2.3.0            py36he8ac12f_0    defaults
mkl_fft                   1.2.0            py36h23d657b_0    defaults
mkl_random                1.1.1            py36h0573a6f_0    defaults
ncurses                   6.2                  he6710b0_1    defaults
ninja                     1.10.2           py36hff7bd54_0    defaults
numpy                     1.19.2           py36h54aff64_0    defaults
numpy-base                1.19.2           py36hfa32c7d_0    defaults
openssl                   1.1.1i               h27cfd23_0    defaults
pip                       20.3.3           py36h06a4308_0    defaults
pycparser                 2.20                       py_2    defaults
python                    3.6.12               hcff3b4d_2    defaults
pytorch                   1.4.0           cuda101py36h02f0884_0    defaults
readline                  8.0                  h7b6447c_0    defaults
regex                     2020.11.13               pypi_0    pypi
requests                  2.25.1                   pypi_0    pypi
sacremoses                0.0.43                   pypi_0    pypi
sentencepiece             0.1.94                   pypi_0    pypi
setuptools                51.0.0           py36h06a4308_2    defaults
six                       1.15.0           py36h06a4308_0    defaults
sqlite                    3.33.0               h62c20be_0    defaults
tk                        8.6.10               hbc83047_0    defaults
tokenizers                0.7.0                    pypi_0    pypi
tqdm                      4.55.1                   pypi_0    pypi
transformers              2.9.0                    pypi_0    pypi
urllib3                   1.26.2                   pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0    defaults
xz                        5.2.5                h7b6447c_0    defaults
zlib                      1.2.11               h7b6447c_3    defaults

What is responsible fo this core dump (I have a VM with 30 GB of memory) ? Seems to be related to transformers. Some dependency issues not catched by conda ? This piece of code seems to work with the latest version of transformers 4.1.1 but this is not compatible with layoutlm. Any suggestions?

Output: It seems something was broken on layoutlm with pytorch 1.4 related issue. Switching to pytorch 1.6 fix the issue with the core dump, and the layoutlm code run without any modification.


Instruction: Add a index selected tensor to another tensor with overlapping indices in pytorch

Input:
This is a follow up question to this question. I want to do the exactly same thing in pytorch. Is it possible to do this? If yes, how?
import torch
image = torch.tensor([[246,  50, 101], [116,   1, 113], [187, 110,  64]])
iy = torch.tensor([[1, 0, 2], [1, 0, 2], [2, 2, 2]])
ix = torch.tensor([[0, 2, 1], [1, 2, 0], [0, 1, 2]])
warped_image = torch.zeros(size=image.shape)

I need something like torch.add.at(warped_image, (iy, ix), image) that gives the output as
[[  0.   0.  51.]
 [246. 116.   0.]
 [300. 211.  64.]]

Note that the indices at (0,1) and (1,1) point to the same location (0,2). So, I want warped_image[0,2] = image[0,1] + image[1,1] = 51.

Output:
What you are looking for is torch.Tensor.index_put_ with the accumulate argument set to True:
>>> warped_image = torch.zeros_like(image)

>>> warped_image.index_put_((iy, ix), image, accumulate=True)
tensor([[  0,   0,  51],
        [246, 116,   0],
        [300, 211,  64]])

Or, using the out-place version torch.index_put:
>>> torch.index_put(torch.zeros_like(image), (iy, ix), image, accumulate=True)
tensor([[  0,   0,  51],
        [246, 116,   0],
        [300, 211,  64]])


Instruction: How to convert PyTorch graph to ONNX and then inference from OpenCV?

Input:
I'm attempting to convert a PyTorch graph to ONNX via the torch.onnx.export function and then use the OpenCV functions blobFromImage, setInput, and forward to inference the converted graph.  I think I'm on the right track but I keep running into errors and there are very few helpful examples of how to do this that I could find.
I realize the general stack overflow policy is to post only relevant portions of code, however with the errors I'm getting it seems this is a case where the devil is in the details so I suspect I'll have to post a full example to make the cause of the errors clear.
Here is my training net (pretty standard for MNIST):
# MnistNet.py

# Net Layout:
# batchSize x 1 x 28 x 28
#     conv1 Conv2d(1, 6, 5)
# batchSize x 6 x 24 x 24
#     relu(x)
#     max_pool2d(x, kernel_size=2)
# batchSize x 6 x 12 x 12
#     conv2 Conv2d(6, 16, 5)
# batchSize x 16 x 8 x 8
#     relu(x)
#     max_pool2d(x, kernel_size=2)
# batchSize x 16 x 4 x 4
#     view(-1, 16 * 4 * 4)    Note: 16 * 4 * 4 = 256
# batchSize x 1 x 256
#     fc1 Linear(256, 120)
#     relu(x)
# batchSize x 1 x 120
#     fc2 Linear(120, 84)
#     relu(x)
# batchSize x 1 x 84
#     fc3 Linear(84, 10)
# batchSize x 1 x 10

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision

class MnistNet(nn.Module):

    TRANSFORM = torchvision.transforms.Compose([
        torchvision.transforms.Resize((28, 28)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize([0.5], [0.5])
    ])

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(256, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    # end function

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=2)
        x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=2)
        x = x.view(-1, 256)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    # end function

# end class

Here is my training script (again, pretty standard for MNIST):
# 1_train.py

from MnistNet import MnistNet

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import torchvision

from termcolor import colored

BATCH_SIZE = 64
NUM_EPOCHS = 10

GRAPH_NAME = 'MNIST.pt'

def main():
    trainDataset = torchvision.datasets.MNIST('built_in_mnist_download', train=True, transform=MnistNet.TRANSFORM, download=True)

    trainDataLoader = DataLoader(trainDataset, batch_size=BATCH_SIZE, shuffle=True)

    # declare net, loss function, and optimizer
    mnistNet = MnistNet()
    lossFunction = nn.CrossEntropyLoss()
    optimizer = optim.Adam(mnistNet.parameters())

    # get device (cuda or cpu)
    if torch.cuda.is_available():
        device = torch.device('cuda')
        print(colored('using cuda', 'green'))
    else:
        device = torch.device('cpu')
        print(colored('GPU does not seem to be available, using CPU', 'red'))
    # end if

    # set network to device
    mnistNet.to(device)

    # set network to train mode
    mnistNet.train()

    print('beginning training . . .')

    # for each epoch . . .
    for epoch in range(1, NUM_EPOCHS+1):

        # variables to calculate loss and accuracy within the epoch
        epochLosses = []
        epochAccuracies = []

        # for each batch . . .
        for i, element in enumerate(trainDataLoader):
            # break out the input images and labels, note these are Tensors
            inputImages, labels = element

            inputImages = inputImages.to(device)
            labels = labels.to(device)

            # clear gradients from the previous step
            optimizer.zero_grad()

            # get net output
            outputs = mnistNet(inputImages)
            # calculate loss
            loss = lossFunction(outputs, labels)
            # call backward() to compute gradients
            loss.backward()
            # update parameters using gradients
            optimizer.step()

            # append the current classification loss to the list of epoch losses
            epochLosses.append(loss.item())

            # calculate current classification accuracy

            # get the highest scoring classification for each prediction
            _, predictions = torch.max(outputs.data, 1)

            # number of labels and predictions should always be the same, log an error if this is not the case
            if labels.size(0) != predictions.size(0):
                print(colored('ERROR: labels.size(0) != predictions.size(0)', 'red'))
            # end if

            # determine the number of correct predictions for the current batch
            correctPredictions = 0
            for j in range(len(labels)):
                if predictions[j].item() == labels[j].item():
                    correctPredictions += 1
                # end if
            # end for

            # append the current batch accuracy to the list of accuracies
            epochAccuracies.append(correctPredictions / labels.size(0))
        # end for

        # calculate epoch loss and accuracy from the respective lists
        epochLoss = sum(epochLosses) / len(epochLosses)
        epochAccuracy = sum(epochAccuracies) / len(epochAccuracies)

        print('epoch ' + str(epoch) + ', epochLoss = ' + '{:.4f}'.format(epochLoss) +
              ', epochAccuracy = ' + '{:.4f}'.format(epochAccuracy * 100) + '%')
    # end for

    print('finished training')

    # save the model
    torch.save(mnistNet.state_dict(), GRAPH_NAME)

    print('saved graph as ' + str(GRAPH_NAME))

# end function

if __name__ == '__main__':
    main()

Here is my best attempt so far at a script to convert a saved graph from PyTorch to ONNX (I'm not sure if this is correct, I can at least say it runs without error):
# 3_convert_graph_to_onnx.py

from MnistNet import MnistNet

import torch

GRAPH_NAME = 'MNIST.pt'
ONNX_GRAPH_NAME = 'MNIST.onnx'

def main():

    net = MnistNet()
    net.load_state_dict(torch.load(GRAPH_NAME))

    net.eval()

    # make a dummy input with a batch size of 1, 1 channel, 28 x 28
    dummyInput = torch.randn(10, 1, 28, 28)

    torch.onnx.export(net, dummyInput, ONNX_GRAPH_NAME, verbose=True)

# end function

if __name__ == '__main__':
    main()

Here is my attempt to inference the ONNX graph with OpenCV (Note that PyTorch is included, but is only used to load the test MNIST dataset, and the images are converted to OpenCV format before inferencing):
# 4_onnx_opencv_inf.py

from MnistNet import MnistNet

import torchvision

import cv2
import numpy as np
from termcolor import colored

ONNX_GRAPH_NAME = 'MNIST.onnx'

def main():
    testDataset = torchvision.datasets.MNIST('built_in_mnist_download', train=False, transform=MnistNet.TRANSFORM, download=True)

    labels = [ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' ]

    net = cv2.dnn.readNetFromONNX(ONNX_GRAPH_NAME)

    # test on 3 images
    for i in range(3):
        # get PyTorch tensor image and ground truth index from dataset
        ptImage, gndTrIdx = testDataset[i]
        # convert to PIL image
        pilImage = torchvision.transforms.ToPILImage()(ptImage)
        # convert to OpenCV image, would convert RGB to BGR here if image was color
        openCvImage = np.array(pilImage)

        gndTr = labels[gndTrIdx]

        # can show OpenCV image here if desired
        # cv2.imshow('openCvImage', openCvImage)
        # cv2.waitKey()

        blob = cv2.dnn.blobFromImage(image=openCvImage, scalefactor=1.0/255.0, size=(64, 64))

        net.setInput(blob)
        preds = net.forward()

        predIdx = np.array(preds)[0].argmax()

        prediction = str(predIdx)
        if prediction == gndTr:
            print(colored('i = ' + str(i) + ', predIdx = ' + str(predIdx) + ', gndTrIdx = ' + str(gndTrIdx) + ', correct answer', 'green'))
        else:
            print(colored('i = ' + str(i) + ', predIdx = ' + str(predIdx) + ', gndTrIdx = ' + str(gndTrIdx) + ', incorrect answer', 'red'))
        # end if

    # end for

# end function

if __name__ == '__main__':
    main()

Currently this final script crashes with this error:
$ python3 4_onnx_opencv_inf.py
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3441) getLayerShapesRecursively OPENCV/DNN: [Reshape]:(18): getMemoryShapes() throws exception. inputs=1 outputs=1/1 blobs=0
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3447) getLayerShapesRecursively     input[0] = [ 1 16 13 13 ]
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3451) getLayerShapesRecursively     output[0] = [ 1 256 ]
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3457) getLayerShapesRecursively Exception message: OpenCV(4.4.0) /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/layers/reshape_layer.cpp:154: error: (-1:Backtrace) Can't infer a dim denoted by -1 in function 'computeShapeByReshapeMask'

Traceback (most recent call last):
  File "4_onnx_opencv_inf.py", line 54, in <module>
    main()
  File "4_onnx_opencv_inf.py", line 38, in main
    preds = net.forward()
cv2.error: OpenCV(4.4.0) /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/layers/reshape_layer.cpp:154: error: (-1:Backtrace) Can't infer a dim denoted by -1 in function 'computeShapeByReshapeMask'

I'm not really sure what to do next based on this error, can anybody please advise on this?  I suspect that I'm at least doing the procedure generally correctly and missing a few small details.

Output:
I was using the wrong size in the ONNX inference script.
In 4_onnx_opencv_inf.py changing:
blob = cv2.dnn.blobFromImage(image=openCvImage, scalefactor=1.0/255.0, size=(64, 64))

to
blob = cv2.dnn.blobFromImage(image=openCvImage, scalefactor=1.0/255.0, size=(28, 28))

makes it run (I'm using Ubuntu 20.04 and PyTorch 1.7.0), however the accuracy is worse.  With regular PyTorch inferencing as above (2nd script) I'm getting 98.5% accuracy, with the OpenCV ONNX version I'm getting 95% accuracy.
I suspect the difference is due to the parameters in cv2.dnn.blobFromImage not being set to handle the normalization correctly, but that is a different post entirely.


Instruction: Pytorch, how to extend a tensor

Input:
I want to extend a tensor in PyTorch in the following way:
Let C be a 3x4 tensor which requires_grad = True. I want to have a new C which is 3x5 tensor and C = [C, ones(3,1)]
(the last column is a one-vector, and others are the old C)
Moreover, I need requires_grad = True for new C.
What is an efficient way to do this?

Output:
You could do something like this -
c = torch.rand((3,4), requires_grad=True)  # a random matrix c of shape 3x4
ones = torch.ones(c.shape[0], 1)  # creating a vector of 1's using shape of c

# merging vector of ones with 'c' along dimension 1 (columns)
c = torch.cat((c, ones), 1)

# c.requires_grad will still return True...