Instruction: How does the nn.Embedding module relate intuitively to the idea of an embedding in general?

Input:
So, I'm having a hard time understanding nn.Embedding. Specifically, I can't connect the dots between what I understand about embeddings as a concept and what this specific implementation is doing.
My understanding of an embedding is that it is a smaller dimension representation of some larger dimension data point. So it maps data in N-d to a M-d latent/embedding space such that M < N.
As I understand it, this mapping is achieved through the learning process, as in an auto-encoder. The encoder learns the optimal embedding so that the decoder can reconstruct the original input.
So my question is, how does this relate to nn.Embedding module:

A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

does this layer "learn" a lower dimensional representation of a larger input space? Or is it something else entirely?
What I'm looking for is to take the very abstract language of the documentation to something real:
Let's say I have some input x. This input might be a vectorized image or maybe some sequence daily temperature data. In any case, this input x has 100 elements (100 days of temperature, or a 10x10 image).
How can you explain the use of nn.Embedding() in this case?
What does each argument mean in a real world context?

Output:
As you said, the aim when using an embedding is to reduce the dimension of your data. However, it does not learn a lower dimensional representation of a larger input space on its own. Starting from a random initialization you can improve this embedding through a learning process. This requires finding a suitable task to train the embedding on, I think, for another question. I believe it's called a "pretext task", where ultimately the objective is to have an accurate embedding matrix.
You can check the parameters of any nn.Module with .parameters(). It will return an generator.
<< [x for x in nn.Embedding(10, 2).parameters()][0].shape
>> torch.Size([10, 2])

Here, there are 10*2 parameters (i.e. dimension_input*dimension_output or by PyTorch's naming num_embeddings*embedding_dims). However it is, still, a lookup table: given an index it will return an embedding of size embedding_dims. But you these embeddings (the values of this matrix) can be changed.
Here's a little experiment:
E = nn.Embedding(10, 2)
optim = optim.SGD(E.parameters(), lr=0.01)
X = torch.randint(0, 10, size=(100,))

loss_before = E(X).mean()
loss_before.backward()
optim.step()

loss_after = E(X).mean()

As expected, loss_before and loss_after are different which shows nn.Embedding's parameters are learnable.

Edit: your question comes down to, "how do I encode my data?".
For those examples you gave precisely:

Let's say I have some input x. This input might be a vectorized image or maybe some sequence daily temperature data. In any case, this input x has 100 elements (100 days of temperature, or a 10x10 image).

You can't use a nn.Embedding to solve these cases. Embedding layers are different to a reduction matrix. The latter can be used to reduce every single vector of dimension d into dimension n where n<<d. The prerequisite to using an embedding layer is having a finite dictionnary of possible elements. For example, you might want to represent a word with a vector of size n then you would use a embedding of nb_possible_words x n. This way, for any given word in the dictionnary the layer will produce the corresponding n-size vector.
As I said in the comments below, num_embeddings is the number of unique elements you are working with and embedding_dim is the size of the embedding, i.e. the size of the output vector.
nn.Embedding is usually used at the head of a network to cast encoded data into a lower dimensionality space. It won't solve your problem by magically reducing your dimensions.
If you have a sequence of temperatures you want to analyse. You could encode each temperature into a one-hot-encoding. But this vector representation might be very large (depending on the number of different temperatures). Using an embedding layer would allow to reduce the size of these vectors. This is important when the aim is to analyse the data with a RNN any other MLP for that matter. Since the bigger your input size, the more paramaters you will have!


Instruction: Visualize Autoencoder Output

Input:
I come with a pretty noob question but I'm stuck...
I have created a Autoencoder with Pytorch and I trained it with the typical MNIST dataset and so on:
class Autoencoder(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=kwargs["embedding_dim"]
        )
        self.encoder_output_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["embedding_dim"]
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["embedding_dim"]
        )
        self.decoder_output_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)

        code = self.encoder_output_layer(activation)
        code = torch.relu(code)

        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)

        activation = self.decoder_output_layer(activation)
        reconstructed = torch.relu(activation)

        return reconstructed

model = Autoencoder(input_shape=784, embedding_dim=128)

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

What I want now is to visualize the reconstructed images, but I don't know how to do it. I know it's quite simple but I cannot find a way. I know that the shape of the output is [128,784] because the batch_size is 128 and 784 is 28x28(x1channel).
Could anyone please tell me how could I get an image from my reconstructed tensor?
Thank you so much!

Output:
First you will have to broadcast the tensor into 128x28x28:
reconstructed = x.reshape(128, 1, 28, 28)

Then, you can convert one of the batch elements into a PIL image using torchvision's functions. The following will show the first image:
import torchvision.transforms as T
img = T.ToPILImage()(reconstructed[0])
img.show()


Instruction: how to load one type of image in cifar10 or stl10 with pytorch

Input:
This is a very simple question, I'm just trying to select a specific class of images (eg "car") from a standard pytorch image dataset. At the moment the data loader looks like this:
def cycle(iterable):
    while True:
        for x in iterable:
            yield x

train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.STL10('drive/My Drive/training/stl10', split='train+unlabeled', transform=torchvision.transforms.Compose([
        torchvision.transforms.ToTensor(),
    ])),
shuffle=True, batch_size=8)
train_iterator = iter(cycle(train_loader))
class_names = ['airplane', 'bird', 'car', 'cat', 'deer', 'dog', 'horse', 'monkey', 'ship', 'truck']

train_iterator = iter(cycle(train_loader))

The iterator returns a batch of shuffled images of all types, but I would like to be able to select what types of images are returned, eg. just images of deer, or ships

Output:
Done it!
def cycle(iterable):
    while True:
        for x in iterable:
            yield x

# Return only images of certain class (eg. aeroplanes = class 0)
def get_same_index(target, label):
    label_indices = []
    for i in range(len(target)):
        if target[i] == label:
            label_indices.append(i)
    return label_indices

# STL10 dataset
train_dataset = torchvision.datasets.STL10('drive/My Drive/training/stl10', split='train+unlabeled', download=True, transform=torchvision.transforms.Compose([
        torchvision.transforms.ToTensor()]))

label_class = 1# birds

# Get indices of label_class
train_indices = get_same_index(train_dataset.labels, label_class)

bird_set = torch.utils.data.Subset(train_dataset, train_indices)

train_loader = torch.utils.data.DataLoader(dataset=bird_set, shuffle=True,
                                           batch_size=batch_size, drop_last=True)
train_iterator = iter(cycle(train_loader))


Instruction: Building wheel for neural-renderer-pytorch (setup.py) ...installing multiperson and neural mesh renderer doesn't work for pytorch 1.6

Input:
I am trying to install a github repo named multiperson for PyTorch 1.6 and I get the following error. How can I make it work for PyTorch 1.6?
(base) mona@mona:~/research$ cd phosa/
(base) mona@mona:~/research/phosa$ mkdir -p external
(base) mona@mona:~/research/phosa$ git clone https://github.com/JiangWenPL/multiperson.git external/multiperson
Cloning into 'external/multiperson'...
remote: Enumerating objects: 752, done.
remote: Counting objects: 100% (752/752), done.
remote: Compressing objects: 100% (566/566), done.
remote: Total 752 (delta 189), reused 723 (delta 173), pack-reused 0
Receiving objects: 100% (752/752), 48.29 MiB | 46.26 MiB/s, done.
Resolving deltas: 100% (189/189), done.
(base) mona@mona:~/research/phosa$ pip install external/multiperson/neural_renderer
Processing ./external/multiperson/neural_renderer
Building wheels for collected packages: neural-renderer-pytorch
  Building wheel for neural-renderer-pytorch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/mona/anaconda3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-nl9m5bw0
       cwd: /tmp/pip-req-build-ma51z6r7/
  Complete output (210 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/load_obj.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/perspective.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/vertices_to_faces.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/visibility.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/get_points_from_angles.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/look.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/projection.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/rasterize.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/save_obj.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/look_at.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/lighting.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/mesh.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/renderer.py -> build/lib.linux-x86_64-3.7/neural_renderer
  creating build/lib.linux-x86_64-3.7/neural_renderer/cuda
  copying neural_renderer/cuda/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer/cuda
  running build_ext
  building 'neural_renderer.cuda.load_textures' extension
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda
  Emitting ninja build file /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/2] /usr/bin/nvcc -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-req-

    copying neural_renderer/look_at.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/lighting.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/mesh.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/renderer.py -> build/lib.linux-x86_64-3.7/neural_renderer
    creating build/lib.linux-x86_64-3.7/neural_renderer/cuda
    copying neural_renderer/cuda/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer/cuda
    running build_ext
    building 'neural_renderer.cuda.load_textures' extension
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda
    Emitting ninja build file /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    [1/2] c++ -MMD -MF /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o.d -pthread -B /home/mona/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    FAILED: /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o
    c++ -MMD -MF /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o.d -pthread -B /home/mona/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:149,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
       84 | #pragma omp parallel for if ((end - begin) >= grain_size)
          |
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp: In function ‘at::Tensor load_textures(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int)’:
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
       28 |     CHECK_INPUT(image);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:23: error: ‘AT_CHECK’ was not declared in this scope; did you mean ‘DCHECK’?
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                       ^~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
       28 |     CHECK_INPUT(image);
          |     ^~~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:29:5: note: in expansion of macro ‘CHECK_INPUT’
       29 |     CHECK_INPUT(faces);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:30:5: note: in expansion of macro ‘CHECK_INPUT’
       30 |     CHECK_INPUT(is_update);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:31:5: note: in expansion of macro ‘CHECK_INPUT’
       31 |     CHECK_INPUT(textures);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    [2/2] /usr/bin/nvcc -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build
        env=env)
      File "/home/mona/anaconda3/lib/python3.7/subprocess.py", line 512, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-ma51z6r7/setup.py", line 40, in <module>
        cmdclass = {'build_ext': BuildExtension}
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/__init__.py", line 163, in setup
        return distutils.core.setup(**attrs)
      File "/home/mona/anaconda3/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/home/mona/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/mona/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
        build_ext.build_extensions(self)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
        _build_ext.build_extension(self, ext)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
        depends=ext.depends)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 478, in unix_wrap_ninja_compile
        with_cuda=with_cuda)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects
        error_prefix='Error compiling objects for extension')
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
        raise RuntimeError(message)
    RuntimeError: Error compiling objects for extension
    ----------------------------------------
  Rolling back uninstall of neural-renderer-pytorch
  Moving to /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/
   from /home/mona/anaconda3/lib/python3.7/site-packages/~eural_renderer
  Moving to /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer_pytorch-1.1.3.dist-info/
   from /home/mona/anaconda3/lib/python3.7/site-packages/~eural_renderer_pytorch-1.1.3.dist-info
ERROR: Command errored out with exit status 1: /home/mona/anaconda3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ul8nk1jn/install-record.txt --single-version-externally-managed --compile --install-headers /home/mona/anaconda3/include/python3.7m/neural-renderer-pytorch Check the logs for full command output.

I have:
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

$ nvidia-smi
Sun Dec  6 16:36:36 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8    21W /  N/A |   1546MiB /  7982MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Output:
You would need to change all AT_CHECK in neural mesh renderer to TORCH_CHECK
(base) mona@mona:~/research/phosa/external/multiperson/neural_renderer$ rg AT_CHECK
neural_renderer/cuda/load_textures_cuda.cpp
15:#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
16:#define CHECK_CONTIGUOUS(x) AT_CHECK(x.is_contiguous(), #x " must be contiguous")

neural_renderer/cuda/create_texture_image_cuda.cpp
13:#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
14:#define CHECK_CONTIGUOUS(x) AT_CHECK(x.is_contiguous(), #x " must be contiguous")

neural_renderer/cuda/rasterize_cuda.cpp
69:#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
70:#define CHECK_CONTIGUOUS(x) AT_CHECK(x.is_contiguous(), #x " must be contiguous")

(base) mona@mona:~/research/phosa$ pip install external/multiperson/neural_renderer
Processing ./external/multiperson/neural_renderer
Building wheels for collected packages: neural-renderer-pytorch
  Building wheel for neural-renderer-pytorch (setup.py) ... done
  Created wheel for neural-renderer-pytorch: filename=neural_renderer_pytorch-1.1.3-cp37-cp37m-linux_x86_64.whl size=6321659 sha256=5e2f4afc2346a90c5cd804b226dd7c424ab95f477aad67f2ca3f15530484fbc6
  Stored in directory: /tmp/pip-ephem-wheel-cache-_rf6c5ld/wheels/c7/1b/84/10bf7a286a267887d8c7d382677c292cf18e1bba4e2508ed33
Successfully built neural-renderer-pytorch
Installing collected packages: neural-renderer-pytorch
  Attempting uninstall: neural-renderer-pytorch
    Found existing installation: neural-renderer-pytorch 1.1.3
    Uninstalling neural-renderer-pytorch-1.1.3:
      Successfully uninstalled neural-renderer-pytorch-1.1.3
Successfully installed neural-renderer-pytorch-1.1.3


Instruction: truncated bptt pytorch implementation question

Input:
i'm trying to implement tbptt in pytorch.
I've found an implementation below in a forum and I get the logic behind the code but I keep getting a "inplace operation" error.
class TBPTT():
    def __init__(self, one_step_module, loss_module, k1, k2, optimizer):
        self.one_step_module = one_step_module
        self.loss_module = loss_module
        self.k1 = k1
        self.k2 = k2
        self.retain_graph = k1 < k2
        # You can also remove all the optimizer code here, and the
        # train function will just accumulate all the gradients in
        # one_step_module parameters
        self.optimizer = optimizer

    def train(self, input_sequence, init_state):
        states = [(None, init_state)]
        for j, (inp, target) in enumerate(input_sequence):

            state = states[-1][1].detach()
            state.requires_grad=True
            output, new_state = self.one_step_module(inp, state)
            states.append((state, new_state))

            while len(states) > self.k2:
                # Delete stuff that is too old
                del states[0]

            if (j+1)%self.k1 == 0:
                loss = self.loss_module(output, target)

                optimizer.zero_grad()
                # backprop last module (keep graph only if they ever overlap)
                start = time.time()
                loss.backward(retain_graph=self.retain_graph)
                for i in range(self.k2-1):
                    # if we get all the way back to the "init_state", stop
                    if states[-i-2][0] is None:
                        break
                    curr_grad = states[-i-1][0].grad
                    states[-i-2][1].backward(curr_grad, retain_graph=self.retain_graph)
                print("bw: {}".format(time.time()-start))
                optimizer.step()

seq_len = 20
layer_size = 50

idx = 0

class MyMod(nn.Module):
    def __init__(self):
        super(MyMod, self).__init__()
        self.lin = nn.Linear(2*layer_size, 2*layer_size)

    def forward(self, inp, state):
        global idx
        full_out = self.lin(torch.cat([inp, state], 1))
        # out, new_state = full_out.chunk(2, dim=1)
        out = full_out.narrow(1, 0, layer_size)
        new_state = full_out.narrow(1, layer_size, layer_size)
        def get_pr(idx_val):
            def pr(*args):
                print("doing backward {}".format(idx_val))
            return pr
        new_state.register_hook(get_pr(idx))
        out.register_hook(get_pr(idx))
        print("doing fw {}".format(idx))
        idx += 1
        return out, new_state

one_step_module = MyMod()
loss_module = nn.MSELoss()
input_sequence = [(torch.rand(200, layer_size), torch.rand(200, layer_size))] * seq_len

optimizer = torch.optim.SGD(one_step_module.parameters(), lr=1e-3)

runner = TBPTT(one_step_module, loss_module, 5, 7, optimizer)

runner.train(input_sequence, torch.zeros(200, layer_size))
print("done")

Here is the weird thing. When I tried to run the code the first time, I kept getting another error and after a thorough speculation I found that in some of the variables such as "one_step_module", "input_sequence" where shadowing other variables outer scope. so after renaming those variables the code ran just fine.
And then, I tried to revise the code a bit further for my own project, I started getting the "inplace operation" error. So, in order to see what went wrong, I fixed the code back to the original code above but I kept getting the error.. I even tried open a new file and copy paste the implementation right from the beginning, and I still can't get the code to run. This is driving me CRAZY.
Here's the "inplace operation" error I started getting from the implementation above.
C:\Users\bboyj\anaconda3\envs\jinkyu\python.exe C:/Users/bboyj/PycharmProjects/pythonProject/test1.py
doing fw 0
doing fw 1
doing fw 2
doing fw 3
doing fw 4
doing backward 4
doing backward 3
doing backward 2
doing backward 1
doing backward 0
bw: 0.17385029792785645
doing fw 5
doing fw 6
doing fw 7
doing fw 8
doing fw 9
doing backward 9
doing backward 8
doing backward 7
doing backward 6
doing backward 5
doing backward 4
C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\autograd\__init__.py:130: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  ..\c10\cuda\CUDAFunctions.cpp:100.)
  Variable._execution_engine.run_backward(
Traceback (most recent call last):
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 100, in <module>
    runner.train(input_sequence, torch.zeros(200, layer_size))
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 59, in train
    states[-i-2][1].backward(curr_grad, retain_graph=self.retain_graph)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\autograd\__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [100, 100]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Process finished with exit code 1

just in case you want to see the specific code triggering the error. Here's the error log with torch anomaly detection.
C:\Users\bboyj\anaconda3\envs\jinkyu\python.exe C:/Users/bboyj/PycharmProjects/pythonProject/test1.py
doing fw 0
doing fw 1
doing fw 2
doing fw 3
doing fw 4
doing backward 4
doing backward 3
doing backward 2
doing backward 1
doing backward 0
bw: 0.17083358764648438
doing fw 5
doing fw 6
doing fw 7
doing fw 8
doing fw 9
doing backward 9
doing backward 8
doing backward 7
doing backward 6
doing backward 5
doing backward 4
C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\autograd\__init__.py:130: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  ..\c10\cuda\CUDAFunctions.cpp:100.)
  Variable._execution_engine.run_backward(
C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\autograd\__init__.py:130: UserWarning: Error detected in AddmmBackward. Traceback of forward call that caused the error:
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 101, in <module>
    runner.train(input_sequence, torch.zeros(200, layer_size))
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 41, in train
    output, new_state = self.one_step_module(inp, state)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 78, in forward
    full_out = self.lin(torch.cat([inp111, state111], 1))
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\nn\modules\linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\nn\functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())
 (Triggered internally at  ..\torch\csrc\autograd\python_anomaly_mode.cpp:104.)
  Variable._execution_engine.run_backward(
Traceback (most recent call last):
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 101, in <module>
    runner.train(input_sequence, torch.zeros(200, layer_size))
  File "C:/Users/bboyj/PycharmProjects/pythonProject/test1.py", line 60, in train
    states[-i-2][1].backward(curr_grad, retain_graph=self.retain_graph)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\bboyj\anaconda3\envs\jinkyu\lib\site-packages\torch\autograd\__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [100, 100]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The main problem is that the first iteration is fine because the loss is calculated only with the new hidden state and the "detached and required_grad = True" state, but the second iteration when it tries to backward on the previous set of hidden states which already have "backwarded" it raises error. So in this case, after forwarding and backward on t =0,1,2,3,4 and forwarding on t = 5,6,7,8,9, when it tries to backward on t=9,8,7,6,5,4,3 (because k2 is 7), backward works fine for t=9,8,7,6,5 but fails on t = 4.
Can anyone please shed some light on this??

Output:
After a careful speculation of the code, I've tracked the bug down. The problem was that after "backwarding" on the previous hidden states, the optimizer was trying to step on the hidden states that had been calculated already. I moved the optimizer out of scope of the for loop and every thing works fine!
I'm leaving this answer for those of you who are trying to implement truncated bptt.


Instruction: Pytorch: Converting a VGG model into a sequential model, but getting different outputs

Input:
Background:
I'm working on an adversarial detector method which requires to access the outputs from each hidden layer.
I loaded a pretrained VGG16 from torchvision.models.
To access the output from each hidden layer, I put it into a sequential model:
vgg16 = models.vgg16(pretrained=True)

vgg16_seq = nn.Sequential(*(
    list(list(vgg16.children())[0]) +
    [nn.AdaptiveAvgPool2d((7, 7)), nn.Flatten()] +
    list(list(vgg16.children())[2])))

Without nn.Flatten(), the forward method will complaint about dimensions don't match between mat1 and mat2.
I looked into the torchvision VGG implementation, it uses the [feature..., AvgPool, flatten, classifier...] structure.
Since AdaptiveAvgPool2d layer and Flatten layer have no parameters, I assume this should work, but I have different outputs.
output1 = vgg16(X_small)
print(output1.size())
output2 = vgg16_seq(X_small)
print(output2.size())
torch.equal(output1, output2)

Problem: They are in the same dimension but different outputs.

torch.Size([32, 1000])
torch.Size([32, 1000])
False

I tested the outputs right after the AdaptiveAvgPool2d  layer, the outputs are equal:
output1 = nn.Sequential(*list(vgg16.children())[:2])(X_small)
print(output1.size())
output2 = nn.Sequential(*list(vgg16_seq)[:32])(X_small)
print(output2.size())
torch.equal(output1, output2)

torch.Size([32, 512, 7, 7])
torch.Size([32, 512, 7, 7])
True

Can someone point out what went wrong?
Thank you

Output:
You need to call the eval mode before doing inference.
i.e.
vgg16.eval()
vgg16_seq.eval()


Instruction: generating segment labels for a Tensor given a value indicating segment boundaries

Input:
Does anyone know of a way to generate a 'segment label' for a Tensor, given a unique value that represents segment boundaries within the Tensor?
For example, given a 1D input tensor where the value 1 represents a segment boundary,
x = torch.Tensor([5, 4, 1, 3, 6, 2])
the resulting segment label Tensor should have the same shape with values representing the two segments:
segment_label = torch.Tensor([1, 1, 1, 2, 2, 2])
Likewise, for a batch of inputs, e.g. batch size = 3,
x = torch.Tensor([
    [5, 4, 1, 3, 6, 2],
    [9, 4, 5, 1, 8, 10],
    [10, 1, 5, 4, 8, 9]
    ])

the resulting segment label Tensor (using 1 as the segment separator) should look something like this:
segment_label = torch.Tensor([
    [1, 1, 1, 2, 2, 2],
    [1, 1, 1, 1, 2, 2],
    [1, 1, 2, 2, 2, 2]
    ])

Context: I'm currently working with Fairseq's Transformer implementation in PyTorch for a seq2seq NLP task. I am looking for a way to incorporate BERT-like segment embeddings in Transformer during the encoder's forward pass, rather than modifying an exisiting dataset used for translation tasks such as language_pair_dataset.
Thanks in advance!

Output:
You can use torch.cumsum to pull the trick:
mask = (x == 1).to(x)  # mask with only the boundaries
segment_label = mask.cumsum(dim=-1) - mask + 1

Results with the desired segment_label.


Instruction: How to generate the architectural figure of the network in pytorch when the type of the output of the network is list?

Input:
I wonder how to use tochviz to generate network architecture, when the output is a list type?
the demo code is as follows:
 import torch
 import torch.nn as nn
 class ConvNet(nn.Module):
     def __init__(self):
         super(ConvNet, self).__init__()
         self.conv1 = nn.Sequential(
             nn.Conv2d(1, 16, 3, 1, 1),
             nn.ReLU(),
             nn.AvgPool2d(2, 2)
         )
         self.conv2 = nn.Sequential(
             nn.Conv2d(16, 32, 3, 1, 1),
             nn.ReLU(),
             nn.MaxPool2d(2, 2)
         )
         self.fc = nn.Sequential(
             nn.Linear(32 * 7 * 7, 128),
             nn.ReLU(),
             nn.Linear(128, 64),
             nn.ReLU()
         )
         self.out = nn.Linear(64, 10)
     def forward(self, x):
         x = self.conv1(x)
         x = self.conv2(x)
         x = x.view(x.size(0), -1)
         x = self.fc(x)
         output = []
         output.append(x)
         output.append(self.out(x))
         return output
 MyConvNet = ConvNet()

and I use torchviz to view this network's architecture like
 from torchviz import make_dot
 x = torch.randn(1, 1, 28, 28).requires_grad_(True)
 y = MyConvNet(x)
 MyConvNetVis = make_dot(y, params=dict(list(MyConvNet.named_parameters()) + [('x', x)]))
 MyConvNetVis.format = "png"
 MyConvNetVis.directory = "data"
 MyConvNetVis.view()

then, I was blocked with this problem
AttributeError                            Traceback (most recent call last)
<ipython-input-23-c8e3cd3a8b4e> in <module>
      2 x = torch.randn(1, 1, 28, 28).requires_grad_(True)
      3 y = MyConvNet(x)
----> 4 MyConvNetVis = make_dot(y, params=dict(list(MyConvNet.named_parameters()) + [('x', x)]))
      5 MyConvNetVis.format = "png"
      6 MyConvNetVis.directory = "data"

~/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torchviz/dot.py in make_dot(var, params)
     35         return '(' + (', ').join(['%d' % v for v in size]) + ')'
     36
---> 37     output_nodes = (var.grad_fn,) if not isinstance(var, tuple) else tuple(v.grad_fn for v in var)
     38
     39     def add_nodes(var):

AttributeError: 'list' object has no attribute 'grad_fn'

Any advice will be appreciated.

Output:
The error indicates torchviz is trying to navigate through the network using grad_fn in order to compute its own graph. However, a tuple is not a tensor and doesn't have the gra_fn property.
I'm not quite sure you can have multiple outputs (i.e. a tuple as output) when working with torchviz. As a workaround, if you just want to visualize the network you could replace the tuple with concatenation of the two tensors using torch.cat:
def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)
    x = x.view(x.size(0), -1)
    x = self.fc(x)
    out = self.out(x)
    output = torch.cat([x, out], dim=1)
    return output

The result is:

Notice how the last node is a CatBackward with two incoming branches, one from AddmmBackward (out) and the other from ReluBackward0 (x). This last node is fictitious and is not present in your actual model, so you could remove it from the graph, by hand.


Instruction: pytorch can't shuffle the dataset

Input:
i am trying to make an ai with the mnist dataset from torchvision and makeing it with pytorch, but when i type some of the code that shuffle the data, and run it, it say:
    trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'

i tried a difrent method but it still do not work, and it says:
    trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
AttributeError: type object 'Variable' has no attribute 'DataLoader'

the code i use is:
import torch
import numpy as np
import torchvision
from torchvision import transforms, datasets

train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
testset = torch.utils.data.Dataloader(test, batch_size=10, shuffle=True)

for data in trainset:
    print(data)
    break

the eror from this code:
    trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'

i tried a new version but it still not working:
import torch
import numpy as np
import torchvision
from torchvision import transforms, datasets

train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()])

trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
testset = torch.autograd.Variable.DataLoader(test, batch_size=10, shuffle=True)

for data in trainset:
    print(data)
    break

the eror from this code:
    trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
AttributeError: type object 'Variable' has no attribute 'DataLoader'

i am stil confused why it is not working,i was folowing a tutorial but it did not work

Output:
You have a simple typo: Dataloader -> DataLoader (capital L).
Try:
 trainset = torch.utils.data.DataLoader(train, batch_size=10, shuffle=True)


Instruction: model.to(device) for Pytorch Lighting

Input:
I currently train my model using GPUs using Pytorch Lightning
trainer = pl.Trainer( gpus=[0,1],
        distributed_backend='ddp',
        resume_from_checkpoint=hparams["resume_from_checkpoint"])

    trainer.fit(model, train_dataloader=train_loader, val_dataloaders=val_loader)

Instructions are also clear for how to run test samples with trainer defined to use GPU
trainer.test(test_dataloader=test_dataloader)

and also how to load a model and use it interactively
model = transformer.Model.load_from_checkpoint('/checkpoints/run_300_epoch_217.ckpt')
results = model(in_data,

I use the later to interface with an interactive system via sockets in a docker container.
Is there a proper way to make this Pytorch Lightning model run on GPU?
Lightning instructions say not to use model.to(device), but it appears to work just like Pytorch. Reason for instructions to avoid a side effect?
I started reading about ONNX, but would rather just have an easy way to specify GPU since the interactive setup works perfectly with cpu.

Output:
My understanding is that "Remove any .cuda() or to.device() calls" is only for using with the Lightning trainer, because the trainer handles that itself.
If you don't use the trainer, a LightningModule module is basically just a regular PyTorch model with some naming conventions. So using model.to(device) is how to run on GPU.


Instruction: Difference between these implementations of LSTM Autoencoder?

Input:
Specifically what spurred this question is the return_sequence argument of TensorFlow's version of an LSTM layer.
The docs say:

Boolean. Whether to return the last output. in the output sequence,
or the full sequence. Default: False.

I've seen some implementations, especially autoencoders that use this argument to strip everything but the last element in the output sequence as the output of the 'encoder' half of the autoencoder.
Below are three different implementations. I'd like to understand the reasons behind the differences, as the seem like very large differences but all call themselves the same thing.
Example 1 (TensorFlow):
This implementation strips away all outputs of the LSTM except the last element of the sequence, and then repeats that element some number of times to reconstruct the sequence:
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_in,1)))
# Decoder below
model.add(RepeatVector(n_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))

When looking at implementations of autoencoders in PyTorch, I don't see authors doing this. Instead they use the entire output of the LSTM for the encoder (sometimes followed by a dense layer and sometimes not).
Example 1 (PyTorch):
This implementation trains an embedding BEFORE an LSTM layer is applied... It seems to almost defeat the idea of an LSTM based auto-encoder... The sequence is already encoded by the time it hits the LSTM layer.
class EncoderLSTM(nn.Module):
  def __init__(self, input_size, hidden_size, n_layers=1, drop_prob=0):
    super(EncoderLSTM, self).__init__()
    self.hidden_size = hidden_size
    self.n_layers = n_layers

    self.embedding = nn.Embedding(input_size, hidden_size)
    self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers, dropout=drop_prob, batch_first=True)

  def forward(self, inputs, hidden):
    # Embed input words
    embedded = self.embedding(inputs)
    # Pass the embedded word vectors into LSTM and return all outputs
    output, hidden = self.lstm(embedded, hidden)
    return output, hidden

Example 2 (PyTorch):
This example encoder first expands the input with one LSTM layer, then does its compression via a second LSTM layer with a smaller number of hidden nodes. Besides the expansion, this seems in line with this paper I found: https://arxiv.org/pdf/1607.00148.pdf
However, in this implementation's decoder, there is no final dense layer. The decoding happens through a second lstm layer that expands the encoding back to the same dimension as the original input. See it here. This is not in line with the paper (although I don't know if the paper is authoritative or not).
class Encoder(nn.Module):
  def __init__(self, seq_len, n_features, embedding_dim=64):
    super(Encoder, self).__init__()
    self.seq_len, self.n_features = seq_len, n_features
    self.embedding_dim, self.hidden_dim = embedding_dim, 2 * embedding_dim
    self.rnn1 = nn.LSTM(
      input_size=n_features,
      hidden_size=self.hidden_dim,
      num_layers=1,
      batch_first=True
    )
    self.rnn2 = nn.LSTM(
      input_size=self.hidden_dim,
      hidden_size=embedding_dim,
      num_layers=1,
      batch_first=True
    )
  def forward(self, x):
    x = x.reshape((1, self.seq_len, self.n_features))
    x, (_, _) = self.rnn1(x)
    x, (hidden_n, _) = self.rnn2(x)
    return hidden_n.reshape((self.n_features, self.embedding_dim))

Question:
I'm wondering about this discrepancy in implementations. The difference seems quite large. Are all of these valid ways to accomplish the same thing? Or are some of these mis-guided attempts at a "real" LSTM autoencoder?

Output:
There is no official or correct way of designing the architecture of an LSTM based autoencoder... The only specifics the name provides is that the model should be an Autoencoder and that it should use an LSTM layer somewhere.
The implementations you found are each different and unique on their own even though they could be used for the same task.
Let's describe them:

TF implementation:

It assumes the input has only one channel, meaning that each element in the sequence is just a number and that this is already preprocessed.
The default behaviour of the LSTM layer in Keras/TF is to output only the last output of the LSTM, you could set it to output all the output steps with the return_sequences parameter.
In this case the input data has been shrank to (batch_size, LSTM_units)
Consider that the last output of an LSTM is of course a function of the previous outputs (specifically if it is a stateful LSTM)
It applies a Dense(1) in the last layer in order to get the same shape as the input.

PyTorch 1:

They apply an embedding to the input before it is fed to the LSTM.
This is standard practice and it helps for example to transform each input element to a vector form (see word2vec for example where in a text sequence, each word that isn't a vector is mapped into a vector space). It is only a preprocessing step so that the data has a more meaningful form.
This does not defeat the idea of the LSTM autoencoder, because the embedding is applied independently to each element of the input sequence, so it is not encoded when it enters the LSTM layer.

PyTorch 2:

In this case the input shape is not (seq_len, 1) as in the first TF example, so the decoder doesn't need a dense after. The author used a number of units in the LSTM layer equal to the input shape.

In the end you choose the architecture of your model depending on the data you want to train on, specifically: the nature (text, audio, images), the input shape, the amount of data you have and so on...


Instruction: pytorch tensors cat on dim =0 not worked for me

Input:
I have a problem with cat in pytorch. I want to concatenate tensors on dim=0, for exampe, I want something like this
>>> x = torch.randn(2, 3)
>>> x
tensor([[ 0.6580, -1.0969, -0.4614],
        [-0.1034, -0.5790,  0.1497]])
>>> torch.cat((x, x, x), 0)
tensor([[ 0.6580, -1.0969, -0.4614],
        [-0.1034, -0.5790,  0.1497],
        [ 0.6580, -1.0969, -0.4614],
        [-0.1034, -0.5790,  0.1497],
        [ 0.6580, -1.0969, -0.4614],
        [-0.1034, -0.5790,  0.1497]])

but, when I try to do it in my program I have
def create_batches_to_device(train_df, test_df, device,batch_size=2):
    train_tensor = torch.tensor([])
    for i in range (batch_size):
        rand_2_strs = train_df.sample(2)
        tmp_tensor = torch.tensor([rand_2_strs.get('Sigma').iloc[0],rand_2_strs.get('Sigma').iloc[1],
                               rand_2_strs.get('mu').iloc[0],rand_2_strs.get('mu').iloc[1],
                               rand_2_strs.get('th').iloc[0],rand_2_strs.get('th').iloc[1],
                               np.log(weighted_mse(np.array(rand_2_strs.get('Decay').iloc[0]),np.array(rand_2_strs.get('Decay').iloc[1]),t)[0])])
        print("it is tmp tensor")
        print(tmp_tensor)
        train_tensor = torch.cat((train_tensor,tmp_tensor),dim=0)
        print("this is after cat")
        print(train_tensor)
create_batches_to_device(train_data, test_data, device)

I have a result
it is tmp tensor
tensor([ 0.3244, -0.6401, -0.7959,  0.9019,  0.1468, -1.7093, -6.4419],
       dtype=torch.float64)
this is after cat
tensor([ 0.3244, -0.6401, -0.7959,  0.9019,  0.1468, -1.7093, -6.4419],
       dtype=torch.float64)
it is tmp tensor
tensor([ 1.2923, -0.3088, -0.1275,  0.6417, -1.3383,  1.4020, 28.9065],
       dtype=torch.float64)
this is after cat
tensor([ 0.3244, -0.6401, -0.7959,  0.9019,  0.1468, -1.7093, -6.4419,  1.2923,
        -0.3088, -0.1275,  0.6417, -1.3383,  1.4020, 28.9065],
       dtype=torch.float64)

and the result has no matter was dim=0 or dim=-1, the result is the same for both variance
This is example(look what dim=-1)
def create_batches_to_device(train_df, test_df, device,batch_size=2):
    train_tensor = torch.tensor([])
    for i in range (batch_size):
        rand_2_strs = train_df.sample(2)
        tmp_tensor = torch.tensor([rand_2_strs.get('Sigma').iloc[0],rand_2_strs.get('Sigma').iloc[1],
                               rand_2_strs.get('mu').iloc[0],rand_2_strs.get('mu').iloc[1],
                               rand_2_strs.get('th').iloc[0],rand_2_strs.get('th').iloc[1],
                               np.log(weighted_mse(np.array(rand_2_strs.get('Decay').iloc[0]),np.array(rand_2_strs.get('Decay').iloc[1]),t)[0])])
        print("it is tmp tensor")
        print(tmp_tensor)
        train_tensor = torch.cat((train_tensor,tmp_tensor),dim=-1)
        print("this is after cat")
        print(train_tensor)
create_batches_to_device(train_data, test_data, device)

and the result is the same
it is tmp tensor
tensor([  1.0183,   0.2162,   0.4987,  -0.0165,   0.2094,   0.9425, -14.4564],
       dtype=torch.float64)
this is after cat
tensor([  1.0183,   0.2162,   0.4987,  -0.0165,   0.2094,   0.9425, -14.4564],
       dtype=torch.float64)
it is tmp tensor
tensor([ 0.2389, -1.0108, -0.2350,  0.7105, -0.9200,  0.3282,  7.5456],
       dtype=torch.float64)
this is after cat
tensor([  1.0183,   0.2162,   0.4987,  -0.0165,   0.2094,   0.9425, -14.4564,
          0.2389,  -1.0108,  -0.2350,   0.7105,  -0.9200,   0.3282,   7.5456],
       dtype=torch.float64)

Output:
The problem was what tmp_tensor had shape ([7]) so I could to concatenate only on one dimension. The solution was that I shold to add one new string tmp_tensor = torch.unsqueeze(tmp_tensor, 0) and now tmp_tensor([1,7]) and I could using torch.cat without problem
def create_batches_to_device(train_df, test_df, device,batch_size=3):
    train_tensor = torch.tensor([])
    for i in range (batch_size):
        rand_2_strs = train_df.sample(2)
        tmp_tensor = torch.tensor([rand_2_strs.get('Sigma').iloc[0],rand_2_strs.get('Sigma').iloc[1],
                               rand_2_strs.get('mu').iloc[0],rand_2_strs.get('mu').iloc[1],
                               rand_2_strs.get('th').iloc[0],rand_2_strs.get('th').iloc[1],
                               np.log(weighted_mse(np.array(rand_2_strs.get('Decay').iloc[0]),np.array(rand_2_strs.get('Decay').iloc[1]),t)[0])])
        print("it is tmp tensor")
        tmp_tensor = torch.unsqueeze(tmp_tensor, 0)
        print(tmp_tensor.shape)
        train_tensor = torch.cat((train_tensor,tmp_tensor),dim=0)
        print("this is after cat")
        print(train_tensor)
create_batches_to_device(train_data, test_data, device)

and the result is
it is tmp tensor
torch.Size([1, 7])
this is after cat
tensor([[ 0.9207, -0.9658,  0.0492,  1.6959,  0.4620, -0.2433, -6.4764]],
       dtype=torch.float64)
it is tmp tensor
torch.Size([1, 7])
this is after cat
tensor([[ 0.9207, -0.9658,  0.0492,  1.6959,  0.4620, -0.2433, -6.4764],
        [-0.5921, -0.1198,  0.6192, -0.0977, -0.1704,  1.2384,  9.4497]],
       dtype=torch.float64)
it is tmp tensor
torch.Size([1, 7])
this is after cat
tensor([[ 0.9207, -0.9658,  0.0492,  1.6959,  0.4620, -0.2433, -6.4764],
        [-0.5921, -0.1198,  0.6192, -0.0977, -0.1704,  1.2384,  9.4497],
        [ 0.3839, -0.3153,  0.6467, -0.9995, -0.7415, -0.5487, -6.5500]],
       dtype=torch.float64)


Instruction: Translating Conv1D Layer from pytorch to tensorflow/keras

Input:
I want to create an equal keras layer from this source:
Layer=torch.nn.Conv1d(in_features, out_features, 1)

My Input is shaped (Batch_size,Channel,Width)
This Layer is compiled to:
Conv1d(10, 256, kernel_size=(1,), stride=(1,))

By pytorch.
How can I express this Layer in tensorflow?
I have so far this:
layer1 = tf.keras.layers.Conv1D(in_features-out_features+1, kernel_size=1)

But I am not confident that this will is the right approach.

Output:
In tensorflow's keras you write something like:
layer1 = tf.keras.layers.Conv1D(filters=256, kernel_size=1)(layer0)


Instruction: how can I clear graphic card memory after training in pytorch?

Input:
I am dealing with pytorch in colab
While training, pytorch consumes enormous memory
after training, I saved model, and loaded model to another notebook(note 2).
in note 2, after loading state_dict and everything, pytorch consumes way less memory than in training state.
So, I wonder 'useless' data is stored in graphic card memory while training(in my case, about 13gb)...
If so, how do I delete useless data after training?
plus. I tried to delete variables used while training, but wasn't big enough(about 2gb)

Output:
This is to be expected while training. During the training process, the operations themselves will take up memory.
For example, consider the following operation -
a = np.random.rand(100, 500, 300)
b = np.random.rand(200, 500, 300)
c = (a[:, None, :, :] * b[None, :, :, :]).sum(-1).sum(-1)

The memory size of a, b and c individually is around 400 MB. However, if you check
%memit (a[:, None, :, :] * b[None, :, :, :]).sum(-1).sum(-1)

That's 23 GB! The line itself takes up a lot of memory to actually do the operation because there are massive intermediate arrays involved. These arrays are temporary and are automatically deleted after the operation is over. So you deleting some variables isn't going to do much for reducing the footprint.
The way to get around this is to use memory optimized operations.
For example, doing np.tensordot(a, b, ((1, 2), (1, 2))) instead of multiplying by broadcasting leaves a much better memory footprint.
So what you need to do is to identify which operation in your code is requiring such a huge memory and see if you can replace that with a more memory efficient equivalent (which might not even be possible depending on your specific use-case).


Instruction: PyTorch LogSoftmax vs Softmax for CrossEntropyLoss

Input:
I understand that PyTorch's LogSoftmax function is basically just a more numerically stable way to compute Log(Softmax(x)). Softmax lets you convert the output from a Linear layer into a categorical probability distribution.
The pytorch documentation says that CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class.
Looking at NLLLoss, I'm still confused...Are there 2 logs being used? I think of negative log as information content of an event. (As in entropy)
After a bit more looking, I think that NLLLoss assumes that you're actually passing in log probabilities instead of just probabilities. Is this correct? It's kind of weird if so...

Output:
Yes, NLLLoss takes log-probabilities (log(softmax(x))) as input. Why?. Because if you add a nn.LogSoftmax (or F.log_softmax) as the final layer of your model's output, you can easily get the probabilities using torch.exp(output), and in order to get cross-entropy loss, you can directly use nn.NLLLoss. Of course, log-softmax is more stable as you said.
And, there is only one log (it's in nn.LogSoftmax). There is no log in nn.NLLLoss.
nn.CrossEntropyLoss() combines nn.LogSoftmax() (log(softmax(x))) and nn.NLLLoss() in one single class. Therefore, the output from the network that is passed into nn.CrossEntropyLoss needs to be the raw output of the network (called logits), not the output of the softmax function.


Instruction: Why are the MNIST images 1x28x28 tensors?

Input:
I made the MNIST images which are 28x28 pixel images into tensors with
dataset = MNIST(root='data/', train=True, transform=transforms.ToTensor())

and when I run
img_tensor, label = dataset[0]
print(img_tensor.shape, label)

It says the shape is torch.Size([1, 28, 28]).
Why is it a 1x28x28? What does the first dimension mean? and what is the point of a 1x28x28 opposed to 28x28?

Output:
An image seen as a matrix has always 3 dimensions: channels, width and height. 28  and 28  are width and height of course. The 1 in this case is the channel. So what's the channel? Every pixel is represented by three colors: red, blue and green. For each color, you will have one color-channel, so normally 3 (RGB). This makes a pictures dimension (3, W, H). So why do you have a 1 there? Because the MNIST images are black and white and therefore dont need three different color-channel to represent the final color, one channel is enough, therefore for black and white images you dimension is (1, W, H).
Here is a picture below to visualize the dimensions:

source: https://commons.wikimedia.org/wiki/File:RGB_channels_separation.png
So you see, for black and white images you only need one channel.
Normally you could ignore the 1 dimension, but pytorch demands the channel dimension.


Instruction: TypeError: backward() got an unexpected keyword argument 'grad_tensors' in pytorch

Input:
I have the following
w = torch.tensor([1.], requires_grad=True)
x = torch.tensor([2.], requires_grad=True)

a = torch.add(w, x)
b = torch.add(w, 1)

y0 = torch.mul(a, b)    # y0 = (x+w) * (w+1)
y1 = torch.add(a, b)    # y1 = (x+w) + (w+1)

loss = torch.cat([y0, y1], dim=0)       # [y0, y1]

weight = torch.tensor([1., 2.])

loss.backward(grad_tensors=weight)

The above give me TypeError: backward() got an unexpected keyword argument 'grad_tensors'
I check the website , the grad_tensors does live in the backward.
However, when I use
loss.backward(gradient=weight)

It works. gradient is not a parameters in backward. Any idea of that? my pytorch version is 1.7.0. Thanks.

Output:
You are calling the torch.Tensor.backward, not torch.autograd.backward.
As for your second question about the difference b/w the two,  torch.Tensor.backward internally calls torch.autograd.backward, which calculates gradients of given tensors w.r.t. graph leaves.
torch.autograd.backward(self, gradient, retain_graph, create_graph)

which corresponds to
torch.autograd.backward(tensors: self, grad_tensors: gradient, retain_graph, create_graph)

Thus, below two are equivalent:
loss.backward(gradient=weight)
torch.autograd.backward(loss, weight)


Instruction: LSTM Autoencoder problems

Input:
TLDR:
Autoencoder underfits timeseries reconstruction and just predicts average value.
Question Set-up:
Here is a summary of my attempt at a sequence-to-sequence autoencoder. This image was taken from this paper: https://arxiv.org/pdf/1607.00148.pdf

Encoder: Standard LSTM layer. Input sequence is encoded in the final hidden state.
Decoder: LSTM Cell (I think!). Reconstruct the sequence one element at a time, starting with the last element x[N].
Decoder algorithm is as follows for a sequence of length N:

Get Decoder initial hidden state hs[N]: Just use encoder final hidden state.
Reconstruct last element in the sequence: x[N]= w.dot(hs[N]) + b.
Same pattern for other elements: x[i]= w.dot(hs[i]) + b
use x[i] and hs[i] as inputs to LSTMCell to get x[i-1] and hs[i-1]

Minimum Working Example:
Here is my implementation, starting with the encoder:
class SeqEncoderLSTM(nn.Module):
    def __init__(self, n_features, latent_size):
        super(SeqEncoderLSTM, self).__init__()

        self.lstm = nn.LSTM(
            n_features,
            latent_size,
            batch_first=True)

    def forward(self, x):
        _, hs = self.lstm(x)
        return hs

Decoder class:
class SeqDecoderLSTM(nn.Module):
    def __init__(self, emb_size, n_features):
        super(SeqDecoderLSTM, self).__init__()

        self.cell = nn.LSTMCell(n_features, emb_size)
        self.dense = nn.Linear(emb_size, n_features)

    def forward(self, hs_0, seq_len):

        x = torch.tensor([])

        # Final hidden and cell state from encoder
        hs_i, cs_i = hs_0

        # reconstruct first element with encoder output
        x_i = self.dense(hs_i)
        x = torch.cat([x, x_i])

        # reconstruct remaining elements
        for i in range(1, seq_len):
            hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
            x_i = self.dense(hs_i)
            x = torch.cat([x, x_i])
        return x

Bringing the two together:
class LSTMEncoderDecoder(nn.Module):
    def __init__(self, n_features, emb_size):
        super(LSTMEncoderDecoder, self).__init__()
        self.n_features = n_features
        self.hidden_size = emb_size

        self.encoder = SeqEncoderLSTM(n_features, emb_size)
        self.decoder = SeqDecoderLSTM(emb_size, n_features)

    def forward(self, x):
        seq_len = x.shape[1]
        hs = self.encoder(x)
        hs = tuple([h.squeeze(0) for h in hs])
        out = self.decoder(hs, seq_len)
        return out.unsqueeze(0)

And here's my training function:
def train_encoder(model, epochs, trainload, testload=None, criterion=nn.MSELoss(), optimizer=optim.Adam, lr=1e-6,  reverse=False):

    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(f'Training model on {device}')
    model = model.to(device)
    opt = optimizer(model.parameters(), lr)

    train_loss = []
    valid_loss = []

    for e in tqdm(range(epochs)):
        running_tl = 0
        running_vl = 0
        for x in trainload:
            x = x.to(device).float()
            opt.zero_grad()
            x_hat = model(x)
            if reverse:
                x = torch.flip(x, [1])
            loss = criterion(x_hat, x)
            loss.backward()
            opt.step()
            running_tl += loss.item()

        if testload is not None:
            model.eval()
            with torch.no_grad():
                for x in testload:
                    x = x.to(device).float()
                    loss = criterion(model(x), x)
                    running_vl += loss.item()
                valid_loss.append(running_vl / len(testload))
            model.train()

        train_loss.append(running_tl / len(trainload))

    return train_loss, valid_loss

Data:
Large dataset of events scraped from the news (ICEWS). Various categories exist that describe each event. I initially one-hot encoded these variables, expanding the data to 274 dimensions. However, in order to debug the model, I've cut it down to a single sequence that is 14 timesteps long and only contains 5 variables. Here is the sequence I'm trying to overfit:
tensor([[0.5122, 0.0360, 0.7027, 0.0721, 0.1892],
        [0.5177, 0.0833, 0.6574, 0.1204, 0.1389],
        [0.4643, 0.0364, 0.6242, 0.1576, 0.1818],
        [0.4375, 0.0133, 0.5733, 0.1867, 0.2267],
        [0.4838, 0.0625, 0.6042, 0.1771, 0.1562],
        [0.4804, 0.0175, 0.6798, 0.1053, 0.1974],
        [0.5030, 0.0445, 0.6712, 0.1438, 0.1404],
        [0.4987, 0.0490, 0.6699, 0.1536, 0.1275],
        [0.4898, 0.0388, 0.6704, 0.1330, 0.1579],
        [0.4711, 0.0390, 0.5877, 0.1532, 0.2201],
        [0.4627, 0.0484, 0.5269, 0.1882, 0.2366],
        [0.5043, 0.0807, 0.6646, 0.1429, 0.1118],
        [0.4852, 0.0606, 0.6364, 0.1515, 0.1515],
        [0.5279, 0.0629, 0.6886, 0.1514, 0.0971]], dtype=torch.float64)

And here is the custom Dataset class:
class TimeseriesDataSet(Dataset):
    def __init__(self, data, window, n_features, overlap=0):
        super().__init__()
        if isinstance(data, (np.ndarray)):
            data = torch.tensor(data)
        elif isinstance(data, (pd.Series, pd.DataFrame)):
            data = torch.tensor(data.copy().to_numpy())
        else:
            raise TypeError(f"Data should be ndarray, series or dataframe. Found {type(data)}.")

        self.n_features = n_features
        self.seqs = torch.split(data, window)

    def __len__(self):
        return len(self.seqs)

    def __getitem__(self, idx):
        try:
            return self.seqs[idx].view(-1, self.n_features)
        except TypeError:
            raise TypeError("Dataset only accepts integer index/slices, not lists/arrays.")

Problem:
The model only learns the average, no matter how complex I make the model or now long I train it.
Predicted/Reconstruction:

Actual:

My research:
This problem is identical to the one discussed in this question: LSTM autoencoder always returns the average of the input sequence
The problem in that case ended up being that the objective function was averaging the target timeseries before calculating loss. This was due to some broadcasting errors because the author didn't have the right sized inputs to the objective function.
In my case, I do not see this being the issue. I have checked and double checked that all of my dimensions/sizes line up. I am at a loss.
Other Things I've Tried

I've tried this with varied sequence lengths from 7 timesteps to 100 time steps.
I've tried with varied number of variables in the time series. I've tried with univariate all the way to all 274 variables that the data contains.
I've tried with various reduction parameters on the nn.MSELoss module. The paper calls for sum, but I've tried both sum and mean. No difference.
The paper calls for reconstructing the sequence in reverse order (see graphic above). I have tried this method using the flipud on the original input (after training but before calculating loss). This makes no difference.
I tried making the model more complex by adding an extra LSTM layer in the encoder.
I've tried playing with the latent space. I've tried from 50% of the input number of features to 150%.
I've tried overfitting a single sequence (provided in the Data section above).

Question:
What is causing my model to predict the average and how do I fix it?

Output:
Okay, after some debugging I think I know the reasons.
TLDR

You try to predict next timestep value instead of difference between current timestep and the previous one
Your hidden_features number is too small making the model unable to fit even a single sample

Analysis
Code used
Let's start with the code (model is the same):
import seaborn as sns
import matplotlib.pyplot as plt

def get_data(subtract: bool = False):
    # (1, 14, 5)
    input_tensor = torch.tensor(
        [
            [0.5122, 0.0360, 0.7027, 0.0721, 0.1892],
            [0.5177, 0.0833, 0.6574, 0.1204, 0.1389],
            [0.4643, 0.0364, 0.6242, 0.1576, 0.1818],
            [0.4375, 0.0133, 0.5733, 0.1867, 0.2267],
            [0.4838, 0.0625, 0.6042, 0.1771, 0.1562],
            [0.4804, 0.0175, 0.6798, 0.1053, 0.1974],
            [0.5030, 0.0445, 0.6712, 0.1438, 0.1404],
            [0.4987, 0.0490, 0.6699, 0.1536, 0.1275],
            [0.4898, 0.0388, 0.6704, 0.1330, 0.1579],
            [0.4711, 0.0390, 0.5877, 0.1532, 0.2201],
            [0.4627, 0.0484, 0.5269, 0.1882, 0.2366],
            [0.5043, 0.0807, 0.6646, 0.1429, 0.1118],
            [0.4852, 0.0606, 0.6364, 0.1515, 0.1515],
            [0.5279, 0.0629, 0.6886, 0.1514, 0.0971],
        ]
    ).unsqueeze(0)

    if subtract:
        initial_values = input_tensor[:, 0, :]
        input_tensor -= torch.roll(input_tensor, 1, 1)
        input_tensor[:, 0, :] = initial_values
    return input_tensor

if __name__ == "__main__":
    torch.manual_seed(0)

    HIDDEN_SIZE = 10
    SUBTRACT = False

    input_tensor = get_data(SUBTRACT)
    model = LSTMEncoderDecoder(input_tensor.shape[-1], HIDDEN_SIZE)
    optimizer = torch.optim.Adam(model.parameters())
    criterion = torch.nn.MSELoss()
    for i in range(1000):
        outputs = model(input_tensor)
        loss = criterion(outputs, input_tensor)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        print(f"{i}: {loss}")
        if loss < 1e-4:
            break

    # Plotting
    sns.lineplot(data=outputs.detach().numpy().squeeze())
    sns.lineplot(data=input_tensor.detach().numpy().squeeze())
    plt.show()

What it does:

get_data either works on the data your provided if subtract=False or (if subtract=True) it subtracts value of the previous timestep from the current timestep
Rest of the code optimizes the model until 1e-4 loss reached (so we can compare how model's capacity and it's increase helps and what happens when we use the difference of timesteps instead of timesteps)

We will only vary HIDDEN_SIZE and SUBTRACT parameters!
NO SUBTRACT, SMALL MODEL

HIDDEN_SIZE=5
SUBTRACT=False

In this case we get a straight line. Model is unable to fit and grasp the phenomena presented in the data (hence flat lines you mentioned).

1000 iterations limit reached
SUBTRACT, SMALL MODEL

HIDDEN_SIZE=5
SUBTRACT=True

Targets are now far from flat lines, but model is unable to fit due to too small capacity.

1000 iterations limit reached
NO SUBTRACT, LARGER MODEL

HIDDEN_SIZE=100
SUBTRACT=False

It got a lot better and our target was hit after 942 steps. No more flat lines, model capacity seems quite fine (for this single example!)

SUBTRACT, LARGER MODEL

HIDDEN_SIZE=100
SUBTRACT=True

Although the graph does not look that pretty, we got to desired loss after only 215 iterations.

Finally

Usually use difference of timesteps instead of timesteps (or some other transformation, see here for more info about that). In other cases, neural network will try to simply... copy output from the previous step (as that's the easiest thing to do). Some minima will be found this way and going out of it will require more capacity.
When you use the difference between timesteps there is no way to "extrapolate" the trend from previous timestep; neural network has to learn how the function actually varies
Use larger model (for the whole dataset you should try something like 300 I think), but you can simply tune that one.
Don't use flipud. Use bidirectional LSTMs, in this way you can get info from forward and backward pass of LSTM (not to confuse with backprop!). This also should boost your score

Questions

Okay, question 1: You are saying that for variable x in the time
series, I should train the model to learn x[i] - x[i-1] rather than
the value of x[i]? Am I correctly interpreting?

Yes, exactly. Difference removes the urge of the neural network to base it's predictions on the past timestep too much (by simply getting last value and maybe changing it a little)

Question 2: You said my calculations for zero bottleneck were
incorrect. But, for example, let's say I'm using a simple dense
network as an auto encoder. Getting the right bottleneck indeed
depends on the data. But if you make the bottleneck the same size as
the input, you get the identity function.

Yes, assuming that there is no non-linearity involved which makes the thing harder (see here for similar case). In case of LSTMs there are non-linearites, that's one point.
Another one is that we are accumulating timesteps into single encoder state. So essentially we would have to accumulate timesteps identities into a single hidden and cell states which is highly unlikely.
One last point, depending on the length of sequence, LSTMs are prone to forgetting some of the least relevant information (that's what they were designed to do, not only to remember everything), hence even more unlikely.

Is num_features * num_timesteps not a bottle neck of the same size as
the input, and therefore shouldn't it facilitate the model learning
the identity?

It is, but it assumes you have num_timesteps for each data point, which is rarely the case, might be here. About the identity and why it is hard to do with non-linearities for the network it was answered above.
One last point, about identity functions; if they were actually easy to learn, ResNets architectures would be unlikely to succeed. Network could converge to identity and make "small fixes" to the output without it, which is not the case.

I'm curious about the statement : "always use difference of timesteps
instead of timesteps" It seem to have some normalizing effect by
bringing all the features closer together but I don't understand why
this is key ? Having a larger model seemed to be the solution and the
substract is just helping.

Key here was, indeed, increasing model capacity. Subtraction trick depends on the data really. Let's imagine an extreme situation:

We have 100 timesteps, single feature
Initial timestep value is 10000
Other timestep values vary by 1 at most

What the neural network would do (what is the easiest here)? It would, probably, discard this 1 or smaller change as noise and just predict 1000 for all of them (especially if some regularization is in place), as being off by 1/1000 is not much.
What if we subtract? Whole neural network loss is in the [0, 1] margin for each timestep instead of [0, 1001], hence it is more severe to be wrong.
And yes, it is connected to normalization in some sense come to think about it.


Instruction: PyTorch alternative for tf.data.experimental.sample_from_datasets

Input:
Suppose I have two datasets, dataset one with 100 items and dataset two with 5000 items.
Now I want that during training my model sees as much items from dataset one as from dataset two.
In Tensorflow I can do:
dataset = tf.data.experimental.sample_from_datasets(
    [dataset_one, dataset_two], weights=[50,1], seed=None
)

Is there an alternative in PyTorch that does the same?
I think this is not too difficult to implement by creating a custom dataset (not working example)
from torch.utils.data import Dataset

class SampleDataset(Dataset):
    def __init__(self, datasets, weights):
        self.datasets = datasets
        self.weights = weights

    def __len__(self):
        return sum([len(dataset) for dataset in self.datasets])

    def __getitem__(self, idx):
        # sample a random number and based on that sample an item

        return self.datasets[dataset_idx][sample_idx]

However, this seems quite common. Is there already something like this available?

Output:
I don't think there is a direct equivalent in PyTorch.
However, there's a function called torch.utils.data.WeightedRandomSampler which samples indices based on a list of probabilities. You can use this in combination with torch.data.utils.ConcatDataset and torch.utils.data.DataLoader's sampler option.
I'll give an example with two datasets: SetA has 500 elements and SetB which only has 10.
First, you can create a concatenation of all your datasets with ConcaDataset:
ds = ConcatDataset([SetA(), SetB()])

Then, we need to sample it. The problem is, you can't just give WeightedRandomSampler [50, 1], as you did in Tensorflow. As a workaround, you can create a list of probabilities of the same length as the size of the total dataset.
The corresponding probability list for this example would be:
dist = np.array([1/51]*500 + [50/51]*10)

Essentially, the first 500 indices (i.e. indices 'pointing' to SetA) will have a probability of 1/51 of being choosen while the following 10 indices (i.e. indices in SetB) will have a probability of 50/51 (i.e much more likely to being sampled since there are less elements in SetB, this is the desired result!)
We can create a sampler from that distribution:
WeightedRandomSampler(dist, 10)

Where 10 is the number of sampled elements. I would put the size of the smallest dataset, otherwise you would likely be going over the same datapoints multiple times during the same epoch...
Finally, we just have to instanciate the dataloader with our dataset and sampler:
dl = DataLoader(ds, sampler=sampler)

To summarize:
ds = ConcatDataset([SetA(), SetB()])
dist = np.array([1/51]*500 + [50/51]*10)
sampler = WeightedRandomSampler(dist, 10)
dl = DataLoader(ds, sampler=sampler)

Edit, for any number of datasets:
sets = [SetA(), SetB(), SetC()]
ds = ConcatDataset(sets)

dist = np.concatenate([[(len(ds) - len(s))/len(ds)]*len(s) for s in sets])
sampler = WeightedRandomSampler(weights=dist, num_samplesmin([len(s) for s in sets])
dl = DataLoader(ds, sampler=sampler)


Instruction: How to use the BitShift operator in Pytorch?

Input: Does anyone has an example of how to use the BitShift operator in Pytorch?

Output:
Bitwise shift operator performs element-wise operation.

It works the same way it works in python, and numpy i.e. shift the bits of an integer to the left or right. The << and >> denotes the left and right shift respectively.
x = torch.tensor([16, 4, 1])
y = torch.tensor([1, 2, 3])
z = x << y
print(z)
tensor([32, 16,  8])

It's equivalent to 16 << 1 (np.left_shift(16, 1)), 4 << 2, and 1 << 3.

For each input element, if the attribute "direction" is "RIGHT", this operator moves its binary representation toward the right side so that the input value is effectively decreased. If the attribute "direction" is "LEFT", bits of binary representation moves toward the left side, which results the increase of its actual value.
This operator supports multidirectional (i.e., Numpy-style) broadcasting.


Instruction: Creating new channels from nearby pixels in pytorch

Input:
Given a batch image tensor like B x C x W x H (batchSize,channels,width,height),
I would like to create a new tensor in which the new channels are the channels from nearby pixels (padded with 0s).
For instance, if I choose the nearby pixel size to be 3 x 3 (like a 3 x 3 filter) then there are 9 total nearby pixels and the final tensor size would be B x ( 9 * C ) x W x H.
Any recommendations on doing this, or do I just need to go the brute-force approach through iteration?

Output:
If you want to cut the edges short (img is your image tensor):
from skimage.util import view_as_windows
B,C,W,H = img.shape
img_ = view_as_windows(img,(1,1,3,3)).reshape(B,C,W-2,H-2,-1).transpose(0,1,4,2,3).reshape(B,C*9,W-2,H-2)

And if you want to pad them with 0 instead:
from skimage.util import view_as_windows
img = np.pad(img,((0,0),(0,0),(1,1),(1,1)))
B,C,W,H = img.shape
img_ = view_as_windows(img,(1,1,3,3)).reshape(B,C,W-2,H-2,-1).transpose(0,1,4,2,3).reshape(B,C*9,W-2,H-2)


Instruction: Using an ellipsis for the middle dimensions of a PyTorch tensor

Input:
Suppose I have a torch.Tensor t of shape (8, 3, 32, 32). I want to index along the first and last 2 dimensions only.
In my usecase, t is a batch of 8 images, of which I want to modify a patch. Suppose the patch is given by indices idx_last = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]]).
I also have idx1 = torch.arange(4) : I want the patch for the first 4 images.
The following does not work:
t[idx1, ..., idx_last]
Is there any way to do this ?

Output:
I found one workaround, although it may not be the most efficient.
In the case where idx1 is 1-dimensional (datapoint selection), and idx_last is multidimensional, the following gets the wanted result:
t[(idx1, ...) + tuple(idx_last.T)]

Better solutions are definitely welcome.