Instruction: Meaning of the outlined code written in C?

Input:
I am currently building an understanding around PyTorch and I have come across few codes written in C language. Since I am not that familiar with C conventions, could anyone elaborate what does this code mean?
typedef float float4 __attribute__((ext_vector_type(4)));
typedef float float8 __attribute__((ext_vector_type(8)));
#define LoadFloat8(PTR) *((const float8 *)(PTR))

The code is taken from here

Output:
GNU GCC Vector extensions
the instruction set contains SIMD vector instructions which operate on multiple values contained in one large register at the same time.
for using these extensions we need to provide the necessary data types (like your example). This should be done using an appropriate typedef, for example:
typedef int v4si __attribute__ ((vector_size (16)));

The int type specifies the base type, while the attribute specifies the vector size for the variable, measured in bytes. the declaration above causes the compiler to set the mode for the v4si type to be 16 bytes wide and divided into int sized units. For a 32-bit int this means a vector of 4 units of 4 bytes, and the corresponding mode of foo is V4SI.
int the first typedef in your example it is a way to let the compiler know you only care about the value of the first 4 elements of a 4 element SIMD register. So compiler can  do optimizations that leave different garbage in the high elements/Bytes when manipulating this data type.
keep reading in the link below:
 Using Vector Instructions through Built-in Functions


Instruction: Installed pytorch through conda, but cannot import in Windows 10

Input:
When I issue an Anaconda prompt conda search pytorch
then I get pytorch installed even issuing conda list command gives me:
pytorch                   1.5.1           py3.7_cuda102_cudnn7_0    pytorch

But when I start python on command prompt and then issue import pytorch i get ModuleNotFoundError: No module named 'pytorch'.
Even I tried to do the same after issuing
conda create -n pytorch_env -c pytorch pytorch torchvision
conda activate pytorch_env
conda install -c pytorch pytorch torchvision

as written in Installing PyTorch via Conda but of no use. BTW does it require a restart of the machine after installing Anaconda? Also, let me know how I can use PyTorch in Jupyter notebooks.

Output:
Verify the installation with import torch not pytorch. Example code below, source.
from __future__ import print_function
import torch
x = torch.rand(5, 3)
print(x)

If above throws same issue in Jupyter Notebooks and if you already have GPU enabled, try restarting the Jupyter notebook server as sometimes it requires restarting, user reported.

when I tried to import this package from Jupyter notebook, I got following error message: ModuleNotFoundError: No module named 'torch'. Then, I tried installing Jupyter notebook application from Anaconda navigator for my environment(torch).
Restarted my Jupyter notebook and ran import torch and this time it worked

Otherwise, you need CPU only version of PyTorch.
conda install pytorch torchvision cpuonly -c pytorch


Instruction: What is the param `last_epoch` on Pytorch Optimizer's Schedulers is for?

Input:
Regarding Pytorch Optimizer's Schedulers, what actually means the last_epoch argument?
It says

last_epoch (int) – The index of last epoch. Default: -1.

But it doesn't really explains much for those, like me, are just learning about these schedules.
I read most of that documentation, if not all, and I could understand what it does and why.

Output:
The last_epoch parameter is used when resuming training and you want to start the scheduler where it left off earlier. Its value is increased every time you call .step() of scheduler. The default value of -1 indicates that the scheduler is started from the beginning.
From the docs:

Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning.

For example,
>>> import torch
>>> cc = torch.nn.Conv2d(10,10,3)
>>> myoptimizer = torch.optim.Adam(cc.parameters(), lr=0.1)
>>> myscheduler = torch.optim.lr_scheduler.StepLR(myoptimizer,step_size=1, gamma=0.1)
>>> myscheduler.last_epoch, myscheduler.get_lr()
(0, [0.1])
>>> myscheduler.step()
>>> myscheduler.last_epoch, myscheduler.get_lr()
(1, [0.001])
>>> myscheduler.step()
>>> myscheduler.last_epoch, myscheduler.get_lr()
(2, [0.0001])

Now, if you decide to stop the training in the middle, then resume it, you can provide last_epoch parameter to schedular so that it start from where it was left off, not from the beginning again.
>>> mynewscheduler = torch.optim.lr_scheduler.StepLR(myoptimizer,step_size=1, gamma=0.1, last_epoch=myscheduler.last_epoch)
>>> mynewscheduler.last_epoch, mynewscheduler.get_lr()
(3, [1.0000000000000004e-05])


Instruction: PyTorch can't see GPU (torch.cuda.is_availble() returns False)

Input:
I have a problem where
import torch
print(torch.cuda_is_available())

will print False, and I can't use the GPU available. I've tried it on conda environment, where I've installed the PyTorch version corresponding to the NVIDIA driver I have. I've also tried it in docker container, where I've done the same. I've tried both of these options on a remote server, but they both failed. I know that I've installed the correct driver versions because I've checked the version with nvcc --version before installing PyTorch, and I've checked the GPU connection with nvidia-smi which displays the GPUs on the machines correctly.
Also, I've checked this post and tried exporting CUDA_VISIBLE_DEVICES, but had no luck.
On the server I have NVIDIA V100 GPUs with CUDA version 10.0 (for conda environment) and version 10.2 on a docker container I've built. Any help or push in the right direction would be greatly appreciated. Thanks!

Output:
For anyone else having this problem, it turned out my server manager has not updated the drivers for the server.
I switched to a different server, installed anaconda and things started working like it should, i.e., torch.cuda.is_available() returns True after setting up a fresh environment.


Instruction: How is a minibatch processed by the GPU in PyTorch?

Input:
I am trying to understand how PyTorch actually performs a forward pass over a minibatch. When a minibatch is processed by a network, is each example in the minibatch (e.g. each image) sent forwards individually, one after the other? Or are all examples in the minibatch sent forwards at the same time?
When an example is sent forwards through a network, the additional memory requirement is the activations at each layer. And as long as the network does not take up the entire GPU, then it seems that multiple instantiations of these activations could be stored at the same time. Each instantiation could then be used to store the activations for one example in the minibatch. And therefore, multiple examples could be sent through the network simultaneously. However, I'm unsure whether this is actually done in practice.
I have done some simple experiments, and the time for a forward pass is roughly proportional to the minibatch size. This suggests that the examples are sent through one after the other. If so, then why is it that people say that training is faster when the minibatch size is larger? It seems that the processing time for an entire epoch would not be dependent on the minibatch size.

Output:
I am trying to understand how PyTorch actually performs a forward pass over a minibatch. When a minibatch is processed by a network, is each example in the minibatch (e.g. each image) sent forwards individually, one after the other? Or are all examples in the minibatch sent forwards at the same time?

All at the same time. To do so, it relies on batch processing, broadcasting, element-wise vectorization for non-linear operations (basically, a highly optimized for-loop, sometimes in parrallel) and matrix linear algebra. The later is much more efficient than a for-loop, since it can leverage dedicated hardware component designed for parallel linear algebra (this is true for both cpu and gpu, but gpu are especially well suited for this).

Each instantiation could then be used to store the activations for one example in the minibatch. And therefore, multiple examples could be sent through the network simultaneously. However, I'm unsure whether this is actually done in practice.

This is not how it works, torch is keeping track of "operations", each of them having a backward used computing the gradient of the inputs wrt to the outputs. It is designed to support batch processing and vectorization, such that processing a bunch of samples is done at once as in single backward pass.

I have done some simple experiments, and the time for a forward pass is roughly proportional to the minibatch size.

This is not true. It may be because you are already eating up 100% of the available resources (cpu or gpu), or because you are not doing the profiling properly (which is not so easy to do). If you post an example, one you try to help you on this point.


Instruction: Using torchtext for inference

Input:
I wonder what is the right way to use torchtext for inference.
Let's assume I've trained the model and dump all Fields with built vocabularies. It seems the next step is to use torchtext.data.Example to load one single example. Somehow I should numeralize it by using loaded Fields and create an Iterator.
I would appreciate any simple examples of using torchtext for inference.

Output:
For a trained model and vocabulary (which is part of the text field , you don't have to save the whole class) :
    def read_vocab(path):
        #read vocabulary pkl
        import pickle
        pkl_file = open(path, 'rb')
        vocab = pickle.load(pkl_file)
        pkl_file.close()
        return vocab

    def load_model_and_vocab():
        import torch
        import os.path

        my_path = os.path.abspath(os.path.dirname(__file__))
        vocab_path = os.path.join(my_path, vocab_file)
        weights_path = os.path.join(my_path, WEIGHTS)

        vocab = read_vocab(vocab_path)
        model = classifier(vocab_size=len(vocab))
        model.load_state_dict(torch.load(weights_path))
        model.eval()
        return model, vocab

    def predict(model, vocab, sentence):
        tokenized = [w.text.lower() for w in nlp(sentence)]  # tokenize the sentence
        indexed = [vocab.stoi[t] for t in tokenized]         # convert to integer sequence
        length = [len(indexed)]                              # compute no. of words
        tensor = torch.LongTensor(indexed).to('cpu')         # convert to tensor
        tensor = tensor.unsqueeze(1).T                       # reshape in form of batch,no. of words
        length_tensor = torch.LongTensor(length)             # convert to tensor
        prediction = model(tensor, length_tensor)            # prediction
        return round(1-prediction.item())

"classifier" is the class I defined for my model.
For saving the vocabulary pkl :
    def save_vocab(vocab):
        import pickle
        output = open('vocab.pkl', 'wb')
        pickle.dump(vocab, output)
        output.close()

And for saving the model after training you can use :
    torch.save(model.state_dict(), 'saved_weights.pt')

Tell me if it worked for you!


Instruction: ImportError: cannot import name 'hf_bucket_url' in HuggingFace Transformers

Input:
So I installed the latest version of transformers on Google Colab
!pip install transformers

When trying to invoke the conversion file using
!python /usr/local/lib/python3.6/dist-packages/transformers/convert_pytorch_checkpoint_to_tf2.py .py --help

Or trying to use
from transformers.file_utils import hf_bucket_url.                                 // works
from transformers.convert_pytorch_checkpoint_to_tf2 import *.                      // fails

convert_pytorch_checkpoint_to_tf("gpt2", pytorch_file, config_file, tf_file).

I get this error
 ImportError                               Traceback (most recent call last)

<ipython-input-3-dadaf83ecea0> in <module>()
      1 from transformers.file_utils import hf_bucket_url
----> 2 from transformers.convert_pytorch_checkpoint_to_tf2 import *
      3
      4 convert_pytorch_checkpoint_to_tf("gpt2", pytorch_file, config_file, tf_file)
/usr/local/lib/python3.6/dist-packages/transformers/convert_pytorch_checkpoint_to_tf2.py in <module>()
     20 import os
     21
---> 22 from transformers import (
     23     ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
     24     BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,

ImportError: cannot import name 'hf_bucket_url'

What's going on?

Output: It turns out to be a bug. This PR solves the issue by importing the function hf_bucket_url properly.


Instruction: Python print file name from a Data class

Input:
I'm trying to print a filename from a class. The class is defined in here in the below code. The code runs without any issues but it doesn't print the filename, my guess is that i have an issue with calling the class.
I am trying to call the class in another file to print the file name into a CSV during training, and using this code to do this:
filename=CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
print(filename)

Output:
A CellsDataSet contains a list of filenames in its files attribute, so you can print all of them like this:
filename=CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
print(filename.files)


Instruction: How to get a specific sample from pytorch DataLoader?

Input:
In Pytorch, is there any way of loading a specific single sample using the torch.utils.data.DataLoader class? I'd like to do some testing with it.
The tutorial uses
trainloader = torch.utils.data.DataLoader(...)
images, labels = next(iter(trainloader))

to fetch a random batch of samples. Is there are way, using DataLoader, to get a specific sample?
Cheers

Output:
Turn off the shuffle in DataLoader
Use batch_size to calculate the batch in which the desired sample you are looking for falls in
Iterate to the desired batch

Code
import torch
import numpy as np
import itertools

X= np.arange(100)
batch_size = 2

dataloader = torch.utils.data.DataLoader(X, batch_size=batch_size, shuffle=False)
sample_at = 5
k = int(np.floor(sample_at/batch_size))

my_sample = next(itertools.islice(dataloader, k, None))
print (my_sample)

Output:
tensor([4, 5])


Instruction: Parallelization strategies for deep learning

Input:
What strategies and forms of parallelization are feasible and available for training and serving a neural network?:

inside a machine across cores (e.g. GPU / TPU / CPU)
across machines on a network or a rack

I'm also looking for evidence for how they may also be used in e.g. TensorFlow, PyTorch or MXNet.
Training
To my knowledge, when training large neural networks on large datasets, one could at least have:

Different cores or machines operate on different parts of the graph ("graph splitting"). E.g. backpropagation through the graph itself can be parallelized e.g. by having different layers hosted on different machines since (I think?) the autodiff graph is always a DAG.
Different cores or machines operate on different samples of data ("data splitting"). In SGD, the computation of gradients across batches or samples can also be parallelized (e.g. the gradients can be combined after computing them independently on different batches). I believe this is also called gradient accumulation (?).

When is each strategy better for what type of problem or neural network? Which modes are supported by modern libraries? and can one combine all four (2x2) strategies?
On top of that, I have read about:

Asynchronous training
Synchronous training

but I don't know what exactly that refers to, e.g. is it the computation of gradients on different data batches or the computation of gradients on different subgraphs? Or perhaps it refers to something else altogether?
Serving
If the network is huge, prediction / inference may also be slow, and the model may not fit on a single machine in memory at serving time. Are there any known multi-core and multi-node prediction solutions that work that can handle such models?

Output:
As the question is quite broad, I'll try to shed a little different light and touch on different topics than what was shown in
@Daniel's in-depth answer.
Training
Data parallelization vs model parallelization
As mentioned by @Daniel data parallelism is used way more often and is easier to do correctly. Major caveat of model parallelism is the need to wait for part of neural network and synchronization between them.
Say you have a simple feedforward 5 layer neural network spread across 5 different GPUs, each layer for one device. In this case, during each forward pass each device has to wait for computations from the previous layers. In this simplistic case, copying data between devices and synchronization would take a lot longer and won't bring benefits.
On the other hand, there are models better suited for model parallelization like Inception networks, see picture below:

Here you can see 4 independent paths from previous layer which could go in parallel and only 2 synchronization points (Filter concatenation and Previous Layer).
Questions

E.g. backpropagation through the graph itself can be parallelized e.g.
by having different layers hosted on different machines since (I
think?) the autodiff graph is always a DAG.

It's not that easy. Gradients are calculated based on the loss value (usually) and you need to know gradients of deeper layers to calculate gradients for the more shallow ones. As above, if you have independent paths it's easier and may help, but it's way easier on a single device.

I believe this is also called gradient accumulation (?)

No, it's actually reduction across multiple devices. You can see some of that in PyTorch tutorial. Gradient accumulation is when you run your forward pass (either on single or multiple devices) N times and backpropagate (the gradient is kept in the graph and the values are added during each pass) and optimizer only makes a single step to change neural network's weights (and clears the gradient). In this case, loss is usually divided by the number of steps without optimizer. This is used for more reliable gradient estimation, usually when you are unable to use large batches.
Reduction across devices looks like this:

This is all-reduce in data parallelization, each device calculates the values which are send to all other devices and backpropagated there.

When is each strategy better for what type of problem or neural
network?

Described above, data parallel is almost always fine if you have enough of data and the samples are big (up to 8k samples or more can be done at once without very big struggle).

Which modes are supported by modern libraries?

tensorflow and pytorch both support either, most modern and maintained libraries have those functionalities implemented one way or another

can one combine all four (2x2) strategies

Yes, you can parallelize both model and data across and within machines.

synchronous vs asynchronous

asynchronous
Described by @Daniel in brief, but it's worth mentioning updates are not totally separate. That would make little sense, as we would essentially train N different models based on their batches.
Instead, there is a global parameter space, where each replica is supposed to share calculated updates asynchronously (so forward pass, backward, calculate update with optimizer and share this update to global params).
This approach has one problem though: there is no guarantee that when one worker calculated forward pass another worker updated the parameters, so the update is calculated with respect to old set of params and this is called stale gradients. Due to this, convergence might be hurt.
Other approach is to calculate N steps and updates for each worker and synchronize them afterwards, though it's not used as often.
This part was based on great blogpost and you should definitely read it if interested (there is more about staleness and some solutions).
synchronous
Mostly described previously, there are different approaches but PyTorch gathers output from network and backpropagates on them (torch.nn.parallel.DistributedDataParallel)[https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel]. BTW. You should solely this (no torch.nn.DataParallel) as it overcomes Python's GIL problem.
Takeaways

Data parallelization is always almost used when going for speed up as you "only" have to replicate neural network on each device (either over the network or within single machine), run part of batch on each during the forward pass, concatenate them into a single batch (synchronization) on one device and backpropagate on said.
There are multiple ways to do data parallelization, already introduced by @Daniel
Model parallelization is done when the model is too large to fit on single machine (OpenAI's GPT-3 would be an extreme case) or when the architecture is suited for this task, but both are rarely the case AFAIK.
The more and the longer parallel paths the model has (synchronization points), the better it might be suited for model parallelization
It's important to start workers at similar times with similar loads in order not to way for synchronization processes in synchronous approach or not to get stale gradients in asynchronous (though in the latter case it's not enough).

Serving
Small models
As you are after large models I won't delve into options for smaller ones, just a brief mention.
If you want to serve multiple users over the network you need some way to scale your architecture (usually cloud like GCP or AWS). You could do that using Kubernetes and it's PODs or pre-allocate some servers to handle requests, but that approach would be inefficient (small number of users and running servers would generate pointless costs, while large numbers of users may halt the infrastructure and take too long to process resuests).
Other way is to use autoscaling based on serverless approach. Resources will be provided based on each request so it has large scaling abilities + you don't pay when the traffic is low. You can see Azure Functions as they are on the path to improve it for ML/DL tasks, or torchlambda for PyTorch (disclaimer, I'm the author) for smaller models.
Large models
As mentioned previously, you could use Kubernetes with your custom code or ready to use tools.
In the first case, you can spread the model just the same as for training, but only do forward pass. In this way even giant models can be put up on the network (once again, GPT-3 with 175B parameters), but requires a lot of work.
In the second, @Daniel provided two possibilities. Others worth mentioning could be (read respective docs as those have a lot of functionalities):

KubeFlow - multiple frameworks, based on Kubernetes (so auto-scaling, multi-node), training, serving and what not, connects with other things like MLFlow below
AWS SageMaker - training and serving with Python API, supported by Amazon
MLFlow - multiple frameworks, for experiment handling and serving
BentoML - multiple frameworks, training and serving

For PyTorch, you could read more here, while tensorflow has a lot of serving functionality out of the box via Tensorflow EXtended (TFX).
Questions from OP's comment

Are there any forms of parallelism that are better within a machine vs
across machines

The best for of parallelism would probably be within one giant computer as to minimize transfer between devices.
Additionally, there are different backends (at least in PyTorch) one can choose from (mpi, gloo, nccl) and not all of them support direct sending, receiving, reducing etc. data between devices (some may support CPU to CPU, others GPU to GPU). If there is no direct link between devices, those have to be first copied to another device and copied again to target device (e.g. GPU on other machine -> CPU on host -> GPU on host). See pytorch info.
The more data and the bigger network, the more profitable it should be to parallelize computations. If whole dataset can be fit on a single device there is no need for parallelization. Additionally, one should take into account things like internet transfer speed, network reliability etc. Those costs may outweigh benefits.
In general, go for data parallelization if you have lots of of data (say ImageNet with 1.000.000 images) or big samples (say images 2000x2000). If possible, within a single machine as to minimize between-machines transfer. Distribute model only if there is no way around it (e.g. it doesn't fit on GPU). Don't otherwise (there is little to no point to parallelize when training MNIST as the whole dataset will easily fit in RAM and the read will be fastest from it).

why bother build custom ML-specific hardware such as TPUs?

CPUs are not the best suited for highly parallel computations (e.g. matrices multiplication) + CPU may be occupied with many other tasks (like data loading), hence it makes sense to use GPU.
As GPU was created with graphics in mind (so algebraic transformation), it can take some of CPU duties and can be specialized (many more cores when compared to CPU but simpler ones, see V100 for example).
Now, TPUs are tailored specificially for tensor computations (so deep learning mainly) and originated in Google, still WIP when compared to GPUs. Those are suited for certain types of models (mainly convolutional neural networks) and can bring speedups in this case. Additionally one should use the largest batches with this device (see here), best to be divisible by 128. You can compare that to NVidia's Tensor Cores technology (GPU) where you are fine with batches (or layer sizes) divisible by 16 or 8 (float16 precision and int8 respectively) for good utilization (although the more the better and depends on number of cores, exact graphic card and many other stuff, see some guidelines here).
On the other hand, TPUs support still isn't the best, although two major frameworks support it (tensorflow officially, while PyTorch with torch_xla package).
In general, GPU is a good default choice in deep learning right now, TPUs for convolution heavy architectures, though might give some headache tbh. Also (once again thanks @Daniel), TPUs are more power effective, hence should be cheaper when comparing single floating point operation cost.


Instruction: Index multidimensional torch tensor by another multidimensional tensor

Input:
I have a tensor x in pytorch let's say of shape (5,3,2,6) and another tensor idx of shape (5,3,2,1) which contain indices for every element in first tensor. I want a slicing of the first tensor with the indices of the second tensor. I tried x= x[idx] but I get a weird dimensionality when I really want it to be of shape (5,3,2) or (5,3,2,1).
I'll try to give an easier example:
Let's say
x=torch.Tensor([[10,20,30],
                 [8,4,43]])
idx = torch.Tensor([[0],
                    [2]])

I want something like
y = x[idx]

such that 'y' outputs [[10],[43]] or something like.
The indices represent the position of the wanted elements the last dimension. for the example above where x.shape = (2,3) the last dimension are the columns, then the indices in 'idx' is the column. I want this but for more than 2 dimensions

Output:
From what I understand from the comments, you need idx to be index in the last dimension and each index in idx corresponds to similar index in x (except for the last dimension). In that case (this is the numpy version, you can convert it to torch):
ind = np.indices(idx.shape)
ind[-1] = idx
x[tuple(ind)]

output:
[[10]
 [43]]


Instruction: How can I fix the “TypeError: 'Tensor' object is not callable” error in Pytorch?

Input:
I am trying to compute a linear function an image's pixels, followed by log softmax (it's for a classification task). I am not sure how to do this without getting errors. Here is the relevant code:
...
torch.nn.functional.nll_loss(output, target) # error happens here
...

def __init__(self):
    super(NetLin, self).__init__()
    self.in_out = torch.nn.Linear(28, 2)

def forward(self, input):
    out_sum = self.in_out(input)
    output = torch.nn.LogSoftmax(out_sum)
    return output

and the full error message I get is:
Traceback (most recent call last):
  File "copy.py", line 98, in <module>
    main()
  File "copy.py", line 94, in main
    train(args, net, device, train_loader, optimizer, epoch)
  File "copy.py", line 21, in train
    loss = torch.nn.functional.nll_loss(output, target)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/functional.py", line 2107, in nll_loss
    dim = input.dim()
TypeError: 'Tensor' object is not callable

I have tried a few different solutions to this based on other answers online but they just result in different error messages. Clearly I am doing something fundamentally wrong here but I haven't used Pytorch before so I'm not sure what it is. Thank you
Edit:
My code is now:
def train(args, model, device, train_loader, optimizer, epoch):
    if args.net == 'lin':
        model = NetLin()
    model.train()
    loss = nn.NLLLoss()
    for batch_idx, (data, target) in enumerate(train_loader):
        data.requires_grad = True
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = loss(model(input), target)
        F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

class NetLin(nn.Module):
    def __init__(self):
        super(NetLin, self).__init__()
        self.in_out = torch.nn.Linear(28 * 28, 2)

    def forward(self, input):
        input = input.view(-1, 28 * 28)
        out_sum = self.in_out(input)
        output = torch.nn.LogSoftmax(out_sum, dim=1)
        return output

and my error message is now:
Traceback (most recent call last):
  File "copy.py", line 102, in <module>
    main()
  File "copy.py", line 98, in main
    train(args, net, device, train_loader, optimizer, epoch)
  File "copy.py", line 24, in train
    output = loss(model(input), target)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/.../copy.py", line 15, in forward
    input = input.view(-1, 28 * 28)
AttributeError: 'builtin_function_or_method' object has no attribute 'view'

As you can kind of see the data and target are read in from a file (they are from KMNIST actually) so I can't control their format exactly, but I do know the image sizes are all [1,28,28], i.e. a 28*28 greyscale image. Also the batch size is 64 in case that matters.

Output:
Did you remember to set your model to training mode in your train loop with model.train()? Also, nll_loss takes in 2 tensors, but the first entry (the input tensor) needs to have requires_grad=True before it goes through the model, which is also why you need to set model.train() before training.
So you would have something like this:
model = NetLin()
model.train()
loss = nn.NLLLoss()
input = torch.randn(7, 4, requires_grad=True) # your input image (tensor)
target = torch.tensor([1, 0]) # image label for image belonging to first class
output = loss(model(input), target)

I am also a bit concerned about your self.in_out = torch.nn.Linear(28, 2). This says that your linear layer is expecting 28 features, implying that your input images are either 7x4, 14x2 or 28x1, which doesn't seem right in my opinion? Aren't you using images of size 28x28 (very typical size in this context)? In which case, you would have your linear layer modified as self.in_out = torch.nn.Linear(28*28, 2), and your forward pass will have to be modified as follows:
def forward(self, input):
    input = input.view(-1, 28*28)
    out_sum = self.in_out(input)
    output = torch.nn.LogSoftmax(out_sum)
    return output


Instruction: model.parameters() not updating in Linear Regression with Pytorch

Input:
I'm a newbie in Deep Learning with Pytorch. I am using the Housing Prices dataset from Kaggle here. I tried sampling with first 50 rows. But the model.parameters() is not updating as I perform the training. Can anyone help?
import torch
import numpy as np
from torch.utils.data import TensorDataset
import torch.nn as nn
from torch.utils.data import DataLoader
import torch.nn.functional as F

inputs = np.array(label_X_train[:50])
targets = np.array(train_y[:50])

# Tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
targets = targets.view(-1, 1)
train_ds = TensorDataset(inputs, targets)
batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

model = nn.Linear(10, 1)
# Define Loss func
loss_fn = F.mse_loss
# Optimizer
opt = torch.optim.SGD(model.parameters(), lr = 1e-5)

num_epochs = 100
model.train()
for epoch in range(num_epochs):
    # Train with batches of data
    for xb, yb in train_dl:

        # 1. Generate predictions
        pred = model(xb.float())

        # 2. Calculate loss
        loss = loss_fn(pred, yb.float())

        # 3. Compute gradients
        loss.backward()

        # 4. Update parameters using gradients
        opt.step()

        # 5. Reset the gradients to zero
        opt.zero_grad()

    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch +
                                                   1, num_epochs,
                                                   loss.item()))

Output:
The weight does update, but you weren't capturing it correctly. model.weight.data is a torch tensor, but the name of the variable is just a reference, so setting w = model.weight.data does not create a copy but another reference to the object. Hence changing model.weight.data would change w too.
So by setting w = model.weight.data and w_new = model.weight data in different part of the loops means you're assigning two reference to the same object making their value equal at all time.
In order to assess that the model weight are changing, either print(model.weight.data) before and after the loop (since you got one linear layer of 10 parameters it's still okay to do that) or simply set w = model.weight.data.clone(). In that case your output will be:
tensor([[False, False, False, False, False, False, False, False, False, False]])

Here's an example that shows you that your weights are changing:
import torch
import numpy as np
from torch.utils.data import TensorDataset
import torch.nn as nn
from torch.utils.data import DataLoader
import torch.nn.functional as F

inputs = np.random.rand(50, 10)
targets = np.random.randint(0, 2, 50)

# Tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
targets = targets.view(-1, 1)
train_ds = TensorDataset(inputs, targets.squeeze())
batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

model = nn.Linear(10, 1)
# Define Loss func
loss_fn = F.mse_loss
# Optimizer
opt = torch.optim.SGD(model.parameters(), lr = 1e-1)

num_epochs = 100
model.train()
w = model.weight.data.clone()
for epoch in range(num_epochs):
    # Train with batches of data
    for xb, yb in train_dl:

        # 1. Generate predictions
        pred = model(xb.float())

        # 2. Calculate loss
        loss = loss_fn(pred, yb.float())

        # 3. Compute gradients
        loss.backward()

        # 4. Update parameters using gradients
        opt.step()

        # 5. Reset the gradients to zero
        opt.zero_grad()

    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch +
                                                   1, num_epochs,
                                                   loss.item()))
print(w == model.weight.data)


Instruction: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm

Input:
I recently have access to a GPU and have tried to run my code but there is an error. I've read about adding .cuda() to the layers could help but I have ran similar code without the "qml" part and it worked just fine without changing the layer (with cuda I mean). For the sake of brevity, I only added the class for the neural network and the entire loop of the code. The missing part of the code is for other aspects. But if there is a need for the rest of the code, I would be happy to include everything.
Code:
class DQN(nn.Module):

    def __init__(self, img_height, img_width):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(in_features=img_height * img_width * 3, out_features=12)
        self.fc2 = nn.Linear(in_features=12, out_features=8)
       # self.fc3 = nn.Linear(in_features=10, out_features=8)
        self.clayer_in = torch.nn.Linear(in_features=8, out_features=wires)
        self.clayer_out = torch.nn.Linear(wires, out_dim)

        dev = qml.device('strawberryfields.fock', wires=wires, cutoff_dim=3)
        self.layer_qnode = qml.QNode(layer, dev)

        weights = qml.init.cvqnn_layers_all(n_quantum_layers, wires)
        weight_shapes = {"w{}".format(i): w.shape for i, w in enumerate(weights)}

        self.qlayer = qml.qnn.TorchLayer(self.layer_qnode, weight_shapes)

    def forward(self, t):
        t = self.flatten(t)
        t = self.fc1(t)
        t = self.fc2(t)
       # t = self.fc3(t)
        t = self.clayer_in(t)
        t = self.qlayer(t)
        t = self.clayer_out(t)
        t = t.sigmoid()
        return t

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
#print(device)
em = CartPoleEnvManager(device)
strategy = EpsilonGreedyStrategy(eps_start, eps_end, eps_decay)
agent = Agent(strategy, em.num_actions_available(), device)
memory = ReplayMemory(memory_size)
#learning_rate = LearningRate(lr_start,lr_end,lr_decay)
#learn = lr(learning_rate)

policy_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net = DQN(em.get_screen_height(), em.get_screen_width()).to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval() #tells pytorch that target_net is only used for inference, not training
optimizer = optim.Adam(params=policy_net.parameters(), lr=0.01)

i = 0
episode_durations = []
for episode in range(num_episodes): #iterate over each episode
    program_starts = time.time()
    em.reset()
    state = em.get_state()

    for timestep in count():
        action = agent.select_action(state, policy_net)
        reward = em.take_action(action)
        next_state = em.get_state()
        memory.push(Experience(state, action, next_state, reward))
        state = next_state
        #i+=1
        #print(i)
        if memory.can_provide_sample(batch_size):
            scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)
            experiences = memory.sample(batch_size)
            states, actions, rewards, next_states = extract_tensors(experiences)

            current_q_values = QValues.get_current(policy_net, states, actions)
            next_q_values = QValues.get_next(target_net, next_states) #will get the max qvalues of the next state, q values of next state are used via next state
            target_q_values = (next_q_values * gamma) + rewards

            loss = F.mse_loss(current_q_values, target_q_values.unsqueeze(1))
            optimizer.zero_grad() # sets the gradiesnt of all weights n biases in policy_net to zero
            loss.backward() #computes gradient of loss with respect to all weights n biases in the policy net
            optimizer.step() # updates the weights n biases with the gradients that were computed form loss.backwards
            scheduler.step()
        if em.done:
            episode_durations.append(timestep)
            plot(episode_durations, 100)
            break
    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())
    now = time.time()
    print("Episode hat {0} Sekunden gedauert".format(now - program_starts))

em.close()

And the error:
Traceback (most recent call last):
  File "qdqn.py", line 328, in <module>
    loss.backward() #computes gradient of loss with respect to all weights n biases in the policy net
  File "/home/ubuntu/anaconda3/envs/gymm/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/ubuntu/anaconda3/envs/gymm/lib/python3.8/site-packages/torch/autograd/__init__.py", line 98, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'mat2' in call to _th_mm

Output: The solution is that the python package "pennylane" doesn't yet have the resources built out to work on GPUs. Since Pennylane is a relatively new package, this is something they plan to build out in the future.


Instruction: Pytorch: Differentiable counting

Input: I need to count the number of elements in a tensor that satisfy a condition, such as counting the number of people with age == 60, or people with age >= 50. Is there a differentiable approximation to the counting function?

Output:
Use torch.Tensor.sum or torch.sum on boolean tensor returned by age >= 50
age = torch.arange(75)

(age >= 50).sum()
tensor(25)


Instruction: PyTorch TypeError: 'ToTensor' object is not iterable

Input:
I'm trying to print an image name at each iteration. However, I'm getting an error TypeError: 'ToTensor' object is not iterable. Please could some advise where I am going wring please? Many Thanks
from torchvision import datasets
import torch.utils.data
from torch.utils.data import DataLoader
from torchvision import transforms
from dataset2 import CellsDataset
from torchvision import datasets
import torch
import torchvision
import torchvision.transforms as transforms

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

# override the __getitem__ method. this is the method that dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
    # the image file path
    path = self.imgs[index][0]
    # make a new tuple that includes original and the path
    tuple_with_path = (original_tuple + (path,))
    return tuple_with_path

# EXAMPLE USAGE:
# instantiate the dataset and dataloader
data_dir = "/Users/nubstech/Documents/GitHub/CellCountingDirectCount/Eddata/"
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
#dataloader = DataLoader(dataset)
transform = transforms.Compose([
    # you can add other transformations in this list
    transforms.ToTensor()
])
dataset = DataLoader(data_dir, transforms.Compose(transforms.ToTensor()))
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
   print(inputs, labels, paths)

Traceback Message:
Traceback (most recent call last):
  File "file_location2.py", line 37, in <module>
    dataset = DataLoader(data_dir, transforms.Compose(transforms.ToTensor()))
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 219, in __init__
    batch_sampler = BatchSampler(sampler, batch_size, drop_last)
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 190, in __init__
    "but got batch_size={}".format(batch_size))
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 66, in __repr__
    for t in self.transforms:
TypeError: 'ToTensor' object is not iterable

Output:
It's because transforms.Compose() needs to be a list (probably some other iterables are accepted too). The problem is here:
dataset = DataLoader(data_dir, transforms.Compose(transforms.ToTensor()))

Try:
transforms = transforms.Compose([transforms.ToTensor()])

This will create a callable in which you can pass your data.


Instruction: volatile was removed and now has no effect. Use with torch.no_grad(): instead

Input:
I have got this error, can you help me please?
This is my error in utils.py file
 content/gdrive/My Drive/DeepFakeDetection/utils.py:21: UserWarning:
 volatile was removed and now has no effect. Use with torch.no_grad():
 instead. return Variable(x, volatile=volatile)

And this is my code in utils.py file
 def to_var(x, volatile=False):

     if torch.cuda.is_available():
        x = x.cuda()
     return Variable(x, volatile=volatile)

Thank you

Output:
I changed it to this:
def to_var(x, volatile=False):
>
>     with torch.no_grad():
>
>         if torch.cuda.is_available():
>             x = x.cuda()
>     return x


Instruction: PyTorch running out of memory: DefaultCPUAllocator can't allocate memory

Input:
I'm trying to optimize some weighs (weigts) in Pytorch but I keep getting this error:

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 8000000000000 bytes. Error code 12 (Cannot allocate memory).

Namely, things blow up when I run (weights * col).sum() / weights.sum(). Weights is a tensor of size (1000000,1) and col is also a tensor of size (1000000, 1). Both tensors are decently sized, but it seems odd that I'm using up all the memory in my computer (8GB) for these operations.

Output: It could be that your weights and col tensors are not aligned (i.e. one of them is transposed so it is (1,1000000) instead of (1000000,1). Then when you do (weights * col) the shapes are broadcast together and it makes a tensor that is (1000000,1000000) which is probably where you are getting the extreme memory usage (as the resulting tensor is 1000000 times bigger than your original tensor).


Instruction: GPU out of memory on evaluation : Pytorch

Input:
The model trains fine when I only train and don't validate, however it runs out of memory during evaluation, but I don't understand why this might be a problem especially since I am using torch.no_grad() any ideas?
def test(epoch,net,testloader,optimizer):
    net.eval()
    test_loss = 0
    correct = 0
    total = 0
    idx = 0
    features_all = []
    for batch_idx, (inputs, targets) in enumerate(testloader):
        with torch.no_grad():
            idx = batch_idx
            # inputs, targets = inputs.cpu(), targets.cpu()
            if use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()
            inputs, targets = Variable(inputs), Variable(targets)
            save_features, out, ce_loss = net(inputs,targets)
            test_loss += ce_loss.item()
            _, predicted = torch.max(out.data, 1)
            total += targets.size(0)
            correct += predicted.eq(targets.data).cpu().sum().item()
            features_all.append((save_features, predicted, targets.data))
    test_acc = 100.*correct/total
    test_loss = test_loss/(idx+1)
    logging.info('test, test_acc = %.4f,test_loss = %.4f' % (test_acc,test_loss))
    print('test, test_acc = %.4f,test_loss = %.4f' % (test_acc,test_loss))
    return features_all, test_acc

Output:
features_all.append((save_features, predicted, targets.data))

This line is saving references to tensors in GPU memory and so the CUDA memory won't be released when loop goes to next iteration (which eventually leads to the GPU running out of memory). Move the tensors to CPU (using .cpu()) while saving them.


Instruction: Non-intersection of two n-dimentional pytorch tensors

Input:
Thanks everyone in advance for your help! What I'm trying to do in PyTorch is to compute non-intersection ( lets call it torch.nonintersection) for tensors of many dimenstions (without for loops because I want it to be effectively performed on GPU). So here is the example how it should work:
a = torch.tensor([[ 0.,  0.], [ 0.,  1.], [ 0.,  2.], [ 1.,  0.], [ 1.,  1.], [ 1.,  2.], [ 1.,  3.],
                  [ 2.,  0.], [ 2.,  1.], [ 2.,  2.]])
b = torch.tensor([[ 2.,  0.], [ 2.,  1.], [ 2.,  2.], [ 1.,  0.], [ 1.,  1.], [ 1.,  2.], [ 1.,  3.]])

torch.spec_unique(a,b) = torch.tensor([ 0.,  0.], [ 0.,  1.], [ 0.,  2.])

I have analogs with for loops, but they take too much time right now. Any Ideas how it can be done? Much appreciated!

Output:
Shape of a and b is different in your case (a is (10, 2), while b is (7, 2)), so you have to "cut" the bigger tensor to the size of the smaller one. Function below handles it with simple if (it won't slow down your computations):
def non_intersection(a, b):
    if a.shape[0] > b.shape[0]:
        return torch.nonzero(a[: b.shape[0]] != b, as_tuple=False)
    return torch.nonzero(b[: a.shape[0]] != a, as_tuple=False)

This would return:
tensor([[0, 0],
        [1, 0],
        [2, 0]])

So the columns are in opposite order. If you wish to get columns just like in your example you can do this on non_intersection output:
torch.index_select(non_intersection(a, b), 1, torch.tensor([1, 0])))


Instruction: How do i iterate over pytorch 2d tensors?

Input:
X = np.array([
    [-2,4,-1],
    [4,1,-1],
    [1, 6, -1],
    [2, 4, -1],
    [6, 2, -1],
    ])

for epoch in range(1,epochs):
    for i, x in enumerate(X):

X = torch.tensor([
    [-2,4,-1],
    [4,1,-1],
    [1, 6, -1],
    [2, 4, -1],
    [6, 2, -1],
    ])

The looping was fine when it was an numpy array. But i want to work with pytorch tensors so what's alternative to enumerate or How can i loop through the above tensor in the 2nd line?

Output:
enumerate expects an iterable, so it works just fine with pytorch tensors too:
X = torch.tensor([
    [-2,4,-1],
    [4,1,-1],
    [1, 6, -1],
    [2, 4, -1],
    [6, 2, -1],
    ])

for i, x in enumerate(X):
    print(x)
    tensor([-2,  4, -1])
    tensor([ 4,  1, -1])
    tensor([ 1,  6, -1])
    tensor([ 2,  4, -1])
    tensor([ 6,  2, -1])

If you want to iterate over the underlying arrays:
for i, x in enumerate(X.numpy()):
    print(x)
    [-2  4 -1]
    [ 4  1 -1]
    [ 1  6 -1]
    [ 2  4 -1]
    [ 6  2 -1]

Do note however that pytorch's underlying data structure are numpy arrays, so you probably want to avoid looping over the tensor as you're doing, and should be looking at vectorising any operations either via pytorch or numpy.


Instruction: What does it mean for the shape of an image tensor to be (64, 64)? Does that mean there are no channels?

Input:
I was exploring the images of the dataset : Tiny Imagenet and I found that most of the image tensors have shapes (64, 64, 3) i.e. images of height and width 64 and 64 respectively and three channels for Red, Green and Blue. But some of the image tensors in the dataset have shape (64, 64).

Does it mean that there are no channels? How is it possible?
EDIT - I downloaded this image with tensor shape (64, 64):

EDIT - Is it possible that it has one channel of some other colour space like Lab? (here b refers to the colours from blue to yellow).

Output: Usually, it is equivalent to (64, 64, 1) but most libraries collapse the last axis. It means you have only one channel, probably a grayscale image. Is it possible that your dataset is mixed between RGB and grayscale images?


Instruction: How to save a model in PyTorch?

Input:
Say we have a model that we can validate like this and that Model inherits torch.nn.Module
def validate(logger, config, valid_loader, model, criterion, epoch, main_proc):
    meters = AverageMeterGroup()
    model.eval()

    with torch.no_grad():
        for step, (x, y) in enumerate(valid_loader):
            x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
            logits, _ = model(x)
            loss = criterion(logits, y)
            prec1, prec5 = utils.accuracy(logits, y, topk=(1, 5))
            metrics = {"prec1": prec1, "prec5": prec5, "loss": loss}
            metrics = utils.reduce_metrics(metrics, config.distributed)
            meters.update(metrics)

            if main_proc and (step % config.log_frequency == 0 or step + 1 == len(valid_loader)):
                logger.info("Epoch [%d/%d] Step [%d/%d]  %s", epoch + 1, config.epochs, step + 1, len(valid_loader), meters)

    if main_proc:
        logger.info("Train: [%d/%d] Final Prec@1 %.4f Prec@5 %.4f", epoch + 1, config.epochs, meters.prec1.avg, meters.prec5.avg)
    return meters.prec1.avg, meters.prec5.avg

How to change validation procedure in order to save model after validation into file system so that it would be runnable on other data?

Output:
with torch.no_grad():
    for step, (x, y) in enumerate(valid_loader):
        x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
        logits, _ = model(x)
        loss = criterion(logits, y)
        prec1, prec5 = utils.accuracy(logits, y, topk=(1, 5))
        metrics = {"prec1": prec1, "prec5": prec5, "loss": loss}
        metrics = utils.reduce_metrics(metrics, config.distributed)
        meters.update(metrics)

        if main_proc and (step % config.log_frequency == 0 or step + 1 == len(valid_loader)):
            logger.info("Epoch [%d/%d] Step [%d/%d]  %s", epoch + 1, config.epochs, step + 1, len(valid_loader), meters)

torch.save(model,'model'+str(epoch)+'.pt')
if main_proc:
    logger.info("Train: [%d/%d] Final Prec@1 %.4f Prec@5 %.4f", epoch + 1, config.epochs, meters.prec1.avg, meters.prec5.avg)
return meters.prec1.avg, meters.prec5.avg


Instruction: Is there a PyTorch with CUDA Unified GPU-CPU Memory fork?

Input: So Training a DNN model can be a pain when a batch of one image takes 15GB. Speed is not so important for me, yet to fit bigger batches (and models is). So I wonder if there is a PyTorch with CUDA Unified Memory fork or something like that to fit giant models (having 16gb per GPU RAM, yet 250 on CPU side it seems quite resonable)?

Output:
If you do not care about the time it takes, but need large batches, you can use a more slow approach. Say you need to have a batch of 128 samples but your gpu memory fits only 8 samples. You can create smaller batches of 8 samples and then average their gradients.
For each small batch of 8 samples that you evaluate, you keep the .grad of each parameter in your cpu memory. You keep a list of grads for each of your models parameters. After you have gathered the grads for 16 batches of 8 samples (128 samples in total) you can average the gradients of each parameter and put the result back into the .grad attribute of each parameter.
You can then call the .step() of your optimizer. This should yield exactly the same results as if you were using a large batch of 128 samples.


Instruction: PyTorch Lightning move tensor to correct device in validation_epoch_end

Input:
I would like to create a new tensor in a validation_epoch_end method of a LightningModule. From the official docs (page 48) it is stated that we should avoid direct .cuda() or .to(device) calls:

There are no .cuda() or .to() calls. . . Lightning does these for you.

and we are encouraged to use type_as method to transfer to the correct device.
new_x = new_x.type_as(x.type())
However, in a step validation_epoch_end I do not have any tensor to copy device from(by type_as method) in a clean way.
My question is what should I do if I want to create a new tensor in this method and transfer it to the device where is the model?
The only thing I can think of is to find a tensor in the outputs dictionary but it feels kinda messy:
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
output = self(self.__test_input.type_as(avg_loss))

Is there any clean way to achieve that?

Output:
did you check part 3.4 (page 34) in the doc you linked ?

LightningModules know what device they are on!  construct tensors on the device directly to avoid CPU->Device transfer

t = tensor.rand(2, 2).cuda()# bad
(self is lightningModule)t = tensor.rand(2,2, device=self.device)# good

I had a similar issue to create tensors this helped me. I hope it will help you too.


Instruction: How to use with torch.cuda.device() conditionally

Input:
I have some code encapsulated in a:
with torch.cuda.device(self.device):
    # do a bunch of stuff

And in my __init__ I have:
self.device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')

But I'm a little confused about how to deal with a situation where the device is cpu. Because torch.cuda.device is already explicitly for cuda. Should I just write a decorator for the function? Seems a bit overkill

Output:
According to the documentation for torch.cuda.device

device (torch.device or int) – device index to select. It’s a no-op if this argument is a negative integer or None.

Based on that we could use something like
with torch.cuda.device(self.device if self.device.type == 'cuda' else None):
    # do a bunch of stuff

which would simply be a no-op when self.device isn't a CUDA device.


Instruction: save to csv - numpy.ndarray' object has no attribute 'append'

Input:
I'm trying to append the filenames of the images during training into a csv file, but getting an error. I can print the values okay but cannot append to csv file for some reason. Here is the full code:
  File "train_filename.py", line 140, in <module>
    names_preds.append() + '\n'
AttributeError: 'numpy.ndarray' object has no attribute 'append'

        with open('preds_base_model_teste1016.csv','a') as fd:
            #dict_writer = csv.writer(fd)
            #dict_writer.writerow('Target' + '\n')
            #dict_writer.writerow( ','.join(map(str, preds.detach().tolist())) + '\n')
            #dict_writer.writerow('Prediction' + '\n')
            #dict_writer.writerow( ','.join(map(str, targets.detach().tolist())) + '\n')
            #fd.write('Target' + '\n')
            fd.write(','.join(map(str, preds.detach().tolist())) + '\n')
            #fd.write('Prediction' + '\n')
            fd.write( ','.join(map(str, targets.detach().tolist())) + '\n')
            #fd.write('Image_Name' + '\n')
            np.append(paths)
            #fd.write([filename])

Output:
Use concatenate instead of append as described in the docs.
It does look like you are trying to add nothing to the array, however.
You should use it like this:
names_preds.concatenate(some_value) where the value inside concatenate is what you want to add to the array.
To save a numpy array to a csv file, there is an easier way:
numpy.savetxt("foo.csv", names_preds, delimiter=",")

See details in this answer.
Replace your line:
np.append(paths)

with
with open('preds_base_model_teste1016.csv','a') as fd:
    np.savetxt(fd, names_preds, delimiter=",")

That will append all of the paths to the file you already specified.
Update:
Per your question in the comment, to put multiple arrays together so you end up with columns, you could try:
with open('preds_base_model_teste1016.csv','a') as fd:
    np.savetxt(fd, list(zip(a,b,c)), delimiter=",")

where a, b, and c are the arrays you want as columns.


Instruction: How can I install Auto-PyTorch on Windows 10 from requirements.txt?

Input:
I have been trying to install Auto-PyTorch, an automatic Neural Network tuning system (more info about installation here: https://github.com/automl/Auto-PyTorch), in a Windows 10 system. The installation steps are as follow:
$ cd install/path
$ git clone https://github.com/automl/Auto-PyTorch.git
$ cd Auto-PyTorch
$ cat requirements.txt | xargs -n 1 -L 1 pip install
$ python setup.py install

However, I can't get around the following line of code:
cat requirements.txt | xargs -n 1 -L 1 pip install

Once I get there, the following Windows PowerShell error is raised:
xargs : The term 'xargs' is not recognized as the name of a cmdlet, function, script file, or

operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:86
+ ... read.CurrentUICulture = 'en-US'; cat requirements.txt | xargs -n 1 -L ...
+                                                             ~~~~~
    + CategoryInfo          : ObjectNotFound: (xargs:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Note that I passed the following line of code so that the error is thrown in English (I am Spanish):
[Threading.Thread]::CurrentThread.CurrentUICulture = 'en-US'; cat requirements.txt | xargs -n 1 -L 1 pip install

I'm not an expert on PowerShell, so I would greatly appreciate your help.
Best regards!

Output:
Since you're running on Windows PowerShell, only command-line utilities natively available on Windows can be assumed to be available - and xargs, a Unix utility, is not among them.
(While git also isn't natively available, it looks like you've already installed it).
Here's a translation of your code into native PowerShell code (note that cd is a built-in alias for Set-Location, and, on Windows only, cat is a built-in alias for Get-Content; % is a built-in alias for the ForEach-Object cmdlet):
Set-Location install/path
git clone https://github.com/automl/Auto-PyTorch.git
Set-Location Auto-PyTorch
Get-Content requirements.txt | % { pip install $_ }
python setup.py install


Instruction: pytorch KLDivLoss loss is negative

Input:
my target is training a span prediction model
which can predict the position in the BERT output sequence
my input's shape is (batch_size, max_sequence_len(512),embedding_size(768))
output's shape will be (batch_size , max_sequence_len , 1) and the third dim is stand for kind a probability , then I will reshape output to (batch_size,max_sequence_len)
my label's shape is (batch_size , max_sequence_len), and in the max_sequence_len(512), only one position will be 1 and the others will be zero
and I've already check this
(batch_size is 2)
start_pos_labels.sum(dim=1)
output >>
tensor([1.0000, 1.0000], device='cuda:0', dtype=torch.float64)

start_pred.sum(dim=1)
tensor([1., 1.], device='cuda:0', dtype=torch.float64, grad_fn=<SumBackward1>)

but when I use nn.KLDivLoss() , output still be negative, I really dont know why
can somebody help me? thanks!
Here is my code
Model Code
class posClassfication_new(nn.Module):
    def __init__(self):
        super(posClassfication_new, self).__init__()
        self.start_task = nn.Sequential(
            nn.Linear(768, 1),
#             nn.ReLU(),
#             nn.Linear(256, 128),
#             nn.ReLU(),
#             nn.Linear(128, 1)
        )
        self.end_task = nn.Sequential(
            nn.Linear(768, 1),
#             nn.ReLU(),
#             nn.Linear(256, 128),
#             nn.ReLU(),
#             nn.Linear(128, 1)
        )
#
    def forward(self, start_x,end_x):
        start_x = start_x.double()
        end_x = end_x.double()

        start_out = self.start_task(start_x)
        end_out = self.end_task(end_x)

        return start_out,end_out

training code
BATCH_SIZE = 8
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("device:", device)

class PosTrainDataset(Dataset):
    def __init__(self, x, start_y,end_y):
        self.x = x
        self.start_y = start_y
        self.end_y = end_y

    def __getitem__(self,idx):
        x = self.x[idx]
        start_y = self.start_y[idx]
        end_y = self.end_y[idx]
        return x, start_y, end_y

    def __len__(self):
        return len(self.x)

trainset = PosTrainDataset(pos_train_x , start_pos_labels_train , end_pos_labels_train)
trainloader = DataLoader(trainset, batch_size=BATCH_SIZE)

pos_model = posClassfication_new()
pos_model = pos_model.to(device)
pos_model = pos_model.double()
pos_model.train()

pos_loss = nn.KLDivLoss()
# pos_loss = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(pos_model.parameters(), lr=1e-5)

EPOCHS = 5
for epoch in range(EPOCHS):
    running_loss = 0.0
    for data in trainloader:
        x, start_pos_labels, end_pos_labels = [t.to(device) for t in data]
        mini_batch = x.size()[0]
        optimizer.zero_grad()

        start_pred , end_pred = pos_model(x,x)

        start_pred = start_pred.reshape((mini_batch,512))
        end_pred = end_pred.reshape((mini_batch,512))

        start_pred = torch.nn.functional.softmax(start_pred,dim=1)
        end_pred = torch.nn.functional.softmax(end_pred,dim=1)

        start_pos_labels = start_pos_labels + 0.0001
        start_pos_labels = torch.nn.functional.softmax(start_pos_labels,dim=1)

        end_pos_labels = end_pos_labels  + 0.0001
        end_pos_labels = torch.nn.functional.softmax(end_pos_labels,dim=1)

#         start_pos_labels = torch.argmax(start_pos_labels,dim=1)
#         end_pos_labels = torch.argmax(end_pos_labels,dim=1)

        start_loss = pos_loss(start_pred,start_pos_labels)
        end_loss = pos_loss(end_pred,end_pos_labels)

        loss = start_loss + end_loss
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

    torch.save(pos_model,'pos_model_single_task.pkl')
    print('[epoch %d] loss: %.3f' %(epoch + 1, running_loss))

Output:
nn.KLDivLoss expects the input to be log-probabilties.
From the documentation:

As with NLLLoss, the input given is expected to contain log-probabilities and is not restricted to a 2D Tensor. The targets are given as probabilities (i.e. without taking the logarithm).

You can apply nn.functional.log_softmax to your predictions to get log-probabilities.
start_pred = torch.nn.functional.log_softmax(start_pred,dim=1)
end_pred = torch.nn.functional.log_softmax(end_pred,dim=1)


Instruction: Extract tensor from string

Input:
Is it possible to extract directly the tensor included in this string tensor([-1.6975e+00,  1.7556e-02, -2.4441e+00, -2.3994e+00, -6.2069e-01])? I'm looking for some tensorflow or pytorch function that can do it, like the ast.literal_eval function does for dictionaries and lists.
If not, could you provide a pythonic method, please?
I'm thinking about something like this:
tensor_list = "tensor([-1.6975e+00,  1.7556e-02, -2.4441e+00, -2.3994e+00, -6.2069e-01])"
str_list = tensor_list.replace("tensor(", "").replace(")", "")
l = ast.literal_eval(str_list)
torch.from_numpy(np.array(l))

But I'm not sure this is the best way.

Output:
You can use eval:
import torch.tensor as tensor

eval(tensor_list)
>>> tensor([-1.6975,  0.0176, -2.4441, -2.3994, -0.6207])


Instruction: How do I use torchaudio with torch_xla on google colab TPU?

Input:
I was using google colab (GPU enabled) to train my Automatic Speech Recognition Model based on pytorch and torchaudio. But when I tried to use google colab TPU I got the following error when training my model :
ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceISt7complexIfEEEPKNS_6detail12TypeMetaDataEv site:stackoverflow.com

I'm almost sure it's a conflict between torchaudio and pytorch_xla versions
Here's the code that I use to install the librairies :
VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!pip install torchvision==0.6.1
!pip install torch==1.5.1
!pip install torchaudio==0.5.1
!python pytorch-xla-env-setup.py --version $VERSION

The solution proposed here https://stackoverflow.com/a/60929133/13847989 generated the following error :
ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

What am I doing wrong ?

Output: Can you try installing torchaudio from source after running the setup script (https://github.com/pytorch/audio#from-source)? Sounds like its due to a version mismatch (nightly torch vs stable torchaudio).


Instruction: How to access individual data points of MNIST data and check their size, shape etc from trainset object using torchvision's import dataset

Input:
the following code is
    from torchvision import datasets, transforms
    trainset = datasets.MNIST('./data/', download=True, train=True, transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))]))

I would like to visualize the first data point in the trainset variable above.
I want to have a look at the pixel values of the first data point by doing something like print(trainset[0]) or check the size by doing print(trainset[0].size) or check the shape by doing print(trainset[0].shape) etc.

Output:
For shape:
trainset.data.shape

torch.Size([60000, 28, 28])

For the first example:
trainset.data[0]

tensor([[[0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         ...,
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0]]], dtype=torch.uint8)


Instruction: OSError: [E050] Can't find model 'de'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory

Input:
So I am trying to make a seq to seq model for translating german to english using pytorch on online notebook like kaggle notebook and google colab
import torch
import torch.nn as nn
import torch.optim as optim
from torchtext.datasets import Multi30k
from torchtext.data import Field, BucketIterator
import numpy as np
import spacy
import random
from torch.utils.tensorboard import SummaryWriter  # to print to tensorboard

Libraries imported, when i load dataset using the function with spacy, as below,
spacy_ger = spacy.load("de")
spacy_eng = spacy.load("en")

This error comes. :
OSError: [E050] Can't find model 'de'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
Everywhere, an explanation is given for 'en', but not for 'de'. If anyone could help with this.
Specification:
Package : Version

Spacy : 2.3.1

pytorch-crf : 0.7.0

torch : 1.5.1

torchnlp : 0.0.0.1

torchtext : 0.4.0

torchvision : 0.6.1

jupyter-tensorboard : 0.2.0

tensorboard : 2.2.2

tensorboard-plugin-wit : 1.7.0

Thanks in advance for helping.

Output:
so after whole one month, trying on other things and exploring issues and questions related to this topic, I found a way to do so,
  import spacy.cli
  spacy.cli.download("en_core_web_md")

With this method, you can use and import any spacy model, whether medium-sized or larger size datasets also, which always gives an error if you try to import the dataset using
spacy.load because it is not effective for loading datasets other then sm or smallest size datasets in Google colab or Kaggle notebook or any other online notebook.


Instruction: How to handle transforms.FiveCrop change in tensor size

Input:
I am trying to add transforms.FiveCrop to my model. I understand that this method of data augmentation adds dimensions to my tensor but I am not sure how to handle it. The documentation notes:

This transform returns a tuple of images and there may be a mismatch in the number of inputs and targets your Dataset returns. See below for an example of how to deal with this.

Example
>>> transform = Compose([
>>>    FiveCrop(size), # this is a list of PIL Images
>>>    Lambda(lambda crops: torch.stack([ToTensor()(crop) for crop in crops])) # returns a 4D tensor
>>> ])
>>> #In your test loop you can do the following:
>>> input, target = batch # input is a 5d tensor, target is 2d
>>> bs, ncrops, c, h, w = input.size()
>>> result = model(input.view(-1, c, h, w)) # fuse batch size and ncrops
>>> result_avg = result.view(bs, ncrops, -1).mean(1) # avg over crops

...But I am not sure how to implement this.
Train loop:
    for batch_idx, (data, target) in enumerate(train_loader):

        print(f'Data: {data.shape}, Target: {target.shape}')
        # Before Fivecrop: Data: torch.Size([32, 3, 224, 224]), Target: torch.Size([32])
        # After Fivecrop: Data: torch.Size([32, 5, 3, 224, 224]), Target: torch.Size([32])

        indx_target = target.clone()
        data = data.to(train_config.device)
        target = target.to(train_config.device)

        optimizer.zero_grad()
        output = model(data)
        loss = F.cross_entropy(output, target)
        loss.backward()
        optimizer.step()

Can someone help explain how this is implemented in my train loop and what I am not understanding?
Thanks

Output:
This makes sense now. Here is for anyone who needs assistance. The new code looks as such:
    for batch_idx, (data, target) in enumerate(train_loader):

        bs, ncrops, c, h, w = data.size()
        print(f'Data: {data.view(-1, c, h, w).shape}, Target: {target.shape}')

        indx_target = target.clone()
        data = data.to(train_config.device)
        target = target.to(train_config.device)

        optimizer.zero_grad()
        output = model(data.view(-1, c, h, w)) # fuse batch size and ncrops
        output_avg = output.view(bs, ncrops, -1).mean(1) # average the output over fivecrops
        loss = F.cross_entropy(output_avg, target)
        loss.backward()
        optimizer.step()

        batch_loss = np.append(batch_loss, [loss.item()])
        prob = F.softmax(output_avg, dim=1)
        pred = prob.data.max(dim=1)[1]
        correct = pred.cpu().eq(indx_target).sum()
        accuracy = float(correct) / float(len(data))
        batch_acc = np.append(batch_acc, [accuracy])


Instruction: matplotlib - segmentation fault (core dumped) while trying to plot a grid

Input:
I'm trying to plot a grid but exists with the error message -
(feature_vis.py:27730): Gdk-WARNING **: 03:23:47.745: Native Windows wider or taller than 32767 pixels are not supported
Segmentation fault (core dumped)

model_weights = './Mutual-Channel-Loss-master/model_info_best.pth'
model_features = './Mutual-Channel-Loss-master/features_best_new.pth'
weights = torch.load(model_weights, map_location = torch.device('cpu'))
features = torch.load(model_features, map_location = torch.device('cpu'))
activations = features['features']
feature_map = activations[0][0][3]
# feature_maps = np.load(path, allow_pickle = True)
# test = feature_maps[0][0][0]
fig = plt.figure(figsize = (300, 500))
grid = ImageGrid(fig, 111,  # similar to subplot(111)
                 nrows_ncols=(20, 30),  # creates 2x2 grid of axes
                 axes_pad=0.1,  # pad between axes in inch.
                 )
for ax, im in zip(grid, feature_map):
    # Iterating over the grid returns the Axes.
    im = im.to('cpu').numpy()
    ax.imshow(im, cmap = 'gray')

plt.savefig('test.png', ppi = 300)

How should I fix / Debug this?
I am using python 3.6.9 and on ubuntu 18.04

Output: The unit used by the attribute figsize is the inch. So an image of size 300x500 with dpi=300 is insanely huge (45M pixels) and mpl cannot handle that. If you want a 300 by 500 (pixels) image you have to first convert it to inches. In this case it would be something like ~1x1.7.


Instruction: No Parameters in Net class even after inheriting from nn.Module

Input:
For some reason, my Class Net has no parameters, even though i have imported everything correctly., its still showing that net has no parameters or attributes and even on printing net its just prints empty list. Can anyone tell me what's wrong ?
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
def ___init__(self):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(784, 64)
    self.fc2 = nn.Parameter(nn.Linear(64, 64))
    self.fc3 = nn.Parameter(nn.Linear(64, 64))
    self.fc4 = nn.Parameter(nn.Linear(64, 10))

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = self.fc4(x)
    return F.log_softmax(x, dim=1)

net = Net()
print(net)
print(list(net.parameters()))

Output:
Net()
[]

Output:
In the code
def ___init__(self):

you're using three underscores before init, use two: __init__.


Instruction: Cnn model using pytorch

Input:
I have images and labels. I divided them into test and train sets.
(xtrain,
ytrain,
xtest,
ytest).
x refers to images and y refers to label.
How to use these sets in the following train model
 **# Train the model
  total_step = len(train_loader)
 for epoch in range(num_epochs):
   for i, (images, labels) in enumerate(train_loader):
    images = images.to(device)
    labels = labels.to(device)

    # Forward pass
    outputs = model(images)
    loss = criterion(outputs, labels)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (i+1) % 100 == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
               .format(epoch+1, num_epochs, i+1, total_step, loss.item()))
 # Test the model
 model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch
                       mean/variance)**

Output:
from torch.utils.data import Dataset, DataLoader
training_set = Dataset(xtrain, ytrain)
test_set = Dataset(xtest, ytest)
params = {'batch_size': 64,
        'shuffle': True}
train_loader = DataLoader(training_set, **params)


Instruction: PyTorch: 'ToTensor()' turns color image into 9 grayscale pictures

Input:
I have found that when I use 'ToTensor' to a images, one image becomes 9 displayed.I checked the official documents but couldn't find the reason. so why a picture become 9 pictures???questioon as the following figure.
a = plt.imread('test.jpg')
plt.imshow(a)
plt.show()

transform = transforms.Compose([transforms.ToTensor()])
b = transform(a)
b = b.view(375,500,3)
plt.imshow(b)

Output:
When you use transforms.ToTensor(), by default it changes the input arrays from HWC to CHW order. For plotting, you'll need to push back the channels to the last dimension.
plt.imshow(b.permute(2, 0, 1))


Instruction: Tensor(1.0).item() vs float(Tensor(1.0))

Input: If x is a torch.Tensor of dtype torch.float then are the operations x.item() and float(x) exactly the same?

Output:
The operations x.item() and float(x) are not the same.
From the documentation of item(), it can be used to get the value of tensor as a Python number(only from tensors containing a single value). It basically returns the value of the tensor as it is. It does not make any modifications to the tensor.
Where as float() is for converting its input to a floating point number, when possible. Find the documentation here.
To see the difference, consider another Tensor y of dtype int64:
import torch

y = torch.tensor(2)
print(y, y.dtype)
>>> tensor(2) torch.int64

print('y.item(): {}, float(y): {}'.format(y.item(), float(y)))
>>> y.item(): 2, float(y): 2.0

print(type(y.item()), type(float(y)))
>>> <class 'int'> <class 'float'>

Note that float(y) does not convert the type in-place. You would need to assign it in case you need that change. Like:
z = float(y)
print('y.dtype: {}, type(z): {}'.format(y.dtype, type(z)))
>>> y.dtype: torch.int64, type(z): <class 'float'>

We can see that z is not a torch.Tensor. It is simply a floating point number.
The float() operation is not to be confused with self.float(). This operation performs the Tensor dtype conversion (not in-place, needs assignment).
print('y.float(): {},\n y.float().dtype: {},\n y: {},\n y.dtype'.format(y.float(), y.float().dtype, y, y.dtype))

y.float(): 2.0,
y.float().dtype: torch.float32,
y: 2,
y.dtype: torch.int64


Instruction: How to convert a ByteTensor into an image tensor?

Input:
I am converting images into binary files via matlab and try to load the binary matlab vectors (1d) into python and convert them into ByteTensors:
img = np.fromfile(dir_train + image_name)
img = torch.ByteTensor(img)

This works fine. The ByteTensors are of the dimension 2. Afterwards I would like to convert them back into into images, like reshaping them, because the neural network (resnet18) needs 4-dimensional tensors. What is the best way to do that?
At the moment my network needs tensors of the dimension [64, 3, 7, 7], but the ByteTensors are [8, 1914].

Output:
I solved it this way:
img = np.fromfile(dir_train + image_name, 'bool')    # read in the binary file
img = img.reshape(1, 350, 350)    # reshape binary file
img = torch.ByteTensor(img)    # convert to ByteTensor
img = img.type(torch.FloatTensor)    # convert to FloatTensor

I think there is no better way than typecasting the ByteTensor. On top of that I changed the first convolutional layer of my network architecture (resnet18) in order to make sure, the input expected input matches the convertet binary input.
resnet18.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)


Instruction: Applying kernels to separate dimensions in gpytorch

Input:
I am translating three-dimensional Gaussian process regression code from GPflow into GPyTorch to take advantage of PyTorch's native scalability for exact GPR.
This example is helpful, but can't find an explanation for how to apply distinct kernels to individual dimensions in a multi-dimensional GPR (e.g. a Matern52 kernel to dimensions [0,1], and a Matern 32 kernel to dimension [2]).
Does GPyTorch support that functionality, and if so how is it implemented/where can I find an example?

Output:
Use "active_dims" in the kernel functions :
from(https://docs.gpytorch.ai/en/v1.1.1/kernels.html#rbfkernel)
active_dims (tuple of ints, optional):
Set this if you want to compute the covariance of only a few input dimensions. The ints corresponds to the indices of the dimensions. Default: None.


Instruction: Creating MLP model to predict the ratings that a user will give to an unseen movie using PyTorch

Input:
For my project , i’m trying to predict the ratings that a user will give to an unseen movie, based on the ratings he gave to other movies. I’m using the movielens dataset.The Main folder, which is ml-100k contains informations about 100,000 movies.
Before processing the data, the main data (ratings data) contains user ID, movie ID, user rating from 0 to 5 and timestamps(not considered for this project).I then split the data into Training set(80%) and test data(20%) using sklearn Library.
To create the recommendation systems, the model ‘Stacked-Autoencoder’ is being used. I’m using PyTorch and the code is implemented on Google Colab. The project is based on this https://towardsdatascience.com/stacked-auto-encoder-as-a-recommendation-system-for-movie-rating-prediction-33842386338
I'm new to deep Learning and I want to compare this model(Stacked_Autoencoder) to another Deep Learning model. For Instance,I want to use Multilayer Perception(MLP). This is for research purposes. This is the code below for creating Stacked-Autoencoder model and training  the model.
### Part 1 : Archirecture of the AutoEncoder

#nn.Module is a parent class
# SAE is a child class of the parent class nn.Module
class SAE(nn.Module):
# self is the object of the SAE class

# Archirecture
    def __init__(self, ):
    # self can use alll the methods of the class nn.Module
        super(SAE,self).__init__()
    # Full connected layer  n°1, input and 20 neurons-nodes of the first layer
    # one neuron can be the genre of the movie

    # Encode step
        self.fc1 = nn.Linear(nb_movies,20)
    # Full connected layer n°2
        self.fc2 = nn.Linear(20,10)

    # Decode step
    # Full connected layer n°3
        self.fc3 = nn.Linear(10,20)
    # Full connected layer n°4
        self.fc4 = nn.Linear(20,nb_movies)
    # Sigmoid activation function
        self.activation = nn.Sigmoid()

# Action : activation of the neurons
def forward(self, x) :
        x = self.activation(self.fc1(x))
        x = self.activation(self.fc2(x))
        x = self.activation(self.fc3(x))
        # dont's use the activation function
        # use the linear function only
        x = self.fc4(x)
        # x is th evector of predicted ratings
        return x

# Create the AutoEncoder object
sae=SAE()
#MSE Loss : imported from torch.nn
criterion=nn.MSELoss()
# RMSProp optimizer (update the weights) imported from torch.optim
#sea.parameters() are weights and bias adjusted during the training
optimizer=optim.RMSProp(sae.parameters(),lr=0.01, weight_decay=0.5)

### Part 2 : Training of the SAE
# number of epochs
nb_epochs = 200
# Epoch forloop
for epoch in range(1, nb_epoch+1):
        # at the beginning the loss is at zero
        s=0.
        train_loss = 0

        #Users forloop
        for id_user in range(nb_users)
            # add one dimension to make a two dimension vector.
            # create a new dimension and put it the first position .unsqueeze[0]
            input = Variable(training_set[id_user].unsqueeze[0])

            # clone the input to obtain the target
            target= input.clone()

            # target.data are all the ratings
            # ratings > 0
            if torch.sum(target.data >0) > 0
                output = sae(input)
                # don't compute the gradients regarding the target
                target.require_grad=False
                # only deal with true ratings
                output[target==0]=0

                # Loss Criterion
                loss =criterion(output,target)

                # Average the error of the movies that don't have zero ratings
                mean_corrector=nb_movies/float(torch.sum(target.data>0)+1e-10)

                # Direction of the backpropagation
                loss.backward()
                train_loss+=np.sqrt(loss.data[0]*mean_corrector)
                s+=1.

                # Intensity of the backpropagation
                optimizer.step()

    print('epoch:' +str (epoch)+'loss:' +str(train_loss/s)

)
If I want to train using the MLP model. How can I implement this class model?
Also, What other deep learning model(Beside MLP) that I can use to compare with Stacked-Autoencoder?
Thanks.

Output:
An MLP is not suited for recommendations. If you want to go this route, you will need to create an embedding for your userid and another for your itemid and then add linear layers on top of the embeddings. Your target will be to predict the rating for a userid-itemid pair.
I suggest you take a look at variational autoencoders (VAE). They give state-of-the-art results in recommender systems. They will also give a fair comparaison with your stacked-autoencoder. Here's the research paper applying VAE for collaborative filtering : https://arxiv.org/pdf/1802.05814.pdf


Instruction: select sub elements from another batch

Input:
I have first tensor with size  torch.Size([12, 64, 8, 8, 3]) ,  (8,8,3) is the image size, 64 is the patches, 12 is the batch size
There is another tensor with size torch.Size([12, 10]) which select 10 patches for each item in the batch (select 10 patches from total of 64). Thus it store the indexes. How to use this to query for the first tensor  with list comprehension?

Output:
You can use index_select:
c = [torch.index_select(i, dim=0, index=j) for i, j in zip(a,b)]

a and b are your tensor and indices respectively.
You could stack it in the zero dimension afterwards.


Instruction: PyTorch forward propagation returns different logits on same samples

Input:
Consider the following LeNet model for MNIST
import torch
from torch import nn
import torch.nn.functional as F

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)
        self.ceriation = nn.CrossEntropyLoss()
    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(x)
        x = x.view(-1, 4*4*50)
        x = self.fc1(x)
        x = self.fc2(x)
        return x

Now, I use this model to do a single forward step on a batch of samples like
network=LeNet()
optimizer = torch.optim.SGD(self.network.parameters(), lr=0.001, momentum=0.9)
device = torch.device("cpu")
network.to(device)
network.train()
optimizer.zero_grad()
# X_batch= ... some batch of 50 samples pulled from a train_loader defined as
# torch.manual_seed(42)
# training_set = datasets.MNIST('./mnist_data', train=True, download=False,
#                               transform=transforms.Compose([
#                                   transforms.ToTensor(),
#                                   transforms.Normalize((0.1307,), (0.3081,))]))
# train_loader = torch.utils.data.DataLoader(training_set,
#                                            batch_size=50,
#                                            shuffle=False)
logits = network(X_batch)

Note that shuffle=False and download=False for the loader since the data set is already downloaded and I don't want to shuffle. My problem is that if I run this code twice I will get different values for logits and I don't understand why since everything else seems to be unchanged. For an extra check, I also extract X_batch to a numpy array and verify that the batch of samples is exactly the same as of previous execution. I do this check with numpy.array_equal() function.
I really can't figure out what I am missing here unless there are precision issues.

Output:
The reason is because every time you run this code you call
network = LeNet()

and end up having different random initialization for the network's weights. If you set random seed before doing that, e.g. like this:
torch.manual_seed(42)
network = LeNet()

then you should get same results on first forward step given you use same data as input.


Instruction: Pytorch: Fast create score tensor method

Input:
I am new to Pytorch and am looking for a quick get score function. That, given a bunch of samples and a distribution, outputs a tensor consisting of the corresponding score for each individual sample. For instance, consider the following code:
norm = torch.distributions.multivariate_normal.MultivariateNormal(torch.zeros(2),torch.eye(2))
samples = norm.sample((1000,))
samples.requires_grad_(True)

Using samples I would like to create a score tensor of size [1000,2] where the ith component score[i] is the gradient of log p(samples[i]), where p is the density of the given distribution. The method I have come up with is the following:
def get_score(samples,distribution):
    log_probs = distribution.log_prob(samples)
    for i in range(log_probs.size()[0]):
        log_probs[i].backward(retain_graph = True)

The resulting score tensor is then samples.grad. The issue is that my method is quite slow for larger samples (e.g. for a sample of size [50000,2] it takes about 25-30 seconds on my CPU). Is this as fast as it can get?
The only alternative I can think of is to hard-code the score function for each distribution I will use, this doesn't seem like a good solution!

From experimentation, for 50000 samples, the following is about 50% quicker:
for i in range(50000):
    sample = norm.sample((1,))
    sample.requires_grad_(True)
    log_prob = norm.log_prob(a)
    log_prob.backward()

This indicates that there should be a better way!

Output:
I'm assuming that log_probs is stored as a pytorch tensor.
You can take advantage of the linearity of differentiation to calculate the derivative for all samples at once: log_probs.sum().backward(retain_graph = True)
At least with GPU acceleration this will be a lot faster.
If log_probs is not a tensor but a list of scalars (represented as pytorch tensors of rank 0), you can use log_probs = torch.stack(log_probs) first.


Instruction: Does PyTorch's nn.DataParallel load the same model in each GPU?

Input: The only way it would seem to work (logically) is if the model was loaded in each of the GPUs. This would mean that when weights are updated, each GPU would need to update the weights as well, increasing the workload compared to a single GPU. Is this line of reasoning correct?

Output:
First of all, it is advised to use torch.nn.parallel.DistributedDataParallel instead.
You can check torch.nn.DataParallel documentation where the process is described (you can also check source code and dig a little deeper on github, here is how replication of module is performed).
Here is roughly how it's done:
Initialization
All (or chosen) devices ids are saved in constructor and dimension along which data will be scattered (almost always 0 meaning it will be splitted to devices along batch)
Forward
This is done during every forward run:

Inputs are scattered (tensors along dimensions, tuple, list, dict shallowed copied, other data is shared among threads).
If there is only one device just return module(*args, **kwargs)
If there are multiple devices, copy the network from source machine to other devices (it is done each time!)
Run forward on each device with it's respective input
Gather outputs from devices onto a single source device (concatenation of outputs) onto a source machine.
Run the rest of the code, backprop, update weights on source machine etc.

Source machine is the cuda:0 by default, but it can be chosen. Also weights are updated on a single device, only batch is scattered and the outputs gathered.


Instruction: Loss not reducing in Linear Regression with Pytorch

Input:
I'm working on a Linear Regression problem with Pytorch. The dataset I'm using is the Housing Prices from Kaggle. While training the model I see the loss is not reducing. It shows an erratic pattern. This is the Loss I'm getting after 100 epochs:
Epoch [10/100], Loss: 222273830912.0000
Epoch [20/100], Loss: 348813688832.0000
Epoch [30/100], Loss: 85658296320.0000
Epoch [40/100], Loss: 290305572864.0000
Epoch [50/100], Loss: 59399933952.0000
Epoch [60/100], Loss: 80360054784.0000
Epoch [70/100], Loss: 90352918528.0000
Epoch [80/100], Loss: 534457679872.0000
Epoch [90/100], Loss: 256064503808.0000
Epoch [100/100], Loss: 102400483328.0000

This is the code:
import torch
import numpy as np
from torch.utils.data import TensorDataset
import torch.nn as nn
from torch.utils.data import DataLoader
import torch.nn.functional as F

inputs = normalized_X
targets = np.array(train_y)

# Tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
targets = targets.view(-1, 1)
train_ds = TensorDataset(inputs, targets.squeeze())
batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

model = nn.Linear(10, 1)
# Define Loss func
loss_fn = F.mse_loss
# Optimizer
opt = torch.optim.SGD(model.parameters(), lr = 1e-1)

num_epochs = 100
model.train()
for epoch in range(num_epochs):
    # Train with batches of data
    for xb, yb in train_dl:

        # 1. Generate predictions
        pred = model(xb.float())

        # 2. Calculate loss
        yb = yb.view(yb.size(0), -1)
        loss = loss_fn(pred, yb.float())

        # 3. Compute gradients
        loss.backward()

        # 4. Update parameters using gradients
        opt.step()

        # 5. Reset the gradients to zero
        opt.zero_grad()

    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch +
                                                   1, num_epochs,
                                                   loss.item()))

Output:
I have run the code you give and I get this error :
    p.py:38: UserWarning: Using a target size (torch.Size([50])) that is
different to the input size (torch.Size([50, 1])). This will likely lead
to incorrect results due to broadcasting. Please ensure they have the same size.

Your problem is due to the difference of dimension between pred and yb.
this code show how to resolve it
import torch
import numpy as np
from torch.utils.data import TensorDataset
import torch.nn as nn
from torch.utils.data import DataLoader
import torch.nn.functional as F

inputs = np.random.rand(50, 10)
targets = np.random.randint(0, 2, 50)

# Tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
train_ds = TensorDataset(inputs, targets.squeeze())
batch_size = 100
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

model = nn.Linear(10, 1)
# Define Loss func
loss_fn = F.mse_loss
# Optimizer
opt = torch.optim.SGD(model.parameters(), lr = 1e-1)

num_epochs = 100
model.train()
for epoch in range(num_epochs):
    # Train with batches of data
    for xb, yb in train_dl:

# 1. Generate predictions
    pred = model(xb.float())

    # 2. Calculate loss
    yb = yb.view(yb.size(0), -1)
    loss = loss_fn(pred, yb.float())

    # 3. Compute gradients
    loss.backward()

    # 4. Update parameters using gradients
    opt.step()

    # 5. Reset the gradients to zero
    opt.zero_grad()

    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch +
                                               1, num_epochs,
                                               loss.item()))

this discusion show to you in the detail
https://discuss.pytorch.org/t/target-size-torch-size-10-must-be-the-same-as-input-size-torch-size-2/72354/6


Instruction: How to split test and train data in a dataset based on number of targets in each category

Input:
I have an imageFolder in PyTorch which holds my categorized data images. Each folder is the name of the category and in the folder are images of that category.
I've loaded data and split train and test data via a sampler with random train_test_split. But the problem is my data distribution isn't good and some classes have lots of images and some classes have fewer.
So for solving this problem I want to choose 20% of each class as my test data and the rest would be the train data
ds = ImageFolder(filePath, transform=transform)
batch_size = 64
validation_split = 0.2

indices = list(range(len(ds))) # indices of the dataset

# TODO: fix spliting
train_indices,test_indices = train_test_split(indices,test_size=0.2)

# Creating PT data samplers and loaders:
train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(ds, batch_size=batch_size, sampler=train_sampler, num_workers=16)
test_loader = torch.utils.data.DataLoader(ds, batch_size=batch_size, sampler=test_sampler, num_workers=16)

Any idea of how I should fix it?

Output:
Use the stratify argument in train_test_split according to the docs. If your label indices is an array-like called y, do:
train_indices,test_indices = train_test_split(indices, test_size=0.2, stratify=y)


Instruction: Can't import ToTensorV2 in Colab

Input:
from albumentations.pytorch.transforms import ToTensorV2

I used the above code, and it doesn't work.

Output:
Just add a code block with the line
! pip install albumentations==0.4.6

above the block where you do the import. I tried installing it without the specific version and it failed.
When i did not specify the version number in pip install, version 0.1.12 was installed which does not contain ToTensorV2.