Instruction: Memory leak when using lambda in Python class

Input:
I have detected a memory leak in Python if I use lambda function inside a class.
Here's the code to reproduce the leak:
import torch
# import numpy as np

class Class1(object):
    def __init__(self, x0):
        self.x0 = x0
        self.obj2 = Class2()
        self._leak_fcn = lambda: self.obj2.fcn()  # source of memory leak!

class Class2(object):
    def fcn(self):
        pass

def fcn(x0):
    obj1 = Class1(x0)
    return x0

def test_fcn():
    shape = (50000000, 3)
    y0 = torch.randn(shape).to(torch.double)
    # y0 = np.random.randn(*shape)
    y = fcn(y0)
    return y

for i in range(1000):
    print(i)
    test_fcn()

The memory leak happens even if I changed it to numpy (without using pytorch).
No memory leak is detected if comment the line containing self._leak_fcn or if I write the _leak_fcn as a method instead of a lambda.
What is happening here?
I am not sure if this leak is from Python, or both PyTorch and NumPy are suffering the same leak.
FYI: I am using Python 3.8.5.
EDIT:
I know there is a memory leak here because if I run it for a long period, my memory is filled up (observed with htop) and the process is killed when it is run out of memory.

Output:
There is no leak here but simply a race as to whether the instances of Class1 get garbage collected soon enough to allow the torch or numpy buffers indirectly anchored by those Class1 buffers to be freed before the process no longer has enough memory to allocate another torch or numpy buffer.  Garbage collection is needed to break the PyObject reference cycle mentioned by @ThierryLathuille.
The fact that a delay in garbage collection is the issue is provable by simply changing the source from the example to import from gc and to add a call to gc.collect() in the last loop of the program, because this makes it so that garbage collection will always happen soon enough (assuming you have enough memory available on the system that the program can make it even one time through the loop).  To prove this, add "import gc" to the top of the program and also make the last loop look like this:
for i in range(1000):
    print(i)
    test_fcn()
    gc.collect()

You will then see that the program can run to completion (again, assuming you have enough memory to make it through the loop at least once).
The second thing one might want to confirm is that the garbage collection is simply not happening soon enough for the particular configuration but would happen eventually.  This is certainly the case, and provably so.  The way to do this is to reduce the memory used per numpy or torch buffer enough that the program does not run out of memory before garbage collection starts allowing some of those buffers to be freed, but not reduce it so much that the program could run to completion without any of those numpy or torch buffers being garbage collected at all.
To understand exactly what these numbers are, one needs understand how large the program will be allowed to grow.  On Linux, one limit is the total available memory to be used as "committed memory", but the system may be configured to allow some over-commit to occur.
This can be roughly checked by looking at /proc/meminfo.  On my system, subtracting Committed_AS from CommitLimit would suggest that if I run the program I will have less than 10 GB available to that program (assuming that other programs don't start or stop or change how much committed memory they use in the meantime).
$ grep Commit /proc/meminfo
CommitLimit:     9325344 kB
Committed_AS:     573964 kB

As already reported, the space used per torch or numpy buffer is roughly 1.2 GB (50,000,000 * 3 * 8) so even with some over-commit of memory allowed I would expect my program to store around 8 of those numpy or torch buffers before it crashes.  It fact, on my system (using numpy rather than torch by starting with the original program from the question and commenting out the torch lines and removing the # from the numpy lines) it crashes at around 10 times through the loop:
$ python3 junk.py
0
1
2
3
4
5
6
7
8
9
10
Traceback (most recent call last):
  File "junk.py", line 29, in <module>
    test_fcn()
  File "junk.py", line 23, in test_fcn
    y0 = np.random.randn(*shape)
  File "mtrand.pyx", line 1233, in numpy.random.mtrand.RandomState.randn
  File "mtrand.pyx", line 1390, in  numpy.random.mtrand.RandomState.standard_normal
  File "_common.pyx", line 577, in numpy.random._common.cont
numpy.core._exceptions.MemoryError: Unable to allocate 1.12 GiB for an array with shape (50000000, 3) and data type float64

So suppose I make the following change in the program to the arguments to shape() to reduce the size of the numpy buffer by a factor of 25:
# shape = (50000000, 3)
shape = (2000000, 3)

Now I expect the numpy buffer to take at least 2,000,000 * 3 * 8 = 48,000,000 bytes.  There is no way on my small system that the program could have 1,000 of those allocated but not yet freed (because that would take at least 48,000,000,000 bytes).  However, the program runs to completion with the modified size, showing that garbage collection must be working.
The next question one might ask is how someone who was not able to spot the python reference cycle using just the source as Thierry apparently did, could figure this out by analysis.
One way to do this analysis to use chap (an open source tool that runs on Linux and for which source is available at https://github.com/vmware/chap).
The input required by chap is a core from the program to be analyzed.  As seen above, on my system the program crashed after around 10 times through the loop so I chose to gather a live core for the program (using the original program with the minor commenting changes to use numpy) by running "gcore " after the program had run 8 or so times through the loop.
Here is the analysis using chap, starting from the point where chap has reached the chap prompt.  We know the numpy buffers are large, so we can find them just by using a command that finds any buffers that are at least 0x1000000 bytes.  Running that command shows that there were 7 such allocations at the time the core was gathered:
chap> describe used /minsize 1000000
Anchored allocation at 7fc6255b6010 of size 47868ff0

Anchored allocation at 7fc66ce1f010 of size 47868ff0

Anchored allocation at 7fc6b4688010 of size 47868ff0

Anchored allocation at 7fc6fbef1010 of size 47868ff0

Anchored allocation at 7fc74375a010 of size 47868ff0

Anchored allocation at 7fc78afc3010 of size 47868ff0

Anchored allocation at 7fc7d282c010 of size 47868ff0

7 allocations use 0x1f4adef90 (8,400,007,056) bytes.

If we pick one of those large allocations, we can see how it is referenced, to see why it is still in memory.  One way to do this is to use the following command, which specifies that we should start from the given allocation, extend to any allocations that contain a pointer to the start (offset 0) of that allocation, then stop:
chap> describe allocation 7fc7d282c010 /extend @0<-=>StopHere
Anchored allocation at 7fc7d282c010 of size 47868ff0

Anchored allocation at 7fc82242c620 of size 50
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x7fc8220a88e0 (numpy.ndarray)

2 allocations use 0x47869040 (1,200,001,088) bytes.

The result shows that there is just one such allocation and, not surprisingly given the source code, it is of type numpy.ndarray.  It matches the chap pattern SimplePythonObject because allocations of type numpy.ndarray cannot reference other python objects (the big buffer is not one) and don't have garbage collection headers.  Such an object will only be freed when the reference count transitions to 0.  The key thing to observe here is that the reference count is 1, meaning we are just looking for one reference to that numpy.darray to understand why it is in memory.
Continuing from the numpy.darray we see that 7 things reference the start of that allocation.  6 of them are instances of python type frame, and are unlikely uninteresting because there are 6 of them and we also only want to explain one reference to the numpy.darray.  There is also a single allocation that matches pattern PyDictValuesArray (because it holds the values for a split python dict) and this one is of more interest (remember to scroll through the output below):
chap> describe allocation 7fc82242c620 /extend @0<-=>StopHere
Anchored allocation at 7fc82242c620 of size 50
This allocation matches pattern SimplePythonObject.
This has reference count 1 and python type 0x7fc8220a88e0 (numpy.ndarray)

Anchored allocation at 562868cdae20 of size 268
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 562868e44ba0 of size 218
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 562868e48830 of size 238
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 562868e65b10 of size 238
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 7fc81a1f5030 of size 1f8
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 7fc81e5a3200 of size 1d0
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc82393aa00 (frame)

Anchored allocation at 7fc8223f9968 of size 28
This allocation matches pattern PyDictValuesArray.
It contains values for a split python dict.

8 allocations use 0xd30 (3,376) bytes.

An allocation that matches pattern PyDictValuesArray must be referenced by the ma_values field of a python dict.  The allocation holding the values is not itself reference counted, but depends on being freed either when the dict is freed or when the dict no longer needs that allocation.  Continuing from that allocation we can see the dict:
chap> describe allocation 7fc8223f9968 /extend @0<-=>StopHere
Anchored allocation at 7fc8223f9968 of size 28
This allocation matches pattern PyDictValuesArray.
It contains values for a split python dict.

Anchored allocation at 7fc823a646a8 of size 48
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc823939780 (dict)

2 allocations use 0x70 (112) bytes.

It is worth noting here that the dict has reference count 1 (so again only one reference needs to be explained), that the garbage collector is tracking this object and that the header for the actual dict, as opposed to the preceding garbage collection header starts at offset 0x18 to the dict.  This means that for references to the dict we need to look for things that point to offset 0x18 of the allocation.  (Links to offset 0 of the allocation would generally be used by garbage collection).
Using this information we can continue to see who references the dict and see that the dict is referenced by an instance of Class1.
chap> describe allocation 7fc823a646a8 /extend @18<-=>StopHere
Anchored allocation at 7fc823a646a8 of size 48
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc823939780 (dict)

Anchored allocation at 7fc822428d50 of size 38
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x562868afee88 (Class1)

2 allocations use 0x80 (128) bytes.

This is not surprising, given the program.  The dict must be being used to hold the various fields of the instance of Class1.  Again, note that the instance of Class1 has a reference count of 1, meaning we need to explain just one reference to understand why that instance of Class1 could still be in memory.
Continuing, again noticing that the allocation has a garbage collector, we can see that the Class1 instance is referenced by an instance of the python cell type, which again is being tracked for garbage collection and has reference count 1:
chap> describe allocation 7fc822428d50 /extend @18<-=>StopHere
Anchored allocation at 7fc822428d50 of size 38
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x562868afee88 (Class1)

Anchored allocation at 7fc823a1a870 of size 30
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc82393d6c0 (cell)

2 allocations use 0x68 (104) bytes.

Continuing, we can see that the cell is referenced by a tuple, which again has reference count 1:
chap> describe allocation 7fc823a1a870 /extend @18<-=>StopHere
Anchored allocation at 7fc823a1a870 of size 30
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc82393d6c0 (cell)

Anchored allocation at 7fc822428fb8 of size 38
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc823936320 (tuple)

2 allocations use 0x68 (104) bytes.

When checking for the tuple, it appears to be referenced by both a tuple and a function but the referencing tuple in this case does not appear to be of interest for our purposes because the garbage collector is not tracking it and also the reference count is 0.  The function is definitely interesting because it is being tracked, because it has reference count 1, and because from the source we know that we are looking for such a thing:
chap> describe allocation 7fc822428fb8 /extend @18<-=>StopHere
Anchored allocation at 7fc822428fb8 of size 38
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc823936320 (tuple)

Anchored allocation at 562868aff290 of size 2b8
This allocation matches pattern ContainerPythonObject.
This allocation is not currently tracked by the garbage collector.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 0 and python type 0x7fc823936320 (tuple)

Anchored allocation at 7fc81a114938 of size 88
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc82393a860 (function)

3 allocations use 0x378 (888) bytes.

Continuing from the function, we complete the cycle because the allocation at 0x7fc8223f9968 that matches pattern PyDictKeysArray has already been seen earlier in our traversal.
chap> describe allocation 7fc81a114938 /extend @18<-=>StopHere
Anchored allocation at 7fc81a114938 of size 88
This allocation matches pattern ContainerPythonObject.
The garbage collector considers this allocation to be reachable.
This has a PyGC_Head at the start so the real PyObject is at offset 0x18.
This has reference count 1 and python type 0x7fc82393a860 (function)

Anchored allocation at 7fc8223f9968 of size 28
This allocation matches pattern PyDictValuesArray.
It contains values for a split python dict.

2 allocations use 0xb0 (176) bytes.

So we have a cycle that looks like this:
Class1 -> dict -> %PyDictValuesArray -> function -> tuple -> cell ->

The allocations in the cycle itself do not take all that much memory, but it does require garbage collection to free it, and the %PyDictValuesArray in that cycle also holds the numpy.darray which holds the big buffer.
People including the author of the question and several people in the comments have discussed valid solutions to avoid the growth, so I will avoid any discussion of the fix itself and limit this answer to the above analysis of the reason for the growth.


Instruction: Installing YOLOv5 dependencies - torchvision?

Input:
I want to use yolov5. According to https://pytorch.org/hub/ultralytics_yolov5/, you should have Python>=3.8 and PyTorch>=1.7 installed, as well as YOLOv5 dependencies.
Python and pytorch are up to date:
pip show torch
Version: 1.7.1

python --version
Python 3.9.1

But when I try to install the yolov5 dependencies, I get an error message:
pip install -qr https://raw.githubusercontent.com/ultralytics/yolov5/master/requirements.txt
ERROR: Could not find a version that satisfies the requirement torchvision>=0.8.1
ERROR: No matching distribution found for torchvision>=0.8.1

An updated version of torchvision is needed (???). So I run the update, but when I check the version it hasn't worked.
pip show torchvision
Version: 0.2.2.post3

-m pip install --upgrade torchvision

pip show torchvision
Version: 0.2.2.post3

Is torchvision needed for installing the yolov5 dependencies? How do I move forward? I'm on Windows 10.
Thanks!

Output:
I was getting similar error. Installing fresh compatible libraries (torch==1.7.1 & torchvision==0.8.2) worked for me.
virtualenv -p python3.8 torch17
source torch17/bin/activate
pip install cython matplotlib tqdm scipy ipython ninja yacs opencv-python ffmpeg opencv-contrib-python Pillow scikit-image scikit-learn lmfit imutils pyyaml jupyterlab==3
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Be careful about your cuda version and install accordingly. Mine is 10.1. Simply using pip install torch==1.7.1 will install latest cuda version (11) compatible lib which might not be your case.
Virtual environment activation command will be different for Windows. I am on Linux


Instruction: how to convert HuggingFace's Seq2seq models to onnx format

Input:
I am trying to convert the Pegasus newsroom in HuggingFace's transformers model to the ONNX format. I followed this guide published by Huggingface. After installing the prereqs, I ran this code:
!rm -rf onnx/
from pathlib import Path
from transformers.convert_graph_to_onnx import convert

convert(framework="pt", model="google/pegasus-newsroom", output=Path("onnx/google/pegasus-newsroom.onnx"), opset=11)

and got these errors:
ValueError                                Traceback (most recent call last)
<ipython-input-9-3b37ed1ceda5> in <module>()
      3 from transformers.convert_graph_to_onnx import convert
      4
----> 5 convert(framework="pt", model="google/pegasus-newsroom", output=Path("onnx/google/pegasus-newsroom.onnx"), opset=11)
      6
      7

6 frames
/usr/local/lib/python3.6/dist-packages/transformers/models/pegasus/modeling_pegasus.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, head_mask, encoder_head_mask, past_key_values, inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
    938             input_shape = inputs_embeds.size()[:-1]
    939         else:
--> 940             raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
    941
    942         # past_key_values_length

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds

I have never seen this error before. Any ideas?

Output:
Pegasus is a seq2seq model, you can't directly convert a seq2seq model (encoder-decoder model) using this method. The guide is for BERT which is an encoder model. Any only encoder or only decoder transformer model can be converted using this method.
To convert a seq2seq model (encoder-decoder) you have to split them and convert them separately, an encoder to onnx and a decoder to onnx. you can follow this guide (it was done for T5 which is also a seq2seq model)
Why are you getting this error?
while converting PyTorch to onnx
_ = torch.onnx._export(
                        model,
                        dummy_input,
                        ...
                       )

you need to provide a dummy variable to both encoder and to the decoder separately. by default when converting using this method it provides the encoder the dummy variable. Since this method of conversion didn't accept decoder of this seq2seq model, it won't give a dummy variable to the decoder and you get the above error.
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds


Instruction: How to tell PyTorch which CUDA version to take?

Input:
I have two version of CUDA installed on my Ubuntu 16.04 machine: 9.0 and 10.1.
They are located in /usr/local/cuda-9.0 and /usr/local/10.1 respectively.
If I install PyTorch 1.6.0 (which needs CUDA 10.1) via pip (pip install torch==1.6.0), it uses version 9.0 and thus detects no GPUs. I already changed my LD_LIBRARY_PATH to "/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/cuda/extras/CUPTI/lib64" but PyTorch is still using CUDA 9.0.
How do I tell PyTorch to use CUDA 10.1?

Output:
Prebuilt wheels for torch built with different versions of CUDA are available at torch stable releases page. For example you can install torch v1.9.0 built with CUDA v11.1 like this:
pip install --upgrade torch==1.9.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

But not all the combinations are available.


Instruction: Is there a way to make a model that creates a mask to drop certain inputs before feeding the masked data to another network?

Input:
This might be a question that is kind of dumb but I'm trying to construct a model that is able to filter out inputs before feeding the filtered output to another network.
For example, I have an image that I would to match with a database of about 100 pictures, then I would apply the first network to do some operations that would output the top 10 pictures that is most likely to correctly match. Afterwards, I would apply a second network to rematch those top 10 pictures using a secondary network.
INPUT --> | NETWORK 1 | --> FILTERED OUTPUT --> | NETWORK 2 | --> FINAL OUTPUT

Wondering if there is a way to accomplish this sort of filtration behaviour where that filtered output is fed to the second model like that.

Output:
You could maybe take a look at Boolean index arrays with numpy
    >>> import numpy as np
    >>> x = np.array(range(20))
    >>> x
    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9,
    10, 11, 12, 13, 14, 15, 16, 17, 18, 19])
    >>> x[x > 10]
    array([11, 12, 13, 14, 15, 16, 17, 18, 19])

x > 10 returns an array with 20 booleans, so you can maybe try something like this:
    x = pic_arr[network1(pic_arr)]
    network2(x)

Where pic_arr is an array with your pictures and network1 returns a list with booleans of which pictures to select.


Instruction: while running netron on colab, getting this "OSError: [Errno 98] Address already in use" error

Input:
I'm using Netron, for visualizing the model on Colab. as shown in this notebook line 11.
when I run the following script to view the model,
import netron

enable_netron = True
if enable_netron:
    netron.start(optimized_model_path)

am getting this error:
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-13-86b0c5c2423c> in <module>()
      5 if enable_netron:
----> 6     netron.start(optimized_model_path)

5 frames
/usr/lib/python3.6/socketserver.py in server_bind(self)
    468         if self.allow_reuse_address:
    469             self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
--> 470         self.socket.bind(self.server_address)
    471         self.server_address = self.socket.getsockname()
    472

OSError: [Errno 98] Address already in use

how can I fix this issue?
Ik I can use its desktop version of the app and give it the model but how can use Netron in colab?

Output:
Should be able to use portpicker.pick_unused_port().
Here's a simple example:
https://colab.research.google.com/gist/blois/227d21df87fe8a390c2a23a93b3726f0/netron.ipynb


Instruction: Partial Backwards in PyTorch Graph

Input:
I have a medium sized tensor x. On this mediums sized tensor an computationally expensive function (both forward and backwards) q is applied to obtain another medium sized tensor y.
Using y I evaluate many functions f producing a scalar, they are not particularly computationally expensive however use large internal states causing a large computational graph for torch that.
Now I want to compute the gradient on x in the following way
y = q(x)

for f in functions
    res += f(y)

res.backward()

The proble with this implementation is that the graphs of all the functions f are retained. That causes the total memory usage to explode.
The other possibility would be to compute
y = q(x)

for f in functions
    partial = f(y)
    partial.backward(retain_graph = True)

The advantage is that after every function evaluation f the result goes out of scope and the graph is freed, saving massive amounts of memory. However, in this situation the function q(x) is evaluated backwards multiple times which is very time expensive.
In the ideal situation I would want to first compute the gradient for y using code similar to the second example and then only once evaluate q backwards to get the gradient for x. What is the proper way to do that with PyTorch?

Output:
I think that would be the way to achieve it :
y = q(x)
z = y.detach()
z.requires_grad_(True)

for f in functions:
    partial = f(y)
    partial.backward(retain_graph = True)
y.backward(z.grad)

You accumulate all the gradients in z, which y but in another computational graph, then you propagate these gradients (z.grad) in the first graph


Instruction: How can i detect if a callback is triggered in pytorch?

Input: I am fine-tuning a BERT model. First, I want to freeze layers and train a bit. When a certain callback is triggered (let's say ReduceLROnPlateau) I want to unfreeze layers. How can I do it?

Output:
I'm afraid learning rate schedulers in PyTorch don't provide hooks. Looking at the implementation of ReduceLROnPlateau here, two properties are reset when the scheduler is triggered (i.e. when it identifies a plateau and reduces the learning rate):
    if self.num_bad_epochs > self.patience:
        self._reduce_lr(epoch)
        self.cooldown_counter = self.cooldown
        self.num_bad_epochs = 0

Based on that, you could wrap your scheduler step call and find out if _reduce_lr was triggered by checking that both scheduler.cooldown_counter == scheduler.cooldown and scheduler.num_bad_epochs == 0 are true.


Instruction: PyTorch nn.Conv2d output comptation

Input:
I am using Python 3.8 and PyTorch 1.7.1. I saw a code which defines a Conv2d layer as follows:
Conv2d(3, 6, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)

The input 'X' being passed to it is a 4D tensor-
X.shape
# torch.Size([4, 3, 6, 6])

The output volume for this conv layer is:
c1(X).shape
# torch.Size([4, 6, 3, 3])

I am trying to use the formula to compute output spatial dimensions for any conv layer: O = ((W - K + 2P)/S) + 1, where W = spatial dimension of image, K = filter/kernel size, P = zero padding & S = stride.
For 'c1' conv layer, we get, W = 6, K = 3, S = 2 & P = 1. Using the formula, you get O = ((6 - 3 + (2 x 1)) / 2) + 1 = 5/2 + 1 = 3.5.
The output volume: (4, 6, 3, 3) since number of filters used = 6.
How is the spatial output from 'c1' then (3, 3)? What am I not getting?
Thanks!

Output:
How would you have half a pixel?
You're missing the floor function:
O = floor(((W - K + 2P)/S) + 1)

So the shape of the outputted maps is (3, 3).

Here's the complete formula (with dilation) for nn.Conv2d:


Instruction: Exporting ONNX Model from Deep Learning Frameworks at Operator-level

Input:
Hi, I have some questions regarding exporting ONNX models.
Let's say we have an LSTM cell from PyTorch.
Using torch.onnx.export produces ONNX model with LSTM layer.
However, I am interested in whether it can produce the ONNX model at the operator level, i.e, matmul, add.
Is there a way to do so?
If not, is there another way to make an operator level ONNX model?
Thanks,
Jake

Output:
When you export a model from PyTorch to onnx using the torch.onnx.export() function, it records all the operations that the initial model has used. as mentioned here.

we call the torch.onnx.export() function. This will execute the model, recording a trace of what operators are used to computing the outputs.

so, yes it does produce the onnx model at the operator level, you can even visualize the exported .onnx model graph using netron

if you still want to use the onnx operator, here is the ONNX Operator Schemas.


Instruction: What does the difference between 'torch.backends.cudnn.deterministic=True' and 'torch.set_deterministic(True)'?

Input:
My network includes 'torch.nn.MaxPool3d' which throw a RuntimeError when cudnn deterministic flag is on according to the PyTorch docs (version 1.7 - https://pytorch.org/docs/stable/generated/torch.set_deterministic.html#torch.set_deterministic), however, when I inserted the code 'torch.backends.cudnn.deterministic=True' at the beginning of my code, there was no RuntimeError. Why doesn't that code throw a RuntimeError?
I wonder whether that code guarantees the deterministic computation of my training process.

Output:
torch.backends.cudnn.deterministic=True only applies to CUDA convolution operations, and nothing else. Therefore, no, it will not guarantee that your training process is deterministic, since you're also using torch.nn.MaxPool3d, whose backward function is nondeterministic for CUDA.
torch.set_deterministic(), on the other hand, affects all the normally-nondeterministic operations listed here (note that set_deterministic has been renamed to use_deterministic_algorithms in 1.8): https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html?highlight=use_deterministic#torch.use_deterministic_algorithms
As the documentation states, some of the listed operations don't have a deterministic implementation. So if torch.use_deterministic_algorithms(True) is set, they will throw an error.
If you need to use nondeterministic operations like torch.nn.MaxPool3d, then, at the moment, there is no way for your training process to be deterministic--unless you write a custom deterministic implementation yourself. Or you could open a GitHub issue requesting a deterministic  implementation: https://github.com/pytorch/pytorch/issues
In addition, you might want to check out this page: https://pytorch.org/docs/stable/notes/randomness.html


Instruction: Why do I get a ValueError when using CrossEntropyLoss

Input:
I am trying to train a pretty simple 2-layer neural network for a multi-class classification class. I am using CrossEntropyLoss and I get the following error: ValueError: Expected target size (128, 4), got torch.Size([128]) in my training loop at the point where I am trying to calculate the loss.
My last layer is a softmax so it outputs the probabilities of each of the 4 classes. My target values are a vector of dimension 128 (just the class values). Am I initializing the CrossEntropyLoss object incorrectly?
I looked up existing posts, this one seemed the most relevant:
https://discuss.pytorch.org/t/valueerror-expected-target-size-128-10000-got-torch-size-128-1/29424 However, if I had to squeeze my target values, how would that work? Like right now they are just class values for e.g., [0 3 1 0]. Is that not how they are supposed to look? I would think that the loss function maps the highest probability from the last layer and associates that to the appropriate class index.
Details:

This is using PyTorch
Python version is 3.7
NN architecture is: embedding -> pool -> h1 -> relu -> h2 -> softmax
Model Def (EDITED):

self.embedding_layer = create_embedding_layer(embeddings)
self.pool = nn.MaxPool1d(1)
self.h1 = nn.Linear(embedding_dim, embedding_dim)
self.h2 = nn.Linear(embedding_dim, 4)
self.s = nn.Softmax(dim=2)

forward pass:
    x = self.embedding_layer(x)
    x = self.pool(x)
    x = self.h1(x)
    x = F.relu(x)
    x = self.h2(x)
    x = self.s(x)
    return x

Output:
The issue is that the output of your model is a tensor shaped as (batch, seq_length, n_classes). Each sequence element in each batch is a four-element tensor corresponding to the predicted probability associated with each class (0, 1, 2, and 3). Your target tensor is shaped (batch,) which is usually the correct shape (you didn't use one-hot-encodings). However, in this case, you need to provide a target for each one of the sequence elements.
Assuming the target is the same for each element of your sequence (this might not be true though and is entirely up to you to decide), you may repeat the targets seq_length times. nn.CrossEntropyLoss allows you to provide additional axes, but you have to follow a specific shape layout:

Input: (N, C) where C = number of classes, or (N, C, d_1, d_2, ..., d_K) with K≥1 in the case of K-dimensional loss.
Target: (N) where each value is 0 ≤ targets[i] ≤ C−1 , or (N, d_1, d_2, ..., d_K) with K≥1 in the case of K-dimensional loss.

In your case, C=4 and seq_length (what you referred to as D) would be d_1.
>>> seq_length = 10
>>> out = torch.rand(128, seq_length, 4) # mocking model's output
>>> y = torch.rand(128).long()           # target tensor

>>> criterion = nn.CrossEntropyLoss()

>>> out_perm = out.permute(0, 2, 1)
>>> out_perm.shape
torch.Size([128, 4, 10])  # (N, C, d_1)

>>> y_rep = y[:, None].repeat(1, seq_length)
>>> y_rep.shape
torch.Size([128, 10])     # (N, d_1)

Then call your loss function with criterion(out_perm, y_rep).


Instruction: Pytorch GRU error RuntimeError : size mismatch, m1: [1600 x 3], m2: [50 x 20]

Input:
Currently, I'm trying to make the training model for an LSTM and GRU. The LSTM is working perfectly,  but once I switched into GRU training, errors pop-out such as size mismatch error.
This is my code
    path = "new_z_axis"
    device = "cuda:0"
    in_size = 3
    h_size = 50
    n_layers = 3
    fc = 20
    out = 1
    batch_size = 16
    seq = 100
    epoch = 100
    learning_rate = 1e-3
    ratio = 0.8
    checkpoint = os.path.join("checkpoints","model_"+path+"_"+str(in_size)+".pth")

    class GRUNet(nn.Module):
        def __init__(self,in_size,h_size,n_layers,fc_out,out_size,dropout=0.5):
            super(GRUNet, self).__init__()
            self.gru = nn.GRU(input_size=in_size,hidden_size=h_size,num_layers=n_layers,dropout=dropout,bias=False)
            self.fc = nn.Linear(in_features=h_size,out_features=fc_out,bias=False)
            self.relu = nn.ReLU(inplace=True)
            self.out = nn.Linear(in_features=fc_out,out_features=out_size,bias=False)
            self.tanh = nn.Tanh()
        def forward(self, x, hidden):
            out, hidden = self.gru(x, hidden)
            x = self.fc(x)
            x = self.relu(x)
            x = self.out(x)
            x = self.tanh(x)
            return x, hidden

    class MyLstm(nn.Module):
        def __init__(self,in_size,h_size,n_layers,fc_out,out_size,dropout=0.5):
            super(MyLstm, self).__init__()
            self.lstm = nn.LSTM(input_size=in_size,hidden_size=h_size,num_layers=n_layers,dropout=dropout,bias=False)
            self.fc = nn.Linear(in_features=h_size,out_features=fc_out,bias=False)
            self.relu = nn.ReLU(inplace=True)
            self.out = nn.Linear(in_features=fc_out,out_features=out_size,bias=False)
            self.tanh = nn.Tanh()
        def forward(self,x,hidden):
            x, hidden = self.lstm(x,hidden)
    #         x = x[-1:]
            x = self.fc(x)
            x = self.relu(x)
            x = self.out(x)
            x = self.tanh(x)
            return x, hidden

    def train(model,train_list,val_list,path,seq,epoch,batch_size,criterion,optimizer,model_type):
        for e in range(epoch):
            train_data = load_data(train_list,batch_size)
            a_loss = 0
            a_size = 0
            model.train()
            for x,y in train_data:
                x,y = x.to(device),y.to(device)
                bs = x.size()[1]

    #             hidden = (hidden[0].detach(),hidden[1].detach())
    #             print(x.size(),hidden[0].size())
                if model_type == "GRU":
                    h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    hidden = h1
                    hidden = hidden.data
                else:
                    h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    h2 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    hidden = (h1,h2)
                    hidden = tuple([e.data for e in hidden])

                model.zero_grad()
                print (len(hidden))
                pred,hidden = model(x,hidden)
                loss = criterion(pred,y)
                loss.backward()
                nn.utils.clip_grad_norm_(model.parameters(),5)
                optimizer.step()
                a_loss += loss.detach()
                a_size += bs
    #         print(e,a_loss/a_size*1e+6)
            model.eval()
            with torch.no_grad():
                val_data = load_data(val_list,batch_size)
                b_loss = 0
                b_size = 0
                for x,y in val_data:
                    x,y = x.to(device),y.to(device)
                    bs = x.size()[1]
                    if model_type == "GRU":
                        h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        hidden = h1
                        hidden = hidden.data
                    else:
                        h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        h2 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        hidden = (h1,h2)
                        hidden = tuple([e.data for e in hidden])
                    pred,hidden = model(x,hidden)
                    loss = criterion(pred,y)
                    b_loss += loss.detach()
                    b_size += bs
            print("epoch: {} - train_loss: {} - val_loss: {}".format(e+1,float(a_loss.item()/a_size*1e+6),b_loss.item()/b_size*1e+6))

train(modelGRU,train_list,val_list,path,seq,epoch,batch_size,criterionGRU,optimizerGRU,model_type="GRU")

This is the error i got

-------------------------------------------------- -------------------------  RuntimeError                               Traceback (most recent call last)   in

---- > 1  train ( modelGRU , train_list , val_list , path , seq , epoch , batch_size , criterionGRU , optimizerGRU , model_type = "GRU"
)
 in train (model, train_list, val_list,
path, seq, epoch, batch_size, criterion, optimizer, model_type)
61              model . zero_grad ( )
62              print ( len ( hidden ) )
---> 63              pred , hidden = model ( x , hidden )
64              loss = criterion ( pred , y )
65              loss .backward ( )
~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in call
(self, *input, **kwargs)
539              result = self . _slow_forward ( * input ,  ** kwargs )
540          else :
--> 541              result = self . forward ( * input ,  ** kwargs )
542          for hook in self . _forward_hooks . values ( ) :
543              hook_result = hook ( self , input , result )
 in forward (self, x, hidden)
11      def forward ( self , x , hidden ) :
12          out , hidden = self . gru ( x , hidden )
---> 13          x = self . fc ( x )
14          x = self . relu ( x )
15          x =self . out ( x )
~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in call
(self, *input, **kwargs)
539              result = self . _slow_forward ( * input ,  ** kwargs )
540          else :
--> 541              result = self . forward ( * input ,  ** kwargs )
542          for hook in self . _forward_hooks . values ( ) :
543              hook_result = hook ( self , input , result )
~ \ Anaconda3 \ lib \ site-packages \ torch \ nn \ modules \ linear.py
in forward (self, input)
85
86      def forward ( self , input ) :
---> 87 return F . Linear ( the Input , Self . Weight , Self . negative bias )      88      89 def extra_repr ( Self ) :
~\Anaconda3\lib\site-packages\torch\nn\functional.py in linear (input,
weight, bias)     1370          ret = torch . addmm ( bias , input ,
weight . t ( ) )     1371      else :
-> 1372          output = input . matmul ( weight . t ( ) )     1373          if bias is  not  None :     1374              output += bias
RuntimeError : size mismatch, m1: [1600 x 3], m2: [50 x 20] at
C:/w/1/s/tmp_conda_3.7_104508/conda/conda-bld/pytorch_1572950778684/work/aten/src\THC/
generic/THCTensorMathBlas.cu:290

Any advice?
Thank you

Output:
This might have to do with the fact that you are not passing the output of your nn.GRU to the first linear layer in GRUNet's forward function:
    def forward(self, x, hidden):
        out, hidden = self.gru(x, hidden)
        x = self.fc(out)


Instruction: Launching the right number of CUDA blocks for a custom PyTorch activation function

Input:
I'm currently writing a CUDA kernel for a custom operation (an activation) for PyTorch, but I'm quite unfamiliar with any form of GPU programming. For reference, I was following the Custom C++ & CUDA extension tutorial.
A simplified example of the sort of operation I want to do:
Let's say I have an input tensor, X_in, which can be of any shape and dims (e.g. something like (16, 3, 50, 100) ). Let's say I also have a 1D tensor, T (for example, T can be a tensor of shape (100,) ).
For each value in X_in, I want to calculate an "index" value that should be < len(T). Then the output would be basically the value of that index in T, multiplied or added with some constant. This is something like a "lookup table" operation.
An example kernel:
__global__ void inplace_lookup_kernel(
    const scalar_t* __restrict__ T,
    scalar_t* __restrict__ X_in,
    const int N) {

    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    const int idx = int(X_in[i]) % N;

    X_in[i] = 5 * T[idx] - 3;
}

I also wish to do the operation in-place, which is why the output is being computed into X_in.
My question is, for an operation like this, which is to be applied pointwise on each value of X_in, how to determine the way to launch a good number of threads/blocks? In the Custom C++ & CUDA extension tutorial, they do so by:
const int threads = 1024;
const dim3 blocks((state_size + threads - 1) / threads, batch_size);

For their use-case, the operation (an lstm variant) has a specific format of input, and thus a fixed number of dimensions, from which blocks can be calculated.
But the operation I'm writing should accept inputs of any dimensions and shape. What is the right way to calculate the block number for this situation?
I'm currently doing the following:
const int threads = 1024;
const int nelems = int(X_in.numel());
const dim3 blocks((nelems + threads - 1) / threads);

However I'm doing this by intuition, and not with any certainty. Is there a better or correct way to do this? And is there any computational advantage if I define blocks in the format blocks(otherdim_size, batch_size) like in the tutorial?

Output:
I'm speculating here, but - since your operations seems to be completely elementwise (w.r.t. X_in); and you don't seem to using interesting SM-specific resources like shared memory, nor a lot of registers per thread, I don't think the grid partition matters all that much. Just treat X_in as a 1-D array according to its layout in memory, and use a 1D grid with a block size of, oh, 256, or 512 or 1024.
Of course - always try out your choices to make sure you don't get bitten by unexpected behavior.


Instruction: PyTorch training with Batches of different lenghts?

Input: Is it possible to train  model with batches that have unequal lenght during an epoch? I am new to pytorch.

Output:
If you take a look at the dataloader documentation, you'll see a drop_last parameter, which explains that sometimes when the dataset size is not divisible by the batch  size, then you get a last batch of different size. So basically the answer is yes, it is possible, it happens often and it does not affect (too much) the training of a neural network.
However you must a bit careful, some pytorch layers deal poorly with very small batch sizes. For example if you happen to have Batchnorm layers, and if you get a batch of size 1, you'll get errors due to the fact that batchnorm at some point divides by len(batch)-1. More generally, training a network that has batchnorms generally require batches of significant sizes, say at least 16 (literature generally aims for 32 or 64). So if you happen to have variable size batches, take the time to check whether your layers have requirement in terms of batch size for optimal training and convergence. But except in particular cases, your network will train anyway, no worries.
As for how to make your batches with custom sizes, I suggest you look at and take inspiration from the pytorch implementation of dataloader and sampler. You may want to implement something similar to BatchSampler and use the batch_sampler argument of Dataloader


Instruction: How to convert raw image bytes, read from stream, to a tensor of a shape valid to perform Conv2d?

Input:
Community,
I have a file of bytes organized into chunks of 16384 bytes. Each of the chunks contains an uncompressed image of size 64x64 pixels. The format of the image pixel is 8-bit ABGR.
Let's say I have successfully read the chunk into numpy.array:
buf = numpy.fromfile( dataFile, dtype=np.uint8, count=16384, offset=offs)

The question is how could I convert this array of bytes into a Pytorch tensor so that to be able to perform convolution (Conv2d)?
If I understand it properly, the aforementioned convolution (Conv2d) expects the tensor to have separate channel planes instead of a single plane of multichannel pixels.
And the extra question is how to get rid of alpha-channel in the meanwhile?

Output:
In the following code, I'm assuming that your image is stored in a row-major fashion (so your bytes are abgrabgrabgr.... and not aaaa....bbbb....gggg....rrrr)
# First I reshape the buf into an image with 4 channels
buf = buf.reshape((4, 64, 64))
# Then I remove the alpha channel by taking only the 3 last :
bgr_buf = buf[1:, :, :]
# Then I make it into a pytorch tensor :
tensor = torch.from_numpy(bgr_buf)
# And finally, pytorch convolutions expects batchs of images. So let's make it a batch of 1
batch = tensor.view(1, 3, 64, 64)


Instruction: Pytorch Linear Regression with squared features

Input:
I am new to PyTorch and I would like to implement linear regression partly with PyTorch and partly on my own. I want to use squared features for my regression:
import torch

# init
x = torch.tensor([1,2,3,4,5])
y = torch.tensor([[1],[4],[9],[16],[25]])
w = torch.tensor([[0.5], [0.5], [0.5]], requires_grad=True)

iterations = 30
alpha = 0.01

def forward(X):
    # feature transformation [1, x, x^2]
    psi = torch.tensor([[1.0, x[0], x[0]**2]])
    for i in range(1, len(X)):
        psi = torch.cat((psi, torch.tensor([[1.0, x[i], x[i]**2]])), 0)
    return torch.matmul(psi, w)

def loss(y, y_hat):
    return ((y-y_hat)**2).mean()

for i in range(iterations):

    y_hat = forward(x)

    l = loss(y, y_hat)
    l.backward()

    with torch.no_grad():
        w -= alpha * w.grad
    w.grad.zero_()

    if i%10 == 0:
        print(f'Iteration {i}: The weight is:\n{w.detach().numpy()}\nThe loss is:{l}\n')

When I execute my code, the regression doesn't learn the correct features and the loss increases permanently. The output is the following:
Iteration 0: The weight is:
[[0.57 ]
 [0.81 ]
 [1.898]]
The loss is:25.450000762939453

Iteration 10: The weight is:
[[ 5529.5835]
 [22452.398 ]
 [97326.12  ]]
The loss is:210414632960.0

Iteration 20: The weight is:
[[5.0884394e+08]
 [2.0662339e+09]
 [8.9567642e+09]]
The loss is:1.7820802835250162e+21

Does somebody know, why my model is not learning?
UPDATE
Is there a reason why it performs so poorly? I thought it's because of the low number of training data. But also with 10 data points, it is not performing well :

Output:
You should normalize your data. Also, since you're trying to fit x -> ax² + bx + c, c is essentially the bias. It should be wiser to remove it from the training data (I'm referring to psi here) and use a separate parameter for the bias.
What could be done:

normalize your input data and targets with mean and standard deviation.

separate the parameters into w (a two-component weight tensor) and b (the bias).

you don't need to construct psi on every inference since x is identical.

you can build psi with torch.stack([torch.ones_like(x), x, x**2], 1), but here we won't need the ones, as we've essentially detached the bias from the weight tensor.

Here's how it would look like:
x = torch.tensor([1,2,3,4,5]).float()
psi = torch.stack([x, x**2], 1).float()
psi = (psi - psi.mean(0)) / psi.std(0)

y = torch.tensor([[1],[4],[9],[16],[25]]).float()
y = (y - y.mean(0)) / y.std(0)

w = torch.tensor([[0.5], [0.5]], requires_grad=True)
b = torch.tensor([0.5], requires_grad=True)

iterations = 30
alpha = 0.02
def loss(y, y_hat):
    return ((y-y_hat)**2).mean()

for i in range(iterations):
    y_hat = torch.matmul(psi, w) + b
    l = loss(y, y_hat)
    l.backward()

    with torch.no_grad():
        w -= alpha * w.grad
        b -= alpha * b.grad
    w.grad.zero_()
    b.grad.zero_()

    if i%10 == 0:
        print(f'Iteration {i}: The weight is:\n{w.detach().numpy()}\nThe loss is:{l}\n')

And the results:
Iteration 0: The weight is:
[[0.49954653]
 [0.5004535 ]]
The loss is:0.25755801796913147

Iteration 10: The weight is:
[[0.49503425]
 [0.5049657 ]]
The loss is:0.07994867861270905

Iteration 20: The weight is:
[[0.49056274]
 [0.50943726]]
The loss is:0.028329044580459595


Instruction: Pytorch model runtime error when testing U-Net

Input:
I've defined a U-Net model using Pytorch but it won't accept my input. I've checked the model layers and they seem to be applying the operations as I would expect them to but I still get an error.
I've just switched to Pytorch after mostly using Keras so I'm not really sure how to debug this issue, the error I get is:

RuntimeError: Given groups=1, weight of size [32, 64, 3, 3], expected
input[1, 128, 65, 65] to have 64 channels, but got 128 channels
instead

Here's the code I'm using:
class UNET(nn.Module):
    def __init__(self, in_channels=2, out_channels=2):
        super().__init__()

        self.conv1 = self.contract_block(in_channels, 32, 3, 1)
        self.conv2 = self.contract_block(32, 64, 3, 1)
        self.conv3 = self.contract_block(64, 128, 3, 1)

        self.upconv3 = self.expand_block(128, 64, 3, 1)
        self.upconv2 = self.expand_block(64, 32, 3, 1)
        self.upconv1 = self.expand_block(32, out_channels, 3, 1)

    def __call__(self, x):

        # downsampling part
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)

        upconv3 = self.upconv3(conv3)

        upconv2 = self.upconv2(torch.cat([upconv3, conv2], 1))
        upconv1 = self.upconv1(torch.cat([upconv2, conv1], 1))

        return upconv1

    def contract_block(self, in_channels, out_channels, kernel_size, padding):

        contract = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=1)
            )

        return contract

    def expand_block(self, in_channels, out_channels, kernel_size, padding):

        expand = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.ConvTranspose2d(out_channels, out_channels, kernel_size=2, stride=2, padding=1, output_padding=1)
            )
        return expand

model = UNET()
x = torch.randn(1, 2, 256, 256)
print(model(x))

Output:
Your problem is in the model layer definition.
You defined self.upconv2 = self.expand_block(64, 32, 3, 1) but what you do is concatenating 2 tensors each with 64 channels so in total you get 128.
You should fix the channels of the up-sampling part of the U-Net to match the number of channels after the concatenation.
Doing the mentioned fix will give you:
class UNET(nn.Module):
    def __init__(self, in_channels=2, out_channels=2):
        super().__init__()

        self.conv1 = self.contract_block(in_channels, 32, 3, 1)
        self.conv2 = self.contract_block(32, 64, 3, 1)
        self.conv3 = self.contract_block(64, 128, 3, 1)

        self.upconv3 = self.expand_block(128, 64, 3, 1)
        self.upconv2 = self.expand_block(64 + 64, 32, 3, 1)
        self.upconv1 = self.expand_block(32 + 32, out_channels, 3, 1)

    def __call__(self, x):

        # downsampling part
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)

        upconv3 = self.upconv3(conv3)
        upconv2 = self.upconv2(torch.cat([upconv3, conv2], 1))
        upconv1 = self.upconv1(torch.cat([upconv2, conv1], 1))

        return upconv1

    def contract_block(self, in_channels, out_channels, kernel_size, padding):

        contract = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=1)
            )

        return contract

    def expand_block(self, in_channels, out_channels, kernel_size, padding):

        expand = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.Tanh(),
            torch.nn.ConvTranspose2d(out_channels, out_channels, kernel_size=2, stride=2, padding=1, output_padding=1)
            )
        return expand

model = UNET()
x = torch.randn(1, 2, 256, 256)
print(model(x))

Based on the comment, if you have other spatial dimensions that might not fit the parameters of the convolutions operations you can do one of 2 options:

Start play with the parameter based on the formula at the bottom of the Conv2d so that you will match the input dimension.

You could force pad the target to desired dimension using the following 2 functions:
import torch
import torch.nn as nn
import torch.nn.functional as F

def pad_tensor(source, target):
    """
    Pad source tensor to match target tensor size

    :param source: tensor that need to get padding
    :param target: tensor of the desired shape
    :return: source tensor with shape equal to target
    """

    diff_y = target.size()[2] - source.size()[2]
    diff_x = target.size()[3] - source.size()[3]

    source = F.pad(source, [diff_x // 2, diff_x - diff_x // 2, diff_y // 2, diff_y - diff_y // 2])
    return source

def concatenate_tensors(x1, x2):
    """
    Concatenate both tensors

    :param x1: first tensor to be concatenated
    :param x2: second tensor to be concatenated
    :return: concatenation of both tensors
    """

    x1 = pad_tensor(x1, x2)
    return torch.cat([x1, x2], dim=1)

Now call concatenate_tensors instead of torch.cat this will fix the dimensions to match the size you need.


Instruction: How to assign a name for a pytorch layer?

Input:
Following a previous question, I want to plot weights, biases, activations and gradients to achieve a similar result to this.
Using
for name, param in model.named_parameters():
    summary_writer.add_histogram(f'{name}.grad', param.grad, step_index)

as was suggested in the previous question gives sub-optimal results, since layer names come out similar to '_decoder._decoder.4.weight', which is hard to follow, especially since the architecture is changing due to research. 4 in this run won't be the same in the next, and is really meaningless.
Thus, I wanted to give my own string names to each layer.

I found this Pytorch forum discussion, but no single best practice was agreed upon.
What is the recommended way to assign names to Pytorch layers?
Namely, layers defined in various ways:

Sequential:

self._seq = nn.Sequential(nn.Linear(1, 2), nn.Linear(3, 4),)

Dynamic:

self._dynamic = nn.ModuleList()
    for _ in range(self._n_features):
        self._last_layer.append(nn.Conv1d(in_channels=5, out_channels=6, kernel_size=3, stride=1, padding=1,),)

Direct:

self._direct = nn.Linear(7, 8)

Other ways I didn't think about

I would like to be able to give a string name to each layer, defined in each of the above ways.

Output:
Sequential
Pass an instance of collections.OrderedDict. Code below gives conv1.weights, conv1.bias, conv2.weight, conv2.bias (notice lack of torch.nn.ReLU(), see end of this answer).
import collections

import torch

model = torch.nn.Sequential(
    collections.OrderedDict(
        [
            ("conv1", torch.nn.Conv2d(1, 20, 5)),
            ("relu1", torch.nn.ReLU()),
            ("conv2", torch.nn.Conv2d(20, 64, 5)),
            ("relu2", torch.nn.ReLU()),
        ]
    )
)

for name, param in model.named_parameters():
    print(name)

Dynamic
Use ModuleDict instead of ModuleList:
class MyModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.whatever = torch.nn.ModuleDict(
            {f"my_name{i}": torch.nn.Conv2d(10, 10, 3) for i in range(5)}
        )

Will give us whatever.my_name{i}.weight (or bias) for each created module dynamically.
Direct
Just name it however you want and that's how it will be named
self.my_name_or_whatever = nn.Linear(7, 8)

You didn't think about

If you want to plot weights, biases and their gradients you can go along this route
You can't plot activations this way (or output from activations). Use PyTorch hooks instead (if you want per-layer gradients as they pass through network use this also)

For last task you can use third party library torchfunc (disclaimer: I'm the author) or go directly and write your own hooks.


Instruction: PyTorch conv2d doesn't propagate torch.channels_last memory format

Input:
When I try to use the torch.nn.conv2d operator on tensors which have the channels_last memory format, then the output does not keep this memory format.
I don't understand why, since conv2d is listed in the PyTorch wiki listing operators with channels last support.
Am I missing something?
Code to reproduce (tested with pytorch 1.6.0 and 1.7.0, on CPU, on Ubuntu 20.04):
import torch
import torch.nn.functional as F

N, C, H, W = 10, 4, 64, 128
out_channels = 2
kernel_size = (3, 3)

memory_format = torch.channels_last

tsr = torch.randn(N, C, H, W).to(memory_format=memory_format)
kernel = torch.randn(out_channels, C, *kernel_size).to(memory_format=memory_format)

conv_out = F.conv2d(tsr, kernel)

print(conv_out.is_contiguous(memory_format=memory_format)) # False

Output:
The conv2d operator is listed under the list of GPU operators supporting channels_last. This is not true for the CPU version of conv2d:
If you switch to cuda device, it will return True:
tsr = torch.randn(N, C, H, W).to('cuda', memory_format=memory_format)
kernel = torch.randn(out_channels, C, *kernel_size).to('cuda', memory_format=memory_format)
conv_out = F.conv2d(tsr, kernel)

>>> conv_out.is_contiguous(memory_format=memory_format)
True


Instruction: Best way to cut a pytorch tensor into overlapping chunks?

Input:
If for instance I have:
eg6 = torch.tensor([
    [ 1.,  7., 13., 19.],
    [ 2.,  8., 14., 20.],
    [ 3.,  9., 15., 21.],
    [ 4., 10., 16., 22.],
    [ 5., 11., 17., 23.],
    [ 6., 12., 18., 24.]])
batch1 = eg6
batch2 = -eg6
x = torch.cat((batch1,batch2)).view(2,6,4)

And then I want to slice it up into overlapping chunks, like a sliding window function, and have the chunks be batch-processable. For example, and just looking at the first dimension, I want 1,2,3, 3,4,5, 5,6 (or 5,6,0).
It seems unfold() kind of does what I want. It transposes the last two dimensions for some reason, but that is easy enough to repair. Changing it from shape [2,3,3,4] to [6,3,4] requires a memory copy, but I believe that is unavoidable?
SZ=3
x2 = x.unfold(1,SZ,2).transpose(2,3).reshape(-1,SZ,4)

This works perfectly when x is of shape [2,7,4]. But with only 6 rows, it throws away the final row.
Is there a version of unfold() that can be told to use all data, ideally taking a pad character?
Or do I need to pad x before calling unfold()? What is the best way to do that? I'm wondering if "pad" is the wrong word, as I'm only finding functions that want to put padding characters at both ends, with convolutions in mind.

Aside: Looking at the source of unfold, it seems the strange transpose is there deliberately and explicitly?! For that reason, and the undesired chop behaviour, it made me think the correct answer to my question might be write a new low-level function. But that is too much effort for me, at least for today... (I think a second function for the backwards pass also needs to be written.)

Output:
The operation performed here is similar to what a 1D convolution would behave like. With kernel_size=SZ and stride=2. As you noticed if you don't provide sufficient padding (you're correct on the wording) the last element won't be used.
A general approach (for any SZ and any input shape x.size(1)) is to figure out if padding is necessary, and if so what amount is needed.

The size of the output is given by out = floor((x.size(1) - SZ)/2 + 1).

The number of unused elements is x.size(1) - out*(SZ-1) - 1.

If the number of unused elements is non zero, you need to add a padding of (out+1)*(SZ-1) + 1 - x.size(1)

This example won't need padding:
>>> x = torch.stack((torch.tensor([
            [ 1.,  7., 13., 19.],
            [ 2.,  8., 14., 20.],
            [ 3.,  9., 15., 21.],
            [ 4., 10., 16., 22.],
            [ 5., 11., 17., 23.]]),)*2)

>>> x.shape
torch.Size([2, 5, 4])

>>> out = floor((x.size(1) - SZ)/2 + 1)
2

>>> unused = x.size(1) - out*(SZ-1) - 1
0

While this one will:
>>> x = torch.stack((torch.tensor([
          [ 1.,  7., 13., 19.],
          [ 2.,  8., 14., 20.],
          [ 3.,  9., 15., 21.],
          [ 4., 10., 16., 22.],
          [ 5., 11., 17., 23.],
          [ 6., 12., 18., 24.]]),)*2)

>>> x.shape
torch.Size([2, 6, 4])

>>> out = floor((x.size(1) - SZ)/2 + 1)
2

>>> unused = x.size(1) - out*(SZ-1) - 1
1

>>> p = (out+1)*(SZ-1) + 1 - x.size(1)
1

Now, to actually add padding you could just use torch.cat. Although I am the built-in, nn.functional.pad, would work...
>>> torch.cat((x, torch.zeros(x.size(0), p, x.size(2))), dim=1)
tensor([[[ 1.,  7., 13., 19.],
         [ 2.,  8., 14., 20.],
         [ 3.,  9., 15., 21.],
         [ 4., 10., 16., 22.],
         [ 5., 11., 17., 23.],
         [ 6., 12., 18., 24.],
         [ 0.,  0.,  0.,  0.]],

        [[ 1.,  7., 13., 19.],
         [ 2.,  8., 14., 20.],
         [ 3.,  9., 15., 21.],
         [ 4., 10., 16., 22.],
         [ 5., 11., 17., 23.],
         [ 6., 12., 18., 24.],
         [ 0.,  0.,  0.,  0.]]])


Instruction: Number of learnable parameters of MultiheadAttention

Input:
While testing (using PyTorch's MultiheadAttention), I noticed that increasing or decreasing the number of heads of the multi-head attention does not change the total number of learnable parameters of my model.
Is this behavior correct? And if so, why?
Shouldn't the number of heads affect the number of parameters the model can learn?

Output:
The standard implementation of multi-headed attention divides the model's dimensionality by the number of attention heads.
A model of dimensionality d with a single attention head would project embeddings to a single triplet of d-dimensional query, key and value tensors (each projection counting d2 parameters, excluding biases, for a total of 3d2).
A model of the same dimensionality with k attention heads would project embeddings to k triplets of d/k-dimensional query, key and value tensors (each projection counting d×d/k=d2/k parameters, excluding biases, for a total of 3kd2/k=3d2).

References:
From the original paper:

The Pytorch implementation you cited:


Instruction: How to save a Pytorch Model directly in s3 Bucket?

Input:
The title says it all - I want to save a pytorch model in an s3 bucket. What I tried was the following:
import boto3

s3 = boto3.client('s3')
saved_model = model.to_json()
output_model_file = output_folder + "pytorch_model.json"
s3.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=saved_model)

Unfortunately this doesn't work, as .to_json() only works for tensorflow models. Does anyone know how to do it in pytorch?

Output:
Try serializing model to a buffer and write it to S3:
buffer = io.BytesIO()
torch.save(model, buffer)
s3.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=buffer.getvalue())


Instruction: Focal loss implementation

Input:
In the paper introducing focal loss, they state that the loss function is formulated as such:

Where

I found an implementation of it on a Github page from another author who used it in their paper. I tried the function out on a segmentation problem dataset I have and it seems to work quite well.
Below is the implementation:
def binary_focal_loss(pred, truth, gamma=2., alpha=.25):
    eps = 1e-8
    pred = nn.Softmax(1)(pred)
    truth = F.one_hot(truth, num_classes = pred.shape[1]).permute(0,3,1,2).contiguous()

    pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))
    pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))

    pt_1 = torch.clamp(pt_1, eps, 1. - eps)
    pt_0 = torch.clamp(pt_0, eps, 1. - eps)

    out1 = -torch.mean(alpha * torch.pow(1. - pt_1, gamma) * torch.log(pt_1))
    out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))

    return out1 + out0

The part that I don't understand is the calculation of pt_0 and pt_1. I created a small example for myself to try and figure it out but it still confuses me a bit.
# one hot encoded prediction tensor
pred = torch.tensor([
                     [
                      [.2, .7, .8], # probability
                      [.3, .5, .7], # of
                      [.2, .6, .5]  # background class
                     ],
                     [
                      [.8, .3, .2], # probability
                      [.7, .5, .3], # of
                      [.8, .4, .5]  # class 1
                     ]
                    ])

# one-hot encoded ground truth labels
truth = torch.tensor([
                      [1, 0, 0],
                      [1, 1, 0],
                      [1, 0, 0]
                     ])
truth = F.one_hot(truth, num_classes = 2).permute(2,0,1).contiguous()

print(truth)
# gives me:
# tensor([
#         [
#          [0, 1, 1],
#          [0, 0, 1],
#          [0, 1, 1]
#         ],
#         [
#          [1, 0, 0],
#          [1, 1, 0],
#          [1, 0, 0]
#         ]
#       ])

pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))
pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))

print(pt_0)
# gives me:
# tensor([[
#         [0.2000, 0.0000, 0.0000],
#         [0.3000, 0.5000, 0.0000],
#         [0.2000, 0.0000, 0.0000]
#         ],
#        [
#         [0.0000, 0.3000, 0.2000],
#         [0.0000, 0.0000, 0.3000],
#         [0.0000, 0.4000, 0.5000]
#        ]
#      ])

print(pt_1)
# gives me:
# tensor([[
#          [1.0000, 0.7000, 0.8000],
#          [1.0000, 1.0000, 0.7000],
#          [1.0000, 0.6000, 0.5000]
#         ],
#         [
#          [0.8000, 1.0000, 1.0000],
#          [0.7000, 0.5000, 1.0000],
#          [0.8000, 1.0000, 1.0000]
#         ]
#       ])

What I don't understand is why in pt_0 we are placing zeros where the torch.where statement is false, and in pt_1 we place ones. From how I understood the paper, I would have thought that instead of placing zeros or ones, you would place 1-p.
Can anyone help explain this to me?

Output:
So the part you try to understand is a procedure people usually do when they want zero out the additional calculations that not needed.
Take another look at the formula of pt:

The following code is does exactly this by separate the two condition:
# if y=1
pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))
# otherwise
pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))

Where it set to zero in pt_0 and one in pt_1 will result zero in output thus have no effect for contribute loss value, i.e:
# Because pow(0., gamma) == 0. and log(1.) == 0.
# out1 == 0. if pt_1 == 1.
out1 = -torch.mean(alpha * torch.pow(1. - pt_1, gamma) * torch.log(pt_1))
# out0 == 0. if pt_0 == 0.
out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))

And the reason for pt_0 to using value of p instead of 1-p is the same reason as your last question, i.e:
1 - (1 - p) == 1 - 1 + p == p

So it can later calculate the FL(pt) by:
# -a * pow(1 - (1 - p), gamma )* log(1 - p) == -a * pow(p, gamma )* log(1 - p)
out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))


Instruction: PyTorch Geometric: What is the function utility to split train/validation/test for node classification

Input: I am new to PyTorch Geometric. I know there is a function that gives you the train, test, and validation node mask of a custom ratio in the node classification task. Does anyone know it?

Output: I found the function AddTrainValTestMask


Instruction: model loss does not changing in pytorch

Input:
Im dealing with titanic data with pytorch
these are my model & training code
import torch.nn.functional as F

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1_1=torch.nn.Linear(13, 512)
        self.fc1_2=torch.nn.Linear(512, 64)
        self.fc1_3=torch.nn.Linear(64, 10)

        self.fc2_1=torch.nn.Linear(13, 64)
        self.fc2_2=torch.nn.Linear(64, 512)
        self.fc2_3=torch.nn.Linear(512, 10)

        self.fc3_1=torch.nn.Linear(13, 128)
        self.fc3_2=torch.nn.Linear(128, 128)
        self.fc3_3=torch.nn.Linear(128, 10)

        self.fc_full_1=torch.nn.Linear(30, 64)
        self.fc_full_2=torch.nn.Linear(64, 128)
        self.fc_full_3=torch.nn.Linear(128, 2)

    def forward(self, x):
        x1=self.fc1_1(x)
        x1=F.relu(x1)
        x1=self.fc1_2(x1)
        x1=F.relu(x1)
        x1=self.fc1_3(x1)
        x1=F.relu(x1)

        x2=self.fc2_1(x)
        x2=F.relu(x2)
        x2=self.fc2_2(x2)
        x2=F.relu(x2)
        x2=self.fc2_3(x2)
        x2=F.relu(x2)

        x3=self.fc3_1(x)
        x3=F.relu(x3)
        x3=self.fc3_2(x3)
        x3=F.relu(x3)
        x3=self.fc3_3(x3)
        x3=F.relu(x3)

        x=torch.cat((x1, x2, x3), dim=1)
        x=self.fc_full_1(x)
        x=F.relu(x)
        x=self.fc_full_2(x)
        x=F.relu(x)
        x=self.fc_full_3(x)

        return x

model=Net()

as seen above, they are just fully connected layers
model loss function and optimization
cross ehtropy loss and adam
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model1.parameters(), lr=0.05)

these are training code
for epoch in range(100):
    model.train()
    x_var = Variable(torch.FloatTensor(x_train))
    y_var = Variable(torch.LongTensor(y_train))

    optimizer.zero_grad()
    train_pred = model(x_var)
    loss =criterion(train_pred, y_var)
    loss.backward()
    optimizer.step()

    train_acc=calc_accuracy(train_pred, y_var)
    loss=loss.data.numpy()

and finaly, the accuracy and loss printed
Epoch  0
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  10
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  20
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  30
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  40
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  50
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  60
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  70
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  80
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  90
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121

as seen above, model training loss and valid loss does not change at all.
what does seems to be the problem?

Output:
Your optimizer does not use your model's parameters, but some other model1's.
optimizer = torch.optim.Adam(model1.parameters(), lr=0.05)

BTW, you do not have to use model.train() for each epoch.


Instruction: Libtorch works with g++, but fails with Intel compiler

Input:
I want to use a neural network developed in Python (PyTorch) in a Fortran program. My OS is Ubuntu 18.04.
What I am doing:

save it as torchscript: TurbNN.pt
call it from c++ program: call_ts.cpp, call_ts.h
call c++ program from Fortran program (using bind©): main.f90

I successfully compiled the codes using CMake (3.19.4) and g++ (7.5.0). However, I cannot compile them using Intel compilers (HPCKit 2021.1.0.2684):
# downloaded torchlib
export Torch_DIR=/home/aiskhak/nn/fortran_calls_torchscript3/build/libtorch/share/cmake/Torch/

# set environment for Intel compilers
. /opt/intel/oneapi/setvars.sh

# cmake
cmake \
-DCMAKE_CXX_COMPILER=icpc \
-DCMAKE_Fortran_COMPILER=ifort ..

# make
make

After “cmake” everything looks fine (just like for g++):
-- The CXX compiler identification is Intel 20.2.1.20201112
-- The Fortran compiler identification is Intel 20.2.1.20201112
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/oneapi/compiler/2021.1.1/linux/bin/intel64/icpc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/compiler/2021.1.1/linux/bin/intel64/ifort - skipped
-- Checking whether /opt/intel/oneapi/compiler/2021.1.1/linux/bin/intel64/ifort supports Fortran 90
-- Checking whether /opt/intel/oneapi/compiler/2021.1.1/linux/bin/intel64/ifort supports Fortran 90 - yes
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Torch: /home/aiskhak/nn/fortran_calls_torchscript3/build/libtorch/lib/libtorch.so
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/aiskhak/nn/fortran_calls_torchscript3/build

However, after “make” I am getting:
Scanning dependencies of target call_ts_cpp
[ 25%] Building CXX object CMakeFiles/call_ts_cpp.dir/src/call_ts.cpp.o
[ 50%] Linking CXX shared library lib/libcall_ts_cpp.so
[ 50%] Built target call_ts_cpp
Scanning dependencies of target fortran_calls_ts.x
[ 75%] Building Fortran object CMakeFiles/fortran_calls_ts.x.dir/src/main.f90.o
[100%] Linking Fortran executable bin/fortran_calls_ts.x
lib/libcall_ts_cpp.so: undefined reference to `c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
lib/libcall_ts_cpp.so: undefined reference to `torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&)'
lib/libcall_ts_cpp.so: undefined reference to `c10::DeviceTypeName[abi:cxx11](c10::DeviceType, bool)'
lib/libcall_ts_cpp.so: undefined reference to `torch::jit::Object::find_method(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
lib/libcall_ts_cpp.so: undefined reference to `torch::jit::Method::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&)'
CMakeFiles/fortran_calls_ts.x.dir/build.make:106: recipe for target 'bin/fortran_calls_ts.x' failed
make[2]: *** [bin/fortran_calls_ts.x] Error 1
CMakeFiles/Makefile2:123: recipe for target 'CMakeFiles/fortran_calls_ts.x.dir/all' failed
make[1]: *** [CMakeFiles/fortran_calls_ts.x.dir/all] Error 2
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2

What I am supposed to do with? I fixed similar problem when I had old g++, but my Intel compiler is new.
Below are my codes:
call_ts.cpp
#include "call_ts.h"
#include <torch/script.h>
#include <iostream>
#include <memory>

// c++ function invariant_nn
//
// takes inputs, reads a neural network TurbNN.pt, do a forward pass
// and returns outputs
//
// inputs: 5 tensor invariants I[0:4] (float)
// outputs: 10 tensor basis coefficients G[0:9] (float)

void invariant_nn(float I[], float G[])
{
    // deserialize scriptmodule from a .pt file
    torch::jit::script::Module module;
    const char *arg;
    arg = "../src/TurbNN.pt";
    module = torch::jit::load(arg);

    // create inputs
    std::vector<torch::jit::IValue> inputs;
    float data[] = {I[0], I[1], I[2], I[3], I[4]};
    inputs.push_back(torch::from_blob(data, {1, 5}));
    //std::cout << "inputs\n" << inputs;
    //std::cout << "\n";

    // do forward pass and turn its output into a tensor
    at::Tensor outputs = module.forward(inputs).toTensor();
    //std::cout << "outputs\n" << outputs;
    //std::cout << "\n";

    // return values
    for (int k = 0; k < 10; k++) {
        G[k] = outputs[0][k].item().to<float>();
        //std::cout << "G\n" << G[k];
    }

    return;
}

call_ts.h
#pragma once

/* export macros for library generated by CMake */
#ifndef CALL_TS_API
#include "call_ts_export.h"
#define CALL_TS_API CALL_TS_EXPORT
#endif

#ifdef __cplusplus
extern "C" {
#endif

CALL_TS_API
void invariant_nn(float I[], float G[]);

#ifdef __cplusplus
}
#endif

main. f90
! fortran program main
!
! calls c++ function invariant_nn, which calls a torchscript with
! a neural network
!

program main

    ! define interface to interact with c++
    use, intrinsic :: iso_c_binding, only: c_float
    implicit none
    interface invariant_nn
        subroutine invariant_nn(I, G) bind (c)
            import :: c_float
            real(c_float) :: I(5)
            real(c_float) :: G(10)
        end subroutine
    end interface

    ! fortran program
    real(4) I(5), G(10)

    ! invariants
    I(1) = 1.01
    I(2) = 1.01
    I(3) = 1.01
    I(4) = 1.01
    I(5) = 1.01
    print *, "Tensor invariants ", I

    ! tensor basis coefficients
    call invariant_nn(I, G)
    print *, "Tensor basis coefficients ", G

end program main

CMakeLists.txt
# stop configuration if cmake version is below 3.0
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)

# project name and enabled languages
project(fortran_calls_ts CXX Fortran)

# find libtorch
find_package(Torch REQUIRED)

# if CMAKE_BUILD_TYPE undefined, set it to Release
if(NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE "Release")
endif()

# compiler flags for release mode
set(CMAKE_CXX_FLAGS_RELEASE "-O3")
set(CMAKE_Fortran_FLAGS_RELEASE "-O3")

# set default build paths
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/bin)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/lib)

# generated export header will be placed here
include_directories(${PROJECT_BINARY_DIR})

# c library
add_library(call_ts_cpp SHARED src/call_ts.cpp)

# fortran executable
add_executable(fortran_calls_ts.x src/main.f90)

# linked against c library
target_link_libraries(fortran_calls_ts.x call_ts_cpp)
target_link_libraries(call_ts_cpp "${TORCH_LIBRARIES}")

# we let cmake generate the export header
include(GenerateExportHeader)
generate_export_header(call_ts_cpp BASE_NAME call_ts)

install(TARGETS call_ts_cpp LIBRARY DESTINATION lib ARCHIVE DESTINATION lib)

install(FILES src/call_ts.h ${PROJECT_BINARY_DIR}/call_ts_export.h DESTINATION include)

set_property(TARGET fortran_calls_ts.x PROPERTY CXX_STANDARD 14)

Output:
Do you see cxx11 in the linker errors?  It looks like your libcall_ts_cpp is compiled in a way that expects the new C++11 ABI for std::string but perhaps the library where those functions are implemented was compiled with the old ABI.  Here's a PyTorch forum post about the same problem: https://discuss.pytorch.org/t/issues-linking-with-libtorch-c-11-abi/29510/11
The solution is to download a new copy of the PyTorch libraries built with the new C++11 ABI.


Instruction: What should be the input shape for 3D CNN on a sequence of images?

Input:
https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#conv3d Describes that the input to do convolution on 3D CNN is (N,Cin,D,H,W). Imagine if I have a sequence of images which I want to pass to 3D CNN. Am I right that:

N -> number of sequences (mini batch)
Cin -> number of channels (3 for rgb)
D -> Number of images in a sequence
H -> Height of one image in the sequence
W -> Width of one image in the sequence

The reason why I am asking is that when I stack image tensors: a = torch.stack([img1, img2, img3, img4, img5]) I get shape of a torch.Size([5, 3, 396, 247]), so is it compulsory to reshape my tensor to torch.Size([3, 5, 396, 247]) so that number of channels would go first or it does not matter inside the Dataloader?
Note that Dataloader would add one more dimension automatically which would correspond to N.

Output:
Yes it matters, you need to ensure that dimensions are ordered correctly (assuming you use DataLoader's default collate function). One way to do this is to invoke torch.stack using dim=1 instead of the default of dim=0. For example
a = torch.stack([img1, img2, img3, img4, img5], dim=1)

results in a being the desired shape of [3, 5, 396, 247].


Instruction: RuntimeError: Expected hidden[0] size (2, 1, 100), got (1, 1, 100)

Input:
I put together a LSTM model and it works. But only as long as I set num_layers = 1.
If I set it for example to 2 it gives my a long error message that tells me:
RuntimeError: Expected hidden[0] size (2, 1, 100), got (1, 1, 100)
I am pretty new at Python and deep learning in general, so I could need some advice how to fix my problem.
Code:
import torch
import torch.nn as nn
import numpy as np
import dataset
import time

amount_hidden_layers = 2
amount_neurons_hidden_layers = 100

input_layer_size = 12

output_layer_size = 48
small = True
hours = 20
learning_rate = 1/1000 #0,001
early_stoping = False

train_obs_dataset = dataset.return_observation_dataset(hours=hours, split="train", small=small)
val_obs_dataset = dataset.return_observation_dataset(hours=hours, split="val", small=small)
test_obs_dataset = dataset.return_observation_dataset(hours=hours, split="test", small=small)

class LSTM(nn.Module):
    def __init__(self, input_layer_size, hidden_layer_size, output_layer_size,
                 num_layers):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.input_layer_size = input_layer_size
        self.num_layers = num_layers
        self.output_layer_size = output_layer_size

        # lstm_layers:
        self.lstm = nn.LSTM(input_size=self.input_layer_size, hidden_size=self.hidden_layer_size, num_layers=num_layers)

        self.hidden_cell = (torch.zeros(self.num_layers, 1, self.hidden_layer_size),
                            torch.zeros(self.num_layers, 1, self.hidden_layer_size))
        # output_layer:
        self.linear = nn.Linear(self.hidden_layer_size, self.output_layer_size)

    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)
        predictions = self.linear(lstm_out.view(len(input_seq), -1))
        return predictions[-1]

model = LSTM(input_layer_size=input_layer_size, hidden_layer_size=amount_neurons_hidden_layers,
             output_layer_size=output_layer_size, num_layers=amount_hidden_layers)
loss_function = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

def training(dataset=train_obs_dataset, epochs=5):
    start_time = time.time()
    val_test = float(100000000)                                                     # for early stoping, saves the last evaluation value
    val_worse_counter = 0                                                           # for early stoping, couts how often tests with validation data get worse
    for i in range(epochs):                                                         # every epoch
        for datapoint in range(dataset[0].__len__()):                               # every datapoint
            optimizer.zero_grad()
            model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                                    torch.zeros(1, 1, model.hidden_layer_size))

            y_pred = model(torch.FloatTensor(dataset[0][datapoint]))                # model needs a tensor of shape [sequence][features]
            # squeeze: does not change data, prevents error,
            single_loss = loss_function(y_pred, torch.FloatTensor(dataset[1][datapoint]).squeeze(-1))

            single_loss.backward()
            optimizer.step()
            if datapoint%100 == 1:
                print(f' Time: {time.time() - start_time: 10.1f} sec,   Epoch: {i:4} Datapoint: {datapoint:3} Loss:  {single_loss.item():10.8f} ')
                #print(y_pred[0])
            "early stoping: if tests with the validation data does not get better for 100 steps, the model stops"
            if (early_stoping == True):
                if datapoint%20 == 1:
                    val = test(val_obs_dataset)
                    if (val[0] <= val_test):
                        val_test = val[0]
                        val_worse_counter = 0
                    if(val[0] > val_test):
                        val_worse_counter = val_worse_counter + 1
                    if(val[0] >= 100):
                        break
                    print("Val: ", val[0], val[1][0], "val_test: ", val_test, " val_worse_counter: ", val_worse_counter)

"""
returns a tupel with 1. the average loss and 2. an array with the losses for average losses seperated for all 48 hours
"""
def test(dataset=val_obs_dataset):
    with torch.no_grad():
        losses = []                 #sperated per hour
        loss = 0
        for i in range(48):
            losses.append(0)
        for datapoint in range(dataset.__len__()):
            val_pred = model(torch.FloatTensor(dataset[0][datapoint]))
            "seperates the loss values per hour"
            for i in range(48):
                losses[i] = losses[i] + float(loss_function(val_pred[i], torch.FloatTensor(dataset[1][datapoint]).squeeze(-1)[i]))
            loss = loss + float(loss_function(val_pred, torch.FloatTensor(dataset[1][datapoint]).squeeze(-1)))
        for i in range(48):                                         # calculates the average
            losses[i] = losses[i]/dataset.__len__()
            loss = loss/dataset.__len__()
    return (loss, losses)

print(model)

training(epochs=200)

Thanks in advance for every helpfull comment

Output:
That is because of this line in your training loop:
model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                     torch.zeros(1, 1, model.hidden_layer_size))

Even though you correctly defined hidden_cell in your model, here you hard coded num_layers to be 1 and replaced the one you did correctly.
To fix it, you can change it to this;
model.hidden_cell = (torch.zeros(model.num_layers, 1, model.hidden_layer_size),
                     torch.zeros(model.num_layers, 1, model.hidden_layer_size))

or even remove it totally as you are basically repeating what you've already done. And I don't suppose hidden_layer_size will change mid-training.
Maybe when you batch your data, then leaving it here would make more sense, but it is clear that your batch_size = 1.


Instruction: How can I convert the dimension in the model form 2D to 1D?

Input:
I am beginner of using pytorch. I would like to classify 2d binary array (17 * 20 ) to 8 classes, I am using cross entropy as loss function . I have 512 batch size . the input is 512 batches of size (17 * 20 )and the final outpu 512 batches of size 8. I applied the following model , I would like to get the final output to be only list of length 8. like [512,8] but I got that dim [512,680,8] (I printed the dimensions i git from the model after the code). How can I get [512,8] from that network as final output.
 def __init__(self, M=1):
        super(PPS, self).__init__()
        #input layer
        self.layer1 = nn.Sequential(
             nn.Conv2d(17, 680,  kernel_size=1, stride=1, padding=0),
             nn.ReLU())
        self.drop1 = nn.Sequential(nn.Dropout())
        self.batch1 = nn.BatchNorm2d(680)
        self.lstm1=nn.Sequential(nn.LSTM(
        input_size=20,
        hidden_size=16,
        num_layers=1,
        bidirectional=True,
        batch_first= True))
        self.gru = nn.Sequential(nn.GRU(
            input_size=16*2,
            hidden_size=16,
            num_layers=2,
            bidirectional=True,
            batch_first=True))
        self.fc1 = nn.Linear(16*2,8)

    def forward(self, x):

        out = self.layer1(x)
        out = self.drop1(out)
        out = self.batch1(out)
        out = out.squeeze()
        out,_ = self.lstm1(out)
        out,_ = self.gru(out)
        out = self.fc1(out)
        return out
cov2d torch.Size([512, 680, 20, 1])
drop torch.Size([512, 680, 20, 1])
batch torch.Size([512, 680, 20])
lstm1 torch.Size([512, 680, 32])
lstm2 torch.Size([512, 680, 32])
linear1 torch.Size([512, 680, 8])

Output:
If you want the output to be (512, 8) then you would have to change your last linear layer to something like this:
def __init__(self, M=1):
    ...
    self.gru = nn.Sequential(nn.GRU(
            input_size=16*2,
            hidden_size=16,
            num_layers=2,
            bidirectional=True,
            batch_first=True))
    self.fc1 = nn.Linear(680 * 16*2, 8)

    def forward (self, x):
        ...
        out, _ = self.gru(out)
        out = self.fc1(out.reshape(-1, 680 * 16*2))
        return out

The goal is to reduce the number of feature from 680 * 16 * 2 to 8. You can (and probably should) add more final linear layers that will do this reduction for you.