Instruction: How to rescale a pytorch tensor to interval [0,1]?

Input:
I would like to rescale a pytorch tensor named outmap to the [0, 1] interval. I tried this:
outmap_min = torch.min(outmap, dim=1, keepdim=True)
outmap_max = torch.max(outmap, dim=1, keepdim=True)
outmap = (outmap - outmap_min) / (outmap_max - outmap_min)

And I am getting this error:
TypeError: unsupported operand type(s) for -: 'Tensor' and 'torch.return_types.min'

Output:
There are many ways to answer the question posed in your title (e.g., min-max normalization, or one of many non-linear functions mapping (-infinity, infinity) to [0, 1]). Based on the context of the post, I'm assuming you just want to implement min-max normalization.
torch.min() does not return a tensor; it returns a type akin to a tuple (and in fact the documentation says it's a namedtuple). It's a two-tuple; the first element is the tensor you're looking for (min), and the second element is an indexing tensor specifying the indices of the minimum values (argmin). The same goes for torch.max().
So, suppose your tensor is of size [N, M]. If you're trying to min-max normalize each "row" (dimension 0) based on the min and max of the M elements (columns) in that row, you'd compute the min and max along dimension 1. This would give you N mins and N maxes -- a min and max for each row. You'd then apply the normalization. This is what your code was intended to do, but you have to unpack the return values of torch.min() and torch.max(), discarding the argmin and argmax in this case:
outmap_min, _ = torch.min(outmap, dim=1, keepdim=True)
outmap_max, _ = torch.max(outmap, dim=1, keepdim=True)
outmap = (outmap - outmap_min) / (outmap_max - outmap_min) # Broadcasting rules apply

If your tensor is of size [N, ...], where ... indicates an arbitrary number of dimensions each of arbitrary size, and you want to min-max normalize each row as before, then the solution is a bit more tricky. Unfortunately, torch.min() and torch.max() each only accept a single dimension. If you want to compute extrema along multiple dimensions, you either have to call torch.min() / torch.max() once for each dimension, or you have to reshape / flatten your tensor so that all of the targeted dimensions are merged into one. So if your tensor is of size [N, ...], it just has to be reshaped to size [N, R, 1, 1, ...], where R is the product of the sizes of the remaining dimensions, and the 1's preserve the number of dimensions (this makes broadcasting easier later). For instance, if your tensor is of size [N, A, B, C], you should reshape it to size [N, A*B*C, 1, 1]. Following that, you can compute mins / maxes along the R-sized dimension 1, as before:
flattened_outmap = outmap.view(outmap.shape[0], -1, 1, 1) # Use 1's to preserve the number of dimensions for broadcasting later, as explained
outmap_min, _ = torch.min(flattened_outmap, dim=1, keepdim=True)
outmap_max, _ = torch.max(flattened_outmap, dim=1, keepdim=True)
outmap = (outmap - outmap_min) / (outmap_max - outmap_min) # Broadcasting rules apply

If you wanted to, you could programmatically construct the placeholder 1's as a list whose length is equal to the number of dimensions minus 2 (to account for the [N, R] dimensions)), and then expand the list. e.g.:
flattened_outmap = outmap.view(outmap.shape[0], -1, *([1] * (len(outmap.shape) - 2)))

In more complicated cases where you want the min / max normalization statistics to be computed across multiple non-contiguous dimensions, you'll probably have to permute the tensor prior to reshaping it. This can't be done with a view, so you'd have to create an extra copy of the tensor. This would surely require more memory than simply calling torch.min() / torch.max() once per dimension, but the tensorization of it might make it faster (it'd probably depend on the number of dimensions; you'd have to benchmark it).


Instruction: Convert from tensor to cpu for dictionary values

Input:
I have a dictionary which has the following values and I am trying to convert my tensors in 'train_acc' to a list of float values like the rest so that I can use it to plot graph but I have no idea how to do it.
defaultdict(list,
            {'train_acc': [tensor(0.9889, device='cuda:0', dtype=torch.float64),
              tensor(0.9909, device='cuda:0', dtype=torch.float64),
              tensor(0.9912, device='cuda:0', dtype=torch.float64)],
             'train_loss': [0.049552333343110315,
              0.040933397413570306,
              0.04100083970214572],
             'val_acc': [0.9779669504256384,
              0.9779669504256384,
              0.9779669504256384],
             'val_loss': [0.11118546511442401,
              0.11118546511442401,
              0.11118546511442401]})

Output:
It can be done .cpu() - moving to cpu then get the value of the tensor by .item().
If the dict looks like below:
dict = {

'train_acc': [tensor(0.9889, device='cuda:0', dtype=torch.float64),
              tensor(0.9909, device='cuda:0', dtype=torch.float64),
              tensor(0.9912, device='cuda:0', dtype=torch.float64)],

'train_loss':[0.049552333343110315,
              0.040933397413570306,
              0.04100083970214572],

 'val_acc':   [0.9779669504256384,
              0.9779669504256384,
              0.9779669504256384],

 'val_loss':  [0.11118546511442401,
              0.11118546511442401,
              0.11118546511442401]
}

Then, the below code can modify the dict:
dict['train_acc'] = [x.cpu().item() for x in dict['train_acc']]


Instruction: How to stop updating the parameters of a part of a layer in a CNN model (not the parameters of the whole layer)?

Input: For example, there are ten parameters(filters) in a CNN layer, how I can do to only update five of them and keep the rest unchanged?

Output:
In Pythorch is easy to freeze only part of the net thanks to the requires_grad property:
Here is a simple script:
def freeze_layers(model, num_of_layers):
    freezed = 0
    for layer in model.children():
        freezed += 1
        if layer < num_of_layers:
            layer.requires_grad = False

Consider however that every model has a different structure and it can have leyer nested into each other, with this code you are iterating through the first level of layers in the net.
I suggest you print the layers before, to understand the network structure spefic to your case.


Instruction: Training on GPU produces slightly different results then when trained on CPU

Input:
I just tried my new Script to train a model on the GPU rather than on the CPU. And the training values (loss, metrics) differ to when trained on the CPU.
I was under the impression that running on cuda vs on cpu should not make a difference. Was I wrong or has it something to do with my code?
Using pytorch=10.1.2 and cudatoolkit=10.1

Output:
How big a difference? Tiny differences are to be expected. Order of  commutative operations matters for floating point computations. That is:
serialised_total = 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1
parallelised_total = (0.1 + 0.1 + 0.1) + (0.1 + 0.1 + 0.1)
# No actual parallelisation is performed. The above is just example of how
# the serialised summation could be broken up into two separate summations.
assert serialised_total != parallelised_total
#                   0.6 != 0.6000000000000001

The results of each side of the equation are still very very close, they're just not exactly the same. See this answer for why.
If you are using the GPU then it will be making use of parallelisation, and so the order of operations will not be the same. For instance, if you sum a series of floating point values then you can speed things up by breaking the list up into chunks and sending each chunk to a different core to be summed. You can then sum the results of each chunk. This will be much quicker, but the order of operations will be different than if you summed the values serially.
In the above example, it is the "parallelised" total that is less accurate than the "serialised" total. This is not a rule, and sometimes it will be the "parallelised" total that is more accurate. For example:
# n = 8
serialised_total = 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1
parallelised_total = (0.1 + 0.1 + 0.1 + 0.1) + (0.1 + 0.1 + 0.1 + 0.1)
assert serialised_total != parallelised_total
#    0.7999999999999999 != 0.8

Without knowing more about your problem, any answers are just speculation about the issue. Including this one.


Instruction: how to transform a posting file to a pytorch tensor

Input:
Is there a python package that transforms a postings file to a pytorch tensor?
By a posting file I mean a csv file with the following format:
"docID" ,"wordID" ,"count"
I also have a dictionary.txt which associates each wordID to a word.
At the end, my text data consists of postings file and a dictionary and I want to use it with a deep learning model that I have implemented with Pytorch.

Output:
No, you have to do it youself. You can simply convert each elemnt into a pytorch tensor or use the pytorch dataset api like this.
import csv
import torch
from torch.utils.data import Dataset
from typing import List, NamedTuple

CsvRowItem = NamedTuple("CsvRowItem", [
    ("docId", int),
    ("wordId", int),
    ("count", int)
])

data: List[CsvRowItem] = []
with open("data.csv", mode='r') as file:
    reader = csv.reader(file)
    next(reader) # Skip the header
    for row in reader:
        data.append(
            CsvRowItem(docId=int(row[0]),
                       wordId=int(row[1]),
                       count=int(row[2]))
        )

class YourDataset(Dataset):

    def __init__(self, data: List[CsvRowItem]):
        self.data = data

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index: int) -> torch.IntTensor:
        item = self.data[index]
        return torch.IntTensor([item.docId, item.wordId, item.count])

dataset = YourDataset(data=data)
print(f"Length of data: {len(dataset):,}")
print(dataset[0])


Instruction: What is the purpose of with torch.no_grad():

Input:
Consider the following code for Linear Regression implemented using PyTorch:
X is the input, Y is the output for the training set, w is the parameter that needs to be optimised
import torch

X = torch.tensor([1, 2, 3, 4], dtype=torch.float32)
Y = torch.tensor([2, 4, 6, 8], dtype=torch.float32)

w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True)

def forward(x):
    return w * x

def loss(y, y_pred):
    return ((y_pred - y)**2).mean()

print(f'Prediction before training: f(5) = {forward(5).item():.3f}')

learning_rate = 0.01
n_iters = 100

for epoch in range(n_iters):
    # predict = forward pass
    y_pred = forward(X)

    # loss
    l = loss(Y, y_pred)

    # calculate gradients = backward pass
    l.backward()

    # update weights
    #w.data = w.data - learning_rate * w.grad
    with torch.no_grad():
        w -= learning_rate * w.grad

    # zero the gradients after updating
    w.grad.zero_()

    if epoch % 10 == 0:
        print(f'epoch {epoch+1}: w = {w.item():.3f}, loss = {l.item():.8f}')

What does the 'with' block do? The requires_grad argument for w is already set to True. Why is it then being put under a with torch.no_grad() block?

Output: The requires_grad argument tells PyTorch that we want to be able to calculate the gradients for those values. However, the with torch.no_grad() tells PyTorch to not calculate the gradients, and the program explicitly uses it here (as with most neural networks) in order to not update the gradients when it is updating the weights as that would affect the back propagation.


Instruction: How to setup CMake project to use PyTorch C++ API installed via Conda

Input:
I have Miniconda3 on a Linux system (Ubuntu 22.04). The environment has Python 3.10 as well as a functioning (in Python) installation of PyTorch (installed following official instructions).
I would like to setup a CMake project that uses PyTorch C++ API. The reason is not important and also I am aware that it's beta (the official documentation states that), so instability and major changes are not excluded.
Currently I have this very minimal CMakeLists.txt:
cmake_minimum_required(VERSION 3.19) # or whatever version you use
project(PyTorch_Cpp_HelloWorld CXX)

set(PYTORCH_ROOT "/home/$ENV{USER}/miniconda/envs/ML/lib/python3.10/site-packages/torch")
list(APPEND CMAKE_PREFIX_PATH "${PYTORCH_ROOT}/share/cmake/Torch/")
find_package(Torch REQUIRED CONFIG)

...
# Add executable
# Link against PyTorch library

When I try to configure the project I'm getting error:

CMake Error at CMakeLists.txt:21 (message):   message called with
incorrect number of arguments
-- Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR)
-- Found Threads: TRUE   CMake Warning at /home/USER/miniconda/envs/ML/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/protobuf.cmake:88
(message):   Protobuf cannot be found.  Depending on whether you are
building Caffe2 or   a Caffe2 dependent library, the next warning /
error will give you more   info. Call Stack (most recent call first):
/home/USER/miniconda/envs/ML/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:56
(include)
/home/USER/miniconda/envs/ML/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68
(find_package)   CMakeLists.txt:23 (find_package)
CMake Error at
/home/USER/miniconda/envs/ML/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:58
(message):   Your installed Caffe2 version uses protobuf but the
protobuf library cannot   be found.  Did you accidentally remove it,
or have you set the right   CMAKE_PREFIX_PATH? If you do not have
protobuf, you will need to install   protobuf and set the library path
accordingly. Call Stack (most recent call first):
/home/USER/miniconda/envs/ML/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68
(find_package)   CMakeLists.txt:23 (find_package)

I installed libprotobuf (again via conda) but, while I can find the library files, I can't find any *ProtobufConfig.cmake or anything remotely related to protobuf and its CMake setup.
Before I go fight against wind mills I would like to ask here what the proper setup would be. I am guessing building from source is always an option, however this will pose a huge overhead on people, who I collaborate with.

Output:
Using this conda env:
name: pytorch_latest
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - pytorch=1.11.0
  - torchvision
  - torchaudio
  - cpuonly

I copied the small example from here and got it to run. The key was to set the correct library directories (both torch in site-packages, but also the lib folder of the environment). The cmake file is written so that the folders are automatically found :
example-app.cpp
#include <torch/torch.h>
#include <iostream>

int main() {
  torch::Tensor tensor = torch::rand({2, 3});
  std::cout << tensor << std::endl;
}

CMakeLists.txt:
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

#Add the torch library directory
list(APPEND CMAKE_PREFIX_PATH "$ENV{CONDA_PREFIX}/lib/python3.10/site-packages/torch")

#This is needed to be able to find the mkl and other dependent libraries
link_directories("$ENV{CONDA_PREFIX}/lib")
set(ENV{MKLROOT} "$ENV{CONDA_PREFIX}/lib")
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app example-app.cpp)

#We need to add pthread and omp manually here
target_link_libraries(example-app "${TORCH_LIBRARIES}" pthread omp)

set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

Exact environment (in case there are problems with reproducibility):
name: pytorch_latest
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_kmp_llvm
  - blas=2.115=mkl
  - blas-devel=3.9.0=15_linux64_mkl
  - brotlipy=0.7.0=py310h5764c6d_1004
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2022.5.18.1=ha878542_0
  - certifi=2022.5.18.1=py310hff52083_0
  - cffi=1.15.0=py310h0fdd8cc_0
  - charset-normalizer=2.0.12=pyhd8ed1ab_0
  - cpuonly=2.0=0
  - cryptography=37.0.1=py310h9ce1e76_0
  - ffmpeg=4.3=hf484d3e_0
  - freetype=2.10.4=h0708190_1
  - giflib=5.2.1=h36c2ea0_2
  - gmp=6.2.1=h58526e2_0
  - gnutls=3.6.13=h85f3911_1
  - idna=3.3=pyhd8ed1ab_0
  - jpeg=9e=h166bdaf_1
  - lame=3.100=h7f98852_1001
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libblas=3.9.0=15_linux64_mkl
  - libcblas=3.9.0=15_linux64_mkl
  - libdeflate=1.10=h7f98852_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.1.0=h8d9b700_16
  - libgfortran-ng=12.1.0=h69a702a_16
  - libgfortran5=12.1.0=hdcd56e2_16
  - libiconv=1.17=h166bdaf_0
  - liblapack=3.9.0=15_linux64_mkl
  - liblapacke=3.9.0=15_linux64_mkl
  - libnsl=2.0.0=h7f98852_0
  - libpng=1.6.37=h21135ba_2
  - libstdcxx-ng=12.1.0=ha89aaad_16
  - libtiff=4.4.0=h0fcbabc_0
  - libuuid=2.32.1=h7f98852_1000
  - libuv=1.43.0=h7f98852_0
  - libwebp=1.2.2=h3452ae3_0
  - libwebp-base=1.2.2=h7f98852_1
  - libxcb=1.13=h7f98852_1004
  - libzlib=1.2.12=h166bdaf_0
  - llvm-openmp=14.0.4=he0ac6c6_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - mkl=2022.1.0=h84fe81f_915
  - mkl-devel=2022.1.0=ha770c72_916
  - mkl-include=2022.1.0=h84fe81f_915
  - ncurses=6.3=h27087fc_1
  - nettle=3.6=he412f7d_0
  - numpy=1.22.4=py310h4ef5377_0
  - openh264=2.1.1=h780b84a_0
  - openjpeg=2.4.0=hb52868f_1
  - openssl=3.0.3=h166bdaf_0
  - pillow=9.1.1=py310he619898_1
  - pip=22.1.2=pyhd8ed1ab_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pycparser=2.21=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pysocks=1.7.1=py310hff52083_5
  - python=3.10.4=h2660328_0_cpython
  - python_abi=3.10=2_cp310
  - pytorch=1.11.0=py3.10_cpu_0
  - pytorch-mutex=1.0=cpu
  - readline=8.1=h46c0cb4_0
  - requests=2.27.1=pyhd8ed1ab_0
  - setuptools=62.3.2=py310hff52083_0
  - sqlite=3.38.5=h4ff8645_0
  - tbb=2021.5.0=h924138e_1
  - tk=8.6.12=h27826a3_0
  - torchaudio=0.11.0=py310_cpu
  - torchvision=0.12.0=py310_cpu
  - typing_extensions=4.2.0=pyha770c72_1
  - tzdata=2022a=h191b570_0
  - urllib3=1.26.9=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xz=5.2.5=h516909a_1
  - zlib=1.2.12=h166bdaf_0
  - zstd=1.5.2=h8a70e8d_1


Instruction: Is GradScaler necessary with Mixed precision training with pytorch?

Input:
So going the AMP: Automatic Mixed Precision Training tutorial for Normal networks, I found out that there are two versions, Automatic and GradScaler. I just want to know if it's advisable / necessary to use the GradScaler with the training becayse it is written in the document that:

Gradient scaling helps prevent gradients with small magnitudes from flushing to zero (“underflowing”) when training with mixed precision.

scaler = torch.cuda.amp.GradScaler()
for epoch in range(1):
    for input, target in zip(data, targets):
        with torch.cuda.amp.autocast():
            output = net(input)
            loss = loss_fn(output, target)

        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
        opt.zero_grad()

Also, looking at NVIDIA Apex Documentation for PyTorch, they have used it as,
from apex import amp

model, optimizer = amp.initialize(model, optimizer)

loss = criterion(…)
with amp.scale_loss(loss, optimizer) as scaled_loss:
    scaled_loss.backward()
optimizer.step()

I think this is what GradScaler does too so I think it is a must. Can someone help me with the query here.

Output:
Short answer: yes, your model may fail to converge without GradScaler().
There are three basic problems with using FP16:

Weight updates: with half precision, 1 + 0.0001 rounds to 1. autocast() takes care of this one.
Vanishing gradients: with half precision, anything less than (roughly) 2e-14 rounds to 0, as opposed to single precision 2e-126. GradScaler() takes care of this one.
Explosive loss: similar to the above, overflow is also much more likely with half precision. This is also managed by autocast() context.


Instruction: module 'torch' has no attribute 'has_mps'

Input:
I just followed a youtube video that teaches how to install PyTorch nightly for MacBook to accelerate by m1 chip. However, I came across a  problem really wierd. I can see in the jupyter notebook that torch.has_mps = True. But in jupyter notebook in vscode, it shows that module 'torch' has no attribute 'has_mps'.
Can anyone kindly tell me why? really confusing.

Output:
Just make sure you installed the nightly build of PyTorch.
Apple Silicon support in PyTorch is currently available only in nightly builds.
e.g., if you're using conda, try this:
conda install pytorch torchvision -c pytorch-nightly

or with pip
pip3 install --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu

See more here:
https://pytorch.org/get-started/locally/
The script to verify that you're using the correct version is like you write, you can simply open python REPL in your env where you installed the above:
import torch
torch.has_mps

And you should get
True

To select the device, use "mps" instead of "cuda" (what you see in tutorials):
device = "mps" if torch.has_mps else "cpu"
print(f'Using device: {device}')

P.S. Although the guide suggest to install torchaudio, it will not work, at least with conda environments.
P.P.S. Also, try with the environment in this github repo:
https://github.com/causevic/mlboxm1/blob/main/pytorch_mac_m1.yml


Instruction: RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast

Input:
I am trying to implement U^2 Net for Salient Object detection. Since this code is not optimised for training, following this official documentation for AMP, I have made some changes to the original code in my fork to check the effects.
I have used the code exactly and when you run my version of training code on colab
as :
! git clone https://github.com/deshwalmahesh/U-2-Net
%cd ./U-2-Net/
!python u2net_train.py

It'll throw you some error. The whole stack is posted in the end. I dug up and found that it is due to the custom loss function as muti_bce_loss_fusion which the authors have used as:
bce_loss = nn.BCELoss(size_average=True)

def muti_bce_loss_fusion(d0, d1, d2, d3, d4, d5, d6, labels_v):

    loss0 = bce_loss(d0,labels_v)
    loss1 = bce_loss(d1,labels_v)
    loss2 = bce_loss(d2,labels_v)
    loss3 = bce_loss(d3,labels_v)
    loss4 = bce_loss(d4,labels_v)
    loss5 = bce_loss(d5,labels_v)
    loss6 = bce_loss(d6,labels_v)

    loss = loss0 + loss1 + loss2 + loss3 + loss4 + loss5 + loss6
    return loss0, loss

Also, in the last line i.e line 526 of the model definition, the model returns 7 sigmoid values which are passed to the loss function.
F.sigmoid(d0), F.sigmoid(d1), F.sigmoid(d2), F.sigmoid(d3), F.sigmoid(d4), F.sigmoid(d5), F.sigmoid(d6)

Now what can be done to avoid this error?
Error trace
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:780: UserWarning: Note that order of the arguments: ceil_mode and return_indices will changeto match the args list in nn.MaxPool2d in a future release.
  warnings.warn("Note that order of the arguments: ceil_mode and return_indices will change"
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3704: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:1944: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
Traceback (most recent call last):
  File "u2net_train.py", line 148, in <module>
    loss2, loss = muti_bce_loss_fusion(d0, d1, d2, d3, d4, d5, d6, labels_v)
  File "u2net_train.py", line 33, in muti_bce_loss_fusion
    loss0 = bce_loss(d0,labels_v)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py", line 612, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 3065, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast.
Many models use a sigmoid layer right before the binary cross entropy layer.
In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits
or torch.nn.BCEWithLogitsLoss.  binary_cross_entropy_with_logits and BCEWithLogits are
safe to autocast.

Output: The main reason why it was due to unstable nature of Sigmoid + BCE. Referring to documentation and torch community, all I had to to do was to replace the models from F.sigmoid(d0)... to d0..... and then in turn replace nn.BCELoss(size_average=True) with nn.BCEWithLogitsLoss(size_average=True). Now the model is running fine.


Instruction: Find the biggest of two pytorch tensor on size

Input:
How to find the biggest of two pytorch tensors on size
>>> tensor1 = torch.empty(0)
>>> tensor2 = torch.empty(1)
>>> tensor1
tensor([])
>>> tensor2
tensor([5.9555e-34])

torch.maximum is returrning the empty tensor as the biggest tensor
>>> torch.maximum(tensor1,tensor2)
tensor([])

Is there a way to find the biggest tensor among two tensors (mostly 1d), base on the number of elements in the tensor.

Output:
Why not comparing their first dimension size? To do so you can use equivalents: x.size(0), x.shape[0], and len(x). To return the tensor with longest size, you can use the built-in max function with the key argument:
>>> max((tensor1, tensor2), key=len)


Instruction: Keras to Pytorch -> troubles with layers and shape

Input:
I am in the process of converting a Keras model to PyTorch and would need your help.
Keras Code:
def model(input_shape):
    input_layer = keras.layers.Input(input_shape)
    conv1 = keras.layers.Conv1D(filters=16, kernel_size=3, padding="same")(input_layer)
    conv1 = keras.layers.BatchNormalization()(conv1)
    conv1 = keras.layers.ReLU()(conv1)
    global_average_pooling = keras.layers.GlobalAveragePooling1D()(conv1)
    output_layer = keras.layers.Dense(number_of_classes, activation="softmax")(global_average_pooling )
    return keras.models.Model(inputs=input_layer, outputs=output_layer)

Summary of Model:

My Code Is:
class model(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        #number_of_classes = data_config.number_of_classes
        self.conv1 = nn.Conv1d(256,128,1) # PyTorch does not support same padding,
        self.bn1=nn.BatchNorm1d(128)
        #self.relu=nn.functional.relu_(16)
        self.avg = nn.AvgPool1d(1)

    def forward(self,x):
        x = self.conv1(x)
        x = self.bn1(x)
        #x = self.relu(x)
        x = self.avg(x)
        output = F.log_softmax(x)
        return output

Could someone please help me with this?
Greets!

Output:
My Current Code is:
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1 = nn.Conv1d(256,128,1)
        self.batch1 = nn.BatchNorm1d(128)
        self.avgpl1 = nn.AvgPool1d(1, stride=1)
        self.fc1 = nn.Linear(128,3)

    def forward(self,x):
        x = self.conv1(x)
        x = self.batch1(x)
        x = F.relu(x)
        x = self.avgpl1(x)
        x = torch.flatten(x,1)
        x = F.log_softmax(self.fc1(x))
        return x

Keras Model Parameters:

And my Parameters currently are:


Instruction: pip install does not work with python3 on windows

Input:
I am getting acquainted with python development and it's been some time since I wrote python code. I am setting up my IDE (Pycharm) and Python3 binaries on Windows 10.
I want to start working with the pytorch library and I am used to before in python2 just typing pip install  and it works fine.
Now it seems pip is not installed by default and has been replaced by something called conda?
What's the best way to install the pytorch package from the command line?
here is a screenshot link

Output:
PyTorch Documentation does have a selector that will give you the commands for pip and conda.
However, Python3 should have pip installed, it may just be pip3 (that's what it was for me).


Instruction: Vectorised pairwise distance

Input:
TLDR: given two tensors t1 and t2 that represent b samples of a tensor with shape c,h,w (i.e, every tensor has shape b,c,h,w), i'm trying to calculate the pairwise distance between t1[i] and t2[j] for all i,j efficiently

some more context - I've extracted ResNet18 activations for both my train and test data (CIFAR10) and I'm trying to implement k-nearest-neighbours. A possible pseudo-code might be:
for te in test_activations:
    distances = []
    for tr in train_activations:
        distances.append(||te-tr||)
    neighbors = k_smallest_elements(distances)
    prediction(te) = majority_vote(labels(neighbors))

I'm trying to vectorise this process given batches from the test and train activations datasets. I've tried iterating the batches (and not the samples) and using torch.cdist(train_batch,test_batch), but I'm not quite sure how this function handles multi-dimensional tensors, as in the documentation it states

torch.cdist(x1, x2,...):
If x1 has shape BxPxM and x2 has shape BxRxM then the output will have shape BxPxR

Which doesn't seem to handle my case (see below)

A minimal example can be found here:
b,c,h,w = 1000,128,28,28 # actual dimensions in my problem
train_batch = torch.randn(b,c,h,w)
test_batch = torch.randn(b,c,h,w)
d = torch.cdist(train_batch,test_batch)

You can think of test_batch and train_batch as the tensors in the for loop for test_batch in train: for train_batch in test:...

EDIT: im adding another example:
both t1[i] and t2[j] are tensors shaped (c,h,w), and the distance between them is a scalar d. so for example, if we have
t1 = torch.randn(2,128,28,28)
t2 = torch.randn(2,128,28,28)

the distance matrix would look something like
[[d(t1[0],t2[0]), d(t1[0],t2[1])],
 [d(t1[1],t2[0]), d(t1[1],t2[1])]]

and have a shape (2,2) (or (b,b) more generally)
where d is the scalar distance between the two tensors t1[i] and t2[j].

Output:
It is common to have to reshape your data before feeding it to a builtin PyTorch operator. As you've said torch.cdist works with two inputs shaped (B, P, M) and (B, R, M) and returns a tensor shaped (B, P, R).
Instead, you have two tensors shaped the same way: (b, c, h, w). If we match those dimensions we have: B=b, M=c, while P=h*w (from the 1st tensor) and R=h*w (from the 2nd tensor). This requires flattening the spatial dimensions together and swapping the last two axes. Something like:
>>> x1 = train_batch.flatten(2).transpose(1,2)
>>> x2 = test_batch.flatten(2).transpose(1,2)

>>> d = torch.cdist(x1, x2)

Now d contains distance between all possible pairs (train_batch[b, :, iy, ix], test_batch[b, :, jy, jx]) and is shaped (b, h*w, h*w).
You can then apply a knn using argmax to retrieve the k closest neighbour from one element of the training batch to the test batch.


Instruction: How to prevent NVIDIA from automatically upgrading the driver on Ubuntu?

Input:
I was training models last night on my Ubuntu workstation, and then woke up this morning and saw this message:
Failed to initialize NVML: Driver/library version mismatch

Apparently the NVIDIA system driver automatically updated itself, and now I need to reboot the machine to use my GPUs... How do I prevent automatic updates from NVIDIA?

Output:
I think I have had the same issue. It is because of so-called unattended upgrades on Ubuntu.
Solution 1: check the changed packages and revert the updates
Check the apt history logs
less /var/log/apt/history.log

Then you can see what packages have changed. Use apt or aptitude to revert the changes.
Solution 2: disable unattended upgrades
Use this guide to disable unattended upgrades. Please consider if this solution works for you as you have to install security updates manually after this change.
Solution 3: hold specific packages
Use this guide on how to hold certain packages. Read the apt history as mentioned above to determine which packages you have to put on hold. Probably CUDA related packages such as nvidia-cuda-toolkit. Hard to say since some information is missing from your post. You can see all nvidia related packages like this
dpkg -l *nvidia*

I hope at least one of my solutions works for you :)
P.S. you have to change the title. NVIDIA isn't upgrading anything on your system by itself. Ubuntu is the one causing your trouble ;)


Instruction: how to expand the dimensions of a tensor in pytorch

Input:
i'm a newcomer for pytorch. if i have a tensor like that:

A = torch.tensor([[1, 2, 3], [ 4, 5, 6]]),

but my question is how to get a 2 dimensions tensor like:
B =  Tensor([[[1, 2, 3],
                           [4, 5, 6]],

                          [[1, 2, 3],
                           [4, 5, 6]]])

Output:
You can concatenate ...
A
tensor([[[1., 2., 3.],
         [4., 5., 6.]]])
B = torch.cat((a, a))

B
tensor([[[1., 2., 3.],
         [4., 5., 6.]],

        [[1., 2., 3.],
         [4., 5., 6.]]])


Instruction: FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version

Input:
I am building a model to train  it for binary classification. While processing the data before feeding it to the model i come across this warning
FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.

Here is my code
import torch
import torch.nn as nn
import matplotlib.pyplot as pyp
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix,accuracy_score
from sklearn.metrics import precision_score,recall_score,roc_curve,auc,roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle

#loading the dataset
path='G:/My Drive/datasets/bank.csv'
df=pd.read_csv(path)
print(df.head(5))
print(df.shape)

#distirbuting the target values
print("Distribution of Target Values in Dataset -")
df.deposit.value_counts()

#check f we have na values in the datset
df.isna().sum()

#extracting columns whith strings
cartegorical_columns=df.select_dtypes(include='object').columns
print('cartegprical columns:',list(cartegorical_columns))

#for all cartegorical column if values in(yes/no) convert into a 1/10 flag
for col in cartegorical_columns:
    if df[col].nunique()==2:
        df[col]=np.where(df[col]=='yes',1,0)

print(df.head(5))

#for the remaining cartegorical values that have no binary values
#crate one hot encoded version of the dataset
new_df=pd.get_dummies(df)

#define the target and predictors for the model
target='deposit'
predictors=set(new_df.columns) - set([target])
print('new_df shape:',new_df.shape)
print(new_df[predictors].head())

The specific error
FutureWarning: Passing a set as an indexer is deprecated and will raise in a future version. Use a list instead.
  print(new_df[predictors].head())

What could be raising this error in my code and how can i solve it

Output:
you are trying to access the new_df with predictors which is set.
convert it to list.
example:
print(new_df[list(predictors)].head())


Instruction: How to use pytorch to perform gradient descent, and print out the true minimum?

Input:
I'm trying to use a gradient method to find a minimizer of a function f, where x(k+1)=x(k)-α▽f(x), and α=0.1. The function f is f(x) = 2*(x^3+1)^(-1/2), x(0)=1
Here is the pytorch sample
import torch
import torch.nn as nn
from torch import optim

alpha = 0.1

class MyModel(nn.Module):
     def __init__(self):
          super().__init__()
          self.x = 1

def forward(self, x):
     return 2*pow((pow(x,3)+1),(-1/2))

model = MyModel()
optimizer = optim.SGD(model.parameters(),lr=alpha)
terminationCond = False

while not terminationCond:
     f = model.forward(x)
     f.backward()
     optimizer.step()
     if x.grad < 0.001:
           terminationCond = True
     optimizer.zero_grad()

But I cannot output the correct value of x, how to modify my code in order to find a minizer of a function f?

Output:
There are a few things that need to be considered ...

For x to be a parameter (model.parameter()) x should be nn.Parameter(torch.as_tensor([1.]))
You are passing what (and why) x to the forward method?
The function f(x) = 2*(x^3+1)^(-1/2) is inversely proportional to x. When x goes up in value, the function goes down. The SGD will minimize the value of the function, hence maximize x.

f(0) = 2.0
f(1) = 1.41421
f(2) = 0.66666

Here is a working example of minimizing f(x) ie maximizing x.
import torch
import torch.nn as nn
from torch import optim

alpha = 0.1

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.x = nn.Parameter(torch.as_tensor([1.0]))
    def forward(self):
        return 2*pow((pow(self.x, 3)+1), (-1/2))

model = MyModel()
optimizer = optim.SGD(model.parameters(), lr=alpha)
terminationCond = False
print(model.x)

f = model()
f.backward()
optimizer.step()
optimizer.zero_grad()
print(model.x)


Instruction: Pytorch NLP Huggingface: model not loaded on GPU

Input:
I have this code that init a class with a model and a tokenizer from Huggingface.
On Google Colab this code works fine, it loads the model on the GPU memory without problems.
On Google Cloud Platform it does not work, it loads the model on gpu, whatever I try.

class OPT:
    def __init__(self, model_name: str = "facebook/opt-2.7b", use_gpu: bool = False):
        self.model_name = model_name
        self.use_gpu = use_gpu and torch.cuda.is_available()
        print(f"Use gpu:: {self.use_gpu}")

        if self.use_gpu:
            print("Using gpu")
            self.model = AutoModelForCausalLM.from_pretrained(
                self.model_name, torch_dtype=torch.float16
            ).cuda()
        else:
            print("Using cpu")
            self.model = AutoModelForCausalLM.from_pretrained(
                self.model_name, torch_dtype=torch.float32, low_cpu_mem_usage=True
            )

        # the fast tokenizer currently does not work correctly
        self.tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)

The printed output is correct:

Use gpu:: True
Using gpu

But the nvidia-smi says that there is no process running on the gpu:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And with htop I can see that the process is using the cpu ram.

Output:
You should use the .to(device) method like this:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = nameofyourmodel.to(device)


Instruction: How to learn multiple binary classifiers?

Input:
I have an input of shape 14 x 10 x 128 x 128, where 14 is batch_size, 10 is the sequence_length and each item in the sequence is of shape 128 x 128. I want to learn to map this input to output of shape 14 x 10 x 128, i.e., for each item in the sequence I want to learn 128-binary classifiers.
Does the following model make sense? So, first I reshape my input to 140 x 128 x 128 and then pass it through the model and reshape the output back to 14 x 10 x 128.
classifier = nn.Sequential(
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Conv1d(128, 128, 1),
    nn.ReLU(),
    nn.BatchNorm1d(128),
    nn.Conv1d(128, 1, 1)
)

Thank you.

Output:
Not really convinced a 1D convolution will get you anywhere since it reasons in two dimensions only. In your case, you are dealing with a sequence of 2D elements. Naturally a nn.Conv2d would seem more appropriate for this kind of task.
You are looking to do a one-to-one mapping with your sequence elements and can therefore consider each one of them as an independent instance. Then a straightforward approach is to simply collapse the sequence into the batch axis and use a CNN coupled with a fully-connected layer.
Here is a very minimal example with a single layer:
model = nn.Sequential(nn.Conv2d(1, 8, 2),
                      nn.ReLU(),
                      nn.AdaptiveAvgPool2d(1),
                      nn.Flatten(),
                      nn.LazyLinear(128))

This requires you to reshape the tensor before and after to collapse and expand the sequence dimensions:
>>> x = torch.rand(14, 10, 128, 128)
>>> y = model(x.view(-1,1,128,128)).view(-1,10,128)


Instruction: Loss for Multi-label Classification

Input:
I am working on a multi-label classification problem. My gt labels are of shape 14 x 10 x 128, where 14 is the batch_size, 10 is the sequence_length, and 128 is the vector with values 1 if the item in sequence belongs to the object and 0 otherwise.
My output is also of same shape: 14 x 10 x 128. Since, my input sequence was of varying length I had to pad it to make it of fixed length 10. I'm trying to find the loss of the model as follows:
total_loss = 0.0
unpadded_seq_lengths = [3, 4, 5, 7, 9, 3, 2, 8, 5, 3, 5, 7, 7, ...] # true lengths of sequences

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()

for data in training_dataloader:
    optimizer.zero_grad()

    # shape of input 14 x 10 x 128
    output = model(data)

    batch_loss = 0.0
    for batch_idx, sequence in enumerate(output):
        # sequence shape is 10 x 128
        true_seq_len = unpadded_seq_lengths[batch_idx]

        # only keep unpadded gt and predicted labels since we don't want loss to be influenced by padded values
        predicted_labels = sequence[:true_seq_len, :] # for example, 3 x 128
        gt_labels =  gt_labels_padded[batch_idx, :true_seq_len, :] # same shape as above, gt_labels_padded has shape 14 x 10 x 128

        # loop through unpadded predicted and gt labels and calculate loss
        for item_idx, predicted_labels_seq_item in enumerate(predicted_labels):

            # predicted_labels_seq_item and gt_labels_seq_item are 1D vectors of length 128
            gt_labels_seq_item = gt_labels[item_idx]
            current_loss = criterion(predicted_labels_seq_item, gt_labels_seq_item)
            total_loss += current_loss
            batch_loss += current_loss

    batch_loss.backward()
    optimizer.step()

Can anybody please check to see if I'm calculating loss correctly. Thanks
Update:
Is this the correct approach for calculating accuracy metrics?
# batch size: 14
# seq length: 10

for epoch in range(10):
    TP = FP = TN = FN = 0.
    for x, y, mask in tr_dl:
        # mask shape: (10,)
        out = model(x) # out shape: (14, 10, 128)
        y_pred = (torch.sigmoid(out) >= 0.5).float().type(torch.int64) # consider all predictions above 0.5 as 1, rest 0
        y_pred = y_pred[mask] # y_pred shape: (14, 10, 10, 128)
        y_labels = y[mask] # y_labels shape: (14, 10, 10, 128)

        # do I flatten y_pred and y_labels?
        y_pred = y_pred.flatten()
        y_labels = y_labels.flatten()

        for idx, prediction in enumerate(y_pred):
            if prediction == 1 and y_labels[idx] == 1:
                # calculate IOU (overlap of prediction and gt bounding box)
                iou = 0.78 # assume we get this iou value for objects at idx
                if iou >= 0.5:
                    TP += 1
                else:
                    FP += 1
            elif prediction == 1 and y_labels[idx] == 0:
                FP += 1
            elif prediction == 0 and y_labels[idx] == 1:
                FN += 1
            else:
                TN += 1

     EPOCH_ACC = (TP + TN) / (TP + TN + FP + FN)

Output:
It is usually recommended to stick with batch-wise operations and avoid going into single-element processing steps while in the main training loop. One way to handle this case is to make your dataset return padded inputs and labels with additionally a mask that will come useful for loss computation. In other words, to compute the loss term with sequences of varying sizes, we will use a mask instead of doing individual slices.
Dataset
The way to proceed is to make sure you build the mask in the dataset and not in the inference loop. Here I am showing a minimal implementation that you should be able to transfer to your dataset without much hassle:
class Dataset(data.Dataset):
    def __init__(self):
        super().__init__()

    def __len__(self):
        return 100

    def __getitem__(self, index):
        i = random.randint(5, SEQ_LEN) # for demo puporse, generate x with random length
        x = torch.rand(i, EMB_SIZE)
        y = torch.randint(0, N_CLASSES, (i, EMB_SIZE))

        # pad data to fit in batch
        pad = torch.zeros(SEQ_LEN-len(x), EMB_SIZE)
        x_padded = torch.cat((pad, x))
        y_padded = torch.cat((pad, y))

        # construct tensor to mask loss
        mask = torch.cat((torch.zeros(SEQ_LEN-len(x)), torch.ones(len(x))))

        return x_padded, y_padded, mask

Essentially in the __getitem__, we not only pad the input x and target y with zero values, we also construct a simple mask containing the positions of the padded values in the currently processed element.
Notice how:

x_padded, shaped (SEQ_LEN, EMB_SIZE)
y_padded, shaped (SEQ_LEN, N_CLASSES)
mask, shaped (SEQ_LEN,)

are all three tensors which are shape invariant across the dataset, yet mask contains the padding information necessary for us to compute the loss function appropriately.
Inference
The loss you've used nn.BCEWithLogitsLoss, is the correct one since it's a multi-dimensional loss used for binary classification. In other words, you can use it here in this multi-label classification task, considering each one of the 128 logits as an individual binary prediction. Do not use nn.CrossEntropyLoss) as suggested elsewhere, since the softmax will push a single logit (i.e. class), which is the behaviour required for single-label classification tasks.
Therefore, in the training loop, we simply have to apply the mask to our loss.
for x, y, mask in dl:
   y_pred = model(x)
   loss = mask*bce(y_pred, y)
   # backpropagation, loss postprocessing, logs, etc.


Instruction: AttributeError: module 'torch.distributed' has no attribute 'is_initialized' in pytorch==1.11.X

Input:
whenever we are creating the object for TrainingArguments
from transformers import Trainer, TrainingArguments
batch_size = 64
logging_steps = len(emotions_encoded["train"]) // batch_size
model_name = f"{model_ckpt}-finetuned-emotion"
training_args = TrainingArguments(output_dir=model_name,
                                      num_train_epochs=2,
                                      learning_rate=2e-5,
                                      per_device_train_batch_size=batch_size,
                                      per_device_eval_batch_size=batch_size,
                                      weight_decay=0.01,
                                      evaluation_strategy="epoch",
                                      disable_tqdm=False,
                                      logging_steps=logging_steps,
                                      push_to_hub=True,
                                      log_level="error")

I am getting
AttributeError: module 'torch.distributed' has no attribute 'is_initialized'

Output:
In order to solve this problem
Actually Window and Mac doesn't support distributed training facility
so this issue is occuring
To solve this problem
go to your transformers package where you install it
in my case it is
Desktop/rajesh/pytorch_env/env/lib/python3.8/site-packages/transformers/training_args.py

replace line-1024
if torch.distributed.is_initialized() and self.local_rank == -1:
with
if True and self.local_rank == -1:
Restart your kernal


Instruction: How do I get my conda environment to recognize my GPU?

Input:
I'm pretty new to this and I'm wondering how do I get my conda environment to be able to recognize my GPU? Purpose is for me to emulate the Linux environment on my windows 10 device. I'm using the Anaconda prompt.
I'm following the instructions in this set up guide : https://medium.com/analytics-vidhya/4-steps-to-install-anaconda-and-pytorch-onwindows-10-5c9cb0c80dfe
But when I run the torch.cuda.is_available() it returns "false"

Not sure where I'm going wrong, would appreciate any help on how to get it to show "true" instead.

Output: Check your CUDA version. Not all the version of CUDA supports official pytorch. Check the official website of pytorch


Instruction: fastest way to calculate edges (derivatives) of a big torch tensor

Input:
Given a tensor with shape (b,c,h,w), I want to extract edges of the spatial data, that is, calculate x, y direction derivatives of the (h,w) and calculate the magnitude I=sqrt(|x_amplitude|^2+|y_amplitude|^2)

My current implementation is as followed
row_mat = np.asarray([[0, 0, 0], [1, 0, -1], [0, 0, 0]])
col_mat = row_mat.T
row_mat = row_mat[None, None, :, :]  # expand dim to convolve with tensor (batch,channel,width,height)
col_mat = col_mat[None, None, :, :]  # expand dim to convolve with tensor (batch,channel,width,height)

def derivative(batch: torch.Tensor) -> torch.Tensor:
    """
    uses convolution to perform x and y derivatives
    :param batch: input tensor batch
    :return: image derivative magnitudes
    """
    x_amplitude = ndimage.convolve(batch, row_mat)
    y_amplitude = ndimage.convolve(batch, col_mat)
    magnitude = np.sqrt(np.abs(x_amplitude) ** 2 + np.abs(y_amplitude) ** 2)
    return torch.tensor(magnitude)

I was wondering if there's a faster way, as this approach actually convolves using the definition of a derivative, so there might be downsides to that.

PS. to test this you can use the tensor torch.randn(1000,128,28,28), as these are the dimension I'm dealing with

Output:
For this specific operation you might be able to speed things up a bit by doing it "manually":
import torch.nn.functional as nnf

def derivative(batch: torch.Tensor) -> torch.Tensor:
  # pad batch
  x = nnf.pad(batch, (1, 1, 1, 1), mode='reflect')
  dx2 = (x[..., 1:-2, :-2] - x[..., 1:-2, 2:])**2
  dy2 = (x[..., :-2, 1:-2] - x[..., 2:, 1:-2])**2
  mag = torch.sqrt(dx2 + dy2)
  return mag


Instruction: How can I Modify pytorch dataset __getitem__ function to return a bag of 10 images?

Input:
I have a directory with multiple images separated into folders. Each folder has up to 3000 images. I would like to modify the pytorch dataset getitem function so that it returns bags of images, where each bag contains 10 images.
Here is what I have so far:
transform = transforms.Compose([transforms.Resize(255),
                            transforms.CenterCrop(224),
                            transforms.ToTensor()
                           ])
dataset = datasets.ImageFolder('./../BCNB/patches/WSI_1', transform=transform)
data_loader = torch.utils.data.DataLoader(dataset, batch_size = 1)

My output of DataLoader should be a tensor with a shape of [1, 10, 3, 256, 256].
Any input would be very helpful!
Thank you very much in advance!

Output:
Why do you need "bags of 10 images"? If you need them as mini batches for training -- don't change the Dataset, but use a DataLoader for that. A DataLoader takes a dataset and does the "batching" for you.
Alternatively, you can overload the __getitem__ method and implement your own that returns 10 images instead of just one.


Instruction: Deep Convolutional GAN (DCGAN) works really well on one set of data but not another

Input:
I am working on using PyTorch to create a DCGAN to use to generate trajectory data for a robot based on a dataset of 20000 other simple paths.
The DCGAN works great on the MNIST dataset, but does not work well on my custom dataset. I am trying to tweak/tune the DCGAN to give good results after being trained with my custom dataset.
Below is an example of output from the GAN (top), along with example training data for MNIST (bottom) after 20 epochs. The loss for the Generator and Discriminator each plateau at around 0.7.

Below is the output for my custom trajectory dataset after a similar number of epochs. The top figure shows the output, and bottom figure shows the training set of the batch.

It is clear that the same GAN is much better at making predictions for the MNIST dataset then for my custom dataset. It is interesting to note that the Discriminator and Generator losses also plateau at similar values of about 0.7 for this dataset too. This makes me think that there is some limit to the network of how low the loss can go.
Discriminator Code:
class Discriminator(nn.Module):
    def __init__(self, channels_img, features_d):
        super(Discriminator, self).__init__()
        self.disc = nn.Sequential(
            # input: N x channels_img x 64 x 64
            nn.Conv2d(
                channels_img, features_d, kernel_size=4, stride=2, padding=1
            ),
            nn.LeakyReLU(0.2),
            # _block(in_channels, out_channels, kernel_size, stride, padding)
            self._block(features_d, features_d * 2, 4, 2, 1),
            self._block(features_d * 2, features_d * 4, 4, 2, 1),
            self._block(features_d * 4, features_d * 8, 4, 2, 1),
            # After all _block img output is 4x4 (Conv2d below makes into 1x1)
            nn.Conv2d(features_d * 8, 1, kernel_size=4, stride=2, padding=0),
            nn.Sigmoid(),
        )

    def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.Conv2d(
                in_channels,
                out_channels,
                kernel_size,
                stride,
                padding,
                bias=False,
            ),
            nn.BatchNorm2d(out_channels),
            nn.LeakyReLU(0.2),
        )

    def forward(self, x):
        return self.disc(x)

Generator Code:
class Generator(nn.Module):
    def __init__(self, channels_noise, channels_img, features_g):
        super(Generator, self).__init__()
        self.net = nn.Sequential(
            # Input: N x channels_noise x 1 x 1
            self._block(channels_noise, features_g * 16, 4, 1, 0),  # img: 4x4
            self._block(features_g * 16, features_g * 8, 4, 2, 1),  # img: 8x8
            self._block(features_g * 8, features_g * 4, 4, 2, 1),  # img: 16x16
            self._block(features_g * 4, features_g * 2, 4, 2, 1),  # img: 32x32
            nn.ConvTranspose2d(
                features_g * 2, channels_img, kernel_size=4, stride=2, padding=1
            ),
            # Output: N x channels_img x 64 x 64
            nn.Tanh(),
        )

    def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.ConvTranspose2d(
                in_channels,
                out_channels,
                kernel_size,
                stride,
                padding,
                bias=False,
            ),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)

Training loop:
opt_gen = optim.Adam(gen.parameters(), lr=LEARNING_RATE_GEN, betas=(0.5, 0.999))
opt_disc = optim.Adam(disc.parameters(), lr=LEARNING_RATE_DISC, betas=(0.5, 0.999))
criterion = nn.BCELoss()

for epoch in range(NUM_EPOCHS):
    # Target labels not needed! <3 unsupervised
    # for batch_idx, (real, _) in enumerate(dataloader):
    for batch_idx, real in enumerate(dataloader):
        real = real.to(device)
        noise = torch.randn(BATCH_SIZE, NOISE_DIM, 1, 1).to(device)
        fake = gen(noise)

        ### Train Discriminator: max log(D(x)) + log(1 - D(G(z)))
        disc_real = disc(real.float()).reshape(-1)
        loss_disc_real = criterion(disc_real, torch.ones_like(disc_real))
        disc_fake = disc(fake.detach()).reshape(-1)
        loss_disc_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
        loss_disc = (loss_disc_real + loss_disc_fake) / 2
        disc.zero_grad()
        loss_disc.backward()
        opt_disc.step()

        ### Train Generator: min log(1 - D(G(z))) <-> max log(D(G(z))
        output = disc(fake).reshape(-1)
        loss_gen = criterion(output, torch.ones_like(output))
        gen.zero_grad()
        loss_gen.backward()
        opt_gen.step()

        # Print losses occasionally and print to tensorboard
        if batch_idx % 100 == 0:
            print(
                f"Epoch [{epoch}/{NUM_EPOCHS}] Batch {batch_idx}/{len(dataloader)} \
                  Loss D: {loss_disc:.4f}, loss G: {loss_gen:.4f}"
            )

            with torch.no_grad():
                fake = gen(fixed_noise)
                # take out (up to) 32 examples
                img_grid_real = torchvision.utils.make_grid(
                    real[:BATCH_SIZE], normalize=True
                )
                img_grid_fake = torchvision.utils.make_grid(
                    fake[:BATCH_SIZE], normalize=True
                )

                writer_real.add_image("Real", img_grid_real, global_step=step)
                writer_fake.add_image("Fake", img_grid_fake, global_step=step)

            step += 1

Output:
(Not enough reputation to comment yet, so I'll reply)
Since your code works fine on the MNIST dataset, the main problem here is that the trajectories in your training data are practically imperceptible, even to a human observer. Note that the trajectory lines are much thinner than the digit lines in the MNIST dataset.


Instruction: How can I extract features from pytorch fasterrcnn_resnet50_fpn

Input:
I tried to extract features from following code.
However, it says 'FasterRCNN' object has no attribute 'features'
I want to extract features with (36, 2048) shape features when it has 36 classes.
Is there any method to extract with pretrained pytorch models.
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True).to(device)
features = list(model.features)

dummy_img = torch.zeros((1, 3, 800, 800)).float() # test image array

req_features = []
output = dummy_img.clone().to(device)

for feature in features:
    output = feature(output)
    if output.size()[2] < 800//16: # 800/16=50
        break
    req_features.append(feature)
    out_channels = output.size()[1]

faster_rcnn_feature_extractor = nn.Sequential(*req_features)

output_map = faster_rcnn_feature_extractor(dummy_img )
print(output_map.shape)

Output:
The function you are calling returns a FasterRCNN object which is based on GeneralizedRCNN. As you have experienced, this object doesn't indeed have a feature attribute. Looking at its source code, if you want to acquire the feature maps, you can follow L83 and L101:
>>> images, _= model.transform(images, None)
>>> features = model.backbone(images.tensors)


Instruction: Find jaccard similarity among a batch of vectors in PyTorch

Input:
I'm having a batch of vectors of shape (bs, m, n) (i.e., bs vectors of dimensions mxn). For each batch, I would like to calculate the Jaccard similarity of the first vector with the rest (m-1) of them
Example:
a = [
    [[3, 8, 6, 8, 7],
    [9, 7, 4, 8, 1],
    [7, 8, 8, 5, 7],
    [3, 9, 9, 4, 4]],

    [[7, 3, 8, 1, 7],
    [3, 0, 3, 4, 2],
    [9, 1, 6, 1, 6],
    [2, 7, 0, 6, 6]]
]

Find pairwise jaccard similarity between a[:,0,:] and a[:,1:,:]
i.e.,
[3, 8, 6, 8, 7] with each of [[9, 7, 4, 8, 1], [7, 8, 8, 5, 7], [3, 9, 9, 4, 4]] (3 scores)
and
[7, 3, 8, 1, 7] with each of [[3, 0, 3, 4, 2], [9, 1, 6, 1, 6], [2, 7, 0, 6, 6]] (3 scores)

Here's the Jaccard function I have tried
def js(la1, la2):
    combined = torch.cat((la1, la2))
    union, counts = combined.unique(return_counts=True)
    intersection = union[counts > 1]
    torch.numel(intersection) / torch.numel(union)

While this works with unequal-sized tensors, the problem with this approach is that the number of uniques in each combination (pair of tensors) might be different and since PyTorch doesn't support jagged tensors, I'm unable to process batches of vectors at once.
If I'm not able to express the problem with the expected clarity, do let me know. Any help in this regard would be greatly appreciated
EDIT: Here's the flow achieved by iterating over the 1st and 2nd dimensions. I wish to have a vectorised version of the below code for batch processing
bs = 2
m = 4
n = 5
a = torch.randint(0, 10, (bs, m, n))
print(f"Array is: \n{a}")

for bs_idx in range(bs):
    first = a[bs_idx,0,:]
    for row in range(1, m):
        second = a[bs_idx,row,:]
        idx = js(first, second)
        print(f'comparing{first} and {second}: {idx}')

Output:
I don't know how you could achieve this in pytorch, since AFAIK pytorch doesn't support set operations on tensors. In your js() implementation, union calculation should work, but intersection = union[counts > 1] doesn't give you the right result if one of the tensors contains duplicated values. Numpy on the other hand has built-on support with union1d and intersect1d. You can use numpy vectorization to calculate pairwise jaccard indices without using for-loops:
import numpy as np

def num_intersection(vec1: np.ndarray, vec2: np.ndarray) -> int:
    return np.intersect1d(vec1, vec2, assume_unique=False).size

def num_union(vec1: np.ndarray, vec2: np.ndarray) -> int:
    return np.union1d(vec1, vec2).size

def jaccard1d(vec1: np.ndarray, vec2: np.ndarray) -> float:
    assert vec1.ndim == vec2.ndim == 1 and vec1.shape[0] == vec2.shape[0], 'vec1 and vec2 must be 1D arrays of equal length'
    return num_intersection(vec1, vec2) / num_union(vec1, vec2)
jaccard2d = np.vectorize(jaccard1d, signature='(m),(n)->()')

def jaccard(vecs1: np.ndarray, vecs2: np.ndarray) -> np.ndarray:
    """
    Return intersection-over-union (Jaccard index) between two sets of vectors.
    Both sets of vectors are expected to be flattened to 2D, where dim 0 is the batch
    dimension and dim 1 contains the flattened vectors of length V (jaccard index of
    an n-dimensional vector and of its flattened 1D-vector is equal).
    Args:
        vecs1 (ndarray[N, V]): first set of vectors
        vecs2 (ndarray[M, V]): second set of vectors
    Returns:
        ndarray[N, M]: the NxM matrix containing the pairwise jaccard indices for every vector in vecs1 and vecs2
    """
    assert vecs1.ndim == vecs2.ndim == 2 and vecs1.shape[1] == vecs2.shape[1], 'vecs1 and vecs2 must be 2D arrays with equal length in axis 1'
    return jaccard2d(vecs1, vecs2)

This is of course suboptimal because the code doesn't run on the GPU. If I run the jaccard function with vecs1 of shape (1, 10) and vecs2 of shape (10_000, 10) I get a mean loop time of 200 ms ± 1.34 ms on my machine, which should probably be fast enough for most use cases. And conversion between pytorch and numpy arrays is very cheap.
To apply this function to your problem with array a:
a = torch.tensor(a).numpy()  # just to demonstrate
ious = [jaccard(batch[:1, :], batch[1:, :]) for batch in a]
np.array(ious).squeeze()  # 2 batches with 3 scores each -> 2x3 matrix

# array([[0.28571429, 0.4       , 0.16666667],
#        [0.14285714, 0.16666667, 0.14285714]])

Use torch.from_numpy() on the result to get a pytorch tensor again if needed.

Update:
If you need a pytorch version for calculating the Jaccard index, I partially implemented numpy's intersect1d in torch:
from torch import Tensor

def torch_intersect1d(t1: Tensor, t2: Tensor, assume_unique: bool = False) -> Tensor:
    if t1.ndim > 1:
        t1 = t1.flatten()
    if t2.ndim > 1:
        t2 = t2.flatten()
    if not assume_unique:
        t1 = t1.unique(sorted=True)
        t2 = t2.unique(sorted=True)
    # generate a m x n intersection matrix where m is numel(t1) and n is numel(t2)
    intersect = t1[(t1.view(-1, 1) == t2.view(1, -1)).any(dim=1)]
    if not assume_unique:
        intersect = intersect.sort().values
    return intersect

def torch_union1d(t1: Tensor, t2: Tensor) -> Tensor:
    return torch.cat((t1.flatten(), t2.flatten())).unique()

def torch_jaccard1d(t1: Tensor, t2: Tensor) -> float:
    return torch_intersect1d(t1, t2).numel() / torch_union1d(t1, t2).numel()

To vectorize the torch_jaccard1d function, you might want to look into torch.vmap, which lets you vectorize a function over an arbitrary batch dimension (similar to numpy's vectorize). The vmap function is a prototype feature and not yet available in the usual pytorch distributions, but you can get it using nightly builds of pytorch. I haven't tested it but this might work.


Instruction: Print activations of a neural network in Pytorch

Input:
I was doing as suggested in this StackOverflow's questions: Pytorch: why print(model) does not show the activation functions?
Basically I want to train a neural network on MNIST, and I want to print the activations.
This is the neural networks:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 128, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(128, 256, 5)
        self.fc1 = nn.Linear(out_features, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.myRelu = nn.ReLU()

    def forward(self, x):
        x = self.conv1(x)
        x = self.myRelu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = self.myRelu(x)
        x = self.pool(x)
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = self.fc1(x)
        x = self.myRelu(x)
        x = self.fc2(x)
        x = self.myRelu(x)
        x = self.fc3(x)
        return x

However, if I do list(model.myRelu.parameters()) the result is an empty list: []. Any suggestion on how to print the activations?

Output:
I think you are getting mixed up between what the an activation function is an what you might be trying to show which are the activations of your intermediate layers. The question you are linking is asking about why isn't the activation function not showing in the summary of the PyTorch model.
On one hand, activation functions are non-linear functions, most generally non-parameterized, such as a ReLU function. This is the reason why you are not getting any parameters when looking at the content of model.myRelu.parameters(). It simply doesn't have any learned parameters!
On the other, you can be looking to inspect the intermediate results of your network. This means printing the output of a given layer, i.e. the activation of that layer. This is of course with respect to an input: for a given input x you are looking at the output of a given intermediate layer.

But reading your question again it seems you are just trying to print your model summary showing the activation functions inside it. In your case, this can be done by defining your model as a sequential model:
class Net(nn.Sequential):
    def __init__(self, out_features):
        super().__init__(nn.Conv2d(1, 128, 5),
                         nn.ReLU(),
                         nn.MaxPool2d(2, 2),
                         nn.Conv2d(128, 256, 5),
                         nn.ReLU(),
                         nn.MaxPool2d(2, 2),
                         nn.Flatten(),
                         nn.Linear(out_features, 120),
                         nn.ReLU(),
                         nn.Linear(120, 84),
                         nn.ReLU(),
                         nn.Linear(84, 10))

Then in the summary, you have:
>>> Net(10)
Net(
  (0): Conv2d(1, 128, kernel_size=(5, 5), stride=(1, 1))
  (1): ReLU()
  (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (3): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1))
  (4): ReLU()
  (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (6): Flatten(start_dim=1, end_dim=-1)
  (7): Linear(in_features=10, out_features=120, bias=True)
  (8): ReLU()
  (9): Linear(in_features=120, out_features=84, bias=True)
  (10): ReLU()
  (11): Linear(in_features=84, out_features=10, bias=True)
)


Instruction: Does Huggingface's "resume_from_checkpoint" work?

Input:
I currently have my trainer set up as:
training_args = TrainingArguments(
    output_dir=f"./results_{model_checkpoint}",
    evaluation_strategy="epoch",
    learning_rate=5e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=2,
    weight_decay=0.01,
    push_to_hub=True,
    save_total_limit = 1,
    resume_from_checkpoint=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_qa["train"],
    eval_dataset=tokenized_qa["validation"],
    tokenizer=tokenizer,
    data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
    compute_metrics=compute_metrics
)

After training, in my output_dir I have several files that the trainer saved:
['README.md',
 'tokenizer.json',
 'training_args.bin',
 '.git',
 '.gitignore',
 'vocab.txt',
 'config.json',
 'checkpoint-5000',
 'pytorch_model.bin',
 'tokenizer_config.json',
 'special_tokens_map.json',
 '.gitattributes']

From the documentation it seems that resume_from_checkpoint will continue training the model from the last checkpoint:
resume_from_checkpoint (str or bool, optional) — If a str, local path to a saved checkpoint as saved by a previous instance of Trainer. If a bool and equals True, load the last checkpoint in args.output_dir as saved by a previous instance of Trainer. If present, training will resume from the model/optimizer/scheduler states loaded here.
But when I call trainer.train() it seems to delete the last checkpoint and start a new one:
Saving model checkpoint to ./results_distilbert-base-uncased/checkpoint-500
...
Deleting older checkpoint [results_distilbert-base-uncased/checkpoint-5000] due to args.save_total_limit

Does it really continue training from the last checkpoint (i.e., 5000) and just starts the count of the new checkpoint at 0 (saves the first after 500 steps -- "checkpoint-500"), or does it simply not continue the training? I haven't found a way to test it and the documentation is not clear on that.

Output:
You also should add resume_from_checkpoint parametr to trainer.train with link to checkpoint

trainer.train(resume_from_checkpoint="{<path-where-checkpoint-were_stored>/checkpoint-0000")

0000- example of checkpoin number.
Don't forget to mount your drive during whole this process.


Instruction: How to bin a pytorch tensor into an array of values

Input:
Say I have an array of values w = [w1, w2, w3, ...., wn] and this array is sorted in ascending order, all values being equally spaced.
I have a pytorch tensor of any arbitrary shape. For the sake of this example, lets say that tensor is:
import torch

a = torch.rand(2,4)

assuming w1=torch.min(a) and wn=torch.max(a), I want to create two separate tensors, amax and amin, both of shape (2,4) such that amax contains values from w that are the nearest maximum value to the elements of a, and vice-versa for amin.
As an example, say:
a = tensor([[0.7192, 0.6264, 0.5180, 0.8836],
            [0.1067, 0.1216, 0.6250, 0.7356]])

w = [0.0, 0.33, 0.66, 1]

therefore, I would like amax and amin to be,
amax = tensor([[1.000, 0.66, 0.66, 1.000],
               [0.33, 0.33, 0.66, 1.00]])

amin = tensor([[0.66, 0.33, 0.33, 0.66],
               [0.00, 0.00, 0.33, 0.66]])

What is the fastest way to do this?

Output:
You could compute for all points in a, the difference with each bin inside w. For this, you need a little bit of broadcasting:
>>> z = a[...,None]-w[None,None]
tensor([[[ 0.7192,  0.3892,  0.0592, -0.2808],
         [ 0.6264,  0.2964, -0.0336, -0.3736],
         [ 0.5180,  0.1880, -0.1420, -0.4820],
         [ 0.8836,  0.5536,  0.2236, -0.1164]],

        [[ 0.1067, -0.2233, -0.5533, -0.8933],
         [ 0.1216, -0.2084, -0.5384, -0.8784],
         [ 0.6250,  0.2950, -0.0350, -0.3750],
         [ 0.7356,  0.4056,  0.0756, -0.2644]]])

We have to identify for each point in a, at which index (visually represented as columns here) the sign change occurs. We can apply the sign operator, then compute difference z[i+1]-z[i] between columns with diff, retrieve the non zero values with nonzero, then finally select and reshape the resulting tensor:
>>> index = z.sign().diff(dim=-1).nonzero()[:,2].view(2,4)
tensor([[2, 1, 1, 2],
        [0, 0, 1, 2]])

To get amin, simply index w with index:
>>> w[index]
tensor([[0.6600, 0.3300, 0.3300, 0.6600],
        [0.0000, 0.0000, 0.3300, 0.6600]])

And to get amax, we can offset the indices to jump to the upper bound:
>>> w[index+1]
tensor([[1.0000, 0.6600, 0.6600, 1.0000],
        [0.3300, 0.3300, 0.6600, 1.0000]])


Instruction: PyTorch running under WSL2 getting "Killed" for Out of memory even though I have a lot of memory left?

Input:
I'm on Windows 11, using WSL2 (Windows Subsystem for Linux). I recently upgraded my RAM from 32 GB to 64 GB.
While I can make my computer use more than 32 GB of RAM, WSL2 seems to be refusing to use more than 32 GB. For example, if I do
>>> import torch
>>> a = torch.randn(100000, 100000) # 40 GB tensor

Then I see the memory usage go up until it hit's 30-ish GB, at which point, I see "Killed", and the python process gets killed. Checking dmesg, it says that it killed the process because "Out of memory".
Any idea what the problem might be, or what the solution is?

Output:
According to this blog post, WSL2 is automatically configured to use 50% of the physical RAM of the machine.  You'll need to add a memory=48GB (or your preferred setting) to a .wslconfig file that is placed in your Windows home directory (\Users\{username}\).
[wsl2]
memory=48GB

After adding this file, shut down your distribution and wait at least 8 seconds before restarting.
Assume that Windows 11 will need quite a bit of overhead to operate, so setting it to use the full 64 GB would cause the Windows OS to run out of memory.


Instruction: Practically understanding gradient dimensions on backprop

Input:
I've wrote this piece of torch code that implements a Linear(1,1) -> Linear(1,1) -> MSE forward/backward pass:
l1 = torch.nn.Linear(1, 1, bias=False)
l2 = torch.nn.Linear(1, 1, bias=False)
l1.weight = torch.nn.Parameter(
        torch.tensor([[ 0.6279 ]]))
l2.weight = torch.nn.Parameter(
        torch.tensor([[ 0.731 ]]))
loss_fn = torch.nn.MSELoss()
optimizer_l1 = optim.SGD(l1.parameters(), lr=0.001)
optimizer_l2 = optim.SGD(l2.parameters(), lr=0.001)
x = torch.tensor([0.3])
y = torch.tensor([1]).float()
l1_o = l1(x)
l2_o = l2(l1_o)
loss = loss_fn(l2_o, y)
optimizer_l1.zero_grad()
optimizer_l2.zero_grad()
loss.backward()

which outputs that:
l1 weight Parameter containing:
tensor([[0.6279]], requires_grad=True)
l2 weight Parameter containing:
tensor([[0.7310]], requires_grad=True)
======================
x tensor([0.3000])
y tensor([1.])
>>>>>>>>>>>>>>>>>>>>>>
l1_o tensor([0.1884], grad_fn=<SqueezeBackward3>)
l2_o tensor([0.1377], grad_fn=<SqueezeBackward3>)
loss tensor(0.7436, grad_fn=<MseLossBackward>)
----------------------
l1_grad tensor([[-0.3782]])
l2_grad tensor([[-0.3249]])

Now, backwards logic of this makes sense:
x = 0.3
l1w = 0.6279
l2w = 0.7310
l1o = 0.1884 -> l2w * x -> 0.6279 * 0.3
l2o = 0.1377

l2_grad = mse_grad(l2o, y, l1o) -> -0.3782
l1_grad = mse_grad(l2o, y, l2w*x) -> -0.3249

However, it stops making sense after we go from 1x1 layers to 2x2:
l1 = torch.nn.Linear(2, 2, bias=False)
l2 = torch.nn.Linear(2, 2, bias=False)
l1.weight = torch.nn.Parameter(
        torch.tensor([[ -0.6279, -0.4686 ], [ -0.0907, 0.6363 ]]))
l2.weight = torch.nn.Parameter(
        torch.tensor([[ 0.731, 0.6026 ], [ -0.1873, -0.6037 ]]))
loss_fn = torch.nn.MSELoss()
optimizer_l1 = optim.SGD(l1.parameters(), lr=0.001)
optimizer_l2 = optim.SGD(l2.parameters(), lr=0.001)
x = torch.tensor([0.3, 0.5])
y = torch.tensor([1, 0]).float()
l1_o = l1(x)
l2_o = l2(l1_o)
loss = loss_fn(l2_o, y)
optimizer_l1.zero_grad()
optimizer_l2.zero_grad()
loss.backward()

Which gives this output:
l1 weight Parameter containing:
tensor([[-0.6279, -0.4686],
        [-0.0907,  0.6363]], requires_grad=True)
l2 weight Parameter containing:
tensor([[ 0.7310,  0.6026],
        [-0.1873, -0.6037]], requires_grad=True)
x tensor([0.3000, 0.5000])
y tensor([1., 0.])
l1_o tensor([-0.4227,  0.2909], grad_fn=<SqueezeBackward3>)
l2_o tensor([-0.1337, -0.0965], grad_fn=<SqueezeBackward3>)
loss tensor(0.6472, grad_fn=<MseLossBackward>)
---------------
l1_grad tensor([[-0.2432, -0.4053],
        [-0.1875, -0.3124]])
l2_grad tensor([[ 0.4792, -0.3298],
        [ 0.0408, -0.0281]])

When I try to reproduce this from scratch, I can't figure out how to keep gradients shape equal to weights shape without messing with my mse_grad function (I want to take gradient with respect to outputs, not weights, this should be possible since they're proportionate):
l1_o = x.dot(l1.T)
l2_o = l1_o.dot(l2.T)
loss = mse(l2_o, y)
l2_o_grad = mse_grad(l2_o, y, l1_o) # shape of l2 (weights) is (2,2), shape of my gradient is (2), since both the layer output and y are of (2) shape

My forward pass, mse loss and mse gradient when called standalone give exact same values as torch.

Output:
I'm not a 100% sure if I understood your problem correctly, but the gradient w.r.t a particular layer does have the same dimensionality as that layer's output. The gradient w.r.t (linearly applied) weights in that layer, is not computed by backpropagation but by multiplying the input activations (those before the weights) with the backpropagated gradient. Hence, input activations of [2, 1] multiplied by backpropagated gradients of [1, 2] give you a [2, 2] matrix. Pay attention that the input activations are a vector "standing up" and the backpropagated gradients are "lying on the side".
I would also recommend you google "The Matrix Cookbook" and familiarize yourself with https://en.wikipedia.org/wiki/Matrix_calculus. Not so much because of the mathematical insights but rather to make sure that derivatives between matrices and vectors and scalars have the right dimensionality (numerator-layout and denominator-layout notation).


Instruction: How to select fairseq option `--ddp-backend`

Input:
I'm learning how to use fairseq to implement a simple translation model based on Transformer.
I would like to use 2 GeForce RTX 3090 GPUs on my lab server. Which option should I select for --ddp-backend of fairseq-train?
Furthermore, could you explain about the meaning of all following options for --ddp-backend and when to use them respectively?

From fairseq Documentation: Command-line Tools => fairseq-train => distributed_training

--ddp-backend:
Possible choices: c10d, fully_sharded, legacy_ddp, no_c10d, pytorch_ddp, slowmo
DistributedDataParallel backend
Default: “pytorch_ddp”

I'm new to stack exchange community, sorry if there is any inappropriate action.

Output:
I am not too sure, but I found this on GitHub
DDP_BACKEND_CHOICES = ChoiceEnum(
    [
        "c10d",  # alias for pytorch_ddp
        "fully_sharded",  # FullyShardedDataParallel from fairscale
        "legacy_ddp",
        "no_c10d",  # alias for legacy_ddp
        "pytorch_ddp",
        "slowmo",
    ]
)

Might be helpful, but I am also struggling with this


Instruction: What are the PyTorch's model.eval() + no_grad() equivalent in TensorFlow?

Input:
I am trying to extract BERT embeddings and reproduce this code using tensorflow instead of pytorch. I know tf.stop_gradient() is the equivalent of torch.no_grad() but what about model.eval() / combination of both ?
# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()

# Run the text through BERT, and collect all of the hidden states produced
# from all 12 layers.
with torch.no_grad():

    outputs = model(tokens_tensor, segments_tensors)

    # Evaluating the model will return a different number of objects based on
    # how it's  configured in the `from_pretrained` call earlier. In this case,
    # becase we set `output_hidden_states = True`, the third item will be the
    # hidden states from all layers. See the documentation for more details:
    # https://huggingface.co/transformers/model_doc/bert.html#bertmodel
    hidden_states = outputs[2]

Output:
TLDR; eval and no_grad are two completely different things but will often be used in conjunction, primarily for performing fast inference in the case of evaluation/testing loops.
The nn.Module.eval function is applied on a PyTorch module and gives it the ability to change its behaviour depending on the stage type: training or evaluation. Only for a handful of layers does this actually have an effect on the layer. Functions such as dropout layers and normalization layers have different behaviours depending on whether they are in training or evaluation mode. You can read more about it on this thread.
The torch.no_grad utility is however a context manager, it changes the way the code contained inside that scope runs. When applied no_grad has the effect of preventing gradient computation. In practice, this means no layer activation is been cached in memory. This is most generally used for evaluation and testing loops where no backpropagation is expected after an inference. However, it can also be used during training, for example when an inference on a frozen component and the gradient is not required to pass through it.


Instruction: Create custom connection/ non-fully connected layers in Pytorch

Input:
As shown in the figure, it is a 3 layer with NN, namely input layer, hidden layer and output layer. I want to design the NN(in PyTorch, just the arch) where the input to hidden layer is fully-connected. However, from hidden layer to output, the first two neurons of the hidden layer should be connected to first neuron of the output layer, second two should be connected to the second in the output layer and so on. How shall this should be designed ?
from torch import nn
layer1 = nn.Linear(input_size, hidden_size)
layer2 = ??????

Output:
As @Jan said here, you can overload nn.Linear and provide a point-wise mask to mask the interaction you want to avoid having. Remember that a fully connected layer is merely a matrix multiplication with an optional additive bias.
Looking at its source code, we can do:
class MaskedLinear(nn.Linear):
    def __init__(self, *args, mask, **kwargs):
        super().__init__(*args, **kwargs)
        self.mask = mask

    def forward(self, input):
        return F.linear(input, self.weight, self.bias)*self.mask

Having F defined as torch.nn.functional
Considering the constraint you have given to the second layer:

the first two neurons of the hidden layer should be connected to the first neuron of the output layer

It seems you are looking for this pattern:
tensor([[1., 0., 0.],
        [1., 0., 0.],
        [0., 1., 0.],
        [0., 1., 0.],
        [0., 0., 1.],
        [0., 0., 1.]])

Which can be obtained using torch.block_diag:
mask = torch.block_diag(*[torch.ones(2,1),]*output_size)

Having this, you can define your network as:
net = nn.Sequential(nn.Linear(input_size, hidden_size),
                    MaskedLinear(hidden_size, output_size, mask))

If you feel like it, you can even implement it inside the custom layer:
class LocalLinear(nn.Linear):
    def __init__(self, *args, kernel_size=2, **kwargs):
        super().__init__(*args, **kwargs)

        assert self.in_features == kernel_size*self.out_features
        self.mask = torch.block_diag(*[torch.ones(kernel_size,1),]*self.out_features)

def forward(self, input):
    return F.linear(input, self.weight, self.bias)*self.mask

And defining it like so:
net = nn.Sequential(nn.Linear(input_size, hidden_size),
                    LocalLinear(hidden_size, output_size))


Instruction: How convert this Pytorch loss function to Tensorflow?

Input:
This Code for a paper I read had a loss function written using Pytorch, I tried to convert it as best as I could but am getting all Zero's as model predictions, so would like to ask the following:

Are the methods I used the correct equivalent in Tensorflow?
Why is the model predicting only Zero's?

Here is the function:
#Pytorch

class AdjMSELoss1(nn.Module):
def __init__(self):
    super(AdjMSELoss1, self).__init__()

def forward(self, outputs, labels):
    outputs = torch.squeeze(outputs)
    alpha = 2
    loss = (outputs - labels)**2
    adj = torch.mul(outputs, labels)
    adj[adj>0] = 1 / alpha
    adj[adj<0] = alpha
    loss = loss * adj
    return torch.mean(loss)

#Tensorflow

def custom_loss_function(outputs,labels):
    outputs = tf.squeeze(outputs)
    alpha = 2.0
    loss = (outputs - labels) ** 2.0
    adj = tf.math.multiply(outputs,labels)

    adj = tf.where(tf.greater(adj, 0.0), tf.constant(1/alpha), adj)
    adj = tf.where(tf.less(adj, 0.0), tf.constant(alpha), adj)

    loss = loss * adj

    return  tf.reduce_mean(loss)

The function compiles correctly and is being used in the loss and metric parameters, it is outputing results in metrics logs that appear to be correct (Similar to val_loss) but the output of the model after running is just predicting all 0's
 model.compile(
        loss= custom_loss_function,
        optimizer=optimization,
        metrics = [custom_loss_function]
    )

MODEL
#Simplified for readability
model = Sequential()
model.add(LSTM(32,input_shape=(SEQ_LEN,feature_number),return_sequences=True,))
model.add(Dropout(0.3))
model.add(LSTM(96, return_sequences = False))
model.add(Dropout(0.3))
model.add(Dense(1))

return model

Inputs/Features are pct_change Price for the previous SEQ_LEN days. (Given SEQ_LEN days tries to predict next day: Target)
Outputs/Targets are the next day's price pct_change * 100 (Ex: 5 for 5%). (1 value per row)
Note: The model predicts normally when RMSE() is set as the loss function, as mentioned when using the custom_loss_function above it's just predicting Zero's

Output:
Try this custom_loss:
def custom_loss(y_pred, y_true):
    alpha = 2.0
    loss = (y_pred - y_true) ** 2.0
    adj = tf.math.multiply(y_pred,y_true)
    adj = tf.where(tf.greater(adj, 0.0), tf.constant(1/alpha), adj)
    adj = tf.where(tf.less(adj, 0.0), tf.constant(alpha), adj)
    loss = loss * adj
    return  tf.reduce_mean(loss)

I check with the below code and work correctly (Code for creating a model for learning and predicting the sum of two variables with the custom_loss):
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
import numpy as np

x = np.random.rand(1000,2)
y = x.sum(axis=1)
y = y.reshape(-1,1)

def custom_loss(y_pred, y_true):
    alpha = 2.0
    loss = (y_pred - y_true) ** 2.0
    adj = tf.math.multiply(y_pred,y_true)
    adj = tf.where(tf.greater(adj, 0.0), tf.constant(1/alpha), adj)
    adj = tf.where(tf.less(adj, 0.0), tf.constant(alpha), adj)
    loss = loss * adj
    return  tf.reduce_mean(loss)

model = Sequential()
model.add(Dense(128, activation='relu', input_dim=2))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1,))

model.compile(optimizer='adam', loss=custom_loss)
model.fit(x, y, epochs=200, batch_size=16)

for _ in range(10):
    rnd_num = np.random.randint(50, size=2)[None, :]
    pred_add = model.predict(rnd_num)
    print(f'predict sum of {rnd_num[0]} -> {pred_add}')

Output:
Epoch 1/200
63/63 [==============================] - 1s 2ms/step - loss: 0.2903
Epoch 2/200
63/63 [==============================] - 0s 2ms/step - loss: 0.0084
Epoch 3/200
63/63 [==============================] - 0s 2ms/step - loss: 0.0016
...
Epoch 198/200
63/63 [==============================] - 0s 2ms/step - loss: 3.3231e-07
Epoch 199/200
63/63 [==============================] - 0s 2ms/step - loss: 5.1004e-07
Epoch 200/200
63/63 [==============================] - 0s 2ms/step - loss: 9.8688e-08

predict sum of [43 44] -> [[82.81973]]
predict sum of [39 13] -> [[48.97299]]
predict sum of [36 46] -> [[78.05187]]
predict sum of [46  7] -> [[49.445843]]
predict sum of [35 11] -> [[43.311478]]
predict sum of [33  1] -> [[31.695848]]
predict sum of [6 8] -> [[13.433815]]
predict sum of [14 38] -> [[49.54941]]
predict sum of [ 1 40] -> [[39.709686]]
predict sum of [10  2] -> [[11.325197]]


Instruction: undo torch.chunk to rebuild image in right order

Input:
I am trying to process a 3D image in chunks (non-overlapping windows). Once this is done I want to put the chunks back together in the right order.
I have been chunking the image as below:
tens = torch.tensor(range(64))
tens = tens.view((4,4,4))
print(tens)
>>>tensor([[[ 0,  1,  2,  3],
         [ 4,  5,  6,  7],
         [ 8,  9, 10, 11],
         [12, 13, 14, 15]],

        [[16, 17, 18, 19],
         [20, 21, 22, 23],
         [24, 25, 26, 27],
         [28, 29, 30, 31]],

        [[32, 33, 34, 35],
         [36, 37, 38, 39],
         [40, 41, 42, 43],
         [44, 45, 46, 47]],

        [[48, 49, 50, 51],
         [52, 53, 54, 55],
         [56, 57, 58, 59],
         [60, 61, 62, 63]]])

tens = torch.chunk(tens,2, -1)
tens = torch.stack(tens)
tens = torch.chunk(tens,2, -2)
tens = torch.concat(tens)
tens = torch.chunk(tens,2, -3)
tens = torch.concat(tens)
print(torch.shape)
>>>torch.Size([8, 2, 2, 2])

Then I want to put it back together in the original order
tens = tens.view([4,4,2,2])
tens = tens.view([2,4,4,2])
tens = tens.view([4,4,4])
print(tens)
>>>tensor([[[ 0,  1,  4,  5],
         [16, 17, 20, 21],
         [ 2,  3,  6,  7],
         [18, 19, 22, 23]],

        [[ 8,  9, 12, 13],
         [24, 25, 28, 29],
         [10, 11, 14, 15],
         [26, 27, 30, 31]],

        [[32, 33, 36, 37],
         [48, 49, 52, 53],
         [34, 35, 38, 39],
         [50, 51, 54, 55]],

        [[40, 41, 44, 45],
         [56, 57, 60, 61],
         [42, 43, 46, 47],
         [58, 59, 62, 63]]])

and I can't figure out how to get the elements in the right order. I realise I probably missed something in the docs or something else obvious but I can't find it. Any ideas?

Output:
Operator torch.chunk doesn't reduce dimensions so its inverse is torch.cat, not torch.stack.
Here are the transforms with the corresponding inverse operations:

Splitting the last dimension in two:

>>> chunks = tens.chunk(chunks=2, dim=-1)
>>> torch.cat(chunks, dim=-1)

Splitting the second dimension into two:

>>> chunks = tens.chunk(chunks=2, dim=-2)
>>> torch.cat(chunks, dim=-2)

Splitting the first dimension into two:

>>> chunks = tens.chunk(chunks=2, dim=-3)
>>> torch.cat(chunks, dim=-3)

If you want to invert the whole sequence, you just have to keep in mind that torch.cat is the reverse operation of torch.chunk:
>>> tens = torch.cat(tens.chunk(2), dim=-3)
>>> tens = torch.cat(tens.chunk(2), dim=-2)
>>> tens = torch.cat(tens.chunk(2), dim=-1)


Instruction: How to flow gradient through parameter updates?

Input:
I have a parameter
import torch
import torch.nn as nn
x = nn.Parameter(torch.tensor([1.0]))

I need to update the value of this parameter while maintaining the gradient flow (for a meta learning problem).
PyTorch does not allow in-place operations on leaf-variable like
a = torch.tensor([2.0], requires_grad=True)
x.copy_(a)
x.backward()
>>> RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

In addition, I also tried the following, but while working, the gradient didn't flow:
x = nn.Parameter(torch.tensor([1.0]))
a = torch.tensor([2.0], requires_grad=True)
x = nn.Parameter(a)
x.backward()
print(a.grad)
>>> None

How can I update the value of a parameter while maintaining gradient flow?

Output:
What you are looking for is torch.Tensor.clone:

This function is differentiable, so gradients will flow back from the
result of this operation to input. To create a tensor without an
autograd relationship to input see detach().

>>> a = torch.tensor([2.0], requires_grad=True)
>>> x = a.clone()
>>> x.backward()

>>> a.grad
tensor([1.])


Instruction: Fine Tuning Blenderbot

Input:
I have been trying to fine-tune a conversational model of HuggingFace: Blendebot. I have tried the conventional method given on the official hugging face website which asks us to do it using the trainer.train() method. I also tried it using the .compile() method. I have tried fine-tuning using PyTorch as well as TensorFlow on my dataset. Both methods seem to fail and give us an error saying that there is no method called compile or train for the Blenderbot model.
I have also looked everywhere online to check how Blenderbot could be fine-tuned on my custom data and nowhere does it mention properly that runs without throwing an error. I have gone through Youtube tutorials, blogs, and StackOverflow posts but none answer this question. Hoping someone would respond here and help me out. I am open to using other HuggingFace Conversational Models as well for fine-tuning.
Thank you! :)

Output:
Here is a link I am using to fine-tune the blenderbot model.
Fine-tuning methods: https://huggingface.co/docs/transformers/training
Blenderbot: https://huggingface.co/docs/transformers/model_doc/blenderbot
from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration
mname = "facebook/blenderbot-400M-distill"
model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = BlenderbotTokenizer.from_pretrained(mname)

#FOR TRAINING:

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.train()

#OR

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)

model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3)

None of these work! :(


Instruction: Blenderbot FineTuning

Input:
I have been trying to fine-tune a conversational model of HuggingFace: Blendebot. I have tried the conventional method given on the official hugging face website which asks us to do it using the trainer.train() method. I tried it using the .compile() method. I have tried fine-tuning using PyTorch as well as TensorFlow on my dataset. Both methods seem to fail and give us an error saying that there is no method called compile or train for the Blenderbot model. I have even looked everywhere online to check how Blenderbot could be fine-tuned on my custom data and nowhere does it mention properly that runs without throwing an error. I have gone through Youtube tutorials, blogs, and StackOverflow posts but none answer this question. Hoping someone would respond here and help me out. I am open to using other HuggingFace Conversational Models as well for fine-tuning.
Here is a link I am using to fine-tune the blenderbot model.
Fine-tuning methods: https://huggingface.co/docs/transformers/training
Blenderbot: https://huggingface.co/docs/transformers/model_doc/blenderbot
from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration
mname = "facebook/blenderbot-400M-distill"
model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = BlenderbotTokenizer.from_pretrained(mname)

#FOR TRAINING:

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.train()

#OR

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)

model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3)

None of these work.

Output:
Maybe try using the TFBlenderbotForConditionalGeneration class for Tensorflow. It has what you need:
import tensorflow as tf
from transformers import BlenderbotTokenizer, TFBlenderbotForConditionalGeneration

mname = "facebook/blenderbot-400M-distill"
model = TFBlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = BlenderbotTokenizer.from_pretrained(mname)

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=tf.metrics.SparseCategoricalAccuracy(),
)
....

See the docs for more information.