Instruction: AWS EC2 causing issue with Streamlit ML App

Input:
It's strange because on my local machine, this issue doesn't happen, and my app works fine.
However, when I run the app on an AWS EC2 instance, it gives me an error regarding a matplotlib import. Below the matplotlib import, I have matplotlib.use('TkAgg'). When the code is like this, the Streamlit app gives me this error (only on the EC2 instance):
ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running
Traceback:
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/ubuntu/extremely_unnecessary/app.py", line 16, in <module>
    matplotlib.use('TkAgg')
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/matplotlib/__init__.py", line 1171, in use
    plt.switch_backend(name)
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/matplotlib/pyplot.py", line 287, in switch_backend
    newbackend, required_framework, current_framework))

After doing some research, I tried changing the offending line to matplotlib.use('agg'). When I do this, the app works properly, however none of the models except one work when selected.
The app is hosted here: http://54.193.229.139:8501/ The way it works is you upload an image, then select a pretrained model from the drop down menu to apply "style transfer" to the image you uploaded.
For some bizarre reason, the 12th model in the list (chicken-strawberries-market-069_10000.pth) works, but none of the other models do. Again, this only happens on the EC2 instance - even when I use matplotlib.use('agg'), all the models work when running the streamlit app locally.
I also tried using some other variations including matplotlib.use('GTK3Agg') and matplotlib.use('WebAgg'), which give me various other error messages.
Does anyone know how to fix this so I can get all the models working on the EC2 instance?
Edit: I've started receiving a new error message, I'm working on trying to change the code now. I use CUDA through my GPU, apparently I have to make some cpu-bound changes so it'll work on the ubuntu server. Not sure why the chicken strawberries model works though...
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Traceback:
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/ubuntu/extremely_unnecessary/app.py", line 91, in <module>
    main()
File "/home/ubuntu/extremely_unnecessary/app.py", line 54, in main
    transformer.load_state_dict(torch.load(checkpoint))
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 774, in _legacy_load
    result = unpickler.load()
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 730, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 151, in _cuda_deserialize
    device = validate_cuda_device(location)
File "/home/ubuntu/anaconda3/envs/streamlit/lib/python3.6/site-packages/torch/serialization.py", line 135, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '

The code for the app:
import matplotlib.pyplot as plt
from PIL import Image
from torchvision.utils import save_image
import tqdm
import streamlit as st
from models import TransformerNet
from utils import *
import torch
import numpy as np
from torch.autograd import Variable
import argparse
import tkinter as tk
import os
import cv2
import matplotlib
matplotlib.use('agg')

def main():

    uploaded_file = st.file_uploader(
        "Choose an image", type=['jpg', 'png', 'webm', 'mp4', 'gif', 'jpeg'])
    if uploaded_file is not None:
        st.image(uploaded_file, width=200)

    folder = os.path.abspath(os.getcwd())
    folder = folder + '/models'

    fnames = []

    for basename in os.listdir(folder):
        print(basename)
        fname = os.path.join(folder, basename)

        if fname.endswith('.pth'):
            fnames.append(fname)

    checkpoint = st.selectbox('Select a pretrained model', fnames)

    os.makedirs("images/outputs", exist_ok=True)

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    # device = torch.device("cpu")
    transform = style_transform()

    # Define model and load model checkpoint
    transformer = TransformerNet().to(device)
    transformer.load_state_dict(torch.load(checkpoint))
    transformer.eval()

    # Prepare input
    image_tensor = Variable(transform(Image.open(
        uploaded_file).convert('RGB'))).to(device)
    image_tensor = image_tensor.unsqueeze(0)

    # Stylize image
    with torch.no_grad():
        stylized_image = denormalize(transformer(image_tensor)).cpu()

    fn = str(np.random.randint(0, 100)) + 'image.jpg'
    save_image(stylized_image, f"images/outputs/stylized-{fn}")

    st.image(f"images/outputs/stylized-{fn}")

if __name__ == "__main__":
    main()

Output:
Turns out all I needed to do was implement the line in the error message - in line 53, I just had to change it from this:
transformer.load_state_dict(torch.load(checkpoint))

to this
transformer.load_state_dict(torch.load(
    checkpoint, map_location=torch.load('cpu')))

And it works!


Instruction: mat1 and mat2 shapes cannot be multiplied

Input:
I am new to AI and python, I'm trying to build an architecture to train a set of images. and later to aim to overfit. but up till now, I couldn't understand how to get the inputs and outputs correctly. I keep seeing the error whenever I try to train the network:
mat1 and mat2 shapes cannot be multiplied (48x13456 and 16x64)
my network:
net2 = nn.Sequential(

nn.Conv2d(3,8, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Conv2d(8,16, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Flatten(),
nn.Linear(16,64),
nn.ReLU(),
nn.Linear(64,10)
)

this is a part of a task I'm working on and I really don't get why it's not running. any hints!

Output:
its because you have flattened your 2D cnn into  1D FC layers...
& you have to manually calculate your changed input shape from 128 size to your Maxpool layer just before flattening layer ...In your case its 29*29*16
So your code must be rewritten as
net2 = nn.Sequential(

nn.Conv2d(3,8, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Conv2d(8,16, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Flatten(),

nn.Linear(13456,64),
nn.ReLU(),
nn.Linear(64,10)
)

This should work
EDIT: This is a simple formula to calculate output size :
  (((W - K + 2P)/S) + 1)
    Here W = Input size
    K = Filter size
    S = Stride
    P = Padding

So 1st conv block will make your output of size 124
Then you do Maxpool which will make it half i.e 62
2nd conv block will make your output of size 58
Then your last Maxpool will make it 29...
So final flattened output would be 29*29*16 where 16 is output channels


Instruction: UnpicklingError: A load persistent id instruction was encountered, but no persistent_load function was specified

Input:
I was trying to run a python file named api.py. In this file, I'm loading the pickle file of the Deep Learning model that was built and trained using PyTorch.
api.py
In api.py the below-given functions are the most important ones.
def load_model_weights(model_architecture, weights_path):
  if os.path.isfile(weights_path):
      cherrypy.log("CHERRYPYLOG Loading model from: {}".format(weights_path))
      model_architecture.load_state_dict(torch.load(weights_path))
  else:
      raise ValueError("Path not found {}".format(weights_path))

def load_recommender(vector_dim, hidden, activation, dropout, weights_path):

    rencoder_api = model.AutoEncoder(layer_sizes=[vector_dim] + [int(l) for l in hidden.split(',')],
                               nl_type=activation,
                               is_constrained=False,
                               dp_drop_prob=dropout,
                               last_layer_activations=False)
    load_model_weights(rencoder_api, weights_path)
    rencoder_api.eval()
    rencoder_api = rencoder_api.cuda()
    return rencoder_api

The directory structure
MP1
 ┣ .ipynb_checkpoints
 ┃ ┗ RS_netflix3months_100epochs_64,128,128-checkpoint.ipynb
 ┣ data
 ┃ ┣ AutoEncoder.png
 ┃ ┣ collaborative_filtering.gif
 ┃ ┣ movie_titles.txt
 ┃ ┗ shut_up.gif
 ┣ DeepRecommender
 ┃ ┣ data_utils
 ┃ ┃ ┣ movielens_data_convert.py
 ┃ ┃ ┗ netflix_data_convert.py
 ┃ ┣ reco_encoder
 ┃ ┃ ┣ data
 ┃ ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┃ ┣ input_layer.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┣ input_layer_api.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┃ ┣ input_layer.py
 ┃ ┃ ┃ ┣ input_layer_api.py
 ┃ ┃ ┃ ┗ __init__.py
 ┃ ┃ ┣ model
 ┃ ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┃ ┣ model.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┃ ┣ model.py
 ┃ ┃ ┃ ┗ __init__.py
 ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┗ __init__.py
 ┃ ┣ __pycache__
 ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┣ compute_RMSE.py
 ┃ ┣ infer.py
 ┃ ┣ run.py
 ┃ ┗ __init__.py
 ┣ model_save
 ┃ ┣ model.epoch_99
 ┃ ┃ ┗ archive
 ┃ ┃ ┃ ┣ data
 ┃ ┃ ┃ ┃ ┣ 92901648
 ┃ ┃ ┃ ┃ ┣ 92901728
 ┃ ┃ ┃ ┃ ┣ 92901808
 ┃ ┃ ┃ ┃ ┣ 92901888
 ┃ ┃ ┃ ┃ ┣ 92901968
 ┃ ┃ ┃ ┃ ┣ 92902048
 ┃ ┃ ┃ ┃ ┣ 92902128
 ┃ ┃ ┃ ┃ ┣ 92902208
 ┃ ┃ ┃ ┃ ┣ 92902288
 ┃ ┃ ┃ ┃ ┣ 92902368
 ┃ ┃ ┃ ┃ ┣ 92902448
 ┃ ┃ ┃ ┃ ┗ 92902608
 ┃ ┃ ┃ ┣ data.pkl
 ┃ ┃ ┃ ┗ version
 ┃ ┣ model.epoch_99.zip
 ┃ ┗ model.onnx
 ┣ Netflix
 ┃ ┣ N1Y_TEST
 ┃ ┃ ┗ n1y.test.txt
 ┃ ┣ N1Y_TRAIN
 ┃ ┃ ┗ n1y.train.txt
 ┃ ┣ N1Y_VALID
 ┃ ┃ ┗ n1y.valid.txt
 ┃ ┣ N3M_TEST
 ┃ ┃ ┗ n3m.test.txt
 ┃ ┣ N3M_TRAIN
 ┃ ┃ ┗ n3m.train.txt
 ┃ ┣ N3M_VALID
 ┃ ┃ ┗ n3m.valid.txt
 ┃ ┣ N6M_TEST
 ┃ ┃ ┗ n6m.test.txt
 ┃ ┣ N6M_TRAIN
 ┃ ┃ ┗ n6m.train.txt
 ┃ ┣ N6M_VALID
 ┃ ┃ ┗ n6m.valid.txt
 ┃ ┣ NF_TEST
 ┃ ┃ ┗ nf.test.txt
 ┃ ┣ NF_TRAIN
 ┃ ┃ ┗ nf.train.txt
 ┃ ┗ NF_VALID
 ┃ ┃ ┗ nf.valid.txt
 ┣ test
 ┃ ┣ testData_iRec
 ┃ ┃ ┣ .part-00199-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt.crc
 ┃ ┃ ┣ part-00000-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt
 ┃ ┃ ┣ part-00003-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt
 ┃ ┃ ┗ _SUCCESS
 ┃ ┣ testData_uRec
 ┃ ┃ ┣ .part-00000-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt.crc
 ┃ ┃ ┣ ._SUCCESS.crc
 ┃ ┃ ┣ part-00161-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┃ ┣ part-00196-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┃ ┗ part-00199-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┣ data_layer_tests.py
 ┃ ┣ test_model.py
 ┃ ┗ __init__.py
 ┣ __pycache__
 ┃ ┣ api.cpython-37.pyc
 ┃ ┣ load_test.cpython-37.pyc
 ┃ ┣ parameters.cpython-37.pyc
 ┃ ┗ utils.cpython-37.pyc
 ┣ api.py
 ┣ compute_RMSE.py
 ┣ load_test.py
 ┣ logger.py
 ┣ netflix_1y_test.csv
 ┣ netflix_1y_train.csv
 ┣ netflix_1y_valid.csv
 ┣ netflix_3m_test.csv
 ┣ netflix_3m_train.csv
 ┣ netflix_3m_valid.csv
 ┣ netflix_6m_test.csv
 ┣ netflix_6m_train.csv
 ┣ netflix_6m_valid.csv
 ┣ netflix_full_test.csv
 ┣ netflix_full_train.csv
 ┣ netflix_full_valid.csv
 ┣ parameters.py
 ┣ preds.txt
 ┣ RS_netflix3months_100epochs_64,128,128.ipynb
 ┗ utils.py

I am getting such an error (serialization.py). Can someone help me with this error?
D:\Anaconda\envs\practise\lib\site-packages\torch\serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    762             "functionality.")
    763
--> 764     magic_number = pickle_module.load(f, **pickle_load_args)
    765     if magic_number != MAGIC_NUMBER:
    766         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

Output:
After searching through PyTorch documentation, I ended up saving the model in the ONNX format and later loaded that ONNX model into PyTorch model and used it for inference.
import onnx
from onnx2pytorch import ConvertModel

def load_model_weights(model_architecture, weights_path):
    if os.path.isfile("model.onnx"):
        cherrypy.log("CHERRYPYLOG Loading model from: {}".format(weights_path))
        onnx_model = onnx.load("model.onnx")
        pytorch_model = ConvertModel(onnx_model)
        ## model_architecture.load_state_dict(torch.load(weights_path))
    else:
        raise ValueError("Path not found {}".format(weights_path))

def load_recommender(vector_dim, hidden, activation, dropout, weights_path):

    rencoder_api = model.AutoEncoder(layer_sizes=[vector_dim] + [int(l) for l in hidden.split(',')],
                               nl_type=activation,
                               is_constrained=False,
                               dp_drop_prob=dropout,
                               last_layer_activations=False)
    load_model_weights(rencoder_api, weights_path)
    rencoder_api.eval()
    rencoder_api = rencoder_api.cuda()
    return rencoder_api

Some useful resources:
torch.save
torch.load
ONNX tutorials


Instruction: How to pad a numpy 3D array (or torch tensor) with values from surrounding 3D arrays

Input:
I have a 3D numpy array of shape 3,3,3 to which I want to pad 2 layers of values from arrays surrounding it spatially, so that it becomes a 5,5,5 array.

What I have done so far using torch cat function (which works the same as numpy concat) to pad the y array, is the following:
x = torch.from_numpy(np.arange(1,28,1).reshape(3,3,3))
y = torch.from_numpy(np.arange(28,55,1).reshape(3,3,3))
z = torch.from_numpy(np.arange(55,82,1).reshape(3,3,3))

torch.cat((y,z[:,:2,:]), dim=1) #To concat z+ with 2 pads
torch.cat((x[:,1:,:],y), dim=1) #To concat z- with 2 pads

torch.cat((y,z[:,:,:2]), dim=2) #To concat x+ with 2 pads
torch.cat((x[:,1:,:],y), dim=1) #To concat x- with 2 pads

torch.cat((x,z[:2,:,:]), dim=2) #To concat y+ with 2 pads
torch.cat((x[1:,:,:],y), dim=1) #To concat y- with 2 pads

But it does not give me the right values. How can I acheive this?

Output:
If I understand correctly, what you want is not a regular array since each dimension have different range depending on the observed axis (i.e., it can't be geometrically represented as a cube - your picture is not a (n,n,n) array).
Anyways, in the somewhat lengthy following snippet, we create a (5,5,5) test 3D array from which the (3,3,3) array can be sampled. Then we consecutively concatenate to obtain the original array, after which we mask the unwanted cells so the output is what your picture shows. Note that you can replace logic operations with + or * when using boolean Numpy arrays.
import numpy as np

# Define dummy 3D field
n = 5
xx, yy, zz = np.ogrid[0:n, 0:n, 0:n]
field = np.sin(xx) + np.cos(yy) + np.tan(zz)

# Indices of 3 innermost elements - to form (3,3,3) array
i1, i2 = n//2 - 1, n//2 + 1
# Inner 3D array
subfield = np.copy(field)[i1:i2+1, i1:i2+1, i1:i2+1]

# Indices of "inferior" pads
x1 = y1 = z1 = np.arange(i1 - 1, i1)
# Indices of "superior" pads
x2 = y2 = z2 = np.arange(i2 + 1, i2 + 2)

# Padding in axis 0 (x)
padded = np.concatenate((field[x1, i1:i2+1, i1:i2+1], subfield))
padded = np.concatenate((padded, field[x2, i1:i2+1, i1:i2+1]))
# Padding in axis 1 (y)
padded = np.concatenate((field[i1-1:i2+2, y1, i1:i2+1], padded), axis = 1)
padded = np.concatenate((padded, field[i1-1:i2+2, y2, i1:i2+1]), axis = 1)
# Padding in axis 2 (z)
padded = np.concatenate((field[i1-1:i2+2, i1-1:i2+2, z1], padded), axis = 2)
padded = np.concatenate((padded, field[i1-1:i2+2, i1-1:i2+2, z2]), axis = 2)

# Check padded array is equal to original 3D array
print(np.all(padded == field))

## We now mask unwanted cells
indices = np.indices(padded.shape)
idx1, idx2 = indices == 0, indices == n - 1

xi, xf = idx1[0], idx2[0]
yi, yf = idx1[1], idx2[1]
zi, zf = idx1[2], idx2[2]

# Logical operations to mask proper slices
xm = (xi + xf) * (yi + yf + zi + zf)            # masking in axis 0
ym = (yi + yf) * (xi + xf + zi + zf)            # masking in axis 1
zm = (zi + zf) * (yi + yf + xi + xf)            # masking in axis 2

mask = xm + ym + zm
# Masked (5,5,5) array
masked_padded = np.ma.masked_where(mask, padded)

By the way, there must be more elegant ways to achieve the same result, but I have not used Numpy's advanced indexing that much :P


Instruction: Python: append images in for loop use a lot of memory

Input:
I'm actually facing a problem while working on a python project.
I'm appending some images in a for loop and it uses a lot of RAM memory.
If you guys, have any solution to optimize this for loop, It'll help me a lot.
Thanks!
augment_img = []
augment_label = []
augment_weight = []
for i in range(4):
    for j in range(len(train_dataset)):
        single_img, single_label, single_weight = train_dataset[j]
        augment_img.append(single_img)
        augment_label.append(single_label)
        augment_weight.append(single_weight)
        if j % 1000==0:
            print(j)```

Output:
Instead of loading all images at once, I would suggest loading them in batches. There are different ways to handle that. In both pytorch and tensorflow, you can save your weights and continue training after some point. So, you can:

iterate through your images in batches
load the model weights and train the model
save the weights for the next round

But easier way is to use default functions implemented in pytorch for example that does the same. For example using ImageFolder and DataLoader:
def load_data(data_folder, batch_size, train, kwargs):
    transform = {
        'train': transforms.Compose(
            [transforms.Resize([256, 256]),
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                  std=[0.229, 0.224, 0.225])]),
        'test': transforms.Compose(
            [transforms.Resize([224, 224]),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                  std=[0.229, 0.224, 0.225])])
        }
    data = datasets.ImageFolder(root = data_folder, transform=transform['train' if train else 'test'])
    data_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=True, **kwargs, drop_last = True if train else False)
    return data_loader

You can specify the folder that contains images ending with various extensions (e.g. .jpg, .png. etc.) and then training your model by passing data_loader to.


Instruction: Training with parametric partial derivatives in pytorch

Input:
Given a neural network with weights theta and inputs x, I am interested in calculating the partial derivatives of the neural network's output w.r.t. x, so that I can use the result when training the weights theta using a loss depending both on the output and the partial derivatives of the output. I figured out how to calculate the partial derivatives following this post. I also found this post that explains how to use sympy to achieve something similar, however, adapting it to a neural network context within pytorch seems like a huge amount of work and a recipee for very slow code.
Thus, I tried something different, which failed. As a minimal example, I created a function (substituting my neural network)
theta = torch.ones([3], requires_grad=True, dtype=torch.float32)
def trainable_function(time):
    return theta[0]*time**3 + theta[1]*time**2 + theta[2]*time

Then, I defined a second function to give me partial derivatives:
def trainable_derivative(time):
    deriv_time = torch.tensor(time, requires_grad=True)
    fun_value = trainable_function(deriv_time)
    gradient = torch.autograd.grad(fun_value, deriv_time, create_graph=True, retain_graph=True)
    deriv_time.requires_grad = False
    return gradient

Given some noisy observations of the derivatives, I now try to train theta. For simplicity, I create a loss that only depends on the derivatives. In this minimal example, the derivatives are used directly as observations, not as regularization, to avoid complicated loss functions that are besides the point.
def objective(train_times, observations):
    predictions = torch.squeeze(torch.tensor([trainable_derivative(a) for a in train_times]))
    return torch.sum((predictions - observations)**2)

optimizer = Adam([theta], lr=0.1)
for iteration in range(200):
    optimizer.zero_grad()
    loss = objective(data_times, noisy_targets)
    loss.backward()
    optimizer.step()

Unfortunately, when running this code, I get the error
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I suppose that when calculating the partial derivatives in the way I do, I do not really create a computational graph through which autodiff could differentiate through. Thus, the connection to the parameters theta somehow gets lost and now it looks to the optimizer as if the loss is completely independent of the parameters theta. However, I could be totally wrong..
Does anyone know how to fix this?
Is it possible to include this type of derivatives in the loss function in pytorch?
And if so, what would be the most pytorch-style way of doing this?
Many thanks for your help and advise, it is much appreciated.
For completeness:
To run the above code, some training data needs to be generated. I used the following code, which works perfectly and has been tested against the analytical derivatives:
true_a = 1
true_b = 1
true_c = 1

def true_function(time):
    return true_a*time**3 + true_b*time**2 + true_c*time

def true_derivative(time):
    deriv_time = torch.tensor(time, requires_grad=True)
    fun_value = true_function(deriv_time)
    return torch.autograd.grad(fun_value, deriv_time)

data_times = torch.linspace(0, 1, 500)
true_targets = torch.squeeze(torch.tensor([true_derivative(a) for a in data_times]))
noisy_targets = torch.tensor(true_targets) + torch.randn_like(true_targets)*0.1

Output:
Your approach to the problem appears overly complicated.
I believe that what you're trying to achieve is within reach in PyTorch.
I include here a simple code snippet that I believe showcases what you would like to do:
import torch
import torch.nn as nn

# Data and Function
torch.manual_seed(0)
input_dim  = 1
output_dim = 2
n = 10 # batchsize
simple_function = nn.Sequential(nn.Linear(1, 2), nn.Sigmoid())
t = (torch.arange(n).float() / n).view(n, 1)
x = torch.randn(n, output_dim)
t.requires_grad = True

# Actual computation
xhat = simple_function(t)
jac = torch.autograd.functional.jacobian(simple_function, t, create_graph=True)
grad = jac[torch.arange(n),:,torch.arange(n),0]
loss = (x -xhat).pow(2).sum() + grad.pow(2).sum()
loss.backward()


Instruction: How to create a 1 to 1 feed forward layer?

Input:
I'm familiar with a fully connected layer, but how can I create a custom layer in PyTorch that is just 1 to 1? That is, each neuron is only connected to 1 other neuron.
Example: Layer 1 neurons: a,b,c
Layer 2 neurons d,e,f
Connections:
a-d
b-e
c-f

Output:
Linear layers are basically just describing a matrix multiplication. And since this is not what you want you can't use the Pytorch implementation nn.Linear. You want to have each weight correspond to just one input neuron and one output neuron. That would mean that the amount of output neurons must be the same as the amount of input neurons (which limits the possibilities of a neural network very much, are you sure you want that?).
So it is basically just elementwise multiplication where one factor is learnable.
An example of what this kind if linear layer could look like:
class ElementwiseLinear(nn.Module):
    def __init__(self, input_size: int) -> None:
        super(ElementwiseLinear, self).__init__()

        # w is the learnable weight of this layer module
        self.w = nn.Parameter(torch.rand(input_size), requires_grad=True)

    def forward(self, x: torch.tensor) -> torch.tensor:
        # simple elementwise multiplication
        return self.w * x

Now an example of how to use it in a model:
class Model(nn.Module):
    def __init__(self, input_size: int) -> None:
        super(Model, self).__init__()

        self.elementWiselinear1 = ElementwiseLinear(input_size)
        self.elementWiselinear2 = ElementwiseLinear(input_size)
        self.elementWiselinear3 = ElementwiseLinear(input_size)

    def forward(self, x: torch.tensor) -> torch.tensor:
        x = F.relu(self.elementWiselinear1(x))
        x = F.relu(self.elementWiselinear2(x))
        x = torch.sigmoid(self.elementWiselinear3(x))

        return x

Again, I dont know what this could be useful for but I hope thats what you wanted!


Instruction: Am getting error trying to predict on a single image CNN pytorch

Input:
Error message
Traceback (most recent call last):
File "pred.py", line 134, in
output = model(data)
Runtime Error: Expected 4-dimensional input for 4-dimensional weight [16, 3, 3, 3], but got 3-dimensional input of size [1, 32, 32] instead.
Prediction code
normalize = transforms.Normalize(mean=[0.4914, 0.4824, 0.4467],
                                     std=[0.2471, 0.2435, 0.2616])
train_set = transforms.Compose([
                                 transforms.RandomCrop(32, padding=4),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.ToTensor(),
                                 normalize,
                                     ])

model = models.condensenet(args)
model = nn.DataParallel(model)
PATH = "results/savedir/save_models/checkpoint_001.pth.tar"

model.load_state_dict(torch.load(PATH)['state_dict'])

device = torch.device("cpu")

model.eval()

image = Image.open("horse.jpg")
input = train_set(image)
train_loader = torch.utils.data.DataLoader(
        input,
        batch_size=1,shuffle=True, num_workers=1)
for i, data in enumerate(train_loader):

    #input_var = torch.autograd.Variable(data, volatile=True)
    #input_var = input_var.view(1, 3, 32,32)

    **output = model(data)
topk=(1,5)
maxk = max(topk)

_, pred = output.topk(maxk, 1, True, True)

Am getting this error when am trying to predict on a single image
Image shape/size error message
Link to saved model
Training code repository

Output:
Instead of doing the for loop and train_loader, solved this by just passing the input directly into the model. like this
input = train_set(image)
input = input.unsqueeze(0)
model.eval()
output = model(input)

More details can be found here link


Instruction: Pytorch DDP get stuck in getting free port

Input:
I try to get a free port in DDP initialization of PyTorch. However, my code get stuck. The following snippet could repeat my description:
def get_open_port():
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return s.getsockname()[1]

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    port = get_open_port()
    os.environ['MASTER_PORT'] = str(port)   # '12345'

    # Initialize the process group.
    dist.init_process_group('NCCL', rank=rank, world_size=world_size)

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 5)

    def forward(self, x):
        print(f'x device={x.device}')
        return self.net1(x)

def demo_basic(rank, world_size):
    setup(rank, world_size)

    logger = logging.getLogger('train')
    logger.setLevel(logging.DEBUG)
    logger.info(f'Running DPP on rank={rank}.')

    # Create model and move it to GPU.
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)  # optimizer takes DDP model.

    optimizer.zero_grad()
    inputs = torch.randn(20, 10)  # .to(rank)

    print(f'inputs device={inputs.device}')
    outputs = ddp_model(inputs)
    print(f'output device={outputs.device}')

    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()

def run_demo(demo_func, world_size):
    mp.spawn(
        demo_func,
        args=(world_size,),
        nprocs=world_size,
        join=True
    )

run_demo(demo_basic, 4)

The function get_open_port is supposed to free the port after invocation. My questions are: 1. How does it happen? 2. How to fix it?

Output:
The answer is derived from here. The detailed answer is: 1. Since each free port is generated from individual process, ports are different in the end; 2. We could get a free port at the beginning and pass it to processes.
The corrected snippet:
def get_open_port():
    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return s.getsockname()[1]

def setup(rank, world_size, port):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = str(port)

    # Initialize the process group.
    dist.init_process_group('NCCL', rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 5)

    def forward(self, x):
        print(f'x device={x.device}')
        # return self.net2(self.relu(self.net1(x)))
        return self.net1(x)

def demo_basic(rank, world_size, free_port):
    setup(rank, world_size, free_port)

    logger = logging.getLogger('train')
    logger.setLevel(logging.DEBUG)
    logger.info(f'Running DPP on rank={rank}.')

    # Create model and move it to GPU.
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)  # optimizer takes DDP model.

    optimizer.zero_grad()
    inputs = torch.randn(20, 10)  # .to(rank)

    print(f'inputs device={inputs.device}')
    outputs = ddp_model(inputs)
    print(f'output device={outputs.device}')

    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()

    optimizer.step()

    cleanup()

def run_demo(demo_func, world_size, free_port):
    mp.spawn(
        demo_func,
        args=(world_size, free_port),
        nprocs=world_size,
        join=True
    )

free_port = get_open_port()
run_demo(demo_basic, 4, free_port)


Instruction: MT5ForConditionalGeneration with Pytorch-lightning gives attribute_error

Input:
Can't handle that problem for several days
I'm new to NLP and the solution is probably very simple
class QAModel(pl.LightningDataModule):

  def __init__(self):
    super().__init__()
    self.model = MT5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)

  def forward(self, input_ids, attention_mask, labels=None):
    output = model(
        input_ids=input_ids,
        attention_mask=attention_mask,
        labels=labels
    )

    return output.loss, output.logits

  def training_step(self, batch, batch_idx):
    input_ids = batch['input_ids']
    attention_mask = batch['attention_mask']
    labels = batch['labels']
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log('train_loss', loss, prog_bar=True, logger=True)
    return loss

  def validation_step(self, batch, batch_idx):
    input_ids = batch['input_ids']
    attention_mask = batch['attention_mask']
    labels = batch['labels']
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log('val_loss', loss, prog_bar=True, logger=True)
    return loss

  def test_step(self, batch, batch_idx):
    input_ids = batch['input_ids']
    attention_mask = batch['attention_mask']
    labels = batch['labels']
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log('test_loss', loss, prog_bar=True, logger=True)
    return loss

  def configure_optimizers(self):
    return AdamW(self.parameters(), lr=0.0001)
model = QAModel()
from pytorch_lightning.callbacks import ModelCheckpoint

checkpoint_callback = ModelCheckpoint(
    dirpath='/content/checkpoints',
    filename='best-checkpoint',
    save_top_k=1,
    verbose=True,
    monitor='val_loss',
    mode='min'
)
trainer = pl.Trainer(
    checkpoint_callback=checkpoint_callback,
    max_epochs=N_EPOCHS,
    gpus=1,
    progress_bar_refresh_rate=30
)
trainer.fit(model, data_module)

Running this code gives me
AttributeError: 'QAModel' object has no attribute 'automatic_optimization'
after fit() function
Probably, the problem is in MT5ForConditionalGeneration, as after passing it to funtion() we've got the same error

Output: Try inheriting pl.LightingModule instead of pl.LightningDataModule. It is the right choice for defining a model class.


Instruction: how do I eliminate the for-loop in this pytorch code

Input:
I have this custom pytorch module (below). It does exactly what I need; it just does it very slowly. What can I do to speed this up? I know that I'm not supposed to have a for-loop in there; it just wasn't clear how to do that divide operation without it. How do I broadcast the x tensor to the divide without that loop? I can move the back-weights to their own layer if that helps.
class StepLayer(nn.Module):
    def __init__(self):
        super(StepLayer, self).__init__()
        w = init_weights()
        self.front_weights = nn.Parameter(torch.DoubleTensor([w, w]).T, requires_grad=True)
        self.back_weights = nn.Parameter(torch.DoubleTensor([w]).T, requires_grad=True)

    def forward(self, x):
        # x shape is batch by feature
        results = []
        for batch in x:
            b = batch.divide(self.front_weights)
            b = torch.some_math_function(b)
            b = b.sum(dim=1)
            b = torch.some_other_math_function(b)
            b = b @ self.back_weights
            results.append(b)
        stack = torch.vstack(results)
        return stack

Output:
Below is a source code with shape after each step described (read code comments please).
I've assumed a few things like F=100, x=Bx2, front_weights=100x2, back_weights=100, you should be able to easily adjust it to your case.
class StepLayer(nn.Module):
    def __init__(self):
        super().__init__()
        F = 100
        # Notice I've added `1` dimension in front_weights
        self.front_weights = nn.Parameter(torch.randn(1, F, 2), requires_grad=True)
        self.back_weights = nn.Parameter(torch.randn(F), requires_grad=True)

    def forward(self, x):
        # x.shape == (B, 2)
        x = x.unsqueeze(dim=1)  # (B, 1, 2)
        x = x / self.front_weights  # (B, F, 2)
        # I just took some element-wise math function from PyTorch
        x = torch.sin(x)  # (B, F, 2)
        x = torch.sum(x, dim=-1)  # (B, F)
        x = torch.sin(x)  # (B, F)
        return x @ self.back_weights  # (B, )

        # results = []
        # for batch in x:
        #     # batch - (1, 2)
        #     b = batch.divide(self.front_weights)  # (F, 2)
        #     b = torch.some_math_function(b)  # (F, 2)
        #     b = b.sum(dim=1)  # (F, )
        #     b = torch.some_other_math_function(b)  # (F, )
        #     b = b @ self.back_weights  # (1, )
        #     results.append(b)
        # stack = torch.vstack(results)  # (B, )
        # return stack  # (B,)

layer = StepLayer()

print(layer(torch.randn(64, 2)).shape)

Main trick is to use 1 dimensions for broadcasting when necessary (division especially) and smart weight initialization so you don't have to do any transpose operations.
Other things

You might want to reconsider Double, float (as used above) is much faster, especially on CUDA and takes half the memory (and neural network should compensate for precision loss, if any).
Use half precision and mixed training if the speed is still an issue (float16 dtype instead of float32), but only on CUDA, see here to know more about automated mixed precision


Instruction: Is there a good way to modify some values in a pytorch tensor while preserving the autograd functionality?

Input:
Sometimes I need to modify some of the values in a pytorch tensor. For example, given a tensor x, I need to multiply its positive part by 2 and multiply its negative part by 3:
import torch

x = torch.randn(1000, requires_grad=True)
x[x>0] = 2 * x[x>0]
x[x<0] = 3 * x[x<0]

y = x.sum()
y.backward()

However such inplace operations always break the graph for autograd:
Traceback (most recent call last):
  File "test_rep.py", line 4, in <module>
    x[x>0] = 2 * x[x>0]
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

Therefore, so far I've been using the following workaround:
import torch

x = torch.randn(1000, requires_grad=True)
y = torch.zeros_like(x, device=x.device)

y[x>0] = 2 * x[x>0]
y[x<0] = 3 * x[x<0]

z = y.sum()
z.backward()

which results in manually creating new tensors. I wonder if there is a better way to do this.

Output:
How about like following?
import torch

x = torch.randn(1000, requires_grad=True)
x = torch.where(x>0, x*2, x)
x = torch.where(x<0, x*3, x)

y = x.sum()
y.backward()


Instruction: Reward of Pong game - (OpenAI gym)

Input:
I know that the Pong Game initializes to new game when one side scores 20 points.
By the way, the reward shows that it goes down below -20.
Why is that so?
One thing to expect is that after one side gets 20 points, the game is reset by playing one more time. Does the game need to get 21 points to initialize?
(Use 8 workers, A2C, PongNoFrameskip-v4)

Output: Pong is played to 21 points, not 20.


Instruction: DCGANs discriminator accuracy metric using PyTorch

Input:
I am implementing DCGANs using PyTorch.
It works well in that I can get reasonable quality generated images, however now I want to evaluate the health of the GAN models by using metrics, mainly the ones introduced by this guide https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/
Their implementation uses Keras which SDK lets you define what metrics you want when you compile the model, see https://keras.io/api/models/model/. In this case the accuracy of the discriminator, i.e. percentage of when it successfully identifies an image as real or generated.
With the PyTorch SDK, I can't seem to find a similar feature that would help me easily acquire this metric from my model.
Does Pytorch provide the functionality to be able to define and extract common metrics from a model?

Output:
Pure PyTorch does not provide metrics out of the box, but it is very easy to define those yourself.
Also there is no such thing as "extracting metrics from model". Metrics are metrics, they measure (in this case accuracy of discriminator), they are not inherent to the model.
Binary accuracy
In your case, you are looking for binary accuracy metric. Below code works with either logits (unnormalized probability outputed by discriminator, probably last nn.Linear layer without activation) or probabilities (last nn.Linear followed by sigmoid activation):
import typing
import torch

class BinaryAccuracy:
    def __init__(
        self,
        logits: bool = True,
        reduction: typing.Callable[
            [
                torch.Tensor,
            ],
            torch.Tensor,
        ] = torch.mean,
    ):
        self.logits = logits
        if logits:
            self.threshold = 0
        else:
            self.threshold = 0.5

        self.reduction = reduction

    def __call__(self, y_pred, y_true):
        return self.reduction(((y_pred > self.threshold) == y_true.bool()).float())

Usage:
metric = BinaryAccuracy()
target = torch.randint(2, size=(64,))
outputs = torch.randn(size=(64, 1))

print(metric(outputs, target))

PyTorch Lightning or other third party
You can also use PyTorch Lightning or other framework on top of PyTorch which defines metrics like accuracy


Instruction: Pytorch crop images from Top Left Corner in Transforms

Input:
I'm using Pytorch's transforms.Compose and in my dataset I have 1200x1600 (Height x Width) images.
I want to crop the images starting from the Top Left Corner (0,0) so that I can have 800x800 images.
I was looking in Pytorch documentation but I didn't find anything to solve my problem, so I copied the source code of center_crop in my project and modified it as follows:
def center_crop(img: Tensor, output_size: List[int]):
    # .... Other stuff of Pytorch

    # ....
    # Original Pytorch Code (that I commented)
    crop_top = int((image_height - crop_height + 1) * 0.5)
    crop_left = int((image_width - crop_width + 1) * 0.5)

    # ----
    # My modifications:
    crop_top = crop_left = 0

    return crop(img, crop_top, crop_left, crop_height, crop_width)

But basically I think this is quite an overkill, if it's possible I'd like to avoid to copy their code and modify it.
Isn't there anything that already implements the desired behaviour by default, is there?

Output:
I used Lambda transforms in order to define a custom crop
from torchvision.transforms.functional import crop

def crop800(image):
    return crop(image, 0, 0, 800, 800)

data_transforms = {
    'images': transforms.Compose([transforms.ToTensor(),
                                  transforms.Lambda(crop800),
                                  transforms.Resize((400, 400))])}


Instruction: How does PyTorch DataLoader interact with a PyTorch dataset to transform batches?

Input:
I'm creating a custom dataset for NLP-related tasks.
In the PyTorch custom datast tutorial, we see that the __getitem__() method leaves room for a transform before it returns a sample:
def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = os.path.join(self.root_dir,
                                self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)

        ### SOME DATA MANIPULATION HERE ###

        sample = {'image': image, 'landmarks': landmarks}
        if self.transform:
            sample = self.transform(sample)

        return sample

However, the code here:
        if torch.is_tensor(idx):
            idx = idx.tolist()

implies that multiple items should be able to be retrieved at a time which leaves me wondering:

How does that transform work on multiple items? Take the custom transforms in the tutorial for example. They do not look like they could be applied to a batch of samples in a single call.

Related, how does a DataLoader retrieve a batch of multiple samples in parallel and apply said transform if the transform can only be applied to a single sample?

Output:
How does that transform work on multiple items? They work on multiple items through use of the data loader. By using transforms, you are specifying what should happen to a single emission of data (e.g., batch_size=1). The data loader takes your specified batch_size and makes n calls to the __getitem__ method in the torch data set, applying the transform to each sample sent into training/validation. It then collates n samples into your batch size emitted from the data loader.

Related, how does a DataLoader retrieve a batch of multiple samples in parallel and apply said transform if the transform can only be applied to a single sample? Hopefully above makes sense to you. Parallelization is done by the torch data set class and the data loader, where you specify num_workers. Torch will pickle the data set and spread it across workers.


Instruction: Adding custom weights to training data in PyTorch

Input: Is it possible to add custom weights to the training instances in PyTorch? More explicitly, I'd like to add a custom weight for every row in my dataset. By default, the weights are 1, which means every data is equally important for my model.

Output:
Loss functions support class weights not sample weights. For sample weights you can do something like below (commented inline):
import torch

x = torch.rand(8, 4)
# Ground truth
y = torch.randint(2, (8,))
# Weights per sample
weights = torch.rand(8, 1)

# Add weights as a columns, so that it will be passed trough
# dataloaders in case you want to use one
x = torch.cat((x, weights), dim=1)

model = torch.nn.Linear(4, 2)

loss_fn = torch.nn.CrossEntropyLoss(reduction='none')
def weighted_loss(y, y_hat, w):
  return (loss_fn(y, y_hat)*w).mean()

loss = weighted_loss(model(x[:, :-1]), y, x[:, -1])
print (loss)


Instruction: Computing Cosine Distance with Differently shaped tensors

Input:
I have the following tensor  representing a word vector
A = (2, 500)

Where the first dimension is the BATCH dimension (i.e. A contains two word vectors each with 500 elements)
I also have the following tensor
B = (10, 500)

I want to compute the cosine distance between A and B such that I get
C = (2, 10, 1)

i.e for each row in A compute the cosine distance with each row in B
I looked at using torch.nn.functional.F.cosine_similarity however this doesn't work as  the dimensions must be the same.
Whats the best efficient way of achieving this in pytorch?

Output:
Use broadcasting technique with unsqueeze
import torch.nn.functional as F

C = F.cosine_similarity(A.unsqueeze(1), B, dim=-1)

print(C.shape)
# torch.size([2,10])


Instruction: How to get a column from a tensor?

Input:
Let's say I have a tensor consisting of 1 and 0's as shown below. How can I get the index of a specific column to replace with new values ? If I want to replace the values of column 1 with the [3.,4.,5.,6.], how do I accomplish this ?
a = torch.tensor([[[1., 0., 0., 0.]],
        [[0., 1., 0., 0.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]]])

Output:
Calling them 'columns' is a bit tricky, given that this is a 3D tensor.
This will do what you need, setting 'column' 1 to the values you gave.
a = torch.tensor([[[1., 0., 0., 0.]],
        [[0., 1., 0., 0.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]]])

# Change values in 'column' 1 (zero-indexed):
# The 0 is there because of the size-1 second dimension.
a[1, 0, :] = torch.tensor([3., 4., 5., 6.])

print(a)
# tensor([[[1., 0., 0., 0.]],
#         [[3., 4., 5., 6.]],
#         [[1., 0., 0., 0.]],
#         [[0., 0., 0., 1.]],
#         [[1., 0., 0., 0.]],
#         [[0., 0., 0., 1.]],
#         [[1., 0., 0., 0.]]])


Instruction: Why are embeddings necessary in Pytorch?

Input:
I (think) I understand the basic principle behind embeddings: they're a shortcut to quickly perform the operation (one hot encoded vector) * matrix without actually performing that operation (by utilizing the fact that this operation is equivalent to indexing into the matrix) while still maintaining the same gradient as if that operation was actually performed.
I know in the following example:
e = Embedding(3, 2)
n = e(torch.LongTensor([0, 2]))

n will be a tensor of shape 2, 2.
But, we could also do:
p = nn.Parameter(torch.zeros([3, 2]).normal_(0, 0.01))
p[tensor([0, 2])]

and get the same result without an embedding.
This in and of itself wouldn't be confusing since in the first example n has a grad_fn called EmbeddingBackward whereas in the 2nd example p has a grad_fn called IndexBackward, which is what we expect since we know embeddings simulate a different derivative.
The confusing part is in chapter 8 of the fastbook they use embeddings to compute movie recommendations. But then, they do it without embeddings in basically the same manner & the model still works. I would expect the version without embeddings to fail because the derivative would be incorrect.
Version with embeddings:
class DotProductBias(Module):
    def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
        self.user_factors = Embedding(n_users, n_factors)
        self.user_bias = Embedding(n_users, 1)
        self.movie_factors = Embedding(n_movies, n_factors)
        self.movie_bias = Embedding(n_movies, 1)
        self.y_range = y_range

    def forward(self, x):
        users = self.user_factors(x[:,0])
        movies = self.movie_factors(x[:,1])
        res = (users * movies).sum(dim=1, keepdim=True)
        res += self.user_bias(x[:,0]) + self.movie_bias(x[:,1])
        return sigmoid_range(res, *self.y_range)

Version without:
def create_params(size):
    return nn.Parameter(torch.zeros(*size).normal_(0, 0.01))

class DotProductBias(Module):
    def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
        self.user_factors = create_params([n_users, n_factors])
        self.user_bias = create_params([n_users])
        self.movie_factors = create_params([n_movies, n_factors])
        self.movie_bias = create_params([n_movies])
        self.y_range = y_range

    def forward(self, x):
        users = self.user_factors[x[:,0]]
        movies = self.movie_factors[x[:,1]]
        res = (users*movies).sum(dim=1)
        res += self.user_bias[x[:,0]] + self.movie_bias[x[:,1]]
        return sigmoid_range(res, *self.y_range)

Does anyone know what is going on?

Output:
Besides the points provided in the comments:

I (think) I understand the basic principle behind embeddings: they're
a shortcut to quickly perform the operation (one hot encoded vector) *
matrix without actually performing that operation

This one is only partially correct. Indexing into matrix is fully differentiable operation, it's just taking part of the data and propagate gradient this path only. Multiplication here would be wasteful and unnecessary.

and get the same result without an embedding.

This one is true

called EmbeddingBackward whereas in the 2nd example p has a grad_fn called IndexBackward, which is what we expect since we know embeddings simulate a different derivative. (emphasis mine)

This one isn't (or often is not). Embedding also chooses part of the data, just like you did, grad_fn is different due to "extra functionalities", but in principle it is the same (choosing some vector(s) from the matrix), think of it like IndexBackward on steroids.

The confusing part is in chapter 8 of the fastbook they use embeddings
to compute movie recommendations. But then, they do it without
embeddings in basically the same manner & the model still works. I
would expect the version without embeddings to fail because the
derivative would be incorrect. (emphasis mine)

In this exact case, both approaches are equivalent (give or take different results from random initialization).
Why nn.Embedding at all?

Easier to understand your intent when reading the code
Common utilities and functionality added if needed (as pointed out in the comments)


Instruction: Tensor creation and destruction within a while loop in libtorch C++

Input:
I have just started with libtorch and I am having some trouble with while loops and tensors :roll_eyes:
So, my main func looks like so:
int main()
{

  auto tensor_create_options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCPU).requires_grad(false);
  torch::Tensor randn_tensor = torch::randn({10}, tensor_create_options);

  int randn_tensor_size = randn_tensor.sizes()[0];

  while (randn_tensor_size > 5)
  {
    std::cout << "==> randn_tensor shape: " << randn_tensor.sizes() << std::endl;
    randn_tensor.reset(); //reset();
    torch::Tensor randn_tensor = torch::randn({3}, tensor_create_options);
    std::cout << "==> randn_tensor shape 3: " << randn_tensor.sizes() << std::endl;
randn_tensor_size--;
  }

  return 0;
}

and I get thrown this:
==> randn_tensor shape: [10]
==> randn_tensor shape 3: [3]
terminate called after throwing an instance of 'c10::Error'
  what():  sizes() called on undefined Tensor

Essentially, what I want to do is recreate the tensor within the while loop and ensure that I can access it again in the whileloop.
Interestingly, it seems to have cerated the tensor of reduced size but the while loop does not seem to recognise this.
Thank you!

Output:
You have a shadowing issue here, try the loop that way:
while (randn_tensor_size > 5)
{
    std::cout << "==> randn_tensor shape: " << randn_tensor.sizes() << std::endl;
    randn_tensor.reset(); //reset();
    randn_tensor = torch::randn({3}, tensor_create_options);
    std::cout << "==> randn_tensor shape 3: " << randn_tensor.sizes() << std::endl;
    randn_tensor_size--;
}

Maybe, the reset of the tensor isn't necessary further on, depends of the internals of this class. If that's not the actual intention of your code and you want only the original tensor to be deleted, then simply reset that one right before the loop. Indepently of this, try to make the code clearer in terms of intention emphasis! I do not really understand what you want to achieve exactly. Your loop counter is misued at all since you mix size and counting semantics, depending on the initial size only. Within the loop, you simply recreate the tensor on the stack again and again, not affecting your counter.


Instruction: How to obtain the values from each row at given position in pytorch?

Input:
How to obtain the values from a 2-d torch array based on 1-d array containing the positions at each row:
for example:
a = torch.randn((5,5))
>>> a
tensor([[ 0.0740, -0.3129,  0.7814, -0.0519,  1.3503],
        [ 1.1985,  0.2098, -0.0326,  0.3922,  0.5037],
        [-1.4334,  1.4047, -0.6607, -1.8024, -0.0088],
        [ 1.2116,  0.5928,  1.4041,  1.0494, -0.1146],
        [ 0.4173,  1.0482,  0.5244, -2.1767,  0.5264]])

b = torch.randint(0,5, (5,))
>>> b
tensor([1, 0, 1, 3, 2])

I wanted to get the elements of tensor a at the position given by tensor b
For example:
desired output:
tensor([-0.3129,
        1.1985,
        1.4047,
        1.0494,
        0.5244])

Here, each element in a given position by tensor b is chosen row-wise.
I have tried:
for index in range(b.size(-1)):
    val = torch.cat((val,a[index,b[index]].view(1,-1)), dim=0) if val is not None else a[index,b[index]].view(1,-1)

>>> val
tensor([[-0.3129],
        [ 1.1985],
        [ 1.4047],
        [ 1.0494],
        [ 0.5244]])

However, is there a tensor indexing way to do it?
I tried couple of solutions using tensor indexing, but none of them worked.

Output:
You can use torch.gather
>>> a.gather(1, b.unsqueeze(1))
tensor([[-0.3129],
        [ 1.1985],
        [ 1.4047],
        [ 1.0494],
        [ 0.5244]])

Or
>>> a[range(len(a)), b].unsqueeze(1)
tensor([[-0.3129],
        [ 1.1985],
        [ 1.4047],
        [ 1.0494],
        [ 0.5244]])


Instruction: How can I add more neurons / filters to a neural network model after training?

Input:
I'm interested in training both a CNN model and a simple linear feed forward model in PyTorch, and after training to add more filters -- to the CNN layers, & neurons -- to the linear model layers and the outputs (e.g. from binary classification to multiclass classification) of both. By adding them I specifically mean to keep the weights that were trained constant, and to add random initialized weights to the new, incoming weights.
There's an example of a CNN model here, and an example of a simple linear feed forward model here

Output:
This one was a bit tricky and requires slice (see this answer for more info about slice, but it should be intuitive). Also this answer for slice trick. Please see comments for explanation:
import torch

def expand(
    original: torch.nn.Module,
    *args,
    **kwargs
    # Add other arguments if needed, like different stride
    # They won't change weights shape, but may change behaviour
):
    new = type(original)(*args, **kwargs)

    new_weight_shape = torch.tensor(new.weight.shape)
    new_bias_shape = torch.tensor(new.bias.shape)

    original_weight_shape = torch.tensor(original.weight.shape)
    original_bias_shape = torch.tensor(original.bias.shape)
    # I assume bias and weight exist, if not, do some checks
    # Also quick check, that new layer is "larger" than original
    assert torch.all(new_weight_shape >= original_weight_shape)
    assert new_bias_shape >= original_bias_shape

    # All the weights will be inputted from top to bottom, bias 1D assumed
    new.bias.data[:original_bias_shape] = original.bias.data

    # Create slices 0:n for each dimension
    slicer = tuple([slice(0, dim) for dim in original_weight_shape])
    # And input the data
    new.weight.data[slicer] = original.weight.data

    return new

layer = torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3)

new = expand(layer, in_channels=32, out_channels=64, kernel_size=3)

This should work for any layer (which has weight and bias, adjust if needed). Using this approach you can recreate your neural network or use PyTorch's apply (docs here)
Also remember, that you have to explicitly pass creational *args and **kwargs for "new layer" which will have trained connections inputted.


Instruction: How to decipher these 2 lines of code? What does it mean?

Input:
This is the code:
real_idx = torch.rand(len(real)) > 0.2
real_selected = real[real_idx][: int(len(real)*p_real)]

p_real is actually 0.2
We're supposed to mix 2 tensors, namely the real and the fake tensor.

Output:
The first line creates a random mask the size of real. The second line applies this mask on real and slices out 20% the size of real from whatever the masking operation returned.
Now if what you meant to ask was why is this being done, then you'd have to give more details.


Instruction: can't convert pytorch tensor dtype from float64 to double

Input:
I need to convert an int to a double tensor, and I've already tried several ways including torch.tensor([x], dtype=torch.double), first defining the tensor and then converting the dtype to double with x_tensor.double(), and also defining the tensor with torch.DoubleTensor([x]) but none actually change the dtype from torch.float64. Here's the code snippet
>>> x = 6
>>> x = torch.tensor([x], dtype=torch.double)
>>> x
tensor([6.], dtype=torch.float64)

>>> x = 6
>>> x_tensor = torch.Tensor([x])
>>> x_double = x_tensor.double()
>>> x_double
tensor([6.], dtype=torch.float64)

>>> x_tensor = torch.DoubleTensor([x])
>>> x_tensor
tensor([6.], dtype=torch.float64)

Any ideas as to why it's not converting?

Output: It is converting; torch.float64 and torch.double are the same thing


Instruction: TypeError: Can't convert re.compile('[A-Z]+') (re.Pattern) to Union[str, tokenizers.Regex]

Input:
I'm having issues applying a Regex expression to a Split() operation found in the HuggingFace Library. The library requests the following input for Split().

pattern (str or Regex) – A pattern used to split the string. Usually a
string or a Regex

In my code I am applying the Split() operation like so:
tokenizer.pre_tokenizer = Split(pattern="[A-Z]+", behavior='isolated')

but it's not working because [A-Z]+ is being interpreted as  a string not a Regex expression. I've used the following to no avail:
pattern = re.compile("[A-Z]+")
tokenizer.pre_tokenizer = Split(pattern=pattern, behavior='isolated')

Getting the following error:

TypeError: Can't convert re.compile('[A-Z]+') (re.Pattern) to Union[str, tokenizers.Regex]

Output:
The following solution worked by importing Regex from the tokenizers library:
from tokenizers import Regex

tokenizer.pre_tokenizer = Split(pattern=Regex("[A-Z]+"),
                                behavior='isolated')


Instruction: Torch: How to inspect weights after training?

Input:
I am wondering what I am doing wrong when looking to see how the weights changed during training.
My loss goes down considerably but it appears that the initialized weights are the same as trained weights. Am I looking in the wrong location? I would appreciate any insight that you might have!
import torch
import numpy as np
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import torch.nn.functional as F

# setup GPU/CPU processing
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# initialize model
class mlp1(torch.nn.Module):
    def __init__(self, num_features, num_hidden, num_classes):
        super(mlp1, self).__init__()
        self.num_classes = num_classes
        self.input_layer = torch.nn.Linear(num_features, num_hidden)
        self.out_layer = torch.nn.Linear(num_hidden, num_classes)

    def forward(self, x):
        x = self.input_layer(x)
        x = torch.sigmoid(x)
        logits = self.out_layer(x)
        probas = torch.softmax(logits, dim=1)
        return logits, probas

# instantiate model
model = mlp1(num_features=28*28, num_hidden=100, num_classes=10).to(device)

# check initial weights
weight_check_pre = model.state_dict()['input_layer.weight'][0][0:25]

# optim
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# download data
train_dataset = datasets.MNIST(root='data',
                               train=True,
                               transform=transforms.ToTensor(),
                               download=True)
# data loader
train_dataloader = DataLoader(dataset=train_dataset,
                          batch_size=100,
                          shuffle=True)

# train
NUM_EPOCHS = 1
for epoch in range(NUM_EPOCHS):
    model.train()
    for batch_idx, (features, targets) in enumerate(train_dataloader):
        # send data to device
        features = features.view(-1, 28*28).to(device)
        targets = targets.to(device)
        # forward
        logits, probas = model(features)
        # loss
        loss = F.cross_entropy(logits, targets)
        optimizer.zero_grad()
        loss.backward()
        # now update weights
        optimizer.step()
        ### LOGGING
        if not batch_idx % 50:
            print ('Epoch: %03d/%03d | Batch %03d/%03d | Loss: %.4f'
                   %(epoch+1, NUM_EPOCHS, batch_idx,
                     len(train_dataloader), loss))

# check post training
weight_check_post = model.state_dict()['input_layer.weight'][0][0:25]

# compare
weight_check_pre == weight_check_post  # all equal

Output:
That is because both variables are referencing the same object (dictionary) in memory and so will always equal to each other.
You can do this to get actual copies of the state_dict.
import copy

# check initial weights
weight_check_pre = copy.deepcopy(model.state_dict()['input_layer.weight'][0][0:25])
...
# check post training
weight_check_post = copy.deepcopy(model.state_dict()['input_layer.weight'][0][0:25])


Instruction: which object detection algorithms can extract the QR code from an image efficiently

Input:
I am new to Object Detection, for now, I want to predict the QR code in images. I want to extract the QR code from the images, and only predict the QR code without the background information, and finally predict the exact number the QR code is representing, since I am using PyTorch, is there any object detection algorithm which is compatible to PyTorch that I could apply to this task?
(for what I mean by extracting, the raw input is an image,I want to change the input of the image to the QR code in the image).

Output:
There are two ways for this task:
Computer Vision based approach:
OpenCV library's QRCodeDetector() function can detect and read QR codes easily. It returns data in QR code and bounding box information of the QR code:
import cv2
detector = cv2.QRCodeDetector()
data, bbox, _ = detector.detectAndDecode(img)

Deep learning  based approach:
Using common object detection framework - Yolo (Yolo v5 in PyTorch), you can achieve your target. However, you need data to train it. For computer vision based approach, you don't need to do training or data collection.
You may consider reading these two.


Instruction: Pytorch Resnet model error if FC layer is changed in Colab

Input:
If I simply import the Resnet Model from Pytorch in Colab, and use it to train my dataset, there are no issues. However, when I try to change the last FC layer to change the output features from 1000 to 9, which is the number of classes for my datasets, the following error is obtained.
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

Working version:
import torchvision.models as models
#model = Net()
model=models.resnet18(pretrained=True)

# defining the optimizer
optimizer = Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()
# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()

Version with error:
import torchvision.models as models
#model = Net()
model=models.resnet18(pretrained=True)

# defining the optimizer
optimizer = Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()
# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()
model.fc = torch.nn.Linear(512, 9)

Error occurs in the stage where training occurs, aka
outputs = model(images)

How should I go about fixing this issue?

Output:
Simple error, the fc layer should be instantiated before declaring model as cuda.
I.e
model=models.resnet18(pretrained=True)
model.fc = torch.nn.Linear(512, 9)
if torch.cuda.is_available():
    model = model.cuda()


Instruction: In PyTorch, what exactly does the grad_fn attribute store and how is it used?

Input:
In PyTorch, the Tensor class has a grad_fn attribute. This references the operation used to obtain the tensor: for instance, if a = b + 2, a.grad_fn will be AddBackward0. But what does "reference" mean exactly?
Inspecting AddBackward0 using inspect.getmro(type(a.grad_fn)) will state that the only base class of AddBackward0 is object. Additionally, the source code for this class (and in fact, any other class which might be encountered in grad_fn) is nowhere to be found in the source code!
All of this leads me to the following questions:

What precisely is stored in grad_fn and how is it called during back-propagation?
How come the objects that get stored in grad_fn do not have some sort of common super class, and why is there no source code for them on GitHub?

Output:
grad_fn is a function "handle", giving access to the applicable gradient function.  The gradient at the given point is a coefficient for adjusting weights during back-propagation.
"Handle" is a general term for an object descriptor, designed to give appropriate access to the object.  For instance, when you open a file, open returns a file handle.  When you instantiate a class, the __init__ function returns a handle to the created instance.  The handle contains references (usually memory addresses) to the data and functions for the item in question.
It appears as the generic object class because it's from the underlying implementation in another language, such that it does not map exactly to the Python function type.  PyTorch handles the inter-language call and return.  This hand-off is part of the pre-complied (shared-object) run-time system.
Is that enough to clarify what you see?


Instruction: What is a good use of the intermediate hidden states of an RNN?

Input:
So I've used RNN/LSTMs in three different capacities:

Many to many: Use every output of the final layer to predict the next. Could be classification or regression.
Many to one: Use the final hidden state to perform regression or classification.
One to many: Take a latent space vector, perhaps the final hidden state of an LSTM encoder and use it to generate a sequence (I've done this in the form of an autoencoder).

In none of these cases do I use the intermediate hidden states to generate my final output. Only the last layer outputs in case #1 and only the last layer hidden state in case #2 and #3. However, PyTorch nn.LSTM/RNN returns a vector containing the final hidden state of every layer, so I assume they have some uses.
I'm wondering what some use cases of those intermediate layer states are?

Output:
There’s nothing explicitly requiring you to use the last layer only. You could feed in all of the layers to your final classifier MLP for each position in the sequence (or at the end, if you’re classifying the whole sequence).
As a practical example, consider the ELMo architecture for generating contextualized (that is, token-level) word embeddings. (Paper here: https://www.aclweb.org/anthology/N18-1202/) The representations are the hidden states of a multi-layer biRNN.
Figure 2 in the paper shows how different layers differ in usefulness depending on the task. The authors suggest that lower levels encode syntax, while higher levels encode semantics.


Instruction: How to split json dataset and save it?

Input:
I have taken a json dataset. Dataset name is v2_OpenEnded_mscoco_train2014_questions.json How can I split some of the data from the dataset and save the split data into another json file?
This is the example of my dataset:

{"image_id": 426004, "question": "How many buns are on the plate?",
"question_id": 426004002}, {"image_id": 92846, "question": "What is
the color of the vase without flowers?", "question_id": 92846000},
{"image_id": 92846, "question": "Is there anything red in this
photo?", "question_id": 92846002}, {"image_id": 92846, "question":
"What does that vase represent?", "question_id": 92846003},
{"image_id": 262166, "question": "What color is the couch?",
"question_id": 262166002}, {"image_id": 262166, "question": "How many
seats are available?", "question_id": 262166003}

I have about 443,757 data in my dataset. I want to split the dataset into 400 different dataset each having 100 data. How can I automate this using python?
It will be a great help if this can be done with pytorch.

Output:
Try this:
n_rows = 100
current_data = []
for i, e in enumerate(data):
    if i % n_rows == 0 and i > 0:
        with open(f'dataset_{i - n_rows}-{i}.json', 'w') as f:
            json.dump(current_data, f)
        current_data = []

    current_data.append(e)

data is a list with jsons which you have. We iterate over it and every n_rows lines are written in a new file. The last several rows aren't written.


Instruction: How to get the indexes of equal elements in two different size PyTorch tensors?

Input:
Let's say I have two PyTorch tensors:
t_1d = torch.Tensor([6, 5, 1, 7, 8, 4, 7, 1, 0, 4, 11, 7, 4, 7, 4, 1])
t = torch.Tensor([4, 7])

I want to get the indices of exact match intersection between the sets for
the tensor t_1d with tensor t.
Desired output of t_1d and t: [5, 12] (first index of exact intersection)
Preferably on GPU for large Tensors, so no loops or Numpy casts.

Output:
In general, we can check where each element in t is equal to elements in t_1d.
After that, shift back the last element by as many places as it misses from the first element (in general case, here shift by -1) and check whether arrays are equal:
intersection = (t_1d == t[0]) & torch.roll(t_1d == t[1], shifts=-1)
torch.where(intersection)[0] # torch.tensor([5, 12])


Instruction: How do I pass an array of tensors into the criterion/loss function in PyTorch?

Input:
My loss function gives an error:
self.loss_fn = nn.MSELoss()

#### -- Snip ####

loss = self.loss_fn(predictions, targets) # Error here: 'list' object has no attribute 'size'
loss.backward()

My predictions are an array of tensors as follows:
predictions = []
for _ in range(100):
   prediction = MyNeuralNet(inputs)
   predictions.append(prediction)

How can I pass an array of tensors into my loss criterion function without getting the above error?

Output:
By using torch.stack I could fix my issue:
predictions = torch.stack(predictions)
loss = self.loss_fn(predictions, targets)


Instruction: What is torch.randn((1, 5))?

Input:
I'm confused as to why there are double parantheses instead of just torch.randn(1,5).
Is torch.randn(1,5) the same thing as torch.randn((1,5))?

Output:
You should check the definition of this function here.

size (int...) – a sequence of integers defining the shape of the output tensor. Can be a variable number of arguments or a collection like a list or tuple.

>>> import torch
>>> a = torch.randn(1,5)
>>> b = torch.randn((1,5))
>>> a.shape == b.shape
True

Therefore, you can use either a or b since they have the same shape.


Instruction: Target and output shape/type for binary classification using PyTorch

Input:
so I have some annotated images that I want to use to train a binary image classifier but I have been having issues creating the dataset and actually getting a test model to train. Each image is either of a certain class or not so I want to set up a binary classification dataset/model using PyTorch. I had some questions:

should labels be float or long?
what shape should my labels be?
I am using a resnet18 class from torchvision model, should my final softmax layer have one or two outputs?
what shapes should my target be, during training, if my batch size is 200?
what shape should my outputs be?

Thanks in advance
Quote
Delete

Output:
Binary classification is slightly different than multi-label classification: while for multilabel your model predicts a vector of "logits", per sample, and uses softmax to converts the logits to probabilities; In the binary case, the model predicts a scalar "logit", per sample, and uses the sigmoid function to convert it to class probability.
In pytorch the softmax and the sigmoind are "folded" into the loss layer (for numerical stability considerations) and therefore there are different Cross Entropy loss layers for the two cases: nn.BCEWithLogitsLoss for the binary case (with sigmoid) and nn.CrossEntropyLoss for the multilabel case (with softmax).
In your case you want to use the binary version (with sigmoid): nn.BCEWithLogitsLoss.
Thus your labels should be of type torch.float32 (same float type as the output of the network) and not integers.
You should have a single label per sample. Thus, if your batch size is 200, the target should have shape (200,1).

I'll leave it here as an exercise to show that training a model with two outputs and CE+softmax is equivalent to binary output+sigmoid ;)


Instruction: How to transform tensor size from [a, b] to [a, b, k] in PyTorch

Input:
I want to add one extra dimension to a tensor and set the value of this dimension as a specific value.
For examples,
print(a.size)
torch.Size([10, 5])

# after tranformation and set a to b
print(b.size)
torch.Size([10, 5, 1])

# after transformation and set a to c
print(c.size)
torch.Size([10, 2, 5])

Thansk in advances.

Output:
torch.stack allows you to concatenate two arrays along a new dimension
value = 1.37
a = torch.normal(0, 1, size=(5, 10))
c = torch.stack([a, torch.ones(a.shape) * value], dim=1)
c.shape

Out: torch.Size([5, 2, 10])

c[:, 0, :]

Out: tensor([[-0.2944, -0.7366,  0.6882, -0.7106,  0.0182, -0.1156, -1.0394, -0.7524,
          0.7587, -0.6066],
        [-1.0445, -2.7990,  0.0232,  0.5246, -0.7383,  0.0306, -1.0277, -0.8969,
          0.4026,  0.2006],
        [-1.2622, -0.6563, -1.9218, -0.6932, -1.9633,  1.8271,  0.6753, -0.7564,
          0.0107, -0.2312],
        [-0.8111, -1.0776, -0.8583,  0.2782, -0.8116,  0.0984,  0.4799,  0.6854,
          0.4408, -0.4280],
        [-1.1083,  1.8509,  0.1209,  0.5571, -1.1472,  0.2342,  0.3912,  0.7858,
          0.5879,  0.4139]])

c[:, 1, :]

Out: tensor([[1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700,
         1.3700],
        [1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700,
         1.3700],
        [1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700,
         1.3700],
        [1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700,
         1.3700],
        [1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700, 1.3700,
         1.3700]])


Instruction: How to import module only when I need to use it?

Input:
I have a project which has optional dependencies, depending on the use case it may need torch library.  PyTorch takes a lot of memory, it's not optimal to install it when you don't need to use it. In my project, there are parts of the code which use torch, but most of the logic does not need it. I'm trying to implement optional import so that when I need to use torch I use it, here is my idea:
def _lazy_import_torch():
    import torch

def main():
    _lazy_import_torch()
    print(f'You have torch: {torch.__version__}')

if __name__ == '__main__':
    main()

When I run this code it appears that torch was not imported:
$ python main.py
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    main()
  File "main.py", line 6, in main
    print(f'You have torch: {torch.__version__}')
NameError: name 'torch' is not defined

torch is installed in my environment.
I was wondering why it did not work and how can I make it works, please help me.

Output:
This code makes torch a local name of the _lazy_import_torch function:
def _lazy_import_torch():
    import torch

After it's executed, this name is gone, so you won't be able to access it anywhere outside this function.
You could return torch from the function and then pass it around everywhere:
def _lazy_import_torch():
    import torch
    return torch

def main():
    torch = _lazy_import_torch()
    print(f'You have torch: {torch.__version__}')

Or use a global variable:
torch = None

def _lazy_import_torch():
    global torch
    import torch

If the module hasn't been imported yet, you'll get an error every time you try to use it, but after you run _lazy_import_torch, the global name torch will refer to the torch module.

Here's an example with the CSV module from the standard library:
>>> csv = None
>>> def import_csv():
...   global csv
...   import csv
...
>>> csv
>>> import_csv()
>>> csv
<module 'csv' from '.../lib/python3.9/csv.py'>
>>>


Instruction: How GNN layers can know which graph in a batch has the given node & node feature?

Input:
When we pass input as node features (x) and edge index (edge_index) to pytorch_geometric layer (e.g. GATConv), I am worried whether the layer can differentiate which batch sample the given node elements belong to.
x follows the shape [num of nodes, feature size] and edge_index follows shape [2, num of edges]. However, these 2 do not have the given information to know which input graph of batch size 32 have given node feature in the x.
Anyone can clarify on this ?

Output: PyTorch-Geometric treats all the graphs in a batch as a single huge graph, with the individual graphs disconnected from each other. The node indices correspond to nodes in this big graph. This means there is no need for a batch dimension in x or edge_index.


Instruction: Convert my model predicted array to categorical using a condition

Input:
I've built a MLP neural network -> Trained -> Saved -> loaded -> now I'm testing my loaded model.
I've used y_pred=loaded_model(Variable(featuresTest)) to predict using pytorch then converted my tensor to array y_pred_arr and here's my result:
array([[-1.1663326 ,  0.369073  ],
       [-1.4255922 ,  0.23584184],
       [-2.045612  ,  1.4165274 ],
       ...,
       [ 4.327711  , -4.1331964 ],
       [-1.255816  ,  0.65834284],
       [ 6.642277  , -7.4430957 ]], dtype=float32)

My true labels array y_test which I'm trying to compare to are categorical
array([[0., 1.],
       [0., 1.],
       [0., 1.],
       ...,
       [1., 0.],
       [0., 1.],
       [1., 0.]], dtype=float32)

now I'm trying to use sklearn.metrics ->classification report however as my predicted array is obviously continuous I'm trying to convert it to a categorical which uses a condition converting the highest positive number in each of the array to a 1 for instance:
my first set in the predicted array is  [-1.4255922 ,  0.23584184], <- this would be converted to [ 0 , 1] as 0.2358 is the highest positive class in the prediction.
I've tried to use:
from keras.utils.np_utils import to_categorical
labels123 = to_categorical(y_pred_arr, dtype = "int64")

however, got this error:
IndexError: index -75 is out of bounds for axis 1 with size 74

Could I please get some assistance with this. Any help would be appreciated

Output:
Try this instead.
from keras.utils.np_utils import to_categorical
labels123 = to_categorical(np.argmax(y_pred_arr, 1), dtype = "int64")


Instruction: Append tensor to each element of another tensor

Input:
I have a pytorch tensor: x = torch.zeros(2, 2), and another tensor of variable values: item = torch.tensor([[1, 2], [3, 4]]), I am just giving this tensor for example.
I would like add the item tensor as each element of the x tensor, such that
x = [[item, item],
     [item, item]]

So x is a tensor with tensors inside it.
I have tried assigning item directly to x, but got an error: RuntimeError: The expanded size of the tensor must match the existing size at non-singleton dimension

Output:
use torch.repeat(),
your target_tensor shape will be torch.Size([2, 2, 2, 2]).
item tensor shape is already torch.Size([2, 2])
use :
target_tensor = item.repeat(2, 2, 1, 1)

the first two parameters of the repeat() function are x's shape


Instruction: quantization vector with numpy/pytorch

Input:
I have created quantized values as: {0.0, 0.5, 1.0} and would like to do quantization for a vector based on the set. For instance, given vector [0.1, 0.1, 0.9, 0.8, 0.6, 0.6] would be transferred into [0.0, 0.0, 1.0, 1.0, 0.5, 0.5].
Please guide me fastest way using python/numpy/pytorch. Thank you very much!

Output:
use map() and lambda()
you can get the round off nearest to 0.5 with round(x * 2) / 2
a = [0.1, 0.1, 0.9, 0.8, 0.6, 0.6]

list(map(lambda x: round(x * 2) / 2, a))

output :
[0.0, 0.0, 1.0, 1.0, 0.5, 0.5]


Instruction: ValueError: Expected x_max for bbox (0.65, 0.51, 1.12, 0.64, 3) to be in the range [0.0, 1.0], got 1.1234809015877545

Input:
I want to apply data augmentations from PyTorch's Albumentations to images with bounding boxes.
When I apply the HorizontalFlip Transformation, I receive this error ValueError: Expected x_max for bbox (0.6505353259854019, 0.517013871576637, 1.1234809015877545, 0.6447916687466204, 3) to be in the range [0.0, 1.0], got 1.1234809015877545.
I use the following code
A.Compose([
           A.HorizontalFlip(p=1),
           ToTensorV2(p=1.0)],
           p=1.0,
           bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])
                  )

When I apply the Cutout transformation, I do not have any error regarding the bounding boxes
A.Compose([
          A.Cutout(num_holes=10, max_h_size=32, max_w_size=32, fill_value=0, p=0.5),
          ToTensorV2(p=1.0)],
          p=1.0,
          bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])
                  )

Output: The bounding box exceeded the image since after the transformation it ends up being larger than the image size. Check carefully the dimensions of the bounding box.


Instruction: How to find training accuracy in pytorch

Input:
def train_and_test(e):
    epochs = e
    train_losses, test_losses, val_acc, train_acc= [], [], [], []
    valid_loss_min = np.Inf
    model.train()
    print("Model Training started.....")
    for epoch in range(epochs):
        running_loss = 0
        batch = 0
        for images, labels in trainloader:
            images, labels = images.to(device), labels.to(device)
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            batch += 1
            if batch % 10 == 0:
                print(f" epoch {epoch + 1} batch {batch} completed")
        test_loss = 0
        accuracy = 0
        with torch.no_grad():
            print(f"validation started for {epoch + 1}")
            model.eval()
            for images, labels in validloader:
                images, labels = images.to(device), labels.to(device)
                logps = model(images)
                test_loss += criterion(logps, labels)
                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor))

      train_losses.append(running_loss / len(trainloader))
      test_losses.append(test_loss / len(validloader))
      val_acc.append(accuracy / len(validloader))
      training_acc.append(running_loss / len(trainloader))
      scheduler.step()
      print("Epoch: {}/{}.. ".format(epoch + 1, epochs),"Training Loss: {:.3f}.. ".format(train_losses[-1]), "Valid Loss: {:.3f}.. ".format(test_losses[-1]),
        "Valid Accuracy: {:.3f}".format(accuracy / len(validloader)), "train Accuracy: {:.3f}".format(running_loss / len(trainloader)))
      model.train()
      if test_loss / len(validloader) <= valid_loss_min:
          print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min, test_loss / len(validloader)))
        torch.save({
            'epoch': epoch,
            'model': model,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': valid_loss_min
            }, path)
          valid_loss_min = test_loss / len(validloader)

    print('Training Completed Succesfully !')
    return train_losses, test_losses, val_acc ,train_acc

my output is
Model Training started.....
 epoch 1 batch 10 completed
 epoch 1 batch 20 completed
 epoch 1 batch 30 completed
 epoch 1 batch 40 completed
validation started for 1
Epoch: 1/2..  Training Loss: 0.088..  Valid Loss: 0.072..  Valid Accuracy: 0.979 train Accuracy: 0.088
Validation loss decreased (inf --> 0.072044).  Saving model ...

I am using dataset that is multi-set classification and getting training accuracy and training loss equal so I think there is error in training accuracy code.
training_acc.append(running_loss / len(trainloader))
"train Accuracy: {:.3f}".format(running_loss / len(trainloader))

training_acc.append(accuracy / len(trainloader))
"train Accuracy: {:.3f}".format(accuracy / len(trainloader))

is also not working fine

Output:
this method should be followed to plot training loses as well as accuracy
      for images , labels in trainloader:
         #start = time.time()
        images, labels = images.to(device), labels.to(device)
        optimizer.zero_grad()# Clear the gradients, do this because gradients are accumulated as 0 in each epoch

        # Forward pass  - compute outputs on input data using the model
        outputs = model(images) # modeling for each image batch
        loss = criterion(outputs,labels)  # calculating the loss

        # the backward pass
        loss.backward() # This is where the model learns by backpropagating
        optimizer.step() # And optimizes its weights here - Update the parameters
        running_loss += loss.item()

        # as Output of the network are log-probabilities, need to take exponential for probabilities
        ps = torch.exp(outputs)
        top_p , top_class = ps.topk(1,dim=1)
        equals = top_class == labels.view(*top_class.shape)

          # Convert correct_counts to float and then compute the mean
        acc += torch.mean(equals.type(torch.FloatTensor))


Instruction: Pytorch change format of tensors from BxWxH to B, N, 3

Input:
I have a tensor A with the shape BxWxH (B=Batch size, W=Width, H=Height) and want to change it to a tensor B of shape BxNx3 (B=Batch size, N=Number of points=W*H).
Tensor A represents a depth map, e.g. tensor[0,1,2] => gives the depth value for the pixel (1,2) in batch 0.
Tensor B also represents a depth map but in a different format. Each point in tensor B has the following three dimensions: (x coord, y coord, depth value).
How can I transform tensor A into tensor B?

Output:
You are looking for meshgrid to give you the x and y coordinates of each pixel:
b, w, h = A.shape
x, y = torch.meshgrid(torch.arange(w), torch.arange(h))
B = torch.cat((x[None, ...], y[None, ...], A), dim=0)
B = B.reshape(b, w*h, 3)


Instruction: Turn off console logging for Hydra when using Pytorch Lightning

Input: Is there a way to turn off console logging for Hydra, but keep file logging? I am encountering a problem where Hydra is duplicating all my console prints. These prints are handled by Pytorch Lightning and I want them to stay like that. However, I am fine with hydra logging them to a file (once per print), but I do not want to see my prints twice in the console.

Output:
I struggled a bit with the hydra documentation which is why I wanted to write a detailed explanation here so that other people can have it easy. In order to be able to use the answer proposed by @j_hu, i.e.:
hydra/job_logging=none
hydra/hydra_logging=none

with hydra 1.0 (which is the stable version at the time I am writing this answer) you need to first:

Create a directory called hydra within your config directory.
Create two subdirectories: job_logging and hydra_logging.
Create two none.yaml files in both of those directories as described below.

# @package _group_
version: 1
root: null
disable_existing_loggers: false

After this is done, you can use the none.yaml configuration to either override the logging via the command line:
python main.py hydra/job_logging=none hydra/hydra_logging=none
or via the config.yaml file:
defaults:
  - hydra/hydra_logging: none
  - hydra/job_logging: none


Instruction: How to use a Pytorch DataLoader for a dataset with multiple labels

Input: I'm wondering how to create a DataLoader that supports multiple types of labels in Pytorch. How do I do this?

Output:
You can return a dict of labels for each item in the dataset, and DataLoader is smart enough to collate them for you. i.e. if you provide a dict for each item, the DataLoader will return a dict, where the keys are the label types. Accessing a key of that label type returns a collated tensor of that label type.
See below:
import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np

class M(Dataset):
    def __init__(self):
        super().__init__()
        self.data = np.random.randn(20, 2)
        print(self.data)

    def __getitem__(self, i):
        return self.data[i], {'label_1':self.data[i], 'label_2':self.data[i]}

    def __len__(self):
        return len(self.data)

ds = M()
dl = DataLoader(ds, batch_size=6)

for x, y in dl:
    print(x, '\n', y)
    print(type(x), type(y))

[[-0.33029911  0.36632142]
 [-0.25303721 -0.11872778]
 [-0.35955625 -1.41633132]
 [ 1.28814629  0.38238357]
 [ 0.72908184 -0.09222787]
 [-0.01777293 -1.81824167]
 [-0.85346074 -1.0319562 ]
 [-0.4144832   0.12125039]
 [-1.29546792 -1.56314292]
 [ 1.22566887 -0.71523568]]
tensor([[-0.3303,  0.3663],
        [-0.2530, -0.1187],
        [-0.3596, -1.4163]], dtype=torch.float64)
 {'item_1': tensor([[-0.3303,  0.3663],
        [-0.2530, -0.1187],
        [-0.3596, -1.4163]], dtype=torch.float64), 'item_2': tensor([[-0.3303,  0.3663],
        [-0.2530, -0.1187],
        [-0.3596, -1.4163]], dtype=torch.float64)}
<class 'torch.Tensor'> <class 'dict'>
...


Instruction: How to use neural nets for classification with a large number of classes?

Input:
Are there techniques I could use to classify data (I'm using pytorch) using a large number of classes?
I noticed that if I try to build a multi layer perceptron network I run out of memory on the GPU given the last layer must have too many neurons, even though my GPU has 24Gb of memory. I have about 3000 classes.
Is there a way or technique do handle this type of scenario?
Note that I am NOT asking for an opinion on which technique is better. I am asking for an objective list of techniques that could be used in this scenario. This can be answered in fact-based fashion and include citations, etc if needed.

Output:
A tricky approach that you could try to follow is to divide the model into two sub nn.Modules. You send the first one to the GPU and keep the last layer(s), the classifier in the CPU. By doing so you will lose some training speed and overall performance, but you are going to be able to handle this huge layer in the MLP.
However it is not a common thing to have this large number of classes, are you doing some Computer Vision or NLP task? If so, you could use some task-specific network types such as CNNs or LSTMs, that perform better with a much more efficient number of parameters to deal with (e.g. using pooling layers in the CNNs). If you have to use a MLP try reducing the dimensionality of the penultimate layer.


Instruction: How to get value in order with a small average value

Input:
I've got a data frame like
x  y    cluster
0  112  4
0  113  4
1  111  4

and I'll get locations from this code:
for n in range(0,9):
...
    location = np.array(cluseter_location )

I want to sort it out in the order of column 'cluster' with small columns 'y' means, so I tried:
for n in range(0,9):
    cluster_ = data2[data2['cluster_id']== n]

    ...

Output:
Modifying your code to sort list of tuples
In your code, instead of appending the cluster_int, just append the tuple (n, cluster_int), and then later while sorting, use lambda to sort by the second value of each tuple.
for n in range(0,9):
    cluster_ = data2[data2['cluster_id']== n]

    cluster_list = cluster_['y'].tolist()
    cluster_avg = sum(cluster_list)/len(cluster_list)

    cluster_int = int(cluster_avg)

    print("cluster_id : %d" %n ,"average : %d" %cluster_int)
    lst.append((n,cluster_int))                              #<-------
    a = sorted(lst, key = lambda x:x[1])                     #<-------

print(a)                                                     #<-------

ordered_average = [average for cluster, average in a]        #<-------
ordered_clusters = [cluster for cluster, average in a]       #<-------

print(ordered_average)                                       #<-------
print(ordered_clusters)                                      #<-------

#cluster and average together
[(4,112),(8,121,(1,127),(6,139),(5,149)]

#averages sorted
[112, 121, 127, 139, 149]

#clusters sorted
[4,8,1,6,5]

Alternate way using pandas
A quicker way to do this would be directly sorting the pandas dataframe after groupby.
print(df.groupby('cluster')['y'].mean().reset_index().sort_values('y'))


Instruction: Extracting features of the hidden layer of an autoencoder using Pytorch

Input:
I am following this tutorial to train an autoencoder.
The training has gone well. Next, I am interested to extract features from the hidden layer (between the encoder and decoder).
How should I do that?

Output:
The cleanest and most straight-forward way would be to add methods for creating partial outputs -- this can be even be done a posteriori on a trained model.
from torch import Tensor

class AE(nn.Module):
    def __init__(self, **kwargs):
        ...

    def encode(self, features: Tensor) -> Tensor:
        h = torch.relu(self.encoder_hidden_layer(features))
        return torch.relu(self.encoder_output_layer(h))

    def decode(self, encoded: Tensor) -> Tensor:
        h = torch.relu(self.decoder_hidden_layer(encoded))
        return torch.relu(self.decoder_output_layer(h))

    def forward(self, features: Tensor) -> Tensor:
        encoded = self.encode(features)
        return self.decode(encoded)

You can now query the model for encoder hidden states by simply calling encode with the corresponding input tensor.
If you'd rather not add any methods to the base class (I don't see why), you could alternatively write an external function:
def get_encoder_state(model: AE, features: Tensor) -> Tensor:
   return torch.relu(model.encoder_output_layer(torch.relu(model.encoder_hidden_layer(features))))


Instruction: Why DataLoader return list that has a different length with batch_size

Input:
I am writting a customed dataloader, while the returned value makes me confused.
import torch
import torch.nn as nn
import numpy as np
import torch.utils.data as data_utils

class TestDataset:
    def __init__(self):
        self.db = np.random.randn(20, 3, 60, 60)

    def __getitem__(self, idx):
        img = self.db[idx]
        return img, img.shape[1:]

    def __len__(self):
        return self.db.shape[0]

if __name__ == '__main__':
    test_dataset = TestDataset()
    test_dataloader = data_utils.DataLoader(test_dataset,
                                       batch_size=1,
                                       num_workers=4,
                                       shuffle=False, \
                                       pin_memory=True
                                       )
    for i, (imgs, sizes) in enumerate(test_dataloader):
        print(imgs.size())  # torch.Size([1, 3, 60, 60])
        print(sizes)  # [tensor([60]), tensor([60])]
        break

Why "sizes" returns a list of length 2? I think it should be "torch.Size([1, 2])" which indicates height and width of a image(1 batch_size).
Further more, should the length of the returned list be the same to batch_size? If I want to get the size, I have to write "sizes = [sizes[0][0].item(), sizes[1][0].item()]". And this makes me very confused.
Thank you for your time.

Output:
It is caused by collate_fn function and its default behaviour. It's main purpose is to ease the batch preparation process. So you can customize your batch preparation process updating this function. As stated in documentation collate_fn, it automatically converts NumPy arrays and Python numerical values into PyTorch Tensors and it preserves the data structure. So it returns in your case [tensor([60]), tensor([60])]. In many cases, you return image with labels as tensors(instead of size of image) and feedforward to neural net. I don't know why you return the image size while enumerating, but you can get what you need adding a custom collate_fn as:
def collate_fn(data):
    imgs, lengths = data[0][0],data[0][1]
    return torch.tensor(imgs), torch.tensor([lengths])

Then you should set it to DataLoader's argument:
test_dataloader = DataLoader(test_dataset,
                                    batch_size=1,
                                    num_workers=4,
                                    shuffle=False, \
                                    pin_memory=True, collate_fn=collate_fn
                                    )

Then you can loop as:
for i, (imgs, sizes) in enumerate(test_dataloader):
    print(imgs.size())
    print(sizes)
    print(sizes.size())
    break

and output will be as:
torch.Size([3, 60, 60])
tensor([[60, 60]])
torch.Size([1, 2])

Afterall, I would like to add one more thing, you should not just return the self.db.shape[0] in len function. In this case your batch size is 1 and it's ok; however, when the batch size changes it will not return the true value for #batches. You can update your class as:
class TestDataset:
    def __init__(self, batch_size=1):
        self.db = np.random.randn(20, 3, 60, 60)
        self._batch_size = batch_size

    def __getitem__(self, idx):
        img = self.db[idx]
        return img, img.shape[1:]

    def __len__(self):
        return self.db.shape[0]/self._batch_size