Instruction: Can a PyTorch DataLoader start with an empty dataset?

Input:
I have a dataset which is in a deque buffer, and I want to load random batches from this with a DataLoader. The buffer starts empty. Data will be added to the buffer before the buffer is sampled from.
self.buffer = deque([], maxlen=capacity)
self.batch_size = batch_size
self.loader = DataLoader(self.buffer, batch_size=batch_size, shuffle=True, drop_last=True)

However, this causes the following error:
  File "env/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py", line 78, in __init__
    super().__init__(dataset, batch_size, shuffle,
  File "env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 268, in __init__
    sampler = RandomSampler(dataset, generator=generator)
  File "env/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 102, in __init__
    raise ValueError("num_samples should be a positive integer "
ValueError: num_samples should be a positive integer value, but got num_samples=0

Turns out that the RandomSampler class checks that num_samples is positive when it is initialised, which causes the error.
if not isinstance(self.num_samples, int) or self.num_samples <= 0:
    raise ValueError("num_samples should be a positive integer "
                     "value, but got num_samples={}".format(self.num_samples))

Why does it check for this here, even though RandomSampler does support datasets which change in size at runtime?
One workaround is to use an IterableDataset, but I want to use the shuffle functionality of DataLoader.
Can you think of a nice way to use a DataLoader with a deque? Much appreciated!

Output:
The problem here is neither the usage of deque nor the fact that the dataset is dynamically growable. The problem is that you are starting with a Dataset of size zero - which is invalid.
The easiest solution would be to just start with any arbitrary object in the deque and dynamically remove it afterwards.


Instruction: Training of multi-headed neural network with labels only for certain heads at a time

Input:
I am trying to train NN with 3 heads sharing some initial layers. However each of my training targets has only output for 2 of them.
I would like to create separate batches with samples that contains output only for the same heads and use them to update only respective heads.
Is there any way how to achieve this in any DL framework?

Output:
As your question is somewhat general, I will answer assuming you are using PyTorchLightning.
I suggest you use a model that looks like this:
class MyModel(LightningModule):
  def training_step(self, batch: MyMultiTaskBatch):
    backbone_output = self.backbone(batch.x)
    head = self.heads[batch.task_name]
    head_output = head(backbone_output)
    loss = self.losses[batch.task_name]
    return loss(head_output, batch.y)

Where your batch tells the model which head it should run, and which loss it should use out of dictionaries that map task names to heads and losses. You will also need to implement a dataloader that returns a MyMultiTaskBatch as its batches.


Instruction: Using Pytorch Dataloader with Probability Distribution

Input:
TL;DR: I want to use DataLoader to take a weighted random sample of the available rows. How do?

I've put together some python code that fits a certain kind of input-driven dynamical system to data using batched gradient descent over the parameters that define the model. I have the following snippet of Python code that gets the job done using Pytorch.
k_trn = self.linear.k_gen(in_trn,t)
u_trn = torch.tensor(in_trn.T)
x_trn = torch.tensor(out_trn.T, dtype = torch.float)
data = TensorDataset(u_trn[:-1,:],k_trn[:-1,:],x_trn[1:,:])
loader = DataLoader(data, batch_size = 20, shuffle = True)

Data types:

u_trn: N x 1 tensor (pytorch's array)
k_trn: N x K tensor
x_trn: N x n tensor

The rows of u_trn,k_trn,x_trn correspond to three trajectories (with u corresponding to the "input"). Each time I iterate over the loader (which can be done, e.g. with a loop for u,k,x in loader:), I get a batch of 20 rows from u_trn, 20 rows of k_trn, and 20 rows of x_trn. These rows are selected with a uniform probability, without replacement.
The catch is that I would like to sample these rows with a non-uniform probability. In particular: denote S = (1/1 + 1/2 + ... + 1/N). I would like for the loader to select the jth row with probability 1/(S*j).
After looking at the relevant doumentation, I suspect that this can be done by messing with either the sampler or batch_sampler keyword arguments when initializing the DataLoader object, but I'm having trouble parsing the documentation well enough to implement the behavior that I'm looking for.
I'd appreciate any help with this. I've tried to keep my question brief; please let me know if I've left out any relevant information.

Followup: with the help of Shai's answer, I've gotten things to work properly. Here's a quick script that I used to test this out and make sure that everything was working as expected.
import numpy as np
import torch
from torch.utils.data import DataLoader, TensorDataset, WeightedRandomSampler
import matplotlib.pyplot as plt
import numpy as np
import torch
from torch.utils.data import DataLoader, TensorDataset, WeightedRandomSampler
import matplotlib.pyplot as plt

N = 100
x = np.zeros((N,2))
x[:,0] = 1 + np.arange(N)
data = TensorDataset(torch.Tensor(x))

weights = [1/j for j in range(1, N+1)]  # my weights
sampler = WeightedRandomSampler(weights, 10000, replacement=True)
loader = DataLoader(data, batch_size=20, sampler=sampler)
sums = []

for y, in loader:
    for k in range(len(y)):
        sums.append(np.sum(y[k].numpy()))

h = plt.hist(sums, bins = N)
a = h[0][0]
plt.plot([a/(n+1) for n in range(N)], lw = 3)

And the resulting plot:

Note that weights are automatically normalized, so there is no need to divide by the sum S. Note also that there is no need for shuffle=True in the loader; the sampler takes care of the randomization on its own.

Output:
Why don't you simply use WeightedRandomSampler?
weights = [1./(S*j) for j in range(1, N+1)]  # your weights
sampler = WeightedRandomSampler(weights, replacement=True)
loader = DataLoader(data, batch_size=20, sampler=sampler)


Instruction: How to register a forward hook for PyTorch matmul?

Input:
torch.matmul doesn't seem to have an nn.Module wrapper to allow the standard forward hook registration by name. In this case, the matrix multiply happens in the middle of a forward() function. I suppose the intermediate result can be returned by forward() in addition to the final result, such as return x, mm_res. But what's a good way to collect these additional outputs?
What are the options for offloading torch.matmul outputs? TIA.

Output:
If your primary complaint is the fact that torch.matmul doesn't have a Module wrapper, how about just making one
class Matmul(nn.Module):
    def forward(self, *args):
        return torch.matmul(*args)

Now you can register a forward hook on a Matmul instance
class Network(nn.Module):
    def __init__(self, ...):
        self.matmul = Matmul()
        self.matmul.register_module_forward_hook(...)
    def forward(self, x):
        y = ...
        z = self.matmul(x, y)
        ...

Being said that, you must not overlook the warning (in red) in the doc that it should only be used for debugging purpose.


Instruction: only one element tensors can be converted to Python scalars error with pca.fit_transform

Input:
I am trying to perform dimensionality reduction using PCA, where outputs is a list of tensors where each tensor has a shape of (1, 3, 32,32). Here is the code:
from sklearn.decomposition import PCA
pca = PCA(10)
pca_result = pca.fit_transform(output)

But I keep getting this error, regardless of whatever I tried:
ValueError: only one element tensors can be converted to Python scalars

I know that the tensors with size(1,3, 32,32) is making the issue, since its looking for 1 element as the error puts it, but do not know how to solve it.
I have tried flattening each tensor with looping over output (don't know if its the right way of solving this issue), using the following code but it leads to error in pca:
new_outputs = []
for i in outputs:
            for j in i:
                j = j.cpu()
                j = j.detach().numpy()
                j = j.flatten()
                new_outputs.append(j)

pca_result = pca.fit_transform(new_output)

I would appreciate if anybody can help with this error whether the flattening approach I took, is correct.
PS:I have read the existing posts (post1,post2) discussing this error but none of them could solve my problem.

Output:
Assuming your Tensors are stored in a matrix with shape like (10, 3, 32, 32) where 10 corresponds to number of Tensors, you should flatten each like that:
import torch
from sklearn.decomposition import PCA
data= torch.rand((10, 3, 32, 32))
pca = PCA(10)
pca_result = pca.fit_transform(data.flatten(start_dim=1))

data.flatten(start_dim=1) makes your data to be in shape (10, 3*32*32)
The error you posted is actually related to one of the post you linked. The PCA estimator expects array-like object with fit() method and you provided a list of Tensors.


Instruction: How to extract patches from an image in pytorch/tensorflow into 4 equal parts?

Input:
I am using a 16x16 colour image; I wrote small code for that but could not perform it precisely.
import numpy as np
from patchify import patchify
image = cv2.imread('subbu_i.jpg')
print(image.shape)
patches = patchify(image, (4,4), step=1)
print(patches.shape)

Any ideas?

Output:
For Tensorflow, try tf.image.extract_patches
import tensorflow as tf
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np

image = Image.open('/content/image.png')
plt.imshow(image)

image = tf.expand_dims(np.array(image), 0)
image = tf.expand_dims(np.array(image), -1)
patches = tf.image.extract_patches(images=image,
                        sizes=[1, 4, 4, 1],
                        strides=[1, 4, 4, 1],
                        rates=[1, 1, 1, 1],
                        padding='VALID')

axes=[]
fig=plt.figure()

for i in range(4):
    axes.append( fig.add_subplot(2, 2, i + 1) )
    subplot_title=("Patch "+str(i + 1))
    axes[-1].set_title(subplot_title)
    patch = tf.reshape(patches[0, i, i], (4, 4))
    plt.imshow(patch)
fig.tight_layout()
plt.show()


Instruction: How to define pytorch fullyconnect model more simple and convenient?

Input:
i am a beginner of pytorch, and i want to build a fully connect model using Pytorch;
the model is very simple like:
def forward(self, x):
    x = self.relu(self.fc1(x))
    x = self.relu(self.fc2(x))
    return self.fc3(x)

but when i want to add some layers or adjust the hidden layers, i found i have to write lots of Redundant code like:
def forward(self, x):
    x = self.relu(self.fc1(x))
    x = self.relu(self.fc2(x))
    x = self.relu(self.fc3(x))
    x = self.relu(self.fc4(x))
    x = self.relu(self.fc5(x))
    ...
    return self.fcn(x)

besides, if i want to change some layer's feature nums, i have to change the layer adjacent
so i want to know a way which is more grace(maybe more pythonic and more easy to adjust hyper parameter).
i tried to write code like:
def __init__(self):
        super().__init__()
        self.hidden_num = [2881, 5500, 2048, 20] # i just want to change here! to try some new structure
        self.fc = [nn.Linear(self.hidden_num[i], self.hidden_num[i + 1]).to(DEVICE)  for i in range(len(self.hidden_num) - 1)]
        self.relu = nn.ReLU()

    def forward(self, x):
        for i in range(len(self.fc)):
            x = self.fc[i](x)
            if i != (len(self.fc) - 1):
                x = self.relu(x)
        return x

but i found this way doesn't work, the model can't be built
so could any bro tell me, how to define a fullyconnect model like above??
(so i can adjust the model layers only by adjust the list named hidden_num )

Output:
If you want to keep the same approach then you can use nn.ModuleList to properly register all linear layers inside the module's __init__:
class Model(nn.Module):
    def __init__(self, hidden_num=[2881, 5500, 2048, 20]):
        super().__init__()
        self.fc = nn.ModuleList([
            nn.Linear(hidden_num[i], hidden_num[i+1])
               for i in range(len(hidden_num) - 1)])

    def forward(self, x):
        for i, m in enumerate(self.fc.children()):
            x = m(x)
            print(i)
            if i != len(self.fc) - 1:
                x = torch.relu(x)
        return x

However, you may want to handle the logic inside the __init__ function once. One alternative is to use nn.Sequential.
class Model(nn.Module):
    def __init__(self, hidden_num=[2881, 5500, 2048, 20]):
        super().__init__()

        fc = []
        for i in range(len(hidden_num) - 1):
            fc.append(nn.Linear(hidden_num[i], hidden_num[i+1]))
            if i != len(self.fc) - 1:
                fc.append(nn.ReLU())

        self.fc = nn.Sequential(fc)

    def forward(self, x):
        x = self.fc(x)
        return x

Ideally, you would inherit from nn.Sequential directly to avoid re-writing the forward function which is unnecessary in this case:
class Model(nn.Sequential):
    def __init__(self, hidden_num=[2881, 5500, 2048, 20]):
        fc = []
        for i in range(len(hidden_num) - 1):
            fc.append(nn.Linear(hidden_num[i], hidden_num[i+1]))
            if i != len(self.fc) - 1:
                fc.append(nn.ReLU())

        super().__init__(fc)


Instruction: DQN predicts same action value for every state (cart pole)

Input:
I'm trying to implement a DQN. As a warm up I want to solve CartPole-v0 with a MLP consisting of two hidden layers along with input and output layers. The input is a 4 element array [cart position, cart velocity, pole angle, pole angular velocity] and output is an action value for each action (left or right).  I am not exactly implementing a DQN from the "Playing Atari with DRL" paper (no frame stacking for inputs etc).  I also made a few non standard choices like putting done and the target network prediction of action value in the experience replay, but those choices shouldn't affect learning.
In any case I'm having a lot of trouble getting the thing to work. No matter how long I train the agent it keeps predicting a higher value for one action over another, for example Q(s, Right)> Q(s, Left) for all states s. Below is my learning code, my network definition, and some results I get from training
class DQN:
    def __init__(self, env, steps_per_episode=200):
        self.env = env
        self.agent_network = MlpPolicy(self.env)
        self.target_network = MlpPolicy(self.env)
        self.target_network.load_state_dict(self.agent_network.state_dict())
        self.target_network.eval()
        self.optimizer = torch.optim.RMSprop(
            self.agent_network.parameters(), lr=0.005, momentum=0.95
        )
        self.replay_memory = ReplayMemory()
        self.gamma = 0.99
        self.steps_per_episode = steps_per_episode
        self.random_policy_stop = 1000
        self.start_learning_time = 1000
        self.batch_size = 32

    def learn(self, episodes):
        time = 0
        for episode in tqdm(range(episodes)):
            state = self.env.reset()
            for step in range(self.steps_per_episode):
                if time < self.random_policy_stop:
                    action = self.env.action_space.sample()
                else:
                    action = select_action(self.env, time, state, self.agent_network)
                new_state, reward, done, _ = self.env.step(action)
                target_value_pred = predict_target_value(
                    new_state, reward, done, self.target_network, self.gamma
                )
                experience = Experience(
                    state, action, reward, new_state, done, target_value_pred
                )
                self.replay_memory.append(experience)
                if time > self.start_learning_time:  # learning step
                    experience_batch = self.replay_memory.sample(self.batch_size)
                    target_preds = extract_value_predictions(experience_batch)
                    agent_preds = agent_batch_preds(
                        experience_batch, self.agent_network
                    )
                    loss = torch.square(agent_preds - target_preds).sum()
                    self.optimizer.zero_grad()
                    loss.backward()
                    self.optimizer.step()
                if time % 1_000 == 0:  # how frequently to update target net
                    self.target_network.load_state_dict(self.agent_network.state_dict())
                    self.target_network.eval()

                state = new_state
                time += 1

                if done:
                    break

def agent_batch_preds(experience_batch: list, agent_network: MlpPolicy):
    """
    Calculate the agent action value estimates using the old states and the
    actual actions that the agent took at that step.
    """
    old_states = extract_old_states(experience_batch)
    actions = extract_actions(experience_batch)
    agent_preds = agent_network(old_states)
    experienced_action_values = agent_preds.index_select(1, actions).diag()
    return experienced_action_values

def extract_actions(experience_batch: list) -> list:
    """
    Extract the list of actions from experience replay batch and torchify
    """
    actions = [exp.action for exp in experience_batch]
    actions = torch.tensor(actions)
    return actions

class MlpPolicy(nn.Module):
    """
    This class implements the MLP which will be used as the Q network. I only
    intend to solve classic control problems with this.
    """

    def __init__(self, env):
        super(MlpPolicy, self).__init__()
        self.env = env
        self.input_dim = self.env.observation_space.shape[0]
        self.output_dim = self.env.action_space.n
        self.fc1 = nn.Linear(self.input_dim, 32)
        self.fc2 = nn.Linear(32, 128)
        self.fc3 = nn.Linear(128, 32)
        self.fc4 = nn.Linear(32, self.output_dim)

    def forward(self, x):
        if type(x) != torch.Tensor:
            x = torch.tensor(x).float()
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x

Learning results:

Here I'm seeing one action always valued over the others (Q(right, s) > Q(left, s)). It's also clear that the network is predicting the same action values for every state.
Does anyone have an idea about what's going on? I've done a lot of debugging and careful reading of the original papers (also thought about "normalizing" the observation space even though the velocities can be infinite) and could be missing something obvious at this point. I can include more code for the helper functions if that would be useful.

Output:
There was nothing wrong with the network definition. It turns out the learning rate was too high and reducing it 0.00025 (as in the original Nature paper introducing the DQN) led to an agent which can solve CartPole-v0.
That said, the learning algorithm was incorrect. In particular I was using the wrong target action-value predictions. Note the algorithm laid out above does not use the most recent version of the target network to make predictions. This leads to poor results as training progresses because the agent is learning based on stale target data. The way to fix this is to just put (s, a, r, s', done) into the replay memory and then make target predictions using the most up to date version of the target network when sampling a mini batch. See the code below for an updated learning loop.
def learn(self, episodes):
        time = 0
        for episode in tqdm(range(episodes)):
            state = self.env.reset()
            for step in range(self.steps_per_episode):
                if time < self.random_policy_stop:
                    action = self.env.action_space.sample()
                else:
                    action = select_action(self.env, time, state, self.agent_network)
                new_state, reward, done, _ = self.env.step(action)
                experience = Experience(state, action, reward, new_state, done)
                self.replay_memory.append(experience)
                if time > self.start_learning_time:  # learning step.
                    experience_batch = self.replay_memory.sample(self.batch_size)
                    target_preds = target_batch_preds(
                        experience_batch, self.target_network, self.gamma
                    )
                    agent_preds = agent_batch_preds(
                        experience_batch, self.agent_network
                    )
                    loss = torch.square(agent_preds - target_preds).sum()
                    self.optimizer.zero_grad()
                    loss.backward()
                    self.optimizer.step()
                if time % 1_000 == 0:  # how frequently to update target net
                    self.target_network.load_state_dict(self.agent_network.state_dict())
                    self.target_network.eval()

                state = new_state
                time += 1
                if done:
                    break


Instruction: Modify Python Class to filter files

Input:
New to python. How can I  modify the class to filter files in the folder with a string. Right now it returns all files in folder_containing_the_content_folder which could be millions of items. The following works however I would like to isolate files that contain a specific string, for example, isolate all files that contain 'v_1234_frame':
# Image loader
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x.mul(255))
])
image_dataset = utils.ImageFolderWithPaths(folder_containing_the_content_folder, transform=transform)
image_loader = torch.utils.data.DataLoader(image_dataset, batch_size=batch_size)

The class that works requires a modification to filter file names that contain 'v_1234_frame':
class ImageFolderWithPaths(datasets.ImageFolder):
"""Custom dataset that includes image file paths.
Extends torchvision.datasets.ImageFolder()
Reference: https://discuss.pytorch.org/t/dataloader-filenames-in-each-batch/4212/2
"""

# override the __getitem__ method. this is the method dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)

    # the image file path
    path = self.imgs[index][0]

    # make a new tuple that includes original and the path
    tuple_with_path = (*original_tuple, path)
    return tuple_with_path

I am learning python and just can't seem to come up with the solution. Hope you can help/suggest a change to the class or calling method.

Output:
Built my own data loader to isolate files via a wildcard pattern in glob and then loop through those to create a tensor for each image, passing that to my model which required it to be converted to a float. Extract the base name (image name) from the path (ex. img_frame1.jpg). Save result to my style folder.  This method gives me total control over the files via the wildcard.  I have included the other functions used in the solution. Note: I am not using a gpu to process these so I can run it on a standard python web server.  Hopefully this helps someone in the future. Simple is sometimes better :)
# Load image file
# def load_image(path):
#     # Images loaded as BGR
#     img = cv2.imread(path)
#     return img

# def itot(img, max_size=None):
#     # Rescale the image
#     if (max_size == None):
#         itot_t = transforms.Compose([
#             # transforms.ToPILImage(),
#             transforms.ToTensor(),
#             transforms.Lambda(lambda x: x.mul(255))
#         ])
#     else:
#         H, W, C = img.shape
#         image_size = tuple([int((float(max_size) / max([H, W])) * x) for x in [H, W]])
#         itot_t = transforms.Compose([
#             transforms.ToPILImage(),
#             transforms.Resize(image_size),
#             transforms.ToTensor(),
#             transforms.Lambda(lambda x: x.mul(255))
#         ])
#
#     # Convert image to tensor
#     tensor = itot_t(img)
#
#     # Add the batch_size dimension
#     tensor = tensor.unsqueeze(dim=0)
#     return tensor

folder_data = glob.glob(folder_containing_the_content_folder + "content_folder/" +  video_token + "_frame*.jpg")

# image_dataset = utils.ImageFolderWithPaths(folder_containing_the_content_folder, transform)
# image_loader = torch.utils.data.DataLoader(image_dataset, batch_size=batch_size)

# Load Transformer Network
net = transformer.TransformerNetwork()
net.load_state_dict(torch.load(style_path))
net = net.to(device)

with torch.no_grad():
    for image_name in folder_data:
        img = utils.load_image(image_name)
        img = img / 255.0  # standardize the data/transform
        img_tensor = utils.itot(img)

        # style image tensor
        generated_tensor = net(img_tensor.float())

        # convert image the model modified tensor back to an image
        generated_image = utils.ttoi(generated_tensor)
        image_name = os.path.basename(image_name)

        # save generated image to folder
        utils.saveimg(generated_image, save_folder + image_name)


Instruction: Pytorch RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x246016 and 3136x1000)

Input:
I'm building a CNN on Pytorch.
I'm a little confused about the inputs.
I'm getting the following error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x246016 and 3136x1000)
Images are 250 x 250 in grayscale.
Can anyone take a look at my constructor and tell me where I'm going wrong?
More points if you can explain to me why I was wrong and the why your answer is right! ;)
class CNN(nn.Module):

# Contructor
def __init__(self):
    super(CNN, self).__init__()
    self.cnn1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2)
    self.conv1_bn = nn.BatchNorm2d(32)

    self.maxpool1=nn.MaxPool2d(kernel_size=2, stride=2)

    self.cnn2 = nn.Conv2d(32, 64, kernel_size=5,stride=1, padding=2)
    self.conv2_bn = nn.BatchNorm2d(64)
    self.maxpool2=nn.MaxPool2d(kernel_size=2, stride=2)

    self.drop_out1 = nn.Dropout()
    self.fc1 = nn.Linear(7 * 7 * 64, 1000)
    self.bn_fc1 = nn.BatchNorm2d(1000)

    self.fc2 = nn.Linear(1000, 1)

# Prediction
def forward(self, x):
    x = self.cnn1(x)
    x = self.conv1_bn(x)
    x = torch.relu(x)
    x = self.maxpool1(x)
    x = self.cnn2(x)
    x = self.conv2_bn(x)
    x = torch.relu(x)
    x = self.maxpool2(x)
    x = x.view(x.size(0), -1)
    x = self.drop_out1(x)
    x = self.fc1(x)
    x = self.bn_fc1(x)
    x = torch.relu(x)
    x = self.fc2(x)
    x = torch.sigmoid(x)

    return x

Output:
Your fc1 layer is expecting a tensor of shape (-1, 7*7*64) but you are passing it a tensor of shape [-1, 246016] (-1 being the batch size).
To calculate output size of conv nets see this post or any neural networks textbook.


Instruction: Why i can't import torch?

Input:
I downloaded and installed CUDA10.0 pyhton3.6 pytorch1.2. , also loaded cuda/cuda-10.0-x86_6 envirnment.
Commands like below
module load cuda/cuda-10.0-x86_64

# CUDA 10.0
conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch

Because it needs to be installed on a NO-NETWOEK centOs server! Therefore, all pakages need be downloaded in advance and run command like below.
conda install --offline pytorch-1.2.0-cuda100py36h938c94c_0.tar.bz2
......

The error occured below.
Python 3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/users/CHDHPC/2020224009/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: libmkl_gnu_thread.so: cannot open shared object file: No such file or directory

I have no idea to solve it! Anyone can help me! Thank you!!

Output:
Finally I fixed this issue by changing Python version to 3.6.2 and re-installing the cuda and cudnn.
Pecfect!


Instruction: Finding several means in a large tensor

Input:
I have a large tensor (~10k). Here's a sampler with 200 values:
sample_tensor = tensor([ 0.6676,  0.0917,  0.6083,  0.4536,  1.1882,  0.6672,  0.6058, -0.1615,
         0.5254,  1.1642,  0.1994, -0.2274,  0.0511,  0.3707,  0.3675, -0.1629,
        -0.0638, -0.0118,  0.2668,  0.8586,  0.7027,  0.3018, -0.2930,  1.2613,
         0.9374,  0.3154,  1.0396, -0.0263,  0.2012,  1.5710, -0.4640, -0.1657,
        -0.2670,  0.5783,  0.7420,  0.1886, -1.1255,  0.3682,  0.2597,  0.3697,
         0.1404, -0.0289,  0.5903,  0.0461,  0.2288, -0.0414,  0.9736,  0.4891,
        -0.0593,  0.1694,  0.2426, -0.0339,  0.1683,  0.2374,  0.1349,  0.1672,
         0.4174,  0.8038,  1.4121, -0.1046,  0.1169,  0.6447, -0.1168,  0.7392,
         0.0578, -0.1398,  0.8974,  1.0977,  0.7102,  1.4012,  0.8541,  0.3314,
        -0.2045,  0.1540,  0.2779, -0.3912,  0.4068, -0.1868,  0.1796,  0.0318,
         0.1354, -0.9689,  0.3460,  0.3762,  0.8637, -0.4735,  0.8413,  0.5261,
         0.8362, -0.2226, -0.2772, -0.2757,  0.2079,  0.0895,  0.4352,  0.8868,
         0.3707,  0.8412,  0.3026,  0.1568,  0.4442,  0.0789,  0.5050,  0.0102,
         0.6944,  0.1852,  0.5215, -0.7028, -0.7591,  0.2139,  0.7411,  0.3830,
         0.8048, -0.7532,  0.7710,  0.8526,  1.1322,  0.0939, -0.3318,  1.1003,
         0.3066,  1.6501,  1.1300,  0.0062,  0.2600,  0.2605, -0.2236,  0.2516,
         0.4460,  0.6813,  0.1876, -0.4710, -0.5939,  0.4144,  0.0783,  0.4282,
         0.1744,  0.0569,  0.1043,  0.3329,  0.3561,  0.1618, -0.1184,  0.4183,
         0.5722, -0.4459,  0.3354,  0.3373,  0.2290,  1.0164, -0.5191,  0.0992,
         0.9188, -0.3634,  1.2128,  0.0457,  0.1028, -0.2206,  0.9355,  0.6074,
         0.3834,  0.0802,  0.7016,  0.8777,  0.2769, -0.7512,  0.8667, -0.1056,
         0.5435,  1.4568, -0.3943,  0.5740,  0.6328,  0.4063, -0.7712,  0.5113,
         0.1578,  0.4571,  1.0314,  0.2863, -0.1470,  1.0763, -0.0019,  0.9103,
         1.0114, -0.1229, -0.3118,  0.5383,  0.5566,  0.2280,  0.9320,  0.6770,
         0.0908,  0.5056,  0.0445, -0.0810,  0.2611,  0.1223, -0.0108,  0.0611])

I also have an input value that corresponds to how many means I need to get from this tensor:
sampler_number_of_means = 10

What is an efficient way to get a tensor of 10 means from this tensor, where each mean is a different  set of values with size of len(sample_tensor)/sampler_number_of_means. That is, in this example the first mean will be the first 20 values, the second mean the next 20 values, etc.
I'm currently iterating through the tensor and breaking it into equal size lists, then iterating through each list to get the mean. But it's quite slow with large tensors.

Output:
You can reshape the tensor, then take the mean.
import torch
sample_tensor = torch.tensor([ 0.6676,  0.0917,  0.6083,  0.4536,  1.1882,  0.6672,  0.6058, -0.1615,
         0.5254,  1.1642,  0.1994, -0.2274,  0.0511,  0.3707,  0.3675, -0.1629,
        -0.0638, -0.0118,  0.2668,  0.8586,  0.7027,  0.3018, -0.2930,  1.2613,
         0.9374,  0.3154,  1.0396, -0.0263,  0.2012,  1.5710, -0.4640, -0.1657,
        -0.2670,  0.5783,  0.7420,  0.1886, -1.1255,  0.3682,  0.2597,  0.3697,
         0.1404, -0.0289,  0.5903,  0.0461,  0.2288, -0.0414,  0.9736,  0.4891,
        -0.0593,  0.1694,  0.2426, -0.0339,  0.1683,  0.2374,  0.1349,  0.1672,
         0.4174,  0.8038,  1.4121, -0.1046,  0.1169,  0.6447, -0.1168,  0.7392,
         0.0578, -0.1398,  0.8974,  1.0977,  0.7102,  1.4012,  0.8541,  0.3314,
        -0.2045,  0.1540,  0.2779, -0.3912,  0.4068, -0.1868,  0.1796,  0.0318,
         0.1354, -0.9689,  0.3460,  0.3762,  0.8637, -0.4735,  0.8413,  0.5261,
         0.8362, -0.2226, -0.2772, -0.2757,  0.2079,  0.0895,  0.4352,  0.8868,
         0.3707,  0.8412,  0.3026,  0.1568,  0.4442,  0.0789,  0.5050,  0.0102,
         0.6944,  0.1852,  0.5215, -0.7028, -0.7591,  0.2139,  0.7411,  0.3830,
         0.8048, -0.7532,  0.7710,  0.8526,  1.1322,  0.0939, -0.3318,  1.1003,
         0.3066,  1.6501,  1.1300,  0.0062,  0.2600,  0.2605, -0.2236,  0.2516,
         0.4460,  0.6813,  0.1876, -0.4710, -0.5939,  0.4144,  0.0783,  0.4282,
         0.1744,  0.0569,  0.1043,  0.3329,  0.3561,  0.1618, -0.1184,  0.4183,
         0.5722, -0.4459,  0.3354,  0.3373,  0.2290,  1.0164, -0.5191,  0.0992,
         0.9188, -0.3634,  1.2128,  0.0457,  0.1028, -0.2206,  0.9355,  0.6074,
         0.3834,  0.0802,  0.7016,  0.8777,  0.2769, -0.7512,  0.8667, -0.1056,
         0.5435,  1.4568, -0.3943,  0.5740,  0.6328,  0.4063, -0.7712,  0.5113,
         0.1578,  0.4571,  1.0314,  0.2863, -0.1470,  1.0763, -0.0019,  0.9103,
         1.0114, -0.1229, -0.3118,  0.5383,  0.5566,  0.2280,  0.9320,  0.6770,
         0.0908,  0.5056,  0.0445, -0.0810,  0.2611,  0.1223, -0.0108,  0.0611])

sampler_number_of_means = 10
sample_tensor.reshape((sampler_number_of_means,int(sample_tensor.shape[0]/sampler_number_of_means))).mean(1)

Output
tensor([0.3729, 0.3248, 0.2977, 0.3431, 0.2499, 0.2993, 0.2740, 0.2841, 0.3611,
        0.3170])


Instruction: Pytorch: set top 10% values of the tensor to zero

Input:
I have Pytorch 2d tensor with normal distribution.
Is there a fast way to nullify top 10% max values of this tensor using Python?
I see two possible ways here:

Flatten tensor to 1d and just sort it
Non-vectorized way using some native Python operators (for-if)

But neither of these looks fast enough.
So, what is the fastest way to set X max values of a tensor to zero?

Output:
Well, it seems that Pytorch has a useful operator torch.quantile() that helps here a lot.
The solution (for 1d tensor):
import torch

x = torch.randn(100)
y = torch.tensor(0.) #new value to assign
split_val = torch.quantile(x, 0.9)
x = torch.where(x < split_val, x, y)


Instruction: How to convert a matplotlib spectrogram image into a torch tensor

Input:
import numpy as np
from numpy import asarray
from matplotlib import pyplot as plt
import torch

# generate a signal
fs = 50 # sampling freq
ts = np.arange(0, 10, 1/fs) # times at which signal is sampled
s1 = np.sin(2 * np.pi * 2 * ts) # 2 hz
s2 = np.sin(2 * np.pi * 3 * ts) # 3 hz
s3 = np.sin(2 * np.pi * 6 * ts) # 6 hz
s = s1 + s2 + s3 # aggregate signal

# generate specgram
spectrum, freqs, t, im = plt.specgram(s, Fs=fs, xextent=((0, len(s)/fs)))

# convert matplotlib image to torch tensor
# bypassing the numpy part would be even better!
torch_tensor = torch.from_numpy(asarray(im, np.float32))

print(torch_tensor)

>>> TypeError: float() argument must be a string or a number, not 'AxesImage'

I should add that the 'spectrum' variable is kind of what I am looking for, except that I am a little confused by it since it has only two columns for time, and I think the specgram image has many more than two timesteps. If there is a way to use the spectrum variable to represent the whole image as a torch tensor, then that would also work for me.

Output:
plt.specgram returns the spectrogram in the spectrum variable. This means that you need to pass that variable to the torch.from_numpy function. Additionally, according to this, specgram shows the 10*log10(spectrum) which means that you might want to do that operation ot compare the results shown by specgram with the plot of your tensor. See code below:
import numpy as np
from numpy import asarray
import numpy as np
from matplotlib import pyplot as plt
import torch

# generate a signal
fs = 50 # sampling freq
ts = np.arange(0, 10, 1/fs) # times at which signal is sampled
s1 = np.sin(2 * np.pi * 2 * ts) # 2 hz
s2 = np.sin(2 * np.pi * 3 * ts) # 3 hz
s3 = np.sin(2 * np.pi * 6 * ts) # 6 hz
s = s1 + s2 + s3 # aggregate signal

# generate specgram
ax1=plt.subplot(121)
ax1.set_title('Specgram image')
spectrum, freqs, t, im = ax1.specgram(s, Fs=fs, xextent=((0, len(s)/fs)))
ax1.axis('tight')

torch_tensor = torch.from_numpy(spectrum)

#Plot torch tensor variable
ax2=plt.subplot(122)
ax2.set_title('Torch tensor image')
ax2.imshow(10*np.log10(torch_tensor),origin='lower left',extent=[0,10,0,25])
ax2.axis('tight')

plt.show()

And the output gives:


Instruction: Distance to object python

Input:
I have an image of a car and a corresponding bounding box. For example:
(xmin, ymin, xmax, ymax)
(504.8863220214844, 410.2454833984375,
937.6451416015625, 723.9139404296875)

That's how I draw boxes:
def plot_results(pil_img, prob, boxes):
    plt.figure(figsize=(16,10))
    plt.imshow(pil_img)
    ax = plt.gca()
    for p, (xmin, ymin, xmax, ymax), c in zip(prob, boxes.tolist(), COLORS * 100):
        ax.add_patch(plt.Rectangle((xmin, ymin), xmax - xmin, ymax - ymin,
                                   fill=False, color=c, linewidth=3))
        cl = p.argmax()
        text = f'{CLASSES[cl]}: {p[cl]:0.2f}'
        ax.text(xmin, ymin, text, fontsize=15,
                bbox=dict(facecolor='yellow', alpha=0.5))
    plt.axis('off')
    plt.show()

I want to measure the distance from car to camera. If the car is nearby, the distance value should be around 0.2-0.4 If the car is far from the camera, the distance value should be around 0.6-0.8.
I also found a solution for my problem: https://pythonprogramming.net/detecting-distances-self-driving-car/
But here author uses an old model. This model doesn't work well.

Output:
In comments you requested code that works similarly to the link you provided. I want to make it clear your source example isn't measuring distance. It is only measuring the width of the bounding boxes on the vehicles. The logic is based on the concept that larger widths are closer to the camera, and a smaller widths are further from the camera. This approach has many flaws due to optical illusions and lack of size and scale context. At any rate:
def plot_results(pil_img, prob, boxes):
    granularity = 3 # fiddle with this to scale
    img_width_inches = 16
    img_height_inches = 10
    fig = plt.figure(figsize=(img_width_inches, img_height_inches))
    img_width_pixels = img_width_inches * fig.dpi
    img_height_pixels = img_height_inches * fig.dpi
    plt.imshow(pil_img)
    ax = plt.gca()
    for p, (xmin, ymin, xmax, ymax), c in zip(prob, boxes.tolist(), COLORS * 100):
        ax.add_patch(plt.Rectangle((xmin, ymin), xmax - xmin, ymax - ymin,
                               fill=False, color=c, linewidth=3))
        cl = p.argmax()
        text = f'{CLASSES[cl]}: {p[cl]:0.2f}'
        ax.text(xmin, ymin, text, fontsize=15, bbox=dict(facecolor='yellow', alpha=0.5))
        # get width of bounding box
        box_width_pixels = xmax - xmin
        # normalize the box width with image width
        normalized_width = box_width_pixels / img_width_pixels
        # invert with 1 - apply power of granularity and round to 1 place
        apx_distance = round(((1 - (normalized_width))**granularity), 1)
        # get middle of box in pixels
        mid_x = (xmin + xmax) / 2
        mid_y = (ymin + ymax) / 2
        # draw value
        ax.text(mid_x, mid_y, apx_distance, fontsize=15, color="white")
        # normalize the middle x position with image width
        mid_x_normalized = mid_x / img_width_pixels
        # create arbitrary ranges and logic to consider actionable
        if apx_distance <= 0.5:
            if mid_x_normalized > 0.3 and mid_x_normalized < 0.7:
                ax.text(50, 50, "WARNING!!!", fontsize=26, color="red")

    plt.axis('off')
    plt.show()

Output:

The main difference between this code and the example you provided is that the bounding box values you've given (504.8863220214844, 410.2454833984375, 937.6451416015625, 723.9139404296875) represent pixels. However, the code in the example has bounding box values that are already normalized between 0 and 1 in relation to the image size. This is why I verbosely defined the image width and height in inches and pixels (also for self explaining code). They are needed to normalize the pixel based widths and positions so they are between 0 and 1 to match the logic in your example, and which you requested. These values can also be helpful when trying to actually measure sizes and distances.
If you are interested in taking this further. I recommend reading about the laws of perspective. Here is an interesting place to start: https://www.handprint.com/HP/WCL/perspect2.html#distance


Instruction: how to get sub 2d tensor with specific value condition in pytorch?

Input:
I want to copy a 2-d torch tensor to a destination tensor containing only values until the first occurrence of 202 value and zero for the rest of the items like this:
source_t=tensor[[101,2001,2034,1045,202,3454,3453,1234,202]
         ,[101,1999,2808,202,17658,3454,202,0,0]
         ,[101,2012,3832,4027,3454,202,3454,9987,202]]

destination_t=tensor[[101,2001,2034,1045,202,0,0,0,0]
                    ,[101,1999,2808,202,0,0,0,0,0]
                    ,[101,2012,3832,4027,3454,202,0,0,0]]

how can I do it?

Output:
I make working and pretty efficient solution.
I have made a little bit more complex source tensor with additional rows with 202 in different places:
import copy
import torch

source_t = torch.tensor([[101, 2001, 2034, 1045, 202, 3454, 3453, 1234, 202],
                         [101, 1999, 2808, 202, 17658, 3454, 202, 0, 0],
                         [101, 2012, 3832, 4027, 3454, 2020, 3454, 9987, 2020],
                         [101, 2012, 3832, 4027, 3454, 202, 3454, 9987, 202],
                         [101, 2012, 3832, 4027, 3454, 2020, 3454, 9987, 202]
                         ])

At the start, we should find occurrences of the first 202. We can find all occurrences and then choose the first one:
index_202 = (source_t == 202).nonzero(as_tuple=False).numpy()
rows_for_replace = list()
columns_to_replace = list()

elements = source_t.shape[1]
current_ind = 0

while current_ind < len(index_202)-1:
    current = index_202[current_ind]
    element_ind = current[1] + 1
    rows_for_replace.extend([current[0]]*(elements-element_ind))
    while element_ind < elements:
        columns_to_replace.append(element_ind)
        element_ind += 1
    if current[0] == index_202[current_ind+1][0]:
        current_ind += 1
    current_ind += 1

After this operation, we have all our indexes which we should replace with zeros. 4 elements in the first row, 5 in the second, 3 in the fourth row and nothing in the third and fifth.
rows_for_replace, columns_to_replace
([0, 0, 0, 0, 1, 1, 1, 1, 1, 3, 3, 3], [5, 5, 5, 5, 4, 4, 4, 4, 4, 6, 6, 6])

Then we just copy our source tensor and set new values in place:
destination_t = copy.deepcopy(source_t)
destination_t[rows_for_replace, columns_to_replace] = 0

Summary:
source_t
tensor([[  101,  2001,  2034,  1045,   202,  3454,  3453,  1234,   202],
        [  101,  1999,  2808,   202, 17658,  3454,   202,     0,     0],
        [  101,  2012,  3832,  4027,  3454,  2020,  3454,  9987,  2020],
        [  101,  2012,  3832,  4027,  3454,   202,  3454,  9987,   202],
        [  101,  2012,  3832,  4027,  3454,  2020,  3454,  9987,   202]])

destination_t
tensor([[ 101, 2001, 2034, 1045,  202,    0,    0,    0,    0],
        [ 101, 1999, 2808,  202,    0,    0,    0,    0,    0],
        [ 101, 2012, 3832, 4027, 3454, 2020, 3454, 9987, 2020],
        [ 101, 2012, 3832, 4027, 3454,  202,    0,    0,    0],
        [ 101, 2012, 3832, 4027, 3454, 2020, 3454, 9987,  202]])


Instruction: Pytorch Binary Classification RNN Model not Learning

Input:
I'm working on a binary classification task with Pytorch and my model is failing to learn, I can't figure out if it is a problem with the model or with the data.
Here is my model:
from torch import nn

class RNN(nn.Module):
    def __init__(self, input_dim):
        super(RNN, self).__init__()

        self.rnn = nn.RNN(input_size=input_dim, hidden_size=64,
                          num_layers=2,
                          batch_first=True, bidirectional=True)

        self.norm = nn.BatchNorm1d(128)

        self.rnn2 = nn.RNN(input_size=128, hidden_size=64,
                          num_layers=2,
                          batch_first=True, bidirectional=False)

        self.drop = nn.Dropout(0.5)

        self.fc7 = nn.Linear(64, 2)
        self.sigmoid2 = nn.Softmax(dim=2)

    def forward(self, x):
        out, h_n = self.rnn(x)
        out = out.permute(0, 2, 1)
        out = self.norm(out)
        out = out.permute(0, 2, 1)

        out, h_n = self.rnn2(out)
        out = self.drop(out)

        out = self.fc7(out)
        out = self.sigmoid2(out)
        return out.squeeze()

The model consists in two RNN layers, with a BatchNorm in between, then a Dropout and the last layer, I use Softmax function with two classes instead of Sigmoid for evaluation purposes.
Then I create and train the model:
model = RNN(2476)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
loss_function = nn.CrossEntropyLoss()
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)

model.train()

EPOCHS = 25
BATCH_SIZE = 64

epoch_loss = []
for ii in range(EPOCHS):
    for i in range(1, X_train.size()[0]//BATCH_SIZE+1):
        x_train = X_train[(i-1)*BATCH_SIZE:i*BATCH_SIZE]
        labels = y_train[(i-1)*BATCH_SIZE:i*BATCH_SIZE]

        optimizer.zero_grad()

        y_pred = model(x_train)
        y_pred = y_pred.round()

        single_loss = loss_function(y_pred, labels.long().squeeze())

        single_loss.backward()
        optimizer.step()
        lr_scheduler.step()

        print(f"\rBatch {i}/{X_train.size()[0]//BATCH_SIZE+1} Trained: {i*BATCH_SIZE}/{X_train.size()[0]} Loss: {single_loss.item():10.8f} Step: {lr_scheduler.get_lr()}", end="")

    epoch_loss.append(single_loss.item())
    print(f'\nepoch: {ii:3} loss: {single_loss.item():10.8f}')

This is the output when training the model:
Batch 353/354 Trained: 22592/22644 Loss: 0.86013699 Step: [1.0000000000000007e-21]
epoch:   0 loss: 0.86013699
Batch 353/354 Trained: 22592/22644 Loss: 0.81326193 Step: [1.0000000000000014e-33]
epoch:   1 loss: 0.81326193
Batch 353/354 Trained: 22592/22644 Loss: 0.87576205 Step: [1.0000000000000022e-45]
epoch:   2 loss: 0.87576205
Batch 353/354 Trained: 22592/22644 Loss: 0.92263710 Step: [1.0000000000000026e-57]
epoch:   3 loss: 0.92263710
Batch 353/354 Trained: 22592/22644 Loss: 0.90701210 Step: [1.0000000000000034e-68]
epoch:   4 loss: 0.90701210
Batch 353/354 Trained: 22592/22644 Loss: 0.92263699 Step: [1.0000000000000039e-80]
epoch:   5 loss: 0.92263699
Batch 353/354 Trained: 22592/22644 Loss: 0.82888693 Step: [1.0000000000000044e-92]
epoch:   6 loss: 0.82888693
Batch 353/354 Trained: 22592/22644 Loss: 0.81326193 Step: [1.000000000000005e-104]]
epoch:   7 loss: 0.81326193
Batch 353/354 Trained: 22592/22644 Loss: 0.87576205 Step: [1.0000000000000055e-115]
epoch:   8 loss: 0.87576205
Batch 353/354 Trained: 22592/22644 Loss: 0.82888693 Step: [1.0000000000000062e-127]
epoch:   9 loss: 0.82888693
Batch 353/354 Trained: 22592/22644 Loss: 0.81326199 Step: [1.0000000000000067e-139]
epoch:  10 loss: 0.81326199
Batch 353/354 Trained: 22592/22644 Loss: 0.82888693 Step: [1.0000000000000072e-151]
epoch:  11 loss: 0.82888693
Batch 353/354 Trained: 22592/22644 Loss: 0.89138699 Step: [1.0000000000000076e-162]
epoch:  12 loss: 0.89138699
Batch 353/354 Trained: 22592/22644 Loss: 0.82888699 Step: [1.000000000000008e-174]]
epoch:  13 loss: 0.82888699
Batch 353/354 Trained: 22592/22644 Loss: 0.82888687 Step: [1.0000000000000089e-186]
epoch:  14 loss: 0.82888687
Batch 353/354 Trained: 22592/22644 Loss: 0.82888693 Step: [1.0000000000000096e-198]
epoch:  15 loss: 0.82888693
Batch 353/354 Trained: 22592/22644 Loss: 0.84451199 Step: [1.0000000000000103e-210]
epoch:  16 loss: 0.84451199
Batch 353/354 Trained: 22592/22644 Loss: 0.96951205 Step: [1.0000000000000111e-221]
epoch:  17 loss: 0.96951205
Batch 353/354 Trained: 22592/22644 Loss: 0.87576205 Step: [1.0000000000000117e-233]
epoch:  18 loss: 0.87576205
Batch 353/354 Trained: 22592/22644 Loss: 0.89138705 Step: [1.0000000000000125e-245]
epoch:  19 loss: 0.89138705
Batch 353/354 Trained: 22592/22644 Loss: 0.79763699 Step: [1.0000000000000133e-257]
epoch:  20 loss: 0.79763699
Batch 353/354 Trained: 22592/22644 Loss: 0.84451199 Step: [1.0000000000000138e-268]
epoch:  21 loss: 0.84451199
Batch 353/354 Trained: 22592/22644 Loss: 0.84451205 Step: [1.0000000000000146e-280]
epoch:  22 loss: 0.84451205
Batch 353/354 Trained: 22592/22644 Loss: 0.79763693 Step: [1.0000000000000153e-292]
epoch:  23 loss: 0.79763693
Batch 353/354 Trained: 22592/22644 Loss: 0.87576205 Step: [1.000000000000016e-304]]
epoch:  24 loss: 0.87576205

And this is the loss per epoch:

For the data, each of the features in the input data have a dimension of (2474,), and the targets have 1 dimension (either [1] or [0]) , then I add the sequence length dimension (1) to the input data for the RNN layers :
X_train.size(), X_test.size(), y_train.size(), y_test.size()

(torch.Size([22644, 1, 2474]),
 torch.Size([5661, 1, 2474]),
 torch.Size([22644, 1]),
 torch.Size([5661, 1]))

Distribution of the target classes:

I can't figure out why my model is not learning, the classes are balanced and I haven't notified anything wrong with the data. Any suggestions?

Output:
This is not a direct solution to your problem, but what was the process that led to this architecture? I've found it helpful to build up complexity iteratively if only to make identifying issues more trivial (what did I add just before the issue arose?).
To save time on constructing your RNN iteratively, you can try single-batch training by which you construct a network that can overfit a single training batch. If your network can overfit a single training batch, it should be complex enough to learn the features in the training data.
Once you have an architecture that can easily overfit a single training batch, you can then train with the entire training set and explore additional strategies to account for overfitting through regularization.
Your model doesn't seem overly complex but this may mean starting with a single rnn layer and a single linear layer to see if your loss will budge on a single batch.


Instruction: How can I prepare and split data set for ImageNet with vgg16

Input:
 ''' I am trying to classify image using PyTorch but I did manage to
  stipulate my our data set to use it with vgg16 architecture '''

# ADD YOUR CODE HERE
def evaluate():
  running_loss = 0.0 #    counter = 0
   # Tell torch not to calculate gradients
  with torch.no_grad():
    for i, data in enumerate(testloader, 0):
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data
      # Move to device
      inputs = inputs.to(device = device)
      labels = labels.to(device = device)
      # Forward pass
      outputs = model(inputs)
      # Calculate Loss
      loss = criterion(outputs, labels)
      # Add loss to the validation set's running loss
      running_loss += loss.item()
    # Since our model  find the real percentages by  the  following
  val_loss = running_loss / len(testloader)
  print('val loss: %.3f' % (val_loss))
     # Get the top class of the output
  return val_loss
## 1. Dataset

Load the dataset you were given. Images should be stored in an X variable and your labels in a Y variable. Split your dataset into train, validation and test set and pre-process your data for training.
def eval_acc(train=False):
  correct = 0
  total = 0
  # since we're not training, we don't need to calculate the
  #gradients
  #for our outputs
  with torch.no_grad():
      loader = trainloader if train else testloader
      for data in loader:
          images, labels = data
          images = images.to(device = device)
          labels = labels.to(device = device)
          # calculate outputs by running images through the network
          outputs = model(images)
          # the class with the highest energy is what we choose as
          #prediction
          _, predicted = torch.max(outputs.data, 1)
          total += labels.size(0)
          correct += (predicted == labels).sum().item()

   # Print out the information
  print('Accuracy of the network on the 10000 %s images: %d %%' % ('train' if train else 'test', 100 * correct / total))

Output:
You are missing a return statement in your forward() method.
def forward(self,x):
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x # <--- THIS


Instruction: How to batch a nested list of graphs in pytorch geometric

Input: I am currently training a model which is a mix of graph neural networks and LSTM. However that means for each of my training sample, I need to pass in a list of graphs. The current batch class in torch_geometric supports batching with torch_geometric.data.Batch.from_data_list() but this only allows one graph for each data point. How else can I go about batching the graphs?

Output:
Use diagonal batching:
https://pytorch-geometric.readthedocs.io/en/latest/notes/batching.html
Simply, you will put all the graphs as subgraphs into one big graph. All the subgraphs will be isolated.
See the example from TUDataset:
https://colab.research.google.com/drive/1I8a0DfQ3fI7Njc62__mVXUlcAleUclnb?usp=sharing


Instruction: Linear regression using Pytorch

Input:
I have classification problem. I am using Pytorch, My input is sequence of length 341 and output one of three classes {0,1,2}, I want to train linear regression model using pytorch, I created the following class but during the training, the loss values start to have numbers then inf then NAN. I do not know how to fix that . Also I tried to initialize the weights for linear model but it is the same thing. Any suggestions.
class regression(nn.Module):
    def __init__(self, input_dim):
        super().__init__()
        self.input_dim = input_dim
        # One layer
        self.linear = nn.Linear(input_dim, 1)

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

criterion = torch.nn.MSELoss()

def fit(model, data_loader, optim, epochs):
    for epoch in range(epochs):

        for i, (X, y) in enumerate(data_loader):
            X = X.float()
            y = y.unsqueeze(1).float()
            X = Variable(X, requires_grad=True)
            y = Variable(y, requires_grad=True)
            # Make a prediction for the input X
            pred = model(X)
            #loss = (y-pred).pow(2).mean()
            loss = criterion(y, pred)
            optim.zero_grad()
            loss.backward()
            optim.step()
            print(loss)
            print(type(loss))
        # Give some feedback after each 5th pass through the data
        if epoch % 5 == 0:
            print("Epoch", epoch, f"loss: {loss}")
    return None
regnet = regression(input_dim=341)
optim = SGD(regnet.parameters(), lr=0.01)
fit(regnet, data_loader, optim=optim, epochs=5)
pred = regnet(torch.Tensor(test_set.data_info).float())
pred = pred.detach().numpy()

Output:
I would additionally suggest to replace MSE with CrossEntropy Loss as it is better suited for multi-class classificiation problems.
import random
import torch
from torch import nn, optim
from matplotlib import pyplot as plt

# Generate random dataset with your shape to test
# Replace this with your own dataset
data = []
for label in [0, 1, 2]:
    for i in range(1000):
        data.append((torch.rand(341), label))

# train test split
random.shuffle(data)
train, val = data[:1500], data[1500:]

 def run_gradient_descent(model, data_train, data_val, batch_size=64, learning_rate=0.01, weight_decay=0, num_epochs=10):
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
    iters, losses = [], []
    iters_sub, train_acc, val_acc = [], [] ,[]
    train_loader = torch.utils.data.DataLoader(data_train, batch_size=batch_size, shuffle=True)

    # training
    n = 0 # the number of iterations
    for epoch in range(num_epochs):
        for xs, ts in iter(train_loader):
            if len(ts) != batch_size:
                continue
            zs = model(xs)
            loss = criterion(zs, ts) # compute the total loss
            loss.backward() # compute updates for each parameter
            optimizer.step() # make the updates for each parameter
            optimizer.zero_grad() # a clean up step for PyTorch
            # save the current training information
            iters.append(n)
            losses.append(float(loss)/batch_size) # compute *average* loss
            if n % 10 == 0:
                iters_sub.append(n)
                train_acc.append(get_accuracy(model, data_train))
                val_acc.append(get_accuracy(model, data_val))
             # increment the iteration number
            n += 1

    # plotting
    plt.title("Training Curve (batch_size={}, lr={})".format(batch_size, learning_rate))
    plt.plot(iters, losses, label="Train")
    plt.xlabel("Iterations")
    plt.ylabel("Loss")
    plt.show()
    plt.title("Training Curve (batch_size={}, lr={})".format(batch_size, learning_rate))
    plt.plot(iters_sub, train_acc, label="Train")
    plt.plot(iters_sub, val_acc, label="Validation")
    plt.xlabel("Iterations")
    plt.ylabel("Accuracy")
    plt.legend(loc='best')
    plt.show()
    return model

def get_accuracy(model, data):
    loader = torch.utils.data.DataLoader(data, batch_size=500)
    correct, total = 0, 0
    for xs, ts in loader:
        zs = model(xs)
        pred = zs.max(1, keepdim=True)[1] # get the index of the max logit
        correct += pred.eq(ts.view_as(pred)).sum().item()
        total += int(ts.shape[0])
    return correct / total

class MyRegression(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(MyRegression, self).__init__()
        # One layer
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        return self.linear(x)

model = MyRegression(341, 3)
run_gradient_descent(model, train, val, batch_size=64, learning_rate=0.01, num_epochs=10)


Instruction: convert boundingPoly to yolo format

Input:
Yolov5 doesn't support segmentation labels and I need to convert it into the correct format.
How would you convert this to yolo format?
        "boundingPoly": {
            "normalizedVertices": [{
                "x": 0.026169369
            }, {
                "x": 0.99525446
            }, {
                "x": 0.99525446,
                "y": 0.688811
            }, {
                "x": 0.026169369,
                "y": 0.688811
            }]
        }

The yolo format looks like this
0 0.588196 0.474138 0.823607 0.441645
<object-class> <x> <y> <width> <height>

Output:
After our back and forth in the comments I have enough info to answer your question. This is output from the Google Vision API. The normalizedVertices are similar to the YOLO format, because they  are "normalized" meaning the coordinates are scaled between 0 and 1 as opposed to being pixels from 1 to n. Still, you need to do some transformation to put into the YOLO format. In the YOLO format, the X and Y values in the 2nd and 3rd columns refer to the center of the bounding box, as opposed to one of the corners.
Here is a code snipped that will sample at https://ghostbin.com/hOoaz/raw into the follow string in YOLO format '0 0.5080664305 0.5624289849999999 0.9786587390000001 0.56914843'
#Sample annotation output
json_annotation = """
      [
        {
          "mid": "/m/01bjv",
          "name": "Bus",
          "score": 0.9459266,
          "boundingPoly": {
            "normalizedVertices": [
              {
                "x": 0.018737061,
                "y": 0.27785477
              },
              {
                "x": 0.9973958,
                "y": 0.27785477
              },
              {
                "x": 0.9973958,
                "y": 0.8470032
              },
              {
                "x": 0.018737061,
                "y": 0.8470032
              }
            ]
          }
        }
      ]
"""

import json
json_object = json.loads(json_annotation, strict=False)

#Map all class names to class id
class_dict = {"Bus": 0}
#Get class id for this record
class_id = class_dict[json_object[0]["name"]]

#Get the max and min values from segmented polygon points
normalizedVertices = json_object[0]["boundingPoly"]["normalizedVertices"]
max_x = max([v['x'] for v in normalizedVertices])
max_y = max([v['y'] for v in normalizedVertices])
min_x = min([v['x'] for v in normalizedVertices])
min_y = min([v['y'] for v in normalizedVertices])

width = max_x - min_x
height = max_y - min_y
center_x = min_x + (width/2)
center_y = min_y + (height/2)

yolo_row = str(f"{class_id} {center_x} {center_y} {width} {height}")
print(yolo_row)

If you are trying to train a YOLO model there are a few more steps you will need to do: You need to setup the images and annotations in a particular folder structure. But this should help you convert your annotations.


Instruction: Concat two tensors with different dimensions

Input:
I have two tensors a and b which are of different dimensions. a is of shape [100,100] and b is of the shape [100,3,10]. I want to concatenate these two tensors.
For example:
a = torch.randn(100,100)
tensor([[ 1.3236,  2.4250,  1.1547,  ..., -0.7024,  1.0758,  0.2841],
        [ 1.6699, -1.2751, -0.0120,  ..., -0.2290,  0.9522, -0.4066],
        [-0.3429, -0.5260, -0.7748,  ..., -0.5235, -1.8952,  1.2944],
        ...,
        [-1.3465,  1.2641,  1.6785,  ...,  0.5144,  1.7024, -1.0046],
        [-0.7652, -1.2940, -0.6964,  ...,  0.4661, -0.3998, -1.2428],
        [-0.4720, -1.0981, -2.3715,  ...,  1.6423,  0.0560,  1.0676]])

The tensor b is as follows:
tensor([[[ 0.4747, -1.9529, -0.0448,  ..., -0.9694,  0.8009, -0.0610],
         [ 0.5160,  0.0810,  0.1037,  ..., -1.7519, -0.3439,  1.2651],
         [-0.5975, -0.2000, -1.6451,  ...,  1.3082, -0.4023, -0.3105]],
        ...,

        [[ 0.4747, -1.9529, -0.0448,  ..., -0.9694,  0.8009, -0.0610],
         [ 0.1939,  1.0365, -0.0927,  ..., -2.4948, -0.2278, -0.2390],
         [-0.5975, -0.2000, -1.6451,  ...,  1.3082, -0.4023, -0.3105]]],
       dtype=torch.float64, grad_fn=<CopyBackwards>)

I want to concatenate such that the first row in tensor a of size [100] is concatenated with the first row in tensor b which is of size [3,10]. This should be applicable to all rows in both tensors. That is, in simple words, considering just the first row in a and b, I want to get an output with size [100,130] as follows:
[ 1.3236,  2.4250,  1.1547,  ..., -0.7024,  1.0758,  0.2841, 0.4747, -1.9529, -0.0448,  ..., -0.9694,  0.8009, -0.0610, 0.5160,  0.0810,  0.1037,  ..., -1.7519, -0.3439,  1.2651, -0.5975, -0.2000, -1.6451,  ...,  1.3082, -0.4023, -0.3105]

In order to do this, I performed unsqueezed to tensor a to get the two tensors in the same dimensions as follows.
a = a.unsqueeze(1)

When I perform torch.cat([a,b], I still get an error. Can somebody help me in solving this?
Thanks in advance.

Output:
Reshape b tensor accordingly and then merge it to a using torch.cat on 1 dim
torch.cat((a, b.reshape(100, -1)), dim=1)


Instruction: Numpy arrays comparison

Input: I have 2 pytorch tensors (single column) of 40 elements To compare them element by element I converted them to numpy arrays with a single column of 40 elements. I want to compare both arrays element by element and if the value is greater than 0.5 in one array make it 1 else 0 and convert the result again to pytorch tensor. How do I do that.

Output:
Maybe this helps:
import numpy as np
a = np.array([1, 2, 3, 4, 5])
b = np.array([1.1, 2.6, 3.3, 4.6, 5.5])
(np.abs(a-b)>0.5).astype(int)
>>> array([0, 1, 0, 1, 0])


Instruction: How to transform output of NN, while still being able to train?

Input:
I have a neural network which outputs output. I want to transform output before the loss and backpropogation happen.
Here is my general code:
with torch.set_grad_enabled(training):
                  outputs = net(x_batch[:, 0], x_batch[:, 1]) # the prediction of the NN
                  # My issue is here:
                  outputs = transform_torch(outputs)
                  loss = my_loss(outputs, y_batch)

                  if training:
                      scheduler.step()
                      loss.backward()
                      optimizer.step()

Following the advice in How to transform output of neural network and still train? , I have a transformation function which I put my output through:
def transform_torch(predictions):
    new_tensor = []
    for i in range(int(len(predictions))):
      arr = predictions[i]
      a = arr.clone().detach()

      # My transformation, which results in a positive first element, and the other elements represent decrements of the first positive element.

      b = torch.negative(a)
      b[0] = abs(b[0])
      new_tensor.append(torch.cumsum(b, dim = 0))

      # new_tensor[i].requires_grad = True
    new_tensor = torch.stack(new_tensor, 0)

    return new_tensor

Note: In addition to clone().detach(), I also tried the methods described in Pytorch preferred way to copy a tensor, to similar result.
My problem is that no training actually happens with this tensor that is tranformed.
If I try to modify the tensor in-place (e.g. directly modify arr), then Torch complains that I can't modify a tensor in-place with a gradient attached to it.
Any suggestions?

Output:
Calling detach on your predictions stops gradient propagation to your model. Nothing you do after that can change your parameters.
How about modifying your code to avoid this:
def transform_torch(predictions):
  b = torch.cat([predictions[:, :1, ...].abs(), -predictions[:, 1:, ...]], dim=1)
  new_tensor = torch.cumsum(b, dim=1)
  return new_tensor

A little test you can run, to verify that gradients do propagate through this transformation is:
# start with some random tensor representing the input predictions
# make sure it requires_grad
pred = torch.rand((4, 5, 2, 3)).requires_grad_(True)
# transform it
tpred = transform_torch(pred)

# make up some "default" loss function and back-prop
tpred.mean().backward()

# check to see all gradients of the original prediction:
pred.grad
# as you can see, all gradients are non-zero
Out[]:
tensor([[[[ 0.0417,  0.0417,  0.0417],
          [ 0.0417,  0.0417,  0.0417]],

         [[-0.0333, -0.0333, -0.0333],
          [-0.0333, -0.0333, -0.0333]],

         [[-0.0250, -0.0250, -0.0250],
          [-0.0250, -0.0250, -0.0250]],

         [[-0.0167, -0.0167, -0.0167],
          [-0.0167, -0.0167, -0.0167]],

         [[-0.0083, -0.0083, -0.0083],
          [-0.0083, -0.0083, -0.0083]]],

        [[[ 0.0417,  0.0417,  0.0417],
          [ 0.0417,  0.0417,  0.0417]],

         [[-0.0333, -0.0333, -0.0333],
          [-0.0333, -0.0333, -0.0333]],

         [[-0.0250, -0.0250, -0.0250],
          [-0.0250, -0.0250, -0.0250]],

         [[-0.0167, -0.0167, -0.0167],
          [-0.0167, -0.0167, -0.0167]],

         [[-0.0083, -0.0083, -0.0083],
          [-0.0083, -0.0083, -0.0083]]],

        [[[ 0.0417,  0.0417,  0.0417],
          [ 0.0417,  0.0417,  0.0417]],

         [[-0.0333, -0.0333, -0.0333],
          [-0.0333, -0.0333, -0.0333]],

         [[-0.0250, -0.0250, -0.0250],
          [-0.0250, -0.0250, -0.0250]],

         [[-0.0167, -0.0167, -0.0167],
          [-0.0167, -0.0167, -0.0167]],

         [[-0.0083, -0.0083, -0.0083],
          [-0.0083, -0.0083, -0.0083]]],

        [[[ 0.0417,  0.0417,  0.0417],
          [ 0.0417,  0.0417,  0.0417]],

         [[-0.0333, -0.0333, -0.0333],
          [-0.0333, -0.0333, -0.0333]],

         [[-0.0250, -0.0250, -0.0250],
          [-0.0250, -0.0250, -0.0250]],

         [[-0.0167, -0.0167, -0.0167],
          [-0.0167, -0.0167, -0.0167]],

         [[-0.0083, -0.0083, -0.0083],
          [-0.0083, -0.0083, -0.0083]]]])

If you'll try this little test with your original code you'll either get an error that you are trying to propagate through tensors that do not require_grad, or you'll get no grads for the input pred.


Instruction: Why does the loss decreases and the accuracy dosen't increases? PyTorch

Input:
I'm creating a CNN in Pytorch and I'm having some problem with the training function, I believe.
For each epoch, the loss decreases. But the accuracy remains the same, it doesn't change.
The output of the training function is this:
Epoch: 1
correct: 234, N_test: 468 ------>  loss: 58.2041027, accuracy_val: %50.0
Epoch: 2
correct: 234, N_test: 468 ------>  loss: 51.47981386, accuracy_val: %50.0
Epoch: 3
correct: 234, N_test: 468 ------>  loss: 51.57150275, accuracy_val: %50.0
Epoch: 4
correct: 234, N_test: 468 ------>  loss: 39.14232715, accuracy_val: %50.0
Epoch: 5
correct: 234, N_test: 468 ------>  loss: 32.23730827, accuracy_val: %50.0

I know that although they are correlated, loss and accuracy have their complications, but I believe there may be a problem with the code and I am not able to determine what.
Here's the neural network:
class CNN(nn.Module):

# Contructor
def __init__(self):
    super(CNN, self).__init__()
    # Conv1
    self.cnn1 = nn.Conv2d(in_channels=1, out_channels=64, kernel_size=5, stride=1, padding=0)
    self.conv1_bn = nn.BatchNorm2d(64)
    self.maxpool1=nn.MaxPool2d(kernel_size=2, stride=2)

    # Conv2
    self.cnn2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=5,stride=1, padding=0)
    self.conv2_bn = nn.BatchNorm2d(64)
    self.maxpool2=nn.MaxPool2d(kernel_size=2, stride=2)

    # Conv3
    self.cnn3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5,stride=1, padding=0)
    self.conv3_bn = nn.BatchNorm2d(128)
    self.maxpool3=nn.MaxPool2d(kernel_size=2, stride=2)

    # FCL 1
    self.fc1 = nn.Linear(in_features=128 * 27 * 27, out_features=500)
    self.bn_fc1 = nn.BatchNorm1d(500)

    # FCL 2
    self.fc2 = nn.Linear(in_features=500, out_features=500)
    self.bn_fc2 = nn.BatchNorm1d(500)

    # FCL3
    self.fc3 = nn.Linear(in_features=500, out_features=1)

# Prediction
def forward(self, x):
    # conv1
    x = self.cnn1(x)
    x = self.conv1_bn(x)
    x = torch.relu(x)
    x = self.maxpool1(x)
    # conv2
    x = self.cnn2(x)
    x = self.conv2_bn(x)
    x = torch.relu(x)
    x = self.maxpool2(x)
    # conv3
    x = self.cnn3(x)
    x = self.conv3_bn(x)
    x = torch.relu(x)
    x = self.maxpool3(x)

    # Fcl1
    x = x.view(x.size(0), -1)
    x = self.fc1(x)
    x = self.bn_fc1(x)
    x = torch.relu(x)
    # Fcl2
    x = self.fc2(x)
    x = self.bn_fc2(x)
    x = torch.relu(x)
    # final fcl
    x = self.fc3(x)
    x = torch.sigmoid(x)

    return x

The training function:
def train_model(model,train_loader,test_loader,optimizer,n_epochs=5):

#global variable
N_test=len(dataset_val)
accuracy_list=[]

loss_list=[]
for epoch in range(n_epochs):
    cost = 0
    model.train()
    print(f"Epoch: {epoch + 1}")
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        z = model(x)
        y = y.unsqueeze(-1)
        y = y.float()
        loss = criterion(z, y)
        loss.backward()
        optimizer.step()
        cost+=loss.item()

    correct=0
    model.eval()
    #perform a prediction on the validation  data
    for x_test, y_test in test_loader:
        x_test, y_test = x_test.to(device), y_test.to(device)

        z = model(x_test)
        _, yhat = torch.max(z.data, 1)
        correct += (yhat == y_test).sum().item()

    accuracy = correct / N_test
    accuracy_list.append(accuracy)
    loss_list.append(cost)
    print(f"------>  loss: {round(cost, 8)}, accuracy_val: %{accuracy * 100}")

return accuracy_list, loss_lis

The plot is this:
Plot with accuracy and loss

Output:
Your outputs are all going to be 1 since you have 1 output and you're taking the max over the 2nd dimension:
    _, yhat = torch.max(z.data, 1)
    correct += (yhat == y_test).sum().item()

To do binary classification you need to pick a threshold and then threshold your data into two classes, or have 2 outputs (probably easier in this case).


Instruction: Pytorch DataLoader doesn't return batched data

Input:
My dataset is composed of image patches obtained from the original image (face patches and random outside of face patches). Patches are stored in a folder with a name of an original image from which patches originate. I created my own DataSet and DataLoader but when I iterate over the dataset data is not returned in batches. A batch of size 1 should include an array of tuples of patches and a label, so with the increased batch size, we should get an array of arrays of tuples with labels. But DataLoader returns only one array of tuples no matter the batch size.
My dataset:
  import os
  import cv2 as cv
  import PIL.Image as Image
  import torchvision.transforms as Transforms
  from torch.utils.data import dataset

  class PatchDataset(dataset.Dataset):
    def __init__(self, img_folder, n_patches):
      self.img_folder = img_folder
      self.n_patches = n_patches
      self.img_names = sorted(os.listdir(img_folder))

      self.transform = Transforms.Compose([
        Transforms.Resize((50, 50)),
        Transforms.ToTensor()
      ])

    def __len__(self):
      return len(self.img_names)

    def __getitem__(self, idx):
      img_name = self.img_names[idx]
      patch_dir = os.path.join(self.img_folder, img_name)
      patches = []

      for i in range(self.n_patches):
        face_patch = cv.imread(os.path.join(patch_dir, f'{str(i)}_face.png'))
        face_patch = cv.cvtColor(face_patch, cv.COLOR_BGR2RGB)
        face_patch = Image.fromarray(face_patch)
        face_patch = self.transform(face_patch)

        patch = cv.imread(os.path.join(patch_dir, f'{str(i)}_patch.png'))
        patch = cv.cvtColor(patch, cv.COLOR_BGR2RGB)
        patch = Image.fromarray(patch)
        patch = self.transform(patch)

        patches.append((face_patch, patch))

      return patches, int(img_name.split('-')[0])

Then I use it as such:
X = PatchDataset(PATCHES_DIR, 9)
train_dl = dataloader.DataLoader(
    X,
    batch_size=10,
    drop_last=True
)

for batch_X, batch_Y in train_dl:
  print(len(batch_X))
  print(len(batch_Y))

In this provided case the batch size is 10, so printing of the batch_Y returns the correct number (10). But the printing of the batch_X returns 9 which is number of patch pairs - returns only one sample from dataset instead of batch of 10 samples where each of them is length of 9.

Output:
You should return a one dimension higher tensor instead of a list of tensors in __get_item__ function call. You can use torch.stack(patches).
def __getitem__(self, idx):
   img_name = self.img_names[idx]
   patch_dir = os.path.join(self.img_folder, img_name)
   patches = []

   for i in range(self.n_patches):
       face_patch = cv.imread(os.path.join(patch_dir, f'{str(i)}_face.png'))
       face_patch = cv.cvtColor(face_patch, cv.COLOR_BGR2RGB)
       face_patch = Image.fromarray(face_patch)
       face_patch = self.transform(face_patch)

       patch = cv.imread(os.path.join(patch_dir, f'{str(i)}_patch.png'))
       patch = cv.cvtColor(patch, cv.COLOR_BGR2RGB)
       patch = Image.fromarray(patch)
       patch = self.transform(patch)

       patches.append((face_patch, patch))

   return torch.stack(patches), int(img_name.split('-')[0])


Instruction: How to use conv2d in this case

Input:
I want to create an NN layer such that:

for the input of size 100 assume every 5 samples create "block"
the layer should compute let's say 3 values for every block
so the input/output sizes of this layer should be: 100 -> 20*3
every block of size 5 (and only this block) is fully connected to the result block of size 3

If I understand it correctly I can use Conv2d for this problem. But I'm not sure how to correctly choose conv2d parameters.
Is Conv2d suitable for this task? If so, what are the correct parameters? Is that

input channels = 100
output channels = 20*3
kernel = (5,1)
?

Output:
You can use either Conv2D or Conv1D.
With the data shaped like batch x 100 x n_features you can use Conv1D with this setup:
Input channels:  n_features
Output channels: 3 * output_features
kernel:          5
strides:         5

Thereby, the kernel is applied to 5 samples and generates 3 outputs. The values for n_features and output_features can be anything you like and might as well be 1. Setting the strides to 5 results in a non-overlapping convolution so that each block uniquely contributes to one output.


Instruction: RuntimeError: expected scalar type Float but found Long neural network

Input:
I know there are some questions that are like this question, but when I follow them it seems to lead me down a rabbit hole. As if The problem I just fixed causes another problem.
Here are 2 of the rabbit hole solutions I have kept because they have seemed to fix their problems. I doubt they would be of any help but here they are just in case.
one:
batch_X = batch_X.to(device=device, dtype=torch.int64)
batch_y = batch_y.to(device=device, dtype=torch.int64)

two:
 x = x.view(x.size(0), -1)

This is the error I'm getting.
Traceback (most recent call last):
  File "c:/Users/14055/Desktop/Research/new 1.0.py", line 93, in <module>
    training()
  File "c:/Users/14055/Desktop/Research/new 1.0.py", line 63, in training
    output = model(batch_X)                              # passed input data here!!!!!!!!!!!!!!!!!!!!!!!!!!
  File "C:\Users\14055\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "c:/Users/14055/Desktop/Research/new 1.0.py", line 31, in forward
    x = F.relu(self.fc1(x))
  File "C:\Users\14055\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\14055\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\14055\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: expected scalar type Float but found Long

My code is below
import torch.cuda
import torch
import numpy as np
import sys
import torch.nn as nn
import torch.nn.functional as F
from torchsummary import summary
#-------------------------------------------------------------------------
torch.cuda.set_device(0)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
np.set_printoptions(threshold=sys.maxsize)
#-------------------------------------------------------------------------

input_data = torch.Tensor(np.load("inputData.npy", allow_pickle=True))
predict_data = torch.Tensor(np.load("predict.npy", allow_pickle=True))

input_data = input_data.type(torch.FloatTensor)
predict_data = predict_data.type(torch.FloatTensor)

print(type(input_data))

class NeuralNet(nn.Module):
    def __init__(self, gpu = True):
        super(NeuralNet, self ).__init__()
        self.fc1 = nn.Linear(248, 750).to(device)
        self.fc2 = nn.Linear(750, 10).to(device)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x).to(device)
        return x.to(device)

def training():
    model.to(device)
    training.criterion = nn.CrossEntropyLoss()
    optimizer= torch.optim.SGD(model.parameters(), lr=0.003, weight_decay= 0.00005, momentum = .9, nesterov = True)
    n_epochs = 20000
    a = np.float64([9,9,9,9,9]) #antioverfit
    testing_loss = 0.0

    BATCH_SIZE = 10
    EPOCHS = 3
    for epoch in range(EPOCHS):

        if(testing_loss <= a[4]): # part of anti overfit
            train_loss = 0.0
            testing_loss = 0.0

            model.train()
            for i in (range(0, len(input_data), BATCH_SIZE)):
                batch_X = input_data[i:i+BATCH_SIZE]
                batch_y = predict_data[i:i+BATCH_SIZE]

                optimizer.zero_grad()

                batch_X = batch_X.to(device=device, dtype=torch.int64) #gpu                        # input data here!!!!!!!!!!!!!!!!!!!!!!!!!!
                batch_y = batch_y.to(device=device, dtype=torch.int64) #gpu                    # larget data here!!!!!!!!!!!!!!!!!!!!!!!!!!

                output = model(batch_X)                              # passed input data here!!!!!!!!!!!!!!!!!!!!!!!!!!
                loss = training.criterion(output, batch_y)
                loss.backward()
                optimizer.step()
                train_loss += loss.item()*batch_X.size(0)

            train_loss = train_loss/len(predict_data.train_loader.dataset)
            print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch+1, train_loss))

            model.eval()  # Gets Validation loss
            train_loss = 0.0
            with torch.no_grad():
                for i in (range(0, len(input_data), BATCH_SIZE)):
                    batch_X = input_data[i:i+BATCH_SIZE]
                    batch_y = predict_data[i:i+BATCH_SIZE]

                    batch_X = batch_X.to(device=device, dtype=torch.int64)
                    batch_y = batch_y.to(device=device, dtype=torch.int64)
                    output = model(batch_X)
                    loss = training.criterion(output, batch_y).to(device=device, dtype=torch.int64)
                    testing_loss += loss.item()*batch_X.size(0)
            testing_loss = testing_loss / len(predict_data.test_loader.dataset)
            print('Validation loss = ' , testing_loss)
            a = np.insert(a,0,testing_loss) # part of anti overfit
            a = np.delete(a,5)
    print('Validation loss = ' , testing_loss)

model = NeuralNet().to(device=device)
#summary(model, input_size=(1, 248, 248))
training()

Output:
Why do you cast X and Y to int64? Mainly, this is the problem.
batch_X = batch_X.to(device=device, dtype=torch.int64) #gpu                        # input data here!!!!!!!!!!!!!!!!!!!!!!!!!!
batch_y = batch_y.to(device=device, dtype=torch.int64) #gpu

You cast batch_X and batch_Y to int64 a.k.a long, but float was expected, hence the error. Replace this with
batch_X = batch_X.to(device=device)
batch_y = batch_y.to(device=device, dtype=torch.int64)

or
batch_X = batch_X.to(device=device, dtype=torch.float)
batch_y = batch_y.to(device=device, dtype=torch.int64)

And this should solve your problem.
EDIT: You only need to keep y as int. Since you are using CrossEntropyLoss which expects target labels (expected to be an int or long). Overall, you need to keep the data type of x to be float, and y should be long or int.


Instruction: Locally save a dataframe from a remote server in VSCode

Input: I'm running a python script in VSCode on a remote server and I want to save a dataframe that is generated in that script locally. Is this somehow possible? Thanks!

Output: You can save the dataframe to a directory (maybe in .csv) on the remote server and download it from the explorer in VSCode by right-clicking on that file.


Instruction: Change last layer on pretrained huggingface model

Input:
I want to re-finetuned a transformer model but I get an unknown error when I tried to train the model.
I can't change the "num_labels" on loading the model.
So, I tried to change it manually
model_name = "mrm8488/flaubert-small-finetuned-movie-review-sentiment-analysis"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name).to('cuda')

num_labels = 3
model.sequence_summary.summary = torch.nn.Linear(in_features=model.sequence_summary.summary.in_features, out_features=num_labels, bias=True)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train['train'],
    eval_dataset=tokenized_test['train'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    #data_collator=data_collator,
)

trainer.train()

The error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-93-8139f38c5ec6> in <module>()
     20 )
     21
---> 22 trainer.train()

7 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847
   2848

ValueError: Expected input batch_size (24) to match target batch_size (16).

Output:
So, There is a solution for this
Just add ignore_mismatched_sizes=True when loading the model as:
model = AutoModelForSequenceClassification.from_pretrained(model_name,num_labels=3, ignore_mismatched_sizes=True).to('cuda')


Instruction: AttributeError: 'DataParallel' object has no attribute 'copy'

Input:
I am trying to resume training monkAI pytorch retinanet. I have loaded with .pt file instead of actual model. The changes are made in Monk_Object_Detection/5_pytorch_retinanet/lib/train_detector.py, check for '# change' in the places where its modified.
def Model(self, model_name="resnet18",gpu_devices=[0]):
    '''
    User function: Set Model parameters

        Available Models
            resnet18
            resnet34
            resnet50
            resnet101
            resnet152

    Args:
        model_name (str): Select model from available models
        gpu_devices (list): List of GPU Device IDs to be used in training

    Returns:
        None
    '''

    num_classes = self.system_dict["local"]["dataset_train"].num_classes();
    if model_name == "resnet18":
        retinanet = model.resnet18(num_classes=num_classes, pretrained=True)
    elif model_name == "resnet34":
        retinanet = model.resnet34(num_classes=num_classes, pretrained=True)
    elif model_name == "resnet50":
        # retinanet = model.resnet50(num_classes=num_classes, pretrained=True)
        # change
        retinanet = torch.load('/content/drive/MyDrive/Object_detection_retinanet/trained_retinanet_40.pt')
    elif model_name == "resnet101":
        retinanet = model.resnet101(num_classes=num_classes, pretrained=True)
    elif model_name == "resnet152":
        retinanet = model.resnet152(num_classes=num_classes, pretrained=True)

    if self.system_dict["params"]["use_gpu"]:
        self.system_dict["params"]["gpu_devices"] = gpu_devices
        if len(self.system_dict["params"]["gpu_devices"])==1:
            os.environ["CUDA_VISIBLE_DEVICES"] = str(self.system_dict["params"]["gpu_devices"][0])
        else:
            os.environ["CUDA_VISIBLE_DEVICES"] = ','.join([str(id) for id in self.system_dict["params"]["gpu_devices"]])
        self.system_dict["local"]["device"] = 'cuda' if torch.cuda.is_available() else 'cpu'

        # change - added 3 lines below
        if isinstance(retinanet,torch.nn.DataParallel):
            retinanet = retinanet.module
        retinanet.load_state_dict(torch.load('/content/drive/MyDrive/Object_detection_retinanet/trained_retinanet_40.pt'))

        retinanet = retinanet.to(self.system_dict["local"]["device"])
        retinanet = torch.nn.DataParallel(retinanet).to(self.system_dict["local"]["device"])

    retinanet.training = True
    retinanet.train()
    retinanet.module.freeze_bn()

    self.system_dict["local"]["model"] = retinanet;

I am getting attribute error, when I call the Model() from main function as shown below:
from train_detector import Detector
gtf = Detector()

#Loading the dataset
root_dir = './'
coco_dir = 'coco_dir'
img_dir = 'images'
set_dir ='train'
gtf.Train_Dataset(root_dir, coco_dir, img_dir, set_dir, batch_size=8, use_gpu=True)

gtf.Model(model_name="resnet50", gpu_devices=[0, 1, 2, 3])

error:
AttributeError                            Traceback (most recent call last)
<ipython-input-22-1a0c8d446904> in <module>()
      3 if PRE_TRAINED:
      4   #Initialising Model
----> 5   gtf.Model(model_name="resnet50", gpu_devices=[0, 1, 2, 3])
      6   #Setting up hyperparameters
      7   gtf.Set_Hyperparams(lr=0.001, val_interval=1, print_interval=20)

2 frames
/content/Monk_Object_Detection/5_pytorch_retinanet/lib/train_detector.py in Model(self, model_name, gpu_devices)
    245             if isinstance(retinanet,torch.nn.DataParallel):
    246                 retinanet = retinanet.module
--> 247             retinanet.load_state_dict(torch.load('/content/drive/MyDrive/Object_detection_retinanet/trained_retinanet_40.pt'))
    248
    249             retinanet = retinanet.to(self.system_dict["local"]["device"])

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1453         # copy state_dict so _load_from_state_dict can modify it
   1454         metadata = getattr(state_dict, '_metadata', None)
-> 1455         state_dict = state_dict.copy()
   1456         if metadata is not None:
   1457             # mypy isn't aware that "_metadata" exists in state_dict

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1176                 return modules[name]
   1177         raise AttributeError("'{}' object has no attribute '{}'".format(
-> 1178             type(self).__name__, name))
   1179
   1180     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'DataParallel' object has no attribute 'copy'

kindly, help me with the solution!

Output:
I found this by simply googling your problem:
retinanet.load_state_dict(torch.load('filename').module.state_dict())
The link to the discussion is here.


Instruction: How to load in graph from networkx into PyTorch geometric and set node features and labels?

Input:
Goal: I am trying to import a graph FROM networkx into PyTorch geometric and set labels and node features.
(This is in Python)
Question(s):

How do I do this [the conversion from networkx to PyTorch geometric]? (presumably by using the from_networkx function)
How do I transfer over node features and labels? (more important question)

I have seen some other/previous posts with this question but they weren't answered (correct me if I am wrong).
Attempt: (I have just used an unrealistic example below, as I cannot post anything real on here)
Let us imagine we are trying to do a graph learning task (e.g. node classification) on a group of cars (not very realistic as I said). That is, we have a group of cars, an adjacency matrix, and some features  (e.g. price at the end of the year). We want to predict the node label (i.e. brand of the car).
I will be using the following adjacency matrix: (apologies, cannot use latex to format this)
A = [(0, 1, 0, 1, 1), (1, 0, 1, 1, 0), (0, 1, 0, 0, 1), (1, 1, 0, 0, 0), (1, 0, 1, 0, 0)]
Here is the code (for Google Colab environment):
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import networkx as nx
from torch_geometric.utils.convert import to_networkx, from_networkx
import torch

!pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.10.0+cpu.html

# Make the networkx graph
G = nx.Graph()

# Add some cars (just do 4 for now)
G.add_nodes_from([
      (1, {'Brand': 'Ford'}),
      (2, {'Brand': 'Audi'}),
      (3, {'Brand': 'BMW'}),
      (4, {'Brand': 'Peugot'}),
      (5, {'Brand': 'Lexus'}),
])

# Add some edges
G.add_edges_from([
                  (1, 2), (1, 4), (1, 5),
                  (2, 3), (2, 4),
                  (3, 2), (3, 5),
                  (4, 1), (4, 2),
                  (5, 1), (5, 3)
])

# Convert the graph into PyTorch geometric
pyg_graph = from_networkx(G)

So this correctly converts the networkx graph to PyTorch Geometric. However, I still don't know how to properly set the labels.
The brand values for each node have been converted and are stored within:
pyg_graph.Brand

Below, I have just made some random numpy arrays of length 5 for each node (just pretend that these are realistic).
ford_prices = np.random.randint(100, size = 5)
lexus_prices = np.random.randint(100, size = 5)
audi_prices = np.random.randint(100, size = 5)
bmw_prices = np.random.randint(100, size = 5)
peugot_prices = np.random.randint(100, size = 5)

This brings me to the main question:

How do I set the prices to be the node features of this graph?
How do I set the labels of the nodes? (and will I need to remove the labels from pyg_graph.Brand when training the network?)

Thanks in advance and happy holidays.

Output:
The easiest way is to add all information to the networkx graph and directly create it in the way you need it. I guess you want to use some Graph Neural Networks. Then you want to have something like below.

Instead of text as labels, you probably want to have a categorial representation, e.g. 1 stands for Ford.
If you want to match the "usual convention". Then you name your input features x and your labels/ground truth y.
The splitting of the data into train and test is done via mask. So the graph still contains all information, but only part of it is used for training. Check the PyTorch Geometric introduction for an example, which uses the Cora dataset.

import networkx as nx
import numpy as np
import torch
from torch_geometric.utils.convert import from_networkx

# Make the networkx graph
G = nx.Graph()

# Add some cars (just do 4 for now)
G.add_nodes_from([
      (1, {'y': 1, 'x': 0.5}),
      (2, {'y': 2, 'x': 0.2}),
      (3, {'y': 3, 'x': 0.3}),
      (4, {'y': 4, 'x': 0.1}),
      (5, {'y': 5, 'x': 0.2}),
])

# Add some edges
G.add_edges_from([
                  (1, 2), (1, 4), (1, 5),
                  (2, 3), (2, 4),
                  (3, 2), (3, 5),
                  (4, 1), (4, 2),
                  (5, 1), (5, 3)
])

# Convert the graph into PyTorch geometric
pyg_graph = from_networkx(G)

print(pyg_graph)
# Data(edge_index=[2, 12], x=[5], y=[5])
print(pyg_graph.x)
# tensor([0.5000, 0.2000, 0.3000, 0.1000, 0.2000])
print(pyg_graph.y)
# tensor([1, 2, 3, 4, 5])
print(pyg_graph.edge_index)
# tensor([[0, 0, 0, 1, 1, 1, 2, 2, 3, 3, 4, 4],
#         [1, 3, 4, 0, 2, 3, 1, 4, 0, 1, 0, 2]])

# Split the data
train_ratio = 0.2
num_nodes = pyg_graph.x.shape[0]
num_train = int(num_nodes * train_ratio)
idx = [i for i in range(num_nodes)]

np.random.shuffle(idx)
train_mask = torch.full_like(pyg_graph.y, False, dtype=bool)
train_mask[idx[:num_train]] = True
test_mask = torch.full_like(pyg_graph.y, False, dtype=bool)
test_mask[idx[num_train:]] = True

print(train_mask)
# tensor([ True, False, False, False, False])
print(test_mask)
# tensor([False,  True,  True,  True,  True])


Instruction: How to get generated tokens in T5 training_step for using user-defined metrics?

Input:
I am fine-tuning T5 for question answering generation and want to add additional measures (e.g., BLEU, ROUGE) for the generated answers, in addition to the loss function.
For that, I believe it would be necessary to obtain the generated tokens (answers) at each training_step. However, after reading the source code, I still have no clue how to add that.
Below I leave an excerpt of my code. I can extract the output.loss and output.logits, but I didn't find a way to get the generated tokens to use additional evaluation metrics.
Thanks in advance.
class MyQAModel(pl.LightningModule):
  def __init__(self):
    super().__init__()
    self.model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)

  def forward(self, input_ids, attention_mask, labels=None):
    output = self.model(
        input_ids,
        attention_mask=attention_mask,
        labels=labels)

    return output.loss, output.logits

  def training_step(self, batch, batch_idx):
    input_ids = batch['input_ids']
    attention_mask=batch['attention_mask']
    labels = batch['labels']
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("train_loss", loss, prog_bar=True, logger=True)
    return {"loss": loss, "predictions":outputs, "labels": labels}

    ...
    (code continues...)
    ....

Output:
You can obtain predicted tokens from output.logits [batch, seq_len, vocab_size] using torch.argmax(output.logits, dim=-1) [batch, seq_len]. Then, to decode the generated sentence from a batch of token ids, run
generated_sentences = []
for predicted_token_ids in torch.argmax(output.logits, dim=-1):
    generated_sentences.append(tokenizer.decode(predicted_token_ids))

# For getting original sentences
original_sentences = []
for sent_ids in input_ids:
    original_sentences.append(tokenizer.decode(sent_ids))


Instruction: Is there a difference between Keras Dense layer and Pytorch's nn.linear layer?

Input:
I noticed the definition of Keras Dense layer says: Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).
So if we have a code like:
model.add(Dense(10, activation = None))

Is it basically the same as:
nn.linear(128, 10)

?
Thank you so much!

Output: Yes, it is the same. model.add (Dense(10, activation = None)) or nn.linear(128, 10) is the same, because it is not activated in both, therefore if you don't specify anything, no activation is applied. It is so!!! :)