Instruction: pytorch sets grad attribute to none if I use simple minus instead of -=

Input:
This is a simple code to show the problem
import torch
X = torch.arange(-3, 3, step=0.1)
Y = X * 3
Y += 0.1 * torch.randn(Y.shape)

def my_train_model(iter):
    w = torch.tensor(-15.0, requires_grad=True)
    lr = 0.1
    for epoch in range(iter):
        print(w.grad)
        yhat = w * X
        loss = torch.mean((yhat - Y) ** 2)
        loss.backward()

        with torch.no_grad():
            print(w.grad)
            w = w - lr * w.grad # gradient exists if w-= lr*w.grad
            print(w.grad)
            w.grad.zero_()
        print(loss)

my_train_model(4)

This sets the w.grad to none after performing the w = w - lr * w.grad, but the problem will be solved if I use w -= lr * w.grad instead of the above expression!
What is the problem with the first expression which sets w.grad to none?

Output:
The function torch.no_grad() guarantees that no gradient is computed, which means any component wrapped in there is created with requires_grad=False, as you can see in this example.
The in-place operation only changes the value of the tensor, from this answer from forum:

An in-place operation is an operation that changes directly the content of a given Tensor without making a copy.

Therefore, with the problem that you proposed, w = w - lr * w.grad will replace w by the copy of itself without gradient, and w -= lr * w.grad only change the content of tensor which is still keep the gradient of itself from the operation outside of torch.no_grad().


Instruction: Expand Pytorch tensor

Input:
The tensor a in shape of torch.Size([2, 2, 1, 2])
how to expand this tensor to the shape of torch.Size([2, 4, 1, 2]) without using for loop which takes time
For example, I have tensor a
>>> print(a, a.shape)
tensor([[[[0.2955, 0.8836]],

         [[0.7607, 0.6657]]],

        [[[0.6779, 0.5109]],

         [[0.0785, 0.6564]]]]) torch.Size([2, 2, 1, 2])

I want to expand it to become tensor b
>>> print(b, b.shape)
tensor([[[[0.2955, 0.8836]],

         [[0.2955, 0.8836]],

         [[0.7607, 0.6657]],

         [[0.7607, 0.6657]]],

        [[[0.6779, 0.5109]],

         [[0.6779, 0.5109]],

         [[0.0785, 0.6564]],

         [[0.0785, 0.6564]]]]) torch.Size([2, 4, 1, 2])

I tried torch.expand() but it only expands when the dimension=1.
How to achieve this format? Thank you

Output:
Pytorch provides multiple functions with different styles to repeat the tensors. The function that you are looking for is named repeat_interleave. Simple example:
>>> a.repeat_interleave(2,dim=1)
tensor([[[[0.2955, 0.8836]],

         [[0.2955, 0.8836]],

         [[0.7607, 0.6657]],

         [[0.7607, 0.6657]]],

        [[[0.6779, 0.5109]],

         [[0.6779, 0.5109]],

         [[0.0785, 0.6564]],

         [[0.0785, 0.6564]]]])


Instruction: Is it possible to get the architecture of neural network built with Tensorflow and Pytorch using static analysis?

Input:
I'm currently analyzing hundreds of code repositories to identify parameter settings of ML algorithms. In this regard, I was wondering if it is possible to extract the architecture of neural networks that are built with Tensorflow and Pytorch using static analysis?
To clarify my problem, consider the development of a neural network with TF and Pytorch. Usually, a model is created by implementing a class that inherits from TF or Pytorch. Within the class, the architecture (e.g., layers) is specified. For example, see the code snipped below:
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
    super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

I was wondering if I can extract the architecture using static analysis. TF provides a function called summary() that prints a summary of a network, including its layers, output shape, and number of parameters. That is exactly what I want to extract with static analysis. The rationale using static analysis is that I analyze hundreds of code repositories and therefore it is not feasible to run the code for each repository.

Output:
You have to compute the Abstract Syntax Tree (AST) of the code that contains the model. For that, you can use an open-source parser.
When you have to AST, you can traverse the graph and extract the model architecture by following tracking the data flow. That part is not straightforward, though. You probably need to look at a few examples and write good tests. But this will not work when you have certain parts of the model defined in other files. You need some way to of doing an interfile static analysis but it is hard.
Hacky way to do it
For each repo, you detect files that contain a class inheriting from nn.Module. You can do this by using the computing AST for each file or simply based on the file content. Now once you know the files containing a module, you create a python file. You import that file in this new file, define and instance of that class. Then you can use .summary function and write it to a file. Probably, there will be multiple nn.Module's per repo when the parts of models defined in several files. To find out the network, you can just take the longest summary.
for repo in repositories:
  for file in repo.get_files():
    if containsNNModule(file):
      create_new_file_with_content(file)
      # The file content contains one line for importing file
      # One line for defining an instance of the module
      # One line to print the summary.
      # Create a python subprocess run the new file, capture its output and save it.


Instruction: Extracting specific blocks from a module list

Input:
I'm using a pretrained model in which there are several self_attentions sequentially stacked each one after another and the number of them is 12. I need to extract the output of the fourth and 10th blocks of this sequential layers. In the following script, the BLock represents each self-attention layer:
dpr = [x.item() for x in torch.linspace(0, 0.1, 12)]  # stochastic depth decay rule

    self.blocks = nn.ModuleList([
        Block(
            dim=embed_dim, num_heads=num_heads, mlp_ratio=mlp_ratio, qkv_bias=qkv_bias, qk_scale=qk_scale,
            drop=drop_rate, attn_drop=attn_drop_rate, drop_path=dpr[i], norm_layer=norm_layer, attention_type=self.attention_type)
        for i in range(12)])

The self-attention layers (stack of Block) are as follows:
## Attention blocks
    for blk in self.blocks:
        x = blk(x, B, T, W)

How can I extract the fourth and the 10th layers' output?

Output:
To extract the output of a layer, you'll need to use hooks. A forward hook is a function that is called after the forward method of the module was executed.
Here's an example of how to do it:
vit = model(...)  # your model with 12 transformer blocks

features = {l:[] for l in range(len(vit.blocks))}  # place holder for the extracted features

def make_hook_function(layer):

  def hook(module, input, output):
    features[layer].appned(output)  # save the output of the layer to the place holder

  return hook

# place the hooks on the layers that interest you
vit.blocks[4].register_forward_hook(make_hook_function(4))
vit.blocks[10].register_forward_hook(make_hook_function(10))

pred = vit(x)  # run an image through the model

# now you can inspect features[4] and features[10]

A very comprehensive example of ViT feature extractor can be found here.


Instruction: How to randomly set k elements in a dimension in pytorch tensor to a specific value?

Input:
For example, if I have tensor (shape [2, 3, 5])
[[[0.8823, 0.9150, 0.3829, 0.9593, 0.3904],
  [0.6009, 0.2566, 0.7936, 0.9408, 0.1332],
  [0.9346, 0.5936, 0.8694, 0.5677, 0.7411]],

 [[0.4294, 0.8854, 0.5739, 0.2666, 0.6274],
  [0.2696, 0.4414, 0.2969, 0.8317, 0.1053],
  [0.2695, 0.3588, 0.1994, 0.5472, 0.0062]]]

and k = 2, I want to randomly set k elements in a dimension (e.g., dim=2) to a specific value (e.g., 5):
[[[0.8823, 0.9150, 0.3829, 5, 5],
  [0.6009, 0.2566, 5, 0.9408, 5],
  [5, 5, 0.8694, 0.5677, 0.7411]],

 [[5, 0.8854, 0.5739, 5, 0.6274],
  [5, 0.4414, 0.2969, 5, 0.1053],
  [0.2695, 0.3588, 5, 0.5472, 5]]]

Output:
Create a random binary mask with k True elements in dimension dim using argsortand set those to 5.
import torch

p = torch.tensor(
    [[[0.8823, 0.9150, 0.3829, 0.9593, 0.3904],
     [0.6009, 0.2566, 0.7936, 0.9408, 0.1332],
     [0.9346, 0.5936, 0.8694, 0.5677, 0.7411]],

    [[0.4294, 0.8854, 0.5739, 0.2666, 0.6274],
     [0.2696, 0.4414, 0.2969, 0.8317, 0.1053],
     [0.2695, 0.3588, 0.1994, 0.5472, 0.0062]]], requires_grad=True)

k = 2
mask = torch.rand(p.shape).argsort(2) < k
torch.where(mask, 5, p)

Output
tensor([[[5.0000, 0.9150, 5.0000, 0.9593, 0.3904],
         [5.0000, 0.2566, 0.7936, 5.0000, 0.1332],
         [0.9346, 5.0000, 0.8694, 0.5677, 5.0000]],

        [[5.0000, 0.8854, 0.5739, 0.2666, 5.0000],
         [5.0000, 0.4414, 0.2969, 5.0000, 0.1053],
         [5.0000, 5.0000, 0.1994, 0.5472, 0.0062]]], grad_fn=<WhereBackward0>)


Instruction: Increasing instances of a class with Data Augmentation

Input:
I am working with some classes of the Charades Dataset https://prior.allenai.org/projects/charades to detect indoor actions.
The structure of my dataset is as follows:

Where:

c025, c137 and c142 are actions;
XR436 has frames result of splitting a video where users are performing action c025 and the same for X3803, ... There is a total of 250 folders.
RI495 has frames result of splitting a video where users are performing action c137 and the same for DI402, ... There is a total of 40 folders.
TUCK3 has frames result of splitting a video where users are performing action c142 and the same for the rest. There is a total of 260 folders.

As you can see, the instances of class c137 are quite unbalanced with regard to class c025 and c142. Thus, i would like to increase the number of instances of this class using data augmentation. The idea is creating twin folders with certain transformations. For example, creating A4DID folder as a twin of RI495 with Equalization over each of the frames, A4456 folder as a twin of RI495 in GrayScale, ARTI3 as a twin of DI402 with rotation over the frames, etc. The pattern of transformations can be the same for every folder or not. Just interesting in augmenting the number of instances.
Do you know how to proceed? I am using Pytorch and I tried with torchvision.transforms and DataLoader from torch.utils.data but I have not achieved the result that I am looking for. Any idea on how to proceed?
PS: Undersampling of c025 and c142 is not an option, due to the classifier is not able to learn well with such limited amount of examples.
Thank you in advance

Output:
A few thoughts:

Standard practice is to use transforms dynamically; that is, each time a data example is loaded, a compose or sequential set of transform operations are applied with random parameter settings. Thus, each time the datum is loaded, the resulting x (inputs) are different. This can be achieved by defining a stack of transforms to apply to each data example as it is loaded in a pytorch dataset object (see here). This helps provide data augmentation.

Class imbalance is a somewhat different issue, and is generally solved by either a.) oversampling (this is acceptable if using the above transform solution because the oversampled examples will have different transforms applied) or b.) over-weighting of these examples in the loss calculation. Of course, neither approach can account for the risk of receiving an out-of-distribution testing example which is higher the fewer and less diverse examples you have for a given class. The former can be acheived by defining a custom Sampler object that yields examples from your dataset in a class-balanced manner. The latter can be achieved by passing weights to the loss function (many pytorch loss functions such as CrossEntropyLoss already support weights).


Instruction: Meaning of output shapes of ResNet9 model layers

Input:
I have a ResNet9 model, implemented in Pytorch which I am using for multi-class image classification. My total number of classes is 6. Using the following code, from torchsummary library, I am able to show the summary of the model, seen in the attached image:
INPUT_SHAPE = (3, 256, 256) #input shape of my image
print(summary(model.cuda(), (INPUT_SHAPE)))

However, I am quite confused about the -1 values in all layers of the ResNet9 model. Also, for Conv2d-1 layer, I am confused about the 64 value in the output shape [-1, 64, 256, 256] as I believe the n_channels value of the input image is 3. Can anyone please help me with the explanation of the output shape values? Thanks!

Output:
Yes
your INPUT_SHAPE is torch.Size([3, 256, 256]) if it's channel first format AND (256, 256, 3) if it's channel last format.
As Pytorch model accepts it in channel first format , for you it shows torch.Size([3, 256, 256])
and talking about  our output shape [-1, 64, 256, 256], this is the output shape of your first conv output which has 64 filter each of 256x256 dim and not your input_shape.
-1 represents your variable batch_size which can be fixed in dataloader


Instruction: What is the Difference between asssigning layer and assigning weight of layer?

Input:
Imagine i have two code examples:
model.embedding = new_emb

and
model.shared.weight = new_emb.weight

What would be the difference between them?

Output:
If you assign a layer weight, the assigned layer weight has to have the same shape as the current layer weight.
If you assign a new layer, you can assign any layer (as long as it can handle the input tensor shape and return the correct output tensor shape).


Instruction: How to determine parameters for nn.Conv2d()

Input:
I am reading this research paper (https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf) and trying to follow along with the code on Github.  I don't understand how the parameters for the nn.Conv2d() were determined.  For the first Conv2d:  Does 64@96*96 mean 64 channels with a 96 x 96 kernel size?  And if so then why is the kernel size 10 in the function?  I have googled the parameters and their meanings and from what I read I understand that its (input_channels, output_channels, kernel_size)
Here is the github post: https://github.com/fangpin/siamese-pytorch/blob/master/train.py
For reference page 4 of the research paper has the model schematic.
       self.conv = nn.Sequential(
            nn.Conv2d(1, 64, 10),  # 64@96*96
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),  # 64@48*48
            nn.Conv2d(64, 128, 7),
            nn.ReLU(),    # 128@42*42
            nn.MaxPool2d(2),   # 128@21*21
            nn.Conv2d(128, 128, 4),
            nn.ReLU(), # 128@18*18
            nn.MaxPool2d(2), # 128@9*9
            nn.Conv2d(128, 256, 4),
            nn.ReLU(),   # 256@6*6
        )
        self.liner = nn.Sequential(nn.Linear(9216, 4096), nn.Sigmoid())
        self.out = nn.Linear(4096, 1)

Output:
If you look at the model schematic, it's showing two things,

Parameters of the convolution kernel,
Parameters of the feature maps (output of the nn.Conv2D op)

For example first conv2d layer is 64@10x10, meaning 64 output channels and a 10x10 kernel.
Whereas the feature map is 64@96x96, which comes from applying 64@10x10 convolution op on 105x105x1 sized input. This way you get 64 output channels and a 105-10+1=96 sized width and height.


Instruction: Plotting a quadratic loss using norm function from Pytorch and showing it using Plotly

Input:
I have a quadratic loss z=(1/2)||Aw-b||^2 where A is 4x2 matrix, w=[x,y] is a 2d vector, and b is a 4d vector. If we plot z, there would be a surface in terms of x,y. I want to plot z using Plotly library. To do this, I want to use Pytorch and the function torch.norm for calculating the norm. Here is a worked example for plotting a 3d surface and I want to modify it as follows:
import plotly.graph_objects as go
import numpy as np

A = torch.tensor([[ 0.1542, -0.0682],
        [ 0.8631,  0.6762],
        [-1.4002,  1.1773],
        [ 0.4614,  0.2431]])

b = torch.tensor([-0.2332, -0.7453,  0.9061,  1.2118])

x = np.arange(-1,1,.01)
y = np.arange(-1,1,.01)
X,Y = np.meshgrid(x,y)

W = ??????

Z = 0.5*torch.norm(torch.matmul(A, W)-b)**2

fig = go.Figure(
    data=[go.Surface(z=Z, x=x, y=y, colorscale="Reds", opacity=0.5)])
fig.update_layout(
    title='My title',
    autosize=False,
    width=500,
    height=500,
    margin=dict(l=65, r=50, b=65, t=90),
    scene_aspectmode='cube'
)
fig.show()

Question:
How should I modify W which includes x,y to plot the surface?

Output:
You could simply do:
Z = [[0.5 * torch.norm(torch.matmul(A, torch.tensor([float(xx), float(yy)]))-b)**2 for xx in x] for yy in y]

Update: You can improve the performance significantly by using torch's micro batch feature. For this you have to reshape your data to lists of matrices. That means you have to extend tensor A to a list that contains only one matrix and W to a list that contains all mesh points, each as matrix.
import plotly.graph_objects as go
import torch

A = torch.tensor([[[0.1542, -0.0682],
                   [0.8631,  0.6762],
                   [-1.4002, 1.1773],
                   [0.4614,  0.2431]]])

b = torch.tensor([-0.2332, -0.7453,  0.9061,  1.2118])

x = torch.arange(-1, 1, 0.01)
y = torch.arange(-1, 1, 0.01)
W = torch.reshape(torch.cartesian_prod(x, y), (len(x) * len(y), 2, 1))

V = torch.reshape(torch.matmul(A, W), (len(x), len(y), 4)) - b
Z = 0.5 * torch.norm(V, dim=2)**2


Instruction: Would this be a valid Implementation of an ordinal CrossEntropy?

Input:
Would this be a valid implementation of a cross entropy loss that takes the ordinal structure of the GT y into consideration? y_hat is the prediction from a neural network.
ce_loss = F.cross_entropy(y_hat, y, reduction="none")
distance_weight = torch.abs(y_hat.argmax(1) - y) + 1
ordinal_ce_loss = torch.mean(distance_weight * ce_loss)

Output:
I'll attempt to answer this question by first fully defining the task, since the question is a bit sparse on details.

I have a set of ordinal classes (e.g. first, second, third, fourth,
etc.) and I would like to predict the class of each data example from
among this set. I would like to define an entropy-based loss-function
for this problem. I would like this loss function to weight the loss
between a predicted class torch.argmax(y_hat) and the true class y
according to the ordinal distance between the two classes. Does the
given loss expression accomplish this?

Short answer: sure, it is "valid". You've roughly implemented L1-norm ordinal class weighting. I'd question whether this is truly the correct weighting strategy for this problem.

For instance, consider that for a true label n, the bin n response is weighted by 1, but the bin n+1 and n-1 responses are weighted by 2. This means that a lot more emphasis will be placed on NOT predicting false positives than on correctly predicting true positives, which may imbue your model with some strange bias.
It also means that examples on the edge will result in a larger total sum of weights, meaning that you'll be weighting examples where the true label is say "first" or "last" more highly than the intermediate classes. (Say you have 5 classes: 1,2,3,4,5. A true label of 1 will require distance_weight of [1,2,3,4,5], the sum of which is 15. A true label of 3 will require distance_weight of [3,2,1,2,3], the sum of which is 11.
In general, classification problems and entropy-based losses are underpinned by the assumption that no set of classes or categories is any more or less related than any other set of classes. In essence, the input data is embedded into an orthogonal feature space where each class represents one vector in the basis. This is quite plainly a bad assumption in your case, meaning that this embedding space is probably not particularly elegant: thus, you have to correct for it with sort of a hack-y weight fix. And in general, this assumption of class non-correlation is probably not true in a great many classification problems (consider e.g. the classic ImageNet classification problem, wherein the class pairs [bus,car], and [bus,zebra] are treated as equally dissimilar. But this is probably a digression into the inherent lack of usefulness of strict ontological structuring of information which is outside the scope of this answer...)

Long Answer: I'd highly suggest moving into a space where the ordinal value you care about is instead expressed in a continuous space. (In the first, second, third example, you might for instance output a continuous value over the range [1,max_place]. This allows you to benefit from loss functions that already capture well the notion that predictions closer in an ordered space are better than predictions farther away in an ordered space (e.g. MSE, Smooth-L1, etc.)
Let's consider one more time the case of the [first,second,third,etc.] ordinal class example, and say that we are trying to predict the places of a set of runners in a race. Consider two races, one in which the first place runner wins by 30% relative to the second place runner, and the second in which the first place runner wins by only 1%. This nuance is entirely discarded by the ordinal discrete classification. In essence, the selection of an ordinal set of classes    truncates the amount of information conveyed in the prediction, which means not only that the final prediction is less useful, but also that the loss function encodes this strange truncation and binarization, which is then reflected (perhaps harmfully) in the learned model. This problem could likely be much more elegantly solved by regressing the finishing position, or perhaps instead by regressing the finishing time, of each athlete, and then performing the final ordinal classification into places OUTSIDE of the network training.
In conclusion, you might expect a well-trained ordinal classifier to produce essentially a normal distribution of responses across the class bins, with the distribution peak on the true value: a binned discretization of a space that almost certainly could, and likely should, be treated as a continuous space.


Instruction: How to set all elements of pytorch tensor to zero after a certain index in the given axis, where the index is given by another pytorch tensor?

Input:
I've two PyTorch tensors
mask = torch.ones(1024, 64, dtype=torch.float32)
indices = torch.randint(0, 64, (1024, ))

For every ith row in mask, I want to set all the elements after the index specified by ith element of indices to zero. For example, if the first element of indices is 50, then I want to set mask[0, 50:]=0. Is it possible to achieve this without using for loop?
Solution with for loop:
for i in range(mask.shape[0]):
    mask[i, indices[i]:] = 0

Output:
You can first generate a tensor of size (1024x64) where each row has numbers arranged from 0 to 63. Then apply a logical operation using the indices reshaped as (1024x1)
mask = torch.ones(1024, 64, dtype=torch.float32)
indices = torch.randint(0, 64, (1024, 1))    # Note the dimensions

mask[torch.arange(0, 64, dtype=torch.float32).repeat(1024,1) >= indices] = 0


Instruction: Unable to convert Tensor to FloatTensor in Pytorch

Input:
I have:
def pytorchConvolution(img, kernel):
    img=torch.from_numpy(img)
    kernel=torch.from_numpy(kernel)

    img.type(torch.FloatTensor)
    kernel.type(torch.FloatTensor)

    dtype_inputs = torch.quint8
    dtype_filters = torch.qint8

    scale, zero_point = 1.0, 0
    q_filters = torch.quantize_per_tensor(kernel, scale, zero_point, dtype_filters)
    q_inputs = torch.quantize_per_tensor(img, scale, zero_point, dtype_inputs)
    bias = torch.randn(8, dtype=torch.float)

    print(img.type())

    convolution2d=qF.conv2d(q_inputs,q_filters,bias)
    return(convolution2d)

which is used here:
blur_filter = (1/250)*np.ones([5, 5])
img_blurred_py = pytorchConvolution(img, blur_filter)

However, upon running this code, I get the following output, meaning PyTorch did not convert the tensor to FloatTensor
torch.DoubleTensor

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [190], in <cell line: 6>()
      4 img_blurred = convolution(img, blur_filter)
      5 img_blurred_c = convolutionImplantation(img, blur_filter)
----> 6 img_blurred_py = pytorchConvolution(img, blur_filter)
      8 plt.axis("off")
      9 plt.subplot(1, 2, 1)

Input In [189], in pytorchConvolution(img, kernel)
      9 dtype_filters = torch.qint8
     11 scale, zero_point = 1.0, 0
---> 12 q_filters = torch.quantize_per_tensor(kernel, scale, zero_point, dtype_filters)
     13 q_inputs = torch.quantize_per_tensor(img, scale, zero_point, dtype_inputs)
     14 bias = torch.randn(8, dtype=torch.float)

RuntimeError: Quantize only works on Float Tensor, got Double

enter image description here
I have set the default_tensor_type to FloatTensor, and tried to convert to other Tensor Types, however, PyTorch does not convert the tensor to any type. I need to convert both the tensors to FloatTensor to quantize them.

Output:
The type function does not operate in place:
In [2]: a = torch.tensor([1,2])

In [3]: a.dtype
Out[3]: torch.int64

In [4]: a.type(torch.FloatTensor)
Out[4]: tensor([1., 2.])

In [5]: a.dtype
Out[5]: torch.int64

You need to assign these back into the variable:
In [7]: a = a.type(torch.FloatTensor)

In [8]: a.dtype
Out[8]: torch.float32


Instruction: Parameters can't be updated when using torch.nn.DataParallel to train on multiple GPUs

Input:
import torch
import torch.nn as nn
import os

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.h = -1
    def forward(self, x):
        self.h =x

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
if torch.cuda.is_available():
    print('using Cuda devices, num:', torch.cuda.device_count())

model = nn.DataParallel(Net())
x = 2
print(model.module.h)
model(x)
print(model.module.h)

When I use multiple GPUs to train my model, I find that the Net's params can't be updated correctly, it remains the initial value. However, when I use only one GPU instead, it's can be correctly updated. How can I fix this problem? thx! (The examples are posted in the image)
This is when I using two GPUs, the param 'h' didn't change:

This is when I using only one GPU, the param 'h' had changed:

Output:
From the PyTorch's documentation (https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html):

In each forward, module is replicated on each device, so any updates to the running module in forward will be lost. For example, if module has a counter attribute that is incremented in each forward, it will always stay at the initial value because the update is done on the replicas which are destroyed after forward.

I am guessing PyTorch skips the copying part when there is only one GPU.
Also, your h is just an attribute. It is not a "parameter" in PyTorch.


Instruction: NLP data processing between `BucketIterator` and `build_vocab_from_iterator`

Input:
I am using AG News Dataset to train model for using text classification.
The part using TabularDataset to generate dataset from csv file.
import torchtext
import torch
from torchtext.legacy.data import Field, TabularDataset, BucketIterator, Iterator

import spacy

def des_tokenize(x):
  return x.split(' ')

def title_tokenize(x):
  return x.split(' ')

def category_tokenize(x):
  return x

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

CATEGORY = Field(tokenize=category_tokenize)
TITLE = Field(tokenize=title_tokenize, init_token='<SOS>', eos_token='<EOS>')
DES = Field(tokenize=des_tokenize, init_token='<SOS>', eos_token='<EOS>')

spacy_en = spacy.load('en_core_web_sm')

train_fields = [('id', None), ('category', CATEGORY), ('title', TITLE), ('description', DES)]

test_fields = [('title', TITLE), ('description', DES)]

train_data = TabularDataset(
              path = '/content/drive/MyDrive/summer2/train.csv',
              format = 'csv',
              fields = train_fields,
              skip_header = True)

test_data = TabularDataset(
              path = '/content/drive/MyDrive/summer2/test.csv',
              format = 'csv',
              fields = test_fields,
              skip_header = True)

After dataset being generated, choosing to use pre-train embedding model called
torchtext.vocab.GloVe to build vocab.
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

train_batch_size = 10
test_batch_size = 1
max_length = 256

tokenizer = get_tokenizer('basic_english')

train_iter = torchtext.legacy.data.BucketIterator(
    train_data,
    batch_size=train_batch_size,
)

test_iter = torchtext.legacy.data.BucketIterator(
    test_data,
    batch_size=test_batch_size,
)

DES.build_vocab(
    train_data,
    vectors=torchtext.vocab.GloVe(name="6B", dim=50, max_vectors=50_000),
    max_size=50_000,
)

TITLE.build_vocab(
    train_data,
    vectors=torchtext.vocab.GloVe(name="6B", dim=50, max_vectors=50_000),
    max_size=50_000,
)

CATEGORY.build_vocab(train_data)

And the output looks great after create_batches function
def create_batches(self):
        self.batches = batch(self.data(), self.batch_size, self.batch_size_fn)

# Create batches - needs to be called before each loop.
train_iter.create_batches()

# Loop through BucketIterator.
print('PyTorchText BuketIterator\n')
for batch in train_iter.batches:

  # Let's check batch size.
  print('Batch size: %d\n'% len(batch))
  print('category\ttitle\tdescription'.ljust(10))

  # Print each example.
  for example in batch:
    print('%s \t %s \t %s'.ljust(10) % (example.category, example.title, example.description))
  print('\n')

  # Only look at first batch. Reuse this code in training models.
  break

Output looks like
PyTorchText BuketIterator

Batch size: 10

category    title   description
2    ['UPDATE', '1-Open-Rejuvenated', 'Haas', 'reaches', 'last', 'eight']    ['Germany', '#39;s', 'Tommy', 'Haas', 'continued', 'his', 'resurgence', 'with', 'a', '7-6', '6-1', '7-5', 'victory', 'over', 'Czech', 'teenager', 'Tomas', 'Berdych', 'on', 'Tuesday', 'to', 'reach', 'the', 'quarter-finals', 'of', 'the', 'US', 'Open', 'for', 'the', 'first', 'time.']
3    ['Japan', '#39;s', 'Nikkei', 'Average,', 'Topix', 'Advance;', 'Toyota,', 'Advantest', 'Gain']   ['Japan', '#39;s', 'Nikkei', '225', 'Stock', 'Average', 'rose', '56.74,', 'or', '0.5', 'percent,', 'to', '11,139.97', 'at', '9:01', 'am', 'in', 'Tokyo.', 'The', 'broader', 'Topix', 'index', 'gained', '5.35,', 'or', '0.5', 'percent,', 'to', '1132.']
2    ['Wildcats', 'on', 'the', 'rise', 'with', 'Santos']     ['The', 'University', 'of', 'New', "Hampshire's", 'impressive', '51-40', 'road', 'victory', 'over', '10th-ranked', 'Villanova', 'Saturday', 'night', 'vaulted', 'the', 'Wildcats', 'three', 'spots', 'to', 'ninth', 'in', 'this', "week's", 'Sports', 'Network', '1-AA', 'football', 'poll,', 'while', 'dropping', 'Villanova', 'to', '14th.']
1    ['Cracking', 'under', 'the', 'strain']      ['Severe', 'cracks', 'surfaced', 'inside', 'the', 'Israeli', 'government', 'this', 'week', 'as', 'its', 'senior', 'law', 'officers', 'publicly', 'fell', 'out', 'with', 'the', 'defence', 'establishment', 'and', 'the', 'Foreign', 'Ministry', 'over', 'the', 'country', '#39;s', 'future', 'strategy', 'in', 'the', 'face', 'of', 'the', 'July', 'verdict', 'of', 'the', 'International', '']
1    ['Arab', 'League', 'to', 'hold', 'emergency', 'meeting']    ['The', 'Arab', 'League', 'says', 'it', 'will', 'hold', 'an', 'emergency', 'session', 'to', 'discuss', 'the', 'violence', 'in', 'Gaza,', 'which', 'has', 'claimed', 'at', 'least', '56', 'Palestinians', 'this', 'week.']
2    ['Holmes', 'to', 'decide', 'on', 'double']      ['Kelly', 'Holmes', 'has', 'still', 'to', 'confirm', 'whether', 'she', 'will', 'attempt', 'to', 'repeat', 'her', 'Olympic', 'double', 'at', 'this', 'weekend', '#39;s', 'World', 'Athletics', 'Final', 'after', 'clearing', 'the', 'first', 'hurdle', 'with', 'a', 'victory', 'in', 'the', '1500m', 'yesterday.']
2    ['NBA', 'suspends', 'nine', 'players,', 'Artest', 'for', 'rest', 'of', 'season']    ['NBA', 'on', 'Sunday', 'suspended', 'nine', 'players', 'for', 'involving', 'in', 'a', 'melee', 'during', 'Friday', '#39;s', 'game', 'between', 'Detorit', 'Pistons', 'and', 'Indiana', 'Pacers,', 'with', 'Ron', 'Artest', 'suspended', 'for', 'the', 'rest', 'of', 'the', 'season,', '73', 'games.']
2    ['On', 'the', 'Far', 'Side', 'of', 'the', 'Field,', 'a', 'Familiar', 'Face']    ['Perhaps', 'there', 'will', 'be', 'a', 'moment', 'during', "Sunday's", 'game', 'between', 'the', 'Giants', 'and', 'the', 'Redskins', 'when', 'a', 'coach', 'and', 'his', 'former', 'franchise', 'quarterback', 'will', 'do', 'a', 'double', 'take.']
3    ['', '#39;QUIET', '#39;', 'RULE', 'MAY', 'CHANGE']      ['The', 'Securities', 'and', 'Exchange', 'Commission', 'wants', 'to', 'scrap', 'a', '1933', 'rule', 'that', 'forces', 'a', 'strict', '', 'quot;quiet', 'period', 'quot;', 'on', 'all', 'talk', 'about', 'a', 'company', 'just', 'prior', 'to', 'its', 'stock', 'being', 'sold', 'initially', 'to', 'the', 'public.']
2    ['Denehy', 'boosts', 'Walpole', '']     ['Danvers', 'coach', 'thought', 'he', 'had', 'the', 'perfect', 'game', 'plan', 'against', 'Walpole', 'last', 'night', 'in', 'the', 'Division', '2', 'playoffs', 'at', 'Endicott', 'College.', 'It', 'was', 'the', 'same', 'game', 'plan', 'that', 'earned', 'his', 'team', 'its', 'first', 'playoff', 'berth', 'in', '63', 'years.']

The question is that what if I use build_vocab_from_iterator to create iterator ?
build_vocab_from_iterator
Does the function has same meaning between my part using BucketIterator ?
Also, I think using Pretrained Word Embeddings GloVe is better than FastText in this work, because the model needs to classify the description is which types.

Output:
After all, the solution which I just post can train the model.
And it had better to use stopwords from library to has better accuracy.


Instruction: Can ideal num_workers for a large dataset in PyTorch be 0?

Input:
I am currently testing out different num_workers in DataLoader in PyTorch and it seems that 0 has the shortest running time.
I also tried out https://github.com/developer0hye/Num-Workers-Search, which is an automated num_workers search based on dataset and batch_size (and some other parameters), and it also gives 0 as the ideal num_workers.
The CPU itself is a server based AMD Epyc with 128 cores (256 threads), running on Ubuntu 20.04.
Can the CPU processing power be the answer to why the ideal num_workers is set to 0? It is a bit counter-intuitive, especially with a large number of threads.

Output:
Yes this definitely can be the case.
If e.g. the bottleneck is e.g. a hard drive from which the data is read, then even with multiple workers data cannot be read faster. Then the overhead of having multiple processes really just decreases the performance.
But it can also be the case that wherever you load your data from is so fast that it is just no bottleneck at all, in which case having more workers also just adds overhead.


Instruction: How to assign a 1d tensor to a 2d tensor using a mask matrix in tensorflow?

Input:
1st: mask (11 true)
2nd: array (11 element)
3nd: zero_tensor (same shape as mask tensor): i want the position can be assigned corresponding element in array (position that was true in mask)
In torch, we can use zero_tensor[mask] = array, but how to do in tensorflow?

Output:
Maybe try something like this:
import tensorflow as tf

x = tf.zeros((5, 3), dtype=tf.int32)
new_values = tf.constant([1, 1, 1, 2, 2, 2, 3, 4, 4, 5, 5], dtype=tf.int32)
mask = tf.constant([[True, True, True],
                    [True, True, True],
                    [True, False, False],
                    [True, True, False],
                    [True, True, False]])
print(x)
new_x = tf.tensor_scatter_nd_update(x, tf.where(mask), new_values)
print(new_x)

tf.Tensor(
[[0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]], shape=(5, 3), dtype=int32)
tf.Tensor(
[[1 1 1]
 [2 2 2]
 [3 0 0]
 [4 4 0]
 [5 5 0]], shape=(5, 3), dtype=int32)


Instruction: PyTorch AutoEncoder : mat1 and mat2 shapes cannot be multiplied (1x512 and 12x64)

Input:
I am trying to pass the output features of a CNN through an AutoEncoder. I used a hooklayer to extract the features of the CNN and converted them into a tensor.
extracted_features = torch.tensor(rn_output)
The size of the data after the conversion from tuple to tensor is torch.Size([1014,512])
The decoder section of the AutoEncoder is throwing the 'cannot be multiplied error' but my belief is the error is due to the setup and shape of the input.
AutoEncoder
class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(in_features=512, out_features=256),  # N, 512 -> N,128
            nn.ReLU(),  # Activation Function
            nn.Linear(in_features=256, out_features=128),
            nn.ReLU(),
            nn.Linear(in_features=128, out_features=64),
            nn.ReLU(),  # Activation Function
            nn.Linear(in_features=64, out_features=12),
        )
        self.decoder = nn.Sequential(
            nn.Linear(in_features=12, out_features=64),  # N, 3 -> N,12
            nn.ReLU(),  # Activation Function
            nn.Linear(in_features=64, out_features=128),
            nn.Linear(in_features=128, out_features=256),
            nn.ReLU(),  # Activation Function
            nn.Linear(in_features=256, out_features=512),
            nn.Tanh()
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(x)
        return decoded

Call to Autoencoder

model = AutoEncoder()
criterion = nn.MSELoss()
optimiser = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)

num_epochs = 10
outputs = []
for epoch in range(num_epochs):
    for (img) in extracted_features:
        recon = model(img)
        loss = criterion(recon, img)

        optimiser.zero_grad()
        loss.backward()
        optimiser.step()

    print(f'Epoch:{epoch+1}, Loss:{loss.item():.4f}')
    outputs.append((epoch, img, recon))

I have tried using a dataloader and passing the data in with a smaller batch size. I have also tried reshaping the images within the forward method but I still continue to get the same error

Output:
I'm pretty sure your forward function is incorrectly performing the encoder-decoder step. I think you should change it from this:
encoded = self.encoder(x)
decoded = self.decoder(x)

to this:
encoded = self.encoder(x)
decoded = self.decoder(encoded)

The decoder generally operates on the encoded input not directly on the input itself, unless you're using a non-standard definition of encoder-decoder I'm unfamiliar with.


Instruction: Why the error information "unrecognized arguments" return?

Input:
When I tried to use ArgumentParser() class to define the argument "epochs" as the training epoch of my CNN model with PyTorch, the system informed me this error. This is my code block:
# 2.1 define super arguments (training epochs for example)

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-e', '--epochs', default=10, type=int,
                    help='number of epochs to train the VAE for')
args = vars(parser.parse_args())

epochs = args['epochs']

When I executed it, the error was thrown out as:
usage: ipykernel_launcher.py [-h] [-e EPOCHS]
ipykernel_launcher.py: error: unrecognized arguments: -f /root/.local/share/jupyter/runtime/kernel-ae55ea85-75ab-4308-8b6a-3319e5b09a40.json
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2

I do not know what this error information means and how to fix it. I never made any extra arguments with the prompt "-f". How can I deal with this error? Many thanks!!

Output:
From your error message, I guess you are running the code in the jupyter notebook environment.
In jupyter notebook, if you want to use argparse, please modify the code to the following form：
args = vars(parser.parse_args(args=[]))


Instruction: How to add modules from torchtext.data?

Input:
I'm trying to do this from torchtext.data import TabularDataset, Field, LabelField, BucketIterator, however, I get the following exception:

ImportError: cannot import name 'TabularDataset' from 'torchtext.data' (D:\Users\shuvo\miniconda3\envs\vi38\lib\site-packages\torchtext\data_init_.py)

But I see it in the documentation (figure below). Is there any other way to import it?

Output: torchtext.data and its contents have been obsolete for a big while now (readthedocs are grossly outdated). For a time being, it's been available as torchtext.legacy.data, but it has been removed entirely in the recent versions as far as I know, in a favour of modern torchdata routines, such as torchdata.datapipes.iter.CSVParser.


Instruction: 'NoneType' object has no attribute 'to' in mps mode(for apple m1 computer chip) for pytorch

Input:
um_epochs = 5
device = torch.device("mps")
d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, None, None, optimizer).to(device)

Then the terminal told me that:
 1 num_epochs = 5
      2 device = torch.device("mps")
----> 3 d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, None, None, optimizer).to(device)

AttributeError: 'NoneType' object has no attribute 'to'

Here is the function:
def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size,
              params=None, lr=None, optimizer=None):
    for epoch in range(num_epochs):
        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0
        for X, y in train_iter:
            y_hat = net(X)
            l = loss(y_hat, y).sum()

            # Gradient zeroing
            if optimizer is not None:
                optimizer.zero_grad()
            elif params is not None and params[0].grad is not None:
                for param in params:
                    param.grad.data.zero_()

            l.backward()
            if optimizer is None:
                sgd(params, lr, batch_size)
            else:
                optimizer.step()

            train_l_sum += l.item()
            train_acc_sum += (y_hat.argmax(dim=1) == y).sum().item()
            n += y.shape[0]
        test_acc = evaluate_accuracy(test_iter, net)
        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'
              % (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))

In the process of searching, I noticed that some alike problems are
owing to the overloading mechanism for "=" and "-". But in this function I didn't find any similar slice...
So what's the matter with this NoneType?

Output:
The function train_ch3 does not return anything explicitly, which means that it returns NonType object. And for sure, NoneType object has not attribute named 'to'
I don't know what type of class dl3 is, but if you want to load a tensor or pytorch module onto a specific device, you should use the method to of that tensor or module like: net.to(device) or X.to(device), y.to(device)


Instruction: Calculate accuracy of a tensor compared to a target tensor

Input:
I have my output tensor like this:
tensor([[0.1834, 0.8166],
        [0.3031, 0.6969],
        [0.3104, 0.6896],
        [0.3065, 0.6935],
        [0.3060, 0.6940],
        [0.2963, 0.7037],
        [0.2340, 0.7660],
        [0.2302, 0.7698],
        [0.2581, 0.7419],
        [0.2081, 0.7919]], grad_fn=<PowBackward0>)

I would like to first convert my output tensor to something like this:
tensor([1., 1., 1......])

where the value indicate the index of the larger value(for example, 0.8166 > 0.1834 so the first element is 1).
Any suggestions would by appreciated!

Output:
That's literally just your_tensor.argmax(dim=1).
your_tensor.argmax(dim=1).float() if you truly need it to be float.
After that, the accuracy can be calculated as sum(my_tensor == target_tensor) / len(target_tensor), for example. (See also this question).


Instruction: I use pytorch to train a model to classify iris, but my acc was about 0.4

Input:
I have tried many improvements like increasing epochs, using better loss functions and optimizers, deepening the network and shuffling the dataset, etc, but still to no avail. This problem has been bothering me for a long time, thanks for your help. Below is my code.
load and process dataset(updated)
def Iris_Reader(dataset):
    train_data, test_data, train_label, test_label = train_test_split(dataset.data, dataset.target, test_size=0.4)

    # scaler = StandardScaler()
    # train_data = scaler.fit_transform(train_data)
    # test_data = scaler.transform(test_data)

    return torch.FloatTensor(train_data), torch.LongTensor(train_label), torch.FloatTensor(test_data), torch.LongTensor(test_label)

Define the classifier
class Classifier(nn.Module):
    def __init__(self):
        super().__init__()

        #4*3*3 network
        self.model = nn.Sequential(
            nn.Linear(4,3),
            nn.ReLU(),

            nn.Linear(3,3),
        )

        #SGD
        self.optimiser = torch.optim.SGD(self.parameters(), lr = 0.1)

        #MSE LOSS_FUNCTION
        self.loss_fn = nn.CrossEntropyLoss()

        self.counter = 0
        self.progress = []

    def forward(self, input):
        return self.model(input)

    def train(self, input, target):
        output = self.forward(input)

        loss = self.loss_fn(output, target)

        self.counter += 1
        self.progress.append(loss.item())

        self.optimiser.zero_grad()
        loss.backward()
        self.optimiser.step()
    # plot loss
    def plot_loss(self):
        plt.figure(dpi=100)
        plt.ylim([0,1.0])
        plt.yticks([0, 0.25, 0.5, 1.0])
        plt.scatter(x = [i for i in range(len(self.progress))], y = self.progress, marker = '.', alpha = 0.2)
        plt.grid('on')
        plt.show()

TRAIN
C = Classifier()
epochs = 10
dataset = datasets.load_iris()

for epoch in range(epochs):
    train_data, train_label, _, _ = Iris_Reader(dataset)
    for i, j in zip(train_data, train_label):
        C.train(i, j)

TEST
score = 0
num = 0
# for epoch in range(epochs):
_, _, test_data, test_label = Iris_Reader(dataset)
for i,j in zip(test_data, test_label):
    output = C.forward(i).detach().argmax()
    if output == j:
        # print(C.forward(i).detach(), j)
        score += 1
    num += 1
print(score, num, round(score/num, 3))

OUTPUT: 53 60 0.883

Output:
There's a bunch of problems here:

First, you seem to shuffle data and labels independently, rendering the dataset useless.
Also, you recreate the dataset inside the loop every epoch, wasting the CPU time pointlessly.

Overall, the dataset creation can be shortened to something like this:
def Iris_Reader(dataset):
    train_data, test_data, train_label, test_label = sklearn.model_selection.train_test_split(dataset.data, dataset.target, test_size=0.2)
    return torch.FloatTensor(train_data), torch.LongTensor(train_label), torch.FloatTensor(test_data), torch.LongTensor(test_label)

and should be taken outside the loop.
Next, MSELoss() is suited for regression. For classification, CrossEntropyLoss() is the default choice.
Using sigmoid as activation in an intermediate layer is not the best choice, especially with a short number of epochs. ReLU should converge much better.
Last but not least, your loss chart would look much cleaner if the values were averaged per epoch.
Update: the implementation that ensures the target having the same size as network output, with additional feature scaling:
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

def Iris_Reader(dataset):
    label = nn.functional.one_hot(torch.LongTensor(dataset.target), num_classes=3).float()

    train_data, test_data,train_label, test_label = train_test_split(dataset.data, label, test_size=0.2)
    scaler = StandardScaler()
    train_data = scaler.fit_transform(train_data)
    test_data = scaler.transform(test_data)
    return torch.FloatTensor(train_data), train_label, torch.FloatTensor(test_data), test_label

Oh, and you should probably also remove the final Sigmoid() since CrossEntropyLoss() applies logsoftmax anyway.


Instruction: Import Error: Deploying custom Pytorch Model on streamlit

Input:
Please, so trying to deploy a custom Pytorch based web-app on streamlit, everything works locally, however when deployed, I find the following Error in the Logs:
Downloading: "https://github.com/ultralytics/yolov5/archive/master.zip" to /home/appuser/.cache/torch/hub/master.zip
2022-08-14 12:23:17.584 Uncaught app exception
Traceback (most recent call last):
  File "/home/appuser/venv/lib/python3.9/site-packages/streamlit/scriptrunner/script_runner.py", line 557, in _run_script
    exec(code, module.__dict__)
  File "app.py", line 28, in <module>
    model = torch.hub.load('ultralytics/yolov5', 'custom', path=run_model_path)
  File "/home/appuser/venv/lib/python3.9/site-packages/torch/hub.py", line 339, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/appuser/venv/lib/python3.9/site-packages/torch/hub.py", line 368, in _load_local
    model = entry(*args, **kwargs)
  File "/home/appuser/.cache/torch/hub/ultralytics_yolov5_master/hubconf.py", line 74, in custom
    return _create(path, autoshape=autoshape, verbose=_verbose, device=device)
  File "/home/appuser/.cache/torch/hub/ultralytics_yolov5_master/hubconf.py", line 31, in _create
    from models.common import AutoShape, DetectMultiBackend
  File "/home/appuser/.cache/torch/hub/ultralytics_yolov5_master/models/common.py", line 14, in <module>
    import cv2
  File "/home/appuser/venv/lib/python3.9/site-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

Here's what the requrements.txt contains:
pillow<=9.2.0
numpy<=1.21.0
streamlit==1.11.0
torch<=1.8.2+cu111
opencv<=4.5.1

Would love your assistance.

Output: If you're deploying the app, opencv-python-headless would be the appropriate package rather than opencv


Instruction: Why iterating over a Pytorch dataloader never ends and goes forever?

Input:
I have created a dataloader whose length is 50000. When I calculate its length it prints out 50000.
class MyDataLoader(torch.utils.data.Dataset):
    def __init__(self, data_size=50000):
        self.data_size = data_size

    def __len__(self) -> int:
        return self.data_size

    def __getitem__(self, idx) -> t.Tuple[torch.Tensor, torch.Tensor]:
        image, label = my_function()#(has_star=True)
        return image[None], label
dl = MyDataLoader()
print(len(dl))
50000

However, when I iterate over it, it goes forever like the following:
for j, i in enumerate(dl):
  if j%10000 == 0:
    print(j)
10000
20000
30000
40000
50000
60000
...

How is that possible?

Output:
You have created a Dataset, not a Dataloader.
This should work:
import torch
from torch.utils.data import DataLoader

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, data_size=50000):
        self.data_size = data_size

    def __len__(self) -> int:
        return self.data_size

    def __getitem__(self, idx):
        # print(idx)
        return idx

dataset = MyDataset()
# Assume a default batch size of 1
dl = DataLoader(dataset)
print(len(dl))

for j, i in enumerate(dl):
  if j%10000 == 0:
    print(j)

# And with a different batch size:
dl = DataLoader(dataset, batch_size=2)
print(len(dl))

for j, i in enumerate(dl):
  if j%10000 == 0:
    print(j)

Note how len(dl) changes when the batch size changes.


Instruction: How can we balance off the imbalance generated by the data loader for a binary classification problem

Input:
I have created a synthesized dataset and the corresponding data loader for a binary classification problem using Pytorch. The zero class happens almost 20% and the other one 80%. When I train my model it only predicts the 80% ones which makes sense because it has seen the one with label one 80% of the time.
How can I handle this imbalance after getting the data from the data loader?
Is BCELoss capable of understanding this situation?
import torch
from torch.utils.data import DataLoader

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, data_size=50000):
        self.data_size = data_size

    def __len__(self) -> int:
        return self.data_size

    def __getitem__(self, idx):
        data, label (label=0 (20%) or label=1 (80%)) = my_function()
        return data, label

dataset = MyDataset()
# Assume a default batch size of 1
batch_size = 1000
dl = DataLoader(dataset, batch_size=batch_size)

# network

loss_fn = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-6)

losses = []
for epoch in range(num_epochs):
    print(f"EPOCH: {epoch}")
       for data, label in tqdm(dl, total=len(dl)):
            data= data.to(DEVICE).float()
            label = label.to(DEVICE).float()
            optimizer.zero_grad()
            preds = model(data)
            prob = torch.sigmoid(preds)
            loss = loss_fn(prob, label)
            loss.backward()
            losses.append(loss.detach().cpu().numpy())
            optimizer.step()
            # break

Question:
I have read about the weight argument for BCELoss but I am not sure if I can use it here or not. To me, it does not handle imbalance in the data. Note that I cannot manipulate my dataset since in practice the percentage is inherit it the data and we cannot change it from source.

Output:
I'm not sure what you mean by "after getting the data from the data loader" but I'll suggest anyway that you could oversample the minority class by using a WeightedRandomSampler. This will make sure that the dataloader always returns the same amount of samples for each class. So there will be a 50/50 chance that it returns a sample of class 1 and 0. Here is how to do it:
import torch
from torch.utils.data import DataLoader, WeightedRandomSampler

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, data_size=50000):
        self.data_size = data_size

    def __len__(self) -> int:
        return self.data_size

    def __getitem__(self, idx):
        data, label (label=0 (20%) or label=1 (80%)) = my_function()
        return data, label

dataset = MyDataset()
# Assume a default batch size of 1
batch_size = 1000

class_weights = [1/20, 1/80] # inverse relative amount of samples per class
sample_weights = [0] * len(dataset)

for idx, (data, label) in enumerate(dataset):
    sample_weights[idx] = class_weights[label]

sampler = WeightedRandomSampler(sample_weights, num_samples=len(sample_weights), replacement=True)
dl = DataLoader(dataset, batch_size=batch_size, sampler=sampler)

# continue with your code...

And using a weighted loss can also improve the accuracy. You can simply do something like this:
weights = [1/20, 1/80] # or [4, 1] or [1/2, 1/4] you can try whatever as long as the proportions match the class distribution in your dataset
loss_fn = nn.BCELoss(weights=weights)

You can use both of the methods. And of course the most effective method: augmentation. If it's possible for you to synthetically create more samples from your dataset the do that. An example for augmentation with images is to flip, rotate, crop the images to create slighty different ones. You could add that to your dataset class and have a 80% that the minority class is augmented.


Instruction: Better understanding of what is used to feed YOLO

Input:
I'm trying to construct a YOLO dataset.
A read a lot about the dataset and understood many things :

pc is the confidence and it corresponds to the IoU between predicted and ground truth bboxes
there are C classes
there are 4 coordinates times the number of bounding boxes (here only 1).

What I do not understand is this quote drawn from the YOLO paper :

Our final layer predicts both class probabilities and bounding box coordinates. We normalize the bounding box width and height by the image width and height so that they fall between 0 and 1. We parametrize the bounding box x and y coordinates to be offsets of a particular grid cell location so they are also bounded between 0 and 1.

I construct my bbox coordinates as many website and annotation solutions suggest, it leads to : cx, cy, rh, rw which are respectively the bbox center coordinates and the bbox height and width relative to the image (normalized by the image size). But this method doesn't seems to rely on the paper's method since cx and cy are note relative to a grid cell here.
I read the Hackernoon article but I don't understand how he sets his x and y : he mentions x=(220-149)/149 = 0.48 where 149 is his grid cell size and 220 is the absolute x coordinate of the bbox. But if there are more cells and one split the image by 6 for instance : 447/6 = 75, then x=(220-75)/75 = 1.93 which is not a relative value (greater than 1)...
My questions :

Does that mean that I had to take into account the grid size (and so the grid cell sizes) when I create my dataset ?
Do I need to include a pc in my training set ? And so, cut the image into cells and return a n*n matrice with 0 and 1 as a pc number ?

Thanks !

Output:
I post here a summary of what has been said and give more context about my initial questions.
Bounding box coordinates
First of all, as mentionned by @MichelKok : the initial YOLO model uses offsets relative to grid cell size (at the contrary, YOLOv5 does not employ the grid structure anymore).
In YOLOv1, the bounding box regression is applied into a grid cell. Therefore input coordinates have to be relative to one grid cell . In other words, if an image is "split" into 3x3 grid cells, the first left cell is (0,0) and the botom right is (2,2).
One can implement this as folow :
cell_size = width/S #or height/S
cx_rcell = cx % cell_size / cell_size
cy_rcell = cy % cell_size / cell_size

Where :

cx and cx are respectively x and y coordinates of the center of the bounding box and are relative to the image (i.e. normalized by the with or height).
cell_size is the size of one grid cell, defined by the number of grid cell in one direction S

Finally, one can go back to the image relative coordinates :
cx = cx_rcell * cell_size - (1/cell_size) * int(cx_rcell/cell_size)
cy = cy_rcell * cell_size - (1/cell_size) * int(cy_rcell/cell_size)

YOLOv1 Loss function
There are 4 parts into this loss :

minimizing the squared error of cx and cy coordinates
minimizing the root squared error of w and h
minimizing the squared error of the confidence numbers
minimizing the squared error of the class probabilities

Note that this loss is constructed in such a way that it penalizes errors depending if an object appears in the current grid cell. As I understood it, this means that one need to deal with bounding box coordinate training set as tensors of shape (N, S, S, 5) in order to get the information "In which cell grid the center of this training bounding box is ?".
A simple implementation could be :
bbox_tensor = torch.zeros(N, S, S, 4)
bbox_tensor[int(cy_rcell / cell_size), int(cx_rcell / cell_size)] = torch.Tensor([cx_rcell , cy_rcell, w, h])

Here, the bounding box coordinates will be located in the cell (i,j) in S^2.


Instruction: Pytorch, Pandas, Numpy different result on Windows and Linux

Input:
We are working on an AI project which amongst others calculates the position of a human body lying in a bed. External supporters provided us a code package which does this job and calculates the deviation between the real position of the body (detected by a camera) and the predicted position.
That code package has been developed on Windows and now needs to be ported to Ubuntu 20.04. Until now, in the code package for Ubuntu, we just changed some path names (backslash to slash) to make it work.
If we now run – the exactly same - code with the exactly same training and test data under Windows 10 and Ubuntu we observe the following strange behaviour:
The deviation between real body positions and predicted body positions is 25mm (average value) in Windows, which is a good result for our purposes and excepted as a match. In Ubuntu the deviation is 150mm (average value), which is a not usable result.
We also ran the Windows code package in a Windows 10 VM (Virtualbox) on the respective Ubuntu machine. The result is close to the result we get when running the code under ‘plain‘ Ubuntu (not usable).
When running under Ubuntu, there is no difference whether we utilize CPU or GPU.
The code is written in Python 3 and executed under Python 3.7.12 in an Anaconda 3 virtual environment.
We assume that Linux/Ubuntu does not fully support a certain CPU instruction set. but at the end we do not have an explanation for this behaviour. Does anybody have an idea?
Our CPU:
Intel Core I7 (Caby Lake)
Our env.yaml for conda:
name: cosy-bunk-3.7
channels:
  - conda-forge
  - anaconda
  - defaults
  - pytorch
dependencies:
  - python=3.7
  - numpy
  - matplotlib
  - pandas
  - ipykernel
  - torchvision
  - pytorch
  - cudatoolkit=10.2
  - torchaudio
  - plotly
  - scikit-learn
  - pip
  - pip:
    - optuna
    - dash

Output of 'conda list‘ under Ubuntu:
    # Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
alembic                   1.8.1                    pypi_0    pypi
alsa-lib                  1.2.6.1              h7f98852_0    conda-forge
attr                      2.5.1                h166bdaf_0    conda-forge
attrs                     22.1.0                   pypi_0    pypi
autopage                  0.5.1                    pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
blas                      1.1                    openblas    conda-forge
brotli                    1.0.9                    pypi_0    pypi
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py37h540881e_1004    conda-forge
ca-certificates           2022.6.15            ha878542_0    conda-forge
certifi                   2022.6.15        py37h89c1867_0    conda-forge
cffi                      1.15.1           py37h43b0acd_0    conda-forge
charset-normalizer        2.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3                    pypi_0    pypi
cliff                     3.10.1                   pypi_0    pypi
cmaes                     0.8.2                    pypi_0    pypi
cmd2                      2.4.2                    pypi_0    pypi
colorlog                  6.6.0                    pypi_0    pypi
cryptography              37.0.4           py37h38fbfac_0    conda-forge
cudatoolkit               10.2.89             h713d32c_10    conda-forge
cudnn                     7.6.5.32             h01f27c4_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dash                      2.6.1                    pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.0            py37hd23a5d3_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
expat                     2.4.8                h27087fc_0    conda-forge
fftw                      3.3.10          nompi_ha7695d1_103    conda-forge
flask                     2.2.2                    pypi_0    pypi
flask-compress            1.12                     pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.34.4           py37h540881e_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glib                      2.72.1               h6239696_0    conda-forge
glib-tools                2.72.1               h6239696_0    conda-forge
greenlet                  1.1.2                    pypi_0    pypi
gst-plugins-base          1.20.3               hf6a322e_0    conda-forge
gstreamer                 1.20.3               hd4edc92_0    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.12.0                   pypi_0    pypi
importlib-resources       5.9.0                    pypi_0    pypi
ipykernel                 6.15.1             pyh210e3f2_0    conda-forge
ipython                   7.33.0           py37h89c1867_0    conda-forge
itsdangerous              2.1.2                    pypi_0    pypi
jack                      1.9.18            h8c3723f_1002    conda-forge
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h89c1867_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           15_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcap                    2.64                 ha37c62d_0    conda-forge
libcblas                  3.9.0           15_linux64_openblas    conda-forge
libclang                  14.0.6          default_h2e3cab8_0    conda-forge
libclang13                14.0.6          default_h3a83d3e_0    conda-forge
libcups                   2.3.3                hf5a7f15_1    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.3.4                h27087fc_0    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libglib                   2.72.1               h2d90d5f_0    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           15_linux64_openblas    conda-forge
libllvm14                 14.0.6               he0ac6c6_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h753d276_3    conda-forge
libpq                     14.4                 hd77ab85_0    conda-forge
libprotobuf               3.20.1               h6239696_0    conda-forge
libsndfile                1.0.31               h9c3ff4c_1    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libtool                   2.4.6             h9c3ff4c_1008    conda-forge
libudev1                  249                  h166bdaf_4    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.2.3                h522a892_1    conda-forge
libwebp-base              1.2.3                h166bdaf_2    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.14               h22db469_3    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
llvm-openmp               14.0.4               he0ac6c6_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
magma                     2.5.4                h5da55e3_2    conda-forge
mako                      1.2.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.2            py37h89c1867_1    conda-forge
matplotlib-base           3.5.2            py37hc347a89_1    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mkl                       2022.1.0           h84fe81f_915    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.30               haf5c9bc_0    conda-forge
mysql-libs                8.0.30               h28c427c_0    conda-forge
nccl                      2.13.4.1             h1a5f58c_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numpy                     1.21.6           py37h976b520_0    conda-forge
openblas                  0.3.20          pthreads_h320a7e8_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1q               h166bdaf_0    conda-forge
optuna                    2.10.1                   pypi_0    pypi
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pbr                       5.9.0                    pypi_0    pypi
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py37h44f0d7a_0    conda-forge
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
plotly                    5.9.0              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
portaudio                 19.6.0               h57a0ea0_5    conda-forge
prettytable               3.3.0                    pypi_0    pypi
prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
psutil                    5.9.1            py37h540881e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio                14.0                 h7f54b18_8    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyperclip                 1.8.2                    pypi_0    pypi
pyqt                      5.15.7           py37hf30b843_0    conda-forge
pyqt5-sip                 12.11.0          py37hd23a5d3_0    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytorch                   1.12.0          cuda102py37haad9b4f_202    conda-forge
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.2.0           py37h0c0c2a8_0    conda-forge
qt-main                   5.15.4               ha5833f6_2    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
scikit-learn              1.0.2            py37hf9e9bfc_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
setuptools                59.8.0           py37h89c1867_1    conda-forge
sip                       6.6.2            py37hd23a5d3_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
sqlalchemy                1.4.40                   pypi_0    pypi
sqlite                    3.39.2               h4ff8645_0    conda-forge
stevedore                 3.5.0                    pypi_0    pypi
tbb                       2021.5.0             h924138e_1    conda-forge
tenacity                  8.0.1              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
torchaudio                0.12.0               py37_cu102    pytorch
torchvision               0.13.0          cuda102py37h9785060_0    conda-forge
tornado                   6.2              py37h540881e_0    conda-forge
tqdm                      4.64.0                   pypi_0    pypi
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.3.0                hd8ed1ab_0    conda-forge
typing_extensions         4.3.0              pyha770c72_0    conda-forge
unicodedata2              14.0.0           py37h540881e_1    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xcb-util                  0.4.0                h166bdaf_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h166bdaf_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h166bdaf_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h8a70e8d_3    conda-forge

Output of 'conda list‘ under Windows:
# Name                    Version                   Build  Channel
alembic                   1.8.1                    pypi_0    pypi
attrs                     22.1.0                   pypi_0    pypi
autopage                  0.5.1                    pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
blas                      1.0                         mkl    anaconda
brotli                    1.0.9                    pypi_0    pypi
brotli-bin                1.0.9                h8ffe710_7    conda-forge
ca-certificates           2022.6.15            h5b45459_0    conda-forge
certifi                   2022.6.15        py37h03978a9_0    conda-forge
cffi                      1.15.1           py37hd8e9650_0    conda-forge
click                     8.1.3                    pypi_0    pypi
cliff                     3.10.1                   pypi_0    pypi
cmaes                     0.8.2                    pypi_0    pypi
cmd2                      2.4.2                    pypi_0    pypi
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
colorlog                  6.6.0                    pypi_0    pypi
cudatoolkit               11.3.1              h280eb24_10    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dash                      2.6.1                    pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
debugpy                   1.6.0            py37hf2a7229_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
flask                     2.2.2                    pypi_0    pypi
flask-compress            1.12                     pypi_0    pypi
fonttools                 4.34.4           py37hcc03f2d_0    conda-forge
freetype                  2.10.4               h546665d_1    conda-forge
future                    0.18.2           py37h03978a9_5    conda-forge
greenlet                  1.1.2                    pypi_0    pypi
icu                       58.2             vc14hc45fdbb_0  [vc14]  anaconda
importlib-metadata        4.12.0                   pypi_0    pypi
importlib-resources       5.9.0                    pypi_0    pypi
intel-openmp              2022.1.0          h57928b3_3787    conda-forge
ipykernel                 6.15.1             pyh025b116_0    conda-forge
ipython                   7.33.0           py37h03978a9_0    conda-forge
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.1             pyhd8ed1ab_2    conda-forge
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9b               vc14h4d7706e_1  [vc14]  anaconda
jupyter_client            7.3.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h03978a9_0    conda-forge
kiwisolver                1.4.4            py37h8c56517_0    conda-forge
libblas                   3.9.0              12_win64_mkl    conda-forge
libbrotlicommon           1.0.9                h8ffe710_7    conda-forge
libbrotlidec              1.0.9                h8ffe710_7    conda-forge
libbrotlienc              1.0.9                h8ffe710_7    conda-forge
libcblas                  3.9.0              12_win64_mkl    conda-forge
liblapack                 3.9.0              12_win64_mkl    conda-forge
libpng                    1.6.37               h1d00b33_2    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libsqlite                 3.39.2               h8ffe710_1    conda-forge
libtiff                   4.0.8           vc14h04e2a1e_10  [vc14]  anaconda
libuv                     1.44.2               h8ffe710_0    conda-forge
libwebp                   1.2.4                h8ffe710_0    conda-forge
libwebp-base              1.2.4                h8ffe710_0    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mako                      1.2.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.3            py37h03978a9_0    conda-forge
matplotlib-base           3.5.3            py37h54234da_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mkl                       2021.4.0           h0e2418a_729    conda-forge
mkl-service               2.4.0            py37h75fcce0_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
ninja                     1.11.0               h2d74725_0    conda-forge
numpy                     1.21.6           py37h2830a78_0    conda-forge
openssl                   1.1.1q               h8ffe710_0    conda-forge
optuna                    2.10.1                   pypi_0    pypi
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37h9386db6_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pbr                       5.10.0                   pypi_0    pypi
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py37hdc2b20a_1
pip                       22.2.2             pyhd8ed1ab_0    conda-forge
plotly                    5.10.0             pyhd8ed1ab_0    conda-forge
prettytable               3.3.0                    pypi_0    pypi
prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
psutil                    5.9.1            py37hcc03f2d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyperclip                 1.8.2                    pypi_0    pypi
pyqt                      5.9.2            py37h6538335_4    conda-forge
pyreadline3               3.4.1                    pypi_0    pypi
python                    3.7.12          h7840368_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytorch                   1.10.2          cpu_py37h907fbb5_0    anaconda
pytz                      2022.2             pyhd8ed1ab_0    conda-forge
pywin32                   303              py37hcc03f2d_0    conda-forge
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.2.0           py37hcce574b_0    conda-forge
qt                        5.9.7            vc14h73c81de_0  [vc14]  anaconda
scikit-learn              1.0.2            py37hcabfae0_0    conda-forge
scipy                     1.7.3            py37hb6553fb_0    conda-forge
setuptools                59.8.0           py37h03978a9_1    conda-forge
sip                       4.19.8          py37h6538335_1000    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlalchemy                1.4.40                   pypi_0    pypi
sqlite                    3.39.2               h8ffe710_1    conda-forge
stevedore                 3.5.0                    pypi_0    pypi
tbb                       2021.5.0             h2d74725_1    conda-forge
tenacity                  8.0.1              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.7            vc14hb68737d_1  [vc14]  anaconda
torchaudio                0.10.2               py37_cu113    pytorch
torchvision               0.11.3               py37_cu113    pytorch
tornado                   6.2              py37hcc03f2d_0    conda-forge
tqdm                      4.64.0                   pypi_0    pypi
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.3.0                hd8ed1ab_0    conda-forge
typing_extensions         4.3.0              pyha770c72_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
unicodedata2              14.0.0           py37hcc03f2d_1    conda-forge
vc                        14.2                 hb210afc_6    conda-forge
vs2015_runtime            14.29.30037          h902a5da_6    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.11           vc14h1cdd9ab_1  [vc14]  anaconda

Output: After deeper research we discovered the error in the prepairing of the data caused by the different handling of os.listdir() in python. The order is not the same on windows and linux machines.


Instruction: Class can't read its attribute

Input:
I following this pytorch tutorial
On the basis of this link I wrote the following code:
import torch
import torchvision
from torch.utils.data import Dataset,DataLoader
import numpy as np
import math
class WineDataset(Dataset):
    def __int__(self):
        xy =np.loadtxt("wine.csv",delimiter=',',dtype=np.float32,skiprows=1)
        self.n_samples =xy.shape[0]
        self.x_data = torch.from_numpy(xy[:, 1:])
        self.y_data = torch.from_numpy(xy[:, [0]])
    def __getitem__(self, index):
        return self.x_data[index],self.y_data[index]
    def __len__(self):
        return self.n_samples
# create dataset
dataset = WineDataset()
# get first sample and unpack
first_data = dataset[0]
features, labels = first_data
print(features, labels)

But when I run the code, it gives me following error:
Traceback (most recent call last):
  File "C:\Users\User\PycharmProjects\AI_Project\Dataset_DataLoader.py", line 19, in <module>
    first_data = dataset.__getitem__(0)
  File "C:\Users\User\PycharmProjects\AI_Project\Dataset_DataLoader.py", line 13, in __getitem__
    return self.x_data[index],self.y_data[index]
AttributeError: 'WineDataset' object has no attribute 'x_data'

This line is also  written on GitHub:
# create dataset
dataset = WineDataset()

# get first sample and unpack
first_data = dataset[0]
features, labels = first_data
print(features, labels)

Where is the mistake?

Output: Change def __int__(self) to def __init__(self). Right now you have no constructor.


Instruction: What's the fastest way to slice a portion of a tensor to another in PyTorch?

Input:
I have three tensors as shown below:
a = tensor([[5, 2, 3, 24],
        [8, 66, 7, 89],
        [9, 10, 1, 12]])

b = tensor([[10, 22, 13, 1],
            [35, 6, 17, 3],
            [11, 13, 5,8]])

c = tensor([[0, 0, 0, 0],
                [0, 0, 0, 0],
                [0, 0, 0,0]])

I want to chnage c values using this formula:
Note that the last values (column) in c are not changed at this point.
c[:,:-1]= a[:,:-1] -a[:, 1:] - b[:, 1:]

This means I will have
c = tensor([[5-2-22, 2-3-13, 3-24-1, 0],
            [8-66-6, 66-7-17, 7-89-3, 0],
            [9-10-13, 10-1-5, 1-12-8,0]])

>>>  c = tensor([[-19, -14, -22, 0],
                 [-64, 42, -85, 0],
                 [-14, 4, -19,0]])

Finally, to change the last column, I want to use c[:, -1] = b[:, -1] - 1
And my final tensor will look like this:
c = tensor([[-19, -14, -22, 0],
            [-64, 42, -85, 2],
            [-14, 4, -19,7]])

Output:
I think the fact that you are "overriding" the elements of c is causing you errors.
Try creating c "from scratch", by concatenating its two parts:
c = torch.cat([a[:,:-1] -a[:, 1:] - b[:, 1:],
               b[:, -1:] - 1], dim=-1)


Instruction: Speeding up a pytorch tensor operation

Input:
I am trying to speed up the below operation by doing some sort of matrix/vector-multiplication, can anyone see a nice quick solution?
It should also work for a special case where a tensor has shape 0 (torch.Size([])) but i am not able to initialize such a tensor.
See the image below for the type of tensor i am referring to:
tensor to add to test
def adstock_geometric(x: torch.Tensor, theta: float):
    x_decayed = torch.zeros_like(x)
    x_decayed[0] = x[0]

    for xi in range(1, len(x_decayed)):
        x_decayed[xi] = x[xi] + theta * x_decayed[xi - 1]

    return x_decayed

def adstock_multiple_samples(x: torch.Tensor, theta: torch.Tensor):

    listtheta = theta.tolist()
    if isinstance(listtheta, float):
        return adstock_geometric(x=x,
                                 theta=theta)
    x_decayed = torch.zeros((100, 112, 1))
    for idx, theta_ in enumerate(listtheta):
        x_decayed_one_entry = adstock_geometric(x=x,
                                                theta=theta_)
        x_decayed[idx] = x_decayed_one_entry
    return x_decayed

if __name__ == '__main__':
    ones = torch.tensor([1])
    hundreds = torch.tensor([idx for idx in range(100)])
    x = torch.tensor([[idx] for idx in range(112)])
    ones = adstock_multiple_samples(x=x,
                                    theta=ones)
    hundreds = adstock_multiple_samples(x=x,
                                        theta=hundreds)
    print(ones)
    print(hundreds)

Output:
I came up with the following, which is 40 times faster on your example:
import torch

def adstock_multiple_samples(x: torch.Tensor, theta: torch.Tensor):
    arange = torch.arange(len(x))
    powers = (arange[:, None] - arange).clip(0)
    return ((theta[:, None, None] ** powers[None, :, :]).tril() * x).sum(-1)

It behaves as expected:
>>> x = torch.arange(112)
>>> theta = torch.arange(100)
>>> adstock_multiple_samples(x, theta)
... # the same output

Note that I considered that x was a 1D-tensor, as for your example the second dimension was not needed.
It also works with theta = torch.empty((0,)), and it returns an empty tensor.