Instruction: how does one fix when torch can't find cuda, error: version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference?

Input:
I get this error with a pytorch import python -c "import torch":
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/afs/cs.stanford.edu/u/brando9/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 13, in <module>
    import torch
  File "/dfs/scratch0/brando9/miniconda/envs/metalearning_gpu/lib/python3.9/site-packages/torch/__init__.py", line 191, in <module>
    _load_global_deps()
  File "/dfs/scratch0/brando9/miniconda/envs/metalearning_gpu/lib/python3.9/site-packages/torch/__init__.py", line 153, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/dfs/scratch0/brando9/miniconda/envs/metalearning_gpu/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /dfs/scratch0/brando9/miniconda/envs/metalearning_gpu/lib/python3.9/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11: symbol cublasLtHSHMatmulAlgoInit, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

how does one fix it?
related:

Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
https://github.com/pytorch/pytorch/issues/51080

Output:
I don't know why this works but this worked for me:
source cuda11.1
# To see Cuda version in use
nvcc -V
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

but if you look through the git issue these might also work:
conda install -y -c pytorch -c conda-forge cudatoolkit=11.1 pytorch torchvision torchaudio

pip3 install torch+cu111 torchvision torchaudio -f https://download.pytorch.org/whl/torch_stable.html

I think the conda one looks like the most robust because you can specify exactly the cudatoolkit you need, so I'd recommend that one.


Instruction: Pytorch inverse FFT (ifft) with amplitude and phase

Input:
To use the amplitude and phase, I use the below function to extract:
def calculate_fft(x):
    fft_im = torch.fft.fft(x.clone())  # bx3xhxw
    fft_amp = fft_im.real**2 + fft_im.imag**2
    fft_amp = torch.sqrt(fft_amp)
    fft_pha = torch.atan2(fft_im.imag, fft_im.real)
    return fft_amp, fft_pha

After I modifying the amp and pha, how to use them to perform inverse FFT?
y = fft_amp * torch.sin(fft_pha)

This does not work. I am not good at maths. :-(

Output:
def inverse_fft(fft_amp, fft_pha):
    imag = fft_amp * torch.sin(fft_pha)
    real = fft_amp * torch.cos(fft_pha)
    fft_y = torch.complex(real, imag)
    y = torch.fft.ifft(fft_y)
    return y

This may work.


Instruction: Convolutional layer: does the filter convolves also trough the nlayers_in or it take all the dimensions?

Input:
In the leading DeepLearning libraries, does the filter (aka kernel or weight) in the convolutional layer convolves also across the "channel" dimension or does it take all the channels at once?
To make an example, if the input dimension is (60,60,10) (where the last dimension is often referred as "channels") and the desired output number of channels is 5, can the filter be (5,5,5,5) or should it be (5,5,10,5) instead ?

Output:
It should be (5, 5, 10, 5). Conv2d operation is just like Linear if you ignore the spatial dimensions.
From TensorFlow documentation [link]:

Given an input tensor of shape batch_shape + [in_height, in_width, in_channels] and a filter / kernel tensor of shape [filter_height, filter_width, in_channels, out_channels], this op performs the following:

Flattens the filter to a 2-D matrix with shape [filter_height * filter_width * in_channels, output_channels].
Extracts image patches from the input tensor to form a virtual tensor of shape [batch, out_height, out_width, filter_height * filter_width * in_channels].
For each patch, right-multiplies the filter matrix and the image patch vector.


Instruction: optimizer got an empty parameter list while using adam

Input:
I was creating conv nets for cifar100 and code is give below but I have encounted the error mentioned in the title while initializing optimizer.
Code for model
class CNN(nn.Module):
  def __init__(self,k):
    super(CNN,self).__init__()
    #Convolutional layer
    conv1=nn.Conv2d(3,32,kernel_size=3,padding='same')
    conv2=nn.Conv2d(32,64,kernel_size=3,padding='same')
    conv3=nn.Conv2d(64,128,kernel_size=3,padding='same')
    conv4=nn.Conv2d(128,128,kernel_size=3,padding='same')
    conv5=nn.Conv2d(128,256,kernel_size=3,padding='same')
    conv6=nn.Conv2d(256,256,kernel_size=3,padding='same')
    #feddfordward layer
    f1=nn.Linear(4*4*256,4096)
    f2=nn.Linear(4096,k)
  def forward(self,x):
    x=f.relu(self.conv1(x))
    x=f.relu(self.conv2(x))
    x=f.max_pool2d(x,kernel_size=2,stride=2)
    x=f.relu(self.conv3(x))
    x=f.relu(self.conv4(x))
    x=f.max_pool2d(x,kernel_size=2,stride=2)
    x=f.relu(self.conv5(x))
    x=f.relu(self.conv6(x))
    x=f.max_pool2d(x,kernel_size=2,stride=2)
    x=x.view(-1,4*4*256)
    x=f.relu(self.f1(x))
    x=self.f2(x)
    return x

Model initialization
model=CNN(k)

device=th.device("cuda:0" if th.cuda.is_available() else "cpu")
print(device)
model.to(device)

lossFunction=th.nn.CrossEntropyLoss()
optimizer=th.optim.Adam(model.parameters())
epoch=100

I tried changing maxpool2d function but it didnot help. I am new to pytorch and any help will be appreciated

Output:
Your submodules must be registered as attributes of your parent nn.Module:
class CNN(nn.Module):
    def __init__(self,k):
        super(CNN,self).__init__()
        # convolutional layer
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding='same')
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding='same')
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding='same')
        self.conv4 = nn.Conv2d(128, 128, kernel_size=3, padding='same')
        self.conv5 = nn.Conv2d(128, 256, kernel_size=3, padding='same')
        self.conv6 = nn.Conv2d(256, 256, kernel_size=3, padding='same')

        # feedfordward layer
        self.f1 = nn.Linear(4*4*256, 4096)
        self.f2 = nn.Linear(4096, k)


Instruction: Cannot run the following code in Python terminal

Input:
From the following page (https://github.com/Babelscape/rebel/blob/main/setup.sh) I got the following code.
#!/bin/bash

# setup conda
source ~/miniconda3/etc/profile.d/conda.sh

# create conda env
read -rp "Enter environment name: " env_name
read -rp "Enter python version (e.g. 3.7) " python_version
conda create -yn "$env_name" python="$python_version"
conda activate "$env_name"

# install torch
read -rp "Enter cuda version (e.g. 10.1 or none to avoid installing cuda support): " cuda_version
if [ "$cuda_version" == "none" ]; then
    conda install -y pytorch torchvision cpuonly -c pytorch
else
    conda install -y pytorch torchvision cudatoolkit=$cuda_version -c pytorch -c conda-forge
fi

# install python requirements
pip install -r requirements.txt

Does anyone know what can be done to process this code effectively?
Thank you in advance.

Output: This is a Bash script. You only can run it in a Bash Shell.


Instruction: Is it safe to do a ToTensor() data transform before a ColorJitter and RandomHorizotnalFlip and other data augmentation techniques in PyTorch?

Input:
I am seeing that the ToTensor is always at the end https://github.com/pytorch/examples/blob/main/imagenet/main.py:
        train_dataset = datasets.ImageFolder(
            traindir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))

        val_dataset = datasets.ImageFolder(
            valdir,
            transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,
            ]))

other for mini-imagenet:
        train_data_transform = Compose([
            # ToPILImage(),
            RandomCrop(size, padding=8),
            # decided 8 due to https://datascience.stackexchange.com/questions/116201/when-to-use-padding-when-randomly-cropping-images-in-deep-learning
            ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4),
            RandomHorizontalFlip(),
            ToTensor(),
            normalize,
        ])
        test_data_transform = transforms.Compose([
            transforms.Resize((size, size)),
            transforms.ToTensor(),
            transforms.Normalize(mean=mean,
                                 std=std),
        ])
        validation_data_transform = test_data_transform

is there a reason for this? What happens if we do the ToTensor (or ToTensor and Normalize) first?

Output:
One reason why people usually use PIL format instead of tensor to perform image preprocessing is related to the resizing algorithms and especially downsizing. As pointed out in this paper,

The Lanczos, bicubic, and bilinear implementations by PIL (top row) adjust the antialiasing filter width by the downsampling factor (marked as ). Other implementations (including those used for PyTorch-FID and TensorFlow-FID) use fixed filter widths, introducing aliasing artifacts and resemble naive nearest subsampling.

image source: clean-fid official github repo
Despite that torchvision.transforms.Resize now supports antialias option, as per the documentation, it is only reducing the gap between the tensor outputs and PIL outputs to some extent. Another reason may be related to the reproducibility and portability, i.e. to have a universal preprocessing pipeline.


Instruction: The saved information in the .json file of the torch profiler

Input:
I have profiled my model using the torch profiler:
from torch.profiler import profile, record_function, ProfilerActivity
import torch
with profile(activities=[
    ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    with record_function("model_inference"):
        model(input)
profiling_results.export_chrome_trace("resnet18_trace_cuda.json")

I am not familiar with the terms that are saved in the .json file.
I appreciate it if I know each of the terms:
 "traceEvents": [
  {
    "ph": "X", "cat": "cpu_op",
    "name": "aten::zeros", "pid": 867040, "tid": 867040,
    "ts": 1668551166936236, "dur": 23,
    "args": {
      "Trace name": "PyTorch Profiler", "Trace iteration": 0,
      "External id": 376,
      "Input Dims": [[], [], [], [], []], "Input type": ["", "Scalar", "", "", "Scalar"]
    }
  },...

Thanks for the help!

Output:
You can open the json with chrome. Maybe this will help you to understand the data.
From the documentation:

The checkpoint can be later loaded and inspected under chrome://tracing URL.


Instruction: FileNotFoundError: scipy.libs

Input:
I'm trying to build an exe file using cx_Freeze.
but when I run the resulting file I get an error:

FileNotFoundError: ..\build\exe.win-amd64-3.8\lib\scipy.libs

please tell me how to fix this problem?
I run the following code:
from cx_Freeze import setup, Executable

build_exe_options = {"packages": ["torch", 'tensorflow']}

target = Executable(
    script='sub.py'
)
setup(
    name='my',
    options={'build_exe': build_exe_options},
    executables=[target]
)

Output:
I had this exact problem, this is only a short term fix but if you search for 'scipy.libs' in your python install location 'site-packages' folder (or virtual environment if you're using one) and copy/paste it into the libs folder in your build it should solve the issue.
I'll edit my answer if I come across the root cause and a more permanent fix...
Hope this helps!


Instruction: TypeError: __init__() got multiple values for argument 'dim'

Input:
I am doing testing on two trained models. In first, I am getting below error during testing so I have changed torch.logsoftmax class to nn.LogSoftmax.
Code
from torch.utils.data import Dataset, DataLoader
import pandas as pd
from torchvision import transforms
from PIL import Image
import torch
import torch.nn as nn
from glob import glob
from pathlib import PurePath
import numpy as np
import timm
import torchvision
import time

img_list = glob('/media/cvpr/CM_22/OOD-CV-phase2/phase2-cls/images/*.jpg')

name_list = [
    'aeroplane',
    'bicycle',
    'boat',
    'bus',
    'car',
    'chair',
    'diningtable',
    'motorbike',
    'sofa',
    'train'
]

# conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=10.2 -c pytorch

class PoseData(Dataset):
    def __init__(self, transforms) -> None:
        """
        the data folder should look like
        - datafolder
            - Images
            - labels.csv
        """
        super().__init__()
        self.img_list = glob('/media/cvpr/CM_22/OOD-CV-phase2/phase2-cls/images/*.jpg')
        self.img_list = sorted(self.img_list, key=lambda x: eval(PurePath(x).parts[-1][:-4]))
        self.trs = transforms

    def __len__(self):
        return len(self.img_list)

    def __getitem__(self, index):
        image_dir = self.img_list[index]
        image_name = PurePath(image_dir).parts[-1]
        image = Image.open(image_dir)
        image = self.trs(image)

        return image, image_name

if __name__ == "__main__":
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])
    tfs = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
        normalize,
    ])

    model1 = timm.models.swin_base_patch4_window7_224(pretrained=False, num_classes=15)
    model1 = torch.nn.DataParallel(model1)
    model1.load_state_dict(torch.load('/media/cvpr/CM_22/OOD_CV/swin15_best.pth.tar')['state_dict'],strict=False)
    model1 = model1.cuda()
    model1.eval()

    model2 = timm.models.convnext_base(pretrained=False, num_classes=15)
    model2 = torch.nn.DataParallel(model2)
    model2.load_state_dict(torch.load('convnext15_best.pth.tar')['state_dict'],strict=False)
    model2 = model2.cuda()
    model2.eval()

    dataset = PoseData(tfs)
    loader = DataLoader(dataset, batch_size=128, shuffle=False, drop_last=False, num_workers=4)

    image_dir = []
    preds = []
    for image, pth in loader:
        image_dir.append(list(pth))
        image = image.cuda()

        with torch.no_grad():

            model1.eval()
            pred1 = model1(image)
            model2.eval()
            pred2 = model2(image)

            entropy1 = -torch.sum(torch.softmax(pred1[:, :10], dim=1) * nn.LogSoftmax(pred1[:, :10], dim=1), dim=-1,
                                  keep_dim=True)
            entropy2 = -torch.sum(torch.softmax(pred2[:, :10], dim=1) * nn.LogSoftmax(pred2[:, :10], dim=1), dim=-1,
                                  keep_dim=True)
            entropy = entropy1 + entropy2

            pred = torch.softmax(pred1[:, :10], dim=1) * (entropy - entropy1) / entropy + torch.softmax(pred2[:, :10],
                                                                                dim=1) * (
                               entropy - entropy2) / entropy
            pred = torch.argmax(pred[:, :10], dim=1)
            p = []
            for i in range(pred.size(0)):
                p.append(name_list[pred[i].item()])
        p = np.array(p)
        preds.append(p)
        print(len(np.concatenate(preds)))

    image_dir = np.array(sum(image_dir, []))
    preds = np.concatenate(preds)

    csv = {'imgs': np.array(image_dir), 'pred': np.array(preds),
           }
    csv = pd.DataFrame(csv)
    print(csv)

    csv.to_csv('results.csv', index=False)

Traceback
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "/media/cvpr/CM_22/OOD_CV/test.py", line 93, in <module>
    entropy1 = -torch.sum(torch.softmax(pred1[:, :10], dim=1) * torch.logsoftmax(pred1[:, :10], dim=1), dim=-1,
AttributeError: module 'torch' has no attribute 'logsoftmax'

Due to PyTorch version conflict, I have replaced with recent PyTorch version but now getting dim error
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "/media/cvpr/CM_22/OOD_CV/test.py", line 93, in <module>
    entropy1 = -torch.sum(torch.softmax(pred1[:, :10], dim=1) * nn.LogSoftmax(pred1[:, :10], dim=1), dim=-1,
TypeError: __init__() got multiple values for argument 'dim'

After implementing
nn.LogSoftMax(dim=1)(pred1[:, :10])

Traceback
    entropy1 = -torch.sum(torch.softmax(pred1[:, :10], dim=1) * nn.LogSoftmax(dim=1)(pred1[:, :10]), dim=-1, keep_dim=True)
TypeError: sum() received an invalid combination of arguments - got (Tensor, keep_dim=bool, dim=int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
      didn't match because some of the keywords were incorrect: keep_dim, dim
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)

Then delete keep_dim=True parameter
Traceback
Traceback (most recent call last):
  File "/media/cvpr/CM_22/OOD_CV/test.py", line 97, in <module>
    pred = torch.softmax(pred1[:, :10], dim=1) * (entropy - entropy1) / entropy + torch.softmax(pred2[:, :10],
RuntimeError: The size of tensor a (10) must match the size of tensor b (128) at non-singleton dimension 1

Output:
nn.LogSoftMax is a module that has to be instantiated first and then called (which is when its forward method is executed). Try this instead:
entropy1 = -torch.sum(torch.softmax(pred1[:, :10], dim=1) * nn.LogSoftmax(dim=1)(pred1[:, :10]), dim=-1, keepdim=True)

Instead, you can also use the functional form of this module:
torch.nn.functional.log_softmax(pred1[:, :10], dim=1)


Instruction: How to train real-time LSTM with input and output of varying length?

Input:
I have the data of banner clicks by minute.
I have the following data: hour, minute, and was the banner clicked by someone in that minute. There are some other features (I omitted them in the example dataframe). I need to predict will be any clicks on banner for all following minutes of this hour.
For example I have data for the first 11 minutes of an hour.

hour
minute
is_click

1
1
0

1
2
0

1
3
1

1
4
0

1
5
1

1
6
0

1
7
0

1
8
0

1
9
1

1
10
1

1
11
0

My goal is to make prediction for 12, 13  ... 59, 60 minute.
It will be real-time model that makes predictions every minute using the latest data.
For example, I made the prediction at 18:00 for the next 59 minutes (until 18:59). Now it is 18:01 and I get the real data about  clicks at 18:00, so I want to make more precise prediction for following 58 minutes (from 18:02 to 18:59). And so on.
My idea was to mask-out the passed minutes with -1
I created the example of 11 minutes.There are targets:

minute
target vector

1
-1  0  1  0  1  0  0  0  1  1  0

2
-1 -1 1 0 1 0 0 0 1 1 0

3
-1 -1 -1 0 1 0 0 0 1 1 0

4
-1 -1 -1 -1 1 0 0 0 1 1 0

5
-1 -1 -1 -1 -1 0 0 0 1 1 0

6
-1 -1 -1 -1 -1 -1 0 0 1 1 0

7
-1 -1 -1 -1 -1 -1 -1 0 1 1 0

8
-1 -1 -1 -1 -1 -1 -1 -1 1 1 0

9
-1 -1 -1 -1 -1 -1 -1 -1 -1 1 0

10
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 0

11
-1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1

However it seems strange to me to train a model to predict this mask values of -1. I think for neural network it will be not obvious that these -1 are just a padding.
The another idea was to use a current minute as a feature and ,therefore, to predict always the sequence of 60 - minute length and then cut the extra prediction. However, the input will have  different lengths anyway, so it does not solve the problem.
So how I should preprocess the data to use LSTM? Should I use described above padding so all vectors will be have the same length of 60? Is there any better solution?

Output:
An RNN (or LSTM) will return an output for every input, as well as the final hidden state (and cell state for LSTM). So one possible solution: Pad your input of future minutes with with a different token and use an embedding layer with 3 embeddings (0, 1, 2 where 2 represents unseen value). For example, at timestep 3 the input = [0, 0, 1, 2, 2, 2,...2].
After this goes through an embedding layer each token will mapped to some embedding dimension (e.g. 16) and this would be pass to the LSTM. So the input size for your LSTM would be 16 and the hidden size would be one (so that you get a scalar output for every timestep of the input). Then you pass this output through a sigmoid so each prediction is between 0 and 1 and use binary cross entropy between the predictions and targets as your loss function. Additionally, since you probably don't care how accurate the predictions are for the minutes you've already seen, you could ignore their contribution to the loss.


Instruction: Cannot load Pytorch model in Pytorch mobile

Input:
I attempt to load a model into an android app using the pytorch mobile flutter plugin which is just a wrapper for the android pytorch package, but when I attempt to load the model I get the following debug statement
E/PyTorchMobile( 8111): assets/models/fullap.pt is not a proper model
E/PyTorchMobile( 8111): com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl
E/PyTorchMobile( 8111): (no backtrace available)
E/PyTorchMobile( 8111):     at org.pytorch.NativePeer.initHybrid(Native Method)
E/PyTorchMobile( 8111):     at org.pytorch.NativePeer.<init>(NativePeer.java:25)
E/PyTorchMobile( 8111):     at org.pytorch.Module.load(Module.java:25)
E/PyTorchMobile( 8111):     at org.pytorch.Module.load(Module.java:35)
E/PyTorchMobile( 8111):     at io.fynn.pytorch_mobile.PyTorchMobilePlugin.onMethodCall(PyTorchMobilePlugin.java:58)
E/PyTorchMobile( 8111):     at io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:262)

I saved the model using torch.save(model, path) after loading it from a state dict. I also tried to resave the model. Any ideas as to what the problem may be? the model is not quantized as of right now.

Output:
file not found: archive/constants.pkl

Make sure you've set the right path of the model.


Instruction: About roboflown's assist function

Input:
Annotated in roboflow and trained in YOLOv5(https://github.com/ultralytics/yolov5).
I annotated the model in roboflow and trained it in YOLOv5.
After that, I want to do label assist using the custom model I learned, but I can't do it. What should I do?

Output:
Use the dataset version you generated to train in YOLOv5 to use Roboflow Train: https://docs.roboflow.com/train - once your Training Job is complete, you’ll have access to Model-Assisted Labeling (Label Assist): https://docs.roboflow.com/annotate/model-assisted-labeling
Your workspace comes with 3 free Roboflow Train credits by default.


Instruction: Object Detection with YOLOV7 on custom dataset

Input:
I am trying to predict bounding boxes on a custom dataset using transfer learning on yolov7 pretrained model.
My dataset contains 34 scenes for training, 2 validation scenes and 5 test scenes. Nothing much happens on the scene, just the camera moves 60-70 degree around the objects on a table/flat surface and scales/tilts a bit. So, even though I have around 20k training images (extracted from 34 scenes), from each scene, the images I get are almost the same, with a kind of augmentation effect (scaling, rotation, occlusion and tilting coming from camera movement).
Here is an example of a scene (first frame and last frame)

Now, I tried different things.

transfer learning with Pretrained yolov7 p5 model
transfer learning with Pretrained yolov7 p5 model (with freezing the extractor, 50 layers)
transfer learning with Pretrained yolov7 tiny model
transfer learning with Pretrained yolov7 tiny model (with freezing the extractor, 28 layers)
full training yolov7 p5 network
full training yolov7 tiny network.

Some of them kind of works (correctly predicts the bounding boxes with 100% precision, but lower recall, and sometimes with wrong class label), but the biggest problem I am facing is, for validation, the object loss is never going down (No matter which approach I try). It happens even from the start, so not sure if I am overfitting or not.
The below graph is from transfer learning in tiny model with frozen backbone.

Any suggestions of how to solve the problem and get a better result?

Output:
I would suggest you thoroughly review your dataset, to start.

Check the class distributions.

How many classes do you have, and what are the counts of the objects of these classes in the training set?
What are the counts in the validation set? Are the ratios approximately similar or different?
Is any class lacking examples (i.e. is too few by proportion)?
Do you have enough background samples? (Images where no desired object is present)

Check your dataset's annotations. Are your objects labelled correctly? If you have time, take a random 1000 images and plot the bounding boxes on them and manually check the labels. This is a sort of sanity check, and sometimes you can find wrongly drawn boxes and incorrect labels.

Another issue could be the lack of variety, as you have mentioned. You have 20K images in your training set, but possibly there are at most just ~34 unique mugs inside (assuming mug is a class). Maybe all those mugs are white, blue, or brown in color, but in your validation the mug is a bright red. (I hope you get the idea).

Try playing around with the hyperparameters a bit. Explore a slightly lower or slightly longer learning rate, longer warmup, stronger weight decay. I assume these are the settings you are using; try increasing the mosaic, copy paste, flip up etc. probabilities as well. If stronger augmentation params are having positive results, it could be a hint that the problem is that the dataset is redundant and lacks variety.


Instruction: How to stop a layer updating in pytorch?

Input:
class DR(nn.Module):
    def __init__(self, orginal, latent_dims):
        super(DR, self).__init__()
        self.latent_dims=latent_dims
        self.linear1 = nn.Linear(orginal, 1000)
        self.linear2 = nn.Linear(1000, 2000)
        self.linear3 = nn.Linear(2000, latent_dims)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        return x

In this DR class, i don't want to update linear1, and linear2 in the traning time. That means, the layer should be same as it initialized, I only want to update linear3 in the traning time. How can i do this ?
I expect a solution of this proeblem. Thank you

Output:
What appears like an underused feature of PyTorch is that the requires_grad, which is meant to disable inference caching for gradient computation can be called on torch.Tensor (as an in place and out of place operation) but also on nn.Module (in place).
In your case, you can simply write in two lines:
model.linear1.requires_grad_(False)
model.linear2.requires_grad_(False)


Instruction: ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([13456, 1])) is deprecated

Input:
My code: https://colab.research.google.com/drive/1qjfy2OsHYewhHDej-W83CMNercB7o7r8?usp=sharing
the error: ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([13456, 1])) is deprecated. Please ensure they have the same size.
the dataset consists of 2 folders: 0 and 1, and in each of these two folders, there’re about 2500 512*512 images and a json file for each image.
the code was from the pytorch gan tutorial, i just changed the dataset.
I wonder where does the 13456 come from?

Output: The original code is intended for 64 x 64 images, not 512 x 512 ones. To fix the problem, you have to either downsize the images to 64 x 64 or modify the discriminator and the generator.


Instruction: Extracting features from a pretrained model using hook function

Input:
I want to extract features from some of the layers from a pretrained model. For this aim, I am using thi pretrained model from here. I removed some of the final layers and for loading the pretrained weights, I use strict=False.
The architecture of the model is as follows:
Net(
  (blocks): ModuleList(
    (0): ResNetBasicStem(
      (conv): Conv3d(3, 64, kernel_size=(1, 7, 7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False)
      (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (activation): ReLU()
    )
    (1): ResStage(
      (res_blocks): ModuleList(
        (0): ResBlock(
          (branch1_conv): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
          (branch1_norm): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(64, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(64, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
        (1): ResBlock(
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(256, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(64, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
        (2): ResBlock(
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(256, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(64, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(64, 64, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(64, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
      )
    )
    (2): ResStage(
      (res_blocks): ModuleList(
        (0): ResBlock(
          (branch1_conv): Conv3d(256, 512, kernel_size=(1, 1, 1), stride=(1, 2, 2), bias=False)
          (branch1_norm): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(256, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(128, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
        (1): ResBlock(
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(512, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(128, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
        (2): ResBlock(
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(512, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(128, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
        (3): ResBlock(
          (branch2): BottleneckBlock(
            (conv_a): Conv3d(512, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_a): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_a): ReLU()
            (conv_b): Conv2plus1d(
              (conv_t): Conv3d(128, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False)
              (norm): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activation): ReLU()
              (conv_xy): Conv3d(128, 128, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False)
            )
            (norm_b): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (act_b): ReLU()
            (conv_c): Conv3d(128, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1), bias=False)
            (norm_c): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (activation): ReLU()
        )
      )
    )
  )
)

I use hook function for extracting features from layers and my method for loading the features from (1): ResStage and (2): ResStage is as follows:
class mymodel(nn.Module):
def __init__(self, pretrained=False):
    super(mymodel, self).__init__()

    self.activation = {}
    def get_activation(name):
        def hook(model, input, output):
            self.activation[name] = output.detach()
        return hook

    self.r2plus1d = create_r2plus1d()

    self.r2plus1d.Net.blocks[1].register_forward_hook(get_activation('ResBlock1'))
    self.r2plus1d.Net.blocks[2].register_forward_hook(get_activation('ResBlock2'))

def forward(self, x, out_consp = False):
    x = self.r2plus1d(x)

    block1_output = self.activation['ResBlock1']   # channel_num:256
    block2_output = self.activation['ResBlock2']   # channel_num:512

    return block1_output, block2_output

Unfortunately the error says that there is not Net insised the state_dict of the model (when it comes to use from hook function). For other pretrained models I could use such scenarios for extracting features from intermediate layers but seemingly, if I'm not mistaken, I think maybe it would be tricky to extract features from Net.

Output:
Looking at the link you provided, the function create_r2plus1d() returns the following
 return Net(blocks=nn.ModuleList(blocks))

Your object self.r2plus1d is already a Net instance, so your line
self.r2plus1d.Net.blocks[1].register_forward_hook(get_activation('ResBlock1'))

is basically like calling Net twice.
You probably only have to call it like that and it should work.
self.r2plus1d.blocks[1].register_forward_hook(get_activation('ResBlock1'))

Let me know if this helps.


Instruction: How to use AWS Sagemaker with newer version of Huggingface Estimator?

Input:
When trying to use Huggingface estimator on sagemaker, Run training on Amazon SageMaker e.g.
# create the Estimator
huggingface_estimator = HuggingFace(
        entry_point='train.py',
        source_dir='./scripts',
        instance_type='ml.p3.2xlarge',
        instance_count=1,
        role=role,
        transformers_version='4.17',
        pytorch_version='1.10',
        py_version='py38',
        hyperparameters = hyperparameters
)

When I tried to increase the version to transformers_version='4.24', it throws an error where the maximum version supported is 4.17.
How to use AWS Sagemaker with newer version of Huggingface Estimator?
There's a note on using newer version for inference on https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/9 but it looks like the way to use it for training with the Huggingface estimator is kind of complicated https://discuss.huggingface.co/t/huggingface-pytorch-versions-on-sagemaker/26315/5?u=alvations and it's not confirmed that the complicated steps can work.

Output:
You can use the Pytorch estimator and in your source directory place a requirements.txt with Transformers added to it. This will ensure 2 things

You can use higher version of pytorch 1.12 (current) compared to 1.10.2 in the huggingface estimator.
Install new version of HuggingFace Transformers library.

To achieve this you need to structure your source directory like this
scripts
/train.py
/requirements.txt
and pass the source_dir attribute to the pytorch estimator
pt_estimator = PyTorch(
entry_point="train.py",
source_dir="scripts",
role=sagemaker.get_execution_role(),


Instruction: Using a target size (torch.Size([80670, 1])) that is different to the input size (torch.Size([80670]))

Input:
I want to implement GCN with my own dataset (from Pytorch geometric tutorial). I copied my notebook below.
It is working when I use the original implementation, However, if I want to use custom data then it gives the below error:
My notebook
Original implementation
I used Load CSV from pyG and tried to incorporate it into their GCN implementation. I know in the original implementation they used the same dataset but I want to try with custom data (load datasets for later).
Using a target size (torch.Size([80670, 1])) that is different to the input size (torch.Size([80670])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
The problem arises:
Print (data)
HeteroData(
  movie={ x=[9742, 404] },
  user={ num_nodes=610 },
  (user, rates, movie)={
    edge_index=[2, 100836],
    edge_label=[100836] # mine is edge_label=[100836,1]
  }
)

I don't know how to make edge_label=[100836,1] to edge_label=[100836] which will hopefully solve the issue.
Looking forward to the comments.

Output:
You can use edge_label = edge_label.squeeze()
it will remove the extra dimension


Instruction: What is Pytorch equivalent of Pandas groupby.apply(list)?

Input:
I have the following pytorch tensor long_format:
tensor([[ 1.,  1.],
        [ 1.,  2.],
        [ 1.,  3.],
        [ 1.,  4.],
        [ 0.,  5.],
        [ 0.,  6.],
        [ 0.,  7.],
        [ 1.,  8.],
        [ 0.,  9.],
        [ 0., 10.]])

I would like to groupby the first column and store the 2nd column as a tensor. The result is NOT guranteed to be the same size for each grouping. See example below.
[tensor([ 1., 2., 3., 4., 8.]),
 tensor([ 5.,  6., 7., 9., 10.])]

Is there any nice way to do this using purely Pytorch operators? I would like to avoid using for loops for tracebility purposes.
I have tried using a for loop and empty list of empty tensors but this result in an incorrect trace (different inputs values gave same results)
n_groups = 2
inverted = [torch.empty([0]) for _ in range(n_groups)]
for index, value in long_format:
   value = value.unsqueeze(dim=0)
   index = index.int()
   if type(inverted[index]) != torch.Tensor:
      inverted[index] = value
   else:
      inverted[index] = torch.cat((inverted[index], value))

Output:
You can use this code:
import torch
x = torch.tensor([[ 1.,  1.],
        [ 1.,  2.],
        [ 1.,  3.],
        [ 1.,  4.],
        [ 0.,  5.],
        [ 0.,  6.],
        [ 0.,  7.],
        [ 1.,  8.],
        [ 0.,  9.],
        [ 0., 10.]])

result =  [x[x[:,0]==i][:,1] for i in x[:,0].unique()]

output
[tensor([ 5.,  6.,  7.,  9., 10.]), tensor([1., 2., 3., 4., 8.])]


Instruction: Why are Pytorch transform functions not being differentiated with autograd?

Input:
I have been trying to write a set of transforms on input data. I also need the transforms to be differentiable to compute the gradients. However, gradients do not seem to be calculated for the resize, normalize transforms.
from torchvision import transforms

from torchvision.transforms import ToTensor

resize = transforms.Resize(size=224, interpolation=transforms.InterpolationMode.BICUBIC, max_size=None, antialias=None)

crop = transforms.CenterCrop(size=(224, 224))

normalize = transforms.Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711))

img = torch.Tensor(images[30])

img.requires_grad = True

rgb = torch.dsplit(torch.Tensor(img),3)

transformed = torch.stack(rgb).reshape(3,100,100)

resized = resize.forward(transformed)

normalized = normalize.forward(resized)

image_features = clip_model.encode_image(normalized.unsqueeze(0).to(device))

text_features = clip_model.encode_text(text_inputs)

similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)

When running normalized.backward(), there are no gradients for resized and transformed.
I have tried to find the gradient for each individual transform, but it still does not calculate the gradients.

Output:
Trying to reproduce your error, what I get when backpropagating the gradient from normalized is:

RuntimeError: grad can be implicitly created only for scalar outputs

What this error means is that the tensor you are calling backward onto should be a scalar and not a vector or multi-dimensional tensor. Generally you would want to reduce the dimensionality for example by averaging or summing. For example you could do the following:
> normalized.mean().backward()


Instruction: Why PyTorch BatchNorm1D gives "batch_norm" not implemented for 'Long'" error while normalizing Integer type tensor?

Input:
I am trying to learn some functions in Pytorch framework and was stuck due to below error while normalizing a simple integer tensor. Could someone please help me with this.
Here is the sample code to reproduce the error -
import torch
import torch.nn as nn

#Integer type tensor
test_int_input = torch.randint(size = [3,5],low=1,high=9)

# BatchNorm1D object
batchnorm1D = nn.BatchNorm1d(num_features=5)
test_output = batchnorm1D(test_int_input)

Error -
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-38-6c672cd731fa> in <module>
      1 batchnorm1D = nn.BatchNorm1d(num_features=5)
----> 2 test_output = batchnorm1D(test_input)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py in forward(self, input)
    105             input, self.running_mean, self.running_var, self.weight, self.bias,
    106             self.training or not self.track_running_stats,
--> 107             exponential_average_factor, self.eps)
    108
    109

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
   1668     return torch.batch_norm(
   1669         input, weight, bias, running_mean, running_var,
-> 1670         training, momentum, eps, torch.backends.cudnn.enabled
   1671     )
   1672

RuntimeError: "batch_norm" not implemented for 'Long'

However, if we try to apply the same on a different non-int tensor, then it works. Here is the example -
import torch
import torch.nn as nn

#Integer type tensor
#test_input = torch.randn(size = [3,5])

# BatchNorm1D object
batchnorm1D = nn.BatchNorm1d(num_features=5)
test_output = batchnorm1D(test_input)
test_output

Output -
tensor([[ 0.4311, -1.1987,  0.9059,  1.1424,  1.2174],
        [-1.3820,  1.2492, -1.3934,  0.1508,  0.0146],
        [ 0.9509, -0.0505,  0.4875, -1.2931, -1.2320]],
       grad_fn=<NativeBatchNormBackward>)

Output:
Your input tensor should be a floating point:
>>> batchnorm1D(test_int_input.float())
tensor([[-5.9605e-08, -1.3887e+00, -9.8058e-01,  2.6726e-01,  1.4142e+00],
        [-1.2247e+00,  4.6291e-01,  1.3728e+00, -1.3363e+00, -7.0711e-01],
        [ 1.2247e+00,  9.2582e-01, -3.9223e-01,  1.0690e+00, -7.0711e-01]],
       grad_fn=<NativeBatchNormBackward0>)


Instruction: Taking mean of all rows in a numpy matrix grouped by values based on another numpy matrix

Input:
I have a matrix A of size NXN with float values and another boolean matrix B of size NXN
For every row, I need to find the mean of all values in A belonging to indices where True is the corresponding value for that index in matrix B
Similarly, I need to find the mean of all values in A belonging to indices where False is the corresponding value for that index in matrix B
Finally, I need to find the count of number of rows where "True" mean is lesser than "False" mean
For example :
A = [[1.0, 2.0, 3.0]
     [4.0, 5.0, 6.0]
     [7.0, 8.0, 9.0]]

B = [[True, True, False]
     [False, False, True]
     [True, False, True]]

Initially, count = 0
For row 1, true_mean = 1.0+2.0 / 2 = 1.5 and false_mean = 3.0
true_mean < false_mean, so count = 0+1=1
For row 2, true_mean = 6.0 and false_mean = 4.0+5.0 / 2 = 4.5
true_mean > false_mean, so count remains same
For row 3, true_mean = 7.0+9.0 / 2 = 8.0 and false_mean = 8.0
true_mean == false_mean, so count remains same
Final count value = 1
My attempt:-
true_mat = np.where(B, A, 0)
false_mat = np.where(B, 0, A)

true_mean = true_mat.mean(axis=1)
false_mean = false_mat.mean(axis=1)

But this actually gives wrong answer since denominator is not exactly the count of number of True/False values in that row but instead 'N'
I only need the count, I don't need the true_mean and false_mean
Anyway to fix it?

Output:
I would say your start is good
true_mat = np.where(B, A, 0)
false_mat = np.where(B, 0, A)

But we want to divide by the number of Trues or Falses, respectively, so...
true_sum = np.sum(B, axis = 1) #sum of Trues per row
false_sum = N-true_sum         # if you don't have N given, do N=A.shape[0]

true_mean = np.sum(true_mat, axis = 1)/true_sum      #add up rows of true_mat and divide by true_sum
false_mean = np.sum(false_mat, axis = 1)/false_sum

For your example this gives
[1.5 6.  8. ]
[3.  4.5 8. ]

So now we just have to compare where the second is larger than the first:
count = np.sum(np.where(false_mean > true_mean, 1, 0))


Instruction: Is it possible to perform quantization on densenet169 and how?

Input:
I have been trying to performing quantization on a densenet model without success. I have been trying to implement pytorch post training static quantization. Pytorch has quantized versions of other models, but does not have for densenet. Is it possible to quantize the densenet architecture.
I have searched for tutorials on how to apply quantization on pre-trained models but i havent had any success.

Output:
Here's how to do this on DenseNet169 from torchvision:
from torch.ao.quantization import QuantStub, DeQuantStub
from torch import nn
from torchvision.models import densenet169, DenseNet169_Weights
from tqdm import tqdm
from torch.ao.quantization import HistogramObserver, PerChannelMinMaxObserver
import torch

# Wrap base model with quant/dequant stub
class QuantizedDenseNet169(nn.Module):
    def __init__(self):
        super().__init__()
        self.dn = densenet169(weights=DenseNet169_Weights.IMAGENET1K_V1)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.dn(x)
        return self.dequant(x)

dn = QuantizedDenseNet169()
# move to gpu
dn.cuda()

# Propagate qconfig
dn.qconfig = torch.quantization.QConfig(
    activation=HistogramObserver.with_args(),
    weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8)
)
# fbgemm for x86 architecture
torch.backends.quantized.engine = 'fbgemm'
dn = torch.quantization.prepare(dn, inplace=False)

# calibrate with own dataset (I'm using random inputs to show process)
with torch.no_grad():
    for _ in tqdm(range(5), desc="PTQ progess"):
        input_ = torch.randn([1, 3, 128, 128], device='cuda')
        dn.forward(input_)

# move to cpu before quantization
dn.cpu()
dn = torch.quantization.convert(dn, inplace=False)

# check if it's working
out = dn(torch.randn([1, 3, 128, 128]))


Instruction: Why when the batch size increased, the epoch time will also increasing?

Input:
Epoch time means the time required to train for an epoch.
From my point of view, when the batch size increased, the epoch time will reduce. But actually I observed that when the batch size increased, the epoch time would reduce at first as we expect, but then increase. The relationship between batch size and epoch is like a v-shaped function. For example, when I change batch size from 1 to 256, the epoch time reduce monotonically. But when I change batch size from 256 to 1024, the epoch time will increase monotonically.
I've observed this so many times that I don't think it's a coincidence. I thought about the reason for this phenomenon from several aspects, but none of them were right.
Considering an simple case, one use batchsize 1024, other use batchsize 512. The calculation time of the former must be less than twice the calculation time of the latter, and same is true for the time to transfer data to the GPU. So whatever, the epoch time of batchsize1024 must be smaller than batchsize 512 instead of larger.
Is it because the algorithm adopted by pytorch is not effective in large matrices multiplication?
My configuration is as follows, I use pytorch, the training process is very standard, and I have observed this phenomenon too many times in different tasks, so I don't think it is accidental. My code is as follows.
               for epoch in range(1, total_epoch + 1):
                    for x, y in tqdm(self.loader):
                    with autocast():
                        loss = self.criterion(self.model(x), y)
                    self.optimizer.zero_grad()
                    scaler.scale(loss).backward()
                    scaler.unscale_(self.optimizer)
                    scaler.step(self.optimizer)
                    scaler.update()
                    self.optimizer.step()

Output:
As you already noticed, there are many factors that may affect epoch-time/batch-time.
Some of these factors may be specific to your machine's configuration. In order to get an accurate answer, you'll need to have a more detailed breakdown of the running time. One way of doing so would be using a profiler.
Try and be more specific about what is taking longer as batch size changes:

Loading data from disk
Moving data from CPU mem to device mem
Forward pass
Backward pass
Updating the model

Once you pinpoint the cause of latency, you can speculate as to the reason for it.
For instance, if loading the data from disk is the bottleneck, you might need more workers in the DataLoader as you increase batch size. In that case, you might be hitting a "ceiling" of the number of cores you have on your CPU.
Bottom line - there's not enough information in your question for a good answer.


Instruction: Why does StableDiffusionPipeline return black images when generating multiple images at once?

Input:
I am using the StableDiffusionPipeline from the Hugging Face Diffusers library in Python 3.10.2, on an M2 Mac (I tagged it because this might be the issue). When I try to generate 1 image from 1 prompt, the output looks fine, but when I try to generate multiple images using the same prompt, the images are all either black squares or a random image (see example below). What could be the issue?
My code is as follows (where I change n_imgs from 1 to more than 1 to break it):
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe = pipe.to("mps")  # for M1/M2 chips
pipe.enable_attention_slicing()

prompt = "a photo of an astronaut driving a car on mars"

# First-time "warmup" pass (because of weird M1 behaviour)
_ = pipe(prompt, num_inference_steps=1)

# generate images
n_imgs = 1
imgs = pipe([prompt] * n_imgs).images

I also tried setting num_images_per_prompt instead of creating a list of repeated prompts in the pipeline call, but this gave the same bad results.
Example output (for multiple images):

[edit/update]: When I generate the images in a loop surrounding the pipe call instead of passing an iterable to the pipe call, it does work:
# generate images
n_imgs = 3
for i in range(n_imgs):
    img = pipe(prompt).images[0]
    # do something with img

But it is still a mystery to me as to why.

Output: Apparently it is indeed an Apple Silicon (M1/M2) issue, of which Hugging Face is not yet sure why this is happening, see this GitHub issue for more details.


Instruction: Locating tags in a string in PHP (with respect to the string with tags removed)

Input:
I want to create a function that labels the location of certain HTML tags (e.g., italics tags) in a string with respect to the locations of characters in a tagless version of the string.
(I intend to use this label data to train a neural network for tag recovery from data that has had the tags stripped out.)
The magic function I want to create is label_italics() in the below code.
$string = 'Disney movies: Aladdin, Beauty and the Beast.';
$string_all_tags_stripped_but_italics = strip_tags($string, ''); // same as $string in this example
$string_all_tags_stripped = strip_tags($string); // 'Disney movies: Aladdin, Beauty and the Beast.'
$featr_string = $string_all_tags_stripped.' '; // Add a single space at the end
$label_string = label_italics($string_all_tags_stripped_but_italics);
echo $featr_string; // 'Disney movies: Aladdin, Beauty and the Beast. '
echo $label_string; // '0000000000000001000000101000000000000000000010'

If a character is supposed to have an  or  tag immediately preceding it, it is labeled with a 1 in $label_string; otherwise, it is labeled with a 0 in $label_string.  (I'm thinking I don't need to worry about the difference between  and  because the recoverer will simply alternate between  and  so as to maintain well-formed markup, but I'm open to reasons as to why I'm wrong about this.)
I'm just not sure what the best way to create label_italics() is.
I wrote this function that seems to work in most cases, but it also seems a little clunky and I'm posting here in hopes that there is a better way.  (If this turns out to be the best way, the below function would be easily generalizable to any HTML tag passed in as a second argument to the function, which could be renamed label_tag().)
function label_italics($stripped) {
  while ((stripos($stripped, '') || stripos($stripped, '')) !== FALSE) {
    $position = stripos($stripped, '');
    if (is_numeric($position)) {
      for ($c = 0; $c < $position; $c++) {
        $output .= '0';
      }
      $output .= '1';
    }
    $stripped = substr($stripped, $position + 4, NULL);
    $position = stripos($stripped, '');
    if (is_numeric($position)) {
      for ($c = 0; $c < $position; $c++) {
        $output .= '0';
      }
      $output .= '1';
    }
    $stripped = substr($stripped, $position + 5, NULL);
  }
  for ($c = 0; $c <= strlen($stripped); $c++) {
    $output .= '0';
  }
  return $output;
}

The function produces bad output if the tags are surplus or the markup is badly formed in the input.  For example, for the following input:
$string = 'Disney movies: Aladdin, Beauty and the Beast.';

The following misaligned output is given.
Disney movies: Aladdin, Beauty and the Beast.
0000000000000001000000000101000000000000000000010

(I'm also open to reasons why I'm going about the creation of the label data all wrong.)

Output:
I think I've got something. How about this:
function label_italics($string) {
    return preg_replace(['/<i>/', '/<\/i>/', '/[^#]/', '/##0/', '/#0/'],
                        ['#', '#', '0', '2', '1'], $string);
}

see: https://3v4l.org/cKG46
Note that you need to supply the string with the tags in it.
How does it work?
I use preg_replace() because it can use regular expressions, which I need once. This function goes through the two arrays and execute each replacement in order. First it replace all occurrences of <i> and </i> by # and anything else by 0. Then replaces ##0 by 2 and #0 by 1. The 2 is extra to be able to replace <i></i>. You can remove it, and simplify the function, if you don't need it.
The use of the # is arbitrary. You should use anything that doesn't clash with the content of your string.

Here's an updated version. It copes with tags at the end of the line and it ignores any # characters in the line.
function label_italics($string) {
    return preg_replace(['/[^<\/i\>]/', '/<i>/', '/<\/i>/', '/i/', '/##0/', '/#0/'],
                        ['0', '#', '#', '0', '2', '1'], $string . ' ');
}

See: https://3v4l.org/BTnLc


Instruction: BGR to RGB for CUB_200 images by Image.split()

Input:
I am creating a PyTorch dataset and dataloader from CUB_200. When reading the images as pill, I need to change the BGR channels to RGB and I use the following code:
def _read_images_from_list(imagefile_list):

   imgs = []
   mean=[0.485, 0.456, 0.406]
   std= [0.229, 0.224, 0.225]

   Transformations = transforms.Compose([transforms.Resize([224, 224]), transforms.ToTensor(), transforms.Normalize(mean, std)])

   for imagefile in imagefile_list:
      # read images as PIL instead of NUMPY
      img = Image.open(imagefile)
      b, g, r = img.split()
      img = Image.merge("RGB", (r, g, b))

      img = Transformations(img) # ToTensor and between [0,1], then normalized using image net mean and std, then transposed into shape (C,H,W)

     imgs += [img]
return imgs

After going through a number of classes, I get the following error.
ValueError: not enough values to unpack (expected 3, got 1)

I wonder what should I do now? it means that one of the images has only one channel instead of one. Can this be the case or there is a problem with my code? I had a different implementation before but it worked. The reason I changed this implementation was that I could not normalize my images.
This is the old implementation:
def _read_images_from_list(imagefile_list):
  imgs = []
  for imagefile in imagefile_list:

    img = cv2.imread(imagefile).astype(np.float32)
    img = cv2.resize(img, (224, 224))
    # Convert RGB to BGR
    img_r, img_g, img_b = np.split(img, 3, axis=2)
    img = np.concatenate((img_b, img_g, img_r), axis=2)
    # Extract mean
    img -= np.array((103.94,116.78,123.68), dtype=np.float32)  # BGR mean

    # HWC -> CHW, compatible with pytorch
    img = np.transpose(img, [2, 0, 1])
    imgs += [img]
return imgs

Output:
I would strongly recommend you use skimage.io to load your images, not opencv. It opens the images in RGB format by default, removing your shuffling overhead, but if you want to convert BGR to RGB you can use this:
import numpy as np

img = np.arange(27).reshape(3,3,3)
b = img[:,:,0]
g = img[:,:,1]
r = img[:,:,2]

rgb = np.dstack([r,g,b])

print(img)
print("#"*20)
print(rgb)