Instruction: Convert list of PNGImageFile to array of array

Input:
I have a dataset organized in this way: /dataset/train/class/images.png (the same for test) (and I have 2 classes, positive and negative).
I want to obtain x_train, y_train, x_test and y_test, so I am using this python script:
x_train = []
y_train = []
x_test = []
y_test = []
base_dir_train = 'Montgomery_real_splitted/TRAIN/'
base_dir_test = 'Montgomery_real_splitted/TEST/'
for f in sorted(os.listdir(base_dir_train)):
    if os.path.isdir(base_dir_train+f):
        print(f"{f} is a target class")
        for i in sorted(os.listdir(base_dir_train+f)):
            y_train.append(f)
            im = Image.open(base_dir_train+f+'/'+i)
            x_train.append(im)
for f in sorted(os.listdir(base_dir_test)):
    if os.path.isdir(base_dir_test+f):
        print(f"{f} is a target class")
        for i in sorted(os.listdir(base_dir_test+f)):
            y_test.append(f)
            imt=Image.open(base_dir_test+f+'/'+i)
            x_test.append(imt)

y_train = np.array(y_train)
y_test = np.array(y_test)

Basically I obtain what I want, for example x_train is this:
[<PIL.PngImagePlugin.PngImageFile image mode=L size=4892x4020 at 0x10A98B280>,
 <PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x10A98B040>,
...
 <PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D940>,
 <PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D9A0>]

And y_train is:
array(['neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg',
        ...
       'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos'],
      dtype='<U3')

However, I want that x_train is in this format:
array([[[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

        ...

        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]]], dtype=uint8)

How can I convert it?
EDIT for @Pyaive Oleg If I do im = np.array(im) then the result is this, which is different from the one that I want. The same for the tensor
[array([[ 0,  0,  0, ...,  0,  0,  0],
        [ 0,  0,  0, ...,  0,  0,  0],
        [ 0,  0,  0, ...,  0,  0,  0],
        ...,
        [ 5,  6,  7, ..., 14,  9,  5],
        [ 4,  5,  6, ..., 12,  8,  4],
        [ 0,  1,  2, ...,  3,  2,  0]], dtype=uint8),
 array([[  1,   1,   1, ...,   8, 246,   0],
        [  1,   1,   1, ...,   0,   7,  11],
        [  1,   1,   1, ...,   0,   0,   6],
        ...,
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0]], dtype=uint8),
        ...
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0]], dtype=uint8),
 array([[  0,   0,   0, ...,   0, 255,   1],
        [  0,   0,   0, ...,   0,   3,  11],
        [  0,   0,   0, ...,   2,   0,   7],
        ...,
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0]], dtype=uint8),
 array([[  1,   1,   1, ...,  19, 246,   0],
        [  1,   1,   1, ...,   0,  16,   0],
        [  1,   1,   1, ...,   2,   0,  12],
        ...,
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0],
        [  0,   0,   0, ...,   0,   0,   0]], dtype=uint8),

Output:
Before appending imt to x_train, do this:
imt = np.array(imt)

The following also can help:
from torchvision import transforms
imt = transforms.ToTensor()(imt)


Instruction: ResNet doesn't train because of differences in images' sizes

Input:
So ho I have 30 folders with images inside them, and I wanted to train ResNet50 on them. I created a CustomDataset, and inside I put a Resize(224, 224) so that every image had the same size.
Here's what I did:
class CustomImageDataset(Dataset):
    def __init__(self, annotations_file, img_dir, transform=None, target_transform = None):
        self.img_labels = pd.read_csv(annotations_file, sep= ';')
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0][0:9] + '.tar', self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        transf = transforms.Resize((224, 224))
        image = transf(image)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label

The dataset works, however, when the network is trying to create the batch, at entry 271 (which I don't know how to plot to see the image) it raises this error:
Traceback (most recent call last):
  File "final.py", line 235, in <module>
    num_epochs = epochs, is_inception=(model_name== 'inception'))
  File "final.py", line 111, in train_model
    for input, labels in dataloaders[phase]:
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 172, in default_collate
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 172, in <listcomp>
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/home/fdalligna/.local/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 138, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 224, 224] at entry 0 and [1, 224, 224] at entry 271

Does anyone by chance know how can I make all images of size [3, 224, 224]?
Thank you :)

Output:
As noted in the comments, the error suggests that your dataset contains both gray scale and RGB (color) images. Although all images have indeed been resized to 224 pixels, color images have 3 channels, whereas gray scale images only have a single channel, so a batch cannot be created.
If you insist on training a network on this mixed dataset, you can either

Turn color images into gray scale
Modify gray scale images to have 3 channels to mimic RGB

From training a neural network point of view, the first option makes more sense. This can be achieved by averaging a color image across the RGB channel.
    def __getitem__(self, idx):
        # copied
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0][0:9] + '.tar', self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        transf = transforms.Resize((224, 224))
        image = transf(image)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)

        # check if color image
        if image.size(0) == 3:
            # average across channels
            image = image.mean(dim=0).unsqueeze(0)
        return image, label

You should make sure that the input layer of the network expects mono-channel images.
If you want to choose option 2 instead, you can instead do
        # check if gray scale image
        if image.size(0) == 1:
            # repeat color channel
            image = image.repeat(3, 1, 1)
        return image, label

In this case, the model would expect three input channels, i.e., nn.Conv2d(in_channels=3, out_channels, kernel_size, ...).


Instruction: How to simplify function for standardising images?

Input:
I have a function to calculate the mean and standard deviation of my dataset.
Is there a simpler way to do this? As it takes a while to compute.
def get_mean_std(loader):
  sum = 0
  sum_sq_err = 0
  for data, _ in loader:
    sum += torch.mean(data, dim=[0,2,3])
    sum_sq_err += torch.mean(data**2, dim=[0,2,3])
  mean = sum/len(loader)
  std = (sum_sq_err/(len(loader)) - mean**2)**0.5
  return mean, std

Output:
Note that this approach is not even correct in general, as the mean of a set is not the mean of the means of some subsets in general (it is when all the subsets have the same length, but that may or may not be the case here).
Provided that every batch is of the same size, what I would do is to call torch.sum in the loop, but rather than already accumulating it into a sum, appending it into a list, and then reducing it via torch.sum + a division afterwards. Note that torch.sum implements a highly non-trivial algorithm that is more precise in general than the naïve iterative sum.


Instruction: how to package my python pytorch program for android?

Input:
my codes is python pytorch.
I build it to .exe, it can work at windows.
How to package the codes for android.
I hope it can work at android.
Thank you.

Output:
Sorry, I don't know how to call an exe program in android, but I can give you some other advice. According to your needs, maybe you can learn about Chaquopy.
Chaquopy provides everything you need to include Python components in an Android app, including:
(1)Full integration with Android Studio’s standard Gradle build system.
(2)Simple APIs for calling Python code from Java/Kotlin, and vice versa.
(3)A wide range of third-party Python packages, including SciPy, OpenCV, TensorFlow and many more.
With Chaquopy, you can run python programs directly on Android. I have experience running sklearn programs on Android before. If you have any questions, you can continue to ask.
Chaquopy official website:https://chaquo.com/chaquopy/


Instruction: Sagemaker inference : how to load model

Input:
I have trained a BERT model on sagemaker and now I want to get it ready for making predictions, i.e, inference.
I have used pytorch to train the model and model is saved to s3 bucket after training.
Here is structure inside model.tar.gz file which is present in s3 bucket.

Now, I do not understand how can I make predictions on it. I have tried to follow many guides but still could not understand.
Here is something which I have tried:
inference_image_uri = sagemaker.image_uris.retrieve(
    framework='pytorch',
    version='1.7.1',
    instance_type=inference_instance_type,
    region=aws_region,
    py_version='py3',
    image_scope='inference'
)

sm.create_model(
    ModelName=model_name,
    ExecutionRoleArn=role,
    PrimaryContainer={
        'ModelDataUrl': model_s3_dir,
        'Image': inference_image_uri
    }
)

sm.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            "VariantName": "variant1", # The name of the production variant.
            "ModelName": model_name,
            "InstanceType": inference_instance_type, # Specify the compute instance type.
            "InitialInstanceCount": 1 # Number of instances to launch initially.
        }
    ]
)

sm.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=endpoint_config_name
)

from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONLinesSerializer
from sagemaker.deserializers import JSONLinesDeserializer

inputs = [
    {"inputs": ["I have a question  [EOT]  Hey Manish Mittal ! I'm OneAssist bot. I'm here to answer your queries. [SEP] thanks"]},
#     {"features": ["OK, but not great."]},
#     {"features": ["This is not the right product."]},
]

predictor = Predictor(
    endpoint_name=endpoint_name,
    serializer=JSONLinesSerializer(),
    deserializer=JSONLinesDeserializer(),
    sagemaker_session=sess
)

predicted_classes = predictor.predict(inputs)

for predicted_class in predicted_classes:
    print("Predicted class {} with probability {}".format(predicted_class['predicted_label'], predicted_class['probability']))

I can see the endpoint created but while predicting, its giving me error:

ModelError: An error occurred (ModelError) when calling the
InvokeEndpoint operation: Received server error (0) from primary with
message "Your invocation timed out while waiting for a response from
container primary. Review the latency metrics for each container in
Amazon CloudWatch, resolve the issue, and try again."

I do not understand how to make it work, and also, do I need to give any entry script to the inference, if yes where.

Output:
Here's detailed documentation on deploying PyTorch models - https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#deploy-pytorch-models
If you're using the default model_fn provided by the estimator, you'll need to have the model as model.pt.
To write your own inference script and deploy the model, see the section on Bring your own model. The pytorch_model.deploy function will deploy it to a real-time endpoint, and then you can use the predictor.predict function on the resulting endpoint variable.


Instruction: How to shrink a 2D tensor to another 2D tensor using boolean mask?

Input:
Say I have a 2D pytorch tensor and a 2D numpy boolean as follows,
a = torch.tensor([[ 0.,  1.,  2.],
                  [ 3.,  4.,  5.],
                  [ 6.,  7.,  8.],
                  [ 9., 10., 11.],
                  [12., 13., 14.]])

m = numpy.array([[ False,  True, False],
                 [  True, False,  True],
                 [ False,  True,  True],
                 [ False, False, False],
                 [  True, False, False]])

They have the same dimension and the number of True's in each column of m is the same.
I need to get the 2x3 tensor that is
a.transpose(0,1).masked_select(torch.from_numpy(m.transpose())).reshape(a.shape[1],-1).transpose(0,1)

which is
tensor([[ 3.,  1.,  5.],
        [12.,  7.,  8.]])

The actual tensor is very large, and the operation needs to be performed many times. So I want to ask what is an efficient way of doing this (or the most efficient way).

Output:
In my benchmarks a jitted numba solution is the fastest, I could find
My benchmarks for a, m with shape (10000,200)(equal result tensors)

1
@numba.jit
13.2 ms (3.46x)

2
list comprehension
31.3 ms (1.46x)

3
baseline
45.7 ms (1.00x)

Generation of sufficiently large sample data for benchmarking
import torch
import numpy as np

def generate_data(rows=500, columns=100):

    a = torch.from_numpy(np.random.uniform(1,10, (rows,columns)).astype(np.float32))

    # argsort trick by @divakar https://stackoverflow.com/a/55317373/14277722
    def shuffle_along_axis(a, axis):
        idx = np.random.rand(*a.shape).argsort(axis=axis)
        return np.take_along_axis(a,idx,axis=axis)

    m = shuffle_along_axis(np.full((columns,rows), np.random.randint(2, size=rows)), 1).astype('bool').T
    return a, np.ascontiguousarray(m)

a, m = generate_data(10000,200)

A jitted numba implementation
import numba as nb

@nb.njit
def gather2d(arr1, arr2):
    res = np.zeros((np.count_nonzero(arr2[:,0]), arr1.shape[1]), np.float32)
    counter = np.zeros(arr1.shape[1], dtype=np.intp)
    for i in range(arr1.shape[0]):
        for j in range(arr1.shape[1]):
            if arr2[i,j]:
                res[counter[j], j] = arr1[i,j]
                counter[j] += 1
    return res

torch.from_numpy(gather2d(a.numpy(),m))

Output
# %timeit 10 loops, best of 5: 13.2 ms per loop
tensor([[2.1846, 7.8890, 8.8218,  ..., 4.8309, 9.2853, 6.4404],
        [5.8842, 3.7332, 6.7436,  ..., 1.2914, 3.2983, 3.5627],
        [9.5128, 2.4283, 2.2152,  ..., 4.9512, 9.7335, 9.6252],
        ...,
        [7.3193, 7.8524, 9.6654,  ..., 3.3665, 8.8926, 4.7660],
        [1.3829, 1.3347, 6.6436,  ..., 7.1956, 4.0446, 6.4633],
        [6.4264, 3.6283, 3.6385,  ..., 8.4152, 5.8498, 5.0281]])

Against a vectorized baseline solution
# %timeit 10 loops, best of 5: 45.7 ms per loop
a.gather(0, torch.from_numpy(np.nonzero(m.T)[1].reshape(-1, m.shape[1], order='F')))

A python list comprehension turns out to be surprisingly fast
def g(arr1,arr2):
    return np.array([i[j] for i,j in zip(arr1.T,arr2.T)]).T

# %timeit 10 loops, best of 5: 31.3 ms per loop
torch.from_numpy(g(a.numpy(), m))


Instruction: How to resolve the error: RuntimeError: received 0 items of ancdata

Input:
I have a torch.utils.data.DataLoader. I have created them with the following code.
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

trainset = CIFAR100WithIdx(root='.',
                           train=True,
                           download=True,
                           transform=transform_train,
                           rand_fraction=args.rand_fraction)

train_loader = torch.utils.data.DataLoader(trainset,
                                           batch_size=args.batch_size,
                                           shuffle=True,
                                           num_workers=args.workers)

But when I run the following code I get an error.
train_loader_2 = []
for i, (inputs, target, index_dataset) in enumerate(train_loader):
    train_loader_2.append((inputs, target, index_dataset))

The error is
Traceback (most recent call last):
  File "main_superloss.py", line 460, in <module>
    main()
  File "main_superloss.py", line 456, in main
    main_worker(args)
  File "main_superloss.py", line 374, in main_worker
    train_loader, val_loader = get_train_and_val_loader(args)
  File "main_superloss.py", line 120, in get_train_and_val_loader
    for i, (inputs, target, index_dataset) in enumerate(train_loader):
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 804, in __next__
    idx, data = self._get_data()
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 771, in _get_data
    success, data = self._try_get_data()
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 724, in _try_get_data
    data = self.data_queue.get(timeout=timeout)
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 284, in rebuild_storage_fd
    fd = df.detach()
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/multiprocessing/reduction.py", line 185, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/C00423766/.conda/envs/dp/lib/python3.7/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

The reason I want to get the data inside a list is because I want to reorder the samples. And not in a random way but in a particular way. How can I do that?

Output:
I was facing a similar issue with my code and based on some discussions (check #1, #2, #3). I used ulimit -n 2048 to increase the maximum number of file descriptors a process can have. You can read more about ulimit here.
About the issue - The discussions suggest that it has to do something with pytorch’s forked multiprocessing code.
On the second part of your question, how to reorder a dataloader - You can refer this answer


Instruction: PyTorch: How to calculate output size of the CNN?

Input:
I went through this PyTorch CNN implementation available here: https://machinelearningknowledge.ai/pytorch-conv2d-explained-with-examples/
I am unable to understand how they replace the '?' with some value. What is the formula for calculating the CNN layer output?
This is essential to be calculated in PyTorch; not so in Tensorflow - Keras. If there is any other blog that explains this well, please drop it in the comments.
# Implementation of CNN/ConvNet Model
class CNN(torch.nn.Module):

    def __init__(self):
        super(CNN, self).__init__()
        # L1 ImgIn shape=(?, 28, 28, 1)
        # Conv -> (?, 28, 28, 32)
        # Pool -> (?, 14, 14, 32)
        self.layer1 = torch.nn.Sequential(
            torch.nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2),
            torch.nn.Dropout(p=1 - keep_prob))
        # L2 ImgIn shape=(?, 14, 14, 32)
        # Conv      ->(?, 14, 14, 64)
        # Pool      ->(?, 7, 7, 64)
        self.layer2 = torch.nn.Sequential(
            torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2),
            torch.nn.Dropout(p=1 - keep_prob))
        # L3 ImgIn shape=(?, 7, 7, 64)
        # Conv ->(?, 7, 7, 128)
        # Pool ->(?, 4, 4, 128)
        self.layer3 = torch.nn.Sequential(
            torch.nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2, padding=1),
            torch.nn.Dropout(p=1 - keep_prob))

        # L4 FC 4x4x128 inputs -> 625 outputs
        self.fc1 = torch.nn.Linear(4 * 4 * 128, 625, bias=True)
        torch.nn.init.xavier_uniform(self.fc1.weight)
        self.layer4 = torch.nn.Sequential(
            self.fc1,
            torch.nn.ReLU(),
            torch.nn.Dropout(p=1 - keep_prob))
        # L5 Final FC 625 inputs -> 10 outputs
        self.fc2 = torch.nn.Linear(625, 10, bias=True)
        torch.nn.init.xavier_uniform_(self.fc2.weight) # initialize parameters

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = out.view(out.size(0), -1)   # Flatten them for FC
        out = self.fc1(out)
        out = self.fc2(out)
        return out

#instantiate CNN model

model = CNN()
model

Thanks!

Output:
I assume you calculation is wrong because:

Pytorch support images in format C * H * W  (e.g. 3x32x32 not 32x32x3)
First dimension always batch dimension and must be omitted in calculation because, all nn.Modules handle it by default

So if you want calculate input size for first Linear layer, you can use this trick:
conv = nn.Sequential(self.layer1,self.layer2, self.layer3, nn.Flatten())
out = conv(torch.randn(1,im_height,im_width).unsqueeze(0))
# fc_layer_in_channels  = out.shape[1]
self.fc1 = torch.nn.Linear(out.shape[1], 625, bias=True)

but only if you know im_height,im_width
The best practice is use  torch.nn.AdaptiveAvgPool2d.
With this layer you always can get output of fixed spatial size.


Instruction: Difference in the order of applying linear decoder and average pooling for sequence models

Input:
I am working with sequence modelling in pytorch and trying to determine if the order of the pooling and linear decoding layer matters. Given that I have a sequence with the shape (Batch, Seqlen, dim_model) and I want to transform it into (Batch, dim_output) I will need a pooling layer for reducing the second dimension (SeqLen) and an affine transformation that maps dim_model to dim_output. Assume Batch = 16, SeqLen = 6000, dim_model = 32, dim_output = 5, we have the following input:
import torch

pooler = lambda x: x.mean(dim=1)
decoder = torch.nn.Linear(32, 5)

x = torch.randn(16, 6000, 32)

Would this:
y = decoder(pooler(x))

Be the same as:
y = pooler(decoder(x))

The normalized difference between both outputs suggest that they are close:
torch.norm(decoder(pooler(x)) - pooler(decoder(x)))

output:
tensor(6.5412e-08, grad_fn=<CopyBackwards>)

But can one say they are equivalent? Are the gradients computed in the same way?
I am interesting the case of using arbitrary pooling layer, this includes for instance the "last" pooler:
pooler = lambda x: x[:,-1]
torch.norm(decoder(pooler(x)) - pooler(decoder(x)))

output:
tensor(0., grad_fn=<CopyBackwards>)

Output:
A linear layer does x -> Ax+b for some matrix A and vector b.
If you have a bunch of x (x1, x2, x3, ..., xn) then A[(x1+...+xn)/n] = (Ax1 +... +Axn)/n, so for mean pooling, applying pooling first and then doing the linear layer results (up to floating point errors) in the same value as applying the linear layer first and then doing the pooling.
For "last pooling", the result is the same because it doesn't matter whether you apply A to every element and then afterwards only pick the final one, or if you pick the final one, and apply A to it.
However, for plenty of  other operations,  the result  would not be the same. E.g. for max pooling, the result would in general not be the same.
e.g. if x1 = (1, 0, 0), x2 = (0, 1, 0), x3 = (0, 0, 1), and A = ((1, 1, 1)) then Ax1 = Ax2 =Ax3 = (1), so applying max pooling after the linear layer just gives you (1),
but max pooling applied to x1, x2, x3 gives you (1, 1, 1) and A(1, 1, 1) = 3.


Instruction: How to insert a value in a fixed positon of pytorch

Input:
I have a PyTorch tensor
x = [[1,2,3,4,5]]

Now I want to add a value to a fixed position of the tensor x, for example, I want to add 11 in position 3 then the x will be
x= [[1,2,3,11,4,5]]

How can I perform this operation in Pytorch?

Output:
Dynamically extending arrays to arbitrary sizes along the non-singleton dimensions, such as the ones you mentioned, are unsupported in PyTorch mainly because the memory is pre-allocated during tensor construction and set to fixed size depending on the data type. The only way to grow non-singleton dimension size is to create a new (empty/zero) tensor with the target shape and insert values at the desired position(s), while also copying values.
In [24]: z = torch.zeros(1, 6)
In [27]: t
Out[27]: tensor([[1, 2, 3, 4, 5]])

In [30]: z[:, :3] = t[:, :3]
In [33]: z[:, -2:] = t[:, -2:]
In [36]: z[z == 0] = 11

In [37]: z
Out[37]: tensor([[ 1.,  2.,  3., 11.,  4.,  5.]])

However, if you'd have instead wanted to expand the tensor along the singleton dimension, then that's easy to achieve using tensor.expand(new_shape). In the below example, we expand the tensor t to length 3 along the 0th dimension, which is originally a singleton dimension.
# make a copy for in-place modification since `expand()` returns a view
In [64]: t_expd = t.expand(3, -1).clone()

In [65]: t_expd
Out[65]:
tensor([[1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5],
        [1, 2, 3, 4, 5]])

# modify 2nd and 3rd rows
In [66]: t_expd[1:, ...] = 23

In [67]: t_expd
Out[67]:
tensor([[ 1,  2,  3,  4,  5],
        [23, 23, 23, 23, 23],
        [23, 23, 23, 23, 23]])


Instruction: RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4

Input:
I am having a hard time understanding image segmentation. I have implemented Unet model for image segmentation. I am using PASCAL VOC dataset and I am trying to train my model. However, I got stuck when calculating the loss. I am unsure of what should be the expected shapes of the output and target classes. Can someone please educate me on what I am doing wrong? My only guess is that I am missing something when it comes to the ground truth images since I don't know how the model will learn which class is which. Thank!
Here is my Unet class:
import torch
import torch.nn as nn
from torchvision import transforms

def x2conv(in_channels, out_channels):
    double_conv = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=0),
        nn.ReLU(inplace=True),
        nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=0),
        nn.ReLU(inplace=True))
    return double_conv

class Encoder(nn.Module):
    def __init__(self, chs):
        super().__init__()
        self.enc_blocks = nn.ModuleList(
            [x2conv(chs[i], chs[i+1]) for i in range(len(chs)-1)])
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

    def forward(self, x):
        ftrs = []
        for block in self.enc_blocks:
            x = block(x)
            ftrs.append(x)
            x = self.pool(x)
        return ftrs

class Decoder(nn.Module):
    def __init__(self, chs):
        super().__init__()
        self.chs = chs
        self.upconvs = nn.ModuleList(
            [nn.ConvTranspose2d(chs[i], chs[i+1], kernel_size=2, stride=2) for i in range(len(chs)-1)])
        self.dec_blocks = nn.ModuleList(
            [x2conv(chs[i], chs[i+1]) for i in range(len(chs)-1)])

    def forward(self, x, encoder_features):
        for i in range(len(self.chs)-1):
            x = self.upconvs[i](x)
            enc_ftrs = self.crop(encoder_features[i], x)
            x = torch.cat([x, enc_ftrs], dim=1)
            x = self.dec_blocks[i](x)
        return x

    def crop(self, enc_ftrs, x):
        _, _, H, W = x.shape
        enc_ftrs = transforms.CenterCrop([H, W])(enc_ftrs)
        return enc_ftrs

class UNet(nn.Module):
    def __init__(self, enc_chs, dec_chs, num_class):
        super(UNet, self).__init__()

        self.encoder = Encoder(enc_chs)
        self.decoder = Decoder(dec_chs)
        self.softmax = nn.Conv2d(dec_chs[-1], num_class, kernel_size=1)

    def forward(self, x):
        enc_ftrs = self.encoder(x)
        out = self.decoder(enc_ftrs[::-1][0], enc_ftrs[::-1][1:])
        out = self.softmax(out)
        return out

And here is my dataset class:
from PIL import Image
import torchvision
VOC_CLASSES = [   # How to use?
    "background",
    "aeroplane",
    "bicycle",
    "bird",
    "boat",
    "bottle",
    "bus",
    "car",
    "cat",
    "chair",
    "cow",
    "diningtable",
    "dog",
    "horse",
    "motorbike",
    "person",
    "pottedplant",
    "sheep",
    "sofa",
    "train",
    "tvmonitor",
]

VOC_COLORMAP = [  # How to use?
    [0, 0, 0],  # Background
    [128, 0, 0],  # Aeroplane
    [0, 128, 0],  # Bicycle
    [128, 128, 0],  # Bird
    [0, 0, 128],  # Boat
    [128, 0, 128],  # Bottle
    [0, 128, 128],  # Bus
    [128, 128, 128],  # Car
    [64, 0, 0],  # Cat
    [192, 0, 0],  # Chair
    [64, 128, 0],  # Cow
    [192, 128, 0],  # Diningtable
    [64, 0, 128],  # Dog
    [192, 0, 128],  # Horse
    [64, 128, 128],  # Motorbike
    [192, 128, 128],  # Person
    [0, 64, 0],  # Pottedplant
    [128, 64, 0],  # Sheep
    [0, 192, 0],  # Sofa
    [128, 192, 0],  # Train
    [0, 64, 128],  # tvmonitor
]

class VocDataset(torchvision.datasets.VOCSegmentation):
    def __init__(self, image_set, transform, root="../data/VOCtrainval_11-May-2012/", download=False, year="2012"):
        self.transform = transform
        self.year = year
        super().__init__(root=root, image_set=image_set,
                         download=download, transform=transform, year=year)

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        # open images and do transformation  img = jpg, mask = png
        img = Image.open(self.images[index]).convert("RGB")
        target = Image.open(self.masks[index]).convert("RGB")

        if self.transform:
            img = self.transform(img)
            trfm = T.Compose([T.ToTensor(), T.Resize((388, 388))])
            target = trfm(target)
        return img, target

and lastly here is my train function
import torch
import torch.nn as nn
import torch.optim as optim
from unet import UNet
from torch.utils.data import DataLoader
from dataset import VocDataset
import torchvision.transforms as T
import torch.nn.functional as F

# Hyperparameters etc.
STD = [0.2686, 0.2652, 0.2812]  # Std for dataset
MEAN = [0.4568, 0.4431, 0.4083]  # Mean for dataset
MOMENTUM = 0.9
LEARNING_RATE = 1e-4
BATCH_SIZE = 32
NUM_EPOCHS = 1
NUM_WORKERS = 2
NUM_CLASSES = 20
TRAIN_SET = "train"
VAL_SET = "val"
ENC_CHANNELS = (3, 64, 128, 256, 512, 1024)  # Encoder channels
DEC_CHANNELS = (1024, 512, 256, 128, 64)  # Decoder channels
TRANSFORM = T.Compose(
    [T.ToTensor(), T.Resize(SIZE), T.Normalize(MEAN, STD)]
)

def main():
    training_data = VocDataset(TRAIN_SET, TRANSFORM)
    train_dataloader = DataLoader(
        training_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKERS, drop_last=True)

    # Create instance of unet
    unet = UNet(ENC_CHANNELS, DEC_CHANNELS, NUM_CLASSES)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(
        unet.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM)

    for epoch in range(NUM_EPOCHS):  # loop over the dataset multiple times
        running_loss = 0.0
        for i, data in enumerate(train_dataloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data  # Shape for labels and inputs are: [32,3,388,388]

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = unet(inputs) # output shape is [32, 32, 388, 388]

            loss = criterion(outputs, labels)  # Error here

            loss.backward()
            optimizer.step()

    # print('Finished Training')

if __name__ == "__main__":
    main()

Output:
For starters, your label and outputs have different dimension. (32 vs 3 channels). Cross Entropy Loss expects them to either have the same number of channels, or for the target to have only one channel with integer values indicating the relevant class.
Let's work with the latter case. In this case, we need to reduce the target to be a single channel [32 x 388 x 388] for your input and batch size. (Secondarily, the Unet should ideally have one output channel for each class (looks like there are 22 classes so you should change the final output layer of the Unet decoder to have 22 outputs)).
To convert the label of size [32 x 3 x 388 x 388] to [32 x  388 x 388], you need to use the colormap for conversion. That is, create a new tensor target of size [32 x 1 x 388 x 388]. For each value target[i,j,k], assign the index into VOC_COLORMAP that matches the value stored in the pixels at label[i,:,j,k].


Instruction: Stable Baselines3 - Setting "manually" the q_values

Input:
What I have done
I'm using the DQN Algorithm in Stable Baselines 3 for a two players board type game. In this game, 40 moves are available, but once one is made, it can't be done again.
I trained my first model with an opponent which would choose randomly its move. If an invalid move is made by the model, I give a negative reward equal to the max score one can obtain and stop the game.
The issue
Once it's was done, I trained a new model against the one I obtained with the first run. Unfortunately, ultimately, the training process gets blocked as the opponent seems to loop an invalid move. Which means that, with all I've tried in the first training, the first model still predicts invalid moves. Here's the code for the "dumb" opponent :
while(self.dumb_turn):
    #The opponent chooses a move
    chosen_line, _states = model2.predict(self.state, deterministic=True)
    #We check if the move is valid or not
    while(line_exist(chosen_line, self.state)):
        chosen_line, _states = model2.predict(self.state, deterministic=True)
    #Once a good move is made, we registered it as a move and add it to the space state
    self.state[chosen_line]=1

What I would like to do but don't know how
A solution would be to set manually the Q-values to -inf for the invalid moves so that the opponent avoid those moves, and the training algorithm does not get stuck. I've been told how to access to these values :
import torch as th
from stable_baselines3 import DQN

model = DQN("MlpPolicy", "CartPole-v1")
env = model.get_env()

obs = env.reset()
with th.no_grad():
     obs_tensor, _ = model.q_net.obs_to_tensor(obs)
     q_values = model.q_net(obs_tensor)

But I don't know how to set them to -infinity.
If somebody could help me, I would be very grateful.

Output:
I recently had a similar problem in which I needed to directly alter the q-values produced by the RL model during training in order to influence its actions.
To do this I overwritten some methods of the library:
# Imports
from stable_baselines3.dqn.policies import QNetwork, DQNPolicy

# Override some methods of the class QNetwork used by the DQN model in order to set to a negative value the q-values of
# some actions

# Two possibile methods to override:
# Override _predict ---> alter q-values only during predictions but not during training
# Override forward ---> alter q-values also during training (Attention: here we are working with batches of q-values)

class QNetwork_modified(QNetwork):

    def forward(self, obs: th.Tensor) -> th.Tensor:
        """
        Predict the q-values.
        :param obs: Observation
        :return: The estimated Q-Value for each action.
        """
        # Compute the q-values using the QNetwork
        q_values = self.q_net(self.extract_features(obs))
        # For each observation in the training batch:
        for i in range(obs.shape[0]):
            # Here you can alter q_values[i]

        return q_values

# Override the make_q_net method of the DQN policy used by the DQN model to make it use the new DQN network

class DQNPolicy_modified(DQNPolicy):
    def make_q_net(self) -> DQNPolicy:
        # Make sure we always have separate networks for features extractors etc
        net_args = self._update_features_extractor(self.net_args, features_extractor=None)
        return QNetwork_modified(**net_args).to(self.device)

model = DQN(DQNPolicy_modified, env, verbose=1)

Personally I don’t like too much this approach, and I would suggest you to try first some “more natural” alternatives, like for examples giving in input to your model also some kind of history of what actions have been already selected, in order to help the model learn that pre-selected actions should be avoided.
For example you could enrich the input for the RL model with an additional binary mask where the moves already chosen have their corresponding bit set to 1. (In this case you should modify the gym environment).


Instruction: My model memory size keeps decreasing and isn't clearing. And as a result, I get a CUDA memory issue

Input:
I'm trying to run a deep network in CUDA/Pytorch. But I keep getting a GPU issue that tells me I'm out of memory in in my GPU as follows:

CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.17 GiB total capacity; 10.62 GiB already allocated; 14.81 MiB free; 10.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

And I get the same sort of error even with an HPC, just that the numbers are bigger.
As far as I can tell, my model is fine and as expected. (My professors told us we can run CIFAR with < 5 M parameters and my model has 4.7 M parameters).
I create data loaders as follows:
transform=transforms.Compose([transforms.ToTensor(),
                             transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                             transforms.CenterCrop(32),
                             transforms.RandomRotation(15)
                             ])
#Augmentation helps prevent overfitting. We use Bilinear and not the default (nearest) since Bilinear is a more efficient technique

batch_size=128

train=datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
test=datasets.CIFAR10(root='./data',train=False,download=True,transform=transform)

#train=train.type(torch.IntTensor)
#train=train.to(uint8)

train_size=int(len(train))
test_size=int(len(test))

train_idx=list(range(train_size))
test_idx=list(range(test_size))

split=int(np.floor(0.77*train_size))

train_idx, val_idx=train_idx[:split] , train_idx[split:]

train_sampler = SubsetRandomSampler(train_idx)
val_sampler = SubsetRandomSampler(val_idx)

train_loader=DataLoader(train,batch_size=batch_size,num_workers=1,sampler=train_sampler)
valid_loader=DataLoader(train,batch_size=batch_size,num_workers=1,sampler=val_sampler)

test_loader=DataLoader(test,batch_size=batch_size,shuffle=False, num_workers=2)

So I split it into a train and test set that are split in a 70/30 fashion.
To run my epochs:
for e in range(epochs):
    running_loss=0.0
    v_loss=0.0
    correct=0
    v_correct=0
    for i,data in enumerate(train_loader):
      print("Ran a batch")
      x,y=data
      x=x.to(device)
      y=y.to(device)
      optimizer.zero_grad()
      y_hat=model(x)
      loss=criterion(y_hat,y) #Comparing prediction to actual output
      loss.backward
      optimizer.step()
      running_loss+=loss
      pred=y_hat.argmax(axis=0) #The index associated with correct prediction
      correct+=y[pred] #Either they match and correct. Or different and wrong
      GPUtil.showUtilization()
      del x
      del y
      GPUtil.showUtilization()
      gc.collect()
      torch.cuda.empty_cache()
      GPUtil.showUtilization()
    print("end of epoch")

In each epoch, I load a batch that I feed to my model. Unfortunatley, it seems like my model can never even train a full batch, that is despite constantly deleting my cache and tensors.
When I look at memory, while GPU utilisation stays balanced and doesn't shoot up, it seems like the memory does and I don't know why that's the case/
From this error, I can see that the memory utilized never clears and it only keeps increasing.

| ID | GPU | MEM |
|  0 | 57% | 76% |

....

|  0 | 68% | 100% |
| ID | GPU | MEM  |

When I look at this problem online, I'm told that decreasing my batch size will help with CUDA running out of memory.
but even when my batch size is 1, the memory keeps growing and causing issues.
As a result, I'm not sure how to solve this problem. I can't tell what's going wrong here.

Output: Maybe typo, but there should be loss.backward() like method calling and loss.backward by itself makes nothing. Also when computing running_loss call .item() method on loss, so full line looks like this running_loss += loss.item(). In discussion there is an answer on why it should reduce memory usage (...cumulative loss will hold reference to each individual loss, which in turn holds references to each of the computation nodes that computed it...).


Instruction: Cuda:0 device type tensor to numpy problem for plotting graph

Input:
as mentioned in the title, I am facing the problem of
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
I found out that that need to be a .cpu() method to overcome the problem, but tried various ways and still unable to solve the problem
def plot(val_loss,train_loss,typ):
    plt.title("{} after epoch: {}".format(typ,len(train_loss)))
    plt.xlabel("Epoch")
    plt.ylabel(typ)
    plt.plot(list(range(len(train_loss))),train_loss,color="r",label="Train "+typ)
    plt.plot(list(range(len(val_loss))),val_loss,color="b",label="Validation "+typ)
    plt.legend()
    plt.savefig(os.path.join(data_dir,typ+".png"))
    plt.close()

Output:
I guess during loss calculation, when you try to save the loss, instead of
train_loss.append(loss)

it should be
train_loss.append(loss.item())

item() returns the value of the tensor as a standard Python number, therefore, train_loss will be a list of numbers and you will be able to plot it.
You can read more about item() here
https://pytorch.org/docs/stable/generated/torch.Tensor.item.html


Instruction: Django web Deployment Failed on azure

Input:
10:47:19 AM django-face-restore: ERROR: Could not find a version that satisfies the requirement torch==TORCH_VERSION+cpu (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
10:47:19 AM django-face-restore: ERROR: No matching distribution found for torch==TORCH_VERSION+cpu
10:47:19 AM django-face-restore: WARNING: You are using pip version 21.1.1; however, version 22.0.4 is available.
10:47:19 AM django-face-restore: You should consider upgrading via the '/tmp/8da12d59fc3e034/antenv/bin/python -m pip install --upgrade pip' command.
10:47:19 AM django-face-restore: "2022-03-31 07:00:00"|ERROR|Failed pip installation with exit code: 1
10:47:21 AM django-face-restore: /opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.8 -i /tmp/8da12d59fc3e034 --compress-destination-dir -p virtualenv_name=antenv --log-file /tmp/build-debug.log
10:47:21 AM django-face-restore: Deployment Failed

Output:
To resolve this ERROR: No matching distribution found for torch==TORCH_VERSION+cpu error:
You need to install the specific version of torch, try either of the following ways:
Add the following to your requirements.txt file:
--find-links https://download.pytorch.org/whl/torch_stable.html

torch==1.7.0+cpu

OR
python -m pip install torch==1.7.0 -f https://download.pytorch.org/whl/torch_stable.html

OR
python3.8 -m pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

References: Install PyTorch from requirements.txt , ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu
, and Could not find a version that satisfies the requirement torch==1.3.1


Instruction: How can I implement a simple dot counting problem using a regression instead of a classification approach in Pytorch?

Input:
I'm trying to figure out how to solve the really simple problem of counting how many pixels in an image are white.
I have 20x20 pixel images (zeros matrix) with 1 to 20 pixels randomly set to 1.
Checking some tutorials Im able to solve this problem via a classification approach with the model using CrossEntropyLoss() and Adam optimizer:
 nn.Linear(20*20, 100),
 nn.ReLU(),
 nn.Linear(100, 30),
 nn.ReLU(),
 nn.Linear(30, 20)```

Here my output is a vector of size 20 with the probabilities that the image contains 1 to 20 white pixels (ones)
Using argmax() I can simply retreive the most possible "class"
What I wanna do now is to solve the same problem but not as a classification one, but like a "counter", where, wiht the image as imput, the NN can estimate the number of white pixels as one single Integer output.
I've made some changes to the code using one neuron as output and MSELoss() and other functions as loss functions, but still cannot get the output to be an integer (1 to 20) instead of a real number
At the end, what I try to understand is how to see a NN as a function, that, in this case, goes from Rn to R
Any ideas?
Using one neuron as output, the predictions are real numbers and the accuracy is always around 4.9% with a loss that keeps quite constant across epochs

Output: Keep in mind that you probably have to normalize the outputs. So your model should still output something between 0 and 1 where 0 means 0 white pixels, 1 means 20 white pixels and 0.5 means 10 and so on. Therefore use sigmoid on the output neuron and mulltiply it by 20 to get the estimated amount of white pixels.


Instruction: PyTorch adapt binary classification model to output probabilities of both classes

Input:
My dataset has 14 features and a target containing {0,1}.
I have trained this binary classifier:
class SimpleBinaryClassifier(nn.Module):
    def __init__(self,input_shape):
        super().__init__()
        self.fc1 = nn.Linear(input_shape,64)
        self.fc2 = nn.Linear(64,32)
        self.dropout = nn.Dropout(p=0.1)
        self.fc3 = nn.Linear(32,1)

    def forward(self,x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.dropout(x)
        x = self.fc3(x)
        return x

with the following criterion and training loop:
criterion   = nn.BCEWithLogitsLoss()

def binary_acc(y_pred, y_test):
    y_pred_tag = torch.round(torch.sigmoid(y_pred))
    correct_results_sum = (y_pred_tag == y_test).sum().float()
    acc = correct_results_sum/y_test.shape[0]
    acc = torch.round(acc * 100)

    return acc

model.train()
for e in range(1, EPOCHS+1):
    epoch_loss = 0
    epoch_acc = 0
    for X_batch, y_batch in train_loader:
        X_batch, y_batch = X_batch.to(device), y_batch.to(device)
        optimizer.zero_grad()

        y_pred = model(X_batch)

        loss = criterion(y_pred, y_batch.unsqueeze(1))
        acc = binary_acc(y_pred, y_batch.unsqueeze(1))

        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    print(f'Epoch {e+0:03}: | Loss: {epoch_loss/len(train_loader):.5f} | Acc: {epoch_acc/len(train_loader):.3f}')

This model, when called like sigmoid(model(input_tensor)) outputs a single number in [0,1]. The pipeline I'm working with, expects a model to output probabilities [p_class1, p_class2].
How can I adapt the model and the training loop?
If I set the output of the last layer to 2, I have problems with the criterion inside the training loop.
class SimpleBinaryClassifier2(nn.Module):
    def __init__(self,input_shape):
        super().__init__()
        self.fc1 = nn.Linear(input_shape,64)
        self.fc2 = nn.Linear(64,32)
        self.dropout = nn.Dropout(p=0.1)
        self.fc3 = nn.Linear(32,2) # now it's 2

    def forward(self,x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.dropout(x)
        x = self.fc3(x)
        return x

I use the CrossEntropy
model = SimpleBinaryClassifier2(input_shape=14)
model.to(device)
print(model)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
criterion   = nn.CrossEntropyLoss()

and replace y_pred_tag = torch.round(torch.sigmoid(y_pred)) with argmax(softmax)
def binary_acc2(y_pred, y_test):
    y_pred_tag = torch.argmax(torch.softmax(y_pred), dim=1)
    correct_results_sum = (y_pred_tag == y_test).sum().float()
    acc = correct_results_sum/y_test.shape[0]
    acc = torch.round(acc * 100)
    return acc

Then the train loop rises an error:
model.train()
for e in range(1, EPOCHS+1):
    epoch_loss = 0
    epoch_acc = 0
    for X_batch, y_batch in train_loader:
        X_batch, y_batch = X_batch.to(device), y_batch.to(device)
        optimizer.zero_grad()

        y_pred = model(X_batch)

        loss = criterion(y_pred, y_batch)
        acc = binary_acc(y_pred, y_batch)

        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    print(f'Epoch {e+0:03}: | Loss: {epoch_loss/len(train_loader):.5f} | Acc: {epoch_acc/len(train_loader):.3f}')

The error is the following:
RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Float'

I already looked up on this other post where the cause of that error was that the element was a Float and not a tensor, but in my case the datasets are tensors:
train_dataset = GenericDataset(torch.FloatTensor(X_train), torch.FloatTensor(y_train))
test_dataset = GenericDataset(torch.FloatTensor(X_test), torch.FloatTensor(y_test))

Output: According to nn.CrossEntropyLoss description it expects target as long and not float, while in your train_dataset you clearly convert it to float


Instruction: What is the significance of 1.000000015047466e+30?

Input:
I was attempting to do JIT compilation on a pytorch-based module from an NLP library and I saw that one of the generated fused CUDA kernel code implementations mentions the number 1.000000015047466e+30:
#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)

template<typename T>
__device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a > b ? a : b);
}

template<typename T>
__device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a < b ? a : b);
}

extern "C" __global__
void fused_add_mul_mul_sub(float* tattn_mask1_1, float* tac_2, float* tbd_2, float* output_1, float* aten_mul_1) {
{
if (blockIdx.x<1ll ? 1 : 0) {
if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<169ll ? 1 : 0) {
if (blockIdx.x<1ll ? 1 : 0) {
        float v = __ldg(tattn_mask1_1 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
        aten_mul_1[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = v * 1.000000015047466e+30f;
      }    }  }if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<2028ll ? 1 : 0) {
    float v_1 = __ldg(tac_2 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
    float v_2 = __ldg(tbd_2 + 12ll * (((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) % 169ll) + ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) / 169ll);
    float v_3 = __ldg(tattn_mask1_1 + ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) % 169ll);
    output_1[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = (v_1 + v_2) * 0.125f - v_3 * 1.000000015047466e+30f;
  }}
}

It feels like this should be some sort of FLOAT_MAX constant, but I don't know any numerical type that would have this as a limit.
Google searching for this constant just yields a handful of results that seem to suggest it may be a physical constant of some kind:

This MATLAB forum link suggests it may be a default value for some physical phenomena
This Github repo on whole cell electrophysiology seems to use it as a max limit of some kind.
This Opensea NFT link shows that the number is somehow used as a parameter for some kind of fractal art?

I'm absolutely baffled because I'm doing natural language processing using CUDA and have absolutely no clue why this number is appearing in my CUDA kernel code and why it's also being used in hard science research code. Does this number have some special floating point properties or something? Is it serving as a numerical stability factor somehow? Any leads would be greatly appreciated.

Output:
1.000000015047466e+30f is 1030 rounded to the nearest value representable in float (IEEE-754 binary32, also called “single precision”), then rounded to 16 decimal digits, then formatted as a float literal.
Thus, it likely originated as 1e30 or another representation of 1030 that was subsequently converted to float and then used to produce new source code.
(The float nearest 1030 is exactly 1000000015047466219876688855040.)


Instruction: Object detection from synthetic to real life data with Yolov5

Input:
Currently trying yolov5 with custom synthetic data. The dataset we've created consists of 8 different objects. Each object has a minimum of 1500 pictures/labels, where the pictures are split 500/500/500 of normal/fog/distractors around object. Sample images from the dataset is in the first imgur link. The model is not trained from scratch, but from yolov5 standard .pt.
So far we've tried:

Adding more data (from 300 images per object, to 4500)
Creating more complex data (distractors on/around objects)
Running multiple runs of training
Trained with network size small, medium, large, xlarge
Different batch size between 4-32 (depending on model size)

Everything so far has resulted in good/great detection on synthetic data, but completely off when used on real-life data.
Examples: Thinks that the whole pictures of unrelated objects is a paperbox, walls are pallets, etc. Quick sample images in the last imgur link.
Anyone got clues for how to improve the training or data to be better suited for real life detection? Or how to better interpret the results? I don't understand how the model draws the conclusion that a whole picture, with unrelated objects, is a box/pallet.
Results from training uploaded to imgur:
https://imgur.com/a/P0TQeBl
Example on real life data:
https://imgur.com/a/SGY7w8w

Output:
There are couple of things to improve results.

After training your model with synthetic data, fine tune your model with real training data, with a smaller learning rate (1/10th maybe). This will reduce the gap between synthetic and real life images. In some cases rather than fine tuning, training the model with mixed (synthetic+real) produces better results.
Generate images structurally similar to real life examples. For example, put humans inside forklifts, or pallets or barrels on forks, etc. Models learn from it.
Randomize the texture on items that you want to detect. Models tend to focus on textures for detection. By randomizing textures, with lots of variability including mon natural occurrences, you force model to learn to identify objects not based on its textures. Although, texture of an object sometimes is a good identifier, synthetic data suffers from not replicating that feature good enough, hence the domain gap, so you reduce its impact on model decision.
I am not sure whether the screenshot accurately represent your data generation distribution, if so, you have to randomize the angles of objects, sizes and occlusion amounts more.
Use objects that you don’t want to detect but will be in the images you will do inference as distractors, rather than simple shapes like spheres.
Randomize lighting more. Intensity, color, angles etc.
Increase background and ground randomization. Use hdris, there are lots of free hdris
Balance your dataset

https://imgur.com/a/LdCa8aO


Instruction: Input and Output to the lstms in pytorch

Input:
I want to implement lstms with CNN in pytorch as my data is a time series data i.e. frames of video for heart rate detection, I am struggling with the input and output dimensions for lstms what and how i should properly configure the dimensions/parameters/arguments at input of lstms in pytorch as its quite confusing when considering time steps, hidden state etc.
my output from CNN is “2 batches of 256 frames”, which is now the input to lstms
batch is 2
features =256
the output is also a batch of 2 with 256 frames.

Output:
Generally, the input shape of sequential data takes the form (batch_size, seq_len, num_features). Based on your explanation, I assume your input is of the form (2, 256), where 2 is the batch size and 256 is the sequence length of scalars (1-dimensional tensor). Therefore, you should reshape your input to be (2, 256, 1) by inputs.unsqueeze(2).
To declare and use an LSTM model, simply try
from torch import nn

model = nn.LSTM(
    input_size=1, # 1-dimensional features
    batch_first=True, # batch is the first (zero-th) dimension
    hidden_size=some_hidden_size, # maybe 64, 128, etc.
    num_layers=some_num_layers, # maybe 1 or 2
    proj_size=1, # output should also be 1-dimensional
)
outputs, (hidden_state, cell_state) = model(inputs)


Instruction: TypeError: forward() missing 1 required positional argument in a method

Input:
I use the following model:

model = DeepGraphInfomax(128, pos_summary_t).to(device)

which looks like:

class DeepGraphInfomax(torch.nn.Module):
def __init__(self, hidden_channels, pos_summary):#, encoder):#, summary, corruption):
    super().__init__()
    self.hidden_channels = hidden_channels
    #self.encoder = GCNEncoder() # needs to be defined either here or give it to here (my comment)
    self.pos_summary = pos_summary
    #self.corruption = corruption

    self.weight = Parameter(torch.Tensor(hidden_channels, hidden_channels))

    #self.reset_parameters()

def forward(self, pos_summary, *args, **kwargs):

    #pos_z = self.encoder(*args, **kwargs)
    #cor = self.corruption(*args, **kwargs)
    #cor = cor if isinstance(cor, tuple) else (cor, )
    #neg_z = self.encoder(*cor)
    summary = self.summary(pos_summary, *args, **kwargs)
    return summary# pos_z#, neg_z, summary

but running model() gives me the error:

TypeError: forward() missing 1 required positional argument:
'pos_summary'

Output:
model is an object since you instantiated DeepGraphInfomax.
model() calls the .__call__ function.
forward is called in the .__call__ function i.e. model().
Have a look at here.
The TypeError means that you should write input in forward function i.e.  model(data).
Here is an exmaple:

import torch
import torch.nn as nn

class example(nn.Module):
    def __init__(self):
        super().__init__()
        self.mlp = nn.Sequential(nn.Linear(5,2), nn.ReLU(), nn.Linear(2,1), nn.Sigmoid())

    def forward(self, x):
        return self.mlp(x)

# instantiate object
test = example()

input_data = torch.tensor([1.0,2.0,3.0,4.0,5.0])

# () and forward() are equal
print(test(input_data))
print(test.forward(input_data))

# outputs for me
#(tensor([0.5387], grad_fn=<SigmoidBackward>),
# tensor([0.5387], grad_fn=<SigmoidBackward>))


Instruction: How to Reverse Order of Rows in a Tensor

Input:
I'm trying to reverse the order of the rows in a tensor that I create. I have tried with tensorflow and pytorch. Only thing I have found is the torch.flip() method. This does not work as it reverses not only the order of the rows, but also all of the elements in each row. I want the elements to remain the same. Is there an array operation of this to index the integers? For instance:
 tensor_a = [1, 2, 3]
       [4, 5, 6]
       [7, 8, 9]

I want it to be returned as:
       [7, 8, 9]
       [4, 5, 6]
       [1, 2, 3]

 however, torch.flip(tensor_a) =
       [9, 8, 7]
       [6, 5, 4]
       [3, 2, 1]

Anyone have any suggestions?

Output: According to documentation torch.flip has argument dims, which control what axis to be flipped. In this case torch.flip(tensor_a, dims=(0,)) will return expected result. Also torch.flip(tensor_a) will reverse all tensor, and torch.flip(tensor_a, dims=(1,)) will reverse every row, like [1, 2, 3] --> [3, 2, 1].


Instruction: ConvNeXt torchvision - specify input channels

Input:
How do I change the number of input channels in the torchvision ConvNeXt model? I am working with grayscale images and want 1 input channel instead of 3.
import torch
from torchvision.models.convnext import ConvNeXt, CNBlockConfig

# this is the given configuration for the 'tiny' model
block_setting = [
    CNBlockConfig(96, 192, 3),
    CNBlockConfig(192, 384, 3),
    CNBlockConfig(384, 768, 9),
    CNBlockConfig(768, None, 3),
]

model = ConvNeXt(block_setting)

# my sample image (N, C, W, H) = (16, 1, 50, 50)
im = torch.randn(16, 1, 50, 50)
# forward pass
model(im)

output:
RuntimeError: Given groups=1, weight of size [96, 3, 4, 4], expected input[16, 1, 50, 50] to have 3 channels, but got 1 channels instead

However, if I change my input shape to (16, 3, 50, 50) it seems to work fine.
The torchvision source code seems to be based of their github implementation but where do I specify in_chans with the torchvision interface?

Output:
You can rewrite the whole input layer, model._modules["features"][0][0] is
nn.Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4))

Then, you only need to change the in_channels
>>> model._modules["features"][0][0] = nn.Conv2d(1, 96, kernel_size=(4, 4), stride=(4, 4))
>>> model(im)
tensor([[-0.4854, -0.1925,  0.1051,  ..., -0.2310, -0.8830, -0.0251],
        [ 0.3332, -0.4205, -0.3007,  ...,  0.8530,  0.1429, -0.3819],
        [ 0.1794, -0.7546, -0.7835,  ..., -0.8072, -0.0972,  0.7413],
        ...,
        [ 0.1356,  0.0868,  0.6135,  ..., -0.1382, -0.2001,  0.2415],
        [-0.1612, -0.4812,  0.1271,  ..., -0.6594,  0.2706,  1.0833],
        [ 0.0243, -0.5039, -0.4086,  ...,  0.4233,  0.0389,  0.2787]],
       grad_fn=<AddmmBackward0>)


Instruction: Float64 Normalisation in pytorch

Input:
So when I want to do normalization to float64 in deep learning, I need to make float64_max to be 1 or just every image’s max value as 1?
I read a .nii file to get a 3D array with type float64 and its value is very big. I need to normalize it into 0-1 and float32 type to input my deep learning model. So I was wondering which normalization way is better or correct.
Make float64_max to be 1:
return torch.tensor(img / sys.float_info.max).to(torch.float32)

Make every image's max value as 1:
return torch.nn.functional.normalize(torch.tensor(img)).to(torch.float32)

Also, because the float64 range is much bigger than float32, when I do return torch.tensor(img / sys.float_info.max).to(torch.float32), the value have a chance to be all 0.

Output:
No, you should normalize the max possible value of all images to 1 not the maximum value of each image separately.
For a general image RGB 8-bit image, the maximum value for a pixel channel is 255, so you could divide the entire image's channel value by 255.
In this way, you will obtain a new image of type float32 (or float64) with all pixel's channel values in the range [0, 1], but the min is not always 0 and max is not always 1.


Instruction: CUDA 11.3 not being detected by PyTorch [Anaconda]

Input:
I am running Ubuntu 20.04 on GTX 1050TI. I have installed CUDA 11.3.
nvidia-smi output:
Wed Apr  6 18:27:23 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   44C    P8    N/A /  N/A |     11MiB /  4040MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3060      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      4270      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

nvcc --version output:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

Anaconda PyTorch isn't detecting CUDA:
> import torch
> torch.cuda.is_available()
> False

Any ideas how to solve the issue?

Output:
The solution:
Conda in my case installed cpu build. You can easily identify your build type by running torch.version.cuda which should return a string in case you have the CUDA build. if you get None then you are running the cpu build and it will not detect CUDA
To fix that I installed torch using pip instead :
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113


Instruction: why does pytorch's utils.save_image() change the color of my image

Input:
I am saving two images with pytorch's utils.save_image() function here. one is the real image and the other, the perturbed image (just a patch). However, the latter image lost its real appearance when saved with save_image().
# save source image
utils.save_image(data.data, "./%s/%d_%d_org.png" % ("log", batch_idx, labels),
                            normalize=True)

# save adv image
utils.save_image(noised_data.data, "./%s/%d_%d_adv.png" % ("log", batch_idx, target_class), normalize=True)

This is the saved clean image
This is the saved perturbed image

Output:
So, I managed to solve this by adding make_grid() to the save method and clipping the image to [0,1] without normalizing.
utils.save_image(utils.make_grid(torch.clip(noised_data.data, 0, 1)), "./%s/%d_%d_adv.png" % ("log", batch_idx, target_class))


Instruction: Combine multiple DataLoaders sequentially

Input:
I'm interested in how I'd go about combining multiple DataLoaders sequentially for training. I understand I can use ConcatDataset to combine datasets first, but this does not work for my use case. I have a custom collate_fn that is passed to each dataloader, and this function depends on an attribute of the underlying Dataset. So, I'll have a set of custom DataLoaders like the following:
def custom_collate(sample, ref):
    data = clean_sample(torch.stack([x[0] for x in sample]), ref)
    labels = torch.tensor([x[1] for x in sample])
    return data, labels

class CollateLoader(torch.utils.data.DataLoader):
    def __init__(self, ref, *args, **kwargs):
        collate_fn = functools.partial(custom_collate, ref=ref)
        super().__init__(collate_fn = collate_fn, *args, **kwargs)

Where ref is a property of the custom Dataset class and is passed on initialization of a CollateLoader. Also, I know transforms can be applied in the Dataset, but in my case it must be done batch-wise.
So, how would I go about combining multiple DataLoaders? In the PyTorch-Lightning LightningDataModule, we can do something like
def train_dataloader(self):
    return [data_loader_1, data_loader_2]

But this will return a list of batches, not the batches sequentially.

Output:
I ran into the same problem and found a workaround. I overrided the epoch training loop using the Loops API from PytorchLightning, defining a class CustomLoop which inherits from pytorch_lightning.loops.TrainingEpochLoop, and overrided the advance() method. I copy pasted the source code from pytorch_lightning and replaced these lines with:
if not hasattr(self,'dataloader_idx'):
    self.dataloader_idx=0
if not isinstance(data_fetcher, DataLoaderIterDataFetcher):
    batch_idx = self.batch_idx + 1
    batch = next(data_fetcher.dataloader.loaders[self.dataloader_idx])
    self.dataloader_idx+=1
    if self.dataloader_idx == len(data_fetcher.dataloader.loaders):
        self.dataloader_idx = 0
else:
    batch_idx, batch = next(data_fetcher)

That way, instead of iterating over the CombinedLoader, i make it iterate over one dataloader at a time.
Then, to make use of this custom loop you have to replace the default loop in the Trainer:
trainer.fit_loop.replace(epoch_loop=CustomLoop)
trainer.fit(my_model)


Instruction: pytorch Error: module 'torch.nn' has no attribute 'ReLu'

Input:
i am working in google colab, so i assume its the current version of pytorch.
I tried this:
class Fc(nn.Module):
    def __init__(self):
        super(Fc, self).__init__()
        self.flatt = nn.Flatten()
        self.seq = nn.Sequential(nn.Linear(28*28, 512),
                                 nn.ReLU(),
                                 nn.Linear(512, 512),
                                 nn.ReLu(),
                                 nn.Linear(512, 10), nn.ReLu())

    def forward(x):
        p = self.flatt(x)
        p = self.seq(p)
        return p
m1 = Fc()

and got:
<ipython-input-85-142a1e77b6b6> in <module>()
----> 1 m1 = Fc()
<ipython-input-84-09df3be0b613> in __init__(self)
  4         self.flatt = nn.Flatten()
  5         self.relu = torch.nn.modules.activation.ReLU()
----> 6         self.seq = nn.Sequential(nn.Linear(28*28, 1012), nn.ReLU(),
nn.Linear(1012, 512), nn.ReLu(), nn.Linear(512, 10), nn.ReLu())
AttributeError: module 'torch.nn' has no attribute 'ReLu'

What I am doing wrong here?

Output:
You got a typo regarding casing. It's called ReLU not ReLu.
import torch.nn as nn

class Fc(nn.Module):
    def __init__(self):
        super(Fc, self).__init__()
        self.flatt = nn.Flatten()
        self.seq = nn.Sequential(nn.Linear(28*28, 512),
                                 # TODO: Adjust here
                                 nn.ReLU(),
                                 nn.Linear(512, 512),
                                 nn.ReLU(),
                                 # TODO: Adjust here
                                 nn.Linear(512, 10), nn.ReLU())

    def forward(x):
        p = self.flatt(x)
        p = self.seq(p)
        return p
m1 = Fc()


Instruction: Inputing a torch 3d tensor into a keras.Sequential model

Input:
Here I have a pytorch tensor object which I need to use for training a neural network. Can pytorch tensors be used for training a keras neural network and if so, what will be the value of the input_shape parameter when the training data has 3 dimensions?
Here's the code:
image, label = val_dset.__getitem__(20)
print(image, label)

Output:
tensor([[[0.8549, 0.8549, 0.8588,  ..., 0.8667, 0.8627, 0.8627],
         [0.8549, 0.8549, 0.8588,  ..., 0.8667, 0.8667, 0.8667],
         [0.8549, 0.8549, 0.8549,  ..., 0.8667, 0.8667, 0.8667],
         ...,
         [0.1216, 0.1412, 0.1686,  ..., 0.5490, 0.7529, 0.5686],
         [0.1451, 0.1843, 0.2667,  ..., 0.4510, 0.6627, 0.5765],
         [0.3098, 0.4078, 0.5098,  ..., 0.3529, 0.4588, 0.4431]],

        [[0.8549, 0.8549, 0.8588,  ..., 0.8902, 0.8824, 0.8824],
         [0.8549, 0.8549, 0.8588,  ..., 0.8863, 0.8824, 0.8824],
         [0.8549, 0.8549, 0.8588,  ..., 0.8863, 0.8824, 0.8784],
         ...,
         [0.1294, 0.1451, 0.1725,  ..., 0.4745, 0.6824, 0.4902],
         [0.1451, 0.1843, 0.2627,  ..., 0.3804, 0.5804, 0.5098],
         [0.2863, 0.3686, 0.4745,  ..., 0.3020, 0.3843, 0.3922]],

        [[0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8627, 0.8627],
         [0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8667, 0.8667],
         [0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8667, 0.8627],
         ...,
         [0.1294, 0.1451, 0.1765,  ..., 0.3804, 0.5294, 0.3882],
         [0.1490, 0.1882, 0.2627,  ..., 0.3137, 0.4588, 0.4039],
         [0.2902, 0.3647, 0.4588,  ..., 0.2471, 0.3176, 0.3059]]]) 1

image.shape

Output:
torch.Size([3, 224, 224])

For the network:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, callbacks, Input

model = keras.Sequential([
    Input(shape=(_)),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(512, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(26, activation='softmax')
])

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

early_stopping = keras.callbacks.EarlyStopping(
    patience=50,
    min_delta=0.005,
    restore_best_weights=True
)

Output:
You can get .numpy() then convert to tensor like below:
>>> torch_image = torch.rand(3, 224, 224)
>>> torch_image.shape
torch.Size([3, 224, 224])

>>> tf_image = tf.convert_to_tensor(torch_image.numpy())
>>> tf_image
<tf.Tensor: shape=(3, 224, 224), dtype=float32, numpy=
array([[[0.7935423 , 0.7479191 , 0.976204  , ..., 0.31874692,
         0.70065683, 0.77162033],
        ...,
        [0.19848353, 0.41827488, 0.5245047 , ..., 0.28861862,
         0.5350955 , 0.6847391 ],
        [0.57963634, 0.8628217 , 0.0179103 , ..., 0.19654012,
         0.38167596, 0.5232694 ]]], dtype=float32)>

Or If you have multiple images then want to convert to tensor and train your model, you can use tf.data.Dataset.from_tensor_slices this function by default does this for you and convert torch to tensor like below:
(In the below example I create 100 random arrays as images)
import torch
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, callbacks, Input

torch_image = torch.rand(100, 3, 224, 224)
torch_label = torch.randint(1,5,(100,))

dataset = tf.data.Dataset.from_tensor_slices((torch_image, torch_label))
dataset = dataset.repeat(10).batch(5)

# for check type Tensor or Torch
# tf_image , label_image = next(iter(dataset))
# >>> tf_image
# <tf.Tensor: shape=(5, 3, 224, 224), dtype=float32, numpy=...>

model = keras.Sequential([
    Input(shape=(3,224,224,)),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(1, activation='softmax')
])

model.compile(
    loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

early_stopping = keras.callbacks.EarlyStopping(
    patience=50, min_delta=0.005, restore_best_weights=True)

model.fit(dataset, epochs=10)

Output:
Epoch 1/10
200/200 [==============================] - 7s 37ms/step - loss: 0.0000e+00 - accuracy: 0.2500


Instruction: How can I sum parts of pytorch tensor of variable sizes?

Input:
Let's consider example.
I have a tensor of size (10, 3).
I want to sum first 3 rows, next 2 rows and 5 next rows by 0 axis.
For example from:
t = torch.ones([10, 3])

I want to get:
[
    [3.0, 3.0, 3.0],
    [2.0, 2.0, 2.0],
    [5.0, 5.0, 5.0],
]

I want to specify a tensor with values and a tensor with part sizes and possibly axis and get a tensor summed along this axis by parts of specified sizes.
How can I achieve that?

Output:
Following the great idea of @ben-grossmann I modified it a little to use sparse tensor and make it more efficient. And implemented it as a function:
def sum_var_parts(t, lens):
    t_size_0 = t.size(0)
    ind_x = torch.repeat_interleave(torch.arange(lens.size(0)), lens)
    indices = torch.cat(
        [
            torch.unsqueeze(ind_x, dim=0),
            torch.unsqueeze(torch.arange(t_size_0), dim=0)
        ],
        dim=0
    )
    M = torch.sparse_coo_tensor(
        indices,
        torch.ones(t_size_0, dtype=torch.float32),
        size=[lens.size(0), t_size_0]
    )
    return M @ t


Instruction: How To Return Incorrectly Predicted Images

Input:
I am having trouble figuring out how to create a list containing the first 10 image IDs that were incorrectly predicted.
import os
import torch
import torchvision
from torch.utils.data import random_split
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
from torch.utils.data.dataloader import
DataLoader
import torch.nn as nn
import torch.nn.functional as F
batch_size=128

def predict_image(img, model):
    # Convert to a batch of 1
    xb = to_device(img.unsqueeze(0), device)
    # Get predictions from model
    yb = model(xb)
    # Pick index with highest probability
    _, preds  = torch.max(yb, dim=1)
    # Retrieve the class label
    return dataset.classes[preds[0].item()]

def invalid_predictions(n=10):
    invalid_ids = []
    while invalid_ids
    return invalid_ids
    # This method should return a list of first
    # 10 image ids that the model could not
    # predict correctly.
    # For example [40, 35, 20, ...]

Output:
def invalid_predictions(n=10, images, labels):
    invalid_ids = []
    image_count = 0
    invalid_count = 0
    while invalid_count < n:
        prediction = predict_image(images[image_count], model)
        if prediction != labels[image_count ]:
            invalid_ids.append(image_count )
            invalid_count +=1
        image_count += 1
    return invalid_ids


Instruction: how to overfit a model on a single batch in keras?

Input:
I am trying to overfit my model on a single batch to check model integrity. I am using Keras and TensorFlow for the implementation of my model and coding style for this project.
I know how to get the single batch and overfit the model in PyTorch but don't have an idea in Keras.
to get a single batch in PyTorch I used:

images, labels = next(iter(train_dataset))
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 0.0001)

for epoch in range(epochs):
    print(f"Epoch [{epoch}/{epochs}]")
    # for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)
    data = data.reshape(data.shape[0], -1)
    # forward
    score = model(data)
    loss = criterion(score, target)
    print(f"Loss: {loss.item()}")
        # backward
    optimizer.zero_grad()
    loss.backward()

    optimizer.step()

How to do it in keras any helping matrial?

Output:
Thank you everyone for coming here. I found a solution and here it is:
datagen = ImageDataGenerator(rescale=1 / 255.0,
                            rotation_range=20,
                            zoom_range=0.2,
                            width_shift_range=0.05,
                            height_shift_range=0.05,
                            shear_range=0.2,
                            horizontal_flip=True,
                            fill_mode="nearest"
                            )
# preprocessing_function=preprocess_input,
# Declare an image generator for validation & testing without generation
test_datagen = ImageDataGenerator(rescale = 1./255,)#preprocessing_function=preprocess_input

# Declare generators for training, validation, and testing from DataFrames
train_gen = datagen.flow_from_directory(directory_train,
                                        target_size=(512, 512),
                                        color_mode='rgb',
                                        batch_size=BATCH_SIZE,
                                        class_mode='binary',
                                        shuffle=True)

val_gen = test_datagen.flow_from_directory(directory_val,
                                        target_size=(512, 512),
                                        color_mode='rgb',
                                        batch_size=BATCH_SIZE,
                                        class_mode='binary',
                                        shuffle=False)

test_gen = test_datagen.flow_from_directory(directory_test,
                                        target_size=(512, 512),
                                        color_mode='rgb',
                                        batch_size=BATCH_SIZE,
                                        class_mode='binary',
                                        shuffle=False)
train_images, train_labels = next(iter(train_gen))
val_images, val_labels = next(iter(val_gen))
test_images, test_labels = next(iter(val_gen))

#check shape for selected Batch

print("Length of Train images : {}".format(len(train_images)))
print("shape of Train images : {}".format(train_images.shape))
print("shape of Train labels : {}".format(train_labels.shape))

Length of Train images : 32
shape of Train images : (32, 512, 512, 3)
shape of Train labels : (32,)

history = model.fit(train_images, train_labels,
                            use_multiprocessing=True,
                            workers=16,
                            epochs=100,
                            class_weight=class_weights,
                            validation_data=(val_images, val_labels),
                            shuffle=True,
                            callbacks=call_backs)


Instruction: How can I use 2 images as a training sample in PyTorch？

Input:
I just begin learning deep learning and my first homework is to finish an leaves-classification system based on convolutional neural networks.I built a resnet-34 model with the code on github to do it.However,my teacher told me that the basic training unit in his dataset is an image pair.I should use 2 images(photos of the same leaf under different light conditions) as the input,combining two 3-channel images into one 6-channel image,but I don't know how to input 2 images and combine them into 6 channels.How can I do that?Are there any functions?Should I modify the structure of the resnet network?

this is my dataset,you can see every two images are about the same leaf.

Output:
You have several issues to tackle:

You need a Dataset with a __getitem__ method that returns 2 images (and a label) instead of the basic ones that returns a single image and a label. You'll probably need to customize your own dataset.
Make sure the augmentations you apply to your images are applied in the same manner to each pair.
You need to modify ResNet-34 network to get as an input 2 images, instead of one. See, e.g., this answer how that can be done.
You need to change the first convolution layer to have 6 input channels instead of 3.
If you want to use pre-trained weights you will not be able to load the existing state_dict of ResNet34 because of changes #3 and #4 - you'll have to do it manually for the first time.


Instruction: Combination of features of convolutional layers channel-by-channel in a multi-branch model

Input:
The convolutional model presented below, has two branches and each branch  (for example) has two stages (convolutional layers).
My aim is to combine the weighted feature maps (channels) of the first convolutional layer from the second branch with the channels of the first convolutional layer from the first branch.
I want to extract the channels from the first convolutional layer in the second branch, multiply it by a weight (weight is a class in the code that makes the output a weighted version of its input) and stack it with the channels of its counterpart convolutional layer from the first branch. Afterwards, by utilizing a 1x1 conv2d, the size of the stacked feature maps will be changed to its initial size and this combined channels should be used by the first branch and the next convolutional layers will be computed based on these combined channels. After that, I want to have this kind of combination between the second convolutional layers of the branches. (In other words, I want to combine features channel-by-channels between branches.)
Please find the main_class (the whole model that consists of two branches) and the first_branch and second_branch below:
class main_class(nn.Module):
    def __init__(self, pretrained=False):
        super(main_class, self).__init__()
        self.input=input_data()  # input_data is a class the provides the input data for the each branch

        self.conv_t2 = BasicConv3d(...........)
        self.second_branch=second_branch(512, out_sigmoid=True)

        self.conv_t1 = BasicConv3d(..............)
        self.first_branch=first_branch(512, out_sigmoid=True)

        self.last = nn.Conv2d(4, 1, kernel_size=1, stride=1)

        self.sigmoid = nn.Sigmoid()

        def forward(self, x, par = False):

         x1, x2 = self.input(x)

        #second branch
        y2 = self.conv_t2(x2)
        out2 = self.second_branch(y2)

        #first branch
        y1 = self.conv_t1(x1)
        out1 = self.first_branch(y1)

        x = torch.cat((out2, out1), 1)
        x = self.last(x)
        out = self.sigmoid(x)

        if par:
            return  out1, out2, out

        return out

The first_branch:
class first_branch(nn.Module):
    def __init__(self, in_channel=512, out_channel=[380, 200], out_sigmoid=False):
        super(first_branch, self).__init__()

        self.out_sigmoid=out_sigmoid

        self.deconvlayer1_2 = self._make_deconv(in_channel, out_channel[0], num_conv=3)
        self.upsample1_2=Upsample(scale_factor=2, mode='bilinear')
        self.combined1_2 = nn.conv2d(720, 380, kernel_size=1, stride=1, padding=0)

        self.deconvlayer1_1 = self._make_deconv(out_channel[0], out_channel[1], num_conv=3)
        self.upsample1_1=Upsample(scale_factor=2, mode='bilinear')
        self.combined1_1 = nn.conv2d(400, 200, kernel_size=1, stride=1, padding=0)

    def forward(self, x):
        x=self.deconvlayer1_2(x)
        x = self.upsample1_2(x)
        x=self.deconvlayer1_1(x)
        x = self.upsample1_1(x)

        if self.out_sigmoid:
            x=self.sigmoid(x)

        return x

The second_branch:
class second_branch(nn.Module):
    def __init__(self, in_channel=512, out_channel=[380,200], out_sigmoid=False):
        super(second_branch, self).__init__()

        self.out_sigmoid=out_sigmoid

        self.weight = weight()  # weight is a class that weighted its input

        self.deconvlayer2_2 = self._make_deconv(in_channel, out_channel[0], num_conv=3)
        self.upsample2_2=Upsample(scale_factor=2, mode='bilinear')
        self.deconvlayer2_! = self._make_deconv(out_channel[0], out_channel[1], num_conv=3)
        self.upsample2_1=Upsample(scale_factor=2, mode='bilinear')

    def forward(self, x):
        x=self.deconvlayer2_2(x)
        x = self.upsample2_2(x)
        weighted2_2 = self.weight(x)

        x=self.deconvlayer2_1(x)
        x = self.upsample2_1(x)
        weighted2_1 = self.weight(x)

        if self.out_sigmoid:
            x=self.sigmoid(x)

        return x, weighted2_1, weighted2_2

For implementing the mentioned idea in the main_class, I modified it as follows (instead of using the first_branch class in the forward function of the main_class, I wrote the script lines of the forward function of the first_branch in the forward function of the main_class):
class main_class(nn.Module):
    def __init__(self, pretrained=False):
        super(main_class, self).__init__()
        self.input=input_data()  # input_data is a class the provides the input data for the each branch

        self.conv_t2 = BasicConv3d(....................)
        self.second_branch=second_branch(512, out_sigmoid=True)

        self.conv_t1 = BasicConv3d(............)
        self.first_branch=first_branch(512, out_sigmoid=True)

        self.last = nn.Conv2d(4, 1, kernel_size=1, stride=1)

        self.sigmoid = nn.Sigmoid()

        def forward(self, x, par = False):

        x1, x2 = self.input(x)

        #second branch
        y2 = self.conv_t2(x2)

        out2, weighted2_1, weighted2_2 = self.second_branch(y2)

        #first branch
        y1 = self.conv_t1(x1)

       # instead of using from class first_branch, again I write the script lines of first_branch.forward() in below:
        x=self.deconvlayer1_2(y1)
        x = self.upsample1_2(x)
        stacking_2 = torch.stack(x, weighted2_2)
        x = self.frist_branch.combined1_2(stacking_2)

        x=self.deconvlayer1_1(x)
        x = self.upsample1_1(x)
        stacking_1 = torch.stack(x, weighted2_1)
        x = self.frist_branch.combined1_1(stacking_1)

        out1=self.sigmoid(x)

        x = torch.cat((out2, out1), 1)
        x = self.last(x)
        out = self.sigmoid(x)

        if par:
            return  out1, out2, out

        return out

I face with the following error:

TypeError: Cannot create a consistent method resolution order (MRO) for bases Module, second_branch

How can I fix this problem and how can I make the code able to have the interactions between new branches that may be added later to the model (for example if I have three branches, how can I have this kind of data combination between the third branch and the second one, and between the output of the previous combination and the first branch)?

Output:
In your main_class the second branch is not receiving additional arguments, it's only the first one that needs to be executed as second (in order). You could just add a parameter to the forward method of that branch like so:
class first_branch(nn.Module):
    ...

    def forward(self, x, weighted_x: list = []):
        x = self.deconvlayer1_2(x)
        x = self.upsample1_2(x)
        out1 = None
        if len(weighted_x) > 0:
            x = torch.stack(x, weighted_x[0])
            x = self.combined1_2(x)

        x = self.deconvlayer1_1(x)
        x = self.upsample1_1(x)
        out2 = None
        if len(weighted_x) > 1:
            x = torch.stack(x, weighted_x[1])
            x = self.combined1_1(x)

        if self.out_sigmoid:
            x = self.sigmoid(x)

        return x, out1, out2

As you can see, there's a lot of boilerplate code, which you can avoid by creating a small submodule that do this part of forward. You could then store multiple modules in your first_branch inside a ModuleList and iterate over them.


Instruction: Reduce multiclass image classification to binary classification in Pytorch

Input:
I am working on an stl-10 image dataset that consists of 10 different classes. I want to reduce this multiclass image classification problem to the binary class image classification such as class 1 Vs rest. I am using PyTorch torchvision to download and use the stl data but I am unable to do it as one Vs the rest.
train_data=torchvision.datasets.STL10(root='data',split='train',transform=data_transforms['train'], download=True)
test_data=torchvision.datasets.STL10(root='data',split='test',transform=data_transforms['val'], download=True)

train_dataloader = DataLoader(train_data,batch_size = 64,shuffle=True,num_workers=2)
test_dataloader = DataLoader(test_data,batch_size = 64,shuffle=True,num_workers=2)

Output:
For torchvision datasets, there is an inbuilt way to do this. You need to define a transformation function or class and add that into the target_transform while creating the dataset.
torchvision.datasets.STL10(root: str, split: str = 'train', folds: Union[int, NoneType] = None, transform: Union[Callable, NoneType] = None, target_transform: Union[Callable, NoneType] = None, download: bool = False)

Here is a working example for reference :

import torchvision
from torch.utils.data import DataLoader
from torchvision import transforms

class Multi2UniLabelTfm():
    def __init__(self,pos_label=5):
        if isinstance(pos_label,int) or isinstance(pos_label,float):
            pos_label = [pos_label,]
        self.pos_label = pos_label

    def __call__(self,y):
        # if y==self.pos_label:
        if y in self.pos_label:
            return 1
        else:
            return 0

if __name__=='__main__':

    test_tfms = transforms.Compose([
        transforms.ToTensor()
    ])
    data_transforms = {'val':test_tfms}

    #Original Labels
    # target_transform = None

    # Label 5 is converted to 1. Rest are 0.
    # target_transform = Multi2UniLabelTfm(pos_label=5)

    # Labels 5,6,7 are converted to 1. Rest are 0.
    target_transform = Multi2UniLabelTfm(pos_label=[5,6,7])
    test_data=torchvision.datasets.STL10(root='data',split='test',transform=data_transforms['val'], download=True, target_transform=target_transform)
    test_dataloader = DataLoader(test_data,batch_size = 64,shuffle=True,num_workers=2)

    for idx,(x,y) in enumerate(test_dataloader):
        print(idx,y)

        if idx == 5:
            break


Instruction: Training a u-net for multi-landmark heatmap regression producing the same heatmap for each channel

Input:
I’m training a U-Net (model below) to predict 4 heatmaps (gaussian centered around a keypoint, one in each channel). Each channel is for some reason outputting the same result, an example is given of a test image where the blue is ground truth for that channel and red is the output of the u-net. I have tried using L1, MSE and adaptive wing loss (Wang 2019), and none are able to regress the heatmaps. I'm not sure what I'm doing wrong would appreciate any advice. Thanks
test1:

test2:

test3:

test4:

class CNN(nn.Module):
    def __init__(self):
        super(CNN,self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 64,kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(64))

        self.layer2 = nn.Sequential(
            nn.Conv2d(64, 64,kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(64))

        self.layer3 = nn.Sequential(
            nn.MaxPool2d(2, stride=2, padding=0))

        self.layer4 = nn.Sequential(
            nn.Conv2d(64,128,kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(128))

        self.layer5 = nn.Sequential(
            nn.Conv2d(128, 128,kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(128))

        self.layer6 = nn.Sequential(
            nn.MaxPool2d(2, stride=2, padding=0))

        self.layer7 = nn.Sequential(
            nn.Conv2d(128, 256,kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
            nn.ReLU(),
            nn.BatchNorm2d(256))

Output:
Hard to see how this is a UNet. The only components that will modify the spatial shape of your input is your MaxPool2d's. You have two of these, so for a given input with size [B, 1, H, W], your output will have the shape [B, 256, H/4, W/4].
I think you need to give a more complete code snippet (don't have enough rep to leave this as a comment).


Instruction: attempting to manually download MNIST pytorch dataset in databricks

Input:
I've attempted a couple different iterations now to get the dataset manually loaded into databricks's DBFS.. so that PyTorch can load it.. however the MNIST dataset seems to just be some binary file.. is it expected I unzip it first or just.. point to the GZipped tarball?  So far all my trials have gotten this error
 train_dataset = datasets.MNIST(
     13     'dbfs:/FileStore/tarballs/train_images_idx3_ubyte.gz',
     14     train=True,

RuntimeError: Dataset not found. You can use download=True to download it

I am aware I can turn Download=True , however due to the firewalls this is not an option and I want to just upload the files and wire them in myself via DBFS... anyone done this as well?
EDIT:  @alexey suggested I need to add the extra paths MNIST/raw

And then change the input to
  train_dataset = datasets.MNIST(
    '/dbfs/FileStore/tarballs',
    train=True,
    download=False,
    transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]))
  data_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

But same error

Output:
My code and dir:
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../colabx/data', train=True, download=False,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))

....\colabx\data\MNIST\raw>ls
t10k-images-idx3-ubyte     train-images-idx3-ubyte
t10k-images-idx3-ubyte.gz  train-images-idx3-ubyte.gz
t10k-labels-idx1-ubyte     train-labels-idx1-ubyte
t10k-labels-idx1-ubyte.gz  train-labels-idx1-ubyte.gz