Instruction: Sagemaker creates output folders but no model.tar.gz after successful completion of the Training Job

Input:
I am running a Training Job using the Sagemaker API. The code for configuring the estimator looks as follows (I shrinked the full path names a bit):
s3_input = "s3://sagemaker-studio-****/training-inputs".format(bucket)
s3_images = "s3://sagemaker-studio-****/dataset"
s3_labels = "s3://sagemaker-studio-****/labels"
s3_output = 's3://sagemaker-studio-****/output'.format(bucket)

cfg='{}/input/models/'.format(s3_input)
weights='{}/input/data/weights/'.format(s3_input)
outpath='{}/'.format(s3_output)
images='{}/'.format(s3_images)
labels='{}/'.format(s3_labels)

hyperparameters = {
    "epochs": 1,
    "batch-size": 2
}

inputs = {
    "cfg": TrainingInput(cfg),
    "images": TrainingInput(images),
    "weights": TrainingInput(weights),
    "labels": TrainingInput(labels)
}

estimator = PyTorch(
    entry_point='train.py',
    source_dir='s3://sagemaker-studio-****/input/input.tar.gz',
    image_uri=container,
    role=get_execution_role(),
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    input_mode='File',
    output_path=outpath,
    train_output=outpath,
    base_job_name='visualsearch',
    hyperparameters=hyperparameters,
    framework_version='1.9',
    py_version='py38'
)

estimator.fit(inputs)

Everything runs fine and I get the success message:
Results saved to #033[1mruns/train/exp#033[0m
2022-07-08 08:38:35,766 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.
2022-07-08 08:38:35,766 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.
2022-07-08 08:38:35,767 sagemaker-training-toolkit INFO     Reporting training SUCCESS

2022-07-08 08:39:08 Uploading - Uploading generated training model
2022-07-08 08:39:08 Completed - Training job completed
ProfilerReport-1657268881: IssuesFound
Training seconds: 558
Billable seconds: 558
CPU times: user 1.34 s, sys: 146 ms, total: 1.48 s
Wall time: 11min 20s

When I call estimator.model_data I get a path poiting to a model.tar.gz file s3://sagemaker-studio-****/output/.../model.tar.gz
Sagemaker generated subfoldes into the output folder (which in turn contain a lot of json files and other artifacts):

But the file model.tar.gz is missing. This file is nowhere to be found. Is there anything I need to change or to add, in order to obtain my model?
Any help is much appreciated.

Output:
you need to make sure to store your model output to the right location inside the training container. Sagemaker will upload everything that is stored in the MODEL_DIR directory. You can find the location in the ENV of the training job:
model_dir = os.environ.get("SM_MODEL_DIR")

Normally it is set to opt/ml/model
Ref:

https://github.com/aws/sagemaker-training-toolkit/blob/master/ENVIRONMENT_VARIABLES.md#sm_model_dir
https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo-output.html


Instruction: `from_numpy()` leads to Expected vec.is_mps() to be true, but got false

Input:
When I try to convert the data (numpy nd array) to a tensor with from_numpy() when using the mps backend with torch.
I initiliaze the model as such:
device = "mps" if torch.has_mps else "cpu"
model = NeuralNetwork().to(device)

and it is using the mps backend:
Using mps device
NeuralNetwork(...)

Then using it as follows:
observations = env.reset()
X = torch.from_numpy(observations)
logits = model(X)

the model throws the error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [9], in <cell line: 3>()
      1 observations = env.reset()
      2 X = torch.from_numpy(observations)
----> 3 logits = model(X)

File lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

Input In [2], in NeuralNetwork.forward(self, x)
     13 def forward(self, x):
---> 14     logits = self.linear_relu_stack(x)
     15     return logits

File lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File lib/python3.8/site-packages/torch/nn/modules/container.py:139, in Sequential.forward(self, input)
    137 def forward(self, input):
    138     for module in self:
--> 139         input = module(input)
    140     return input

File lib/python3.8/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File lib/python3.8/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -> Tensor:
--> 114     return F.linear(input, self.weight, self.bias)

RuntimeError: Expected vec.is_mps() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

If I change device to cpuinstead of mps, it works. How do I use numpy arrays with the mps backend?
I am running it on an M1 chip and torch.has_mps is True.

Output:
You would have to first send your model to the mps device, then send your input explicitly to the mps device as well. In code:
model.to('mps')
logits = model(X.to('mps'))

Something like this worked for me using torch nightly on M1 pro, using a model that does not contain ops and data types that mps does not currently support.


Instruction: Is there any way to read .pth(dataset) and turn them into csv?

Input: I have a repo that provided a model architecture, but not pretrained model. it actually provides a .pth file but it's dataset inside the file, is there any way to make the dataset to csv?

Output: .pth files saved with toch.save() are supposed to be binaries. using torch.load() you can get the dataset, and then save it again as a .csv with pandas for example


Instruction: PyTorch UNet semantic segmentation dice score more than 1

Input:
I'm trying to develop a program that finds road lanes using semantic segmentation with UNet backend. But while training the model, it's giving me dice score more than 1. Why is this happening?
Batch size: 16
Num_workers: 2
Epochs: 50
IMAGE_HEIGHT = 80
IMAGE_WIDTH = 120
PIN_MEMORY = True
Here's my accuracy function:
def check_accuracy(loader, model, device="cpu"):
    num_correct = 0
    num_pixels = 0
    dice_score = 0
    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device)
            y = y.to(device).unsqueeze(1)
            preds = torch.sigmoid(model(x))
            preds = (preds > 0.5).float()
            num_correct += (preds == y).sum()
            num_pixels += torch.numel(preds)
            dice_score += (2 * (preds * y).sum()) / ((preds + y).sum() + 1e-8)

        print(f"Got {num_correct/num_pixels} with accuracy {num_correct/num_pixels*100:.2f}")
        print(f"Dice score: {dice_score/len(loader)}")
        model.train()

x's are images, and y's are ground truths.
Here's an example for images and ground truths:
Image:

Ground truth:

Here's my dataset's __getitem__ method:
    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.images[index])
        mask_path = os.path.join(self.val_dir, self.images[index])
        image = np.array(Image.open(img_path).convert("RGB"))
        mask = np.array(Image.open(mask_path).convert("L"), dtype=np.float32)
        mask[mask == 255.0] = 1.0

        if self.transform is not None:
            augmentations = self.transform(image=image, mask=mask)
            image = augmentations["image"]
            mask = augmentations["mask"]

        return image, mask

Here's my transformations:
    train_transform = A.Compose(
        [
            A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
            A.Rotate(limit=35, p=1.0),
            A.HorizontalFlip(p=0.5),
            A.VerticalFlip(p=0.1),
            A.Normalize(
                mean=[0.0, 0.0, 0.0],
                std=[1.0, 1.0, 1.0],
                max_pixel_value=255.0,
            ),
            ToTensorV2(),
        ],
    )

    val_transforms = A.Compose(
        [
            A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
            A.Normalize(
                mean=[0.0, 0.0, 0.0],
                std=[1.0, 1.0, 1.0],
                max_pixel_value=255.0,
            ),
            ToTensorV2(),
        ],
    )

And these are the accuracy metrics:
Epoch: 1/50

100%|██████████| 1/1 [00:00<00:00, 1.99it/s, loss=1.13]

=> Saving checkpoint

Epoch: 2/50

100%|██████████| 1/1 [00:00<00:00, 1.81it/s, loss=1.13]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 3/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=0.7]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 4/50

100%|██████████| 1/1 [00:00<00:00, 1.87it/s, loss=0.354]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 5/50

100%|██████████| 1/1 [00:00<00:00, 1.91it/s, loss=-.094]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 6/50

100%|██████████| 1/1 [00:00<00:00, 1.94it/s, loss=-.419]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 7/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-.914]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 8/50

100%|██████████| 1/1 [00:00<00:00, 1.92it/s, loss=-.7]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 9/50

100%|██████████| 1/1 [00:00<00:00, 1.87it/s, loss=-1.26]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 10/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-1.6]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 11/50

100%|██████████| 1/1 [00:00<00:00, 1.96it/s, loss=-2.04]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 12/50

100%|██████████| 1/1 [00:00<00:00, 1.89it/s, loss=-2.53]

=> Saving checkpoint Got 0.9816353917121887 with accuracy 98.16 Dice score: 0.0

Epoch: 13/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-2.77]

=> Saving checkpoint Got 0.9814687371253967 with accuracy 98.15 Dice score: 0.0

Epoch: 14/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-3.14]

=> Saving checkpoint Got 0.9801874756813049 with accuracy 98.02 Dice score: 0.0

Epoch: 15/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-3.77]

=> Saving checkpoint Got 0.9771562218666077 with accuracy 97.72 Dice score: 0.0

Epoch: 16/50

100%|██████████| 1/1 [00:00<00:00, 1.85it/s, loss=-3.95]

=> Saving checkpoint Got 0.972906231880188 with accuracy 97.29 Dice score: 0.013821512460708618

Epoch: 17/50

100%|██████████| 1/1 [00:00<00:00, 1.92it/s, loss=-4.83]

=> Saving checkpoint Got 0.9672812223434448 with accuracy 96.73 Dice score: 0.09691906720399857

Epoch: 18/50

100%|██████████| 1/1 [00:00<00:00, 1.96it/s, loss=-4.96]

=> Saving checkpoint Got 0.9596353769302368 with accuracy 95.96 Dice score: 0.2153720110654831

Epoch: 19/50

100%|██████████| 1/1 [00:00<00:00, 1.86it/s, loss=-5.43]

=> Saving checkpoint Got 0.9514479041099548 with accuracy 95.14 Dice score: 0.34086111187934875

Epoch: 20/50

100%|██████████| 1/1 [00:00<00:00, 1.98it/s, loss=-5.81]

=> Saving checkpoint Got 0.9496874809265137 with accuracy 94.97 Dice score: 0.385390967130661

Epoch: 21/50

100%|██████████| 1/1 [00:00<00:00, 1.91it/s, loss=-5.86]

=> Saving checkpoint Got 0.9429270625114441 with accuracy 94.29 Dice score: 0.4814487397670746

Epoch: 22/50

100%|██████████| 1/1 [00:00<00:00, 1.98it/s, loss=-6.34]

=> Saving checkpoint Got 0.9388750195503235 with accuracy 93.89 Dice score: 0.5995486974716187

Epoch: 23/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-6.92]

=> Saving checkpoint Got 0.9380520582199097 with accuracy 93.81 Dice score: 0.7058220505714417

Epoch: 24/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-7.42]

=> Saving checkpoint Got 0.9415416717529297 with accuracy 94.15 Dice score: 0.8273581266403198

Epoch: 25/50

100%|██████████| 1/1 [00:00<00:00, 1.94it/s, loss=-7.84]

=> Saving checkpoint Got 0.9451770782470703 with accuracy 94.52 Dice score: 0.9627659916877747

Epoch: 26/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-8.32]

=> Saving checkpoint Got 0.9467916488647461 with accuracy 94.68 Dice score: 1.1096603870391846

Epoch: 27/50

100%|██████████| 1/1 [00:00<00:00, 1.84it/s, loss=-8.75]

=> Saving checkpoint Got 0.9468228816986084 with accuracy 94.68 Dice score: 1.2025865316390991

Epoch: 28/50

100%|██████████| 1/1 [00:00<00:00, 1.92it/s, loss=-8.6]

=> Saving checkpoint Got 0.942562460899353 with accuracy 94.26 Dice score: 1.3215676546096802

Epoch: 29/50

100%|██████████| 1/1 [00:00<00:00, 1.95it/s, loss=-9.3]

=> Saving checkpoint Got 0.9366874694824219 with accuracy 93.67 Dice score: 1.4410816431045532

Epoch: 30/50

100%|██████████| 1/1 [00:00<00:00, 1.85it/s, loss=-9.37]

=> Saving checkpoint Got 0.9291979074478149 with accuracy 92.92 Dice score: 1.5965386629104614

Epoch: 31/50

100%|██████████| 1/1 [00:00<00:00, 1.89it/s, loss=-9.29]

=> Saving checkpoint Got 0.9251145720481873 with accuracy 92.51 Dice score: 1.7157979011535645

Epoch: 32/50

100%|██████████| 1/1 [00:00<00:00, 1.89it/s, loss=-9]

=> Saving checkpoint Got 0.9198125004768372 with accuracy 91.98 Dice score: 1.7378650903701782

Epoch: 33/50

100%|██████████| 1/1 [00:00<00:00, 1.88it/s, loss=-9.82]

=> Saving checkpoint Got 0.9136666655540466 with accuracy 91.37 Dice score: 1.758676290512085

Epoch: 34/50

100%|██████████| 1/1 [00:00<00:00, 1.85it/s, loss=-9.84]

=> Saving checkpoint Got 0.9044270515441895 with accuracy 90.44 Dice score: 1.7790669202804565

Epoch: 35/50

100%|██████████| 1/1 [00:00<00:00, 1.92it/s, loss=-10.4]

=> Saving checkpoint Got 0.8995833396911621 with accuracy 89.96 Dice score: 1.7764993906021118

Epoch: 36/50

100%|██████████| 1/1 [00:00<00:00, 1.95it/s, loss=-10.7]

=> Saving checkpoint Got 0.8992916345596313 with accuracy 89.93 Dice score: 1.775970697402954

Epoch: 37/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-10.8]

=> Saving checkpoint Got 0.8976249694824219 with accuracy 89.76 Dice score: 1.77357017993927

Epoch: 38/50

100%|██████████| 1/1 [00:00<00:00, 1.95it/s, loss=-10.5]

=> Saving checkpoint Got 0.8952499628067017 with accuracy 89.52 Dice score: 1.782752513885498

Epoch: 39/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-10.8]

=> Saving checkpoint Got 0.8901249766349792 with accuracy 89.01 Dice score: 1.7879419326782227

Epoch: 40/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-11]

=> Saving checkpoint Got 0.8890937566757202 with accuracy 88.91 Dice score: 1.7865288257598877

Epoch: 41/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-11.2]

=> Saving checkpoint Got 0.8915520906448364 with accuracy 89.16 Dice score: 1.7899266481399536

Epoch: 42/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-11.1]

=> Saving checkpoint Got 0.8923645615577698 with accuracy 89.24 Dice score: 1.7993781566619873

Epoch: 43/50

100%|██████████| 1/1 [00:00<00:00, 1.96it/s, loss=-10.9]

=> Saving checkpoint Got 0.887333333492279 with accuracy 88.73 Dice score: 1.804529070854187

Epoch: 44/50

100%|██████████| 1/1 [00:00<00:00, 1.85it/s, loss=-11.9]

=> Saving checkpoint Got 0.8823854327201843 with accuracy 88.24 Dice score: 1.7975029945373535

Epoch: 45/50

100%|██████████| 1/1 [00:00<00:00, 1.91it/s, loss=-11.2]

=> Saving checkpoint Got 0.8859270811080933 with accuracy 88.59 Dice score: 1.802530288696289

Epoch: 46/50

100%|██████████| 1/1 [00:00<00:00, 1.92it/s, loss=-11.6]

=> Saving checkpoint Got 0.890500009059906 with accuracy 89.05 Dice score: 1.8090462684631348

Epoch: 47/50

100%|██████████| 1/1 [00:00<00:00, 1.90it/s, loss=-11.7]

=> Saving checkpoint Got 0.9020833373069763 with accuracy 90.21 Dice score: 1.8257639408111572

Epoch: 48/50

100%|██████████| 1/1 [00:00<00:00, 1.81it/s, loss=-12.1]

=> Saving checkpoint Got 0.9160521030426025 with accuracy 91.61 Dice score: 1.8463400602340698

Epoch: 49/50

100%|██████████| 1/1 [00:00<00:00, 1.91it/s, loss=-12]

=> Saving checkpoint Got 0.925125002861023 with accuracy 92.51 Dice score: 1.859954833984375

Epoch: 50/50

100%|██████████| 1/1 [00:00<00:00, 1.93it/s, loss=-11.4]

=> Saving checkpoint Got 0.933968722820282 with accuracy 93.40 Dice score: 1.873431921005249

As you can see 50th dice score is 1.873431921005249. Why dice score tis more than 1?

Output:
The equation you are using to calculate the dice coefficient is wrong.
This will work.
dice_score += (2 * (preds * y).sum()) / (2 * (preds * y).sum()+ ((preds*y)<1).sum())

You can interpret it as 2 x correct_classified/(2 x correct_classified+wrong_classified).
Note that this only works in the binary case.


Instruction: Is the output of GRU in Pytorch just the hidden states?

Input: I understand how GRU works, but now I'm confused by the difference between "hidden" and "output" of GRU in Pytorch: is "output" just the hidden states of the GRU, or the hidden states after some transformations? If "output" is just the hidden states, why do we want both output and h_n as return value of GRU.forward, since we can just get h_n from the last element of output?

Output:
According to the documentation:

output is a tensor of shape (batch_size, sequence_length, num_layers, num_directions * hidden_size)
h_n a tensor of shape (num_directions * num_layers, batch_size, hidden_size)

The first provides you with hidden states across the entire sequence, allowing you to use intermediate representations (e.g. for attention) or train for step-separable tasks (e.g. token-level classification).
The latter provides you with just a single summary vector per input sequence, which is handy if you're only interested in a sequence-level representation that doesn't involve token-wise attention.
There is no implicit hidden-to-output transformation.


Instruction: TypeError: got an unexpected keyword argument 'image'

Input:
I am getting TypeError: get_train_augs() got an unexpected keyword argument 'image', I have my   augmentation functions as follows
Augmentation functions
def get_train_augs():
  return A.Compose([
                    A.Resize(IMAGE_SIZE,IMAGE_SIZE),
                    A.HorizontalFlip(p = 0.5),
                    A.VerticalFlip(p = 0.5),
  ])

def get_valid_augs():
  return A.Compose([
                    A.Resize(IMAGE_SIZE,IMAGE_SIZE),

  ])

Custom segmentation dataset class
class SegmentationDataset(Dataset):
  def __init__(self, df, augmentations=None):
    self.df = df
    self.augmentations = augmentations

  def __len__(self):
    return len(self.df)

  def __getitem__(self,idx):
    row = self.df.iloc[idx]

    image_path = row.images
    mask_path = row.masks

    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE) # (h, w, c)
    mask = np.expand_dims(mask, axis = -1)

    if self.augmentations is not None:
      data = self.augmentations(image = image, mask = mask)

      image = data['image']
      mask = data['mask']

    # (h, w, c) -> (c, h, w)
    image = np.transpose(image,(2,0,1)).astype(np.float32)
    mask = np.transpose(mask,(2,0,1)).astype(np.float32)

    image = torch.Tensor(image)/255.0
    mask = torch.round(torch.Tensor(mask)/255.0)

    return image, mask

when I call trainset like this I am getting an error:
trainset = SegmentationDataset(train_df, get_train_augs)
validset = SegmentationDataset(valid_df, get_valid_augs)

calling a random index
idx = 3
image, mask = trainset[idx]

The error I am getting is:
TypeError                                 Traceback (most recent call last)
<ipython-input-28-9b83781b7e3d> in <module>()
      1 idx = 3
      2
----> 3 image, mask = trainset[idx]
      4
      5 helper.show(image, mask)

<ipython-input-25-39872478644d> in __getitem__(self, idx)
     20
     21     if self.augmentations is not None:
---> 22       data = self.augmentations(image = image, mask = mask)
     23
     24       image = data['image']

TypeError: get_train_augs() got an unexpected keyword argument 'image'

Output:
try
trainset = SegmentationDataset(train_df, get_train_augs())
validset = SegmentationDataset(valid_df, get_valid_augs())


Instruction: Loss is Nan - PyTorch

Input:
I'm trying to train a simple neural net with the following architecture:
class ModifiedNet(nn.Module):
    def __init__(self, num_inputs, num_outputs):
        super(ModifiedNet, self).__init__()
        self.linear = nn.Linear(num_inputs, 1000)
        self.linear2 = nn.Linear(in_features=1000, out_features=num_outputs)

    def forward(self, input):
        input = input.view(-1, num_inputs) # reshape input to batch x num_inputs
        output = self.linear(input)
        output = self.linear2(output)
        return output

The dataset is MNIST (num_inputs=784 and num_outputs=10).
I'm trying to plot the loss (we're using CrossEntropy) for each learning rate (0.01, 0.1, 1, 10), but the loss is NaN when I reach LR=1.
From looking at similar questions, I saw that you're supposed to lower the LR, but my task  is to measure it with the given ones.
What am I doing wrong?
This is the code for the train and test:
# train and test functions
def train(epoch, network, optimizer=None):
    losses = list()
    network.train()

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data), Variable(target)
        if optimizer is not None:
            optimizer.zero_grad()
        output = network(data)
        loss = F.cross_entropy(output, target).to(torch.float64)
        losses.append(loss.item())
        loss.backward()

        if optimizer is not None:
            optimizer.step()

        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
    return np.mean(np.array(losses))

def test(network):
    network.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        #data, target = Variable(data, volatile=True), Variable(target)
        output = network(data)
        test_loss += F.cross_entropy(output, target, reduction='sum').to(torch.double).item() # sum up batch loss
        #test_loss += F.cross_entropy(output, target, sum=True).item() # sum up batch loss
        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))
    return test_loss

and this is where I'm looping over the learning rates:
learning_rates = [0.01, 0.1, 1, 10]

for learning_rate in learning_rates:
    net = ModifiedNet(num_inputs, num_outputs)
    optimizer = optim.SGD(net.parameters(), lr=learning_rate)
    train_losses = dict()
    for epoch_idx in range(10):
        train_losses[epoch_idx] = train(epoch_idx, net, optimizer)

    plot_graph(list(train_losses.keys()), list(train_losses.values()), "epoch", "train loss", str(learning_rate))

And this is the original question:

Retrain the model for 10 epochs with each of the learning rates in the
set {0.01, 0.1, 1, 10} and test the resulting model. Create a figure
and plot the loss curves of each of the four runs for comparison.
Explain the obtained (train and test) results.

Also, the net architecture is a given in the question, so I can't change it.

Output: I had a silly bug as usual, and didn't use an activation function.


Instruction: Is there any replacement for tf.random_gamma in pytorch?

Input:
I'm converting a TensorFlow repository to PyTorch code. I came across this line of code:
tf.squeeze(tf.random_gamma(shape =(self.n_sample,),alpha=self.alpha+tf.to_float(self.B)))

I would like to know the equivalent of tf.random_gamma in PyTorch. I think torch.distributions.gamma.Gamma doesn't work the same way.

Output:
It looks like torch.distributions.gamma.Gamma can be used in this case. Here is an example:
import torch
from torch.distributions.gamma import Gamma

def random_gamma(shape, alpha, beta=1.0):
  alpha = torch.ones(shape) * torch.tensor(alpha)
  beta = torch.ones(shape) * torch.tensor(beta)
  gamma_distribution = Gamma(alpha, beta)

  return gamma_distribution.sample()

print(random_gamma(shape=(10,), alpha=3.0))

Output:
tensor([2.7673, 1.5498, 6.5191, 5.2923, 3.3204, 3.9286, 1.4163, 1.2400, 3.9661, 1.7663])

The difference is that torch.distributions.gamma.Gamma requires complete tensors for alpha and beta instead of shape+values like it is in TF. Also, TF version has default value 1 for beta which I tried to imitate in the example code.
It makes sense to create distribution instance once though in case if the function will be used multiple times.


Instruction: Issue between number of classes and shape of inputs in metric collection torch

Input:
I have a problem because I want to calculate some metrics in torchmetrics. but there is a problem:
ValueError: The implied number of classes (from shape of inputs) does not match num_classes.

The output is from UNet and the loss function is BCEWithLogitsLoss (binary segmentation)
channels = 1 because of grayscale img
Input shape: (batch_size, channels, h, w) torch.float32
Label shape: (batch_size, channels, h, w) torch.float32 for BCE
Output shape: (batch_size, channels, h, w): torch.float32
inputs, labels = batch
outputs = model(input)
loss = self.loss_function(outputs, labels)
prec = torchmetrics.Precision(num_classes=1)(outputs, labels.type(torch.int32)

Output:
It seems that torchmetrics expects different shape. Try to flatten both output and labels:
prec = torchmetrics.Precision(num_classes=1)(outputs.view(-1), labels.type(torch.int32).view(-1))


Instruction: How can I define the activation function as a hyperparameter in PyTorch through RAY.Tune?

Input:
This is the link to the main page of the topic I want.
https://docs.ray.io/en/latest/tune/examples/tune-pytorch-cifar.html#tune-pytorch-cifar-ref
But unfortunately, there is no good documentation to answer all the questions.
Also, if you know how I can define nested cross validation in this environment, please tell me.

Output:
I solved this problem as follows.
n_samples,n_features=X_train.shape
class NeuralNetwork (nn.Module):
def __init__(self,n_input_features,l1, l2,l3,config):
    super (NeuralNetwork, self).__init__()
    self.config = config
    self.linear1=nn.Linear(n_input_features,4*math.floor(n_input_features/2)+l1)
    self.linear2=nn.Linear(l1+4*math.floor(n_input_features/2),math.floor(n_input_features/2)+l2)
    self.linear3=nn.Linear(math.floor(n_input_features/2)+l2,math.floor(n_input_features/3)+l3)
    self.linear4=nn.Linear(math.floor(n_input_features/3)+l3,math.floor(n_input_features/6))
    self.linear5=nn.Linear(math.floor(n_input_features/6),1)

    self.a1 = self.config.get("a1")
    self.a2 = self.config.get("a2")
    self.a3 = self.config.get("a3")
    self.a4 = self.config.get("a4")
@staticmethod
def activation_func(act_str):
    if act_str=="tanh" or act_str=="sigmoid":
        return eval("torch."+act_str)
    elif act_str=="silu" or act_str=="relu" or act_str=="leaky_relu" or act_str=="gelu":
        return eval("torch.nn.functional."+act_str)
def forward(self,x):
    out=self.linear1(x)
    out=self.activation_func(self.a1)(out.float())
    out=self.linear2(out)
    out=self.activation_func(self.a2)(out.float())
    out=self.linear3(out)
    out=self.activation_func(self.a3)(out.float())
    out=self.linear4(out)
    out=self.activation_func(self.a3)(out.float())
    out=torch.sigmoid(self.linear5(out))
    y_predicted=out
    return y_predicted


Instruction: Masking the columns of pytorch matrix

Input:
I have a matrix of shape (batch_size, N, N) and a masking tensor of shape (batch_size, N).
I want to put -infinity values only for the columns (and not rows) of the matrix, according to the given mask.

Output:
The solution is via repeating the mask in the correct dimension:
repeated_mask[mask.unsqueeze(-1).repeat(1,1,mask.shape[-1])!=1] = 0


Instruction: PyTorch learning scheduler order changes loss in a drastic way

Input:
I'm a beginner to PyTorch and am trying to train a MNIST model based on a custom neural network class. My learning rate scheduler, loss function and optimizer are:
optimizer = optim.Adam(model.parameters(), lr=0.003)
loss_fn = nn.CrossEntropyLoss()
exp_lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

I'm also using Learning Rate scheduler for that purpose. Initially, I had my training loop like this:
# this training gives high loss and it doesn't varies that much
def training(epochs):
    model.train()

    for batch_idx, (imgs, labels) in enumerate(train_loader):
        imgs = imgs.to(device=device)
        labels = labels.to(device=device)

        optimizer.zero_grad()
        outputs = model(imgs)
        loss = loss_fn(outputs, labels)

        loss.backward()
        optimizer.step()

        exp_lr_scheduler.step()  # inside the loop and after the optimizer

        if (batch_idx + 1)% 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, (batch_idx + 1) * len(imgs), len(train_loader.dataset),
                100. * (batch_idx + 1) / len(train_loader), loss.data))

But this training was not efficient and my loss was almost the same in every epoch.
Then, I changed my training function to this in the end:
# this training works perfectly
def training(epochs):
    model.train()
    exp_lr_scheduler.step()  # out of the loop but before optimizer step

    for batch_idx, (imgs, labels) in enumerate(train_loader):
        imgs = imgs.to(device=device)
        labels = labels.to(device=device)

        optimizer.zero_grad()
        outputs = model(imgs)
        loss = loss_fn(outputs, labels)

        loss.backward()
        optimizer.step()

        if (batch_idx + 1)% 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, (batch_idx + 1) * len(imgs), len(train_loader.dataset),
                100. * (batch_idx + 1) / len(train_loader), loss.data))

And now, it's working correctly. I just don't get the reason for this.
I have two queries:

Shouldn't exp_lr_scheduler.step() be in the for loop so that it also get's updated with every epoch? ; and
PyTorch latest version says to keep exp_lr_scheduler.step() after optimizer.step() but doing this in the my training function gives me worse loss.

What can be the reason or am I doing it wrong?

Output:
StepLR updates the learning rate after every step_size by gamma, that means if step_size is 7 then learning rate will be updated after every 7 epoch by multiplying the current learning rate to gamma. That means that in your snippet, the learning rate is getting 10 times smaller every 7 epochs.
Have you tried increasing the starting learning rate? I would try 0.1 or 0.01. I think the problem could be at the size of the starting learning rate since the starting point it is already quite small. This  causes that the gradient descent algorithm (or its derivatives, as Adam) cannot move towards the minimum because the step is too small and your results keep being the same (at the same point of the functional to minimize).
Hope it helps.


Instruction: select a specific GPU to use in jupyter notebook

Input:
I'm coding on a .ipynb file on a linux server.
The linux server I use has multiple GPUs on it, but I should only use idle GPU so as not to accidentally abort others' programme.
I've already known that for common .py file we can add some instructions at the command line to choose a common GPU(e.g. export CUDA_VISIBLE_DEVICES=#), but will it work for jupyter notebook? If not, how can I specify a GPU to work on.

Output:
You have to choose its name correctly. For instance there may be 3 GPU devices available namely "cuda:0","cuda:1","cuda:2". To choose the third one you need to run the following code:
if torch.cuda.is_available():
  dev = "cuda:2"
else:
  dev = "cpu"
device = torch.device(dev)


Instruction: How to use yolov5 tflite export with OpenCV

Input:
I have exported a tflite file from Yolov5 and I got the output data using the code below:
import numpy as np
import tensorflow as tf
from PIL import Image
import os

img = Image.open(os.path.join('dataset', 'images','val','IMG_6099.JPG'))
img = img.resize((256,256),Image.ANTIALIAS)
numpydata = np.asarray(img)
interpreter = tf.lite.Interpreter(model_path="yolov5s-fp16.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

input_shape = input_details[0]['shape']
input_data = np.array(img,dtype=np.float32)
input_data = tf.expand_dims(input_data, 0)
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()

output_data = interpreter.get_tensor(output_details[0]['index'])

print out output_data:
[[[1.6754180e-02 3.2771632e-02 8.4546164e-02 ... 2.2025524e-05
   3.0189141e-05 6.1972853e-05]
  [1.5505254e-02 3.5847023e-02 9.6953809e-02 ... 1.9333076e-05
   1.5587271e-05 3.6931968e-05]
  [1.6107641e-02 3.6390714e-02 8.2990780e-02 ... 1.6197217e-05
   1.4623029e-05 3.6216315e-05]
  ...
  [8.6931992e-01 8.8494051e-01 2.4040593e-01 ... 3.1457843e-05
   2.4052188e-05 2.2471884e-05]
  [8.6244017e-01 9.0521729e-01 4.4481179e-01 ... 5.1936011e-05
   3.9207229e-05 3.5609013e-05]
  [8.6841702e-01 9.0255147e-01 7.0057535e-01 ... 1.0812500e-04
   1.0073676e-04 7.7818921e-05]]]

What are these numbers? and more important how can I show the results on the image?
I also see this post already.
and here is my code trying to capture objects in real time:
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
print(ret)
frame = cv2.resize(frame, (256 , 256))

for i in range(len(scores)):
    if ((scores[i] > 0.1) and (scores[i] <= 1.0)):
        H = frame.shape[0]
        W = frame.shape[1]
        xmin = int(max(1,(xyxy[0][i] * W)))
        ymin = int(max(1,(xyxy[1][i] * H)))
        xmax = int(min(H,(xyxy[2][i] * W)))
        ymax = int(min(W,(xyxy[3][i] * H)))

        # cv2.rectangle(frame, (xmin,ymin), (xmax,ymax), (10, 255, 0), 2)
plt.imshow(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

Output:
The post you attached is clear said:
[x ,y ,w ,h , conf, class0, class1, ...] total 85 columns and  many objects (rows) detected.
col 0-3 is boxes, col 4 is conf, and the other 80 is the class
You need filter the rows with the conf. Otherwise you will get a lot false detected object.
Also [x ,y ,w ,h] is not the real scale of your input image.
To obtain the real boxes, you may need do some processing like e.g rescale, xywh to xyxy, NMS ( non max suppression) etc..
You can check the detect.py and utils/general.py in Yolov5 source code.
After you get the real boxes, draw box on image.
Below doc show you how to draw box on image using open cv
https://docs.opencv.org/4.x/dc/da5/tutorial_py_drawing_functions.html
If you google, there are a lot of example teach you how to draw using open cv


Instruction: Forward function with multiple outputs?

Input:
Typically the forward function in nn.module of pytorch computes and returns predictions for inputs happening in the forward pass. Sometimes though, intermediate computations might be useful to return. For example, for an encoder, one might need to return both the encoding and reconstruction in the forward pass to be used later in the loss.
Question: Can Pytorch's nn.Module's forward function, return multiple outputs? Eg a tuple of outputs consisting predictions and intermediate values?
Does such a return value not mess up the backward propagation or autograd?
If it does, how would you handle cases where multiple functions of input are incorporated in the loss function?
(The question should be valid in tensorflow too.)

Output:
"The question should be valid in Tensorflow too", but PyTorch and Tensorflow are different frameworks. I can answer for PyTorch at least.
Yes you can return a tuple containing any final and or intermediate result. And this does not mess up back propagation since the graph is saved implicitly from the tensors outputs using callbacks and cached tensors.


Instruction: How to convert cv::Mat to torch::Tensor and feed it to libtorch model?

Input: I read image with cv2.imread() and trying to feed it to torch model in c++. It has datatype cv::Mat. I think i need to convert it to tensor somehow and then use model.forward(), but i am confused how to do it. Is there some function similar to .Tensor() in python?

Output:
The function torch::from_blob can be used to create a tensor view over the image data, like this:
torch::Tensor to_tensor(cv::Mat img) {
    return torch::from_blob(img.data, { img.rows, img.cols, 3 }, torch::kUInt8);
}


Instruction: Trouble Training Same Tensorflow Model in PyTorch

Input:
I have trained a model in Tensorflow and am having trouble replicating it in PyTorch. The Tensorflow model achieves near 100% accuracy (the task is simple), but the PyTorch model performs at random. I've spent a while trying to figure this out, and can't understand what the problem could be.
The model is trained for the task of binary classification. Given an input utterance describing a quadrant and a (x, y, z) coordinate, the model has to predict if the (x, z) portion of the coordinate is in the quadrant described. For example, if the input text was "quadrant 1" and the coordinate was (0.5, -, 0.5), then the prediction should be true, but if the region was "quadrant 2" with the same coordinate, then the prediction should be false.
I generated some data and trained the model in Tensorflow using this code:
x_data_placeholder = tf.placeholder(tf.float32, [FLAGS.batch_size, 1], name="x_data")
y_data_placeholder = tf.placeholder(tf.float32, [FLAGS.batch_size, 1], name="y_data")
z_data_placeholder = tf.placeholder(tf.float32, [FLAGS.batch_size, 1], name="z_data")
# text and labels placeholders
text_data = tf.placeholder(tf.int32, [FLAGS.batch_size, maxtextlength])
text_lengths = tf.placeholder(tf.int32, [FLAGS.batch_size])
y_labels_placeholder = tf.placeholder(tf.int64, [FLAGS.batch_size])

# encode text and coordinate
embeddings = tf.Variable(tf.random_uniform([100, embedding_size], -1, -1))
rnn_inputs = tf.nn.embedding_lookup(embeddings, text_data)
rnn_layers = [tf.compat.v1.nn.rnn_cell.LSTMCell(size, initializer=tf.compat.v1.keras.initializers.glorot_normal) for size in [256]]
multi_rnn_cell = tf.compat.v1.nn.rnn_cell.MultiRNNCell(rnn_layers, state_is_tuple=True)

text_outputs, text_fstate = tf.compat.v1.nn.dynamic_rnn(cell=multi_rnn_cell,
                                             inputs=rnn_inputs,
                                             dtype=tf.float32, sequence_length=text_lengths)

# have fully connected layers to map them the input coordinates into the same dimension as the LSTM output layer from above
x_output_layer = tf.compat.v1.layers.dense(x_data_placeholder, units=FLAGS.fc_column_size, activation=tf.nn.relu, name='x_coordinate')
y_output_layer = tf.compat.v1.layers.dense(y_data_placeholder, units=FLAGS.fc_column_size, activation=tf.nn.relu, name='y_coordinate')
z_output_layer = tf.compat.v1.layers.dense(z_data_placeholder, units=FLAGS.fc_column_size, activation=tf.nn.relu, name='z_coordinate')

# add the representations
total_output_layer = x_output_layer + y_output_layer + z_output_layer + lstm_output_layer

# make the predictions with two fully connected layers
fc_1 = tf.compat.v1.layers.dense(total_output_layer, units=FLAGS.hidden_layer_size, activation=tf.nn.relu, name='fc_1')
logits = tf.compat.v1.layers.dense(fc_1, units=FLAGS.output_dims, activation=None, name='logits')

# train the model
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_labels_placeholder, logits=logits))
optimizer = tf.train.AdamOptimizer(learning_rate=FLAGS.learning_rate, epsilon=1e-7)
gradients, variables = zip(*optimizer.compute_gradients(loss))
gradients, _ = tf.clip_by_global_norm(gradients, FLAGS.gradient_clip_threshold)
optimize = optimizer.apply_gradients(zip(gradients, variables))

# then it'll be trained with sess.run ...

Now for the PyTorch replication:
class BaselineModel(nn.Module):
    def __init__(self):
        super(BaselineModel, self).__init__()

        self.encode_x = nn.Linear(1, embed_size)
        self.encode_y = nn.Linear(1, embed_size)
        self.encode_z = nn.Linear(1, embed_size)
        self._embeddings = nn.Embedding(vocab_size, self.embedding_table_size)
        nn.init.uniform_(self._embeddings.weight, -1.0, 1.0)
        self.num_layers = 1
        self.rnn = nn.LSTM(self.embedding_table_size, self.hidden_size, batch_first=True)
        self.fc_after_text_lstm = nn.Linear(self.hidden_size, 100)
        self.fc = nn.Linear(100, 256)
        self.fc_final = nn.Linear(256, 2)
        self.relu_activation = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def init_hidden(self, batch_size, device='cuda:0'):
        # for LSTM, we need # of layers
        h_0 = torch.zeros(1, batch_size, self.hidden_size).to(device)
        c_0 = torch.zeros(1, batch_size, self.hidden_size).to(device)
        return h_0, c_0

    def forward(self, input_text, x_coordinate=None, y_coordinate=None, z_coordinate=None):
        x_embed = self.relu_activation(self.encode_x(x_coordinate.cuda().to(torch.float32)).cuda())
        y_embed = self.relu_activation(self.encode_y(y_coordinate.cuda().to(torch.float32))).cuda()
        z_embed = self.relu_activation(self.encode_z(z_coordinate.cuda().to(torch.float32))).cuda()

        embeds = self._embeddings(input_text)
        embedding, hidden = self.rnn(embeds, self.hidden)
        text_fc = self.relu_activation(self.fc_after_text_lstm(embedding[:, -1]))
        representations_so_far_added = torch.sum(torch.stack([text_fc, x_embed, y_embed, z_embed]), dim=0)
        pre_final_embedding = self.relu_activation(self.fc(representations_so_far_added))

        return self.fc_final(pre_final_embedding )

### training code
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, eps=1e-7)
criterion = nn.CrossEntropyLoss()
for input_text, x_coordinate, y_coordinate, z_coordinate, targets, train_data:

    optimizer.zero_grad()
    pred = model(input_text, x_coordinate=x_coordinate, y_coordinate=y_coordinate, z_coordinate=z_coordinate)
    loss = criterion(pred.float(), targets)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
    optimizer.step()
    scheduler.step()

    # accuracy evaluation code, this is evaluated over the entire epoch
    pred_idx = F.log_softmax(pred, dim=1)
    target_labels = targets.cpu().int()
    pred_labels = torch.argmax(pred_idx, dim=-1).cpu().data.int()
    curr_acc = skm.accuracy_score(target_labels, pred_labels)

If anyone can spot any issue with the PyTorch implementation or maybe tell me what could be wrong, that would be much appreciated! I also tried to load the weights of the Tensorflow model into all the appropriate layers, and performance still struggles in PyTorch! Thanks in advance!
EDIT:
I have created a minimally reproducible example, because I still cannot figure out what the problem is. Any help would be still appreciated!
import torch
import torch.nn as nn
import numpy as np
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

lr = 0.0005
n_epochs = 10
input_dim = 4
hidden_dim = 128
layer_dim = 2
output_dim = 2
batch_size = 50

class FeatureDataSet(torch.utils.data.Dataset):

    def __init__(self, x_train, y_train, x_coordinates):
        self.x_train = torch.tensor(x_train, dtype=torch.long)
        self.y_train = torch.tensor(y_train)
        self.x_coordinates = torch.tensor(x_coordinates, dtype=torch.float32)
    def __len__(self):
        return len(self.y_train)
    def __getitem__(self, idx):
        return self.x_train[idx], self.y_train[idx], self.x_coordinates[idx]

class RNN(nn.Module):

    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, batch_size):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim

        # linear layer to encode the coordinate
        self.encode_x = nn.Linear(1, hidden_dim).cuda()
        self._embeddings = nn.Embedding(40, 100).cuda()

        # hidden_dim is 128
        # layer_dim is 2
        self.lstm = nn.LSTM(100, hidden_dim, layer_dim, batch_first=True).cuda()
        self.fc = nn.Linear(2 * hidden_dim, output_dim).cuda()
        self.batch_size = batch_size
        self.hidden = None

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        return [t.cpu() for t in (h0, c0)]

    def forward(self, x, x_coordinate):
        #initializing the hidden states
        h0, c0 = self.init_hidden(x)
        embeds = self._embeddings(x)
        out, (hn, cn) = self.lstm(embeds.cuda(), (h0.cuda(), c0.cuda()))

        x_embed = F.relu(self.encode_x(x_coordinate.cuda().to(torch.float32)).cuda())
        representations_so_far_added = torch.cat([out[:, -1, :], x_embed], dim=1)

        out = self.fc(representations_so_far_added)
        return out

model = RNN(input_dim, hidden_dim, layer_dim, output_dim, batch_size)
criterion = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.001)

print('Start model training')
import sklearn.metrics as skm
import torch.nn.functional as F

x_train = []
x_coordinates = []
y_train = []
for i in range(10000):
    # create the data. if x_coordinate > 0 and the sentence says that (represented by [1, 5, 6, 8]), then we should predict positive else negative (if the x_coordinate > 0)
    # same applies if the x_coordinate < 0, just that the sentence is now [1, 5, 6, 9]
    if np.random.randint(0, 2) == 0:
        if np.random.randint(0, 2) == 0:
            # x coordinate > 0
            x_train.append([1, 5, 6, 8])
            x_coordinates.append([round(np.random.uniform(0.01, 1.00, 1)[0], 2)])
            y_train.append(1.0)
        else:
            # x coordinate > 0 negative
            x_train.append([1, 5, 6, 8])
            x_coordinates.append([round(np.random.uniform(-1.00, 0.00, 1)[0], 2)])
            y_train.append(0.0)
    else:
        if np.random.randint(0, 2) == 0:
            # x coordinate < 0
            x_train.append([1, 5, 6, 9])
            x_coordinates.append([round(np.random.uniform(-1.00, 0.00, 1)[0], 2)])
            y_train.append(1.0)
        else:
            # x coordinate < 0 negative
            x_train.append([1, 5, 6, 9])
            x_coordinates.append([round(np.random.uniform(0.01, 1.00, 1)[0], 2)])
            y_train.append(0.0)

# print a sample of data
print(x_train[:10])
print(y_train[:10])
print(x_coordinates[:10])

# create a dataloader
trainingDataset = FeatureDataSet(x_train=x_train, y_train=y_train, x_coordinates=x_coordinates)
train_loader = torch.utils.data.DataLoader(dataset=trainingDataset, batch_size=batch_size, shuffle=True)

# for each epoch
for epoch in range(1, n_epochs + 1):
    acc_all = []
    # each batch
    for i, (x_batch, y_batch, x_coord_batch) in enumerate(train_loader):
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)

        x_coord_batch = x_coord_batch.to(device)

        opt.zero_grad()
        # pass in the text (x_batch) and coordinate (x_coord_batch)
        out = model(x_batch, x_coordinate=x_coord_batch)
        loss = criterion(out.float(), y_batch.type(torch.LongTensor).cuda())
        loss.backward()
        opt.step()

        pred_idx = F.log_softmax(out, dim=1)
        target_labels = y_batch.cpu().int()
        pred_labels = torch.argmax(pred_idx, dim=-1).cpu().data.int()

        curr_acc = skm.accuracy_score(target_labels, pred_labels)
        acc_all.append(curr_acc)

    print(np.mean(acc_all))

Output:
I suppose perhaps there are some mistakes in your dataset implementation in the PyTorch version.
I tried your pytorch BaselineModel on both the dataset in your "minimally reproducible example" and my own dataset constructed according to your description, and find that it works fine.
The following is my codes for testing on my own dataset. Note that I add several hyperparameters to the code of BaselineModel to make it run. I got accuracy over 99%.
import random

import torch
import torch.nn as nn
import numpy as np

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

lr = 0.0005
n_epochs = 100
input_dim = 4
hidden_dim = 128
layer_dim = 2
output_dim = 2
batch_size = 50

class AverageMeter(object):
    """Computes and stores the average and current value"""

    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

class FeatureDataSet(torch.utils.data.Dataset):

    def __init__(self, x_train, y_train, x_coordinates, y_coordinates, z_coordinates):
        self.x_train = torch.tensor(x_train, dtype=torch.long)
        self.y_train = torch.tensor(y_train)
        self.x_coordinates = torch.tensor(x_coordinates, dtype=torch.float32)
        self.y_coordinates = torch.tensor(y_coordinates, dtype=torch.float32)
        self.z_coordinates = torch.tensor(z_coordinates, dtype=torch.float32)

    def __len__(self):
        return len(self.y_train)

    def __getitem__(self, idx):
        return self.x_train[idx], self.y_train[idx], self.x_coordinates[idx], self.y_coordinates[idx], self.z_coordinates[idx]

class BaselineModel(nn.Module):
    def __init__(self):
        super(BaselineModel, self).__init__()
        vocab_size = 40
        self.hidden_size = 100
        self.embedding_table_size = self.hidden_size
        self.encode_x = nn.Linear(1, self.hidden_size)
        self.encode_y = nn.Linear(1, self.hidden_size)
        self.encode_z = nn.Linear(1, self.hidden_size)
        self._embeddings = nn.Embedding(vocab_size, self.embedding_table_size)
        nn.init.uniform_(self._embeddings.weight, -1.0, 1.0)
        self.num_layers = 1
        self.rnn = nn.LSTM(self.embedding_table_size, self.hidden_size, batch_first=True)
        self.fc_after_text_lstm = nn.Linear(self.hidden_size, 100)
        self.fc = nn.Linear(100, 256)
        self.fc_final = nn.Linear(256, 2)
        self.relu_activation = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

        self.hidden = self.init_hidden(batch_size)

    def init_hidden(self, batch_size, device='cuda:0'):
        # for LSTM, we need # of layers
        h_0 = torch.zeros(1, batch_size, self.hidden_size).to(device)
        c_0 = torch.zeros(1, batch_size, self.hidden_size).to(device)
        return h_0, c_0

    def forward(self, input_text, x_coordinate=None, y_coordinate=None, z_coordinate=None):
        x_embed = self.relu_activation(self.encode_x(x_coordinate.cuda().to(torch.float32)).cuda())
        y_embed = self.relu_activation(self.encode_y(y_coordinate.cuda().to(torch.float32))).cuda()
        z_embed = self.relu_activation(self.encode_z(z_coordinate.cuda().to(torch.float32))).cuda()

        embeds = self._embeddings(input_text)
        embedding, hidden = self.rnn(embeds, self.hidden)
        text_fc = self.relu_activation(self.fc_after_text_lstm(embedding[:, -1]))
        representations_so_far_added = torch.sum(torch.stack([text_fc, x_embed, y_embed, z_embed]), dim=0)
        pre_final_embedding = self.relu_activation(self.fc(representations_so_far_added))

        return self.fc_final(pre_final_embedding)

# model = RNN(input_dim, hidden_dim, layer_dim, output_dim, batch_size)
model = BaselineModel().cuda()
criterion = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.001)

print('Start model training')
import sklearn.metrics as skm
import torch.nn.functional as F

x_train = []
x_coordinates = []
y_coordinates = []
z_coordinates = []
y_train = []
for i in range(10000):
    x_coordinate = round(np.random.uniform(-1, 1.00, 1)[0], 2)
    y_coordinate = round(np.random.uniform(-1, 1.00, 1)[0], 2)
    z_coordinate = round(np.random.uniform(-1, 1.00, 1)[0], 2)
    x_coordinates.append([x_coordinate])
    y_coordinates.append([y_coordinate])
    z_coordinates.append([z_coordinate])

    if np.random.randint(0, 2) == 0: # positive example
        if x_coordinate <= 0 and z_coordinate <= 0:
            x_train.append([1, 5, 6, 8])
        elif x_coordinate <= 0 and z_coordinate > 0:
            x_train.append([1, 5, 6, 9])
        elif x_coordinate > 0 and z_coordinate <= 0:
            x_train.append([1, 5, 6, 10])
        elif x_coordinate > 0 and z_coordinate > 0:
            x_train.append([1, 5, 6, 11])
        y_train.append(1.0)
    else:
        if x_coordinate <= 0 and z_coordinate <= 0:
            x_train.append(random.choice([[1, 5, 6, 9], [1, 5, 6, 10], [1, 5, 6, 11]]))
        elif x_coordinate <= 0 and z_coordinate > 0:
            x_train.append(random.choice([[1, 5, 6, 8], [1, 5, 6, 10], [1, 5, 6, 11]]))
        elif x_coordinate > 0 and z_coordinate <= 0:
            x_train.append(random.choice([[1, 5, 6, 8], [1, 5, 6, 9], [1, 5, 6, 11]]))
        elif x_coordinate > 0 and z_coordinate > 0:
            x_train.append(random.choice([[1, 5, 6, 8], [1, 5, 6, 9], [1, 5, 6, 10]]))
        y_train.append(0.0)

# print a sample of data
print(x_train[:10])
print(y_train[:10])
print(x_coordinates[:10])
print(y_coordinates[:10])
print(z_coordinates[:10])

# create a dataloader
trainingDataset = FeatureDataSet(x_train=x_train, y_train=y_train, x_coordinates=x_coordinates, y_coordinates=y_coordinates, z_coordinates=z_coordinates)
train_loader = torch.utils.data.DataLoader(dataset=trainingDataset, batch_size=batch_size, shuffle=True)

# for each epoch
loss_meter = AverageMeter()
for epoch in range(1, n_epochs + 1):
    acc_all = []
    # each batch
    loss_meter.reset()
    for i, (x_batch, y_batch, x_coord_batch, y_coord_batch, z_coord_batch) in enumerate(train_loader):
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)

        x_coord_batch = x_coord_batch.to(device)
        y_coord_batch = y_coord_batch.to(device)
        z_coord_batch = z_coord_batch.to(device)

        opt.zero_grad()
        # pass in the text (x_batch) and coordinate (x_coord_batch)
        out = model(x_batch, x_coordinate=x_coord_batch, y_coordinate=y_coord_batch, z_coordinate=z_coord_batch)
        loss = criterion(out.float(), y_batch.type(torch.LongTensor).cuda())

        loss.backward()
        opt.step()

        pred_idx = F.log_softmax(out, dim=1)
        target_labels = y_batch.cpu().int()
        pred_labels = torch.argmax(pred_idx, dim=-1).cpu().data.int()

        curr_acc = skm.accuracy_score(target_labels, pred_labels)
        acc_all.append(curr_acc)

        loss_meter.update(loss.item())

    print(np.mean(acc_all))
    print("loss is %f" % loss_meter.val)

As for the "minimally reproducible example", I think the model RNN doesn't work is quite reasonable, as I have stated in the comments. I suppose that tensorflow can not fit as well, although I have not tried it. Your "minimally reproducible example" may be unrelated to your main problem.


Instruction: What is called when `log_every_n_steps` of a pytorch lightning trainer is reached?

Input:
PL lightning trainer offers a parameter log_every_n_steps which it states controls "How often to add logging rows", however what is the function actually being called here? We can do our own logging every step with the example code below
def training_step(self, batch, batch_idx):
    self.log("performance", {"acc": acc, "recall": recall})

But is the trainer doing the same at the every nth step?

Output: log_every_n_steps will make the training log every n batches. This value is used by self.log if on_step=True. If you want a less bloated log-file, with the results per epoch only, you could do self.log(metrics, on_step=False, on_epoch=True)


Instruction: How to compute the outer sum (similar to outer product

Input:
Given tensors x and y, each with shape (num_batches, d), how can I use PyTorch to compute the sum of every combination of x and y within a batch?
This is similar to outer product, except we don't want to multiply, but sum.  (This implies that I could solve this by exponentiating, outer product, and taking the log, but of course that has numerical and performance disadvantages).
It could be done via cartesian product and then summing each of the combinations.
Essentially, I'd like osum[b, i, j] == x[b, i] + y[b, j].  Can PyTorch do this in tensors, without loops?

Output:
This can easily be done, by introducing singleton dimensions into x and y and broadcasting along these singleton dimensions:
osum = x[..., None] + y[:, None, :]

For example:
x = torch.arange(6).view(2,3)
y = x * 10
osum = x[..., None] + y[:, None, :]

Results with:

tensor([[[ 0, 10, 20],
         [ 1, 11, 21],
         [ 2, 12, 22]],

        [[33, 43, 53],
         [34, 44, 54],
         [35, 45, 55]]])

Update (July, 14th): How it works?
You have two tensors, x and y of shape bxn, and you want to compute:
osum[b,i,j] = x[b, i] + y[b, j]

We can, conceptually, create new variables xx and yy by repeating each element of x and y along a third dimension, such that:
xx[b, i, j] == x[b, i]  # for all j
yy[b, i, j] == y[b, j]  # for all i

With these new variables, it is easy to see that:
osum = xx + yy

since, by deinition
osum[b, i, j] == xx[b, i, j] + yy[b, i, j] == x[b, i] + y[b, j]

Now, you can use commands such as torch.expand or torch.repeat  to explicitly create xx and yy - but why bother? since their elements are just trivial repetitions of the elements along specific dimensions, broadcasting does this implicitly for you.


Instruction: How to make the sum of output to 1

Input:
My (PyTorch) sum of model’s output isn’t 1. And this is the structure of model.
LSTM(4433, 64)
LSTM(64, 64)
Linear(64, 4433)
Sigmoid()

And this is the predicted output of the model.
Input
[1, 0, 0, …, 0, 0]
Output
[.7842, .5, .5, …, .5, .5]

Do you know any function that can make its sum 1?

Output:
Sigmoid activation function maps every input to a value between [0, 1], without taking into account other elements in the input vector. However, Softmax does a similar transformation but the output vector sums 1.
TL;DR: use softmax instead of sigmoid.


Instruction: Trouble Understanding ResNet Implementation

Input:
I'm having trouble understanding and replicating the original implementation of ResNet on the CIFAR-10 dataset, as described in the paper "Deep Residual Learning for Image Recognition".  Specifically, I have a few questions about the following passage:

We use a weight decay of 0.0001 and momentum of 0.9,
and adopt the weight initialization in [13] and BN [16] but
with no dropout. These models are trained with a minibatch size of 128 on two GPUs. We start with a learning
rate of 0.1, divide it by 10 at 32k and 48k iterations, and
terminate training at 64k iterations, which is determined on
a 45k/5k train/val split. We follow the simple data augmentation in [24] for training: 4 pixels are padded on each side,
and a 32×32 crop is randomly sampled from the padded
image or its horizontal flip. For testing, we only evaluate
the single view of the original 32×32 image.

What does a minibatch size of 128 on two GPUs entail?  Does this mean the batch size per GPU is 64?

How can I convert from iterations to epochs?  Is the model trained for 64000 * 128/45000 = 182.04 epochs?

How can I implement the training and learning rate scheduling in PyTorch?  Since 45000 isn't divisible by 128, should I drop the last 72 images every epoch?  Also, since the 32k, 48k, and 64k milestones don't fall on a whole number of epochs, should I round them to the nearest epochs?  Or is there a way to change the learning rate and terminate training in the middle of an epoch?

If anyone could point me in the right direction, I greatly appreciate it.  I'm new to deep learning, so thank you for your help and kind understanding.

Output:
What does a minibatch size of 128 on two GPUs entail? Does this mean the batch size per GPU is 64?

When running two GPUs on the same machine then the batch size is split between the GPUs, as you've said. The gradient produced by both GPUs will be transfered, averaged and applied on one of the GPUs, or possibly on the CPU.
Here's more info: https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html

How can I convert from iterations to epochs? Is the model trained for 64000 * 128/45000 = 182.04 epochs?

I encourage everyone to think in terms of iterations rather than epochs. Each iteration equates to a single weight update, which is much more relevant to model convergence than an epoch is. If you think in epochs you have to adjust the number of epochs of training every time you try a different batch size. This isn't the case if you use think in terms of iterations (aka training steps, or weight updates). But your formula is correct in computing epochs.

How can I implement the training and learning rate scheduling in PyTorch?

I think this pytorch post answers the question, it looks like this was added to pytorch (sorry for a non authoritative answer here, I'm more familiar with Tensorflow):
https://forums.pytorchlightning.ai/t/training-for-a-set-number-of-iterations-without-setting-epochs/178
https://github.com/Lightning-AI/lightning/pull/5687
You can also just use epochs of course, and adjusting the learning rate doesn't have to happen exactly at the same point as the paper describes, as near as you can reasonably get with rounding error will work just fine.


Instruction: Using SKORCH with PyCaret for Regression problems

Input:
Using the fantastic article https://towardsdatascience.com/pycaret-skorch-build-pytorch-neural-networks-using-minimal-code-57079e197f33 there is a great example of using SKORCH and PyCaret to do Classification problems, but I am having trouble getting it working for Regression problems.
import pycaret
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
from skorch import NeuralNetRegressor
from sklearn.pipeline import Pipeline
from skorch.helper import DataFrameTransformer
from pycaret.regression import *
from pycaret.datasets import get_data

data = get_data('boston')
target = "medv"

reg1 = setup(data = data,
            target = target,
            train_size = 0.8,
            fold = 5,
            session_id = 123,
            silent = True)

class RegressorModule(nn.Module):
    def __init__(
            self,
            num_units=100,
            nonlin=F.relu,
    ):
        super(RegressorModule, self).__init__()
        self.num_units = num_units
        self.nonlin = nonlin

        self.dense0 = nn.Linear(14, num_units)
        self.nonlin = nonlin
        self.dense1 = nn.Linear(num_units, 10)
        self.output = nn.Linear(10, 1)

    def forward(self, X, **kwargs):
        X = self.nonlin(self.dense0(X))
        X = F.relu(self.dense1(X))
        X = self.output(X)
        return X

net_regr = NeuralNetRegressor(
    RegressorModule,
    max_epochs=20,
    lr=0.1,
    device='cuda'
)

nn_pipe = Pipeline(
    [
        ("transform", DataFrameTransformer()),
        ("net", net_regr),
    ]
)

skorch_model = create_model(nn_pipe)

But it errors with:

ValueError: The target data shouldn't be 1-dimensional but instead
have 2 dimensions, with the second dimension having the same size as
the number of regression targets (usually 1). Please reshape your
target data to be 2-dimensional (e.g. y = y.reshape(-1, 1).

If I take the same data and normalise it, reshape it etc and pass that straight to SKORCH, it works fine, like so:
X = data.copy().to_numpy().astype(np.float32)
mean = X.mean(axis=0)
X -= mean
std = X.std(axis=0)
X /= std

y = data[target].to_numpy().astype(np.float32)
y = y.reshape(-1, 1)
net_regr.fit(X, y)

So the problem is somewhere in how it takes the PyCaret (DataFrame based) data and SKORCH converts for use in PyTorch, that the y is staying single dimension, which is fine for the Classification model in the above link, but not for regression where it needs to be 2D.
Is there anyway I can intercept / transform the y?
Thanks :)

Output:
It is mentioned in Pytorch Dataset with Skorch. Anyway this will not solve the problem. If you overload the Fit of the NeuralNetworkRegressor like:
class MyNet(NeuralNetRegressor):
def fit(self, X, y):
    if y.ndim == 1:
        y = y.values.reshape(-1, 1)
    return super().fit(X, y)

net_regr = MyNet(
    RegressorModule,
    max_epochs=20,
    lr=0.1,
    train_split=None
)

it should work.


Instruction: Combine values from multiple dataframe rows into lists for every user

Input:
I have this csv file below (products rated by users) which into pandas dataframe:
--------------------------------
User_id | Product_id | Rating  |
--------------------------------
1       | 00         | 3       |
1       | 02         | 5       |
2       | 01         | 1       |
2       | 00         | 2       |
2       | 02         | 2       |

I want to change the dataframe so that it has the same number of rows as the source table above, but only two columns:

Column 1: needs to be a list of L length (L = total number of existing kinds of products), and where the n-th value (n = product_id) in the list is the rating given by the user in this row to the product. All all other values in the list need to be zeros
column 2 should be a list of the same L length, where the n-ths values equal to ratings for n-ths products (n = product_id) for all product_ids rated by this user (in the entire table); all other (unrated) values that are not rated by the user need to be zeros

The desired result would be (consistent with the example above):
--------------------------------
User_id | col1       | col2    |
--------------------------------
1       | [3,0,0]    | [3,0,5] |
1       | [0,0,5]    | [3,0,5] |
2       | [0,1,0]    | [2,1,2] |
2       | [2,0,0]    | [2,1,2] |
2       | [0,0,2]    | [2,1,2] |

I will greatly appreciate any help with this. Please do ask questions if i can make the question & explanation more clear.

Output:
I managed to solve this, however it feels like a lot of expensive code & operations for something relatively simple. If you have any ideas how to simplify this, I'd appreciate it a lot.
df = pd.read_csv('interactionsv21test.csv')

number_of_products = df['product_id'].nunique()

#assign indexes to products https://stackoverflow.com/questions/38088652/pandas-convert-categories-to-numbers
df.product_id = pd.Categorical(df.product_id)
df['product_indx'] = df.product_id.cat.codes
print('source table')
print(df.sort_values(['user_id', 'product_id', 'product_indx'], ascending=True).head(n=3))

df1 = (df.groupby([df['user_id']])
       .apply(lambda x: {int(i):int(k) for i,k in zip(x['product_indx'], x['rating'])})
       .reset_index(name='rating'))

#add blank values for non existing dictionary values https://stackoverflow.com/questions/38987/how-do-i-merge-two-dictionaries-in-a-single-expression
df1['rating_y'] = (df1['rating'].apply(lambda x: {int(k): 0 for k in range(number_of_products )} | x))

df['rating_x'] = df.apply(lambda row: {row['product_indx']:row['rating']}, axis=1)
df['rating_x'] = (df['rating_x'].apply(lambda x: {int(k): 0 for k in range(number_of_products)} | x ))

df = df[['user_id', 'rating_x']].merge(df1[['user_id','rating_y']],how='inner',left_on=['user_id'],right_on=['user_id'])

pd.set_option('display.max_columns', 7)
pd.set_option('display.width', 1000)

print('final result')
print(df.head(n=3))

Output:
source table
    user_id prod_id  rating  is_reviewed  prod_indx
40      198      63       5            1          1
0       198    2590       4            1         41
5       198    6960       4            1         51

final result
    user_id  rating_x                                           rating_y
...
40      198  {0: 0, 1: 5, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...  {0: 0, 1: 5, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...
41      198  {0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...  {0: 0, 1: 5, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...
42      198  {0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...  {0: 0, 1: 5, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: ...


Instruction: Numpy/Torch insert smallest value in case of collision

Input:
I have an empty numpy array, a list of indices, and list of values associated with the indices. The issue is that there may be duplicates in the indices. In all these "collision" cases, I'd like the smallest value to be picked. Just wondering what is the best way to go about it.
Eg:
array = [0,0,0,0,0,0,0]
indices = [0, 0, 2, 3, 2, 4]
values = [1.0, 3.0, 3.5, 1.5, 2.5, 8.0]

Result:
out = [1.0, 0, 2.5, 1.5, 8.0, 0.0, 0.0]

Output:
You can always implement something manually like:
import numpy as np

def index_reduce(arr, indices, out, reducer=min):
    touched = np.zeros_like(out, dtype=np.bool_)
    for i, x in enumerate(indices):
        if not touched[x]:
            out[x] = arr[i]
            touched[x] = True
        else:
            out[x] = reducer(out[x], arr[i])
    return out

which essentially loops through the indices and assign the values of arr to out if not already touched (keeping track of this with the touched array) and reducing the output with the specified reducer.
NOTE: The reducer function needs to be such that the final result can only depend on the current and previous value.

The usage of this would be:
indices = [0, 0, 2, 3, 2, 4]
values = [1.0, 3.0, 3.5, 1.5, 2.5, 8.0]
array = np.zeros(7)

index_reduce(values, indices, array)
# array([1. , 0. , 2.5, 1.5, 8. , 0. , 0. ])

If performances are of concern, you can also accelerate the above code with Numba with a simple decoration provided that also the values and indices inputs are NumPy arrays:
import numba as nb

index_reduce_nb = nb.njit(index_reduce)

indices = np.array([0, 0, 2, 3, 2, 4])
values = np.array([1.0, 3.0, 3.5, 1.5, 2.5, 8.0])
array = np.zeros(7)

index_reduce_nb(values, indices, array)
# array([1. , 0. , 2.5, 1.5, 8. , 0. , 0. ])

Benchmarks
The above solutions can be compared to a Torch-based solution (reworked from @Shai's answer):
import torch

def index_reduce_torch(arr, indices, out, reduce_="amin"):
    arr = torch.from_numpy(arr)
    indices = torch.from_numpy(indices)
    out = torch.from_numpy(out)
    return out.index_reduce_(dim=0, index=indices, source=arr, reduce=reduce_, include_self=False).numpy()

or, with additional skipping of Torch gradients:
index_reduce_torch_ng = torch.no_grad()(index_reduce_torch)
index_reduce_torch_ng.__name__ = "index_reduce_torch_ng"

and a Pandas-based solution (reworked from @bpfrd's answer):
import pandas as pd

def index_reduce_pd(arr, indices, out, reducer=min):
    df = pd.DataFrame(data=zip(indices, arr))
    df1 = df.groupby(0, as_index=False).agg(reducer)
    out[df1[0]] = df1[1]
    return out

using the following code:
funcs = index_reduce, index_reduce_nb, index_reduce_pd, index_reduce_torch, index_reduce_torch_ng

timings = {}
for i in range(4, 18):
    n = 2 ** i
    print(f"n = {n}, i = {i}")
    extrema = 0, 2 * n
    indices = np.random.randint(*extrema, n)
    values = np.random.random(n)
    out = np.zeros(extrema[1] + 1)
    timings[n] = []
    base = funcs[0](values, indices, out)
    for func in funcs:
        res = func(values, indices, out)
        is_good = np.allclose(base, res)
        timed = %timeit -r 16 -n 16 -q -o func(values, indices, out)
        timing = timed.best * 1e6
        timings[n].append(timing if is_good else None)
        print(f"{func.__name__:>24}  {is_good}  {timing:10.3f} µs")

to produce with the additional lines:
import matplotlib.pyplot as plt

df = pd.DataFrame(data=timings, index=[func.__name__ for func in funcs]).transpose()
df.plot(marker='o', xlabel='Input size / #', ylabel='Best timing / µs', figsize=(6, 4))
df.plot(marker='o', xlabel='Input size / #', ylabel='Best timing / µs', ylim=[0, 500], figsize=(6, 4))
fig = plt.gcf()
fig.patch.set_facecolor('white')

these plots:

(the second is a zoomed-in version of the first).
These indicate that the Numba accelerated solution could be the fastest, closely followed by the Torch-based solution while the Pandas approach could be the slowest, even slower than the explicit solution without acceleration.


Instruction: How to I reshape the 2D array like this? (By using tensor)

Input:
I want to resize my image from 32 * 32 to 16 * 16. (By using torch.tensor)
Like decreasing the resolution?
Can anyone help me?

Output:
If you have an image (stored in a tensor) and you want to decrease it's resolution, then you are not reshaping it, but rather resizing it.
To that end, you can use pytorch's interpolate:
import torch
from torch.nn import functional as nnf

y = nnf.interpolate(x[None, None, ...], size=(16, 16), mode='bicubic', align_corners=False, antialias=True)

Notes:

nnf.interpolate operates on batches of multi-channel images, that is, it expects its input x to have 4 dimensions: batch-channels-height-width. So, if your x is a single image with a single channel (e.g., an MNIST digit) you'll have to create a singleton batch dimension and a singleton channel dimension.
Pay close attention to align_corners and antialias -- make sure you are using the right configuration for your needs.

For more information regarding aliasing and alignment when resizing images you can look at ResizeRight.


Instruction: Why is PyTorch inference non deterministic even when setting `model.eval()`

Input:
I have fine-tuned a PyTorch transformer model using HuggingFace, and I'm trying to do inference on a GPU. However, even after setting model.eval() I still get slightly different outputs if I run inference multiple times on the same data.
I have tried a number of things and have done some ablation analysis and found out that the only way to get deterministic output is by also setting
torch.cuda.manual_seed_all(42)

(or any other seed number).
Why is this the case? And is this normal? The model's weights are fixed, and there are no undefined or randomly initialized weights (when I load the trained model I get the All keys matched successfully message), so where is the randomness coming from if I don't set the cuda seed manually? Is this randomness to be expected?

Output:
You can use torch.use_deterministic_algorithms to force non-deterministic modules to perform deterministically, where supported e.g:
>>> a = torch.randn(100, 100, 100, device='cuda').to_sparse()
>>> b = torch.randn(100, 100, 100, device='cuda')

# Sparse-dense CUDA bmm is usually nondeterministic
>>> torch.bmm(a, b).eq(torch.bmm(a, b)).all().item()
False

>>> torch.use_deterministic_algorithms(True)

# Now torch.bmm gives the same result each time, but with reduced performance
>>> torch.bmm(a, b).eq(torch.bmm(a, b)).all().item()
True

# CUDA kthvalue has no deterministic algorithm, so it throws a runtime error
>>> torch.zeros(10000, device='cuda').kthvalue(1)
RuntimeError: kthvalue CUDA does not have a deterministic implementation...


Instruction: Replacing THC/THC.h module to ATen/ATen.h module

Input:
I have question about replacing <THC/THC.h> method.
Recently, I'm working on installing different loss functions compiled with cpp and cuda.
However, what I faced was a fatal error of
'THC/THC.h': No such file or directory

I found out that TH(C) methods were currently deprecated in recent version of pytorch, and was replaced by ATen API (https://discuss.pytorch.org/t/question-about-thc-thc-h/147145/8).
For sure, downgrading my pytorch version will solve the problem. However, due to my GPU compatibility issue, I have no choice but to modify the script by myself. Therefore, my question can be summarized into follows.
First, how can I replace codes that have dependency of TH(C) method using ATen API?. Below are codes that I have to modify, replacing those three lines looked enough for my case.
#include <THC/THC.h>
extern THCState *state;
cudaStream_t stream = THCState_getCurrentStream(state);

Second, will single modification on cpp file be enough to clear the issue that I'm facing right now? (This is just a minor question, answer on first question will suffice me).
For reference, I attach the github link of the file I'm trying to build (https://github.com/sshaoshuai/Pointnet2.PyTorch).

Output:
After struggling for a while, I found the answer for my own.
In case of  THCState_getCurrentStream, it could directly be replaced by at::cuda::getCurrentCUDAStream(). Therefore, modified code block was formulated as below.
//Comment Out
//#include <THE/THC.h>
//extern THCState *state;
//cudaStream_t stream = THCState_getCurrentStream(state);

//Replace with
#include <ATen/cuda/CUDAContext.h>
#include <ATen/cuda/CUDAEvent.h>
cudaStream_t stream = at::cuda::getCurrentCUDAStream();

After replacing the whole source code, I was able to successfully build the module.
Hope this helps.


Instruction: Pytorch: Why does altering the scale of the loss functions improve the convergence in some models?

Input:
I have a question surrounding a pretty complex loss function I have.
This is a variational autoencoder loss function and it is fairly complex.  It is made of two reconstruction losses, KL divergence and a discriminator as a regularizer. All of those losses are on the same scale, but I have found out that increasing one of the reconstruction losses by a factor of 20 (while leaving the rest on the previous scale) heavily increases the performance of my model.
Since I am still fairly novice on DL, I dont completely understand why this happens, or how I could identify this sort of thing on successive models.
Any advice/explanation is greatly appreciated.

Output:
To summarize your setting first:
loss = alpha1 * loss1 + alpha2 * loss2

When computing the gradients for backpropagation, we compute back through this formular. By backpropagating through our error function we get the gradient:
dError/dLoss

To continue our propagation downwards, we now want to compute dError/dLoss1 and dError/dLoss2.
dError/dLoss1 can be expanded to dError/dLoss * dLoss/dLoss1 via the cain rule (https://en.wikipedia.org/wiki/Chain_rule).
We already computed dError/dLoss so we only need to compute dLoss derived with respect to dLoss1, which is
dLoss/dLoss1 = alpha1
The backpropagation now continues until we reach our weights (dLoss1/dWeight). The gradient our weight receives is:
dError/dWeight = dError/dLoss * dLoss/dLoss1 * dLoss1/dWeight = dError/dLoss * alpha1 * dLoss1/dWeight

As you can see, the gradient used to update our weight does now depend on alpha1, the factor we use to scale Loss1.
If we increase alpha1 while not changing alpha2 the gradients depending on Loss1 will have higher different impact than the gradients of Loss2 and therefor changing the optimization of our model.


Instruction: pytorch error in multiplying matrices in neural network

Input:
I was trying to make a Neural Network in PyTorch, however I ran into the error below. I'm still new to this topic so I am not able to understand how I should go about solving this.

Code:
class ANN_Model(nn.Module):
    def __init__(self,input_features=8,hidden1=8,hidden2=200,hidden3=200,hidden4=300,hidden5=300,hidden6=400,hidden7=400,hidden8=300,hidden9=300,out_features=2):
        super().__init__()
        self.f_connected1=nn.Linear(input_features,hidden1)
        self.f_connected2=nn.Linear(hidden1,hidden2)
        self.f_connected2=nn.Linear(hidden2,hidden3)
        self.f_connected2=nn.Linear(hidden3,hidden4)
        self.f_connected2=nn.Linear(hidden4,hidden5)
        self.f_connected2=nn.Linear(hidden5,hidden6)
        self.f_connected2=nn.Linear(hidden6,hidden7)
        self.f_connected2=nn.Linear(hidden7,hidden8)
        self.f_connected2=nn.Linear(hidden8,hidden9)
        self.out=nn.Linear(hidden9,out_features)
    def forward(self,x):
        x=F.relu(self.f_connected1(x))
        x=F.relu(self.f_connected2(x))
        x=F.relu(self.f_connected3(x))
        x=F.relu(self.f_connected4(x))
        x=F.relu(self.f_connected5(x))
        x=F.relu(self.f_connected6(x))
        x=F.relu(self.f_connected7(x))
        x=F.relu(self.f_connected8(x))
        x=F.relu(self.f_connected9(x))
        x=self.out(x)
        return x
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)
epochs = 500
final_losses = []
for i in range(epochs):
    i = i + 1
    y_pred = model.forward(X_train)
    loss=loss_function(y_pred, y_train)
    final_losses.append(loss.item())
    if i%10==1:
        print("Epoch number: {} and the loss: {}".format(i, loss.item()))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [13], in <cell line: 3>()
      3 for i in range(epochs):
      4     i = i + 1
----> 5     y_pred = model.forward(X_train)
      6     loss=loss_function(y_pred, y_train)
      7     final_losses.append(loss.item())

Input In [8], in ANN_Model.forward(self, x)
     14 def forward(self,x):
     15     x=F.relu(self.f_connected1(x))
---> 16     x=F.relu(self.f_connected2(x))
     17     x=F.relu(self.f_connected3(x))
     18     x=F.relu(self.f_connected4(x))

File ~/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)
   1126 # If we don't have any hooks, we want to skip the rest of the logic in
   1127 # this function, and just call forward.
   1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1129         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1130     return forward_call(*input, **kwargs)
   1131 # Do not call functions when jit is used
   1132 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -> Tensor:
--> 114     return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (691x8 and 300x300)

I was trying to make a Neural Network in PyTorch, however I ran into the error below. I'm still new to this topic so I am not able to understand how I should go about solving this.

Output: I found it, in your model's constructor __init__ every layer is named self.f_connected2 and because of that it expects a shape of (batch_size,300).


Instruction: Can't import VecFrameStackFrame from Stable-baselines3 - importing problem

Input:
I have a problem when importing some dependencies from stable baselines 3 library, I installed it with this command
pip install stable-baselines3[extra]

But When I import my dependencies
import gym
from stable_baselines3 import A2C
from stable_baselines3.common.vec_env import VecFrameStackFrame
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.env_util import make_atari_env
import os

I face this error
ImportError: cannot import name 'VecFrameStackFrame' from 'stable_baselines3.common.vec_env' (C:\Users\User\anaconda3\envs\rl_learning\lib\site-packages\stable_baselines3\common\vec_env\__init__.py)

Any advice?

Output:
I knew that stable baselines new version has changed the name from
from stable_baselines3.common.vec_env import VecFrameStackFrame

To
from stable_baselines3.common.vec_env import vec_frame_stack

and it worked for me


Instruction: Is it more beneficial to read many small files or fewer large files of the exact same data?

Input:
I am working on a project where I am combining 300,000 small files together to form a dataset to be used for training a machine learning model. Because each of these files do not represent a single sample, but rather a variable number of samples, the dataset I require can only be formed by iterating through each of these files and concatenating/appending them to a single, unified array. With this being said, I unfortunately cannot avoid having to iterate through such files in order to form the dataset I require. As such, the process of data loading prior to model training is very slow.
Therefore my question is this: would it be better to merge these small files together into relatively larger files, e.g., reducing the 300,000 files to 300 (merged) files? I assume that iterating through less (but larger) files would be faster than iterating through many (but smaller) files. Can someone confirm if this is actually the case?
For context, my programs are written in Python and I am using PyTorch as the ML framework.
Thanks!

Output:
Usually working with one bigger file is faster than working with many small files.
It needs less open, read, close, etc. functions which need time to

check if file exists,
check if you have privilege to access this file,
get file's information from disk (where is beginning of file on disk, what is its size, etc.),
search beginning of file on disk (when it has to read data),
create system's buffer for data from disk (system reads more data to buffer and later function read() can read partially from buffer instead of reading partially from disk).

Using many files it has to do this for every file and disk is much slower than buffer in memory.


Instruction: Why does this custom function cost too much time while backward in pytorch?

Input:
I'm revising a baseline method in pytorch. But when I add a custom function in the training phase, the cost time of backward increases 4x on a single V100. Here is an example of the custom function:
def batch_function(M, kernel_size=21, sf=2):
    '''
    Input:
        M: b x (h*w) x 2 x 2 torch tensor
        sf: scale factor
    Output:
        kernel: b x (h*w) x k x k  torch tensor
    '''

    M_t = M.permute(0,1,3,2)  # b x (h*w) x 2 x 2
    INV_SIGMA = torch.matmul(M_t, M).unsqueeze(2).unsqueeze(2)  # b x (h*w) x 1 x 1 x 2 x 2

    X, Y = torch.meshgrid(torch.arange(kernel_size), torch.arange(kernel_size))
    Z = torch.stack((Y, X), dim=2).unsqueeze(3).to(M.device)   # k x k x 2 x 1

    Z = Z.unsqueeze(0).unsqueeze(0) # 1 x 1 x k x k x 2 x 1
    Z_t = Z.permute(0,1,2,3,5,4)  # 1 x 1 x k x k x 1 x 2
    raw_kernel = torch.exp(-0.5 * torch.squeeze(Z_t.matmul(INV_SIGMA).matmul(Z)))  # b x (h*w) x k x k

    # Normalize
    kernel = raw_kernel / torch.sum(raw_kernel, dim=(2,3)).unsqueeze(-1).unsqueeze(-1)   # b x (h*w) x k x k

    return kernel

where b is the batch size, 16; h and w are the spatial dimensions, 100; k is equal to 21. I'm not sure if the large dimension of M causes the cost time longer.
Why does the cost time longer? And are there other methods to rewrite this code to improve it?
I'm new here, so if the problem is not clearly described, please let me know!

Output:
You might be able to get a performance boost on the double tensor multiplication by using torch.einsum:
>>> o = torch.einsum('acdefg,bshigj,kldejm->bsdefm', ZZ_t, INV_SIGMA, ZZ)

The resulting tensor o will be shaped (b, h*w, k, k, 1, 1)

For details on the subscript notation:

b: batch dimension.
s: 's' for spatial, i.e. the h*w dimension.
d and e: the two k dimensions which are paired across ZZ_t and ZZ.

A simple 2D matrix multiplication applying matmul with ij,jk->ik.
Keeping that in mind, we have in your case:

A first multiplication: r = ZZ_t@INV_SIGMA which does something like *fg,*gj->*fj, the asterisk sign * refers to leading dimensions.

A second matrix multiplication: r@INV_SIGMA which comes down to *fj,*jm->*fm.

Overall, if we combine both, we get directly: *fg,*gj,*jm->*fm.
Finally, I have assigned all other dimensions to random but different subscript letters:
a, c, f, h, i, k, l

Replacing the asterisk above with those notations, we get the following subscript input:
#  *  fg, *  gj, *  jm-> *  fm
# acdefg,bshigj,kldejm->bsdefm


Instruction: Pytorch error mat1 and mat2 shapes cannot be multiplied in Autoencoder to compress images

Input:
I receive this error. Whereas the size of my input image is 3x120x120, so I flatten the image by the following code, however, I received this error:
mat1 and mat2 shapes cannot be multiplied (720x120 and 43200x512)
I have tu use an autoencoder to compress my images of a factor of 360 ( So i started from 3x120x120 input to 120 in the encoder).
my code:
class AE(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder = torch.nn.Sequential(
            torch.nn.Linear(3*120*120, 512),
            torch.nn.ReLU(),
            torch.nn.Linear(512, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 120)
        )

        self.decoder = torch.nn.Sequential(
            torch.nn.Linear(120, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 512),
            torch.nn.ReLU(),
            torch.nn.Linear(512, 3*120*120),
            torch.nn.Sigmoid()
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

Output:
I think you missed some basic characteristics of nn.Linear function. Its inputs are input and output channel dimension, respectively. And therefore only 1D input is allowable (considering batch, which is 3 in your case, it will be 2D in total ). Therefore you should first flatten your x for later compute compatibility.
class AE(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.encoder = torch.nn.Sequential(
            torch.nn.Linear(120*120, 512),
            torch.nn.ReLU(),
            torch.nn.Linear(512, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 120)
        )

        self.decoder = torch.nn.Sequential(
            torch.nn.Linear(120, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 256),
            torch.nn.ReLU(),
            torch.nn.Linear(256, 512),
            torch.nn.ReLU(),
            torch.nn.Linear(512, 3*120*120),
            torch.nn.Sigmoid()
        )

    def forward(self, x):
        x = x.reshape(3, -1)
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

I recommend you to carefully search for the structure and dimensional change of common neural networks. It will help you alot for sure.


Instruction: PyTorch CUDA : the provided PTX was compiled with an unsupported toolchain

Input:
I am using Nvidia V100 with the following specs:
(pytorch) [s.1915438@cl1 aneurysm]$ srun nvidia-smi
Sun Jul 17 16:17:27 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  On   | 00000000:D8:00.0 Off |                    0 |
| N/A   31C    P0    25W / 250W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The Python, Pytorch and CUDA version is as follows:
Python 3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.12.0+cu113'

When I run a python file, containing a machine learning model, I get the following error.
(pytorch) [s.1915438@cl1 aneurysm]$ srun python aneurysm.py
terminate called after throwing an instance of 'std::runtime_error'
  what():  the provided PTX was compiled with an unsupported toolchain.
srun: error: ccs2114: task 0: Aborted

Is it some kind of compatibility issue? Should I fallback to CUDA 10
.2 as the V100 is very old GPU?

Output:
Anyone using an old GPU from an HPC cluster is probably out of luck. In my case, I had Nvidia Driver 495 which is not very old. In fact, for CUDA 11.5 they recommend Nvidia Driver 470.
This is the official reply from Nvidia for a similar problem. They also recommend updating the driver. And most of the time HPC centres won't update the driver on personal requests.


Instruction: Why my dataloader output structure differs from structure in pytorch official guide?

Input:
I was following DCGAN tutorial(https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) and I've got problem with format of output they get from their dataloader. So, in short, if they need any data from dataloader they call it like that:
real_batch = next(iter(dataloader))
(real_batch[0].to(device)[:64]

or
for i, data in enumerate(dataloader, 0):
real_cpu = data[0].to(device)

and that data[0] they are calling has batch size (set up to be 128), in the first example they need 64 data samples, so they use [:64] to cut them.
The problem is that my dataloader doesn't follow such behaviour, and it spit out batch size just after next call or in the enumerate cycle, calling data[0] for my dataloader return just one sample, not entire batch like in the example. I found this extremely weird, because just by removing [0] in each data load I make my code run without any errors, but I'm afraid that I'm missing some important part of making data to specific shape and that could cause some errors.
This is how their dataloader has been set up:
dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)

My dataloader set up is a bit tricky, I have a simple custom dataset with data being a list of one channel images (I'm creating this list myself with augmentation function called on images stored on disk, and I'm not sure if this is a good way to store such data...), and basically more or less the same dataloader params.
class MyDataset(torch.utils.data.Dataset):
    def __init__(self, dataset, image_size):
        super(MyDataset, self).__init__()
        # dataset is [list] of PIL images with 1 channel
        self.dataset = dataset
        self.image_size = image_size
        self.transform=transforms.Compose([transforms.Resize(self.image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5), (0.5))])

    def __getitem__(self, idx):
        x = self.dataset[idx]
        return self.transform(x)

    def __len__(self):
        return len(self.dataset)

and dataloader itself:
train_set = MyDataset(data, image_size=image_size)
data_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)

Output:
The difference is that typically the implemented datasets will return both the image AND the corresponding label, i.e. the implementation of the __getitem__ method is something like:
def __getitem__(self, idx):
    return self.image[idx], self.target[idx]

Then, the dataloader returns a tuple: data = (images, targets), both of the same batch size. They access images by taking data[0].
In your case, your __getitem__ returns only one output, and the dataloader will collate it into simple data = images.
So, removing [0], as you tried, is actually a correct thing to do! For compatibility with other already implemented datasets, I sometimes return a dummy label together with the sample in __getitem__, e.g. return self.transform(x), 0 (you can try it -- then calling data[0] will work).