How to use pytorch lightning Accuracy with ignore class?
I have some training pipeline which uses CrossEntropyLoss with an ignore class.
The model outputs log_probs of shape (150, 3) - meaning 3 possible classes in batches of 150.
The label_batch is of shape 150, and torch.max(label_batch) == tensor(3, device='cuda:0'), meaning there is an extra class labeled 3, which is the ignore class.
The loss handles it fine:
self._criterion = nn.CrossEntropyLoss(
    reduction='mean',
    ignore_index=3
)

But the accuracy metric thinks class 3 is valid and gives very wrong results:
self.train_acc = pl.metrics.Accuracy()
wrong result with self.train_acc.update(log_probs, label_batch) because of the 3 label should be ignored.

How to correctly use pl.metrics.Accuracy() with an ignore class?

Copying response from the discussion thread in the github forum https://github.com/PyTorchLightning/pytorch-lightning/discussions/6890

It is currently not supported in the accuracy metric, but we have an open PR for implementing that exact feature PyTorchLightning/metrics#155
Currently what you can is instead calculate the confusion matrix and then ignore some classes based on that (remember that the true positive/correctly classified are found on the diagonal of the confusion matrix):
ignore_index = 3
metric = ConfusionMatrix(num_classes=3)
confmat = metric(preds, target)
confmat = confmat[:2,:2] # remove last column and row corresponding to class 3
acc = confmat.trace() / confmat.sum()

