Pytorch: multiple datasets with multiple losses
I am using multiple datasets. I have multiple losses, each of which must be evaluated on a subset of these datasets. I want to generate a batch from each dataset, and evaluate each loss on all of its appropriate batches. Some of the losses are pairwise (need to load pairs of corresponding datapoints) whereas others are computed on single datapoints. I need to design this in such a way that is open to easily adding new datasets. Is there any pytorch builtin that would help with this? What is the best way to design this in pytorch? Thanks in advance.

It's not clear from your question what exactly your settings are.
However, you can have multiple Datasets instances, one for each of your datasets.
On top of your datasets, you can implement a "tagged dataset", a dataset that adds a "tag" for all samples:

class TaggedDataset(data.Dataset):
  def __init__(dataset, tag):
    super(TaggedDataset, self).__init__()
    self.ds_ = dataset
    self.tag_ = tag

  def __len__(self):
    return len(self.ds_)

  def __getitem__(self, index):
    return self.ds_[index], self.tag_


Give a different tag to each dataset, concat all of them into a single ConcatDataset, and wrap a regular DataLoader around it.

Now, in your training code

for input, label, tag in my_tagged_loader:
  # process each input according to the dataset tag it got.

