How do I use two loss for two dataset in one pytorch nn?
I am quite new to pytorch and deep learning. Here is my question. I have two different datasets with the same feature domain sharing one neural network for a regression problem. The input is the features and the output is the target value. The first dataset uses a normal loss while the second dataset, I am trying to create a new loss for it.
I have searched multi-loss problems, people usually have two loss summed up for the backward process. But I want to use the loss in turn. (When I train the first dataset, the nn uses the first loss and when I train the second dataset, the nn uses the other loss)
Is this possible to do? Appreciate if anyone has some idea.

The loss function does not necessarily have to do with network topology. You can use the corresponding loss with each dataset you use, e.g.
if first_task:
    dataloader = torch.utils.data.DataLoader(first_dataset)
    loss_fn = first_loss_fn
else: 
    dataloader = torch.utils.data.Dataloader(second_dataset)
    loss_fn = second_loss_fn

# The pytorch training loop, very roughly
for batch in dataloader: 
   x, y = batch
   optimizer.zero_grad()
   loss = loss_fn(network.forward(x), y) # calls the corresponding loss function
   loss.backward()
   optimizer.step()

You can do this for the two datasets sequentially (meaning you interleave by epochs):
for batch in dataloader_1: 
    ...
    loss = first_loss_fn(...)

for batch in dataloader_2: 
    ...
    loss = second_loss_fn(...)

or better
dataset = torch.utils.data.ChainDataset([first_dataset, second_dataset])
dataloader = torch.utils.data.DataLoader(dataset) 

You can also do simultaneously (interleave by examples). The standard way I think would be to use torch.utils.data.ConcatDataset
dataset = torch.utils.data.ConcatDataset([first_dataset, second_dataset])
dataloader = torch.utils.data.DataLoader(dataset)

Note that here you need each sample to store information about the dataset it comes from so you can determine which cost to apply.
A simpler way would be to interleave by batches (then you apply the same cost to the entire batch). For this case one way proposed here is to use separate dataloaders (this way you get flexibility on how often to sample each of them).
