Implementing SmoothL1Loss for specific case
I have been experimenting with L1 and MSE losses in Pytorch and noticing that L1Loss performs better because it is more robust to outliers. I discovered SmoothL1Loss which seems to be the best of both worlds. I understand that it behaves like MSELoss for error&lt;1 and like L1Loss otherwise. My dataset only contains values between 0 and 1. Therefore the largest possible error is 1. Does this mean the function behaves identical to MSELoss? Is it possible adjust the threshold in anyway to work better for my problem?

Yes, in this case it acts just like torch.nn.MSELoss, and it is called Huber Loss all in all.

Due to it's nature threshold doesn't make much sense, let's look at example why that is the case:

How it works

Let's compare errors being larger than 1.0 in case of MSELoss and SmoothL1Loss. Assume our absolute error (|f(x) - y|) is 10. MSELoss would give it value of 100 (or 50 in case of pytorch implementation), while SmoothL1Loss gives just this value of 10, hence it won't punish the model so much for large errors.

In case of value below 1.0 SmoothL1Loss punishes the model less than L1Loss. E.g. 0.5 would become 0.5*0.5 so 0.25 for Huber and 0.5 for L1Loss.

It's not "best of both worlds" it depends what you are after. Mean Squared Error - amplifies large errors and downplays the small ones, L1Loss gives errors "equal" weight let's say.

Custom loss function

Though it's not usually done you could use any loss function you'd like, depending on your goal (threshold doesn't really make sense here). If you want smaller errors to be more severe you could, for example, do something like this:

import torch

def fancy_squared_loss(y_true, y_pred):
    return torch.mean(torch.sqrt(torch.abs(y_true - y_pred)))


For value 0.2 you would get ~0.447, for 0.5 ~0.7 and so on. Experiment and check whether any specific loss functions exist for task at hand, though I think it's unlikely those experiments will give you significant boost over L1Loss if any.

Custom threshold

If you really want to set custom threshold for MSELoss and L1Loss you could implement it on your own though:

import torch

class CustomLoss:
    def __init__(self, threshold: float = 0.5):
        self.threshold = threshold

    def __call__(self, predicted, true):
        errors = torch.abs(predicted - true)
        mask = errors &lt; self.threshold
        return (0.5 * mask * (errors ** 2)) + ~mask * errors


Everything below threshold would get MSELoss while all above would have L1Loss.
