Is Loss not a good indication of performance?
To answer the title of your question:


  Is Loss not a good indication of performance?


It is only a relative indicator of performance over the training session.

First, what is loss anyway? In general, the loss is some expression of the difference between the model's predicted output and the target output. Depending on the loss function used (e.g. if log is involved) or on the nominal values of the inputs themselves, the value of the loss can also be very small or very large.

People usually normalise data so the values are smaller, but the point is that you cannot always say that a validation loss of 0.0012345 is actually a good value. Or that 12345 is definitely bad!

Other possible loss functions

Other 3d segmentation models that I have seen it is common to use the Dice Coefficient for your cost/loss. Maybe give that a try.  The Dice coefficient is essentially the same as the F1 score; you are really finding a trade-off in how to penalise a model for its mistakes in classification e.g. of pixels or voxels. Do you want to strongly punich bad cases or rather a more averging approach. As that link points out, it is similar to the difference between the $L_1$ and $L_2$ losses.

There is also the Jaccard index, which is essentially the same as the Dice Coefficient.

The Tversky index is a generalisation of the two - it is an asymmetric similarity measure. :

$$
Tversky(A, B; α, β) = \frac{|TruePos|}{|TruePos| + \alpha |FalsePos| + \beta |FalseNeg|}
$$


The Dice coefficient is this with $\alpha = \frac{1}{2}$ and $\beta = \frac{1}{2}$
The Jaccard index instead has $\alpha = 1$ and $\beta = 1$


This might be a nice approach for your problem to tweak how the loss is computed.
There are plenty of source online to explain more about these well-defined measures and perhaps help you gain an intuition for their results.

Your results

With the losses you posted, it doesn't actually look as good as you explained. I am not sure exactly what your idx values mean, but the loss values are actually going up:

In [1]: import numpy as np                                                         
In [2]: import pandas as pd
In [3]: import matplotlib.pyplot as plt
In [4]: val_loss = np.array([0.029650183394551277, 0.009899887256324291, 0.0504908040
   ...: 1659012, 0.02019292116165161, 0.04724293574690819, 0.02810296043753624, 0.026
   ...: 42594277858734, 0.029894422739744186, 0.04158024489879608, 0.0457481481134
   ...: 8913, 0.05406259000301361])    
In [5]: pd.Series(val_loss, index=range(0, 51, 5)).plot()                           
Out[5]: &lt;matplotlib.axes._subplots.AxesSubplot at 0x7f4405326390&gt;
In [6]: plt.show()  




This suggests you might be overfitting. However, the differences between the predictions and ground truth in the images you show suggest there is a more fundamental problem.