When is a neural network better "traditional" models like decisions trees and lassos?
Neural network are "just" a way of estimating a numerical function. Their power is that this comes out of simple matrix multiplications with non linear functions that can be efficiently parallelized. As such, they are perfectly described by calculus.

The question on whether they are biased or not is then related to how they are trained and their black-box behavior. You can put stats at the end of a neural network as well.

The problem IMHO is that generalization for them is a complex topic. Presented with new data, they can have unexpected behavior compared to more traditional techniques for which we understand the behavior at the edge.

If you consider simple regression, like least squares, then neural networks are not actually in play. We use the frameworks to build equations, but we don't build neural networks. As such, we just benefit from the capability of these to process huge amount of data.

We are still going to estimate $y=ax+b$ if we want to estimate the coefficients. For logistic regression, it's still the same boundaries as if we were doing a logistic regression without neural networks.

Neural networks are only in play when the function is unknown and needs to be estimated.