It is not immediately clear whether the variation in MSE loss across 𝑏 values in Fig. is inherent to the problem of training a NN to perform the LS method or a result of randomness in the training of the NN. There are two sources of randomness in the training process. First, the NN is initialized with random weight parameters (bias parameters are set to zero). The second source of randomness is from the training process. A stochastic gradient descent optimization method, like the Adam method used here , randomly partitions the training data into small batches over which many small gradient descent steps are computed. Each training epoch combines all of these small gradient descents to estimate the true gradient descent direction with respect to the entire training dataset to reduce MSE loss, significantly reducing computational costs. Even if a small batch of training data is not representative of the true gradient descent direction, taken together over an entire epoch, many small gradient steps often approximate the true gradient very well. The random partitioning of the training data into small batches and the random order in which each small batch is used by the optimizer contribute to the randomness of training. Due to the high dimensionality of the NN parameter space it is extremely rare that the NN will be trapped in a local minimum, but also extremely likely that the path taken by the optimizer will differ between training realizations, even with identical initialization of the NN parameters.