Can Neural Networks be trained to smooth values / output the average? Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process

$$f(x) = ax + \mathcal{N}(b, c)$$

with $a, b, c \in \mathbb{R}^+$, e.g. something like

feature  | target
-----------------
0             0.0
0             1.0
0             1.5
0            -1.2
0            -0.9
...


I know that neural networks can deal pretty well with labeling errors in classification problems. Meaning if you have a large dataset and a couple of examples have the wrong label, they get basically ignored.

But for this kind of problem I'm not too sure. A first experiment indicates that they do smooth values.

Are there choices in architecture / training which help the smoothing / averaging / removal of noise?

What I tried

I created a network which can solve this kind of regression problem without noise. It gets a MSE of about 0.0005. When I add a bit of noise to the training set only, I get an MSE of 0.001:

#!/usr/bin/env python

# core modules
import random

# 3rd party modules
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np


def main(add_noise=True):
    # Get data
    xs, ys = create_data_points(10000)
    x_train, x_test, y_train, y_test = train_test_split(xs, ys, test_size=0.20)

    # Add noise to training data
    if add_noise:
        noise = np.random.normal(0, 0.1, len(x_train))
        x_train = x_train + noise

    # Create model
    model = create_model()
    model.compile(optimizer='rmsprop',
                  loss='mse',
                  metrics=['mse'])

    # Fit model to data.
    model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)

    # Evaluate
    y_pred = model.predict(x_test, batch_size=100).flatten()
    print("MSE on test set:")
    print(((y_pred - y_test)**2).sum() / len(y_test))


def create_data_points(nb_points):
    xs = []
    ys = []
    for i in range(nb_points):
        x = random.random()
        xs.append(x)
        ys.append(2 * x)
    return np.array(xs), np.array(ys)


def create_model(input_dim=1, output_dim=1):
    model = Sequential()
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(output_dim, activation='linear'))
    return model


if __name__ == '__main__':
    main()


Outliers

In an earlier version of this question I wrote "outlier" when I meant "label noise". For outliers, there is:


The Effects of Outliers Data on Neural Network Performance
In general simpler models are more robust to noise in the input.  The strength of neural networks is also their biggest 'gotcha' - they are extremely expressive. This means they easily overfit and can be sensitive to noise in inputs.  The strategy of simplifying a model to make it more robust to noise is called regularization.  There are many types:


use smaller hidden layers (i.e. 20 instead of 200 nodes)
use dropout, where during training inputs are randomly set to 0 during training (makes the network more robust to noise overall)
stop training earlier - 'early stopping'
use L1 or L2 regularization, which imposes a cost on the weights


All of these can be done from with Keras.  You want to make your network more robust to noise without decreasing your validation quality.  To do this, I would try the above suggestions in order.  You can measure overfitting by looking at the difference between predictive accuracy on your training data and validation data.  If they are very different - your model has learned structure in your training data that is not in your validation that is NOT what you want.  Try to fiddle regularization nobs until (1) your train-validation AUC or accuracy is very similar, (2) your validation AUC/accuracy is still sufficiently high.