Tensorflow - simple multi-layer perceptron not stabilizing around mean of normally distributed y-values I'm building an FX trading model where I'm trying to predict the +/- movement of a currency pair 5 minutes into the future.  I've had some promising results adapting the model as a classifier (i.e., buy if the currency pair is expected to increase by more than some threshold, sell if the pair is expected to decrease by some threshold, or do nothing otherwise), but when trying to create even the simplest multi-layer perceptron regression model, the mean of predicted values bounces around the mean of actual y values (as expected, this is very close to zero).

The two images below depict the train predictions (green) versus the actual y_train values (blue) after two different epochs after a number of epochs had already been completed.  Rather than move towards the true y mean in some stable way, the prediction mean flies right past it and will seemingly do this over and over with more epochs.





So far, I've tried:


Using a smaller learning rate
Adding more epochs.  The training set has almost 300k items in it, so the classification model is able to learn in only about 5 epochs, but increasing this to 50 or 100 for regression still doesn't seem to help.
Changing the loss function from mean squared error to mean absolute error
Changing the optimizer from Adam to other options like RMSProp (eventually will predict the same value for all items), Adagrad, Adadelta, Nadam, etc.
Using elu or leaky relu instead of relu.  Also, if I don't add 1 to the y values before training, the model ends up predicting the same value for every item after only a few epochs, which might be an extension of the same problem.
A bias constraint on the output node to keep it close to zero (the true y mean).  Ultimately, it doesn't matter and the model still pushes away from zero.
Batch normalization


This is happening even with a very simple model such as this:

self.model = models.Sequential()
self.model.add(layers.Dense(self.num_layers, activation="relu" input_shape=(self.x_train.shape[1],)))
self.model.add(layers.Dense(1))
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])


Any suggestions as to how I can stabilize the model?
Try deeper network. Something like this.

self.model = models.Sequential()
self.model.add(layers.Dense(self.num_layers, activation="relu", input_shape=(self.x_train.shape[1],)))
self.model.add(layers.Dense(self.num_layers//2, activation="relu"))
self.model.add(layers.Dense(self.num_layers//4, activation="linear"))
self.model.add(layers.Dense(1))
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])