neural network to find a very simple linear model (scikit-learn) I'm trying to test different machine learning algorithm to try to find correlation between various data on MRI scans.
Since I'm dealing with medical data, I don't have access to many events, but still I'm trying to see what a simple fully connected NN while provides me.
By debugging, it looks like even if I enter the output almost as is in the input, a simple NN using the default scikit-learn NN regressor is unable to find a satisfactory model.
I tried to reproduce the effect and I'd like to share with you the following code. I guess I'm just doing something wrong, because, basically, I'd like the NN to produce a simple model where label=X2-X1 and it doesn't work to me which is really surprising me.

here is the code:


import sklearn.neural_network as NN
import matplotlib.pyplot as plt
import numpy as np
N_events=100000 # number of rows of dataset
N_features = 2 # number of features
X = np.random.rand(N_events,N_features) # chose the data randomly
labels = X[:,1]-X[:,0] # label = X2-X1

C = NN.MLPRegressor(hidden_layer_sizes=(2,2),max_iter=500000,random_state=1) # very simple scikit-learn NN regressor
n = N_events // 2 # half data for training, half for test

print(X[0:10,1]-X[0:10,0]) # check that indeed label=X2-X1
print(labels[0:10])
C.fit( X[0:n,:] , labels[0:n]) # train the model

# plot y_predicted vs y_true for the test set... 
plt.figure()
y_real = labels[n:] # 
y_pred = C.predict(X[n:,:])
plt.plot(y_real,y_pred,'.')

# plot y_predicted vs y_true for the training set... even this doesn't work
plt.figure()
y_real = labels[0:n] # 
y_pred = C.predict(X[0:n,:])
plt.plot(y_real,y_pred,'.')



I was expecting the NN with 2x2 = 4 degrees of freedom to easily find such a simple model (note that I even don't add unused features which could disturb the fit, since N_features is 2 here), so I guess I'm just not using the code correctly.
Can anyone helps me ?

Many thanks
The problem is relu which is the default activation function. It zeros the inputs smaller than 0. Problem will be solved by using other functions. For example:

C = NN.MLPRegressor(activation="identity", hidden_layer_sizes=(1,1),max_iter=25,random_state=1)


Note that even 1 hidden layer with 1 neuron is enough to find the $X_2 = X_1$ decision boundary.