Should I relabel this data or remove the potentially leaky feature? Putting together a Keras MLP to predict whether a value will exceed a static percent threshold in the next 15 minutes. The incoming data is a rolling percentage which moves smoothly for the most part because the data comes in every few milliseconds and is windowed over some fixed number of minutes. So, when the data is already above the threshold, it tends to stay there for a while.

When putting the data through the NN, it gets high accuracy, but this seems to be due to it correctly predicting that when it is currently over the threshold, it will also be over the threshold at some point (the next point) in the next x timesteps. The usefulness in the model would be if it could accurately predict before it crosses the threshold.

Features:

1) current point is over threshold - 1/0
2) current point is AM - 1/0
3) current day is weekday - 1/0
4) current percentage - 0.0-1.0
5-9) average of percentages in past 1/5/10/20/30 minute - 0.0-1.0


Label:

1 if a point is over threshold at any point in time after now and before now+15 minutes


Features 5-9 are intended to capture the inertia of the current percentage.

From feature importance, it looks like the current value is heavily used, followed by whether or not its over threshold, followed by the rolling means in order of time. I am currently changing the NN architecture and number of epochs in order to increase f1 score. Should I remove features 1 and 4, or rework the label in order to increase the accuracy of the predictive ability before its actually over the threshold?
Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).

Actual / Predicted   Alarm goes off      Alarm did not go off
State changed        True positive       False positive
State stayed         False negative      True negative


Note the true negatives are useless - we don't care if the state stays in the same spot (below/above threshold). Therefore, the accuracy is NOT important for this system. Do not try to minimize the wrong metric. This extends to "do not use ROC/AUC" (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).

We also would like to deal with a huge "lag", trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.