Should I relabel this data or remove the potentially leaky feature?
Consider your problem as a binary classification. We have two kinds of prediction: raise an alarm (state is going to change), do not raise an alarm (state is going to stay the same).

Actual / Predicted   Alarm goes off      Alarm did not go off
State changed        True positive       False positive
State stayed         False negative      True negative


Note the true negatives are useless - we don't care if the state stays in the same spot (below/above threshold). Therefore, the accuracy is NOT important for this system. Do not try to minimize the wrong metric. This extends to "do not use ROC/AUC" (in your problem). You can use a PR curve, but be careful with those (no interpolation, no AUC PR, these are wrong/useless). Basic metrics that you could use are F-score, precision and recall, or a re-weighting of those (if the false alarm is less important than a missed alarm, for example).

We also would like to deal with a huge "lag", trying to predict many time periods before (predicting just before the change is less useful). For this reason, I'd suggest investigating models other than MLP, notably those based on recurrent neural networks, such as LSTM. Also consider using time series prediction instead of classification - the literature on the subject is extensive and matches your problem really well.