Time series data combined with multiple feature. Predicting difference from the mean
With random forest there is no need to modify your output variable (i.e difference from the mean) when performing a regression.

It may be that the features you have are not relevant, or that you don't have enough data. Also try different model parameterisations (num trees, num layers etc).

Try to do some feature engineering to create intelligent features that might help the classifier.