title: Python: How can we smooth a noisy signal using moving average?
body: <p>For an evaluation of a random forest regression, I am trying to improve a result using a <code>moving average filter</code> after fitting a model using a <code>RandomForestRegressor</code> for a dataset found in <a href="https://drive.google.com/open?id=0B2Iv8dfU4fTUdlE0VUJKUnctSkE" rel="nofollow noreferrer">this link</a></p>
  
  <pre><code>import pandas as pd
  import math
  import matplotlib
  import matplotlib.pyplot as plt
  import numpy as np
  from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
  from sklearn.model_selection import GridSearchCV
  from sklearn.metrics import r2_score, mean_squared_error, make_scorer
  from sklearn.model_selection import train_test_split
  from math import sqrt
  from sklearn.cross_validation import train_test_split
  
  
  n_features=3000
  
  df = pd.read_csv('cubic32.csv')
  
  for i in range(1,n_features):
      df['X_t'+str(i)] = df['X'].shift(i)
  
  print(df)
  
  df.dropna(inplace=True)
  
  
  X = df.drop('Y', axis=1)
  y = df['Y']
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.40)
  
  X_train = X_train.drop('time', axis=1)
  X_test = X_test.drop('time', axis=1)
  
  
  parameters = {'n_estimators': [10]}
  clf_rf = RandomForestRegressor(random_state=1)
  clf = GridSearchCV(clf_rf, parameters, cv=5, scoring='neg_mean_squared_error', n_jobs=-1)
  model = clf.fit(X_train, y_train)
  model.cv_results_['params'][model.best_index_]
  math.sqrt(model.best_score_*-1)
  model.grid_scores_
  
  #####
  print()
  print(model.grid_scores_)
  print("The best score: ",model.best_score_)
  
  print("RMSE:",math.sqrt(model.best_score_*-1))
  
  clf_rf.fit(X_train,y_train)
  modelPrediction = clf_rf.predict(X_test)
  print(modelPrediction)
  
  print("Number of predictions:",len(modelPrediction))
  
  meanSquaredError=mean_squared_error(y_test, modelPrediction)
  print("Mean Square Error (MSE):", meanSquaredError)
  rootMeanSquaredError = sqrt(meanSquaredError)
  print("Root-Mean-Square Error (RMSE):", rootMeanSquaredError)
  
  fig, ax = plt.subplots()
  index_values=range(0,len(y_test))
  
  y_test.sort_index(inplace=True)
  X_test.sort_index(inplace=True)
  
  modelPred_test = clf_rf.predict(X_test)
  ax.plot(pd.Series(index_values), y_test.values)
  
  smoothed = np.convolve(modelPred_test, np.ones(10)/10)
  PlotInOne=pd.DataFrame(pd.concat([pd.Series(smoothed), pd.Series(y_test.values)], axis=1))
  plt.figure(); PlotInOne.plot(); plt.legend(loc='best')
  </code></pre>
  
  <p>However, the plot of the predicted values seems (as shown below) to be very coarse (the blue line) even if i am smoothing the prediction values like the following</p>
  
  <pre><code>smoothed = np.convolve(modelPred_test, np.ones(10)/10)
  </code></pre>
  
  <p>The orange line is a plot of the actual value.</p>
  
  <p><a href="https://i.stack.imgur.com/oXDN0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oXDN0.png" alt="enter image description here"></a></p>
  
  <p>Is there any way that we can penalize the prediction error (or smooth the noise of the signal using a moving average or other smoothing techniques) so that we get a plot closer to the actual value? I am also trying to increase the number of estimators for the random forest tree (<code>n_estimators</code>), but it doesn't seem to improve much. If we plot the actual value alone, it looks like the following.</p>
  
  <p><a href="https://i.stack.imgur.com/zIR9o.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zIR9o.png" alt="enter image description here"></a></p>