"""
# Electric Vehicle (EV) Analysis and Temperature Prediction

Being able to predict the operating parameters of an electric vehicle motor is an important emerging task due to the huge demand of electric vehicles in todays society. 

Throughout this notebook, we'll take the measured parameters of a Permanent-Magnet Synchronous Motor (PMSM) and attempt to predict all four output temperatures (permanent magnetic, stator yoke, stator winding and stator tooth) as accurately as possible. This forms an interesting multi-output regression task, and we'll explore various feature engineering, models and hyper-parameter tuning accordingly. 

**Table of Contents:**

1. [Imports](#imports)
2. [Creation of Training / Test Splits](#training-splits)
3. [EDA](#EDA)
4. [Baseline Predictions](#baseline-predictions)
5. [Feature Engineering](#feature-engineering)
6. [Improved Splitting of Data](#data-splitting)
7. [Updated Baseline after Feature Engineering](#updated-baseline)
8. [Model Cross-Validation](#cross-validation)
9. [Model Refinements and Evaluation](#model-refinements)
10. [Test Set Predictions](#final-test-set-predictions)
"""
"""
<a id="imports"></a>
## 1. Import dependencies and dataset
"""
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_predict, cross_validate

from collections import Counter
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.cluster import KMeans
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor, GradientBoostingRegressor, BaggingRegressor, VotingRegressor
from sklearn.linear_model import ElasticNet, LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split, cross_val_score, KFold, GroupKFold, GroupShuffleSplit
from sklearn.multioutput import MultiOutputRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.svm import SVR, LinearSVR

from xgboost import XGBRegressor
from tqdm import tqdm
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
data_df = pd.read_csv("/kaggle/input/electric-motor-temperature/pmsm_temperature_data.csv")
"""
---
"""
"""
<a id="training-splits"></a>
## 2. Creation of our main training and test splits 
"""
"""
We want to allocate a proportion of the total data as our test set. We'll do this using the profile IDs, since we want to fairly test the generalisation performance of our models. If we leak information from various profile ids in the test set into the training set, our models will be assessed as over-optimistically good in terms of performance.
"""
full_profiles = data_df['profile_id'].value_counts().index.values
test_set_profiles = [31, 32, 41, 42, 51, 52, 61, 62, 71, 73]
train_set_profiles = [id for id in full_profiles if id not in test_set_profiles]
train_df = data_df.loc[data_df['profile_id'].isin(train_set_profiles)].copy()
test_df = data_df.loc[data_df['profile_id'].isin(test_set_profiles)].copy()

len(train_df['profile_id'].value_counts()), len(test_df['profile_id'].value_counts())
train_df.shape, test_df.shape
train_df.head(3)
test_df.head(3)
test_profiles = test_df.loc[:, 'profile_id'].copy()
test_labels = test_df.loc[:, ['pm', 'stator_yoke', 'stator_tooth', 'stator_winding']].copy()
test_df.drop(['profile_id', 'pm', 'stator_yoke', 'stator_tooth', 'stator_winding'], axis=1, inplace=True)
test_df.head(3)
# add names to the index of each df
train_df.index.name = 'timestep'
test_df.index.name = 'timestep'
"""
---
"""
"""
<a id="EDA"></a>
## 3. Initial Exploratory Data Analysis
"""
"""
### 3.1 General trends of our data features
"""
train_df.loc[:, 'ambient'].plot(figsize=(16,4))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
train_df.loc[:, 'coolant'].plot(figsize=(16,4))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
Our training data appears to be ordered and not shuffled, as can be seen from the relatively continuous series represented above for the two selected variables.
"""
train_df.describe()
"""
The features within the data appears to have already been standardised with a zero mean and standard deviation of 1. This is unfortunate, since it makes manipulation of the electrical parameters and measured values into more useful and insightful ones a more difficult task.
"""
train_df['profile_id'].value_counts()[:5]
"""
### 3.2 Exploring trends within an example Profile ID
"""
"""
**Lets briefly explore the data associated with an example profile ID**
"""
EXAMPLE_ID = 10

example_profile = train_df.loc[train_df['profile_id'] == EXAMPLE_ID].copy()
example_profile['ambient'].plot(figsize=(16,7))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
This data is extremely noisy in its current form, which could be detrimental to our results for making temperature predictions. To overcome this, we could consider pre-processing strategies such as applying filters over the data to smooth it. A good strategy for this could be to apply one or more low-pass filters to the data. A good means of achieving this could be to apply an exponentially weighted moving average and standard deviation to each of our features, especially those that are particularly noisy.
"""
# choose a range of spans to experiment with
spans = [600, 1200, 2400, 4800]

for span in spans:
    new_col = f"ambient_exp_{span}"
    example_exp_ma = example_profile.ewm(span=span, adjust=False).mean()
    example_profile[new_col] = example_exp_ma['ambient'].copy()
    
ambient_cols = [x for x in example_profile.columns.values if x.startswith('ambient')]
example_profile[ambient_cols].plot(figsize=(16,7))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
By applying exponential weighted moving average functions as shown above, we can effectively reduce the noise within our dataset, which will hopefully help improve the performance of our models.
"""
# choose a range of spans to experiment with
spans = [600, 1200, 2400]

chosen_col = 'coolant'

for span in spans:
    new_col = f"{chosen_col}_exp_{span}"
    example_exp_ma = example_profile.ewm(span=span, adjust=False).mean()
    example_profile[new_col] = example_exp_ma[chosen_col].copy()
    
chosen_cols = [x for x in example_profile.columns.values if x.startswith(chosen_col)]
example_profile[chosen_cols].plot(figsize=(16,8))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Outflow Coolant Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
From the basic measured parameters, we want to be able to predict the temperatures at the rotor, stator yoke, stator tooth, and stator winding as part of the efforts in increasing Electric Vehicle (EV) efficiencies. It's not practical to be able to measure these in real time for commercialised vehicles, so being able to accurately predict these temperatures would be extremely valueble.
"""
output_vars = ['pm', 'stator_yoke', 'stator_tooth', 'stator_winding']

example_profile[output_vars].plot(figsize=(16,6))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
A good thing is that for the example we've looked at, each of the four output dependent variables appear to be highly correlated. This should simply the process of making predictions for each of the variables given the operational motor data. When results are more correlated like this, it makes it easier for our multiple-output regression models to produce a good set of results from a unified model, rather than having to form individual models for each predicted output.

Let's see how these output variables look after applying applying an exponentially weighted average to each variable:
"""
exp_ma = example_profile[output_vars].ewm(span=500).mean()
exp_ma.plot(figsize=(16,6))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Temperature (Exp Weighted Average)", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
Now let's assess the input features over time:
"""
input_cols = ['ambient', 'coolant', 'u_d', 'u_q', 'motor_speed', 'torque', 'i_d', 'i_q']

example_profile[input_cols].plot(figsize=(18,8))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
example_profile[output_vars].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()

example_profile[input_cols].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
### 3.3 Correlation of our features
"""
train_df_correlations = train_df.corr()
plt.figure(figsize=(12,10))
sns.heatmap(train_df_correlations, annot=True)
plt.show()
"""
This is useful to understand the correlation some variables have relative to one-another. For instance, coolant looks like it is highly correlated to the stator yoke temperature. Since stator yoke temperature is also highly correlated to the other output variables, this could come in useful for making inferences of temperature given a set of input features.

A key thing to keep in mind is that a correlation matrix such as that above will only capture linear correlations. When analysing the previous plots, its clear that many of our variables are correlated, but these are more complex relationships that may not be captured well be the pearson correlation coefficient, which the matrix above is based on.
"""
training_sample = train_df.sample(100000)
plt.figure(figsize=(12,7))
sns.scatterplot(x='coolant', y='stator_yoke', data=training_sample, alpha=0.05)
plt.grid()
plt.show()
"""
We have a nice correlation between coolant and one of the dependent outputs - stator yoke.
"""
plt.figure(figsize=(12,7))
ax = sns.regplot(x="i_d", y="pm", data=training_sample, scatter_kws={'alpha':0.01})
plt.grid()
plt.show()
plt.figure(figsize=(12,7))
ax = sns.regplot(x="i_q", y="pm", data=training_sample, scatter_kws={'alpha':0.02})
plt.grid()
plt.show()
"""
**Lets look at how the output variables are correlated to one-another:**
"""
pd.plotting.scatter_matrix(training_sample[output_vars], figsize=(16, 10), alpha=0.02)
plt.show()
"""
All of our output variables are highly correlated to one another, which makes sense; as the temperature of a given component tends to increase in the motor, so do all of the others on average. Again, as stated previously, this makes the multi-output regression problem slightly easier, since the different outputs we are trying to predict are all similar.
"""
"""
---
"""
"""
<a id="baseline-predictions"></a>
## 4. Baseline predictions on the test set using default off-the-shelf models
"""
"""
### 4.1 Basic Multiple-Output Linear Regression OLS 
"""
"""
Just so that the process of making a model and predicting on the dataset is familier, lets make some predictions on the test set using simple baseline models. We'll evaluate these models on the test set just to have an idea of what we should be aiming to beat.
"""
y = train_df[output_vars]
X = train_df.drop(columns=output_vars + ['profile_id'])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=12)
"""
Lets train a normal multiple-output linear regression model on the data and see what results we obtain:
"""
lin_reg = LinearRegression()

lr_scores = cross_val_score(lin_reg, X_train, y_train, 
                         scoring='neg_mean_squared_error', cv=10)

lr_rmse_scores = np.sqrt(-lr_scores)
print(f"Linear Regression RMSE: {lr_rmse_scores.mean():.5f} +/- {lr_rmse_scores.std():.5f}")
lin_reg.fit(X_train, y_train)
val_preds = lin_reg.predict(X_test)
print(f"Linear Regression RMSE on Validation Split: {np.sqrt(mean_squared_error(val_preds, y_test)):.4f}")
"""
### 4.2 Examining the predictions made by our baseline for an example Profile ID
"""
"""
Let's see what this model is like for reproducing a random profile from the dataset:
"""
profile_4_idx = train_df[train_df['profile_id'] == 4].copy().index
profile_4_preds = lin_reg.predict(X.iloc[profile_4_idx])
profile_4_original = train_df.loc[profile_4_idx, output_vars]
profile_4_preds.shape, profile_4_original.shape

new_cols = [f"{x}_pred" for x in profile_4_original.columns]
profile_4_preds = pd.DataFrame(profile_4_preds)
profile_4_preds.columns = new_cols
profile_4_original[output_vars].plot(figsize=(16,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Output Temperature", weight="bold")
plt.title("Original Output Temperatures - Profile 4", weight='bold')
plt.tight_layout()
plt.grid()
plt.show()
profile_4_preds.plot(figsize=(16,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Output Temperature", weight="bold")
plt.title("Predicted Output Temperatures - Profile 4", weight='bold')
plt.tight_layout()
plt.grid()
plt.show()
for column in profile_4_original.columns:
    mse = mean_squared_error(profile_4_original[column], profile_4_preds[column + '_pred'])
    rmse = np.sqrt(mse)
    print(f"RMSE for {column}: {rmse:.4f}")
"""
Not too bad straight off the bat, however this should not be relied on heavily since many of the data instances above could have been within the chosen training set, and therefore our linear regression model may have simply memorised many of those points above. For true generalisation evaluation we would need to test this model on a profile it has never seen before.

A good option for this would be to ensure our training and validation splits used to evaluate models contain unique profile IDs to one-another, so that the data we validate against is always new and unseen.
"""
"""
---
"""
"""
<a id="feature-engineering"></a>
## 5. Feature Engineering and Improvements for better performance
"""
"""
### 5.1 Addition of new features:

From the provided electrical terms and measured data we have, we can combine these using various formulae to form new (and potentially insightful) features for our dataset. A key problem is that the data for this problem is already standardised, and unfortunately we have no access to the original dimensions or standardising parameters to reverse this process. Despite this, using the relationships defined below we can may still get some meaningful features from the standardised data, even if they are no longer mathematically sound products due to the standardised form its currently in.

**Voltage and current magnitudes**

We can work out the magnitudes of the voltage and current vectors by calculating the norm, as shown below:

$ v_{s} = \displaystyle\sqrt{v_{d}^{2} + v_{q}^{2}} $

$ i_{s} = \displaystyle\sqrt{i_{d}^{2} + i_{q}^{2}} $

Where $ v_{s} $ is the voltage magnitude, $ v_{d} $ is the d-component of the voltage, and $ v_{q} $ is the q-component of the voltage.

**Apparent Power**

Using these features, we can then determine electrical apparent power (magnitude of complex power), which is given by:

$ \vert S \vert = 1.5 \times v_{s} \times i_{s} $

**Effective Power**

Using the components of voltage and current we can also calculate effective power like so:

$ P = \big( v_{d} \times v_{q} \big) + \big( i_{d} \times i_{q} \big) $

**Additional Feature Combinations**

We'll also add some additional features, which represent the interaction between parameters of interest. This will include:

- Motor speed and current magnitude: $ \omega \times i_{s} $


- Motor speed and apparent power: $ \omega \times \vert S \vert $
"""
def norm(row, col1, col2):
    return np.linalg.norm([row[col1], row[col2]])
def find_new_features(dataframe):
    """ Calculate new electrical features for our dataset """
    extra_df = dataframe.copy()
    
    extra_df['voltage_s'] = extra_df.apply(norm, args=('u_d', 'u_q'), axis=1)
    extra_df['current_s'] = extra_df.apply(norm, args=('i_d', 'i_q'), axis=1)
    extra_df['apparent_power'] = 1.5 * extra_df['voltage_s'] * extra_df['current_s']
    extra_df['effective_power'] = ((extra_df['u_d'] * extra_df['u_q']) +
                                (extra_df['i_d'] * extra_df['i_q']))
    extra_df['speed_current'] = extra_df['current_s'] * extra_df['motor_speed']
    extra_df['speed_power'] = extra_df['current_s'] * extra_df['apparent_power']
    
    return extra_df
train_df = find_new_features(train_df)
train_df.head()
example_profile = train_df.loc[train_df['profile_id'] == EXAMPLE_ID].copy()
example_profile[output_vars].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()

example_profile[['voltage_s', 'current_s', 'apparent_power']].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
We can see that apparent power actually correlates well to the output variables - at least for this example profile ID.
"""
example_profile[output_vars + ['apparent_power']].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
Lets now standardise our apparent power variable with zero mean and a unit standard deviation, which should fit it to the output variable data better:
"""
example_profile['apparent_power'] = ((example_profile['apparent_power'] - 
                                     example_profile['apparent_power'].mean()) /
                                     example_profile['apparent_power'].std())

example_profile[output_vars + ['apparent_power']].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
example_profile[output_vars + ['effective_power']].plot(figsize=(18,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Ambient Temperature", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
plot_var = 'apparent_power'
alpha = 0.05

plt.figure(figsize=(14,9))

plt.subplot(221)
ax = sns.regplot(x=plot_var, y="pm", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(222)
ax = sns.regplot(x=plot_var, y="stator_yoke", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(223)
ax = sns.regplot(x=plot_var, y="stator_tooth", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(224)
ax = sns.regplot(x=plot_var, y="stator_winding", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()
plt.tight_layout()
plt.show()
"""
### 5.2 Introduction of exponentially weighted averages
"""
"""
Let's try applying a low pass filter (exponentially weighted averages are the equivalent of doing this) to our data and seeing how this impacts the relationship our input features have with the output features.

Let's try with the apparent power variable and see how different variations of it correlate to the output variables when we apply averaging functions with different spans:
"""
# choose a range of spans to experiment with
spans = [600, 1200, 2400, 4800]

chosen_col = 'apparent_power'

for span in spans:
    new_col = f"{chosen_col}_exp_{span}"
    example_exp_ma = example_profile.ewm(span=span, adjust=False).mean()
    example_profile[new_col] = example_exp_ma[chosen_col].copy()
    
chosen_cols = [x for x in example_profile.columns.values if x.startswith(chosen_col)]
example_profile[chosen_cols].plot(figsize=(16,5))
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Apparent Power (Standardised)", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
cols = output_vars + chosen_cols
color_dict = {}

for column in cols:
    if column == 'apparent_power':
        color_dict[column] = 'black'
    elif column.startswith('apparent_power_exp'):
        color_dict[column] = 'tab:blue'
    else:
        color_dict[column] = 'tab:red'
example_profile[output_vars + chosen_cols].plot(figsize=(16,7), 
                                                color=[color_dict.get(x, 'black') for x in cols])
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Standardised Feature Values", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
Its quite interesting how well the apparent power feature correlates to our output temperatures when we apply a low-pass filter (exponentially weighted average) to it. This certainly works well for this specific profile ID. It could be worth exploring others to make sure it also correlates well to those.

In all cases, it seems like the apparent power feature was well worth introducing into our dataset.
"""
# translate apparent power variables by 1.5
example_profile[chosen_cols[1:]] = example_profile[chosen_cols[1:]] - 1.5
example_profile[output_vars + chosen_cols[1:]].plot(figsize=(16,7), 
                                                color=[color_dict.get(x, '#333333') for x in cols])
plt.xlabel("Timestep (0.5s per step)", weight="bold")
plt.ylabel("Standardised Feature Values", weight="bold")
plt.tight_layout()
plt.grid()
plt.show()
"""
Through a simple translation downwards, we can almost fit the apparent power plots onto the various temperature plots!

It will be worth fitting a new model with these variables and seeing how this improves performance relative to the baseline models!
"""
plot_var = 'apparent_power_exp_2400'
alpha = 0.05
plt.figure(figsize=(14,9))

plt.subplot(221)
ax = sns.regplot(x=plot_var, y="pm", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(222)
ax = sns.regplot(x=plot_var, y="stator_yoke", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(223)
ax = sns.regplot(x=plot_var, y="stator_tooth", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()

plt.subplot(224)
ax = sns.regplot(x=plot_var, y="stator_winding", data=example_profile, scatter_kws={'alpha':alpha})
plt.grid()
plt.tight_layout()
plt.show()
corr_matrix = example_profile.corr()
corr_matrix['stator_yoke'].sort_values(ascending=False)
"""
The relationships are not quite linear and are more complex, but they're definitely much better than before we added any features.
"""
"""
### 5.3 Extending our data with the exponentially weighted average features 

Let's add these exponentially weighted averages to our training data and see how they perform when used to train a model. We'll automate this process through the definition of a class to do our pre-processing.
"""
class PMSMDataProcessor(BaseEstimator, TransformerMixin):
    """ PMSM data processor and loader """
    
    def __init__(self, add_exp_terms=True, spans=[600, 1200, 2400, 4800],
                 output_vars=['pm', 'stator_tooth', 'stator_yoke', 'stator_winding']):
        self.add_exp_terms = add_exp_terms
        self.spans = [600, 1200, 2400, 4800]
        self.output_vars = output_vars
        self.training_cols = []
        
        
    def fit(self, X, y=None):      
        return self
                
    
    def transform(self, X):
        """ Create new features from our given data """
        
        extra_df = self._find_new_features(X)
        
        # update training cols
        self._update_train_cols(extra_df)
        
        # if selected add additional exponential terms
        if self.add_exp_terms:
            extra_df = self._find_exp_terms(extra_df).copy()
            
            # update training cols again to reflect new additions
            self._update_train_cols(extra_df)
            
        return extra_df
        
        
    def _find_norm(self, row, col1, col2):
        return np.linalg.norm([row[col1], row[col2]])
    
    
    def _update_train_cols(self, X):
        """ Compile list of base columns used for training """
        self.training_cols = [x for x in X if x != 'profile_id' 
                              and x not in self.output_vars]
    
    
    def _find_new_features(self, dataframe):
        """ Calculate new electrical features for our dataset """
        
        extra_cols = dataframe.copy()
        
        extra_cols['voltage_s'] = extra_cols.apply(self._find_norm, 
                                                   args=('u_d', 'u_q'), axis=1)
        extra_cols['current_s'] = extra_cols.apply(self._find_norm, 
                                                   args=('i_d', 'i_q'), axis=1)
        
        extra_cols['apparent_power'] = 1.5 * extra_cols['voltage_s'] * extra_cols['current_s']
        extra_cols['effective_power'] = ((extra_cols['u_d'] * extra_cols['u_q']) +
                                         (extra_cols['i_d'] * extra_cols['i_q']))
        
        extra_cols['speed_current'] = extra_cols['current_s'] * extra_cols['motor_speed']
        extra_cols['speed_power'] = extra_cols['current_s'] * extra_cols['apparent_power']
    
        return extra_cols
    
    
    def _find_exp_terms(self, dataframe):
        """ Add exponential terms to all features, using the class spans """
        
        extended_df = dataframe.copy()
        
        # add new columns to each feature for each span
        for span in self.spans:
            exp_ma_df = dataframe.ewm(span=span, adjust=False).mean()
            for column in self.training_cols:
                new_col = f"{column}_exp_{span}"
                extended_df[new_col] = exp_ma_df[column].copy()
                
        return extended_df
"""
**Process data from scratch again:**
"""
data_preprocessor = PMSMDataProcessor()

train_df_extra = data_preprocessor.fit_transform(train_df)
test_df_extra = data_preprocessor.transform(test_df)

y = train_df_extra[data_preprocessor.output_vars].copy()
X = train_df_extra.drop(columns=data_preprocessor.output_vars)

X.shape, y.shape, test_df_extra.shape
"""
Although we dont want to use it as training data, we need to keep track of our profile IDs within the training set so that we can ensure appropriate splits of data between training and validation splits.
"""
# select profile id's as groups to split our folds on
groups = X.loc[:, 'profile_id'].copy()
X.drop('profile_id', axis=1, inplace=True)
"""
### 5.4 Standardise all features of our processed data

Although our data was already standardised originally, since we've introduced new features we should standardise again to ensure the new columns are consistent relative to the original features.

We'll need to remove the profile IDs, standardise our features, and then add it in as a column again:
"""
std_scaler = StandardScaler()
X_std = pd.DataFrame()
X_std[X.columns] = pd.DataFrame(std_scaler.fit_transform(X))
X_std['profile_id'] = groups.values
test_std = std_scaler.transform(test_df_extra)

X_std.head(3)
"""
<a id="data-splitting"></a>
## 6. Improved splitting of data - ensuring unique profile IDs between splits
"""
"""
For a better and true appreciation of the generalisation performance of our model we need to ensure that the training, validation and test splits all contain different profile IDs. This ensures that the data is unseen from what we've trained our models on.

Due to the requirement to know the profile_id of the training data, we need to keep track of the profile IDs corresponding to the training instances, which is stored in the profiles variable.

We have two options for grouped kfolds. We can either split without shuffling, using GroupKFold, or Shuffle our groups to get different subsets of groups each time (but still no like-groups in both splits, just as we need).

We'll use the shuffling option, since we want to be able to mix up the splits of profile_id we get between different folds, in order to help assess the true generalisation of our models.
"""
grouped_kfold = GroupShuffleSplit(n_splits=6)
grouped_kfold.get_n_splits(X_std, y, groups)
"""
To illustrate how this works, this kfold iterator returns the following splits:
"""
for train_index, test_index in grouped_kfold.split(X_std, y, groups):
    print("TRAIN:", train_index.shape, "TEST:", test_index.shape)
    train_inputs, val_inputs = X_std.iloc[train_index], X_std.iloc[test_index]
    train_outputs, val_outputs = y.iloc[train_index], y.iloc[test_index]
    
    profiles_train = train_inputs['profile_id'].value_counts().index.values
    profiles_test = val_inputs['profile_id'].value_counts().index.values
    
    print(f"Profile IDs in Training set: \n{profiles_train}\n")
    print(f"Profile IDs in Test set: \n{profiles_test}\n")
"""
This is good, we have a nice random subset of profile IDs for each fold, and in each case, the profile IDs within the training split are not present within the validation split. This ensures a much better evaluation of performance when testing our models.
"""
"""
<a id="updated-baseline"></a>
## 7. Testing the baseline performance with improved splits and additional features
"""
"""
Lets test this using our OLS linear regression model, and repeat it 10 times so that we can find the average performance across multiple attempts of cross-validation:
"""
lr_rmse_scores = []
mean_rmse_scores = []

lr_reg = LinearRegression()

# times to repeat the cross validation
repeats = 10

for i in tqdm(range(repeats)):
    lr_scores = cross_val_score(lr_reg, X_std.drop('profile_id', axis=1), y, groups=groups,
                                scoring='neg_mean_squared_error', cv=grouped_kfold)
    rmse_scores = np.sqrt(-lr_scores)
    lr_rmse_scores.append(rmse_scores)
    mean_rmse_scores.append(rmse_scores.mean())

lr_rmse_scores = np.array(lr_rmse_scores)
mean_rmse_scores = np.array(mean_rmse_scores)

print(f"Linear Regression OLS RMSE: {mean_rmse_scores.mean():.5f} +/- {mean_rmse_scores.std():.5f}")
"""
These results are much better than the original baseline models, which highlights the improvement that adding our extra features has resulted in. We could likely improve by filtering the output of our linear regression model, in order to smooth the output noise.

What's interesting about this result is we might have expected performance to have dropped, due to the improvement in how we split our data. Clearly, the addition of more insightful features has helped overcome this and increase performance considerably. 
"""
"""
<a id="cross-validation"></a>
## 8. Exploring a range of default models using Cross-Validation
"""
"""
It will be heplful to explore a range of different models and test their performance against one-another accordingly, so that we can choose the best and then perform further fine-tuning.

Since the training set is very large computation time can be a problem, especially if we are using techniques like cross-validation and grid-search. To counteract this, we will initially make a subsample of training data to make our comparisons on, which should greatly speed up this process. As above, we'll ensure we split training and validation so that profile ids are unique across splits:
"""
def multi_model_grouped_cross_validation(clf_tuple_list, X, y, groups, K_folds=6, random_seed=0, subsample=False,
                                 score_type='neg_mean_squared_error', subsample_prop=0.1):
    """ Find grouped cross validation scores, and print and return results """
    
    model_names, model_scores = [], []
    
    # if selected, only use a small portion of data for speed
    if subsample:
        X, _, y, _ = train_test_split(X, y, training_size=subsample_prop, random_seed=random_seed)
    
    for name, model in clf_tuple_list:
        grouped_kfold = GroupShuffleSplit(n_splits=K_folds)
        cross_val_results = cross_val_score(model, X, y, groups=groups, 
                                            cv=grouped_kfold, scoring=score_type, n_jobs=-1)
        model_names.append(name)
        rmse_scores = np.sqrt(-cross_val_results)
        model_scores.append(rmse_scores)
        print("{0:<40} {1:.5f} +/- {2:.5f}".format(name, rmse_scores.mean(), rmse_scores.std()))
        
    return model_names, model_scores


def boxplot_comparison(model_names, model_scores, figsize=(14, 10), score_type="RMSE",
                       title="K-Folds Cross-Validation Comparisons"):
    """ Boxplot comparison of a range of models using Seaborn and matplotlib """
    
    fig = plt.figure(figsize=figsize)
    fig.suptitle(title, fontsize=18)
    ax = fig.add_subplot(111)
    sns.boxplot(x=model_names, y=model_scores)
    ax.set_xticklabels(model_names)
    ax.set_xlabel("Model", fontsize=16) 
    ax.set_ylabel("Model Score ({})".format(score_type), fontsize=16)
    ax.set_xticklabels(ax.get_xticklabels(), rotation=60)
    plt.show()
    return
"""
Due to significant computation time and our large dataset, we'll only conduct this on a small sub-sample of the total data:
"""
X_trg_sub, _, y_trg_sub, _ = train_test_split(X_std, y, stratify=groups, test_size=0.99, random_state=12)

# extract our training profiles from the training data prior to fitment
trg_profiles = X_trg_sub.loc[:, 'profile_id'].values
X_trg_sub = X_trg_sub.drop('profile_id', axis=1)

X_trg_sub.shape, y_trg_sub.shape
# list of classifiers to compare
clf_list = [("Linear Regressor", LinearRegression()),
            ("Ridge Regressor", Ridge(alpha=250)),
            ("LASSO Regressor", Lasso(alpha=0.1)),
            ("ElasticNet", ElasticNet(alpha=0.01, l1_ratio=0.1)),
            ("Support Vector Regressor", MultiOutputRegressor(LinearSVR())),
            ("Extra Trees Regressor", ExtraTreesRegressor(n_estimators=10, bootstrap=True, max_depth=60, 
                                                          n_jobs=-1, min_samples_leaf=3, min_samples_split=5)),
            ("Random Forest Regressor", RandomForestRegressor(n_estimators=10, bootstrap=True, max_depth=60, 
                                                              n_jobs=-1, min_samples_leaf=3, min_samples_split=5)),
            ("Multi-layer Perception", MLPRegressor())]

# obtain names and scores for each cross-validation, and print / plot results
print("K-Folds Cross-Validation Results on training set: \n{}".format('-'*60))
%time model_names, model_scores = multi_model_grouped_cross_validation(clf_list, X_trg_sub, y_trg_sub, trg_profiles)
boxplot_comparison(model_names, model_scores)
"""
Interestingly, our linear regression models appear to be the best in this case. This has also been the case in the previous results we obtained, which suggests a simple, yet very effective solution to the problem of predicting real-time temperatures in our motors is using a linear regression OLS model.

Our tree based methods took a long time to train above, so it could be worth reducing the size of our dataset further when exploring the optimisation of these methods.
"""
"""
<a id="model-refinements"></a>
## 9. Exploring and tuning our individual regression models:

Lets explore linear regression and regularisation a bit more, since it is the best performing model from all of those above.
"""
"""
### 9.1 Exploration of a Ridge regressor and the alpha hyper-parameter:

Since our ridge regressor was good, but not quite as good as standard unregularised linear regression, it could be worth exploring a range of hyper-parameters to see whether we can improve performance accordingly:
"""
alpha_values = [0, 0.01, 0.1, 0.5, 1.0, 1.5, 3, 10, 20, 50, 100, 
                200, 250, 500, 750, 1000, 2000, 3000, 4000, 5000]

ridge_rmse_scores = []
mean_ridge_rmse_scores = []

for alpha in tqdm(alpha_values):
    ridge_reg = Ridge(alpha=alpha, fit_intercept=True)
    ridge_scores = cross_val_score(ridge_reg, X, y, groups=groups,
                                   scoring='neg_mean_squared_error', cv=grouped_kfold)
    rmse_scores = np.sqrt(-ridge_scores)
    ridge_rmse_scores.append(rmse_scores)
    mean_ridge_rmse_scores.append(rmse_scores.mean())

ridge_rmse_scores = np.array(ridge_rmse_scores)
mean_ridge_rmse_scores = np.array(mean_ridge_rmse_scores)

print(f"Ridge RMSE: {mean_ridge_rmse_scores.mean():.5f} +/- {mean_ridge_rmse_scores.std():.5f}")

# calculate standard deviation to annotate on our plot
ridge_rmse_scores_std = ridge_rmse_scores.std(axis=1)
# plot mean and standard deviation of cross-val scores
plt.figure(figsize=(18,6))
sns.lineplot(x=alpha_values, y=mean_ridge_rmse_scores)
plt.fill_between(alpha_values, mean_ridge_rmse_scores - ridge_rmse_scores_std, 
                 mean_ridge_rmse_scores + ridge_rmse_scores_std, 
                 color='tab:blue', alpha=0.2)
plt.semilogx(alpha_values, mean_ridge_rmse_scores + ridge_rmse_scores_std, 'b--')
plt.semilogx(alpha_values, mean_ridge_rmse_scores - ridge_rmse_scores_std, 'b--')
plt.ylabel("Cross-Validation RMSE (Average)", weight='bold', size=14)
plt.xlabel("Ridge Alpha Hyper-Parameter", weight='bold', size=14)
plt.show()
"""
### 9.2 Exploring LASSO regression in a similar way:

LASSO tends to perform automatic feature reduction by setting the coefficient of low-importance features to zero during training. The extend to which this occurs is related to the hyper-parameter values we select at model creation. We'll explore a range of these similarly to how we did with Ridge regression above:
"""
# range of alpha values between 0 and 1
alpha_values = np.logspace(-4, 1, 20)

lasso_rmse_scores = []
mean_lasso_rmse_scores = []

for alpha in tqdm(alpha_values):
    lasso_reg = Lasso(alpha=alpha, fit_intercept=True, max_iter=1000, tol=0.5)
    lasso_scores = cross_val_score(lasso_reg, X, y, groups=groups,
                                   scoring='neg_mean_squared_error', cv=grouped_kfold)
    rmse_scores = np.sqrt(-lasso_scores)
    lasso_rmse_scores.append(rmse_scores)
    mean_lasso_rmse_scores.append(rmse_scores.mean())

lasso_rmse_scores = np.array(lasso_rmse_scores)
mean_lasso_rmse_scores = np.array(mean_lasso_rmse_scores)

print(f"LASSO RMSE: {mean_lasso_rmse_scores.mean():.5f} +/- {mean_lasso_rmse_scores.std():.5f}")
"""
It seems our models struggle to converge if not given enough iterations. If our performance is good enough, it may be worth increasing this accordingly in order to maximise the potential performance of our LASSO models.
"""
# calculate standard deviation to annotate on our plot
lasso_rmse_scores_std = lasso_rmse_scores.std(axis=1)

# plot mean and standard deviation of cross-val scores
plt.figure(figsize=(18,6))
sns.lineplot(x=alpha_values, y=mean_lasso_rmse_scores)
plt.fill_between(alpha_values, mean_lasso_rmse_scores - lasso_rmse_scores_std, 
                 mean_lasso_rmse_scores + lasso_rmse_scores_std, 
                 color='tab:blue', alpha=0.2)
plt.semilogx(alpha_values, mean_lasso_rmse_scores + lasso_rmse_scores_std, 'b--')
plt.semilogx(alpha_values, mean_lasso_rmse_scores - lasso_rmse_scores_std, 'b--')
plt.ylabel("Cross-Validation RMSE (Average)", weight='bold', size=14)
plt.xlabel("LASSO Alpha Hyper-Parameter", weight='bold', size=14)
plt.show()
"""
### 9.3 Combining the effects of L1 and L2 regularisation together using Elastic Net:

For this we'll explore a range of alpha and l1_ratio parameters to see whether elastic net has potential or not:
"""
# range of alpha values and l1 ratios for exploring
alpha_values = [0.0001, 0.001, 0.01, 0.1, 1, 5, 10]
l1_ratios = np.arange(0.0, 1.1, 0.1)

results_tuples = []
en_rmse_scores = []
mean_en_rmse_scores = []

for alpha in tqdm(alpha_values):
    for l1 in l1_ratios:
        
        en_reg = ElasticNet(alpha=alpha, l1_ratio=l1, tol=0.5,
                            fit_intercept=True, max_iter=1000)
        en_scores = cross_val_score(en_reg, X, y, groups=groups,
                                    scoring='neg_mean_squared_error', cv=grouped_kfold)
        rmse_scores = np.sqrt(-en_scores)
        en_rmse_scores.append(rmse_scores)
        mean_en_rmse_scores.append(rmse_scores.mean())
        
        results_tuples.append((alpha, l1, rmse_scores.mean(), rmse_scores.std()))

en_rmse_scores = np.array(en_rmse_scores)
mean_en_rmse_scores = np.array(mean_en_rmse_scores)

print(f"Elastic Net RMSE: {mean_en_rmse_scores.mean():.5f} +/- {mean_en_rmse_scores.std():.5f}")
# form dataframe from the results
elasnet_results = pd.DataFrame(results_tuples)
elasnet_results.columns = ['alpha', 'l1_ratio', 'mean', 'std']

# plot results according to hyper-parameters and mean RMSE
plt.figure(figsize=(16,6))
sns.scatterplot(x='alpha', y='l1_ratio', hue='mean', 
                data=elasnet_results, s=150)
plt.xlabel('ElasticNet Alpha Parameter', weight='bold', size=14)
plt.ylabel('ElasticNet l1-ratio', weight='bold', size=14)
plt.legend(loc='best')
plt.xlim(-0.5, 13.0)
plt.show()
"""
It seems that regularisation really isn't helping our linear model perform any better on our dataset; The basic linear regression model without any regularisation tends to perform better on our temperature prediction task when compared to the regularised methods. This can be seen in the plot above, whereby the points towards the lower right of the plot (those more representing normal linear regression with no regularisation) perform much better. This suggests that we're not actually fitting our data enough, and therefore performing regularisation (preventing over-fitting) may actually be harmful rather than good. We could consider adding further features to produce a more complex model, but in this case the results will suffice.
"""
"""
### 9.4 Investigating performance of bootstrapped linear regression ensemble:
"""
lr_bag_reg = BaggingRegressor(LinearRegression(), n_estimators=5, bootstrap=True)

lr_bag_rmse_scores = []
mean_lr_bag_rmse_scores = []

# times to repeat the cross validation
repeats = 2

for i in tqdm(range(repeats)):
    lr_bag_scores = cross_val_score(lr_bag_reg, X, y, groups=groups,
                                scoring='neg_mean_squared_error', cv=grouped_kfold)
    rmse_scores = np.sqrt(-lr_bag_scores)
    lr_bag_rmse_scores.append(rmse_scores)
    mean_lr_bag_rmse_scores.append(rmse_scores.mean())

lr_bag_rmse_scores = np.array(lr_bag_rmse_scores)
mean_lr_bag_rmse_scores = np.array(mean_lr_bag_rmse_scores)

print(f"Lin Reg Bagging RMSE: {mean_lr_bag_rmse_scores.mean():.5f} +/- {mean_lr_bag_rmse_scores.std():.5f}")
"""
### 9.5 Exploration of tree based models:

The performance of our random forest and extra tree regressors were not actually that bad. With refinement, these could become good models to include within a final ensemble, and therefore its worth trying to optimise the performance if we can.

In order to do this, we'll take an even smaller stratified sample of training instances based on profile ID (due to higher time complexity with these methods), followed by unique grouping between training / validation of profile IDs, just like we did above. This will allow us to experiment and fine-tune on a much smaller subsample, which still has the benefit of having different profile IDs between our training and validation splits.
"""
X_trg_sub, _, y_trg_sub, _ = train_test_split(X_std, y, stratify=groups, test_size=0.99, random_state=12)
trg_profiles = X_trg_sub.loc[:, 'profile_id'].values

print(f"Number of profile IDs within sample: {X_trg_sub['profile_id'].value_counts().shape[0]}\n")

# drop profile ID before training
X_trg_sub = X_trg_sub.drop('profile_id', axis=1)

print(f"Shapes of data: X = {X_trg_sub.shape}, y = {y_trg_sub.shape}")

sample_group_kfold = GroupShuffleSplit(n_splits=6)
"""
Good - our sample still retains instances from all 42 profile IDs after our stratified sampling technique. We can now perform grouped K-Folds on this in a similar way to above.
"""
"""
#### Exploration of Extra Trees Hyper-Parameters

Below we'll define a range of parameters to use with grid search for finding optimal hyper-parameters for our Extra Trees Regressor. Note - we'll not bother using the n_estimators parameter, since we know a larger number of estimators helps to reduce overfitting, so for the final model we'll use a large number of n_estimators. By using this during grid search we'll just increase the search time considerably for relatively little insight.
"""
et_param_grid = {'n_estimators' : [10], 'max_depth' : [5, 10, 20, 60], 
                 'min_samples_split' : [2, 5, 10, 20], 'min_samples_leaf' : [1, 3, 5, 10], 
                 'n_jobs': [-1], 'bootstrap' : [True]}

et_reg = ExtraTreesRegressor()
et_grid_search = GridSearchCV(et_reg, et_param_grid, cv=sample_group_kfold, scoring='neg_mean_squared_error')
# fit to our grid search with grouped kfold
%time et_grid_search.fit(X_trg_sub, y_trg_sub, groups=trg_profiles)
# display best parameters found during grid search
best_et_params = et_grid_search.best_params_
best_et_params
# change best params n_estimators to a higher a more appropriate value
best_et_params['n_estimators'] = 600
best_et_params
"""
Best parameters found were: 
{'bootstrap': True,
 'max_depth': 20,
 'min_samples_leaf': 3,
 'min_samples_split': 2,
 'n_jobs': -1}
"""
"""
Let's perform cross-validation again using this model and the optimal hyper-parameters, along with more estimators:
"""
et_reg = ExtraTreesRegressor(**best_et_params)

et_scores = cross_val_score(et_reg, X_trg_sub, y_trg_sub, groups=trg_profiles, 
                            scoring='neg_mean_squared_error', cv=sample_group_kfold)

et_rmse_scores = np.sqrt(-et_scores)
print(f"Extra Trees Reg RMSE: {et_rmse_scores.mean():.5f} +/- {et_rmse_scores.std():.5f}")
"""
The performance of this model is not bad now! If we trained this on all of the training set, I expect this performance would be improved even further, especially when considering this model has only been trained on less than 1% of the total training set!

If we perform the same with Random Forest, the optimal parameters found are similar, although performance is slightly less than that found with the Extra Trees Regressor; Clearly the extra bias provided by Extra Trees lets it perform better to our data than that with a Random Forest.

Either of these tree based models will be ideal to use in a final ensemble model for our regression problem. Extra Trees is preferable, since its much faster to train, and better performing in this case.
"""
"""
**Exploring the performance of a Gradient Boosting Regressor:**
"""
gb_reg = MultiOutputRegressor(GradientBoostingRegressor())

gb_scores = cross_val_score(gb_reg, X_trg_sub, y_trg_sub, groups=trg_profiles, 
                            scoring='neg_mean_squared_error', cv=sample_group_kfold)

gb_rmse_scores = np.sqrt(-gb_scores)
print(f"Gradient Boosting Reg RMSE: {gb_rmse_scores.mean():.5f} +/- {gb_rmse_scores.std():.5f}")
"""
Clearly our gradient boosting regressor has high potential here, and could likely be trained to have better performance than our linear regression OLS model. The problem with this method however is the extremely long training time on the entire dataset, which could become problematic.

If we performed hyper-parameter optimisation on this, I have no doubt that it would be the highest performing model so far on our data.
"""
"""
### 9.6 Production of a stacking ensemble to combine the predictions of multiple models:

A good strategy is to combine the results of multiple models together, which often results in a better overall performance than either of the standalone models produce. This assumes that each model chosen is relatively good in terms of performance, and the more diversity we have between each model used in the ensemble, the better the chance of our performance being better for a given task.

Just to illustrate this concept, and produce a reasonable model, lets combine the benefits of a tree based model, Extra Trees Regressor, along with our multiple-output Linear Regression OLS mode (with no regularisation).
"""
regressors = [('Bagged Linear Reg', BaggingRegressor(LinearRegression(), n_estimators=20, bootstrap=True)), 
              ('Extra Trees Reg', ExtraTreesRegressor(**best_et_params))]
ensemble_reg = MultiOutputRegressor(VotingRegressor(estimators=regressors, n_jobs=-1))

ensemble_scores = cross_val_score(ensemble_reg, X_trg_sub, y_trg_sub, groups=trg_profiles, 
                            scoring='neg_mean_squared_error', cv=sample_group_kfold)

ensemble_rmse_scores = np.sqrt(-ensemble_scores)
print(f"Ensemble Reg RMSE: {ensemble_rmse_scores.mean():.5f} +/- {ensemble_rmse_scores.std():.5f}")
"""
As expected, our ensemble is actually better performing than either model alone! This should suffice for a final submission and evaluation on the test set.
"""
"""
<a id="final-test-set-predictions"></a>
## 10. Producing our models and making predictions on the final test set
"""
data_preprocessor = PMSMDataProcessor()

train_df_extra = data_preprocessor.fit_transform(train_df)
test_df_extra = data_preprocessor.transform(test_df)

y = train_df_extra[data_preprocessor.output_vars].copy()
X = train_df_extra.drop(columns=data_preprocessor.output_vars)

# select profile id's as groups to split our folds on
groups = X.loc[:, 'profile_id'].copy()
X.drop('profile_id', axis=1, inplace=True)

X.shape, y.shape, test_df_extra.shape
std_scaler = StandardScaler()
X_std = pd.DataFrame()
X_std[X.columns] = pd.DataFrame(std_scaler.fit_transform(X))
test_std = std_scaler.transform(test_df_extra)
et_opt_params = {'n_estimators' : 600, 'bootstrap': True, 'max_depth': 20, 
                 'min_samples_leaf': 3, 'min_samples_split': 2, 'n_jobs': -1}
# train final extra trees classifier
et_reg = ExtraTreesRegressor(**et_opt_params)
%time et_reg.fit(X_std, y)
# train final extra trees classifier
bag_lr_reg = BaggingRegressor(LinearRegression(), n_estimators=20, bootstrap=True)
%time bag_lr_reg.fit(X_std, y)
et_test_preds = et_reg.predict(test_std)
print(f"Final Extra Trees RMSE on Test Set: {np.sqrt(mean_squared_error(et_test_preds, test_labels)):.5f}")
lr_test_preds = bag_lr_reg.predict(test_std)
print(f"Final Linear Regression RMSE on Test Set: {np.sqrt(mean_squared_error(lr_test_preds, test_labels)):.5f}")
"""
With both sets of predictions obtained from these different models, we can simply combine them into a final merged dataset by averaging their results. 
"""
ensemble_preds = (et_test_preds + lr_test_preds) / 2.0
print(f"Final Ensemble RMSE on Test Set: {np.sqrt(mean_squared_error(ensemble_preds, test_labels)):.5f}")
# create an ensemble and compare final performance - we should expect a marginal improvement
regressors = [('Bagged Linear Reg', BaggingRegressor(LinearRegression(), n_estimators=20, bootstrap=True)), 
              ('Extra Trees Reg', ExtraTreesRegressor(**et_opt_params))]

# craete our final ensemble model
ensemble_reg = MultiOutputRegressor(VotingRegressor(estimators=regressors))
#print("Training final ensemble model on entire training set.\n")
#ensemble_reg.fit(X_std, y)
#test_preds = ensemble_reg.predict(test_std)
#print(f"Final Ensemble RMSE on Test Set: {np.sqrt(mean_squared_error(test_preds, test_labels)):.5f}")
"""
## Closing Remarks

So there we have it - a final prediction on our test set, which was completely unseen by the trained models due to the unique Profile IDs assigned to the test at the very beginning of this notebook. Our performance was pretty strong, especially when considering the relative simplicity of the models that we used in our final product.

We could definitely improve this model further, which could be done by one or more of the following strategies:

- Production of a larger ensemble with more diversity.

- Use of better models, such as gradient boosting regressors to produce our final model. This wasn't included within this notebook due to the significant training time of a gradient boosting model on the huge dataset provided. However, the basic concepts remain the same and this could easily be added into the above analysis.

- Use of a neural network implementation with a better capacity to learn the complex relationships within our data.

- Better exploitation of time-series within our data, which could be done by the use of models such as RNN LSTMs or CNNs to learn time-dependent patterns within our data. It's expected that if we did this well, it would result in the best overall performance when compared to any other classical model implementation. The downside of this is the inference time could be greater, and the training time could be significant due to model complexity.

I hope you enjoyed this notebook - many thanks for reading!
"""