\section{Theory Background}
\label{background}

The Raft protocol divides the initial problem into two separate problems: leader election and log replication. The leader election is solved with a voting procedure. The log replication is based on maintaining a set of constraints on the contents on each server's log, guaranteeing the consistency of the replicated log entries. The protocol is based on two RPCs: RequestVoteRPC (for leader election) and AppendEntriesRPC (for log replication). The two phases follow each others inside a \emph{term}.

The servers that participate in the protocol may take three roles: \emph{leader, follower, or candidate}.
\begin{itemize}
    \item The leader is responsible for linearizing the insertion of log entries on the other machines. It receives the entries to append from the clients, and then proceeds to replicate them.
    \item The followers respond to RPCs from the leader.
    \item The candidate role is required only when preparing to elect a new leader (see \ref{leader-election}).
\end{itemize}
Each server may take only one role at the time.
\subsection{Leader Election}
\label{leader-election}
\begin{algorithm}[ht]
\caption{Arguments of the RequestVoteRPC}
\label{algo-requestVote-arguments}
bool RequestVoteRPC( \\
    term, \\
    candidateId, \\
    lastLogIndex, \emph{// index of the last log entry (i.e: the most recent one)}\\
    lastLogTerm   \emph{// term of the last log entry}\\
    ) \;
\end{algorithm}
The outline of the leader election protocol is simple. Any server can start the leader election procedure after a certain time has passed without receiving any messages (indicating that the previous leader is dead or not connected anymore). This ``candidate'' server sends a RequestVoteRPC to all other servers (which may be in any other state), and they decide to grant or not the vote to the candidate. If a majority of servers granted their votes, the candidate becomes the new leader. Otherwise, it starts a new election. The timeouts on each server are usually initialized with some random offset, to avoid simultaneous elections. Simultaneous elections do not result in more than one leader being elected, but they may split the majority and the election would not produce a leader. A server may only vote once in a single term, thus it is granted that only one leader will be elected in a single term.

\subsection{Log Replication}
\begin{algorithm}[ht]
\caption{Arguments of the AppendEntriesRPC}
\label{algo-appendEntries-arguments}
bool AppendEntriesRPC( \\
    term, \\
    leaderId, \\
    prevLogIndex, \emph{// index of the last log entry} \\
    prevLogTerm,  \emph{// term of the last log entry} \\
    entries[],    \emph{// entries to append} \\
    leaderCommit  \emph{// leader's commitIndex} \\
    )\;
\end{algorithm}
After electing a leader for the cluster, the log replication phase can begin. When the leader has new entries appended to its own log, it sends an AppendEntriesRPC to all followers. The entry is appended if it satisfies a series of checks (see \ref{when-to-append}), otherwise the RPC fails. If the RPC succeeds on the majority of the followers, the entry is considered committed. The log replication phase ends if one of the followers decides to start a new election; the protocol returns in the leader election phase, until a new leader is elected.
\subsubsection{When to append log entries}
\label{when-to-append}
During normal operation (no messages are lost or delayed) the followers can append entries to their local log just as they receive them. However, if messages are lost or delayed, the followers need to be careful in what they add. For example, if a follower receives entries with index $\{1,2,3,4\}$, but the entry with index 3 is lost, the entries in the local log of the follower will be $\{1,2,4\}$, and the logs would diverge. Instead, a follower appends an entry if it satisfies the following requirements (see algorithm \ref{algo-add-entries-local-log}):
\begin{enumerate}
    \item It has a term number $\geq$ than the current term number
    \item The indicated previous entry has the same term number as the previous entry on the leader
\end{enumerate}
These requirements ensure that a given follower appends entries coming from the current leader and are applied in the same order as all other followers. If the follower cannot append the entries, the RPC fails. The leader does not "give up" on a failed RPC but instead sends previous entries from its own local log (see algorithm \ref{algo-leader-update-status}). By following this scheme, the leader will eventually find an entry that can be appended on the follower, and then will send the entries following that entry.

A follower needs a mechanism to correct its own log when appending new entries. When adding a set of new entries, if there is a new entry with the same index and the same term as an entry already present in the log, the new entry replaces the old one. All the entries that follow the replaced one are deleted. This allows for a follower to correct the contents of its log, if is elected a leader that has a different log from the previous one (within certain limits; see \ref{committing-entries}).
\begin{algorithm}[htp]
\caption{Procedure used by a follower to verify if it can add a new set of entries to its own localLog}
\label{algo-add-entries-local-log}
addEntries(newEntries,term,prevLogIndex,leaderCommit)
\Begin{
    previousEntry $\gets$ localLog[prevLogIndex]\;
    \eIf{term $\geq$ currentTerm $\vee$ previousEntry.term $\neq$ prevLogTerm}
    {
        RPC fails; reply false to leader and terminate\;
    }{
        %else...
        \If{entryConflicts(localLog,newEntries)}
        {
            deleteFollowers()\;
        }
        appendEntriesToLog(newEntries)\;
        \If{leaderCommit $\geq$ commitIndex}
        {
            \emph{// if the leader already committed entries ahead of us, we can safely commit them}
            commitIndex $\gets$ min(leaderCommit, newEntries.lastIndex())\;
        }
    }
}
\end{algorithm}

\begin{algorithm}[ht]
\caption{Procedure used by a leader to update its internal status based on the result of AppendEntriesRPC from a follower}
\label{algo-leader-update-status}
updateInternalStatus(follower\_id)
\Begin{
    \eIf{RPC was successful}
    {
        nextIndex[follower\_id] $\gets$ nextIndex[follower\_id] + 1\;
        matchIndex[follower\_id] $\gets$ matchIndex[follower\_id] + 1\;
    }{
        %else...
        nextIndex[follower\_id] $\gets$ nextIndex[follower\_id] - 1\;
        retry RPC with different nextIndex\;
    }
    \emph{/* Either way, success or failure, we try to update the leader's commitIndex.} \\
    \emph{We look if exists a majority of values of matchIndex, such that matchIndex is greater than the current commit index.
                In this way, we make sure that we increase the commitIndex only if we have previously received a positive RPC from
                the majority of followers */} \\
    \For{N $\gets$ commitIndex; N $<$ localLog.size(); N $\gets$ N + 1}
    {
        aheadFollowers $\gets$ 0\;
        \emph{// search for followers that have a known matchIndex greater than the current N } \\
        \For{i $\gets$ 0; i $<$ matchIndex.size(); i $\gets$ i + 1;}
        {
            \If{matchIndex[i] >= N $\wedge$ localLog[N].term $==$ currentTerm}
            {
                 aheadFollowers $\gets$ aheadFollowers + 1\;
            }
        }
        \If{aheadFollowers $\geq$ (numberOfFollowers $/$ 2)}
        {
            commitIndex $\gets$ N\;
        }
    }
}
\end{algorithm}

\subsubsection{Commits}
\label{committing-entries}
The Raft protocol maintains an index to the most recent committed entry. An entry \emph{e} is committed when a majority of servers have \emph{e} in their logs, with the same index and with the same term. This implies that all parts of the log with \texttt{index} $<$ \texttt{commitIndex} are present on all servers, all with the same term, all with the same index. \\
However, the phrase "a majority of servers have that entry in their logs" is misleading. On a first level, it seems that the leader reads the logs of the followers and then decides which entry is committed. But a more careful read of the protocol brings out an important detail: the leader uses \emph{its own state} to decide which entry to commit, nothing else. The state of the logs is stored inside the \texttt{matchIndex[]} array; each entry contains the index of the higher entry known to be replicated on the server with id \emph{i}. The leader knows that an entry is replicated when receives the response to the AppendEntriesRPC. By virtue of being replicated on a majority of servers, a committed entry is 'safe'; even if the leader changes, it is guaranteed that any committed entry will eventually be present on all followers.

\clearpage
\newpage


\input{settings}

\begin{document}

\lhead{Sergio Arnaud}
\rhead{ Statistical learning (MT7038) \\ Final Project }
\cfoot{\thepage \ of \pageref{LastPage}}

\section{Dataset}

For this proyect, the dataset chosen was \href{https://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant}{\textit{Combined Cycle Power Plant Data Set}}. According to the datasets information:

\begin{displayquote}
A combined cycle power plant is composed of gas turbines, steam turbines and heat recovery steam generators. In a CCPP, the electricity is generated by gas and steam turbines, which are combined in one cycle, and is transferred from one turbine to another. While the Vacuum is colected from and has effect on the Steam Turbine, he other three of the ambient variables effect the GT performance.
\end{displayquote}

My main motivation to use this dataset was that the last month there was a \href{https://www.kaggle.com/c/ashrae-energy-prediction/overview}{kaggle} challenge (ASHRAE - Great Energy Predictor III) that tried to answer the question: 'How much energy will a building consume?'. I think this kind of problems dealing with the estimation of energy production and consumption will start to appear more and more and I personally find them to be both really interesting and challenging.

Furthermore, I think that the data was complicated enough to challenge common regression techniques yet simple enough to be solved without advanced domain knowledge or the requirement of more advanced modelling habilities.
\section{Features}
The dataset contains 9568 observations collected from a CCPP over the period 2006-2011 and the features consist of:
\begin{center}
    \begin{tabular}{ |p{5.5cm}||p{2.25cm}|p{2.25cm}|p{1.8cm}|}
     \hline
     Feature name & Min & Max & Data type \\
     \hline
     Temperature (AT)              & 1.81 \degree C & 37.11\degree C & Continous \\
     Ambient Pressure (AP)         & 992.89 mb  & 1033.30 mb   & Continous \\
     Relative Humidity (RH)        & 25.56\%   &  100.16\% &  Continous \\
     Exhaust Vacuum (V)            & 25.36 cm Hg  & 81.56 cm Hg &   Continous \\
     Electrical energy output (PE) & 420.26 MW  & 495.76 MW & Continous \\
     \hline
    \end{tabular}
\end{center}
The individual distribution of each feature is shown in the following figure:
\begin{figure}[H]
    \begin{center}
        \includegraphics[width=12cm, height=7cm]{img/hist.jpg}
        \caption{Histogram of the variables}
    \end{center}
\end{figure}
\newpage
Both the ambient temperature and the electrical energy output seem to have a bimodal distribution, the relative humidity is unimodal with a negative skew, the ambient pressure appears to be normally distributed and the exhaust vacuum shows the most complicated distribution with several distinct modes.
\section{Relations of the predictors with the response variable}
First, let's take a look to the correlation matrix:
\begin{center}
    \begin{tabular}{ |p{.75cm}|p{.75cm}|p{.75cm}|p{.75cm}|p{.75cm}|p{.75cm}|}
     \hline
      & PE & AP & RH & V & AT \\
     \hline
     PE & 1 &  &  &  & \\
     \hline
     AP & .52 & 1 &  &  & \\
     \hline
     RH & .39 & .1 & 1 &  & \\
     \hline
     V & -.87 & -.41 & -.31 & 1 & \\
     \hline
     AT & -.95 & -.51 & -.54 & .84 & 1\\
     \hline
    \end{tabular}
\end{center}
the response variable has a high negative correlation with both the ambient temperature and the exhaust vacuum, also those features are highly correlated between them.

Now, let's focus on the individual relations of the predictors with the response variable:
\begin{figure}[H]
    \begin{center}
        \includegraphics[width=0.8\textwidth]{img/relations.jpg}
        \caption{Individual relations of predictors with response}
    \end{center}
\end{figure}
There's a clear relationship between the temperature and the energy output: for a higher temperature, a lower energy output corresponds. Also the exhaust vacuum appear to show a relationship of that kind. We can't extract any conclusions from the ambient pressure and the relative humidity from this figure.

To finish the analysis of our features, it's important to mention that the data have no misssing values and it looks like we don't have to deal with outliers or further cleaning so we'll just standardize each variable and proceed with the modelling.
\newpage

\section{Modelling}

All the predictors and the response are continuous variables so the task for this dataset is a regression.

A gradient boosted decision tree was used to solve the regression task since I think it's a good fit for the problem in hand and personally wanted to explore and learn more about this particular method.

A more simple and interpretable model was used as a baseline to compare the performance of the gradient boosting model. A linear regresion was fitted, lasso and ridge methods were tried but multicollinearity weren't a problem and the improvement in terms of test error was minimal so I decided in favor of the more simple linear regresion.

\section{Software}
Tha programming language used for the analysis of the CCPP dataset was R and the following table describes the main packages and libraries utilized:
\begin{center}
    \begin{tabular}{ |p{4cm}||p{10.5cm}|}
         \hline
         Library & Description \\
         \hline
         \href{https://www.tidyverse.org}{\textit{Tidyverse}} & For data manipulation and visualization. \\
         \href{http://topepo.github.io/caret/index.html}{\textit{Caret}} (Classification And Regression Training) &  The package goal is to provide a uniform interface to standardize common modelling tasks such parameter tuning and variable importance. In particular, \textit{caret} was used for the cross validation procedure.  \\
         \href{https://xgboost.ai/about}{\textit{XGBoost}}(Extreme Gradient Boosting) & An optimized distributed library which provides a gradient boosting framework for several programming languages including R. \\
         \href{https://www.rdocumentation.org/packages/xgboostExplainer/versions/0.1}{xgboostExplainer} & An R package which allows the predictions from an XGBoost model to be split into the impact of each feature, making the model more interpretable. \\
         \hline
    \end{tabular}
\end{center}

\section{Linear model}
The linear regresion model was straightforward, all the predictors were significant to the response and a significant regression equation was found $(F(4,7651) = \num{2.517e-4}, p < \num{2.2e-16})$ with an $R^2 = 0.9294$.

\section{Gradient boosted decision tree}

Using the library XGBoost a gradient boosted decision tree model was fitted to predict the electrical energy output. The first issue to solve was how to pick the optimal hyperparameters of the model.

\subsection{Parameter tuning}

According to XGBoost there are many hyperparameters to be tuned. The following table resumes the most important, it's meaning, and the value used.

\begin{tabular}{ |p{2cm}||p{11cm}| p{1cm}|}
\hline
Parameter & Description & Value\\
\hline
eta & After each boosting step, eta shrinks the feature weights to make the boosting process more conservative and less likely to overfit & .1 \\
gamma & Minimum loss reduction required to make a further partition on a leaf node of the tree. A larger gamma implies a more conservative algorithm. & 0 \\
max\_depth & Maximum depth of a tree. Increasing this value will make the model more complex and more likely to overfit. & 6 \\
min\_child \_weight & If the tree partition step results in a leaf node with the sum of instance weight less than min\_child\_weight, then the building process will give up further partitioning. & 2\\
\hline
\end{tabular}

To select, eta, max\_depth and min\_child\_weight a grid search was performed and the optimal values were selected with 5 fold cross-validation using caret \textit{train} function. After selecting the optimal hyperparameters, the optimal number of rounds for boosting was found to be 600.

\subsection{Model training and evaluation}
With those parameters, the \textit{xgb.train} function was used to train the optimal model. The test mean squared error was $0.032$ and, as seen in the predicted vs actual energy output plot the predictions are really good.

\begin{figure}[H]
    \begin{center}
        \includegraphics[width=0.8\textwidth]{img/pred_act.jpg}
        \caption{Actual vs Predicted energy output}
    \end{center}
\end{figure}

\subsection{Feature importance}

After the boosted trees are constructed, we can build an importance score for each attribute. That score indicates how valuable each feature was in the fitting process. With the function \textit{xgb.importance}, the following importance values were calculated.
\newpage

\begin{center}
    \begin{tabular}{ |p{4cm}||p{3cm}|}
         \hline
         Library & Description \\
         \hline
         Ambient temperature & .90 \\
         Exhaust Vacuum & .06 \\
         Ambient Preassure & .015 \\
         Relative Humidity & .013 \\
         \hline
    \end{tabular}
\end{center}

The ambient temperature is by far the most important feature, followed by the exhaust vacuum. The ambient preassure and the relative humidity weren't very relevant.

\subsection{Interpretation}

Using the The \textit{xgboostExplainer} R package, we can extract the log-odds contribution of each feature in a prediction by adding up the contributions of each one of them for every tree in the ensemble. With that information, we can measure the impact that each feature had in a particular prediction.

Each plot in the following figure shows the values of a predictor plotted against the impact associated with that value.

\begin{figure}[H]
    \begin{center}
        \includegraphics[width=0.9\textwidth]{img/expl_all.jpg}
        \caption{Value vs impact}
    \end{center}
\end{figure}
In the first plot we can see that for lower temperatures the impact is positive (lower temperature contributes to a greater energy output) and as the temperature increases, the impact decreases (a greater temperature contributes to a smaller energy output)

The plot of ambient preassure suggests that, for smaller and bigger values of the preassure the impact to the response is negative (lower and higher preasures lead to an smaller energy output) whilst a mean preassure has, in general, a positive impact (a mean preassure leads to a greater energy output).
\newpage
The plots of exhaust vacuum and relative humidity will be analized with the following graph where each dot is colored with red if the temperature was higher than the mean and blue if it was lower than the
mean.
\begin{figure}[H]
    \begin{center}
        \includegraphics[width=11cm, height=8cm]{img/expl_2}
        \caption{Value vs impact}
    \end{center}
\end{figure}

From the first plot we can see that, for bigger values of humidity we may observe greater energy outputs if the temperature is above the mean; and smaller energy outputs if the temperature is below the mean. A similar analysis can be made for the second plot.

\section{Conclusions}

The linear model achieved good results and is definitely a good alternative to analyze the problem since it's the most interpretable model we could use.

The gradient boosted decision tree model was able to capture some nonlinear relationships between the predictors achieving a mean squared error 76\% smaller than the one from the linear model.

Since the gradient boosting is a complicated model, usually the results lack interpretation. Nevertheless, there exist tools to helps us extract important information about the way in which the model is working and how the predictions are made.

I'm really satisfied with the fact that the gradient boosting model achieves good prediction power without renouncing to interpretation. The gradient boosting framework is, indeed, a very powerfull one.

Posible improvements might involve be in several areas:
\begin{itemize}
    \item The dataset only uses the 4 more common variables in a CCPP, more features could be added for a broader analysis.
    \item The results of both the linear and the gradient boosting models could be further interpreted, maybe with more work and domain knowledge.
    \item Only the main parameters of the gradient boosting model were tuned, we could try doing more exhaustive hyperparameter optimization
    \item Other regression techniques could be tried in order to achieve better predictions or further insights.
\end{itemize}

\end{document}