\chapter{Multiple and logistic regression}
\label{multipleRegressionAndANOVA}
\label{multipleAndLogisticRegression}

The principles of simple linear regression lay the foundation for more sophisticated regression methods used in a wide range of challenging settings. In Chapter~\ref{multipleAndLogisticRegression}, we explore multiple regression, which introduces the possibility of more than one predictor, and logistic regression, a technique for predicting categorical outcomes with two possible categories.

\section{Introduction to multiple regression}
\label{introductionToMultipleRegression}

\index{multiple regression|(}

Multiple regression extends simple two-variable regression to the case that still has one response but many predictors (denoted $x_1$, $x_2$, $x_3$, ...). The method is motivated by scenarios where many variables may be simultaneously connected to an output.

\index{data!mario\_kart|(}

We will consider Ebay auctions of a video game called \emph{Mario Kart} for the Nintendo Wii. The outcome variable of interest is the total price of an auction, which is the highest bid plus the shipping cost. We will try to determine how total price is related to each characteristic in an auction while simultaneously controlling for other variables. For instance, all other characteristics held constant, are longer auctions associated with higher or lower prices? And, on average, how much more do buyers tend to pay for additional Wii wheels (plastic steering wheels that attach to the Wii controller) in auctions? Multiple regression will help us answer these and other questions.

The data set \textbf{mario\_\hspace{0.3mm}kart} includes results from 141 auctions.\footnote{Diez DM, Barr CD, and \c{C}etinkaya-Rundel M. 2012. \emph{openintro}: OpenIntro data sets and supplemental functions. \href{http://cran.r-project.org/web/packages/openintro}{cran.r-project.org/web/packages/openintro}.} Four observations from this data set are shown in Table~\ref{marioKartDataMatrix}, and descriptions for each variable are shown in Table~\ref{marioKartVariables}. Notice that the condition and stock photo variables are indicator variables\index{indicator variable}. For instance, the \textbf{cond\_\hspace{0.3mm}new} variable takes value 1 if the game up for auction is new and 0 if it is used. Using indicator variables in place of category names allows for these variables to be directly used in regression. See Section~\ref{categoricalPredictorsWithTwoLevels} for additional details. Multiple regression also allows for categorical variables with many levels, though we do not have any such variables in this analysis, and we save these details for a second or third course.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrlr}
  \hline
 & price & cond\_\hspace{0.3mm}new & stock\_\hspace{0.3mm}photo & duration & wheels \\
  \hline
1 & 51.55 &   1 & 1 & 3 &   1 \\
  2 & 37.04 &  0 &  1 & 7 &   1 \\
$\vdots$ &$\vdots$ &$\vdots$ &$\vdots$ &$\vdots$ &$\vdots$ \\
  140 & 38.76 &  0 &  0 & 7 &   0 \\
  141 & 54.51 &  1 &  1 & 1 &   2 \\
   \hline
\end{tabular}
\caption{Four observations from the \textbf{mario\_\hspace{0.3mm}kart} data set.}
\label{marioKartDataMatrix}
\end{table}
%library(openintro); data(marioKart); d <- marioKart[marioKart$totalPr < 100,]; row.names(d) <- NULL; d

\begin{table}
\centering\small
\begin{tabular}{lp{9.5cm}}
\hline
{\bf variable} & {\bf description} \\
\hline
\textbf{price} & final auction price plus shipping costs, in US dollars \\
\textbf{cond\_\hspace{0.3mm}new} & a coded two-level categorical variable, which takes value \resp{1} when the game is new and \resp{0} if the game is used \\
\textbf{stock\_\hspace{0.3mm}photo} & a coded two-level categorical variable, which takes value \resp{1} if the primary photo used in the auction was a stock photo and \resp{0} if the photo was unique to that auction \\
\textbf{duration} & the length of the auction, in days, taking values from 1 to 10 \\
\textbf{wheels} & the number of Wii wheels included with the auction (a \emph{Wii wheel} is a plastic racing wheel that holds the Wii controller and is an optional but helpful accessory for playing Mario Kart) \\
\hline
\end{tabular}
\caption{Variables and their descriptions for the \textbf{mario\_\hspace{0.3mm}kart} data set.}
\label{marioKartVariables}
\end{table}

\subsection{A single-variable model for the Mario Kart data}
\label{twoSingleVariableModelsForMarioKartData}

Let's fit a linear regression model with the game's condition as a predictor of auction price. The model may be written as
\begin{align*}
\widehat{price} &= 42.87 + 10.90\times cond\_\hspace{0.3mm}new
\end{align*}
Results of this model are shown in Table~\ref{singleVarModelsForPriceUsingCond} and a scatterplot for price versus game condition is shown in Figure~\ref{marioKartSingle}.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & 42.8711 & 0.8140 & 52.67 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 10.8996 & 1.2583 & 8.66 & 0.0000 \\
   \hline
   &&&\multicolumn{2}{r}{$df=139$}
\end{tabular}
\caption{Summary of a linear model for predicting auction price based on game condition.}
\label{singleVarModelsForPriceUsingCond}
\end{table}
% library(openintro); library(xtable); data(marioKart); d <- marioKart[marioKart$totalPr < 100,]; d$cond <- relevel(d$cond, "used"); xtable(lm(d$totalPr ~ d$cond)); xtable(lm(d$totalPr ~ d$duration))

\begin{figure}
\centering
\includegraphics[width=0.65\textwidth]{06/figures/marioKartSingle/marioKartSingle}
\caption{Scatterplot of the total auction price against the game's condition. The least squares line is also shown.}
\label{marioKartSingle}
\end{figure}

\begin{exercise}
Examine Figure~\ref{marioKartSingle}. Does the linear model seem \mbox{reasonable}?\footnote{Yes. Constant variability, nearly normal residuals, and linearity all appear reasonable.}
\end{exercise}

\textA{\pagebreak}

\begin{example}{Interpret the coefficient for the game's condition in the model. Is this coefficient significantly different from 0?}
Note that \textbf{cond\_\hspace{0.3mm}new} is a two-level categorical variable that takes value 1 when the game is new and value 0 when the game is used. So 10.90 means that the model predicts an extra \$10.90 for those games that are new versus those that are used. (See Section~\ref{categoricalPredictorsWithTwoLevels} for a review of the interpretation for two-level categorical predictor variables.) Examining the regression output in Table~\ref{singleVarModelsForPriceUsingCond}, we can see that the p-value for \textbf{cond\_\hspace{0.3mm}new} is very close to zero, indicating there is strong evidence that the coefficient is different from zero when using this simple one-variable model.
\end{example}

\subsection{Including and assessing many variables in a model}
\label{includingAndAssessingManyVariablesInAModel}

Sometimes there are underlying structures or relationships between predictor variables. For instance, new games sold on Ebay tend to come with more Wii wheels, which may have led to higher prices for those auctions. We would like to fit a model that includes all potentially important variables simultaneously. This would help us evaluate the relationship between a predictor variable and the outcome while controlling for the potential influence of other variables. This is the strategy used in \textbf{multiple regression}. While we remain cautious about making any causal interpretations using multiple regression, such models are a common first step in providing evidence of a causal connection.

We want to construct a model that accounts for not only the game condition, as in Section~\ref{twoSingleVariableModelsForMarioKartData}, but simultaneously accounts for three other variables: \textbf{stock\_\hspace{0.3mm}photo}, \textbf{duration}, and \textbf{wheels}.
\begin{align}
\widehat{\textbf{price}}
	&= \beta_0 + \beta_1\times \textbf{cond\_\hspace{0.3mm}new} +
		\beta_2\times \textbf{stock\_\hspace{0.3mm}photo} \notag \\
	&\qquad\  + \beta_3 \times  \textbf{duration} +
		\beta_4 \times  \textbf{wheels} \notag \\
\hat{y}
	&= \beta_0 + \beta_1 x_1 + \beta_2 x_2 +
		\beta_3 x_3 + \beta_4 x_4
\label{eqForMultipleRegrOfTotalPrForAllPredictors}
\end{align}
In this equation, $y$ represents the total price, $x_1$ indicates whether the game is new, $x_2$ indicates whether a stock photo was used, $x_3$ is the duration of the auction, and $x_4$ is the number of Wii wheels included with the game. Just as with the single predictor case, a multiple regression model may be missing important components or it might not precisely represent the relationship between the outcome and the available explanatory variables. While no model is perfect, we wish to explore the possibility that this one may fit the data reasonably well.

We estimate the parameters $\beta_0$, $\beta_1$, ..., $\beta_4$ in the same way as we did in the case of a single predictor. We select $b_0$, $b_1$, ..., $b_4$ that minimize the sum of the squared residuals:
\begin{align}\label{sumOfSqResInMultRegr}
SSE = e_1^2 + e_2^2 + \dots + e_{141}^2
	= \sum_{i=1}^{141} e_i^2
	 = \sum_{i=1}^{141} \left(y_i - \hat{y}_i\right)^2
\end{align}
Here there are 141 residuals, one for each observation. We typically use a computer to minimize the sum in Equation~\eqref{sumOfSqResInMultRegr} and compute point estimates, as shown in the sample output in Table~\ref{outputForMultipleRegrOutputForAllPredictors}. Using this output, we identify the point estimates $b_i$ of each $\beta_i$, just as we did in the one-predictor case.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & 36.2110 & 1.5140 & 23.92 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 5.1306 & 1.0511 & 4.88 & 0.0000 \\
  stock\_\hspace{0.3mm}photo & 1.0803 & 1.0568 & 1.02 & 0.3085 \\
  duration & -0.0268 & 0.1904 & -0.14 & 0.8882 \\
  wheels & 7.2852 & 0.5547 & 13.13 & 0.0000 \\
   \hline
   &&&\multicolumn{2}{r}{$df=136$}
\end{tabular}
\caption{Output for the regression model where \textbf{price} is the outcome and \textbf{cond\_\hspace{0.3mm}new}, \textbf{stock\_\hspace{0.3mm}photo}, \textbf{duration}, and \textbf{wheels} are the predictors.}
\label{outputForMultipleRegrOutputForAllPredictors}
\end{table}
%library(openintro); library(xtable); data(marioKart); d <- marioKart[marioKart$totalPr < 100,]; d$cond <- relevel(d$cond, "used"); g <-lm(totalPr ~ cond + stockPhoto + duration + wheels, d)

\begin{termBox}{\tBoxTitle{Multiple regression model}
A multiple regression model is a linear model with many predictors. In general, we write the model as
\begin{align*}
\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k %+ \epsilon
\end{align*}
when there are $k$ predictors. We often estimate the $\beta_i$ parameters using a computer.\vspace{1mm}}
\end{termBox}

\begin{exercise} \label{eqForMultipleRegrOfTotalPrForAllPredictorsWithCoefficients}
Write out the model in Equation~\eqref{eqForMultipleRegrOfTotalPrForAllPredictors} using the point estimates from Table~\ref{outputForMultipleRegrOutputForAllPredictors}. How many predictors are there in this model?\footnote{$\hat{y} = 36.21 + 5.13x_1 + 1.08x_2 - 0.03x_3 + 7.29x_4$, and there are $k=4$ predictor variables.}
\end{exercise}

\begin{exercise}
What does $\beta_4$, the coefficient of variable $x_4$ (Wii wheels), represent? What is the point estimate of $\beta_4$?\footnote{It is the average difference in auction price for each additional Wii wheel included when holding the other variables constant. The point estimate is $b_4 = 7.29$.}
\end{exercise}

\begin{exercise} \label{computeMultipleRegressionResidualForMarioKart}
Compute the residual of the first observation in Table~\vref{marioKartDataMatrix} using the equation identified in Guided Practice~\ref{eqForMultipleRegrOfTotalPrForAllPredictorsWithCoefficients}. \footnote{$e_i = y_i - \hat{y_i} = 51.55 - 49.62 = 1.93$, where 49.62 was computed using the variables values from the observation and the equation identified in Guided Practice~\ref{eqForMultipleRegrOfTotalPrForAllPredictorsWithCoefficients}.}
\end{exercise}

\begin{example}{We estimated a coefficient for \textbf{cond\_\hspace{0.3mm}new} in Section~\ref{twoSingleVariableModelsForMarioKartData} of $b_1 = 10.90$ with a standard error of $SE_{b_1} = 1.26$ when using simple linear regression. Why might there be a difference between that estimate and the one in the multiple regression setting?} \label{colinearityOfCondNewAndStockPhoto}
If we examined the data carefully, we would see that some predictors are correlated. For instance, when we estimated the connection of the outcome \textbf{price} and predictor \textbf{cond\_\hspace{0.3mm}new} using simple linear regression, we were unable to control for other variables like the number of Wii wheels included in the auction. That model was biased by the confounding variable \textbf{wheels}. When we use both variables, this particular underlying and unintentional bias is reduced or eliminated (though bias from other confounding variables may still remain).
\end{example}

Example~\ref{colinearityOfCondNewAndStockPhoto} describes a common issue in multiple regression: correlation among predictor variables. We say the two predictor variables are \textbf{collinear} (pronounced as \emph{co-linear}) when they are correlated, and this collinearity complicates model estimation. While it is impossible to prevent collinearity from arising in observational data, experiments are usually designed to prevent predictors from being collinear.

\begin{exercise}
The estimated value of the intercept is 36.21, and one might be tempted to make some interpretation of this coefficient, such as, it is the model's predicted price when each of the variables take value zero: the game is used, the primary image is not a stock photo, the auction duration is zero days, and there are no wheels included. Is there any value gained by making this interpretation?\footnote{Three of the variables (\textbf{cond\_\hspace{0.3mm}new}, \textbf{stock\_\hspace{0.3mm}photo}, and \textbf{wheels}) do take value 0, but the auction duration is always one or more days. If the auction is not up for any days, then no one can bid on it! That means the total auction price would always be zero for such an auction; the interpretation of the intercept in this setting is not insightful.}
\end{exercise}

\subsection{Adjusted $R^2$ as a better estimate of explained variance}

\index{adjusted r squared@adjusted $R^2$ ($R_{adj}^2$)|(}

We first used $R^2$ in Section~\ref{fittingALineByLSR} to determine the amount of variability in the response that was explained by the model:
\begin{align*}
R^2 = 1 - \frac{\text{variability in residuals}}{\text{variability in the outcome}}
	= 1 - \frac{Var(e_i)}{Var(y_i)}
\end{align*}
where $e_i$ represents the residuals of the model and $y_i$ the outcomes. This equation remains valid in the multiple regression framework, but a small enhancement can often be even more informative.

\begin{exercise} \label{computeUnadjustedR2ForAllPredictorsInMarioKart}
The variance of the residuals for the model given in Guided Practice~\ref{computeMultipleRegressionResidualForMarioKart} is 23.34, and the variance of the total price in all the auctions is 83.06. Calculate $R^2$ for this model.\footnote{$R^2 = 1 - \frac{23.34}{83.06} = 0.719$.}
\end{exercise}

This strategy for estimating $R^2$ is acceptable when there is just a single variable. However, it becomes less helpful when there are many variables. The regular $R^2$ is actually a biased estimate of the amount of variability explained by the model. To get a better estimate, we use the adjusted $R^2$.

\begin{termBox}{\tBoxTitle{Adjusted $\mathbf{R^2}$ as a tool for model assessment}
The \termsub{adjusted $\mathbf{R^2}$}{adjusted r squared@adjusted $R^2$ ($R_{adj}^2$)} is computed as
\begin{align*}
R_{adj}^{2} = 1-\frac{Var(e_i) / (n-k-1)}{Var(y_i) / (n-1)}
	= 1-\frac{Var(e_i)}{Var(y_i)} \times \frac{n-1}{n-k-1}
\end{align*}
where $n$ is the number of cases used to fit the model and $k$ is the number of predictor variables in the model.}
\end{termBox}

Because $k$ is never negative, the adjusted $R^2$ will be smaller -- often times just a little smaller -- than the unadjusted $R^2$. The reasoning behind the adjusted $R^2$ lies in the \termsub{degrees of freedom}{degrees of freedom (df)!regression} associated with each variance.\footnote{In multiple regression, the degrees of freedom associated with the variance of the estimate of the residuals is $n-k-1$, not $n-1$. For instance, if we were to make predictions for new data using our current model, we would find that the unadjusted $R^2$ is an overly optimistic estimate of the reduction in variance in the response, and using the degrees of freedom in the adjusted $R^2$ formula helps correct this bias.}

\begin{exercise}
There were $n=141$ auctions in the \textbf{mario\_\hspace{0.3mm}kart} data set and $k=4$ predictor variables in the model. Use $n$, $k$, and the variances from Guided Practice~\ref{computeUnadjustedR2ForAllPredictorsInMarioKart} to calculate $R_{adj}^2$ for the Mario Kart model.\footnote{$R_{adj}^2 = 1 - \frac{23.34}{83.06}\times \frac{141-1}{141-4-1} = 0.711$.}
\end{exercise}

\begin{exercise}
Suppose you added another predictor to the model, but the variance of the errors $Var(e_i)$ didn't go down. What would happen to the $R^2$? What would happen to the adjusted $R^2$?\hspace{0.7mm}\footnote{The unadjusted $R^2$ would stay the same and the adjusted $R^2$ would go down.}
\end{exercise}

\index{adjusted r squared@adjusted $R^2$ ($R_{adj}^2$)|)}

%__________________
\section{Model selection}
\label{modelSelection}

\index{model selection|(}

The best model is not always the most complicated. Sometimes including variables that are not evidently important can actually reduce the accuracy of predictions. In this section we discuss model selection strategies, which will help us eliminate from the model variables that are less important.

In this section, and in practice, the model that includes all available explanatory variables is often referred to as the \textbf{full model}. Our goal is to assess whether the full model is the best model. If it isn't, we want to identify a smaller model that is preferable.

\textA{\pagebreak}

\subsection{Identifying variables in the model that may not be helpful}

Table~\ref{outputForMultipleRegrOutputForAllPredictors2} provides a summary of the regression output for the full model for the auction data. The last column of the table lists p-values that can be used to assess hypotheses of the following form:
\begin{itemize}
\setlength{\itemsep}{0mm}
\item[$H_0$:] $\beta_i = 0$ when the other explanatory variables are included in the model.
\item[$H_A$:] $\beta_i \neq 0$ when the other explanatory variables are included in the model.
\end{itemize}

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & 36.2110 & 1.5140 & 23.92 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 5.1306 & 1.0511 & 4.88 & 0.0000 \\
  stock\_\hspace{0.3mm}photo & 1.0803 & 1.0568 & 1.02 & 0.3085 \\
  duration & -0.0268 & 0.1904 & -0.14 & 0.8882 \\
  wheels & 7.2852 & 0.5547 & 13.13 & 0.0000 \\
   \hline
  \vspace{-3.6mm} & & & & \\
\multicolumn{3}{l}{$R_{adj}^2 = 0.7108$}&\multicolumn{2}{r}{$df=136$}
\end{tabular}
\caption{The fit for the full regression model, including the adjusted $R^2$.}
\label{outputForMultipleRegrOutputForAllPredictors2}
\end{table}

\begin{example}{The coefficient of \textbf{cond\_\hspace{0.3mm}new} has a $t$ test statistic of $T=4.88$ and a p-value for its corresponding hypotheses ($H_0: \beta_1 = 0$, $H_A: \beta_1 \neq 0$) of about zero. How can this be interpreted?}
If we keep all the other variables in the model and add no others, then there is strong evidence that a game's condition (new or used) has a real relationship with the total auction price.
\end{example}

\begin{example}{Is there strong evidence that using a stock photo is related to the total auction price?}
The $t$ test statistic for \textbf{stock\_\hspace{0.3mm}photo} is $T=1.02$ and the p-value is about 0.31. After accounting for the other predictors, there is not strong evidence that using a stock photo in an auction is related to the total price of the auction. We might consider removing the \textbf{stock\_\hspace{0.3mm}photo} variable from the model.
\end{example}

\begin{exercise}
Identify the p-values for both the \textbf{duration} and \textbf{wheels} variables in the model. Is there strong evidence supporting the connection of these variables with the total price in the model?\footnote{The p-value for the auction duration is 0.8882, which indicates that there is not statistically significant evidence that the duration is related to the total auction price when accounting for the other variables. The p-value for the Wii wheels variable is about zero, indicating that this variable is associated with the total auction price.}
\end{exercise}

There is not statistically significant evidence that either the stock photo or duration variables contribute meaningfully to the model. Next we consider common strategies for pruning such variables from a model.

\begin{tipBox}{\tipBoxTitle{Using adjusted $R^2$ instead of p-values for model selection}
The adjusted $R^2$ may be used as an alternative to p-values for model selection, where a higher adjusted $R^2$ represents a better model fit. For instance, we could compare two models using their adjusted $R^2$, and the model with the higher adjusted $R^2$ would be preferred. This approach tends to include more variables in the final model when compared to the p-value approach.}
\end{tipBox}

\subsection{Two model selection strategies}

Two common strategies for adding or removing variables in a multiple regression model are called \emph{backward-selection} and \emph{forward-selection}. These techniques are often referred to as \textbf{stepwise} model selection strategies, because they add or delete one variable at a time as they ``step'' through the candidate predictors. We will discuss these strategies in the context of the p-value approach. Alternatively, we could have employed an $R_{adj}^2$ approach.

The \textbf{backward-elimination} strategy starts with the model that includes all potential predictor variables. Variables are eliminated one-at-a-time from the model until only variables with statistically significant p-values remain. The strategy within each elimination step is to drop the variable with the largest p-value, refit the model, and reassess the inclusion of all variables.

\begin{example}{Results corresponding to the \emph{full model} for the \textbf{mario\_\hspace{0.3mm}kart} data are shown in Table~\ref{outputForMultipleRegrOutputForAllPredictors2}. How should we proceed under the backward-elimination strategy?} \label{backwardEliminationExampleWMarioKartData}
There are two variables with coefficients that are not statistically different from zero: \textbf{stock\_\hspace{0.3mm}photo} and \textbf{duration}. We first drop the \textbf{duration} variable since it has a larger corresponding p-value, \emph{then we refit the model}. A regression summary for the new model is shown in Table~\ref{outputForMultipleRegrOutputForAllPredictorsButDuration}.

In the new model, there is not strong evidence that the coefficient for \textbf{stock\_\hspace{0.3mm}photo} is different from zero, even though the p-value decreased slightly, and the other p-values remain very small. Next, we again eliminate the variable with the largest non-significant p-value, \textbf{stock\_\hspace{0.3mm}photo}, and refit the model. The updated regression summary is shown in Table~\ref{outputForMultipleRegrOutputForAllPredictorsButDurationAndStockPhoto}.

In the latest model, we see that the two remaining predictors have statistically significant coefficients with p-values of about zero. Since there are no variables remaining that could be eliminated from the model, we stop. The final model includes only the \textbf{cond\_\hspace{0.3mm}new} and \textbf{wheels} variables in predicting the total auction price:
\begin{align*}
\hat{y} \ &= \ b_0 + b_1x_1 + b_4x_4 \\
	&= \ 36.78 + 5.58x_1 + 7.23x_4
\end{align*}
where $x_1$ represents \textbf{cond\_\hspace{0.3mm}new} and $x_4$ represents \textbf{wheels}.

An alternative to using p-values in model selection is to use the adjusted $R^2$. At each elimination step, we refit the model without each of the variables up for potential elimination. For example, in the first step, we would fit four models, where each would be missing a different predictor. If one of these smaller models has a higher adjusted $R^2$ than our current model, we pick the smaller model with the largest adjusted $R^2$. We continue in this way until removing variables does not increase~$R_{adj}^2$. Had we used the adjusted $R^2$ criteria, we would have kept the \textbf{stock\_\hspace{0.3mm}photo} variable along with the \textbf{cond\_\hspace{0.3mm}new} and \textbf{wheels} variables.
\end{example}

\begin{table}[t]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & 36.0483 & 0.9745 & 36.99 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 5.1763 & 0.9961 & 5.20 & 0.0000 \\
  stock\_\hspace{0.3mm}photo & 1.1177 & 1.0192 & 1.10 & 0.2747 \\
  wheels & 7.2984 & 0.5448 & 13.40 & 0.0000 \\
   \hline
  \vspace{-3.6mm} & & & & \\
\multicolumn{3}{l}{$R_{adj}^2 = 0.7128$}&&\small$df=137$
\end{tabular}
\caption{The output for the regression model where \textbf{price} is the outcome and the duration variable has been eliminated from the model.}
\label{outputForMultipleRegrOutputForAllPredictorsButDuration}
\end{table}

\begin{table}[t]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & t value & Pr($>$$|$t$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & 36.7849 & 0.7066 & 52.06 & 0.0000 \\
  cond\_\hspace{0.3mm}new & 5.5848 & 0.9245 & 6.04 & 0.0000 \\
  wheels & 7.2328 & 0.5419 & 13.35 & 0.0000 \\
   \hline
  \vspace{-3.6mm} & & & & \\
\multicolumn{3}{l}{$R_{adj}^2 = 0.7124$}&&\small$df=138$
\end{tabular}
\caption{The output for the regression model where \textbf{price} is the outcome and the duration and stock photo variables have been eliminated from the model.}
\label{outputForMultipleRegrOutputForAllPredictorsButDurationAndStockPhoto}
\end{table}

Notice that the p-value for \textbf{stock\_\hspace{0.3mm}photo} changed a little from the full model (0.309) to the model that did not include the \textbf{duration} variable (0.275). It is common for p-values of one variable to change, due to collinearity, after eliminating a different variable. This fluctuation emphasizes the importance of refitting a model after each variable elimination step. The p-values tend to change dramatically when the eliminated variable is highly correlated with another variable in the model.

The \textbf{forward-selection} strategy is the reverse of the backward-elimination technique. Instead of eliminating variables one-at-a-time, we add variables one-at-a-time until we cannot find any variables that present strong evidence of their importance in the model.

\begin{example}{Construct a model for the \textbf{mario\_\hspace{0.3mm}kart} data set using the forward-selection strategy.}\label{forwardEliminationExampleWMarioKartData}
We start with the model that includes no variables. Then we fit each of the possible models with just one variable. That is, we fit the model including just the \textbf{cond\_\hspace{0.3mm}new} predictor, then the model including just the \textbf{stock\_\hspace{0.3mm}photo} variable, then a model with just \textbf{duration}, and a model with just \textbf{wheels}. Each of the four models (yes, we fit four models!) provides a p-value for the coefficient of the predictor variable. Out of these four variables, the \textbf{wheels} variable had the smallest p-value. Since its p-value is less than 0.05 (the p-value was smaller than 2e-16), we add the Wii wheels variable to the model. Once a variable is added in forward-selection, it will be included in all models considered as well as the final model.

Since we successfully found a first variable to add, we consider adding another. We fit three new models: (1) the model including just the \textbf{cond\_\hspace{0.3mm}new} and \textbf{wheels} variables (output in Table~\ref{outputForMultipleRegrOutputForAllPredictorsButDurationAndStockPhoto}), (2) the model including just the \textbf{stock\_\hspace{0.3mm}photo} and \textbf{wheels} variables, and (3) the model including only the \textbf{duration} and \textbf{wheels} variables. Of these models, the first had the lowest p-value for its new variable (the p-value corresponding to \textbf{cond\_\hspace{0.3mm}new} was 1.4e-08). Because this p-value is below 0.05, we add the \textbf{cond\_\hspace{0.3mm}new} variable to the model. Now the final model is guaranteed to include both the condition and wheels variables.

We must then repeat the process a third time, fitting two new models: (1) the model including the \textbf{stock\_\hspace{0.3mm}photo}, \textbf{cond\_\hspace{0.3mm}new}, and \textbf{wheels} variables (output in Table~\ref{outputForMultipleRegrOutputForAllPredictorsButDuration}) and (2) the model including the \textbf{duration}, \textbf{cond\_\hspace{0.3mm}new}, and \textbf{wheels} variables. The p-value corresponding to \textbf{stock\_\hspace{0.3mm}photo} in the first model (0.275) was smaller than the p-value corresponding to \textbf{duration} in the second model (0.682). However, since this smaller p-value was not below 0.05, there was not strong evidence that it should be included in the model. Therefore, neither variable is added and we are finished.

The final model is the same as that arrived at using the backward-selection strategy.
\end{example}

\begin{example}{As before, we could have used the $R_{adj}^2$ criteria instead of examining p-values in selecting variables for the model. Rather than look for variables with the smallest p-value, we look for the model with the largest $R_{adj}^2$. What would the result of forward-selection be using the adjusted $R^2$ approach?}
Using the forward-selection strategy, we start with the model with no predictors. Next we look at each model with a single predictor. If one of these models has a larger $R_{adj}^2$ than the model with no variables, we use this new model. We repeat this procedure, adding one variable at a time, until we cannot find a model with a larger $R_{adj}^2$. If we had done the forward-selection strategy using $R_{adj}^2$, we would have arrived at the model including \textbf{cond\_\hspace{0.3mm}new}, \textbf{stock\_\hspace{0.3mm}photo}, and \textbf{wheels}, which is a slightly larger model than we arrived at using the p-value approach and the same model we arrived at using the adjusted $R^2$ and backwards-elimination.
\end{example}

\begin{termBox}{\tBoxTitle{Model selection strategies}
The backward-elimination strategy begins with the largest model and eliminates variables one-by-one until we are satisfied that all remaining variables are important to the model. The forward-selection strategy starts with no variables included in the model, then it adds in variables according to their importance until no other important variables are found.}
\end{termBox}

There is no guarantee that the backward-elimination and forward-selection strategies will arrive at the same final model using the p-value or adjusted $R^2$ methods. If the backwards-elimination and forward-selection strategies are both tried and they arrive at different models, choose the model with the larger $R_{adj}^2$ as a tie-breaker; other tie-break options exist but are beyond the scope of this book.

It is generally acceptable to use just one strategy, usually backward-elimination with either the p-value or adjusted $R^2$ criteria. However, before reporting the model results, we must verify the model conditions are reasonable.

\index{model selection|)}

%%%%%
\textA{\newpage}
\section{Checking model assumptions using graphs}
\label{multipleRegressionModelAssumptions}

\index{multiple regression!model assumptions|(}

Multiple regression methods using the model
\begin{align*}
\hat{y} &= \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_kx_k
\end{align*}
generally depend on the following four assumptions:
\begin{enumerate}
\setlength{\itemsep}{0mm}
\item the residuals of the model are nearly normal,
\item the variability of the residuals is nearly constant,
\item the residuals are independent, and
\item each variable is linearly related to the outcome.
\end{enumerate}
Simple and effective plots can be used to check each of these assumptions. We will consider the model for the auction data that uses the game condition and number of wheels as predictors. The plotting methods presented here may also be used to check the conditions for the models introduced in Chapter~\ref{linRegrForTwoVar}.

\begin{description}
\item[Normal probability plot.] A normal probability plot of the residuals is shown in Figure~\ref{mkDiagnosticNormalQuantilePlot}. While the plot exhibits some minor irregularities, there are no outliers that might be cause for concern. In a normal probability plot for residuals, we tend to be most worried about residuals that appear to be outliers, since these indicate long tails in the distribution of residuals.

\begin{figure}
\centering
\includegraphics[width=0.71\textwidth]{06/figures/marioKartDiagnostics/mkDiagnosticNormalQuantilePlot}
\caption{A normal probability plot of the residuals is helpful in identifying observations that might be outliers.}
\label{mkDiagnosticNormalQuantilePlot}
\end{figure}

\item[Absolute values of residuals against fitted values.] A plot of the absolute value of the residuals against their corresponding fitted values ($\hat{y}_i$) is shown in Figure~\ref{mkDiagnosticEvsAbsF}. This plot is helpful to check the condition that the variance of the residuals is approximately constant. We don't see any obvious deviations from constant variance in this example.

\begin{figure}
\centering
\includegraphics[width=0.72\textwidth]{06/figures/marioKartDiagnostics/mkDiagnosticEvsAbsF}
\caption{Comparing the absolute value of the residuals against the fitted values ($\hat{y}_i$) is helpful in identifying deviations from the constant variance assumption.}
\label{mkDiagnosticEvsAbsF}
\end{figure}

\item[Residuals in order of their data collection.] A plot of the residuals in the order their corresponding auctions were observed is shown in Figure~\ref{mkDiagnosticInOrder}. Such a plot is helpful in identifying any connection between cases that are close to one another, e.g. we could look for declining prices over time or if there was a time of the day when auctions tended to fetch a higher price. Here we see no structure that indicates a problem.\footnote{An especially rigorous check would use \textbf{time series} methods. For instance, we could check whether consecutive residuals are correlated. Doing so with these residuals yields no statistically significant correlations.}

\begin{figure}
\centering
\includegraphics[width=0.72\textwidth]{06/figures/marioKartDiagnostics/mkDiagnosticInOrder}
\caption{Plotting residuals in the order that their corresponding observations were collected helps identify connections between successive observations. If it seems that consecutive observations tend to be close to each other, this indicates the independence assumption of the observations would fail.}
\label{mkDiagnosticInOrder}
\end{figure}

\item[Residuals against each predictor variable.] We consider a plot of the residuals against the \textbf{cond\_\hspace{0.3mm}new} variable and the residuals against the \textbf{wheels} variable. These plots are shown in Figure~\ref{mkDiagnosticEvsVariables}. For the two-level condition variable, we are guaranteed not to see any remaining trend, and instead we are checking that the variability doesn't fluctuate across groups. In this example, when we consider the residuals against the \textbf{wheels} variable, we see some possible structure. There appears to be curvature in the residuals, indicating the relationship is probably not linear.

\begin{figure}
\centering
\includegraphics[width=0.85\textwidth]{06/figures/marioKartDiagnostics/mkDiagnosticEvsVariables}
\caption{In the two-level variable for the game's condition, we check for differences in distribution shape or variability. For numerical predictors, we also check for trends or other structure. We see some slight bowing in the residuals against the \textbf{wheels} variable.}
\label{mkDiagnosticEvsVariables}
\end{figure}

\end{description}

It is necessary to summarize diagnostics for any model fit. If the diagnostics support the model assumptions, this would improve credibility in the findings. If the diagnostic assessment shows remaining underlying structure in the residuals, we should try to adjust the model to account for that structure. If we are unable to do so, we may still report the model but must also note its shortcomings. In the case of the auction data, we report that there may be a nonlinear relationship between the total price and the number of wheels included for an auction. This information would be important to buyers and sellers; omitting this information could be a setback to the very people who the model might assist.

\begin{tipBox}{\tipBoxTitle[]{``All models are wrong, but some are useful'' -George E.P. Box}
The truth is that no model is perfect. However, even imperfect models can be useful. Reporting a flawed model can be reasonable so long as we are clear and report the model's shortcomings.}
\end{tipBox}

\begin{caution}
{Don't report results when assumptions are grossly violated}
{While there is a little leeway in model assumptions, don't go too far. If model assumptions are very clearly violated, consider a new model, even if it means learning more statistical methods or hiring someone who can help.}
\end{caution}

\begin{tipBox}{\tipBoxTitle{Confidence intervals in multiple regression}
\index{confidence interval}
Confidence intervals for coefficients in multiple regression can be computed using the same formula as in the single predictor model:
\begin{align*}
b_i \ \pm\ t_{df}^{\star}SE_{b_{i}}
\end{align*}
where $t_{df}^{\star}$ is the appropriate $t$ value corresponding to the confidence level and model degrees of freedom, $df=n-k-1$.}
\index{multiple regression!model assumptions|)}
\index{data!mario\_kart|)}
\index{multiple regression|)}
\end{tipBox}

%__________________
\section{Logistic regression}
\label{logisticRegression}

\index{logistic regression|(}

In this section we introduce \textbf{logistic regression} as a tool for building models when there is a categorical response variable with two levels. Logistic regression is a type of \textbf{generalized linear model} (GLM) for response variables where regular multiple regression does not work very well. In particular, the response variable in these settings often takes a form where residuals look completely different from the normal distribution.

GLMs can be thought of as a two-stage modeling approach. We first model the response variable using a probability distribution, such as the binomial or Poisson distribution. Second, we model the parameter of the distribution using a collection of predictors and a special form of multiple regression.

\index{data!email|(}

In Section~\ref{logisticRegression} we will revisit the \textbf{email} data set from Chapter~\ref{introductionToData}. These emails were collected from a single email account, and we will work on developing a basic spam filter using these data. The response variable, \textbf{spam}, has been encoded to take value~0 when a message is not spam and~1 when it is spam. Our task will be to build an appropriate model that classifies messages as spam or not spam using email characteristics coded as predictor variables. While this model will not be the same as those used in large-scale spam filters, it shares many of the same features.

\textA{\newpage}

\subsection{Email data}

The \textbf{email} data set was first presented in Chapter~\ref{introductionToData} with a relatively small number of variables. In fact, there are many more variables available that might be useful for classifying spam. Descriptions of these variables are presented in Table~\ref{emailVariables}. The \textbf{spam} variable will be the outcome, and the other 10 variables will be the model predictors. While we have limited the predictors used in this section to be categorical variables (where many are represented as indicator variables), numerical predictors may also be used in logistic regression. See the footnote for an additional discussion on this topic.\footnote{Recall from Chapter~\ref{linRegrForTwoVar} that if outliers are present in predictor variables, the corresponding observations may be especially influential on the resulting model. This is the motivation for omitting the numerical variables, such as the number of characters and line breaks in emails, that we saw in Chapter~\ref{introductionToData}. These variables exhibited extreme skew. We could resolve this issue by transforming these variables (e.g. using a log-transformation), but we will omit this further investigation for brevity.}

\begin{table}
\centering\small
\begin{tabular}{lp{10.5cm}}
\hline
{\bf variable} & {\bf description} \\
\hline
\textbf{spam} & Specifies whether the message was spam. \\
\textbf{to\_\hspace{0.3mm}multiple} & An indicator variable for if more than one person was listed in the \emph{To} field of the email.  \\
\textbf{cc} & An indicator for if someone was CCed on the email. \\
\textbf{attach} & An indicator for if there was an attachment, such as a document or image.   \\
\textbf{dollar} & An indicator for if the word ``dollar'' or dollar symbol (\$) appeared in the email.  \\
\textbf{winner} & An indicator for if the word ``winner'' appeared in the email message.  \\
\textbf{inherit} & An indicator for if the word ``inherit'' (or a variation, like ``inheritance'') appeared in the email.  \\
\textbf{password} & An indicator for if the word ``password'' was present in the email.  \\
\textbf{format} & Indicates if the email contained special formatting, such as bolding, tables, or links    \\
\textbf{re\_\hspace{0.3mm}subj} & Indicates whether ``Re:'' was included at the start of the email subject.   \\
\textbf{exclaim\_\hspace{0.3mm}subj} & Indicates whether any exclamation point was included in the email subject.    \\
\hline
\end{tabular}
\caption{Descriptions for 11 variables in the \textbf{email} data set. Notice that all of the variables are indicator\index{indicator variable} variables, which take the value 1 if the specified characteristic is present and 0 otherwise.}
\label{emailVariables}
\end{table}

\subsection{Modeling the probability of an event}
\label{modelingTheProbabilityOfAnEvent}

\begin{tipBox}{\tipBoxTitle{Notation for a logistic regression model}
The outcome variable for a GLM is denoted by $Y_i$, where the index $i$ is used to represent observation $i$. In the email application, $Y_i$ will be used to represent whether email $i$ is spam ($Y_i=1$) or not ($Y_i=0$). \vspace{3mm}

The predictor variables are represented as follows: $x_{1,i}$ is the value of variable 1 for observation $i$, $x_{2,i}$ is the value of variable 2 for observation $i$, and so on.}
\end{tipBox}

Logistic regression is a generalized linear model where the outcome is a two-level categorical variable. The outcome, $Y_i$, takes the value 1 (in our application, this represents a spam message) with probability $p_i$ and the value 0 with probability $1-p_i$. It~is the probability $p_i$ that we model in relation to the predictor variables.

The logistic regression model relates the probability an email is spam ($p_i$) to the predictors $x_{1,i}$, $x_{2,i}$, ..., $x_{k,i}$ through a framework much like that of multiple regression:
\begin{align}
transformation(p_{i}) = \beta_0 + \beta_1x_{1,i} + \beta_2 x_{2,i} + \cdots \beta_k x_{k,i}
\label{linkTransformationEquation}
\end{align}
We want to choose a transformation in Equation~\eqref{linkTransformationEquation} that makes practical and mathematical sense. For example, we want a transformation that makes the range of possibilities on the left hand side of Equation~\eqref{linkTransformationEquation} equal to the range of possibilities for the right hand side; if there was no transformation for this equation, the left hand side could only take values between 0 and 1, but the right hand side could take values outside of this range. A common transformation for $p_i$ is the \textbf{logit transformation}, which may be written as
\begin{align*}
logit(p_i) = \log_{e}\left( \frac{p_i}{1-p_i} \right)
\end{align*}
The logit transformation is shown in Figure~\ref{logitTransformationFigureHoriz}. Below, we rewrite Equation~\eqref{linkTransformationEquation} using the logit transformation of $p_i$:
\begin{align*}
\log_{e}\left( \frac{p_i}{1-p_i} \right)
	= \beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i} + \cdots + \beta_k x_{k,i}
\end{align*}
In our spam example, there are 10 predictor variables, so $k = 10$. This model isn't very intuitive, but it still has some resemblance to multiple regression, and we can fit this model using software. In fact, once we look at results from software, it will start to feel like we're back in multiple regression, even if the interpretation of the coefficients is more complex.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{06/figures/logitTransformationFigureHoriz/logitTransformationFigureHoriz}
\caption{Values of $p_i$ against values of $logit(p_i)$.}
\label{logitTransformationFigureHoriz}
\end{figure}

\textA{\pagebreak}

\begin{example}{Here we create a spam filter with a single predictor: \textbf{to\_\hspace{0.3mm}multiple}. This variable indicates whether more than one email address was listed in the \emph{To} field of the email. The following logistic regression model was fit using statistical software:
\begin{align*}
\log\left( \frac{p_i}{1-p_i} \right) = -2.12 - 1.81\times\text{\textbf{to\_\hspace{0.3mm}multiple}}
\end{align*}
%library(openintro); data(email); tm <- email$to_multiple > 0; g <- glm(email$spam ~ tm, family=binomial); summary(g); a <- exp(-2.12); a/(1+a); a <- exp(-2.12-1.81); a/(1+a)
If an email is randomly selected and it has just one address in the \emph{To} field, what is the probability it is spam? What if more than one address is listed in the \emph{To} field?}\label{logisticExampleWithToMultiple}
If there is only one email in the \emph{To} field, then \textbf{to\_\hspace{0.3mm}multiple} takes value 0 and the right side of the model equation equals -2.12. Solving for $p_i$: $\frac{e^{-2.12}}{1 + e^{-2.12}} = 0.11$. Just as we labeled a fitted value of $y_i$ with a ``hat'' in single-variable and multiple regression, we will do the same for this probability: $\hat{p}_i = 0.11$.

If there is more than one address listed in the \emph{To} field, then the right side of the model equation is $-2.12 - 1.81\times1 = -3.93$, which corresponds to a probability $\hat{p}_i = 0.02$.

Notice that we could examine -2.12 and -3.93 in Figure~\ref{logitTransformationFigureHoriz} to estimate the probability before formally calculating the value.
\end{example}

To convert from values on the regression-scale (e.g. -2.12 and -3.93 in Example~\ref{logisticExampleWithToMultiple}), use the following formula, which is the result of solving for $p_i$ in the regression model:
\begin{align*}
p_i
	= \frac{e^{\beta_0 + \beta_1 x_{1,i}+\cdots+\beta_k x_{k,i}}}
		{\ 1\ \ +\ \ e^{\beta_0 + \beta_1 x_{1,i}+\cdots+\beta_k x_{k,i}}\ }
\end{align*}
As with most applied data problems, we substitute the point estimates for the parameters (the $\beta_i$) so that we may make use of this formula. In Example~\ref{logisticExampleWithToMultiple}, the probabilities were calculated as
\begin{align*}
&\frac{\ e^{-2.12}\ }{\ 1\ +\ e^{-2.12}\ } = 0.11 && \frac{\ e^{-2.12 - 1.81}\ }{\ 1\ +\ e^{-2.12 - 1.81}\ } = 0.02
\end{align*}
While the information about whether the email is addressed to multiple people is a helpful start in classifying email as spam or not, the probabilities of 11\% and 2\% are not dramatically different, and neither provides very strong evidence about which particular email messages are spam. To get more precise estimates, we'll need to include many more variables in the model.

We used statistical software to fit the logistic regression model with all ten predictors described in Table~\ref{emailVariables}. Like multiple regression, the result may be presented in a summary table, which is shown in Table~\ref{emailLogisticModelResults}. The structure of this table is almost identical to that of multiple regression; the only notable difference is that the p-values are calculated using the normal distribution rather than the $t$ distribution.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & z value & Pr($>$$|$z$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & -0.8362 & 0.0962 & -8.69 & 0.0000 \\
  to\_\hspace{0.3mm}multiple & -2.8836 & 0.3121 & -9.24 & 0.0000 \\
  winner & 1.7038 & 0.3254 & 5.24 & 0.0000 \\
  format & -1.5902 & 0.1239 & -12.84 & 0.0000 \\
  re\_\hspace{0.3mm}subj & -2.9082 & 0.3708 & -7.84 & 0.0000 \\
  exclaim\_\hspace{0.3mm}subj & 0.1355 & 0.2268 & 0.60 & 0.5503 \\
  cc & -0.4863 & 0.3054 & -1.59 & 0.1113 \\
  attach & 0.9790 & 0.2170 & 4.51 & 0.0000 \\
  dollar & -0.0582 & 0.1589 & -0.37 & 0.7144 \\
  inherit & 0.2093 & 0.3197 & 0.65 & 0.5127 \\
  password & -1.4929 & 0.5295 & -2.82 & 0.0048 \\
\hline
\end{tabular}
\caption{Summary table for the full logistic regression model for the spam filter example.}
\label{emailLogisticModelResults}
\end{table}
% library(openintro); ?email # run example
% library(xtable); xtable(g)

Just like multiple regression, we could trim some variables from the model using the p-value. Using backwards elimination with a p-value cutoff of 0.05 (start with the full model and trim the predictors with p-values greater than 0.05), we ultimately eliminate the \textbf{exclaim\_\hspace{0.3mm}subj}, \textbf{dollar}, \textbf{inherit}, and \textbf{cc} predictors. The remainder of this section will rely on this smaller model, which is summarized in Table~\ref{emailLogisticReducedModel}.

\begin{table}[ht]
\centering
\begin{tabular}{rrrrr}
  \hline
  \vspace{-3.7mm} & & & & \\
 & Estimate & Std. Error & z value & Pr($>$$|$z$|$) \\
  \hline
  \vspace{-3.8mm} & & & & \\
(Intercept) & -0.8595 & 0.0910 & -9.44 & 0.0000 \\
  to\_\hspace{0.3mm}multiple & -2.8372 & 0.3092 & -9.18 & 0.0000 \\
  winner & 1.7370 & 0.3218 & 5.40 & 0.0000 \\
  format & -1.5569 & 0.1207 & -12.90 & 0.0000 \\
  re\_\hspace{0.3mm}subj & -3.0482 & 0.3630 & -8.40 & 0.0000 \\
  attach & 0.8643 & 0.2042 & 4.23 & 0.0000 \\
  password & -1.4871 & 0.5290 & -2.81 & 0.0049 \\
\hline
\end{tabular}
\caption{Summary table for the logistic regression model for the spam filter, where variable selection has been performed.}
\label{emailLogisticReducedModel}
\end{table}
% library(openintro); ?email # run example
% g <- glm(spam ~ to_multiple + winner + format + re_subj + attach + password, data=e, family=binomial); xtable(g)

\begin{exercise}
Examine the summary of the reduced model in Table~\ref{emailLogisticReducedModel}, and in particular, examine the \textbf{to\_\hspace{0.3mm}multiple} row. Is the point estimate the same as we found before, -1.81, or is it different? Explain why this might be.\footnote{The new estimate is different: -2.87. This new value represents the estimated coefficient when we are also accounting for other variables in the logistic regression model.}
\end{exercise}

Point estimates will generally change a little -- and sometimes a lot -- depending on which other variables are included in the model. This is usually due to colinearity in the predictor variables. We previously saw this in the Ebay auction example when we compared the coefficient of \textbf{cond\_\hspace{0.3mm}new} in a single-variable model and the corresponding coefficient in the multiple regression model that used three additional variables (see Sections~\ref{twoSingleVariableModelsForMarioKartData} and~\ref{includingAndAssessingManyVariablesInAModel}).

\begin{example}{Spam filters are built to be automated, meaning a piece of software is written to collect information about emails as they arrive, and this information is put in the form of variables. These variables are then put into an algorithm that uses a statistical model, like the one we've fit, to classify the email. Suppose we write software for a spam filter using the reduced model shown in Table~\ref{emailLogisticReducedModel}. If an incoming email has the word ``winner'' in it, will this raise or lower the model's calculated probability that the incoming email is spam?} \label{exampleForSpamAndWinner}
The estimated coefficient of \textbf{winner} is positive (1.7370). A positive coefficient estimate in logistic regression, just like in multiple regression, corresponds to a positive association between the predictor and response variables when accounting for the other variables in the model. Since the response variable takes value 1 if an email is spam and 0 otherwise, the positive coefficient indicates that the presence of ``winner'' in an email raises the model probability that the message is spam.
\end{example}

\begin{example}{Suppose the same email from Example~\ref{exampleForSpamAndWinner} was in HTML format, meaning the \textbf{format} variable took value 1. Does this characteristic increase or decrease the probability that the email is spam according to the model?}\label{exampleForSpamAndFormat}
Since HTML corresponds to a value of 1 in the \textbf{format} variable and the coefficient of this variable is negative (-1.5569), this would lower the probability estimate returned from the model.
\end{example}

\subsection{Practical decisions in the email application}

Examples~\ref{exampleForSpamAndWinner} and~\ref{exampleForSpamAndFormat} highlight a key feature of logistic and multiple regression. In the spam filter example, some email characteristics will push an email's classification in the direction of spam while other characteristics will push it in the opposite direction.

If we were to implement a spam filter using the model we have fit, then each future email we analyze would fall into one of three categories based on the email's characteristics:
\begin{enumerate}
\item The email characteristics generally indicate the email is not spam, and so the resulting probability that the email is spam is quite low, say, under 0.05.
\item The characteristics generally indicate the email is spam, and so the resulting probability that the email is spam is quite large, say, over 0.95.
\item The characteristics roughly balance each other out in terms of evidence for and against the message being classified as spam. Its probability falls in the remaining range, meaning the email cannot be adequately classified as spam or not spam.
\end{enumerate}
If we were managing an email service, we would have to think about what should be done in each of these three instances. In an email application, there are usually just two possibilities: filter the email out from the regular inbox and put it in a ``spambox'', or let the email go to the regular inbox.

\begin{exercise}
The first and second scenarios are intuitive. If the evidence strongly suggests a message is not spam, send it to the inbox. If the evidence strongly suggests the message is spam, send it to the spambox. How should we handle emails in the third category?\footnote{In this particular application, we should err on the side of sending more mail to the inbox rather than mistakenly putting good messages in the spambox. So, in summary: emails in the first and last categories go to the regular inbox, and those in the second scenario go to the spambox.}
\end{exercise}

\begin{exercise}
Suppose we apply the logistic model we have built as a spam filter and that 100 messages are placed in the spambox over 3 months. If we used the guidelines above for putting messages into the spambox, about how many legitimate (non-spam) messages would you expect to find among the 100 messages?\footnote{First, note that we proposed a cutoff for the predicted probability of 0.95 for spam. In a worst case scenario, all the messages in the spambox had the minimum probability equal to about 0.95. Thus, we should expect to find about 5 or fewer legitimate messages among the 100 messages placed in the spambox.}
\end{exercise}

Almost any classifier will have some error. In the spam filter guidelines above, we have decided that it is okay to allow up to 5\% of the messages in the spambox to be real messages. If we wanted to make it a little harder to classify messages as spam, we could use a cutoff of 0.99. This would have two effects. Because it raises the standard for what can be classified as spam, it reduces the number of good emails that are classified as spam. However, it will also fail to correctly classify an increased fraction of spam messages. No matter the complexity and the confidence we might have in our model, these practical considerations are absolutely crucial to making a helpful spam filter. Without them, we could actually do more harm than good by using our statistical model.

\subsection{Diagnostics for the email classifier}

\begin{termBox}{\tBoxTitle{Logistic regression conditions}
There are two key conditions for fitting a logistic regression model: \vspace{-1.5mm}
\begin{enumerate}
\setlength{\itemsep}{0mm}
\item Each predictor $x_i$ is linearly related to logit$(p_i)$ if all other predictors are held constant.
%\item The model relating the parameter $p_i$ to the predictors $x_{1,i}$, $x_{2,i}$, ..., $x_{k,i}$ closely resembles the true relationship between the parameter and the predictors.
\item Each outcome $Y_i$ is independent of the other outcomes.
\end{enumerate}}
\end{termBox}

The first condition of the logistic regression model is not easily checked without a fairly sizable amount of data. Luckily, we have 3,921 emails in our data set! Let's first visualize these data by plotting the true classification of the emails against the model's fitted probabilities, as shown in Figure~\ref{logisticModelPredict}. The vast majority of emails (spam or not) still have fitted probabilities below 0.5.

\begin{figure}[h]
\centering
\includegraphics[width=0.95\textwidth]{06/figures/logisticModel/logisticModelPredict}
\caption{The predicted probability that each of the 3,912 emails is spam is classified by their grouping, spam or not. Noise (small, random vertical shifts) have been added to each point so that points with nearly identical values aren't plotted exactly on top of one another. This makes it possible to see more observations.}
\label{logisticModelPredict}
\end{figure}

This may at first seem very discouraging: we have fit a logistic model to create a spam filter, but no emails have a fitted probability of being spam above 0.75. Don't despair; we will discuss ways to improve the model through the use of better variables in Section~\ref{improvingTheSetOfVariablesForASpamFilter}.

We'd like to assess the quality of our model. For example, we might ask: if we look at emails that we modeled as having a 10\% chance of being spam, do we find about 10\% of them actually are spam? To help us out, we'll borrow an advanced statistical method called \textbf{natural splines} that estimates the local probability over the region 0.00 to 0.75 (the largest predicted probability was 0.73, so we avoid extrapolating). All you need to know about natural splines to understand what we are doing is that they are used to fit flexible lines rather than straight lines.

The curve fit using natural splines is shown in Figure~\ref{logisticModelSpline} as a solid black line. If the logistic model fits well, the curve should closely follow the dashed $y=x$ line. We have added shading to represent the confidence bound for the curved line to clarify what fluctuations might plausibly be due to chance. Even with this confidence bound, there are weaknesses in the first model assumption. The solid curve and its confidence bound dips below the dashed line from about 0.1 to 0.3, and then it drifts above the dashed line from about 0.35 to 0.55. These deviations indicate the model relating the parameter to the predictors does not closely resemble the true relationship.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{06/figures/logisticModel/logisticModelSpline}
\caption{The solid black line provides the empirical estimate of the probability for observations based on their predicted probabilities (confidence bounds are also shown for this line), which is fit using natural splines. A small amount of noise was added to the observations in the plot to allow more observations to be seen.}
\label{logisticModelSpline}
\end{figure}

We could evaluate the second logistic regression model assumption -- independence of the outcomes -- using the model residuals. The residuals for a logistic regression model are calculated the same way as with multiple regression: the observed outcome minus the expected outcome. For logistic regression, the expected value of the outcome is the fitted probability for the observation, and the residual may be written as
\begin{align*}
e_i = Y_i - \hat{p}_i
\end{align*}
We could plot these residuals against a variety of variables or in their order of collection, as we did with the residuals in multiple regression. However, since the model will need to be revised to effectively classify spam and you have already seen similar residual plots in Section~\ref{multipleRegressionModelAssumptions}, we won't investigate the residuals here.

\subsection{Improving the set of variables for a spam filter}
\label{improvingTheSetOfVariablesForASpamFilter}

If we were building a spam filter for an email service that managed many accounts (e.g. Gmail or Hotmail), we would spend much more time thinking about additional variables that could be useful in classifying emails as spam or not. We also would use transformations or other techniques that would help us include strongly skewed numerical variables as predictors.

Take a few minutes to think about additional variables that might be useful in identifying spam. Below is a list of variables we think might be useful:
\begin{enumerate}
\item[(1)] An indicator variable could be used to represent whether there was prior two-way correspondence with a message's sender. For instance, if you sent a message to john@example.com and then John sent you an email, this variable would take value 1 for the email that John sent. If you had never sent John an email, then the variable would be set to~0.
\item[(2)] A second indicator variable could utilize an account's past spam flagging information. The variable could take value 1 if the sender of the message has previously sent messages flagged as~spam.
\item[(3)] A third indicator variable could flag emails that contain links included in previous spam messages. If such a link is found, then set the variable to~1 for the email. Otherwise, set it to~0.
\end{enumerate}
The variables described above take one of two approaches. Variable (1) is specially designed to capitalize on the fact that spam is rarely sent between individuals that have two-way communication. Variables~(2) and~(3) are specially designed to flag common spammers or spam messages. While we would have to verify using the data that each of the variables is effective, these seem like promising ideas.

Table~\ref{emailTableOfSpamAnd} shows a contingency table for spam and also for the new variable described in~(1) above. If we look at the 1,090 emails where there was correspondence with the sender in the preceding 30 days, not one of these message was spam. This suggests variable~(1) would be very effective at accurately classifying some messages as not spam. With this single variable, we would be able to send about 28\% of messages through to the inbox with confidence that almost none are spam.

\begin{table}[h]
\centering
\begin{tabular}{lrrrr}
& \multicolumn{3}{c}{prior correspondence} & \\
\cline{2-4}
 & \ \hspace{5mm}\resp{no} & \ \hspace{8mm}\resp{yes} && \ \hspace{3mm}Total \\
\hline
\vspace{-3.8mm} & & & & \\
\resp{spam} &  367 &    0 &&  367 \\
\resp{not spam}\hspace{3mm}\  & 2464 & 1090 && 3554 \\
\hline
\vspace{-3.8mm} & & & & \\
Total & 2831 & 1090 && 3921 \\
\hline
\end{tabular}
\caption{A contingency table for \textbf{spam} and a new variable that represents whether there had been correspondence with the sender in the preceding 30~days.}
\label{emailTableOfSpamAnd}
\end{table}
%library(openintro); data(email); contTable(email[,c("spam", "sent_email")])

The variables described in (2) and (3) would provide an excellent foundation for distinguishing messages coming from known spammers or messages that take a known form of spam. To utilize these variables, we would need to build databases: one holding email addresses of known spammers, and one holding URLs found in known spam messages. Our access to such information is limited, so we cannot implement these two variables in this textbook. However, if we were hired by an email service to build a spam filter, these would be important next steps.

In addition to finding more and better predictors, we would need to create a customized logistic regression model for each email account. This may sound like an intimidating task, but its complexity is not as daunting as it may at first seem. We'll save the details for a statistics course where computer programming plays a more central role.

For what is the extremely challenging task of classifying spam messages, we have made a lot of progress. We have seen that simple email variables, such as the format, inclusion of certain words, and other circumstantial characteristics, provide helpful information for spam classification. Many challenges remain, from better understanding logistic regression to carrying out the necessary computer programming, but completing such a task is very nearly within your reach.

\index{data!email|)}
\index{logistic regression|)}


\documentclass[11pt,a4paper]{article}

% These are extra packages that you might need for writing the equations:
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{xcolor}
\lstset {language=C++,
		 basicstyle=\ttfamily,
         keywordstyle=\color{blue}\ttfamily,
         stringstyle=\color{red}\ttfamily,
         commentstyle=\color{purple}\ttfamily,
         morecomment=[l][\color{magenta}]{\#},
       	 basicstyle=\tiny}

% You need the following package in order to include figures in your report:
\usepackage{graphicx}

% With this package you can set the size of the margins manually:
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}

\begin{document}

% Enter the exercise number, your name and date here:
\noindent\parbox{\linewidth}{
 \parbox{.25\linewidth}{ \large HPCSE I, Exercise 01 }\hfill
 \parbox{.5\linewidth}{\begin{center} \large Beat Hubmann \end{center}}\hfill
 \parbox{.2\linewidth}{\begin{flushright} \large Sep 30, 2018 \end{flushright}}
}
\noindent\rule{\linewidth}{2pt}

\section{Question 1}
Nothing to report.

\section{Question 2}
Nothing to report.

\section{Question 3}

\subsection{Task a)}
Assumptions for all subsequent calculations of operational intensity:
\begin{itemize}
\item{Cache is empty (i.e. cold)}
\item{"Memory transfer" implies transfer from random access memory (DRAM) to cache (typically L3 cache)}
\end{itemize}

\subsubsection{DAXPY}

\begin{equation}
y = \alpha x + y \quad \alpha \in \mathbb{R}; x, y \in \mathbb{R}^{n}
\end{equation}

According to table~\ref{tab:daxpy}, a DAXPY operation on $x, y \in \mathbb{R}^{n}$ requires $2n$ FLOPS and $3n$ memory accesses. By definition, \textbf{D}AXPY is a double precision operation and thus each involved element has a size of 8 bytes. The resulting total operational intensity is $I(n) = \frac{2n \: \text{FLOPS}}{3n \cdot 8 \: \text{bytes}} = \frac{1}{12} \: \text{ops/byte}$ and hence $\mathcal{O}(1)$.

\begin{table}[ht]
\centering
\begin{tabular}{@{\extracolsep{4pt}}lcccc}
\toprule
{}  & \multicolumn{2}{c}{FLOPS} & \multicolumn{2}{c}{memory accesses}\\
\cmidrule{2-3}
\cmidrule{4-5}
$i = 1,2, \ldots, n$ & multiply & add & read & write \\
\midrule
$n \: \text{times\ldots} $  & 1    & 1   & 2   & 1 \\
\bottomrule
\end{tabular}
\caption{DAXPY operations per element $i \in \{1, 2, \ldots, n\}$}\label{tab:daxpy}
\end{table}

\subsubsection{SGEMV}

\begin{equation}
y = Ax + y \quad x, y \in \mathbb{R}^{n}; A \in \mathbb{R}^{n \times n}
\end{equation}

According to table~\ref{tab:sgemv}, a SGEMV operation on $x, y \in \mathbb{R}^{n}$ requires $2n^2 + n$ FLOPS and $2n^2 + 2n$ memory accesses. To make things easier we focus on the asymptotic bound and only consider the second-order terms. By definition, \textbf{S}GEMV is a single precision operation and thus each involved element has a size of 4 bytes. The resulting total asymptotic operational intensity is $I(n) = \frac{2n^2\: \text{FLOPS}}{2n^2\cdot 4 \: \text{bytes}} = \frac{1}{4} \: \text{ops/byte}$ and hence $\mathcal{O}(1)$.

\begin{table}[ht]
\centering
\begin{tabular}{@{\extracolsep{4pt}}lcccc}
\toprule
{}  & \multicolumn{2}{c}{FLOPS} & \multicolumn{2}{c}{memory accesses}\\
\cmidrule{2-3}
\cmidrule{4-5}
$i = 1,2, \ldots, n$ & multiply & add   & read   & write \\
\midrule
$n \: \text{times\ldots} $                    & $n$      & $n+1$ & $2n+1$ & 1 \\
\bottomrule
\end{tabular}
\caption{SGEMV operations per element $i \in \{1, 2, \ldots, n\}$}\label{tab:sgemv}
\end{table}

\subsubsection{DGEMM}

\begin{equation}
C = AB + C \quad A,B,C \in \mathbb{R}^{n \times n}
\end{equation}

Based on the hint given in the task description, we assume that $n$ and thus the matrices $A, B, C$ are small enough to fit in the cache concurrently and do not require multiple reads during a single DGEMM operation. According to table~\ref{tab:dgemm}, a DGEMM operation on $B, C \in \mathbb{R}^{n \times n}$ requires $2n^3 + n^2$ FLOPS and only $4n^2$ memory accesses because of the above assumption about $n$ versus cache size. As for SGMEV above, we focus on the higher order terms for the following calculations. By definition, \textbf{D}GEMM is a double precision operation and thus each involved element has a size of 8 bytes. The resulting total asymptotic operational intensity is $I(n) = \frac{2n^3\: \text{FLOPS}}{4n^2\cdot 8 \: \text{bytes}} = \frac{n}{16} \: \text{ops/byte}$ and hence $\mathcal{O}(n)$.

\begin{table}[ht]
\centering
\begin{tabular}{@{\extracolsep{4pt}}lcccc}
\toprule
{}  & \multicolumn{2}{c}{FLOPS} & \multicolumn{2}{c}{memory accesses}\\
\cmidrule{2-3}
\cmidrule{4-5}
$i,j = 1,2, \ldots, n$ & multiply & add   & read   & write \\
\midrule
$n\cdot n \: \text{times\ldots} $                    & $n$      & $n+1$ & $3$ & 1 \\
\bottomrule
\end{tabular}
\caption{DGEMM operations per element $i, j \in \{1, 2, \ldots, n\}$}\label{tab:dgemm}
\end{table}

\subsection{Task b)}

\begin{equation}
u_{i}^{n+1} = u_{i}^{n} + \frac{\Delta t \alpha}{\Delta x^2}(u_{i-1}^{n} - 2u_{i}^{n} + u_{i+1}^{n})
\label{eqn:1d_diffusion}
\end{equation}

Based on equation~(\ref{eqn:1d_diffusion}) and considering that $\frac{\Delta t \alpha}{\Delta x^2}$ is a fixed value which can be precalculated and stored, there are 3 additions and 2 multiplications for a total of 5 FLOPS per grid point. As for DGEMM in the previous task, we now must make assumptions about the problem size versus the cache size (the cache size being a potential bottleneck): It would be most realistic but also most cumbersome for our calculations to assume that some part of the problem fits in a cache of limited but realistic size and that we have to deal with a mix of compulsory and capacity cache misses. At the extreme ends of the cache size-vs-problem size scale we could take up either the assumption of an infinite cache and thus only one read and one write operation per grid point or the assumption of zero cache with a worst case maximum of four reads and one write per grid point. Assuming we are dealing with double precision data (8 bytes per element), we can thus make the following estimations: $I(N)_{\text{max}} = \frac{5\: \text{FLOPS}}{2\cdot8 \: \text{bytes}}=\frac{5}{16}\: \text{ops/byte}$ and $I(N)_{\text{min}} = \frac{5\: \text{FLOPS}}{5\cdot8 \: \text{bytes}}=\frac{1}{8}\: \text{ops/byte}$ with the realistic asymptotic operational intensity bounded as in equation~(\ref{eqn:task3b}).
\begin{equation}
0.125  \: \text{ops/byte} \leq I(N) \leq 0.3125  \: \text{ops/byte}
\label{eqn:task3b}
\end{equation}

\section{Question 4}

\subsection{Task a)}
Based on the link~\cite{intel} given in the task description, the CPU used is of the Haswell generation, has 12 cores with a processor base frequency of 2.50 GHz and can use the AVX2 instruction set. As nicely summarized in~\cite{nasa}, AVX2 introduced 256-bit (i.e. 4 times 64-bit double-precision elements) floating-point fused multiply-add (FMA) support which allows for 16 double precision floating point operations per cycle (2 FMA execution units each able to execute 2 operations per cycle). In total, this gives $\pi = 12 \times 2.5 \times 10^9 \frac{\text{cycle}}{s} \times 16 \frac{\text{FLOPS}}{\text{cycle}} = 480 \frac{\text{GFLOPS}}{s}$.\\
 According to~\cite{intel}, the fastest memory supported by the CPU is DDR4 memory running at 2133 MHz on four memory channels of size 64 bit each (the 256 bit as described above divided by 4). We thus get a maximum memory bandwidth of $\beta = 2133 \times 10^6 \frac{\text{cycle}}{s} \times 256 \frac{\text{bit}}{\text{cycle}} = 546.048 \frac{\text{Gbit}}{s} = 68.256 \frac{\text{Gbyte}}{s}$ which conveniently also is equal to the \textit{Max Memory Bandwith} of 68GB/s indicated on~\cite{intel} itself.

\subsection{Task b)}
\subsubsection{$I_b$ when in operating in balance}
As described in~\cite{kou}, the ridge point is where both the CPU and the memory bandwidth are fully exploited and is situated at the intersection of $\pi$ and $I \times \beta$ in a log-log plot of operational intensity $I$ vs $\text{GFLOPS}/s$.\\
In specific, for a full node of Euler II we have $I_b = \frac{\pi}{\beta} = \frac{960 \frac{\text{GFLOPS}}{s}}{136.6\frac{\text{Gbyte}}{s}} \approx 7 \: \text{ops/byte}$.

\subsubsection{$P_{\text{peak}}$}
Also as per the source and calculations cited above, we have that $P_{\text{peak}}(I) = \text{min}(\pi, I\times \beta)$ where $\pi, \beta$ are fixed hardware properties as described above.

\subsubsection{memory bound vs compute bound}
A code is memory bound if its operational intensity $I$ is smaller than the calculated balanced operational intensity $I_b$: $I < I_b$. Likewise, a code is compute bound if we have that $I > I_b$.
The possible optimisations differ according to the bound: For memory bound code, it is recommended~\cite{kou} to first minimise conflict caches misses by data reordering and then to minimise capacity cache misses by reordering and non-temporal writes. All this is to increase operational intensity $I$.\\
For compute bound codes, it is harder to find optimisations as there is no point in further increasing operational intensity. Possible measures include use of SSE/AVX instructions and/or reductions in precision.

\subsection{Task c)}
According to~\cite{euler}, a full compute node of Euler II has two sockets each according to the above Xeon E5-2680v3 CPU and memory bandwidth specifications $\pi, \beta$ with a total of 24 cores per node. Based on the computations This makes for a roofline diagram as in figure~\ref{fig:roofline}.

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=0.9]{roofline.eps}
\end{center}
\caption{Roofline diagram for full compute node of Euler II; measured value of task d).}
\label{fig:roofline}
\end{figure}

\subsection{Task d)}
Based on the code with an attempted CPU data cache flush given in appendix~\ref{app} compiled with GCC 4.8.2 using the flags \lstinline{-fopenmp} \lstinline{-std=c++11} \lstinline{-O3} \lstinline{-march=native} \lstinline{-funroll-loops} (using OpenMP only to get a simple way to measure time), repeated measurements on Euler II result in a performance of about $4.1 \frac{\text{GFLOPS}}{s}$ short of the theoretically achievable single thread/single channel maximum of $5 \frac{\text{GFLOPS}}{s}$. This position is marked by a green dot in figure~\ref{fig:roofline}. We therefore conclude that the chosen implementation falls into to the memory bound region as described above. Potential reasons for not fully utilising the hardware are the naive implementation (unbalanced adds/multiplies) as well as only using a single thread. A first measure would be to use OpenMP to use all available threads on the CPU's cores.

\begin{thebibliography}{99}

\bibitem{nasa}
	NASA,
 	\emph{Haswell Processors},
	online (\url{https://www.nas.nasa.gov/hecc/support/kb/haswell-processors_492.html}),
 	accessed 30-Sep-2018.

\bibitem{intel}
	Intel Corporation,
	\emph{Intel Xeon Processor E5-2680 v3 Product Specifications},
	online (\url{https://ark.intel.com/products/81908/}),
	accessed 30-Sep-2018.

\bibitem{euler}
	ETH Zurich,
 	\emph{scientific computing wiki: Euler},
	online (\url{https://scicomp.ethz.ch/wiki/Euler#Euler_II}),
 	accessed 30-Sep-2018.

\bibitem{kou}
	Petros Koumoutsakos,
	\emph{HPCSE I Lecture Notes},
	online (\url{http://www.cse-lab.ethz.ch/wp-content/uploads/2018/09/HPCSE_I_1_Intro.pdf}),
	accessed 30-Sep-2018.

\end{thebibliography}

\appendix
\section{Question 4, task d)}\label{app}
\lstinputlisting{ex01_q4_task_d.cpp}
\end{document}


\documentclass[12pt]{article}
\usepackage[margin=1.0in]{geometry}
\usepackage{indentfirst}
\usepackage{graphicx}
\usepackage{ amssymb }
\usepackage{listings}
\usepackage[toc,page]{appendix}
\usepackage{wrapfig}
\usepackage{float}
\usepackage{subcaption}
\usepackage{mwe}
\renewcommand{\thesubsection}{\thesection.\alph{subsection}}

\title{CTA200: Assignment 2}
\author{Jessica Patel}
\date{May 8, 2021}

\begin{document}

\maketitle

This write-up contains my methods and analysis for each question in Assignment 2.
\section{Question 1}
\subsection{Methods}
First, functions were defined for the equations of the first and second numerical approximation methods. A variety of h values were arbitrarily chosen between 0 and 1, with a step size of 0.1. The equations were iterated for these $h$ values, giving the rounded derivative approximation values shown in Table 1.

\begin{table}[h]
\centering

\begin{tabular}{l|lcr}
h values & Method 1 & Method 2\\
\hline
0.1 &0.9884 & 0.9933 \\
0.2 & 0.9784&0.9884 \\
0.3 & 0.9653&  0.9801\\
0.4 & 0.9490& 0.9687\\
0.5 &  0.9296& 0.9541\\
0.6 & 0.9073&   0.9157\\
0.7 & 0.8822& 0.9364\\
0.8 & 0.8544& 0.8922 \\
0.9 & 0.8240& 0.8660\\
\hline
\end{tabular}

\caption{Approximate derivative values over h for both methods.}
\end{table}

The error compared to the analytical derivative was then calculated for both methods by subtracting the analytical derivative from the numerical derivative and then dividing by the analytical derivative. By iterating over the various $h$ values, Figure 1 was found.

\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q1.pdf}
\caption{Comparing the errors of the numerical approximations to the analytical derivative for both methods.}
\end{center}
\end{figure}

\subsection{Analysis}
It can be observed from Table 1 that the derivative values get smaller as the step size h increases. Furthermore, the first method appears to have smaller derivative values than the second method for all $h$ values. Considering that the analytical derivative at $x_0 = 0.1$ is $\cos(0.1) \simeq 0.995$, it can be concluded that the second method provides a closer, and thus more accurate approximation of the derivative than the first method.

Considering Figure 1, it can be seen that the slope of the first method slightly increases, whereas the slope of the second method stays constant, showing a straight line. The slope for this plot represents the increase in the log of error over the log of step size. Putting the plot in a log scale allows for better comparison of the slopes since they would normally be curved lines, which are harder to compare. Since the first method appears to be slightly curved in Figure 1, this shows that the derivative approximation using method 1 diverges away from the true value at a quicker rate as the step size $h$ increases.
\section{Question 2}
\subsection{Methods}
First, a function was written to compute and save the iterations of the $z_{i+1} = z_i^2 + c$ equation. This function was made to iterate until a given limit of iterations, or to a limit of the size of iterations. This was done to stop the iterations when enough information is found. In this case, the limit of iterations was chosen to be $20$ since that would provide enough iterations to get an idea of whether the point diverges or not, because if it does, it tends to diverge quickly. A limit on the size of the iterations was chosen to be $1 \times 10^{20}$ so that once the iterations reach this size, it is known that it will diverge, and so less work can be done.

Next, the points were classified as bounded or diverging points by checking if the absolute values of the iterations ever increase past a value of $2$. This value was chosen since one of the basic properties of the Mandelbrot set is that the values are bounded within a disk of radius two from the origin. Figure 2 shows the classification of the points.

Then, the iteration number at which the points diverged was found by making the limit of the iteration size to be $2$, and finding the number of iterations it takes to pass this limit. Figure 3 shows the colour map showing the number of iterations for the various points. It should be noted that the iterations were capped at $20$, so the bounded points seem to correspond to $20$ iterations, but would never actually diverge.

\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q2_1.pdf}
\caption{Differentiating between diverging and bounded points.}
\end{center}
\end{figure}
\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q2_2.pdf}
\caption{Differentiating between number of iterations it takes to diverge.}
\end{center}
\end{figure}

\subsection{Analysis}
For Figure 2, it shows that the bounded points are mostly near the origin and it shows a symmetrical pattern over the x-axis. Figure 3 shows that the number of iterations it takes for the points to diverge decreases as the distance from the origin increases.

\section{Question 3}
\subsection{Methods}
For this question, the SIR model for disease spread was evaluated using $scipy.integrate$ as in the lecture example. The $\beta$ and $\gamma$ values were chosen by keeping $\beta$ greater than $\gamma$ by the same amount, while decreasing both parameters by the same amount. These values were chosen because it was noticed that when both parameters were made equal, or $\gamma$ was made larger than $\beta$, the results did not make sense since it would show that all curves are constant, and the infection and recovered curves would be near or equal to zero over time, which does not provide any information about disease spread. Furthermore, the difference between $\beta$ and $\gamma$  was chosen arbitrarily, but they were decreased to observe the pattern. The values for these parameters were chosen between $0$ and $1$. This is because, looking at the differential equations, it can be deduced that $\beta$ and $\gamma$ represent some fraction of the susceptible, infected and recovered populations. Figures 4, 5 and 6 show these curves for the various $\beta$ and $\gamma$ values.

\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q3_[0.8, 0.5].pdf}
\caption{SIR model for larger $\beta$ and $\gamma$ values.}
\end{center}
\end{figure}
\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q3_[0.6, 0.3].pdf}
\caption{SIR model for medium $\beta$ and $\gamma$ values.}
\end{center}
\end{figure}
\begin{figure}[h]
\begin{center}
\includegraphics[scale=0.5]{Q3_[0.4, 0.1].pdf}
\caption{SIR model for smaller $\beta$ and $\gamma$ values.}
\end{center}
\end{figure}
\subsection{Analysis}
 It can be observed from the figures that, as the $\beta$ and $\gamma$ values decrease, the infected curve has a larger bump and the susceptible and infected curves reach lower and higher amounts of people, respectively. This makes sense, because as $\gamma$ decreases, the number of recovered people decreases. Due to this definition, it is expected for the amount of infected people to hit a larger number. It is also expected for the number of susceptible people to decrease and for the amount of infected people to increase.

\end{document}


\documentclass[12pt]{article}
\newcommand{\kg}{\mathrm{kg}}
\newcommand{\m}{\mathrm{m}}
\newcommand{\s}{\mathrm{s}}
\begin{document}
\newcounter{problem}
\thispagestyle{empty}

\section*{NYU General Physics 1---Problem set 1}

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
\textsl{(a)}~What is the maximum amount of cash you can obtain by
successfully robbing an armored truck?  Assume that it is packed with
twenties; that is, estimate an answer by estimating the likely volume
of the inside of the truck, and (harder) the volume of a 20-dollar
bill.  State your assumptions and explain your work, but please don't
attempt an experiment.  Be sure to explain exactly how you estimated
the volume of a 20-dollar bill.  \emph{Hint: think of a stack of bills
  to estimate the volume.  Feel free to \emph{check} any part of your
  answer on the internet, but make sure you actually make a justified,
  quantitative estimate independently.}

\textsl{(b)}~Do you think that many of the armored trucks in Manhattan
are fully packed with 20-dollar bills?

\textsl{(c)}~Would a similar truck weigh more, less, or about the same
if it contained the same amount of money but in the form of gold bars
instead of 20-dollar bills?

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
\textsl{(a)}~Power $P$ has dimensions of energy per time.  Energy $E$
has dimensions you can infer from the famous formula $E=m\,c^2$.  What
combination can you make of a speed $v$ (length per time), an area $A$
(length squared), and a density $\rho$ (mass per volume) that has
dimensions of power?

\textsl{(b)}~Can you think of any environmental or economic or
engineering significance of that calculation?  \emph{Hint: Imagine
  that the density is the density of the air, and the area is the
  cross-sectional area of a car!}

\textsl{(c)}~What combination can you make of a speed $v$, a mass $m$,
and an acceleration $g$ (length per time squared) that has dimensions
of power?

\textsl{(d)}~Can you think of any biological or physiological
significance of that calculation?  \emph{Hint: Imagine that the speed
  is the speed you can sustainably climb stairs, the mass is your
  mass, and the acceleration is the acceleration due to gravity!}

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
For very small objects, like cells in a centrifuge, there is a
``settling velocity'' that depends on a density $\rho$ (actually a
density difference, but we will worry about that much later), an
acceleration $g$, a particle size (length or radius) $R$, and a
``dynamic viscosity'' $\mu$.  The dynamic viscosity has dimensions
of mass per length per time (for example, $\kg\,\m^{-1}\,\s^{-1}$; if that seems odd, look it up).  Without
worrying about any details---this is only dimensional analysis---find
a combination of these quantities that has the dimensions of velocity
(length per time).  Compare to what is written in the first couple of
paragraphs of the Wikipedia page called ``Stokes Law''.

\end{document}

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Roughly how far does a typical American car have to drive to ``sweep
up'' or drive through a column of air that is comparable in weight to
the car itself?  You will have to estimate the cross-sectional area
and weight of a typical car (or look both things up on the web; if you
look them up, give the make and model).