\chapter{Amplifier Classes}
The generic power amplifier consists of a transistor with a DC bias and AC coupled to the RF input and output ports. For the sake of simplicity this work will only describe the operation of field effect transistor amplifiers. The major source of power consumption of an amplifier comes from the drain source current and voltage that occurs during operation. Power amplifiers come in a variety of classes, and the most commonly used ones are A, B, AB, C, D, E, and F. Classes A through C can be described by the conduction angle of the device and modeling the transistor as a current source. The conduction angle is the amount of current flowing through the transistor from drain to source during a single cycle of the AC signal being amplified and ranges from 0 to $2\pi$ \cite{Colantonio1998}. Classes D through F are defined as switching amplifiers and vary due to the matching networks which shape the waveforms of the current and voltage at the drain and gate of the device \cite{Sokal1975}. This chapter will describe a high level overview of the various amplifier classes and discuss the limitations of each in practice.

%For class A, the conduction angle is $2\pi$ meaning current is always flowing when the amplifier is on increasing the power consumption. Class A does have the advantage of having the highest gain at a given frequency compared to other classes and also can operate near the frequency limit of a transistor .\cite{} Class A also has the highest linearity, lowest harmonic distortion, and is advantageous when high dynamic range is required .\cite{C.Cripps2006} Class B has a conduction angle of $\pi$ so current only flows during half the cycle of the AC signal reducing power consumption. Class AB has a conduction angle between $\pi$ and $2\pi$. Class E is a type of harmonic termination that uses capacitive terminations at the output to achieve 100\% efficiency.

% Citations are wrong above!

\section{Efficiency of Amplifiers}

To reduce the power consumption of a power amplifiers, two major concepts are important: zero voltage switching (ZVS) and zero voltage derivative switching (ZVDS). ZVS is when there is zero voltage when the amplifier is switched on and conducting so while current flows through the output no power is consumed. ZVDS is when there is no overlap between the voltage and current waveforms so no power is consumed during the switching. An amplifier can be 100\% if it achieves ZVS and ZVDS because no power would be consumed at any point in the amplification process. A measure of amplifier efficiency is the power added efficiency (PAE) defined in Equation \ref{pae}. PAE is a measure of how efficient an amplifier is able to convert the input DC power into RF power gain. Ideally all of the DC power used to power the amplifier would be used to increase the input RF power. The maximum limit for a lossless system is 100\%, for example if a power amplifier consumed 1 Watt and was able to output 1.001 Watt of RF power from 1 mW of RF input power, the PAE would be 100.\% because all of the DC power was used to increase the RF power.

\begin{equation}\label{pae}
  PAE,\% = \frac{RF\;Power_{out} - RF\\Power_{in}}{DC\;Power}
\end{equation}

Another measure of amplifier efficiency is the drain efficiency which is the proportion of the output power of the fundamental harmonic over the DC power consumed seen in Equation \ref{drain_eff}. The drain efficiency is typically used when talking about the theory behind the different amplifier classes. In practice, PAE is used as the metric to compare power amplifier efficiency because it takes in account RF input power which is relatively large compared to small signal amplifiers. Small signal amplifiers operate at a bias point that can be assumed to be linear for small input powers. Power amplifiers are large signal devices that operate in the non-linear regions of transistor.

\begin{equation}\label{drain_eff}
  \eta, \% = \frac{P_{Fundamental}}{DC\;Power}
\end{equation}

\section{Current Source Amplifiers}

\subsection{Class A Amplifier}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=6in]{figures/classes/classa}
  \caption{Class A Bias Point and Conduction Cycle \cite{Rosu2001}}\label{classa_bias}
\end{figure}

The class A amplifier uses a bias that results in a conduction angle of $2\pi$, seen in Figure \ref{classa_bias}, and a theoretical quiescent current of 50\% the maximum drain current. This results in power always being consumed due to the constant bias current and voltage present at the drain of the amplifier. The bias point is chosen so the transistor operates in the active region which can be seen in Figure \ref{classa_bias}. The constant power consumption limits the maximum efficiency of the class A amplifier for a continuous wave (CW) signal to 50\% \cite{C.Cripps2006}.

%The class A amplifier is similar to the small signal amplifier in many ways

The class A amplifier is the most linear of all classes due to the conduction angle that provides ideally no clipping of the current or voltage waveforms. The high linearity allows the class A amplifier to have the lowest intermodulation distortion (IMD) of all the amplifier classes. The downside of the linearity is that the input drive level is proportional to the IMD present at the output of the amplifier. Class A amplifiers are also able to operate closer to the highest operating frequency of a transistor because it does't use the harmonics present in the amplifier for it's amplification \cite{Rosu2001}. %ham paper

%Fix all the citations lol

\subsection{Class B Amplifier}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=6in]{figures/classes/classb_bias}\\
  \caption{Class B Bias Point and Conduction Cycle \cite{Rosu2001}}\label{classb_bias}
\end{figure}

Class B uses a conduction angle of $\pi$ resulting in a half sinusoid waveform for the drain current seen in Figure \ref{classb_bias} with a theoretical quiescent current of 0 A. The maximum efficiency for a CW signal is 78.5\%. The reduced conduction angle causes the required input power to achieve the maximum efficiency to be 6 dB higher compared to the class A amplifier \cite{C.Cripps2006}.
The drain current is proportional to the input signal so the Class B amplifier is considered a linear amplifier but will result in harmonic distortion due to the clipping of the drain current. The output is usually filtered to reduce the harmonic distortion\cite{Raab2003}.
In most applications, the class B amplifier is used in a push pull configuration to reduce distortion by providing a full sine wave output and maintain greater efficiency than a class A amplifier. Also instead of setting the quiescent current to 0, the drain current is typically set to 10\% of the max drain current of the transistor to reduce the harmonic distortion. An interesting result of the push pull configuration is that even harmonics subtract one another out while odd harmonics add together \cite{Rosu2001}.

\subsection{Class AB Amplifier}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=6in]{figures/classes/classab_bias}\\
  \caption{Class AB Bias Point and Conduction Cycle \cite{Rosu2001}}\label{classab_bias}
\end{figure}

The class AB amplifier uses a conduction angle ranging between $\pi$ and $2\pi$ and is a compromise between class A and class B amplifiers. The drain efficiency lies between 50\% and 78.5\% with approximately the same output power as the class A amplifier \cite{C.Cripps2006}. The bias point and conduction cycle can be seen in Figure \ref{classab_bias}. The class AB amplifier offers a wider dynamic range than class A or class B because the conduction angle is proportional to the drain current which also results in the class AB amplifier being a non-linear amplifier \cite{C.Cripps2006}.

\subsection{Class C Amplifier}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=6in]{figures/classes/classc_bias}\\
  \caption{Class C Bias Point and Conduction Cycle \cite{Rosu2001}}\label{classc_bias}
\end{figure}

The class C amplifiers is biased such that the quiescent current is 0 A and the conduction angle is between $0$ and $\pi$ seen in Figure \ref{classc_bias}. The class C amplifier is non-linear with a higher efficiency than class B with a maximum drain efficiency of 100\% at zero conduction angle but will have zero gain. In practice, the class C amplifier has a conduction angle slightly less than $\pi$ for a maximum efficiency of 85\% while still having a useful gain \cite{Raab2003}. The class C has the highest harmonic distortion of all the current source amplifier classes which will be shown in the following section. The harmonic distortion requires significant filtering at the output of the amplifier. Class C amplifiers aren't typically used for microwave power amplifiers due to the large negative voltage present at the gate of the transistor which can cause reverse breakdown in the transistor \cite{C.Cripps2006}. The class C amplifier offers a tradeoff of higher efficiency but with higher harmonic distortion and lower output power level compared to the other current source amplifier classes.

\subsection{Harmonic Distortion}

The current source power amplifiers can be generalized to a power amplifier with various conduction angles and the drain efficiencies. The  output power and harmonic distortion can be described as a function of conduction angle seen in Equations \ref{eq:cond_eff}, \ref{eq:cond_pout}, and \ref{eq:cond_current} \cite{Hella}. Table \ref{table:waveform_param} defines the various parameters to the various equations. Figure \ref{fig:bias_current_harmonic} shows the normalized output current harmonics versus conduction angle. At a conduction angle of $2\pi$, the output has minimal harmonic distortion with only the fundamental and DC component  present. The DC component is easily removed by coupling capacitor. As conduction angle increase, the other harmonic amplitudes increase which causes increased harmonic distortion and the DC power is reduced increasing the efficiency. At zero conduction angle, the transistor won't conduct any current and all of the harmonic amplitudes drop to 0. Figure \ref{fig:bias_power_eff} shows the RF power relative to class A and efficiency versus conduction angle assuming the harmonics are removed and the transistor is optimally loaded. A power amplifier is said to be properly loaded if the load resistance is equal to the DC drain voltage minus the knee voltage of the transistor divided by the fundamental drain current, see Equation \ref{eq:opt_load} \cite{Hella}.

The generalized form of the current source amplifiers are not able to ever achieve ZVS due to the constant drain voltage waveform present at the output which is required to bias the amplifier. And current source amplifiers can only achieve ZVDS at a conduction angle of 0 radians which results in 0 power delivered to the load. To achieve 100\% efficiency and have a practical gain we look to the switching amplifier.
%In the equations, $v_{dd}$ is the drain voltage, $v_{sat}$ is the saturation voltage of the transistor, $\theta$ is the conduction angle in radians,

\begin{table}
    \caption{Table of Current Source Waveform Parameters}
    \label{table:waveform_param}
        \begin{center}
            \begin{tabular}{|l|l|l|}
              \hline
              % after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
              Parameter & Definition & Units\\ \hline
              $I_m$ & Maximum drain current of transistor & A \\ \hline
              $I_n$ & Magnitude of current harmonic $n$ & A \\ \hline
              $v_{dd}$ & DC drain voltage & V\\ \hline
              $v_{knee}$ & Knee voltage of transistor & V\\ \hline
              $v_{sat}$ & Saturation voltage of transistor & V \\ \hline
              $\theta$ & Conduction angle & Radians\\ \hline

            \end{tabular}
        \end{center}
\end{table}

\begin{equation}\label{eq:cond_eff}
  \eta_{Drain}, \% = \frac{v_{dd} - v_{sat}}{v_{dd}} \frac{\theta - sin\theta}{4(sin\frac{\theta}{2} - \frac{\theta}{2} cos\frac{\theta}{2}) }
\end{equation}

\begin{equation}\label{eq:cond_pout}
  P_{out} = \frac{1}{2}(v_{dd} - v_{sat} )\frac{I_m}{2\pi}(\theta - sin\theta)
\end{equation}

\begin{equation}\label{eq:cond_current}
  I_n = \frac{1}{\pi}\int_{-\frac{\theta}{2}}^{\frac{\theta}{2}} \frac{I_m}{1 - cos(\frac{\theta}{2})} (cos\alpha - cos(\frac{\theta}{2}))cos(n\alpha) \; d\alpha
\end{equation}

\begin{equation}\label{eq:opt_load}
  R_{opt} = \frac{v_{dd}-v_{knee}}{I_{Fundemetnal}}
\end{equation}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=4in,height=4in,keepaspectratio]{figures/classes/bias_current_harmonic}\\
  \caption{Normalized Current Harmonic Amplitude versus Conduction Angle \cite{C.Cripps2006}}
  \label{fig:bias_current_harmonic}
\end{figure}

\begin{figure}
  \centering
  % Requires \usepackage{graphicx}
  \includegraphics[width=4in,height=4in,keepaspectratio]{figures/classes/bias_power_eff}\\
  \caption{RF Power (relative to Class A) and Efficiency versus Conduction Angle \cite{C.Cripps2006}}
  \label{fig:bias_power_eff}
\end{figure}

\section{Switching Amplifiers}

Switching amplifiers model the transistor as a switch instead a dependent current source. Figure \ref{fig:basic_switch} shows the ideal operation of a switching amplifier. The transistor is modeled as a switch that can only be "on" or "off" depending on the gate voltage. The switching operation allows for current to flow while the switch is on with 0 drain voltage, and for 0 current while the switch is off with a non-zero drain voltage. Ideally this would allow for ZVS and for ZVDS which could result in 100\% efficiency. The switching amplifier classes (D, E, and F) build upon this ideal model and are able to achieve higher efficiencies in practice than current source amplifiers by using matching networks to shape the waveforms at the drain and gate.

\begin{figure}
    \subfigure{\includegraphics[width=3in]{figures/classes/basic_switch}}
    \subfigure{\includegraphics[width=3in]{figures/classes/basic_switch_waveform}}
    \caption{Basic Switching Amplifier Circuit and Waveforms\cite{C.Cripps2006}}\label{fig:basic_switch}
\end{figure}

\subsection{Class D Amplifier}

The class D amplifier can be thought of as push pull version of a switching amplifier. The amplifier consists of two transistors to operate as switches to conduct during the positive half and negative half cycle of the input seen in Figure \ref{fig:class_d} \cite{C.Cripps2006}. This allows for a full wave drain current to flow through the output load that is filtered through a high Q LC network while the drain voltage is a square wave with 0 overlap between the transistor that is on that allows for ZVS and ZVDS \cite{Rosu2001}. This allows the class D amplifier to theoretically have 100\% drain efficiency. The LC network is tuned to the fundamental frequency of the input. Class D looks effective in theory but suffers due to switching resistances and drain capacitances which limit the maximum frequency of operation to VHF in practice \cite{Raab2003}.

\begin{figure}
    \subfigure{\includegraphics[width=3in]{figures/classes/classd_ckt}}
    \subfigure{{\raisebox{-0.25\height}{\includegraphics[width=3in,height=3in,keepaspectratio]{figures/classes/classd_wave}}}}
    \caption{Class D Amplifier Circuit and Waveforms\cite{C.Cripps2006}}\label{fig:class_d}
\end{figure}

\subsection{Class E Amplifier}

The class E amplifier builds up on the ideal switch amplifier takes in account for the shunt output drain capacitance and amplification is the result of DC and RF currents that charge the drain shunt capacitance \cite{Raab2003}. As seen in Figure \ref{fig:class_e}, the switch is turned on from  time $-\alpha_1$ to $\alpha_2$ allowing current to flow solely from the power supply and the load through the switch and the voltage across the drain capacitance is shorted. Then as the switch is turned off from time $\alpha_2$ to $\alpha_1$ the current from the power supply and load will flow through the drain capacitor \cite{C.Cripps2006}. To achieve high efficiency, the switching current waveforms are adjusted so that the capacitor current and charge is 0 when the switch is turned on which allows for ZVS and ZVDS conditions allowing for theoretically 100\% efficiency. \cite{Kee2003}.

The class E amplifier produces significant harmonic distortion due to the switching action and the output must be filtered to suppress harmonics \cite{Sokal1975}. Another concern with using the class E amplifier in practice is the large peak voltages at the drain of the transistor which are around three times the drain supply voltage when the switch turns off \cite{Hella}. In addition the output drain capacitance limits the frequency of operation that is required to create the proper drain waveforms \cite{Rosu2001}.

\begin{figure}
    \subfigure{\includegraphics[width=3in]{figures/classes/classe_ckt}}
    \subfigure{{\raisebox{-0.25\height}{\includegraphics[width=3in,height=3in,keepaspectratio]{figures/classes/classe_wave2}}}}
    \caption{Class E Amplifier Circuit and Waveforms\cite{C.Cripps2006}}\label{fig:class_e}
\end{figure}

\subsection{Class F Amplifier}

The class F amplifier has a theoretical efficiency of 100\% through the use of multi harmonic terminations. The theory behind the class F amplifier relies on the generation of odd voltage harmonics and shorting of even voltage harmonics at the drain of the device to increase the efficiency of the amplifier. By keeping only the odd voltage harmonics of the signal at the output, ideally a square wave would appear and by biasing the amplifier at a conduction angle of pi the amplifier has zero overlap between the current and voltage waveforms and no power consumed when the amplifier is switched on.
The harmonics are generated due to the knee voltage of the transistor. When the AC signal causes the transistor to reach saturation the abrupt change in current flow generates voltage harmonics. In practice it becomes challenging to design matching networks as the number of harmonics to control increases. Also the output capacitance of the transistor will eventually short out higher order harmonics so 100\% efficiency is not realizable in practice. By controlling up to the 5th harmonic, 90.5\% efficiency can be achieved. A more in depth analysis of the class F amplifier follows in the next chapter.

%\subsection{Class J Amplifier}
%
%Shorts even harmonics
%
%Is described by the ratio of the drain reactance and the
%
%Maximum efficiency of 78.5\%


\documentclass{article}
    % General document formatting
    \usepackage[margin=0.7in]{geometry}
    \usepackage[parfill]{parskip}
    \usepackage[utf8]{inputenc}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{tikz}
    \usepackage{fancyhdr}
    \usepackage{listings}
    \usepackage{multicol}
    \usepackage{polynom}

\pagestyle{fancy}
\fancyhf{}
\rhead{Edgar Jacob Rivera Rios - A01184125}

\begin{document}
\begin{titlepage}

    \newcommand{\HRule}{\rule{\linewidth}{0.5mm}} % Defines a new command for the horizontal lines, change thickness here

    \center % Center everything on the page

    %----------------------------------------------------------------------------------------
    %	HEADING SECTIONS
    %----------------------------------------------------------------------------------------

    \textsc{\LARGE Tecnológico de Monterrey}\\[1.5cm] % Name of your university/college
    \textsc{\Large Fundamentos de computación}\\[0.5cm] % Major heading such as course name
    %\textsc{\large Minor Heading}\\[0.5cm] % Minor heading such as course title

    %----------------------------------------------------------------------------------------
    %	TITLE SECTION
    %----------------------------------------------------------------------------------------

    \HRule \\[0.4cm]
    { \huge \bfseries Homework 9}\\[0.4cm] % Title of your document
    \HRule \\[1.5cm]

    %----------------------------------------------------------------------------------------
    %	AUTHOR SECTION
    %----------------------------------------------------------------------------------------

    \begin{minipage}{0.4\textwidth}
    \begin{flushleft} \large
    \emph{Student:}\\
    Jacob \textsc{Rivera} % Your name
    \end{flushleft}
    \end{minipage}
    ~
    \begin{minipage}{0.4\textwidth}
    \begin{flushright} \large
    \emph{Professor:} \\
    Dr. Hugo \textsc{Terashima} % Supervisor's Name
    \end{flushright}
    \end{minipage}\\[2cm]

    % If you don't want a supervisor, uncomment the two lines below and remove the section above
    %\Large \emph{Author:}\\
    %John \textsc{Smith}\\[3cm] % Your name

    %----------------------------------------------------------------------------------------
    %	DATE SECTION
    %----------------------------------------------------------------------------------------

    {\large \today}\\[2cm] % Date, change the \today to a set date if you want to be precise

    %----------------------------------------------------------------------------------------
    %	LOGO SECTION
    %----------------------------------------------------------------------------------------

    \includegraphics[width=0.4\textwidth,height=\textheight,keepaspectratio]{logo-tec-negro.png} % Include a department/university logo - this will require the graphicx package

    %----------------------------------------------------------------------------------------

    \vfill % Fill the rest of the page with whitespace

\end{titlepage}

\section{Problems}
Solve the following problems:
\begin{enumerate}
    \item What is a P problem?

    A problem that can be solved in polynomial time

    \item What is an NP problem?

    A problem which result can be verified in polynomial time, but has a non-deterministic solution time.

    \item What is and NP - complete problem?

    This are the problems that are at least as hard as the hardest NP problems. This means, the ones that are both NP and NP-Hard.

    \item Why P = NP is considered an open problem?

    It is the question that arises from comparing P and NP problems, it is known that P is included in NP, but we do not know if the NP problems can be reduced to be solved in P time, making P and NP the same.
    It is currently considered an open problem there is not a mathematical proof that NP is equal or different to P.

    \item What is the difference between a decision problem and an optimization problem?

    A decision problem returns as result a simple $yes$, $no$ answer, and a optimization problem tries to find the best values for certain defined parameters.

    \item Investigate the 2-dimensional packing problem, describe the problem, and state it as a decision problem and then as an optimization problem?

    In the problem, you are presented with a series of items with different dimensions and bins of fixed dimensions, the objective is to fit the items in as few bins as possible.

    As a decision problem:\\
    Is it possible to fit the items in $n$ bins?

    As a optimization problem:\\
    Find the combinations of items which fit in the least amount of bins possible

    \item Describe a simple heuristic (approximation algorithm) for solving the 2-dimensional packing problem. How could you measure the effectiveness of this heuristic for solving the problem?

    The characterization of the items can be used to solve the problem, you could use the average of width and height to decide if it's better to fill a bin as much as possible or open a new one. The quantity of bins is definitive measure to use when testing the heuristic.

    \item Can you describe the Travelling Salesman Problem? How is this problem related to the Hamiltonian Cycle? Provide a polynomial transformation between those problems. How are these problems related
    to the Vehicle Routing Problem?

    The traveling salesman problem is a problem which states that given a list of cities and the distances between them, to return the shortest path that visits each city and returns to the initial city. This is basically a specialization of the problem of the Hamiltonian circle, which consists on finding a cycle which given a graph, tries to get a cycle which passes for each node once. The vehicle routing problem is a generalization of the the Traveling Salesman Problem.

    \item What is a phase transition?

    It is a transformation for the inputs from one problem to another

    \item How is a phase transition related to the easiness or hardness of NP problems?

    By using a phase transition, you can either simplify or maker harder the problem by converting in another problem. For example, you can use a phase transition to convert an NP problem to an NP-Hard problem

\end{enumerate}
\end{document}


\chapter{About this Book}

This book is an Open Education Resource. It is written for students
with a little background in logic, and some high school mathematics.
It aims to scratch the tip of the surface of the philosophy of set
theory. By the end of this book, students reading it might have a
sense of:
\begin{enumerate}
	\item why set theory came about;
	\item how to embed large swathes of mathematics within set theory + arithmetic;
	\item how to embed arithmetic itself within set theory;
	\item what the cumulative iterative conception of set amounts to;
	\item how one might try to justify the axioms of ZFC.
\end{enumerate}
The book grew out of a short course that I taught in the Cambridge Philosophy department. Before me, it was lectured by Luca Incurvati and Michael
Potter. In writing this book---and the course, more generally---I was
hugely indebted to both Luca and Michael. I hope this comes through in the text of the book itself; but I also want to offer both of them
my heartfelt thanks here.

Most of this book was originally released as \emph{Open
Set Theory}; this book is its successor. I have contributed all of the material for this book to the \href{http://openlogicproject.org}{\emph{Open Logic Project}}, where it is freely available. (\Crefrange{sfr:set::chap}{sfr:siz::chap} were drawn, with only tiny
changes, from previously existing material in the OLP; these changes are now also part of the OLP.) Please see
\href{http://openlogicproject.org/}{openlogicproject.org} for more
information.

\begin{flushright}
Tim Button
\\University College London
\\October 2021
\end{flushright}


\documentclass{proc}
\usepackage{url}

\begin{document}

\title{Access path selection in a relational database management system~\cite{selinger1979access}.}

%\author{Selinger et al.}

\maketitle

\section{Abstract / Introduction}

\emph{1. What opportunities/changes that make this work useful and timely?
2. Why existing approaches fail to make use of these opportunities?
3. How do you propose to do better?
4. Why this problem is relevant to the course?
(1-2 sentences each)}

SQL is a declarative alternative to imperative data manipulation languages used by IMS and proposed by CODASYL.
SQL translates into relational algebra (\emph{RA}), which is built on top of a theoretically sound relational data model.
SQL is good because it makes the programmer's life easier by providing data independence so that changes in how the data is physically stored does not affect the SQL queries that developers write to access and manipulate their data.

This development is important because developers currently spend most of their time writing and re-writing their applications whenever there is a change to the schema of the data or how it is stored.  This means developers don't have time to build new features and improve their applications.  SQL can help address this issue, but current SQL execution engines are unacceptably slow and nowhere near the performance of hand-tuned IMS/CODASYL queries.

SQL can translate into many different, but semantically equivalent, relational algebra statements, and each statement can be executed by many possible \emph{query plans},  it is desirable to pick the plan that is fastest to run. This project proposes a way to estimate the cost of executing a query plan, and automatically search the space of all valid query plans for a SQL query to find one that is fast no execute.

This problem is relevant to the course because data visualization systems are often built on top of data management systems, so making the data management system faster would thus make the data visualization faster.

\section{One Sentence Summary}
\begin{quote}
\emph{Describe your project in one sentence, in other words, your hypothesis.}
\end{quote}

We will devise a cost model to give each query plan a score, and use existing equivalence rules to define and search the space of alternative plans to find one with low cost.  Our hypothesis is that this will be competitive with hand-coded execution plans.

\section{Audience and Needs}
\begin{quote}
\emph{Who are the audiences for this project?
How does it meet their needs?
What happens if their needs remain unmet?}
\end{quote}

This project will impact the academic community as well as every application developer.
The community benefits because it will validate the hypothesis that declarative lanugages such as SQL can run quickly, and open up a new field of research in data management.

Application developers benefit because they will not need to write imperative data manipulation code, worry about how to hand optimize the code to be fast, and can focus on building application features.

\section{Approach}
\begin{quote}
\emph{What is your approach?
Why do you think it's a good approach and will be successful?}
\end{quote}

\begin{enumerate}
\item We will build cost models for individual operators.  Recent advances on disk-based indexes~\cite{bayer2002organization} suggest that we will need different cost models depending on how the data is stored.  We may need to make simplifying assumptions and model the costs based on very simple statistics.
\item We will combine the per-operator cost models estimate the cost for an entire query plan.
\item With a cost model, we can now search through the space of all query plans that return the same results.  This space is huge, so we will develop heuristics to reduce the space.   Past experience with IMS has shown that the dominant cost will be combining multiple relations using joins so we will focus on them. The space of possible joins can also be very large, so we will develop a dynamic programming-based approach to search through the space.
\item Run experiments and show that the plans picked by the optimizer are reasonable.
\end{enumerate}

\section{(Best Case) Impact}
\begin{quote}
\emph{In the best-case scenario, what would be the impact statement (ideal outcome and conclusion) for this project?}
\end{quote}

We can show that the optimizer picks reasonable plans that are comparable to hand-optimized plans selected by expert developers.

\section{Milestones}
\begin{quote}
\emph{List all major milestones for this project}
\end{quote}

\begin{enumerate}
  \item Identify the list of statistics that we can reasonably compute and store in tens of kilobytes of space.
  \item Develop a cost model for individual operators and show that the model is correlated with reality by running them on synthetic datasets based on what existing IMS customer use.
  \item Develop a cost model for a full query plan.
  \item Derive an estimate of the size of the full plan space for a given query plan and show that it is infeasibly large.
  \item Develop dynamic programming heuristic to search the plan space.
  \item Run experiments on synthetic datasets to compare the optimizer-picked query plans with hand-optimized query plans.
\end{enumerate}

\section{Obstacles}
\begin{quote}
\emph{What are the major and minor obstacles that could happen?
Note that major obstacles are situations where you would consider \textbf{killing} the project.
Minor obstacles are things that would delay the project or increase the overall cost in energy, time, people, and money.}
\end{quote}

\subsection{Major obstacles}

\begin{itemize}
  \item If we cannot show decent correlation between individual operator cost models and reality, then the rest of the project may not work.  We will also need to define what ``decent'' means.
\end{itemize}

\subsection{Minor obstacles}

\begin{itemize}
  \item We may not have access to computing resources to run any experiments, in which case we will need to focus on theoretical aspects of the work.  This could reduce the impact of the project, however showing that an optimizer is \emph{possible} is still a contribution.
\end{itemize}

\section{Additional Resources}
\begin{quote}
\emph{What additional resources do you need to complete this project?}
\end{quote}

\begin{itemize}
  \item Some computational time to run our optimizer algorithm to generate some query plans.
  \item Access to a machine where we can install and run experiments using our current database prototype.
 \end{itemize}

\section{Literature Review}
\begin{quote}
\emph{List 5 major publications that are most relevant to this project, and how they are related.}
\end{quote}

\begin{itemize}
\item \emph{Background for the project:} This work builds on prior work on relational algebra and the relational model~\cite{codd1970relational}, and on new relational database systems~\cite{stonebraker1976design,astrahan1976system}

\item \emph{Work the project relies and builds on: } Some preliminary work has suggested a language for specifying query plans~\cite{lorie1979access} that we could borrow from.  Also, Recent work~\cite{bayer2002organization} on different ways to store and index data lend credence to the need for different cost models for access data in relations.

\item \emph{Direct competitors: } We could not find existing works on alternative techniques to automatically optimize query plans.

\item \emph{Alternatives to achieve the broader goal: } As stated above, IMS and CODASYL~\cite{taylor1976codasyl} are the main alternative data management systems, but they do not have any automated query optimization.

 \end{itemize}

\section{Define Success}
\begin{quote}
\emph{When / How do you know if you have succeeded in this project?
In other words, what is the minimum finding that would make this project a success and publishable?}
\end{quote}

Simply developing a set of cost models and search heuristics for query plans should be publishable, because an automated optimizer of any sort does not yet exist.

\bibliographystyle{abbrv}
\bibliography{prospectus}
\end{document}


\chapter{Cost-sensitive decision trees}\label{ch:8}

\begin{remark}{Outline}
In this chapter, we propose the example-dependent cost-sensitive decision tree algorithm. This
algorithm is
based on incorporating the different example-dependent costs into a new cost-based impurity measure
and a new cost-based pruning criteria. First, in Section~\ref{sec:8:dt}, we give the background
behind the decision tree algorithm. Then, in Section~\ref{sec:8:csdt}, we introduce the
cost-sensitive decision tree method. For this, we first show the new cost-sensitive impurity
measure that introduces the different costs when evaluating the performance of a split. Then, we
introduce a new cost-sensitive pruning criteria. Finally, in Section~\ref{sec:8:experiments}, we
compare the results of the proposed algorithm against state-of-the-art methods using the five
real-world cost-sensitive databases.
\end{remark}

\section{Introduction}
\label{sec:8:intro}

Decision trees are one of the most widely used machine learning algorithms \citep{Lior2008}.
The technique is considered to be white box, in the sense that is easy to interpret, and has a
very low computational cost, while maintaining a good performance as compared with more complex
techniques \citep{Hastie2009}. There are two types of decision tree depending on the objective of
the model. They work either for classification or regression. In this section we focus on
binary classification decision tree.

Standard decision tree algorithms focus on inducing trees that maximize accuracy. However this is
not optimal when the misclassification costs are unequal \citep{Elkan2001}. This has led to many
studies that develop algorithms that aim to introduce the cost-sensitivity into the algorithms
\citep{Lomax2013}. These studies have focused on introducing the class-dependent costs
\citep{Draper1994,Ting2002,Ling2004,Li2005,Kretowski2006,Vadera2010}, which is not optimal for
some applications. For example in credit card fraud detection, it is true that false positives
have a different cost than false negatives, nevertheless, false negatives may vary significantly,
which makes class-dependent cost-sensitive methods not suitable for this problem.

In this chapter, we propose a cost-sensitive decision tree algorithm. The algorithm is
based on incorporating the different example-dependent costs into a new cost-based impurity measure
and a new cost-based pruning criteria. We evaluate the proposed example-dependent
cost-sensitive decision tree using three different databases. In particular, a credit card fraud
detection, a credit scoring and a direct marketing databases. The results show that the proposed
method outperforms state-of-the-art example-dependent cost-sensitive methods. Furthermore, when
compared against a standard decision tree, our method builds significantly smaller trees in only a
fifth of the time.

\section{Decision trees}
\label{sec:8:dt}

There are two types of decision tree depending on the objective of the model. They work either for
classification or regression. In this section we focus on binary classification decision tree.

\subsection{Construction of classification trees}

Classification trees is one of the most common types of decision tree, in which the objective
is to find the $Tree$ that best discriminates between classes. In general the decision tree
represents a set of splitting rules organized in levels in a flowchart structure.

\subsubsection{Splitting criteria}

In the $Tree$, each rule is shown as a node, and it is represented as $(\mathbf{x}^j,l^j)$, meaning
that the
set $\mathcal{S}$ is split in two sets $\mathcal{S}^l$ and $\mathcal{S}^r$ according to
$\mathbf{x}^j$ and $l^j$:
\begin{equation}
  \mathcal{S}^l = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge x^j_i \le l^j \},
\end{equation}
and
\begin{equation}
  \mathcal{S}^r = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge x^j_i > l^j \},
\end{equation}
where $\mathbf{x}^j$ is the $j^{th}$ feature represented in the vector
$\mathbf{x}^j=[x_1^j,x_2^j,...,x_N^j]$, and $l^j$ is a value such that $min(\mathbf{x}^j) \le l^j <
max(\mathbf{x}^j)$. Moreover, $\mathcal{S} = \mathcal{S}^l \cup \mathcal{S}^r$.

After the training set have been split, the percentage of positives in the different sets is
calculated. First, the number of positives in each set is estimated by
\begin{equation}
 \mathcal{S}_1 = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S} \wedge y_i =1 \},
\end{equation}
then the percentage of positives is calculates as
\begin{equation}
 \pi_1=\vert \mathcal{S}_1 \vert / \vert \mathcal{S} \vert.
\end{equation}

\newpage
Then, the impurity of each leaf is calculated using either a misclassification error,
entropy or Gini measures:
\begin{itemize}
 \item  Misclassification
 \begin{equation}
 I_m(\pi_1)=1-max(\pi_1,1-\pi_1)
 \end{equation}

 \item Entropy
 \begin{equation}
 I_e(\pi_1)=-\pi_1\log \pi_1 -(1-\pi_1) \log (1-\pi_1)
 \end{equation}

 \item Gini
 \begin{equation}
 I_g(\pi_1)=2\pi_1(1-\pi_1)
 \end{equation}
\end{itemize}

In \figurename{ \ref{fig:8:impurity}}, the three measures are shown. All measures are similar, with
the exception that the gini index and the cross-entropy are differentiable, which makes them
more suitable for numerical optimization.
\begin{figure}[t!]
\includegraphics[scale=0.85]{ch8_fig1}
\caption{Impurity measures for a binary classification, as a function of the proportion of
positive examples in the set (Cross-entropy is scaled) \citep{Hastie2009}.}
\label{fig:8:impurity}
\end{figure}

Finally the gain of the splitting criteria using the rule $(\mathbf{x}^j,l^j)$ is calculated as the
impurity of $\mathcal{S}$ minus the weighted impurity of each leaf:
\begin{equation}\label{eq:8:gain}
  Gain(\mathbf{x}^j,l^j)=I(\pi_1)-\frac{\vert \mathcal{S}^l \vert}{\vert \mathcal{S}
\vert}I(\pi^l_1)  -\frac{\vert \mathcal{S}^r \vert}{\vert \mathcal{S} \vert}I(\pi^r_1),
\end{equation}
where $I(\pi_1)$ can be either of the impurity measures $I_e(\pi_1)$ or $I_g(\pi_1)$.

Subsequently, the gain of all possible splitting rules is calculated. The rule with maximal
gain is selected
\begin{equation}
  (best_x, best_l) = \argmax_{(\mathbf{x}^j,l^j)}Gain(\mathbf{x}^j,l^j),
\end{equation}
and the set $\mathcal{S}$ is split into $\mathcal{S}^l$ and $\mathcal{S}^r$ according to that rule.

\subsubsection{Tree growing}
In order to grow a tree typical algorithms use a top-down induction using a greedy search in each
iteration \citep{Rokach2010}. In each iteration, the algorithms evaluates all possible splitting
rules and pick the one that maximizes the splitting criteria. After the selection of a splitting
rule, each leaf is further selected and it is subdivides into smaller leafs, until one of the
stopping criteria is meet. In Algorithm \ref{alg:8:tree_grow}, the pseudocode of the tree
growing procedure is presented.

\begin{algorithm}\label{alg:8:tree_grow}
Pseudocode of the tree growing procedure.
\textnormal{
\begin{algorithmic}[1]
\Function{TreeGrow}{$\mathcal{S}$}
    \State $Tree = Empty$
    \If{\Call{Stopping}{$Tree, \mathcal{S}$}}
      \State \Return $Tree$
     \EndIf
    \State \# For all features
    \For{$j=1 \  to \ k$}
      \State \# For all possible thresholds
      \For{$m=1 \ to N_k$}
        \State $Gains(j,m) = $ \Call{Gain}{$\mathbf{x}^j,l^j_m, \mathcal{S}$}
      \EndFor
    \EndFor
    \State \# Select the threshold with the highest gain
    \State $(j^*, l^*) = argmax_{(j,m)}Gains$
    \State \# Split the set using the best rule
    \State $\mathcal{S}^l = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S}
            \wedge \mathbf{x}_i^{j^*} \le l^* \}$
    \State $\mathcal{S}^r = \{\mathbf{x}_i^* \vert \mathbf{x}_i^* \in \mathcal{S}
            \wedge \mathbf{x}_i^{j^*} > l^* \}$
    \State \# Recursively continue growing tree
    \State \Call{TreeGrow}{$\mathcal{S}^l$}
    \State \Call{TreeGrow}{$\mathcal{S}^r$}
    \State \# Return the tree
    \State \Return $Tree$
\EndFunction
\end{algorithmic}
}
\end{algorithm}

\subsubsection{Stopping criteria}
As the growing phase of the algorithm continue, at each iteration the stopping criteria are
evaluated \citep{Rokach2010}. The most common stopping criteria are:
 \begin{itemize}
   \item All examples in the set belong to the same class. Meaning that either $\pi_1=1$ or
    $\pi_1=0$.

  \item The maximum number of iterations have been reached.

  \item The number of examples in $\mathcal{S}$ are less than the minimum number of examples
defined for a split.

  \item The number of examples in $\mathcal{S}^l$ or $\mathcal{S}^r$ are less than the minimum
number of examples defined for a leaf.

  \item The best splitting rule has a Gain lower than a defined threshold.
\end{itemize}

\subsubsection{Pruning techniques}
After a decision tree has been fully grown, there is the big chance that  the algorithm generates a
large tree that is probably over fitting the  training data. In order to solve this in
\citep{Breiman1984a} originally suggest the use of pruning techniques after the tree growing phase.
The overall objective of pruning is to eliminate branches that are not contributing
to the generalization accuracy of the \mbox{tree \citep{Rokach2010}.}

In general pruning techniques start from a fully grown tree, and recursively check if by
eliminating
a $Branch$ there is an $Improvement$ in the error rate $Err$ of the $Tree$.  There are two
main
methods to calculate the $Improvement$ of the error rate $Err$ of the $Tree$, cost-complexity
and  error based pruning:

\begin{itemize}
\item Cost-complexity pruning:

Initially proposed by Breiman  \citep{Breiman1984a}, this method evaluate iterative if the removal
of a $Branch$ improve the error rate $Err$ of a $Tree$, weighted by the difference of the
number of leafs trees.
\begin{equation}\label{eq:8:pruning}
PC_{cc} = \frac{Err(EB(Tree,branch),S)-Err(Tree,\mathcal{S}) }
{\vert Tree\vert-\vert EB(Tree,branch)\vert}
\end{equation}
where $Err(Tree,\mathcal{S})$ calculate the error rate of the $Tree$ in set $\mathcal{S}$ and
$EB(Tree,branch)$ is an auxiliary function that removes $branch$ from $Tree$ and return a new
$Tree$. At each iteration, the current $Tree$ is compared against all possible $Branches$.

\item Error based pruning:

This method proposed by Quinlan  \citep{Quinlan1992} is more pessimistic and instead of  using the
error rate $Err$ calculate the upper bound of the confidence interval   of the error
$Err_{UB}$ calculated assuming a normal distribution with a defined significance level
$Z_\alpha$.
\begin{equation}
PC_{eb} = \frac{Err_{UB}(EB(Tree,branch),\mathcal{S})-Err_{UB}(Tree,\mathcal{S}) }
    {\vert Tree\vert-\vert EB(Tree,branch)\vert}
\end{equation}
where
\begin{align}
    Err_{UB}(Tree,&\mathcal{S})= Err(Tree,\mathcal{S}) & \nonumber \\
    &+Z_\alpha \sqrt{
\frac{Err(Tree,\mathcal{S})(1-Err(Tree,\mathcal{S}))}{\vert \mathcal{S} \vert}} &
\end{align}
Moreover, as a difference from cost-complexity pruning, in the error based method, at each
iteration not all the branches are compared, but only the ones on the same level. Meaning leafs
are only compared against other leafs that are in the same level of iterations when the $Tree$
was constructed. This allows for much faster pruning procedure.
\end{itemize}

In Algorithm~\ref{alg:8:tree_pruning}, the general method of pruning is shown. The pruning
procedure consists in evaluating the $Improvement$ of eliminating all possible branches in a
$Tree$,
and then eliminate the one with the higher $Improvement$ and repeat the process until a stopping
criteria is meet.

\begin{algorithm}\label{alg:8:tree_pruning}
Pseudocode of the tree pruning procedure.
\textnormal{
\begin{algorithmic}[1]
\Function{TreePruning}{$\mathcal{S}, Tree$}
    \State \# For all branches
    \For{$branch \  in \ Tree.branches$}
      \State \# Evaluate the pruning criteria
      \State $Improvements(branch) = $ \Call{PC}{$\mathcal{S}, Tree, branch$}
    \EndFor
    \State \# Select the branch to prune
    \State $branch^* = argmax_{branch} Improvements$
    \State \# Prune selected branch
    \State $Tree = $ \Call{EB}{$TreeTree,branch^*$}
    \State \# Recursively continue pruning the tree
    \State \Call{TreePruning}{$\mathcal{S}, Tree$}
    \State \Return $Tree$
\EndFunction
\end{algorithmic}
}
\end{algorithm}

\subsubsection{Categorical features}
When using continuous features the splitting is done by testing all possible thresholds on the
database and  selecting the one that maximizes the desired measure. Nevertheless, this is not
straight forward when using a  categorical feature, since testing all possible ways of binning the
feature becomes prohibitively time  consuming the feature has many categories. Instead, a common
method for doing this is to calculate $\pi_1$ of each category, then sort it, and applied the
method as a continuous feature \citep{Marslan2009}.

\subsection{Decision tree algorithms}
% \begin{remark}{Decision tree algorithms}
Two main branches of decision trees has being studied in the last years. The main difference is
the  impurity measure used when splitting. First the CART algorithm that is based in the Gini
index, and  later the ID3 and C4.5 which uses the entropy measure.

\begin{itemize}
 \item
  CART or classification and regression trees was introduced by Brieman et al. in 1984
\citep{Breiman1984a}. It is based on using the Gini index as the impurity measure and the tree is
grow until all examples in  each leaf belong to the same class. Afterwards, the tree is pruned
using
the cost-complexity  method \citep{Rokach2010,Marslan2009}.

\item ID3
  algorithm uses entropy as the impurity measure. The growing of the tree stop when all
examples belong  of each leaf belongs to the same class. In ID3 no pruning is applied
\citep{Quinlan1992}.

\item C4.5
  the extension of ID3 both proposed by Quinlan \citep{Quinlan1992}.  Both are similar
regarding the measure used, but C4.5 define the stopping criteria during the growth process
  to be when the number of examples in a set is less than a threshold. Moreover, after the tree is
created  a error based pruning is applied \citep{Rokach2010}.
\end{itemize}
% \end{remark}

\section{Example-Dependent Cost-sensitive Decision Trees}
\label{sec:8:csdt}

  In this section, we first propose a new method to introduce the costs into the decision tree
	induction stage, by creating new-cost based impurity measures. Afterwards, we propose a new
	pruning method based on minimizing the cost as pruning criteria.

	\subsection{Cost-sensitive impurity measures}

		Standard impurity measures such as misclassification, entropy or Gini, take into account the
		distribution of classes of each leaf to evaluate the predictive power of a splitting rule,
		leading to an impurity measure that is based on minimizing the misclassification rate. However,
		as has been previously shown in Chapter~\ref{ch:6} and Chapter~\ref{ch:7}, minimizing
  misclassification does not lead to the same results than minimizing cost. Instead, we are
  interested in measuring how good is a splitting rule in terms of cost not only accuracy. For
  doing that, we propose a new example-dependent cost based impurity measure that takes into
  account the cost matrix of each example.

		We define a new cost-based impurity measure taking into account the costs when all the examples
		in a leaf are classified both as negative using $f_0$ and positive using $f_1$
		\begin{equation}\label{eq:8:cost_impurity}
			I_c(\mathcal{S}) = \min \bigg\{ Cost(f_0(\mathcal{S})), Cost(f_1(\mathcal{S})) \bigg\}.
		\end{equation}
		The objective of this measure is to evaluate the lowest expected cost of a splitting rule.
		Following the same logic, the classification of each set is calculated as the prediction that
		leads to the lowest cost
	  \begin{equation}\label{eq:8:pred}
	    f(\mathcal{S}) =
	    \begin{cases}
	      \phantom{-}0 \phantom{-} \mbox{if} \phantom{-} Cost(f_0(\mathcal{S})) \le
        Cost(f_1(\mathcal{S}))\\
	      \phantom{-}1 \phantom{-}\mbox{otherwise}
	    \end{cases}
	  \end{equation}

		Finally, using the cost-based impurity, the splitting criteria cost based gain of using the
		splitting rule $(\mathbf{x}^j,l^j)$ is calculated with (\ref{eq:8:gain}).

	\subsection{Cost-sensitive pruning}

		Most of the literature in class-dependent cost-sensitive decision tree focuses on using the
		misclassification costs during the construction of the algorithms \citep{Lomax2013}. Only few
		algorithms such as AUCSplit \citep{Ferri2002} have \mbox{included} the costs both during and
		after the construction of the tree. However, this approach only used the class-dependent costs,
		and not the example-dependent costs.

		We propose a new example-dependent cost-based impurity measure, by replacing the error rate
		$Err$ in (\ref{eq:8:pruning}) with the cost of using the $Tree$ on $\mathcal{S}$ i.e., by
    replacing with 	$Cost(f(\mathcal{S}))$.
		\begin{equation}\label{eq:8:cost_pruning}
			PC_{c} = \frac{ Cost(f(\mathcal{S})) - Cost(f^*(\mathcal{S})) }
		  {\vert Tree\vert-\vert EB(Tree,node)\vert} ,
		\end{equation}
		where $f^*$ is the classifier of the tree without the selected node $EB(Tree,node)$.

		Using the new pruning criteria, nodes of the tree that do not contribute to the minimization of
		the cost will be pruned, regardless of the impact of those nodes on the accuracy of the
		algorithm. This follows the same logic as in the proposed cost-based impurity measure, since
		minimizing the misclassification is different than minimizing the cost, and in several
		real-world applications the objectives align with the cost not with the misclassification error.

\section{Experiments}
\label{sec:8:experiments}

\begin{table}
  \centering
  \footnotesize
%   \tabcolsep 4pt
  \begin{tabular}{|c|l|c c|c c |c c|}
    \multicolumn{8}{c}{ }\\
    \cline{3-8}
    \multicolumn{2}{c}{}& \multicolumn{2}{|c}{Fraud Detection} & \multicolumn{2}{|c|}{Direct
    Marketing} &
    \multicolumn{2}{|c|}{Credit Scoring}\\
    \hline
    set&Algorithm & $\%Sav$ & $F_1Score$ & $\%Sav$ & $F_1Score$ & $\%Sav$ & $F_1Score$\\
    \hline
    t&$DT_{not p}$& 31.76&0.4458&19.11&0.2976&18.95&0.3062\\
    &$DT_{err p}$&  31.76&0.4458&19.70&0.3147&18.95&0.3062\\
    &$DT_{cost p}$& 35.89&0.4590&28.08&0.3503&27.59&0.3743\\
    &$CSDT_{not p}$&  70.85&0.2529&69.00&0.2920&49.28&0.3669\\
    &$CSDT_{err p}$&  70.85&0.2529&68.97&0.3193&48.85&0.3669\\
    &$CSDT_{cost p}$& 71.16&0.2522&69.105&0.2878&49.39&0.3684\\
    \hline
    u&$DT_{not p}$& 52.39&0.1502&49.80&0.3374&48.91&0.2983\\
    &$DT_{err p}$&  52.39&0.1502&49.80&0.3374&48.91&0.2983\\
    &$DT_{cost p}$& 70.26&0.2333&53.20&0.3565&49.77&0.3286\\
    &$CSDT_{not p}$&  12.46&0.0761&64.21&0.2830&30.68&0.2061\\
    &$CSDT_{err p}$&  14.98&0.0741&66.21&0.2822&41.49&0.2564\\
    &$CSDT_{cost p}$& 15.01&0.0743&68.07&0.2649&44.89&0.2881\\
    \hline
    r&$DT_{not p}$& 34.39&0.4321&68.59&0.3135&48.97&0.3931\\
    &$DT_{err p}$&  34.39&0.4321&68.79&0.3196&48.97&0.3931\\
    &$DT_{cost p}$& 38.99&0.4478&69.27&0.3274&50.48&0.3501\\
    &$CSDT_{not p}$&  70.85&0.2529&66.87&0.2761&31.25&0.1940\\
    &$CSDT_{err p}$&  70.85&0.2529&67.47&0.2581&40.69&0.2529\\
    &$CSDT_{cost p}$& 71.09&0.2515&68.08&0.2642&44.51&0.2869\\
    \hline
    o&$DT_{not p}$& 31.72&0.4495&60.30&0.3674&47.39&0.3994\\
    &$DT_{err p}$&  31.72&0.4495&60.30&0.3674&47.39&0.3994\\
    &$DT_{cost p}$& 37.35&0.4575&69.44&0.3108&50.92&0.3977\\
    &$CSDT_{not p}$&  70.84&0.2529&68.75&0.2935&41.37&0.2205\\
    &$CSDT_{err p}$&  70.84&0.2529&68.75&0.2935&41.38&0.3457\\
    &$CSDT_{cost p}$& 71.09&0.2515&68.75&0.2935&41.65&0.2896\\
    \hline
  \end{tabular}
    \caption{Results on the three datasets of the cost-sensitive and standard decision tree,
  without pruning ($not p$), with error based pruning ($err p$), and with cost-sensitive pruning
  technique ($cost p$). Estimated using the different training sets: training, under-sampling,
  cost-proportionate rejection-sampling and cost-proportionate over-sampling}
  \label{tab:8:1}
\end{table}

In this section we present the experimental results. First, we evaluate the performance of the
proposed $CSDT$ algorithm and compare it against a classical decision tree ($DT$). We evaluate
the different trees using them without pruning ($not p$), with error based pruning ($err p$),
and also with the proposed cost-sensitive pruning technique ($cost p$). The different algorithms
are trained using the training ($t$), under-sampling ($u$), cost-proportionate rejection-sampling
($r$), and cost-proportionate over-sampling ($o$) datasets. We use the decision trees
implementation of \textit{Scikit-learn} \citep{Pedregosa2011}, and the \textit{CostCla} library,
see Appendix~\ref{ch:A}, for the cost-sensitive algorithms.

For the experiments we use three datasets from three different real world example-dependent
cost-sensitive problems: Credit card fraud detection (see Section~\ref{sec:4:fraud}), credit
scoring (see Section~\ref{sec:4:creditscoring}) and direct marketing (see
Section~\ref{sec:5:directmarketing}). The different datasets are summarized
in \tablename{~\ref{tab:4:databases}} and \tablename{~\ref{tab:5:databases}}.

We evaluate a decision tree constructed using the Gini impurity measure, with and without the
pruning defined in (\ref{eq:8:pruning}). We also apply    the cost-based pruning procedure given
in (\ref{eq:8:cost_pruning}). Lastly, we compared against     the proposed $CSDT$ constructed
using the cost-based impurity measure defined in (\ref{eq:8:cost_impurity}), using the two
pruning procedures.

\begin{figure}[t]
  \centering
  \includegraphics[scale=0.8]{ch8_fig2}
  \caption{Results of the $DT$ and the $CSDT$. For both algorithms, the results are
  calculated with and without both types of pruning  criteria.}
\label{fig:8:2}
\end{figure}
\begin{figure}[t]
  \centering
  \includegraphics[scale=0.8]{ch8_fig3}
  \caption{Average savings on the three datasets of the different cost-sensitive and standard
  decision tree,  estimated using the different training sets: training, under-sampling,
  cost-proportionate rejection-sampling  and cost-proportionate over-sampling.}
  \label{fig:8:3}
\end{figure}

In \figurename{~\ref{fig:8:2}}, the results using the three databases are shown. In particular we
first evaluate the impact of the algorithms when trained using the training set. There is a clear
difference between the savings of the $DT$ and the $CSDT$ algorithms. However, that difference
is not observable on the $F_1Score$ results. Since the $CSDT$ is focused on maximizing the
savings not the accuracy or $F_1Score$. There is a small increase in savings when using the
$DT$ with cost-sensitive pruning. Nevertheless, in the case of the $CSDT$ algorithm, there is
no change when using any pruning procedure, neither in savings or $F_1Score$.

In addition, we also evaluate the algorithms on the different sets, under-sampling,
rejection-sampling and over-sampling. The results are shown in \tablename{~\ref{tab:8:1}}.
Moreover, in \figurename{~\ref{fig:8:3}}, the average results of the different algorithms measured
by savings is shown. The best results are found when using the training set. When using the
under-sampling set there is a decrease in savings of the $CSDT$ algorithm. Lastly, in the case of
the cost-proportionate sampling sets, there is a small increase in savings when using the $CSDT$
algorithm.

\begin{figure}
  \centering
  \includegraphics[scale=0.8]{ch8_fig4}
  \caption{Average tree size (a) and training time (b), of the different cost-sensitive and
  standard decision tree, estimated using the different training sets.}
  \label{fig:8:4}
\end{figure}

\begin{table}
  \centering
  \footnotesize
  \begin{tabular}{|c|l|c c|c c|c c|}
    \cline{3-8}
    \multicolumn{2}{c}{}& \multicolumn{2}{|c|}{Fraud Detection} & \multicolumn{2}{|c|}{Direct
    Marketing}
    & \multicolumn{2}{|c|}{Credit Scoring}\\
    \hline
    set&Algorithm & $\vert Tree\vert$ & Time& $\vert Tree\vert$ & Time& $\vert Tree\vert$ & Time\\
    \hline
    t &$DT_{not p}$&488&2.45&298&1.58&292&1.58\\
    &$DT_{err p}$&488&3.90&298&2.13&292&2.13\\
    &$DT_{cost p}$&446&19.19&291&5.23&280&5.23\\
    &$CSDT_{not p}$&89&1.47&51&0.40&69&0.40\\
    &$CSDT_{err p}$&88&1.87&51&0.64&69&0.64\\
    &$CSDT_{cost p}$&89&1.74&51&0.45&69&0.45\\
    \hline
    u &$DT_{not p}$&308&1.10&198&1.00&167&1.00\\
    &$DT_{err p}$&308&1.43&198&1.14&167&1.14\\
    &$DT_{cost p}$&153&2.59&190&1.34&142&1.34\\
    &$CSDT_{not p}$&14&0.20&23&0.17&42&0.17\\
    &$CSDT_{err p}$&14&0.23&23&0.19&42&0.19\\
    &$CSDT_{cost p}$&14&0.24&23&0.18&42&0.18\\
    \hline
    r &$DT_{not p}$&268&0.98&181&0.90&267&0.90\\
    &$DT_{err p}$&268&1.24&181&0.95&267&0.95\\
    &$DT_{cost p}$&153&2.48&162&1.20&261&1.20\\
    &$CSDT_{not p}$&18&0.22&10&0.07&70&0.07\\
    &$CSDT_{err p}$&18&0.23&10&0.07&70&0.07\\
    &$CSDT_{cost p}$&18&0.23&10&0.07&70&0.07\\
    \hline
    o &$DT_{not p}$&425&2.30&340&1.80&277&1.80\\
    &$DT_{err p}$&425&3.98&340&2.65&277&2.65\\
    &$DT_{cost p}$&364&10.15&288&5.99&273&5.99\\
    &$CSDT_{not p}$&37&1.58&51&0.38&70&0.38\\
    &$CSDT_{err p}$&37&1.90&51&0.45&70&0.45\\
    &$CSDT_{cost p}$&37&1.98&51&0.42&70&0.42\\
    \hline
  \end{tabular}
  \caption{Training time and tree size of the different cost-sensitive and standard decision
  tree, estimated using the different training sets: training, under-sampling,
  cost-proportionate rejection-sampling and cost-proportionate over-sampling, for the three
  databases.}
  \label{tab:8:2}
\end{table}

Finally, we also analyze the different models taking into account the complexity and the
training time. In particular we evaluate the size of each $Tree$. In \tablename{~\ref{tab:8:2}},
and \figurename{~\ref{fig:8:4}}, the results are shown. The $CSDT$ algorithm creates significantly
smaller trees, which leads to a lower training time. In particular this is a result of using the
non weighted gain, the $CSDT$ only accepts splitting rules that contribute to the overall reduction
of the cost, which is not the case if instead the weighted gain was used. Even that the $DT$
with cost pruning, produce a good result measured by savings, it is the one that takes the
longer to estimate. Since the algorithm first creates a big decision tree using the $Gini$
impurity, and then attempt to find a smaller tree taking into account the cost. Measured by
training time, the $CSDT$ is by all means faster to train than the $DT$ algorithm, leading to
an algorithm that not only gives better results measured by savings but also one that can be
trained much quicker than the standard $DT$.


\documentclass{article}
\title{Compsci 590T Homework 7}
\usepackage[margin=1in]{geometry}
\usepackage[loose,nice]{units}
\usepackage{enumitem}
\usepackage{amsmath}
\usepackage{multirow,array}
\newcommand{\pvec}[1]{\vec{#1}\mkern2mu\vphantom{#1}}
\usepackage{amssymb}
\usepackage{cancel}
\usepackage{mathabx}
\usepackage{hyperref}
\newcommand\tab[1][1cm]{\hspace*{#1}}
\usepackage{tikz}
\newcommand*\circled[1]{\tikz[baseline=(char.base)]{
            \node[shape=circle,draw,inner sep=2pt] (char) {#1};}}

\usetikzlibrary{positioning}
\tikzset{main node/.style={circle,fill=blue!20,draw,minimum size=4mm,inner sep=0pt},}
\author{Student: Brian Sun, Collaborator: Thai On}
\date{}

\begin{document}
\maketitle

\section{Problem 1.a}
$(\star)$ Base on the given information we know $A^* = max_{A \in A(N,G)} NW(A)$\\
If $A^*$ is not Pareto efficient, then base on the Pareto efficient statement we know for any other allocation $A'$, it is either the cases that there exists no player $i \in N$ for which $v_i(A'_i) < v_i(A^*_i)$, and the two allocations cannot offer the same utility to all players: $v_i(A^*_i) = v_i(A'_i)$ for all $i \in N$. Therefore we know there should not have any player that $v_i(A'_i) < v_i(A^*_i)$ and at least one player such that $v_i(A'_i) > v_i(A^*_i)$.\\\\
There will have two cases: \\
1. There are some players utility are the same and the rest of the players have greater utility in $A'$ than $A^*$.\\
2. There are no player utility are the same, which all the players have greater utility in $A'$ than $A^*$.\\\\
Case $1:$\\
Set $N_{same}$ be all the players who have the same utility between $A'$ and $A^*$\\
$a.$ $i\in N_{same},\ v_i(A'_i) = v_i(A^*_i)$ therefore $i \in N_{same},\ \prod v_i(A'_i) = \prod v_i(A^*_i)$\\
$b.$ $i \in N \setminus N_{same},\ v_i(A'_i) > v_i(A^*_i)$ therefore $i \in N \setminus N_{same},\  \prod v_i(A'_i) > \prod v_i(A^*_i)$\\
Base on $a,b$ we have $i \in N,\  \prod v_i(A'_i) > \prod v_i(A^*_i)$\\
Since $ \prod v_i(A'_i) > \prod v_i(A^*_i)$ which contradict $(\star)$ therefore case 1 cannot happen.\\\\
Case $2:$\\
Since $i \in N,\ v_i(A'_i) > v_i(A^*_i)$ so $\prod v_i(A'_i) > \prod v_i(A^*_i)$ since this contradict $(\star)$ therefore case 2 cannot happen as well.\\\\
Since both cases cannot happen, therefore $A^*$ is Pareto efficient.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 1.b}

Assume the allocation $A^*$ that maximizes the Nash welfare is not envy-free up to one good then that means for any two players $i,j \in N$, there exists no item $g \in A^*_j$ such that $v_i(A^*_i) < v_i(A^*_j \setminus \{g\})$.\\\\
If this is true, consider the  new allocation $A'$ by move the item $g^* \in A^*_j$ to $A^*_i$. Defined $g^*$ as an item that maximizes the value $v_i(g) - v_j(g), \forall g \in A^*_j$.\\\\
Then we know:
\begin{align*}
NW(A') &= v_i(A^*_i \cup g^*) \times v_j(A^*_j \setminus g^*)\\
&= \left(\sum_{g \in A_i} v_i(g) + v_i(g^*)\right) \times v_j(A^*_j \setminus g^*)\\
&= (v_i(A^*_i) + v_i(g^*)) \times v_j(A^*_j \setminus g^*)\\
&= v_i(A^*_i)v_j(A^*_j \setminus g^*) + v_i(g^*)v_j(A^*_j \setminus g^*)\\\\
NW(A^*) &= v_i(A^*_i) \times (v_j(A^*_j \setminus g^*) + v_j(g^*))\\
&= v_i(A^*_i)v_j(A^*_j \setminus g^*) + v_i(A^*_i)v_j(g^*)
\end{align*}
compare $NW(A^*)$ and $NW(A')$:
\begin{align*}
NW(A^*) &: NW(A')\\
= v_i(A^*_i)v_j(A^*_j \setminus g^*) + v_i(A^*_i)v_j(g^*) &: v_i(A^*_i)v_j(A^*_j \setminus g^*) + v_i(g^*)v_j(A^*_j \setminus g^*)\\
= v_i(A^*_i)v_j(g^*) &: v_i(g^*)v_j(A^*_j \setminus g^*) \longrightarrow (\star)
\end{align*}
Then base on the inequality we know:
$$\frac{v_j(g^*)}{v_i(g^*)} \leq \frac{v_j(A^*_j)}{v_i(A^*_j)}$$
Since $v_i(A^*_i) < v_i(A^*_j \setminus g^*)$ so $v_i(A^*_i \cup g^*) < v_i(A^*_j)$ therefore:
$$\frac{v_j(g^*)}{v_i(g^*)} \leq \frac{v_j(A^*_j)}{v_i(A^*_j)} < \frac{v_j(A^*_j)}{v_i(A^*_i \cup g^*)} \longrightarrow (1)$$\\
Base on $(1)$ we know:
\begin{align*}
\frac{v_j(g^*)}{v_i(g^*)} &< \frac{v_j(A^*_j)}{v_i(A^*_i \cup g^*)}\\
&< \frac{\sum_{g \in A^*_j\setminus g^*}v_j(g) + v_j(g^*)}{\sum_{g \in A^*_i}v_i(g) + v_i(g^*)}\\
&< \frac{v_j(A^*_j \setminus g^*) + v_j(g^*)}{v_i(A^*_i) + v_i(g^*)}\\
v_j(g^*) \times (v_i(A^*_i) + v_i(g^*))&< v_i(g^*) \times (v_j(A^*_j \setminus g^*) + v_j(g^*))\\
v_i(A^*_i)v_j(g^*) + v_i(g^*)v_j(g^*) &< v_i(g^*)v_j(A^*_j \setminus g^*) + v_i(g^*)v_j(g^*)\\
v_i(A^*_i)v_j(g^*) &< v_i(g^*)v_j(A^*_j \setminus g^*) \longrightarrow (\blackdiamond)
\end{align*}
Base on $(\star)$ and $(\blackdiamond)$ we know $NW(A^*) < NW(A')$\\
Since $NW(A')$ is greater than $NW(A^*)$ which contradict $A^*$ is the maximizes the Nash welfare. Therefore $A^*$ that maximizes the Nash welfare is envy-free up to one good

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 2}

Assume $g^*$ is not exists, then that means no matter which item $g \in A_i$ we remove from $A_i$ there will at least exist one player $j \in E_i(A)$ still envy $i$. Which another way to say this is in the envy graph if there exist an node $n_i$ that has incoming edge, then no matter which item you remove from $A_i$, $n_i$ will still has at least one incoming edges.\\\\
Since no matter which item we remove from $A_i$, $n_i$ will still has at least one incoming edges. Which means $i$ got assign item at least one time when $i$ already cost someone envy him.\\\\
Then there will have $2$ possible situations for when we assign item to player $i$ when $i$ already cost someone envy him.\\
1. when there are still players exist that don't have incoming edges.\\
2. when every players have at least one incoming edges.\\\\
For situation 1: this situation cannot happen, since the algorithm will only assign item to player $i$ with no incoming edges. So the algorithm will not assign item to player $i$ since he has incoming edges.\\\\
For situation 2: this situation cannot happen. Since if all players has incoming edges, that means in the envy graph every nodes has at least one path to go back to itself, which there will have at least one cycle on the graph. Since if the graph have cycles, the algorithm will decycle the cycle, to make the envy graph to a acyclic envy graph. Since there are no cycle in the graph, which means there are no path exist that can make any nodes go back to itself, which means there exist at least one node don't have incoming edges. So the situation of every players have at least one incoming edges cannot happen.\\\\
Since both possible situations for when we assign item to $i$ when $i$ already has incoming edges cannot happen in the algorithm. Therefore the contrition is false, so $g^*$ exist.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 3}
Given there are at least one MMS allocation, then to show there will exist at least one Pareto-optimal MMS allocation. We can prove it by pick a random MMS allocation point and try to make Pareto improvement to reach to the Pareto-optimal point. Since Pareto improvement is a action that increase player's utility value without decrease any other player's utility value. So keep repeating this type of action until we get to a stage that no matter what we do, there will always exist at least one player decrease his utility value, which base on the Pareto-optimal definition, this stage will be the Pareto-optimal allocation point.\\\\
There will have two situation:\\
1. when there are only one Pareto-optimal allocation.\\
2. when there are more than one Pareto-optimal allocation.\\\\
Situation (1): If there are only one Pareto-optimal allocation, then we need to check if that Pareto-optimal point satisfy MMS or not.\\\\
$\tab$a. Assume the Pareto-optimal allocation point does not satisfy MMS.\\
Given the fact that there exist at least one MMS allocation, but Pareto-optimal allocation does not satisfy MMS which means we cannot make make any Pareto improvement action from this MMS allocation point to the Pareto-optimal point.\\
Because the Pareto-optimal point does not satisfy MMS, which means there will have at least one value in this point that does not reach the MMS threshold. Therefore we cannot make any Pareto improvement from the MMS allocation point to the Pareto-optimal point without decrease any value in the MMS allocation point. Since the space is bounded therefore the Pareto-optimal point must satisfy MMS so we can reach there starting from the MMS allocation point we pick.\\\\
$\tab$b. Assume the Pareto-optimal allocation point satisfy MMS.\\
Randomly pick a MMS allocation point and try to make Pareto improvement actions. Since Pareto improvement is increase player's utility value without decrease any other player's utility value, therefore any Pareto improvement starting from any MMS allocation point will still satisfy MMS. Therefore starting from a MMS allocation point and try to make Pareto improvement until we cannot, then that stage will be the Pareto-optimal allocation point and is satisfy MMS.\\\\
Situation (2): base on situation 1.a, we know the space is bounded, therefore there must exist at least one Pareto-optimal allocation point that satisfy MMS.\\
Since there are at least one Pareto-optimal allocation point that satisfy MMS, then do the same thing in situation 1.b, then we can reach to the Pareto-optimal allocation point and is satisfy MMS.\\\\
Therefore if there exists at least one MMS allocation, then there exists a Pareto-optimal
MMS allocation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 4.a}

Assume the result can output a not EF-1 allocation, then we can try to make a case that given any two player $a,b \in N$ that $a$ envy $b$ more than one good.\\\\
Therefore two different order:\\
1. The order is $a$ starts ahead of $b$\\
2. The order is $b$ starts ahead of $a$\\\\
For order (1): Every round $a$ will pick his most preferred available course $j^*$ if $v_{aj^*} > 0$. Since $b$ pick the class after $a$, therefore the best $b$ can do is to pick the class that is at least as good as what $a$ pick.\\
Therefore we have:
\begin{align*}
\text{first round: } v_a(c_{a1}) &\geq v_a(c_{b1}) \tab \text{define $c_{ik}$ is the class $i$ pick in the $k$ round}\\
\text{second round: } v_a(c_{a2}) &\geq v_a(c_{b2})\\
\text{third round: } v_a(c_{a3}) &\geq v_a(c_{b3})\\
.\ \ &\tab .\\
.\ \ &\tab .\\
.\ \ &\tab .\\
\text{last round: } v_a(c_{an}) &\geq v_a(c_{bn})\\
\end{align*}
Sum all the value from both side we will get:
$$\sum_{k = 1}^n v_a(c_{ak}) \geq \sum_{k = 1}^n v_a(c_{bk})$$
Since student $a$ value his classes weakly greater than what student $b$ has, therefore if $a$ pick before $b$, we cannot construct an allocation that make $a$ envy $b$ more than one good.\\\\
For order (2): Every round $a$ will pick his most preferred available course $j^*$ if $v_{aj^*} > 0$. Therefore the best $b$ can do after $a$ pick his class is to pick an class that is at least as good as what $a$ pick.\\
Therefore we have:
\begin{align*}
v_a(c_{a1}) &\geq v_a(c_{b2})\\
v_a(c_{a2}) &\geq v_a(c_{b3})\\
v_a(c_{a3}) &\geq v_a(c_{b4})\\
.\ \ &\tab .\\
.\ \ &\tab .\\
.\ \ &\tab .\\
v_a(c_{an-1}) &\geq v_a(c_{bn})\\
\end{align*}
Sum all the value from both side we will get:
$$\sum_{k = 1}^n v_a(c_{ak}) \geq \sum_{k = 2}^n v_a(c_{bk})$$
Since student $a$ value his classes always weakly greater than student $b$'s class without the first class (first class must be $v_a(c_{b1}) \geq v_a(c_{a1})$, since we are trying to construct an allocation that make $a$ envy $b$), therefore if $b$ pick before $a$, we still cannot construct an allocation that make $a$ envy $b$ more than one good.\\\\
Therefore the algorithm output a EF-1 allocation, since both order cannot construct an EF-n allocation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 4.b}
For example:\\
\begin{center}
\begin{tabular}{ | m{4em} | m{4em}| m{4em} | }
  \hline
   & 590T (1) & 589 (1) \\
  \hline
  Alice & 1 & 1 \\
  \hline
  Bob & 2 & 0 \\
  \hline
\end{tabular}
\end{center}
We set the order to be A → B. The algorithm proceeds as follows\\
(i) Alice gets 590T\\
(ii) Bob gets 589\\
(iii) Since all courses are at capacity, we terminate.\\\\
The result is not PO allocation. Since if Bob gets 590T and Alice gets 589, the utility for Bob increase and Alice stay the same, which is a parental improvement. At That stage we cannot make parental improvement any more, therefore Bob gets 590T and Alice gets 589 is the PO allocation which is different than the result that RR algorithm gives us.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Problem 5}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}


\documentclass[color=usenames,dvipsnames]{beamer}\usepackage[]{graphicx}\usepackage[]{color}
% maxwidth is the original width if it is less than linewidth
% otherwise use linewidth (to make sure the graphics do not exceed the margin)
\makeatletter
\def\maxwidth{ %
  \ifdim\Gin@nat@width>\linewidth
    \linewidth
  \else
    \Gin@nat@width
  \fi
}
\makeatother

\definecolor{fgcolor}{rgb}{0, 0, 0}
\newcommand{\hlnum}[1]{\textcolor[rgb]{0.69,0.494,0}{#1}}%
\newcommand{\hlstr}[1]{\textcolor[rgb]{0.749,0.012,0.012}{#1}}%
\newcommand{\hlcom}[1]{\textcolor[rgb]{0.514,0.506,0.514}{\textit{#1}}}%
\newcommand{\hlopt}[1]{\textcolor[rgb]{0,0,0}{#1}}%
\newcommand{\hlstd}[1]{\textcolor[rgb]{0,0,0}{#1}}%
\newcommand{\hlkwa}[1]{\textcolor[rgb]{0,0,0}{\textbf{#1}}}%
\newcommand{\hlkwb}[1]{\textcolor[rgb]{0,0.341,0.682}{#1}}%
\newcommand{\hlkwc}[1]{\textcolor[rgb]{0,0,0}{\textbf{#1}}}%
\newcommand{\hlkwd}[1]{\textcolor[rgb]{0.004,0.004,0.506}{#1}}%
\let\hlipl\hlkwb

\usepackage{framed}
\makeatletter
\newenvironment{kframe}{%
 \def\at@end@of@kframe{}%
 \ifinner\ifhmode%
  \def\at@end@of@kframe{\end{minipage}}%
  \begin{minipage}{\columnwidth}%
 \fi\fi%
 \def\FrameCommand##1{\hskip\@totalleftmargin \hskip-\fboxsep
 \colorbox{shadecolor}{##1}\hskip-\fboxsep
     % There is no \\@totalrightmargin, so:
     \hskip-\linewidth \hskip-\@totalleftmargin \hskip\columnwidth}%
 \MakeFramed {\advance\hsize-\width
   \@totalleftmargin\z@ \linewidth\hsize
   \@setminipage}}%
 {\par\unskip\endMakeFramed%
 \at@end@of@kframe}
\makeatother

\definecolor{shadecolor}{rgb}{.97, .97, .97}
\definecolor{messagecolor}{rgb}{0, 0, 0}
\definecolor{warningcolor}{rgb}{1, 0, 1}
\definecolor{errorcolor}{rgb}{1, 0, 0}
\newenvironment{knitrout}{}{} % an empty environment to be redefined in TeX

\usepackage{alltt}
%\documentclass[color=usenames,dvipsnames,handout]{beamer}

%\usepackage[roman]{../../lab1}
\usepackage[sans]{../../lab1}

\hypersetup{pdftex,pdfstartview=FitV}

%% New command for inline code that isn't to be evaluated
\definecolor{inlinecolor}{rgb}{0.878, 0.918, 0.933}
\newcommand{\inr}[1]{\colorbox{inlinecolor}{\texttt{#1}}}
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\begin{document}

%\setlength\fboxsep{0pt}

%\section{Intro}

\begin{frame}[plain]
%  \maketitle
  \LARGE
  \centering \par
  {\color{RoyalBlue}{Lab 6 -- Randomized Complete Block Design \par}}
  \vspace{1cm}
  \large
%  October 1 \& 2, 2018 \\
%  September 24 \& 25, 2018 \\
  FANR 6750 \\
  \vfill
  \large
  Richard Chandler and Bob Cooper
\end{frame}

\section{Recap}

\begin{frame}
  \frametitle{Recap}
  \large
%  \begin{itemize}%[<+->]
%    \item
  Like one-way ANOVA but experimental units are
  organized into blocks to account for extraneous sources of
  variation \\
      \pause
      \vfill
%    \item
      Blocks could be regions, time periods, individual subjects,
      etc$\ldots$ \\
      \pause
      \vfill
%    \item
      Blocking must occur during the design phase of the study \\
      \pause
      \vfill
%  \end{itemize}
      Additive model:
  \[
    y_{ij} = \mu + \alpha_i + \beta_j + \varepsilon_{ij}
  \]
\end{frame}

\section{Blocked ANOVA ``By Hand''}

\begin{comment}
\begin{frame}[fragile]
  \frametitle{Gypsy Moth Data}
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{a} \hlkwb{<-} \hlnum{3} \hlcom{# Number of treatments}
\hlstd{b} \hlkwb{<-} \hlnum{4} \hlcom{# Number of blocks}
\hlstd{larvae} \hlkwb{<-} \hlkwd{c}\hlstd{(}\hlnum{16}\hlstd{,} \hlnum{3}\hlstd{,} \hlnum{10}\hlstd{,} \hlnum{18}\hlstd{,} \hlnum{25}\hlstd{,} \hlnum{10}\hlstd{,} \hlnum{15}\hlstd{,} \hlnum{32}\hlstd{,} \hlnum{14}\hlstd{,} \hlnum{2}\hlstd{,} \hlnum{16}\hlstd{,} \hlnum{12}\hlstd{)}
\hlstd{treatment} \hlkwb{<-} \hlkwd{factor}\hlstd{(}\hlkwd{rep}\hlstd{(}\hlkwd{c}\hlstd{(}\hlstr{"Bt"}\hlstd{,} \hlstr{"Control"}\hlstd{,} \hlstr{"Dimilin"}\hlstd{),} \hlkwc{each}\hlstd{=b))}
\hlstd{block} \hlkwb{<-} \hlkwd{factor}\hlstd{(}\hlkwd{rep}\hlstd{(}\hlnum{1}\hlopt{:}\hlnum{4}\hlstd{,} \hlkwc{times}\hlstd{=a))}
\hlstd{gypsyData} \hlkwb{<-} \hlkwd{data.frame}\hlstd{(}\hlkwc{caterpillars}\hlstd{=larvae,} \hlkwc{pesticide}\hlstd{=treatment,} \hlkwc{region}\hlstd{=block)}
\hlstd{gypsyData}
\end{alltt}
\begin{verbatim}
##    caterpillars pesticide region
## 1            16        Bt      1
## 2             3        Bt      2
## 3            10        Bt      3
## 4            18        Bt      4
## 5            25   Control      1
## 6            10   Control      2
## 7            15   Control      3
## 8            32   Control      4
## 9            14   Dimilin      1
## 10            2   Dimilin      2
## 11           16   Dimilin      3
## 12           12   Dimilin      4
\end{verbatim}
\begin{alltt}
\hlkwd{write.csv}\hlstd{(gypsyData,} \hlstr{"gypsyData.csv"}\hlstd{,} \hlkwc{row.names}\hlstd{=}\hlnum{FALSE}\hlstd{)}
\end{alltt}
\end{kframe}
\end{knitrout}
\end{frame}
\end{comment}

\begin{frame}[fragile]
  \frametitle{Gypsy Moth Data}
%  \small
  \footnotesize
\begin{knitrout}\footnotesize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{gypsyData} \hlkwb{<-} \hlkwd{read.csv}\hlstd{(}\hlstr{"gypsyData.csv"}\hlstd{)}
\hlstd{gypsyData}\hlopt{$}\hlstd{region} \hlkwb{<-} \hlkwd{factor}\hlstd{(gypsyData}\hlopt{$}\hlstd{region)} \hlcom{# Convert to factor}
\hlstd{gypsyData}
\end{alltt}
\begin{verbatim}
##    caterpillars pesticide region
## 1            16        Bt      1
## 2             3        Bt      2
## 3            10        Bt      3
## 4            18        Bt      4
## 5            25   Control      1
## 6            10   Control      2
## 7            15   Control      3
## 8            32   Control      4
## 9            14   Dimilin      1
## 10            2   Dimilin      2
## 11           16   Dimilin      3
## 12           12   Dimilin      4
\end{verbatim}
\end{kframe}
\end{knitrout}
\vfill
\alert{Note:} Numeric grouping variables must be coded as factors.
\end{frame}

\begin{frame}[fragile]
  \frametitle{Compute the means}
  \small
{Grand mean ($\bar{y}_.$)}
\begin{knitrout}\footnotesize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{caterpillars} \hlkwb{<-} \hlstd{gypsyData}\hlopt{$}\hlstd{caterpillars}
\hlstd{(grand.mean} \hlkwb{<-} \hlkwd{mean}\hlstd{(caterpillars))}
\end{alltt}
\begin{verbatim}
## [1] 14.41667
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
%\vspace{0.3cm}
{ Treatment means ($\bar{y}_i$)}
\begin{knitrout}\footnotesize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{pesticide} \hlkwb{<-} \hlstd{gypsyData}\hlopt{$}\hlstd{pesticide}
\hlstd{(treatment.means} \hlkwb{<-} \hlkwd{tapply}\hlstd{(caterpillars, pesticide, mean))}
\end{alltt}
\begin{verbatim}
##      Bt Control Dimilin
##   11.75   20.50   11.00
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
%\vspace{0.3cm}
{ Block means ($\bar{y}_j$)}
\begin{knitrout}\footnotesize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{region} \hlkwb{<-} \hlstd{gypsyData}\hlopt{$}\hlstd{region}
\hlstd{(block.means} \hlkwb{<-} \hlkwd{tapply}\hlstd{(caterpillars, region, mean))}
\end{alltt}
\begin{verbatim}
##        1        2        3        4
## 18.33333  5.00000 13.66667 20.66667
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Treatment sums-of-squares}
  {\Large
  \[
  b \times \sum_{i=1}^a (\bar{y}_i - \bar{y}_.)^2
  \]
  }
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{b} \hlkwb{<-} \hlnum{4}
\hlstd{b} \hlkwb{<-} \hlkwd{nlevels}\hlstd{(region)}
\hlstd{SS.treat} \hlkwb{<-} \hlstd{b}\hlopt{*}\hlkwd{sum}\hlstd{((treatment.means} \hlopt{-} \hlstd{grand.mean)}\hlopt{^}\hlnum{2}\hlstd{)}
\hlstd{SS.treat}
\end{alltt}
\begin{verbatim}
## [1] 223.1667
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Block sums-of-squares}
  {\Large
  \[
  a \times \sum_{j=1}^b (\bar{y}_j - \bar{y}_.)^2
  \]
  }
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{a} \hlkwb{<-} \hlkwd{nlevels}\hlstd{(pesticide)}
\hlstd{SS.block} \hlkwb{<-} \hlstd{a}\hlopt{*}\hlkwd{sum}\hlstd{((block.means} \hlopt{-} \hlstd{grand.mean)}\hlopt{^}\hlnum{2}\hlstd{)}
\hlstd{SS.block}
\end{alltt}
\begin{verbatim}
## [1] 430.9167
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Within groups sums-of-squares}
  {\Large
    \[
    \sum_{i=1}^a \sum_{j=1}^b (y_{ij} - \bar{y}_i - \bar{y}_j + \bar{y}_.)^2
    \]
  }
%  \pause
\begin{knitrout}\small
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{treatment.means.long} \hlkwb{<-} \hlkwd{rep}\hlstd{(treatment.means,} \hlkwc{each}\hlstd{=b)}
\hlstd{block.means.long} \hlkwb{<-} \hlkwd{rep}\hlstd{(block.means,} \hlkwc{times}\hlstd{=a)}
\hlstd{SS.within} \hlkwb{<-} \hlkwd{sum}\hlstd{((caterpillars} \hlopt{-} \hlstd{treatment.means.long} \hlopt{-}
                  \hlstd{block.means.long} \hlopt{+} \hlstd{grand.mean)}\hlopt{^}\hlnum{2}\hlstd{)}
\hlstd{SS.within}
\end{alltt}
\begin{verbatim}
## [1] 114.8333
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
\vfill
{\bf NOTE:} For this to work, \inr{treatment.means} and
\inr{block.means} must be in the same order as in the original
data. \\
\end{frame}

\begin{frame}[fragile]
  \frametitle{Create ANOVA table}
\begin{knitrout}\small
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{df.treat} \hlkwb{<-} \hlstd{a}\hlopt{-}\hlnum{1}
\hlstd{df.block} \hlkwb{<-} \hlstd{b}\hlopt{-}\hlnum{1}
\hlstd{df.within} \hlkwb{<-} \hlstd{df.treat}\hlopt{*}\hlstd{df.block}
\hlstd{ANOVAtable} \hlkwb{<-} \hlkwd{data.frame}\hlstd{(}
    \hlkwc{df} \hlstd{=} \hlkwd{c}\hlstd{(df.treat, df.block, df.within),}
    \hlkwc{SS} \hlstd{=} \hlkwd{c}\hlstd{(SS.treat, SS.block, SS.within))}
\hlkwd{rownames}\hlstd{(ANOVAtable)} \hlkwb{<-} \hlkwd{c}\hlstd{(}\hlstr{"Treatment"}\hlstd{,} \hlstr{"Block"}\hlstd{,} \hlstr{"Within"}\hlstd{)}
\hlstd{ANOVAtable}
\end{alltt}
\begin{verbatim}
##           df       SS
## Treatment  2 223.1667
## Block      3 430.9167
## Within     6 114.8333
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Create ANOVA table, continued$\dots$}
  \small
  { Mean squares}
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{MSE} \hlkwb{<-} \hlstd{ANOVAtable}\hlopt{$}\hlstd{SS} \hlopt{/} \hlstd{ANOVAtable}\hlopt{$}\hlstd{df}
\hlstd{ANOVAtable}\hlopt{$}\hlstd{MSE} \hlkwb{<-} \hlstd{MSE}
\end{alltt}
\end{kframe}
\end{knitrout}
\pause
%\vspace{0.3cm}
{ $F$ values}
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{F} \hlkwb{<-} \hlkwd{c}\hlstd{(MSE[}\hlnum{1}\hlstd{]}\hlopt{/}\hlstd{MSE[}\hlnum{3}\hlstd{], MSE[}\hlnum{2}\hlstd{]}\hlopt{/}\hlstd{MSE[}\hlnum{3}\hlstd{],} \hlnum{NA}\hlstd{)}
\hlstd{ANOVAtable}\hlopt{$}\hlstd{F} \hlkwb{<-} \hlstd{F}
\end{alltt}
\end{kframe}
\end{knitrout}
\pause
%\vspace{0.3cm}
{ $P$-values}
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{P} \hlkwb{<-} \hlkwd{c}\hlstd{(}\hlnum{1} \hlopt{-} \hlkwd{pf}\hlstd{(F[}\hlnum{1}\hlstd{],} \hlnum{2}\hlstd{,} \hlnum{6}\hlstd{),} \hlnum{1} \hlopt{-} \hlkwd{pf}\hlstd{(F[}\hlnum{2}\hlstd{],} \hlnum{3}\hlstd{,} \hlnum{6}\hlstd{),} \hlnum{NA}\hlstd{)}
\hlstd{ANOVAtable}\hlopt{$}\hlstd{P} \hlkwb{<-} \hlstd{P}
\hlkwd{round}\hlstd{(ANOVAtable,} \hlnum{3}\hlstd{)}
\end{alltt}
\begin{verbatim}
##           df      SS     MSE     F     P
## Treatment  2 223.167 111.583 5.830 0.039
## Block      3 430.917 143.639 7.505 0.019
## Within     6 114.833  19.139    NA    NA
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Reminder about $P$-values}
  \footnotesize

Critical value
\begin{knitrout}\scriptsize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlkwd{qf}\hlstd{(}\hlnum{0.95}\hlstd{,} \hlkwc{df1}\hlstd{=}\hlnum{2}\hlstd{,} \hlkwc{df2}\hlstd{=}\hlnum{6}\hlstd{)} \hlcom{# 95% of the distribution is before this value of F}
\end{alltt}
\begin{verbatim}
## [1] 5.143253
\end{verbatim}
\end{kframe}
\end{knitrout}
$P$-value
\begin{knitrout}\scriptsize
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlnum{1}\hlopt{-}\hlkwd{pf}\hlstd{(F[}\hlnum{1}\hlstd{],} \hlkwc{df1}\hlstd{=}\hlnum{2}\hlstd{,} \hlkwc{df2}\hlstd{=}\hlnum{6}\hlstd{)} \hlcom{# Proportion of the distribution beyond this F value}
\end{alltt}
\begin{verbatim}
## [1] 0.03921514
\end{verbatim}
\end{kframe}
\end{knitrout}
\begin{center}
  \includegraphics[width=9cm]{figure/Fdist-1}
\end{center}
\end{frame}

\section{Blocked ANOVA Using {\tt aov}}

\begin{frame}[fragile]
  \frametitle{Using {\tt aov}}
  \small
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{aov1} \hlkwb{<-} \hlkwd{aov}\hlstd{(caterpillars} \hlopt{~} \hlstd{pesticide} \hlopt{+} \hlstd{region, gypsyData)}
\hlkwd{summary}\hlstd{(aov1)}
\end{alltt}
\begin{verbatim}
##             Df Sum Sq Mean Sq F value Pr(>F)
## pesticide    2  223.2  111.58   5.830 0.0392 *
## region       3  430.9  143.64   7.505 0.0187 *
## Residuals    6  114.8   19.14
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlkwd{round}\hlstd{(ANOVAtable,} \hlnum{3}\hlstd{)}
\end{alltt}
\begin{verbatim}
##           df      SS     MSE     F     P
## Treatment  2 223.167 111.583 5.830 0.039
## Block      3 430.917 143.639 7.505 0.019
## Within     6 114.833  19.139    NA    NA
\end{verbatim}
\end{kframe}
\end{knitrout}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Using {\tt aov}}
  { Look what happens if we ignore the blocking variable \par}
  \small
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{aov2} \hlkwb{<-} \hlkwd{aov}\hlstd{(caterpillars} \hlopt{~} \hlstd{pesticide, gypsyData)}
\hlkwd{summary}\hlstd{(aov2)}
\end{alltt}
\begin{verbatim}
##             Df Sum Sq Mean Sq F value Pr(>F)
## pesticide    2  223.2  111.58    1.84  0.214
## Residuals    9  545.8   60.64
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
\vfill
%\centering
  { Why is the effect of pesticide no longer significant? \\}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Treating block effects as random effects}
%  \small
  \footnotesize
\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.878, 0.918, 0.933}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlstd{aov3} \hlkwb{<-} \hlkwd{aov}\hlstd{(caterpillars} \hlopt{~} \hlstd{pesticide} \hlopt{+} \hlkwd{Error}\hlstd{(region), gypsyData)}
\hlkwd{summary}\hlstd{(aov3)}
\end{alltt}
\begin{verbatim}
##
## Error: region
##           Df Sum Sq Mean Sq F value Pr(>F)
## Residuals  3  430.9   143.6
##
## Error: Within
##           Df Sum Sq Mean Sq F value Pr(>F)
## pesticide  2  223.2  111.58    5.83 0.0392 *
## Residuals  6  114.8   19.14
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
\end{verbatim}
\end{kframe}
\end{knitrout}
\pause
\vfill
The values of the ANOVA table are the same as before, and there is
no reason to use random effects here if interest only lies in
testing the null hypothesis concerning pesticides. Later, we will see
cases where it is important to use random and fixed effects.
\end{frame}

\begin{frame}
  \frametitle{Assignment: Due before lab next week}
  \tiny
%  \scriptsize
  Plantations of {\it Pinus caribaea} were established at
  four locations on Puerto Rico.  %As part of a spacing study,
  Four spacings were used at each location to determine the effect of
  stocking density on tree height. Twenty years after the
  plantations were established, the following tree heights were
  recorded: \\ %par
  \vfill
  {  \centering
%  \begin{columns}
  %   \begin{center}
%  \begin{column}{0.5\textwidth}
    \tiny %\scriptsize
    \begin{tabular}{lcc}
      \hline
      Location  & Spacing (ft) & Height (ft) \\
      \hline
      Caracoles & 5            & 72          \\
                & 7            & 80          \\
                & 10           & 85          \\
                & 14           & 91          \\
      Utado     & 5            & 75          \\
                & 7            & 90          \\
                & 10           & 94          \\
                & 14           & 112         \\
      Guzman    & 5            & 88          \\
                & 7            & 95          \\
                & 10           & 94          \\
                & 14           & 91          \\
      Lares     & 5            & 79          \\
                & 7            & 94          \\
                & 10           & 104         \\
                & 14           & 106         \\
      \hline
    \end{tabular} \\
  }
    Create an \R~script to the address the following:
    \begin{enumerate}[(1)]
      \item What are the null and alternative hypotheses?.
      \item Test for effects of location and spacing on plant height
        using the \inr{aov} function. Do the ANOVA again but without
        \inr{aov}. Treat the block effects as fixed, not random. HINTS:
        \begin{itemize}
          \tiny
          \item Spacing must be treated as a factor.
          \item You must put the group means and block means in the
            correct order when computing the sums-of-squares.
        \end{itemize}
      \item Perform a Tukey test to determine which spacings differ.
      \item Summarize the main results in 2-3 sentences. Upload the
        script to ELC the day before your next lab.
    \end{enumerate}
\end{frame}

\end{document}