\documentclass[11pt]{article}
\usepackage{hyperref}
\usepackage{ifthen}
\usepackage{enumerate,fullpage}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{version}
\usepackage{pdfsync}

\pagestyle{plain}

\newcommand{\points}[1]{\textbf{[#1 Points]}}

\begin{document}

\title{CS 186 Assignment 5: Sponsored Search Auctions}
	\author{David C. Parkes  \\
	School of Engineering and Applied Sciences, Harvard University \\
	Out: Wednesday, March 25 2015\\
	Due: {\bf 5pm} sharp: {\bf Wednesday, April 1 2015}
\\({\bf 5pm} sharp: Thursday, April 2nd for Extension students)\\
	Submissions to {\tt Dropbox for assignments in iSites}}
\date{}

\maketitle

\noindent \points{Total: 135} {\em Submit a single zip file containing
  your code and your writeup.} The total number of points includes a
small bonus for good performance in an in-class tournament.

{\bf College}: This is a group-assignment to be completed by groups of
up to 2 students. While you are permitted to discuss your agent
designs with other students, each group must develop its own code and
write-up its own submission. (If you need help finding a partner
please post to Piazza.)

{\bf Extension}: This is an individual project unless you prefer to
work with a partner. You may also post to Piazza if you would like to work with someone else.

\section{Introduction}

You will program bidding agents to participate in a generalized
second-price (GSP) position auction.  The first agent you will design
uses a {\em balanced bidding} strategy, which is inspired by the
spiteful bidding strategy (Defn.  9.3) in the reading.  You will also
design an agent to do as well as possible in a competitive
environment, and in the presence of daily budget constraints on how
much it can spend. In this assignment you will also explore the effect
that the design of payment rules has on revenue.

\section{Setup}
	\begin{description}

		\item[Generating .py-files:] Pick a group name, perhaps based on your initials, so it will be unique in the class. Run \verb+python start.py GROUPNAME+, substituting your group name for \verb+GROUPNAME+. This will create appropriately named template files for your clients. For example, if your group name was ``abxy'':
			\begin{verbatim}> python start.py abxy
				Copying bbagent_template.py to abxybb.py...
				Copying bbagent_template.py to abxybudget.py...
				All done.  Code away!
			\end{verbatim}
			In each of these newly generated files, you will need to change the class name \verb+BBAgent+  to your teamname plus client specification (e.g. for the balanced bidding agent: \verb+class Abxybb+).

		\item[The simulator:] You are given an ad-auction simulator.
The ad-auction does not model bidder effects, so that the only thing that
matters in determining the probability of a click and thus the expected
number of clicks received by an ad in a time period is the position.

The simulator works as follows:
			\begin{itemize}
				\item Time proceeds in discrete periods ($0, 1, 2, \ldots, 47)$. Each period simulates 30 minutes and includes multiple auctions,
and the 48 periods correspond to a day. The simulator can be used to simulate
multiple days by increasing the number of iterations.

				\item In the first period, an agent's
                                  bid is queried through
                                  \verb+initial_bid()+ whereas for all
                                  subsequent periods, the bid is
                                  queried through a call to
                                  \verb+bid()+. The simulator tracks
                                  money and values in integer numbers
                                  of cents.
The same bid value is used for every auction that
occurs within a period.
To encourage competition, the number of available positions is one less than
the number of agents in the market.
%
%
\item Each period is simulated by collecting bids, assigning positions,
  determining the expected number of clicks received by each bidder,
  and determining payments and utilities.  Sometimes there is a {\em
    reserve price} (see Section~4). The positions are assigned
in order of submitted bid.
%
\item Number of clicks: Let $c_j^t$ denote the number of clicks
  received by an ad in position $j$ (from 1 to the number of
  positions) in period $t$. For the top position, this
  follows a cosine curve:
%
\begin{align}
 c_1^t &= \mbox{round}(30 \cos(\frac{\pi t}{24}) + 50),
\quad t = 0, 1, \ldots,
\end{align}
%
where $\mbox{round}(x)$ returns the nearest integer value to $x$.
The number of clicks starts at 80, falls to 20 by period 24,
and  increases  to 80 by the end of a day.

The number of clicks decreases according to a multiplicative
position effect. The number of clicks received by an ad in
position $j$ ($j>1$) in period $t$ is
%
\begin{align}
c_j^t &= 0.75^{(j-1)}c_1^t,
\end{align}
%
so that it decreases by a factor of 75\% from  position to position.
%
\item The price in the GSP auction depends on the next highest bid.
Given this, the utility in period $t$ to
agent $i$ occupying position $j$
is
%
%
\begin{align}
u_i^t &=  c^t_j(w_i - p^t_j) = c^t_j(w_i - b^t_{j+1}),
\end{align}
%
where $w_i$ is the per-click value,
and $p^t_j$ the
price for position $j$ and equal
to $b^t_{j+1}$, which
is the amount of the next highest
bid (or zero
or the reserve if there
is no such bid.)
This is the number of clicks received multiplied by the
utility (=value-price) per click.
%
%
\item Budget constraint: The total payment by an agent in position $j$
  in period $t$ is $c_j^t b^t_{j+1}$. Each agent has a {\em daily
    budget constraint}, but for most of the assignment this will be
  high enough not to matter. For the competition this is \$600/day and
  it will matter.
 An agent  with budget still available at the start
of a period is still allowed to bid. For this reason, an
agent  is allowed to slightly over-spend its budget.
Once the budget is exhausted the
bid in all
subsequent periods  of the day
must be \$0 (the
  simulator will ensure this if you try to bid more than \$0.)
%
\item At the start of each day, the per-click value $w_i$ of an agent
  is sampled independently from the uniform distribution on
  $[0.25,1.75]$.  The simulator also runs additional permutations of
  value assignments to agents to improve the statistical
  significance of the estimated utility.
If the number of agents is at most 5 then
all permutations
 of values to agents
are tested. Otherwise,
  120 random permutations are tested. (You can change this threshold
  with the \verb+--perms+ option)
%
%
\item The score of an agent is the total utility over all 48 periods,
  and averaged over all permutations of values to agents that are
  used.  In addition, multiple iterations can be run. This has the
  effect of repeating the test over multiple days.  In this case the
  score is the average total utility over all permutations and all
  days.  For a single day, the total utility to agent $i$ with value
  $w_i$ is $\sum_{t=0}^{47} u_i^t = \sum_{t=0}^{47} c_j^t(w_i -
  b_{j+1}^t)$, where $j$ is the position assigned in period $t$ (and
  $c_j^t=0$ if the agent is not assigned to a position in period $t$.
{\em Any unspent  budget has no effect on utility or score}.
			\end{itemize}
%
		\item[Source code:] Familiarize yourself with the provided code. You will need to change the agent created by \verb+start.py+, as well as the file \verb+vcg.py+. You'll want to take a look at the simple truthful-bidding agent in \verb+truthful.py+, as well as the the GSP implementation in \verb+gsp.py+.

		\item[Testing:] Here are some initial test commands to run. The following command gives a list of helpful command line parameters:
			\begin{verbatim}
				> ./auction.py --h
			\end{verbatim}

			The following command can be used to test the code. It runs the auction with five truthful agents for two periods (rather than the default of 48 periods). The \verb+--perms 1+ command forces the simulator to assign only one permutation of value to the agents.
			\begin{verbatim}
				> ./auction.py --loglevel debug --num-rounds 2 --perms 1 Truthful,5
			\end{verbatim}

The number of periods defaults to 48 if this is not set.

The following command runs the auction with a reserve price of 40
cents. The \verb+--iters 2+ command specifies that the 48 periods will
be repeated twice, with different value samples (i.e., two days.)
%
The
%
\verb+--seed INT+ (where INT is any integer) command
 sets the seed of the random number generators in the simulator and
allows for repeatability.
%
			\begin{verbatim}
				> ./auction.py  --perms 1 --iters 2 --reserve=40 --seed 1  Truthful,5
			\end{verbatim}

		\item[Tips]
			\begin{itemize}
				\item {\em Permutations}: If you're
running an experiment on a population of agents that each have the
same strategy,
use \verb+--perms 1+ to make the code run faster. In this case this is
probably OK because you're likely comparing the average utility or
revenue of the auction, and not looking at specific agents.
%
				\item {\em Pseudo-random numbers}: If you're trying to track down a bug, or understand what's going on with some specific case, use \verb+--seed INT+ to fix the random seed and get repeatable value distributions and tie breaking.
			\end{itemize}

\item[Comments]
\begin{itemize}
\item {\em Numbering convention}: The slots in this write-up are 1-indexed
(top slot is slot 1) but 0-indexed in the code. Don't be confused by this!
%
\item {\em Ties}: In the case of two tied bids one of the two is randomly chosen to be allocated th higher slot.
%
\item
{\em State}: The agent you write can access history from the previous period within each day.
This could allow for more sophisticated strategies.

\item {\em Workarounds}: The code is designed so that it's hard to mess up the main simulation accidentally, but because everything is in the same process, it is still possible to cheat by directly modifying the simulation data structures and such. Don't!
%
		\item {\em Bugs}: If you find bugs in the code,
please let us know. If you want to improve the logging or stats or performance or add animations or graphs, feel free :) Send those changes along too.
%
    		\item {\em Questions}:
If something is unclear about the assignment, please ask a question on Piazza. Please don't post solution code but it is OK
to post code snippets are fine (use your judgment).
    	\end{itemize}
	\end{description}

\section{The Balanced Bidding Agent}

	\begin{enumerate}

		\item\points{30} {\bf Designing a bidding agent}

You are given a truthful bidding agent in \verb+truthful.py+.

In
\verb+GROUPNAMEbb.py+, write an agent that best-responds to the
bids of agents in the previous period. In particular, the agent follows
a {\em balanced bidding} strategy.  Consider period $t$.
Let $b_{-i}^{t-1}$ denote the bids from the other
agents in the  period $t-1$.
Suppose there are $m$ positions. Balanced-bidding
proceeds as follows:
%
\begin{itemize}
\item Given bids $b_{-i}^{t-1}$,
target the position $k^\ast$ that maximizes
%
\begin{align}
\arg\max_{k\in\{1,\ldots,m\}} [q_k\cdot (w_i - p_k)],
\end{align}
%
where $q_k$ is the position effect (this falls by a multiplicative
factor of 0.75 each slot) and $p_k$ is the price the agent would pay
for position $k$ given bids $b^{t-1}_{-i}$.

%
\item ({\em Not expecting to win})
IF price $p_{k^\ast}\geq w_i$ in this target position,
then bid $b^t_i=w_i$ in period $t$.
%
		\item Otherwise:
%
\begin{itemize}
\item[(a)] ({\em Not going for the top}) If target position $k^\ast>1$,
then set  bid $b^t_i$ to satisfy
%
\begin{align}
q_{k^\ast}(w_i - p_{k^\ast}) = q_{k^\ast - 1}(w_i - b^t_i).
\end{align}
%
%
\item[(b)] ({\em Going for the top}) If $k^\ast=1$,
then bid
$b^t_i=w_i$.
\end{itemize}
	\end{itemize}

        This strategy is motivated by the spiteful bidding strategy in
        the reading.\footnote{Cary et al.  ``Greedy bidding strategies
          for keyword auctions" {\em Proc.  ACM EC 2007} pp 262--271,
          show that for agents with constant values, balanced bidding
          converges to the spiteful Nash equilibrium of the GSP
          auction.}
%
	The file \verb+TEAMNAMEbb.py+ provides a skeleton of a
        balanced bidding agent. You will need to complete
the code to compute the  expected
        utility for each position and the optimal bid.

		\item \points{20} {\bf Experimental Analysis}

{\bf 			To answer the following questions, run the
			simulation with 5 agents. By default the
			budget is \$5000, which is
not binding. Leave it
			this way!}

			\begin{enumerate}
				\item \points{10} What is the average utility of a population of truthful agents? What is the average utility of a population of balanced bidding agents? {\em  Compare the two cases and explain your findings.}

				Make use of the \verb+--perms+, \verb+--seed+, and \verb+--iters+ commands, e.g. \verb+--perms 1 --seed 2+ \verb+--iters 200+ would be a good starting point.

				\item \points{10} In addition, what is the
average utility of one balanced bidding agent against 4 truthful agents, and
one truthful agent against 4 balanced bidding agents?
For the new experiment,
make use of the \verb+--seed+, and \verb+--iters+ commands, but you will now
want to run multiple permutations. Note that you can add multiple agents types using:
			\begin{verbatim}
				> ./auction.py  --perms 10 --iters 200 Truthful,4 abxybb,1
			\end{verbatim}
%

{\em What does this suggest about the incentives to follow the truthful vs. the balanced bidding strategy?}

			\end{enumerate}

\end{enumerate}

\section{Experiments with Revenue: GSP vs VCG auctions}

In this section we compare the revenue properties of the GSP and VCG
auctions with different reserve prices.
%
A reserve price, $r>0$, sets a minimum price for any position.  Used
carefully, reserve prices can increase revenue.
%
The reserve price in the GSP works as follows: only bids (weakly)
above $r$ can be allocated.  The agent in the lowest allocated
position pays the maximum of the reserve price and the maximum bid of
unallocated bidders. The VCG auction works in a similar way and is
explained below.\footnote{One way to think about it is that the
  reserve is made operational by including a ``dummy bidder'' whose
  bid is the amount of the reserve, and ignoring all bids with value
  less than $r$.}

\begin{enumerate}
		\item[3.] \points{55} {\bf Auction Design and Reserve Prices}

{\bf Run all simulations with 5 agents. Leave the budget to its default
of \$5000.}

			\begin{enumerate}
\item \points{20} Complete the code that runs
the VCG auction in \verb+vcg.py+. The
allocation rule is already implemented. You need to implement the
payment rule. Because of the reserve price, it is most convenient to
use the recursive form
of the VCG payment rule (see the proof of Theorem~9.3 in the
reading).

In particular, suppose there are 3 bidders with bids weakly greater
than the reserve price, and $b_1\geq b_2\geq b_3$, and say that $b_4$
is the fourth highest bid. Bidders 1--3 are allocated the top 3
positions.

Let $t_{\mathrm{vcg},i}(b)$ denote the expected payment by bidder $i$
in a single auction given bid profile $b$.  For bidder 3, this is
$t_{\mathrm{vcg},3}(b)=q_3 \max(r,b_{4})$. For bidders 1 and 2, in
positions 1 and 2 respectively, this is
$t_{\mathrm{vcg},i}(b)=(q_i-q_{i+1})b_{i+1} +t_{\mathrm{vcg},i+1}(b)$.

				\item \points{10} What is the auctioneer's revenue under GSP with no reserve price when all the agents use the balanced bidding strategy? What happens as the reserve price increases? What is the revenue-optimal reserve price?

You can set the reserve price in the simulation with the command line argument \verb+--reserve INT+ (where INT is the reserve price in cents). Also use \verb+--perms 1+ and \verb+--iters 200+.

\item \points{10} What is the auctioneer's revenue under VCG with no
reserve price when all agents are truthful? What happens as the
reserve price increases? Explain your findings and compare with the
results of part (a).

				Again use the \verb+--perms+, \verb+--seed+, and \verb+--iters+ commands, e.g. \verb+--perms 1 --seed 2+ \verb+--iters 200+.

\item \points{10} Fix the reserve price to zero.
Explore what might happen if a search engine
switched over from the GSP to VCG design.
For this,
run the balanced bidding agents in
GSP, and at period 24, switch to VCG, by using the \verb+--mech=switch+ parameter. What happens to the revenue?
%

				Again use the \verb+--perms+, \verb+--seed+, and \verb+--iters+ commands, e.g. \verb+--perms 1 --seed 2+ \verb+--iters 200+.

                              \item \points{5} In one paragraph,
state what
                                did you learn from these exercises
                                about agent design and implementation,
                                mechanism design, and revenue? (There
                                is no specific right answer).
			\end{enumerate}

\end{enumerate}

\section{The Competition}

\begin{enumerate}
		\item[4.] \points{30} {\bf Budget constraints}

                  The balanced bidding agent does not consider the
                  budget constraint when deciding how to bid.  In the
                  final part of the assignment, your task is to design
                  a {\em budget-aware agent}.

                  This agent will compete in the simulated GSP auction
                  against the agents submitted by other groups. You
                  might like to test your design against a variety of
                  other strategies.

                  For example, you could write an agent that measures
                  competition, or
tries to drive up the payments of other
                  agents so that they pay more and exhaust their
budgets! Example ideas include:
%
\begin{itemize}
\item Trying not to bid too much when
click-through rates are low.
%
\item Try to bid when other agents are not bidding very much
and the price is lower.
%
\end{itemize}

{\bf For the purpose of the competition, the daily budget constraint
  will be set to \$600 (use the \texttt{--budget} flag). We will run a
  GSP auction with a small reserve price.  Auctions will contain
  around 5 agents.}

The competition will be structured as a tournament, with agents placed
into groups of 5 or 6 and the top few agents making it into a
semi-final and final. The precise structure of the tournament will
depend on the number of agents submitted.

		\begin{enumerate}
			\item \points{25} In \verb+TEAMNAMEbudget.py+, write your  competition agent.

Describe in a few sentences how it works, why
it is designed this way,
and how you expect it to perform in the class competition.
		 {\em You are not expected to spend many hours on writing an optimal  agent, unless you want to. Consider a few possible strategies, try them out, pick the best one.}

			\item \points{5} Win the competition (!)
		Likely parameters for the competition are

		\verb+./auction.py --num-rounds 48 --mech=gsp --reserve=10 --iters 200+

(followed by the list of submitted agents)
		\end{enumerate}
	\end{enumerate}

\end{document}


\begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}9}]:} \PY{n}{train\PYZus{}path} \PY{o}{=} \PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{/Users/hn/Documents/GitHub/job\PYZus{}inventory/Hossein/Code\PYZus{}Question/Zillow/TRAINING.csv}\PY{l+s+s1}{\PYZsq{}}
        \PY{n}{test\PYZus{}path} \PY{o}{=} \PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{/Users/hn/Documents/GitHub/job\PYZus{}inventory/Hossein/Code\PYZus{}Question/Zillow/TEST.csv}\PY{l+s+s1}{\PYZsq{}}

        \PY{n}{zillow\PYZus{}train} \PY{o}{=} \PY{n}{pd}\PY{o}{.}\PY{n}{read\PYZus{}csv}\PY{p}{(}\PY{n}{train\PYZus{}path}\PY{p}{)}
        \PY{n}{zillow\PYZus{}test}  \PY{o}{=} \PY{n}{pd}\PY{o}{.}\PY{n}{read\PYZus{}csv}\PY{p}{(}\PY{n}{test\PYZus{}path}\PY{p}{)}
\end{Verbatim}

\begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}10}]:} \PY{n+nb}{print} \PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{zillow\PYZus{}train.shape}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{zillow\PYZus{}train}\PY{o}{.}\PY{n}{shape}\PY{p}{)}
         \PY{n+nb}{print} \PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{zillow\PYZus{}test.shape}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{zillow\PYZus{}test}\PY{o}{.}\PY{n}{shape}\PY{p}{)}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
zillow\_train.shape (11588, 24)
zillow\_test.shape (4402, 24)

    \end{Verbatim}

\iffalse

\begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}}
\PY{n+nn}{pandas}
\PY{n+nn}{numpy}

        \PY{k+kn}{import} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{preprocessing}
        \PY{k+kn}{import} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{model\PYZus{}selection}
        \PY{k+kn}{import} \PY{n+nn}{time}

        \PY{k+kn}{import} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{pyplot} \PY{k}{as} \PY{n+nn}{plt}
        \PY{k+kn}{import} \PY{n+nn}{seaborn} \PY{k}{as} \PY{n+nn}{sb}

{\color{incolor}In [{\color{incolor}2}]:} \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{utils} \PY{k}{import} \PY{n}{shuffle}
        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{pipeline} \PY{k}{import} \PY{n}{make\PYZus{}pipeline}
        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{kernel\PYZus{}ridge} \PY{k}{import} \PY{n}{KernelRidge}
        \PY{k+kn}{from} \PY{n+nn}{sklearn} \PY{k}{import} \PY{n}{ensemble}\PY{p}{,} \PY{n}{tree}\PY{p}{,} \PY{n}{linear\PYZus{}model}
        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{metrics} \PY{k}{import} \PY{n}{r2\PYZus{}score}\PY{p}{,} \PY{n}{mean\PYZus{}squared\PYZus{}error}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{model\PYZus{}selection} \PY{k}{import} \PY{n}{train\PYZus{}test\PYZus{}split}\PY{p}{,} \PYZbs{}
                                            \PY{n}{KFold}\PY{p}{,} \PYZbs{}
                                            \PY{n}{cross\PYZus{}val\PYZus{}score}\PY{p}{,} \PYZbs{}
                                            \PY{n}{GridSearchCV}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{ensemble} \PY{k}{import} \PY{n}{RandomForestRegressor}\PY{p}{,}\PYZbs{}
                                     \PY{n}{GradientBoostingRegressor}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{linear\PYZus{}model} \PY{k}{import} \PY{n}{ElasticNet}\PY{p}{,} \PYZbs{}
                                         \PY{n}{Ridge}\PY{p}{,} \PYZbs{}
                                         \PY{n}{Lasso}\PY{p}{,} \PYZbs{}
                                         \PY{n}{BayesianRidge}\PY{p}{,} \PYZbs{}
                                         \PY{n}{LassoLarsIC}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{linear\PYZus{}model} \PY{k}{import} \PY{n}{RidgeCV}\PY{p}{,} \PYZbs{}
                                         \PY{n}{LassoCV}\PY{p}{,} \PYZbs{}
                                         \PY{n}{ElasticNetCV}\PY{p}{,} \PYZbs{}
                                         \PY{n}{LassoLarsCV}\PY{p}{,} \PYZbs{}
                                         \PY{n}{LinearRegression}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{base} \PY{k}{import} \PY{n}{BaseEstimator}\PY{p}{,} \PYZbs{}
                                 \PY{n}{TransformerMixin}\PY{p}{,} \PYZbs{}
                                 \PY{n}{RegressorMixin}\PY{p}{,} \PY{n}{clone}

{\color{incolor}In [{\color{incolor}3}]:} \PY{k+kn}{from} \PY{n+nn}{datetime} \PY{k}{import} \PY{n}{date}\PY{p}{,} \PY{n}{datetime}

        \PY{k+kn}{from} \PY{n+nn}{scipy} \PY{k}{import} \PY{n}{stats}
        \PY{k+kn}{from} \PY{n+nn}{scipy}\PY{n+nn}{.}\PY{n+nn}{stats} \PY{k}{import} \PY{n}{norm}\PY{p}{,} \PY{n}{skew}

        \PY{k+kn}{from} \PY{n+nn}{sklearn} \PY{k}{import} \PY{n}{preprocessing}
        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{feature\PYZus{}selection} \PY{k}{import} \PY{n}{chi2}

        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{preprocessing} \PY{k}{import} \PY{n}{OneHotEncoder}\PY{p}{,} \PYZbs{}
                                          \PY{n}{StandardScaler}\PY{p}{,} \PYZbs{}
                                           \PY{n}{Normalizer}\PY{p}{,} \PYZbs{}
                                           \PY{n}{RobustScaler}
\end{Verbatim}
\fi

%%%%%%%%%%%%%%%%%%%%%%%5

    \section{Functions}\label{functions}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}4}]:} \PY{k}{def} \PY{n+nf}{AAPE}\PY{p}{(}\PY{n}{y\PYZus{}true}\PY{p}{,} \PY{n}{y\PYZus{}pred}\PY{p}{)}\PY{p}{:}
            \PY{n}{out} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{n+nb}{abs}\PY{p}{(}\PY{n}{y\PYZus{}true} \PY{o}{\PYZhy{}} \PY{n}{y\PYZus{}pred}\PY{p}{)} \PY{o}{/} \PY{n}{y\PYZus{}true}\PY{p}{)}
            \PY{k}{return} \PY{n}{out}\PY{o}{*}\PY{l+m+mi}{100}

        \PY{k}{def} \PY{n+nf}{median\PYZus{}APE}\PY{p}{(}\PY{n}{y\PYZus{}true}\PY{p}{,} \PY{n}{y\PYZus{}pred}\PY{p}{)}\PY{p}{:}
            \PY{n}{out} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{n+nb}{abs}\PY{p}{(}\PY{n}{y\PYZus{}true} \PY{o}{\PYZhy{}} \PY{n}{y\PYZus{}pred}\PY{p}{)} \PY{o}{/} \PY{n}{y\PYZus{}true}\PY{p}{)}
            \PY{k}{return} \PY{n}{np}\PY{o}{.}\PY{n}{median}\PY{p}{(}\PY{n}{out}\PY{p}{)}\PY{o}{*}\PY{l+m+mi}{100}

        \PY{c+c1}{\PYZsh{} For accurate scoring}
        \PY{k}{def} \PY{n+nf}{get\PYZus{}score}\PY{p}{(}\PY{n}{lables}\PY{p}{,} \PY{n}{prediction}\PY{p}{)}\PY{p}{:}
            \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{R2 = }\PY{l+s+si}{\PYZob{}0:.2f\PYZcb{}}\PY{l+s+s1}{\PYZsq{}}\PY{o}{.}\PY{n}{format}\PY{p}{(}\PY{n}{sklearn}\PY{o}{.}\PY{n}{metrics}\PY{o}{.}\PY{n}{r2\PYZus{}score}\PY{p}{(}\PY{n}{lables}\PY{p}{,} \PY{n}{prediction}\PY{p}{)}\PY{p}{)}\PY{p}{)}
            \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{RMSE = }\PY{l+s+si}{\PYZob{}0:.2f\PYZcb{}}\PY{l+s+s1}{\PYZsq{}}\PY{o}{.}\PY{n}{format}\PY{p}{(}\PY{n}{np}\PY{o}{.}\PY{n}{sqrt}\PY{p}{(}\PY{n}{mean\PYZus{}squared\PYZus{}error}\PY{p}{(}\PY{n}{lables}\PY{p}{,} \PY{n}{prediction}\PY{p}{)}\PY{p}{)}\PY{p}{)}\PY{p}{)}
            \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{AAPE = }\PY{l+s+si}{\PYZob{}:.2f\PYZcb{}}\PY{l+s+s1}{\PYZsq{}}\PY{o}{.}\PY{n}{format}\PY{p}{(}\PY{n}{AAPE}\PY{p}{(}\PY{n}{lables}\PY{p}{,} \PY{n}{prediction}\PY{p}{)}\PY{p}{)}\PY{p}{)}
            \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{MAPE = }\PY{l+s+si}{\PYZob{}:.2f\PYZcb{}}\PY{l+s+s1}{\PYZsq{}}\PY{o}{.}\PY{n}{format}\PY{p}{(}\PY{n}{median\PYZus{}APE}\PY{p}{(}\PY{n}{lables}\PY{p}{,} \PY{n}{prediction}\PY{p}{)}\PY{p}{)}\PY{p}{)}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}5}]:} \PY{k}{def} \PY{n+nf}{convert\PYZus{}date}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{,} \PY{n}{date\PYZus{}column}\PY{p}{)}\PY{p}{:}
            \PY{c+c1}{\PYZsh{} all the data are from 2015.}
            \PY{n}{origin} \PY{o}{=} \PY{n}{date}\PY{p}{(}\PY{l+m+mi}{2015}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{)}

            \PY{c+c1}{\PYZsh{} convert the date\PYZus{}column from string to datetime format.}
            \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{date\PYZus{}column}\PY{p}{]} \PY{o}{=} \PY{n}{pd}\PY{o}{.}\PY{n}{to\PYZus{}datetime}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{date\PYZus{}column}\PY{p}{]}\PY{p}{)}
            \PY{n}{sold\PYZus{}date} \PY{o}{=} \PY{p}{[}\PY{k+kc}{None}\PY{p}{]}\PY{o}{*}\PY{n+nb}{len}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{TransDate}\PY{p}{)}
            \PY{n}{age} \PY{o}{=} \PY{p}{[}\PY{k+kc}{None}\PY{p}{]}\PY{o}{*}\PY{n+nb}{len}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{TransDate}\PY{p}{)}

            \PY{k}{for} \PY{n}{ii} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{TransDate}\PY{p}{)}\PY{p}{)}\PY{p}{:}
                \PY{n}{sold\PYZus{}date}\PY{p}{[}\PY{n}{ii}\PY{p}{]} \PY{o}{=} \PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{TransDate}\PY{p}{[}\PY{n}{ii}\PY{p}{]}\PY{o}{.}\PY{n}{date}\PY{p}{(}\PY{p}{)} \PY{o}{\PYZhy{}} \PY{n}{date}\PY{p}{(}\PY{l+m+mi}{2015}\PY{p}{,}\PY{l+m+mi}{1}\PY{p}{,}\PY{l+m+mi}{1}\PY{p}{)}\PY{p}{)}\PY{o}{.}\PY{n}{days}
                \PY{n}{age}\PY{p}{[}\PY{n}{ii}\PY{p}{]} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{TransDate}\PY{p}{[}\PY{n}{ii}\PY{p}{]}\PY{o}{.}\PY{n}{year} \PY{o}{\PYZhy{}} \PY{n+nb}{int}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{BuiltYear}\PY{p}{[}\PY{n}{ii}\PY{p}{]}\PY{p}{)}

            \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{age}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{n}{age}
            \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{date\PYZus{}column}\PY{p}{]} \PY{o}{=} \PY{n}{sold\PYZus{}date}
            \PY{k}{return} \PY{n}{df\PYZus{}tr}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}6}]:} \PY{k}{def} \PY{n+nf}{train\PYZus{}test}\PY{p}{(}\PY{n}{estimator}\PY{p}{,} \PY{n}{x\PYZus{}trn}\PY{p}{,} \PY{n}{x\PYZus{}tst}\PY{p}{,} \PY{n}{y\PYZus{}trn}\PY{p}{,} \PY{n}{y\PYZus{}tst}\PY{p}{)}\PY{p}{:}
            \PY{n}{prediction\PYZus{}train} \PY{o}{=} \PY{n}{estimator}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{x\PYZus{}trn}\PY{p}{)}
            \PY{c+c1}{\PYZsh{} Printing estimator}
            \PY{n+nb}{print}\PY{p}{(}\PY{n}{estimator}\PY{p}{)}
            \PY{c+c1}{\PYZsh{} Printing train scores}
            \PY{n}{get\PYZus{}score}\PY{p}{(}\PY{n}{y\PYZus{}trn}\PY{p}{,} \PY{n}{prediction\PYZus{}train} \PY{p}{)}
            \PY{n}{prediction\PYZus{}test} \PY{o}{=} \PY{n}{estimator}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{x\PYZus{}tst}\PY{p}{)}
            \PY{c+c1}{\PYZsh{} Printing test scores}
            \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
            \PY{n}{get\PYZus{}score}\PY{p}{(}\PY{n}{y\PYZus{}tst}\PY{p}{,} \PY{n}{prediction\PYZus{}test}\PY{p}{)}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}7}]:} \PY{k}{def} \PY{n+nf}{regularize\PYZus{}cv}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{,} \PY{n}{cols}\PY{p}{,} \PY{n}{target}\PY{p}{,} \PY{n}{method}\PY{p}{,} \PY{n}{no\PYZus{}folds}\PY{o}{=}\PY{l+m+mi}{5}\PY{p}{,} \PY{n}{rand\PYZus{}state}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{)}\PY{p}{:}
            \PY{c+c1}{\PYZsh{} split the data into no\PYZus{}folds folds}
            \PY{n}{kf} \PY{o}{=} \PY{n}{KFold}\PY{p}{(}\PY{n}{n\PYZus{}splits}\PY{o}{=}\PY{n}{no\PYZus{}folds}\PY{p}{,} \PY{n}{shuffle}\PY{o}{=}\PY{k+kc}{False}\PY{p}{,} \PY{n}{random\PYZus{}state}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{)}
            \PY{n}{kf}\PY{o}{.}\PY{n}{get\PYZus{}n\PYZus{}splits}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{)}

            \PY{c+c1}{\PYZsh{} pick up the target}
            \PY{n}{y\PYZus{}tr} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{values}

            \PY{c+c1}{\PYZsh{}}
            \PY{c+c1}{\PYZsh{} mean encoding}
            \PY{c+c1}{\PYZsh{}}
            \PY{k}{if} \PY{n}{method}\PY{o}{==}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{mean}\PY{l+s+s1}{\PYZsq{}}\PY{p}{:}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZus{}mean\PYZus{}target}\PY{l+s+s2}{\PYZdq{}}\PY{p}{]} \PY{o}{=} \PY{k+kc}{None}

                \PY{k}{for} \PY{n}{tr\PYZus{}ind}\PY{p}{,} \PY{n}{val\PYZus{}ind} \PY{o+ow}{in} \PY{n}{kf}\PY{o}{.}\PY{n}{split}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{)}\PY{p}{:}
                    \PY{c+c1}{\PYZsh{} gran validation and training sets.}
                    \PY{n}{X\PYZus{}tr}\PY{p}{,} \PY{n}{X\PYZus{}val} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{tr\PYZus{}ind}\PY{p}{]}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}\PY{p}{,} \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{val\PYZus{}ind}\PY{p}{]}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}
                    \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                        \PY{l+s+sd}{\PYZdq{}\PYZdq{}\PYZdq{}}
        \PY{l+s+sd}{                 map mean\PYZus{}encoding of training set back to validation set}
        \PY{l+s+sd}{                 this is done to prevent data leackage.}
        \PY{l+s+sd}{                 This is a way of regularizing (prevent leackage)}
        \PY{l+s+sd}{                 and would work just fine if size of data is big.}
        \PY{l+s+sd}{                \PYZdq{}\PYZdq{}\PYZdq{}}
                        \PY{n}{means} \PY{o}{=} \PY{n}{X\PYZus{}val}\PY{p}{[}\PY{n}{col}\PY{p}{]}\PY{o}{.}\PY{n}{map}\PY{p}{(}\PY{n}{X\PYZus{}tr}\PY{o}{.}\PY{n}{groupby}\PY{p}{(}\PY{n}{col}\PY{p}{)}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{p}{)}\PY{p}{)}
                        \PY{n}{X\PYZus{}val}\PY{o}{.}\PY{n}{loc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}mean\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{n}{means}
                    \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{val\PYZus{}ind}\PY{p}{]}\PY{o}{=} \PY{n}{X\PYZus{}val}

                \PY{c+c1}{\PYZsh{} replace NaNs with global mean}
                \PY{n}{prior} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{p}{)}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}mean\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]}\PY{o}{.}\PY{n}{fillna}\PY{p}{(}\PY{n}{prior}\PY{p}{,} \PY{n}{inplace}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
                \PY{k}{return} \PY{n}{df\PYZus{}tr}
            \PY{c+c1}{\PYZsh{}}
            \PY{c+c1}{\PYZsh{} count encoding (Why in the log we were getting division by zero,}
            \PY{c+c1}{\PYZsh{}                 while clearly it was not zero?)}
            \PY{k}{elif} \PY{n}{method}\PY{o}{==}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{count}\PY{l+s+s1}{\PYZsq{}}\PY{p}{:}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZus{}count\PYZus{}target}\PY{l+s+s2}{\PYZdq{}}\PY{p}{]} \PY{o}{=} \PY{k+kc}{None}

                \PY{k}{for} \PY{n}{tr\PYZus{}ind}\PY{p}{,} \PY{n}{val\PYZus{}ind} \PY{o+ow}{in} \PY{n}{kf}\PY{o}{.}\PY{n}{split}\PY{p}{(}\PY{n}{df\PYZus{}tr}\PY{p}{)}\PY{p}{:}
                    \PY{c+c1}{\PYZsh{} gran validation and training sets.}
                    \PY{n}{X\PYZus{}tr}\PY{p}{,} \PY{n}{X\PYZus{}val} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{tr\PYZus{}ind}\PY{p}{]}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}\PY{p}{,} \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{val\PYZus{}ind}\PY{p}{]}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}
                    \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                        \PY{n}{means} \PY{o}{=} \PY{n}{X\PYZus{}val}\PY{p}{[}\PY{n}{col}\PY{p}{]}\PY{o}{.}\PY{n}{map}\PY{p}{(}\PY{n}{X\PYZus{}tr}\PY{o}{.}\PY{n}{groupby}\PY{p}{(}\PY{n}{col}\PY{p}{)}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{p}{)}\PY{p}{)}
                        \PY{n}{X\PYZus{}val}\PY{o}{.}\PY{n}{loc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}count\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{n}{means}
                    \PY{n}{df\PYZus{}tr}\PY{o}{.}\PY{n}{iloc}\PY{p}{[}\PY{n}{val\PYZus{}ind}\PY{p}{]}\PY{o}{=} \PY{n}{X\PYZus{}val}

                \PY{c+c1}{\PYZsh{} replace NaNs with global count}
                \PY{n}{prior} \PY{o}{=} \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{count}\PY{p}{(}\PY{p}{)}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{df\PYZus{}tr}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}mean\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]}\PY{o}{.}\PY{n}{fillna}\PY{p}{(}\PY{n}{prior}\PY{p}{,} \PY{n}{inplace}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
                \PY{k}{return} \PY{n}{df\PYZus{}tr}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}8}]:} \PY{k}{def} \PY{n+nf}{apply\PYZus{}train\PYZus{}encode\PYZus{}on\PYZus{}val}\PY{p}{(}\PY{n}{train\PYZus{}df}\PY{p}{,} \PY{n}{val\PYZus{}df}\PY{p}{,} \PY{n}{cols}\PY{p}{,} \PY{n}{method}\PY{p}{,} \PY{n}{target}\PY{p}{)}\PY{p}{:}
            \PY{n}{val\PYZus{}df\PYZus{}new} \PY{o}{=} \PY{n}{val\PYZus{}df}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}
            \PY{k}{if} \PY{n}{method}\PY{o}{==}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{mean}\PY{l+s+s1}{\PYZsq{}}\PY{p}{:}
                \PY{c+c1}{\PYZsh{} pick up the target}
                \PY{n}{y\PYZus{}tr} \PY{o}{=} \PY{n}{train\PYZus{}df}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{values}

                \PY{c+c1}{\PYZsh{} In the new dataFrame create new columns}
                \PY{c+c1}{\PYZsh{} for encodings}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{val\PYZus{}df\PYZus{}new}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZus{}mean\PYZus{}target}\PY{l+s+s2}{\PYZdq{}}\PY{p}{]} \PY{o}{=} \PY{k+kc}{None}

                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{means} \PY{o}{=} \PY{n}{val\PYZus{}df\PYZus{}new}\PY{p}{[}\PY{n}{col}\PY{p}{]}\PY{o}{.}\PY{n}{map}\PY{p}{(}\PY{n}{train\PYZus{}df}\PY{o}{.}\PY{n}{groupby}\PY{p}{(}\PY{n}{col}\PY{p}{)}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{p}{)}\PY{p}{)}
                    \PY{n}{val\PYZus{}df\PYZus{}new}\PY{o}{.}\PY{n}{loc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}mean\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{n}{means}

                \PY{c+c1}{\PYZsh{} replace NaNs with global mean}
                \PY{n}{prior} \PY{o}{=} \PY{n}{train\PYZus{}df}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{mean}\PY{p}{(}\PY{p}{)}
                \PY{n}{val\PYZus{}df\PYZus{}new}\PY{o}{.}\PY{n}{fillna}\PY{p}{(}\PY{n}{prior}\PY{p}{,} \PY{n}{inplace}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
                \PY{k}{return} \PY{n}{val\PYZus{}df\PYZus{}new}

            \PY{k}{elif} \PY{n}{method}\PY{o}{==}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{count}\PY{l+s+s1}{\PYZsq{}}\PY{p}{:}
                \PY{c+c1}{\PYZsh{} pick up the target}
                \PY{n}{y\PYZus{}tr} \PY{o}{=} \PY{n}{train\PYZus{}df}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{values}

                \PY{c+c1}{\PYZsh{} In the new dataFrame create new columns}
                \PY{c+c1}{\PYZsh{} for encodings}
                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{val\PYZus{}df\PYZus{}new}\PY{p}{[}\PY{n}{col}\PY{o}{+}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZus{}count\PYZus{}target}\PY{l+s+s2}{\PYZdq{}}\PY{p}{]} \PY{o}{=} \PY{k+kc}{None}

                \PY{k}{for} \PY{n}{col} \PY{o+ow}{in} \PY{n}{cols}\PY{p}{:}
                    \PY{n}{means} \PY{o}{=} \PY{n}{val\PYZus{}df\PYZus{}new}\PY{p}{[}\PY{n}{col}\PY{p}{]}\PY{o}{.}\PY{n}{map}\PY{p}{(}\PY{n}{train\PYZus{}df}\PY{o}{.}\PY{n}{groupby}\PY{p}{(}\PY{n}{col}\PY{p}{)}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{p}{)}\PY{p}{)}
                    \PY{n}{val\PYZus{}df\PYZus{}new}\PY{o}{.}\PY{n}{loc}\PY{p}{[}\PY{p}{:}\PY{p}{,} \PY{n}{col}\PY{o}{+}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZus{}count\PYZus{}target}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{n}{means}

                \PY{c+c1}{\PYZsh{} replace NaNs with global mean}
                \PY{n}{prior} \PY{o}{=} \PY{n}{train\PYZus{}df}\PY{p}{[}\PY{n}{target}\PY{p}{]}\PY{o}{.}\PY{n}{count}\PY{p}{(}\PY{p}{)}
                \PY{n}{val\PYZus{}df\PYZus{}new}\PY{o}{.}\PY{n}{fillna}\PY{p}{(}\PY{n}{prior}\PY{p}{,} \PY{n}{inplace}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
                \PY{k}{return} \PY{n}{val\PYZus{}df\PYZus{}new}
\end{Verbatim}

\usepackage{pgfplots}
\usetikzlibrary{patterns}

%\usepackage{graphics}
\usepackage{tikz,ifthen,fp,calc}
\usepackage{caption}
\usepackage{subcaption}
\usetikzlibrary{plotmarks}
\usepackage{graphicx}

%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%% Theorem Styles
%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{prop}{Proposition}[section]
\newtheorem{corr}{Corollary}[section]
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{remark}{Remark}[section]
\newtheorem{fact}{Fact}[section]

\usepackage[english]{babel}
\usepackage{babel,blindtext}
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{exmp}{Example}[section]
\usepackage{fullpage}
\usepackage{amsfonts}
\usepackage{lscape}
\usepackage{bbm}

\usepackage{todonotes}
\usepackage{cite}
\usepackage{verbatim}
\usepackage{bm}

\DeclareMathOperator*{\argmax}{arg\,max}
\usepackage[margin=1in]{geometry}

\begin{tcolorbox}
\begin{center}
\textbf{Notation used throughout the paper}
\end{center}
\begin{multicols}{2}
[
]
\begin{itemize}
\item $\psi$ potential function.
\item $\nabla \psi$ gradient of function $\psi$.
\item $\tau$ parameter for some potential functions like (tip of the) tent, (bend point of) BCM potential.
\item $\mathbb{O}$ opinion space.
\item $\mathbb{T}$ topic space, if topic space is finite, $|\mathbb{T}| = n$.
\item $o_i^{(t)}$ opinion of agent $i$ at time $t$ if there is only one topic in the system.
\item $o_i^{(t)}(s_k)$ opinion of agent $i$ at time $t$ about topic $s_k$.
\item $\dot{o}$ derivative of $o$ with respect to time.
\item $\Delta o_i^{(t+1)}(s_k) = o_i^{(t+1)}(s_k) - o_i^{(t)}(s_k) $.
\item  $d_{ij}^{(t)}(s) = o_i^{(t)}(s) - o_j^{(t)}(s)$.

Difference between opinion of agents $i$ and $j$ about topic $s$ at a time $t$.
\columnbreak
\item $N(\mu, \sigma)$ normal distribution with mean $\mu$ and standard deviation $\sigma$.
\item $\alpha$ learning rate.
\item $\sigma(\mathbf{A})$ spectrum of matrix $\mathbf{A}$.
\item $\mathbf{J}$ Jacobian matrix.
\item $\mathbf{C}$ coupling matrix.
\item $c_{ij}$ inverse coupling strength of topic $i$ over topic $j$.
\item $G = (V,E)$ graph $G$ with set of nodes $V$ and set of edges $E$.
\item $n(i)$ set of neighbors of agent $i$.
\item $||.||$ Euclidean norm.
\item vectors and matrices will be bold letters.
\item $\mu_s$ mean of polarization counts
\item $\mu_p$ mean of stabilization time
\end{itemize}
\end{multicols}
\end{tcolorbox}


\documentclass[12pt]{article}
%\documentclass[12pt,landscape]{article}

\include{preamble}

\newcommand{\instr}{\small Your answer will consist of a string (e.g. \texttt{aebgd}) where the order of the letters does not matter nor does upper / lowercase. \normalsize}

\title{Math 390.4 / 650.3 Spring 2020 \\ Midterm Examination Two}
\author{Professor Adam Kapelner}

\date{Thursday, May 14, 2020}

\begin{document}
\maketitle

%\noindent Full Name \line(1,0){410}

\thispagestyle{empty}

\section*{Code of Academic Integrity}

\footnotesize
Since the college is an academic community, its fundamental purpose is the pursuit of knowledge. Essential to the success of this educational mission is a commitment to the principles of academic integrity. Every member of the college community is responsible for upholding the highest standards of honesty at all times. Students, as members of the community, are also responsible for adhering to the principles and spirit of the following Code of Academic Integrity.

Activities that have the effect or intention of interfering with education, pursuit of knowledge, or fair evaluation of a student's performance are prohibited. Examples of such activities include but are not limited to the following definitions:

\paragraph{Cheating} Using or attempting to use unauthorized assistance, material, or study aids in examinations or other academic work or preventing, or attempting to prevent, another from using authorized assistance, material, or study aids. Example: using an unauthorized cheat sheet in a quiz or exam, altering a graded exam and resubmitting it for a better grade, etc.
\\

\noindent By taking this exam, you acknowledge and agree to uphold this Code of Academic Integrity. \\

%\begin{center}
%\line(1,0){250} ~~~ \line(1,0){100}\\
%~~~~~~~~~~~~~~~~~~~~~signature~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ date
%\end{center}

\normalsize

\section*{Instructions}

This exam has 146 total points, is 120 minutes (variable time per question) and closed-book. You are allowed \textbf{two} pages (front and back) of a \qu{cheat sheet}, one table of reference and scrap paper and a graphing calculator. Please read the questions carefully. No food is allowed, only drinks. %If the question reads \qu{compute,} this means the solution will be a number otherwise you can leave the answer in \textit{any} widely accepted mathematical notation which could be resolved to an exact or approximate number with the use of a computer. I advise you to skip problems marked \qu{[Extra Credit]} until you have finished the other questions on the exam, then loop back and plug in all the holes. I also advise you to use pencil. The exam is 100 points total plus extra credit. Partial credit will be granted for incomplete answers on most of the questions. \fbox{Box} in your final answers. Good luck!

\pagebreak

\problem [10min] Consider the following causal diagrams A, B and C below. In each diagram, all three events x, y, z have other causes that are not displayed.

\begin{figure}[htp]
\centering
\includegraphics[width=4in]{basic_causal_diagrams}
\end{figure}

\vspace{-.8cm}
\benum

\subquestionwithpoints{13} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
%\setcounter{enumi}{3}
\item In diagram A, x and y could be spuriously correlated.
\item In diagram A, x and y are correlated.
\item In diagram A, z causes y.
\item In diagram B, x and y could be spuriously correlated.
\item In diagram B, x and y are correlated.
\item In diagram A, x causes y.
\item In diagram C, x and y could be spuriously correlated.
\item In diagram C, x and y are correlated.
\item In diagram C, z could cause x.
\item In diagram A, x causes z and x causes y.
\item Let y be number of car accidents daily in New York City, let z be daily rainfall in New York City and let x be the phase of the moon in degrees. The most likely causal diagram reflecting this situation is A.
\item Let y be number of car accidents daily in New York City, let z be daily rainfall in New York City and let x be the number of people working from home. The most likely causal diagram reflecting this situation is B.
\item Let y be number of car accidents daily in New York City, let z be daily rainfall in New York City and let x be the number of people working from home. The most likely causal diagram reflecting this situation is C.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Consider the same causal diagrams A, B and C below. In each diagram, all three events x, y, z have other causes that are not displayed.

\begin{figure}[htp]
\centering
\includegraphics[width=4in]{basic_causal_diagrams}
\end{figure}

Consider the following two predictive models fit by OLS:

\beqn
&[I]& ~~~ \yhat^I = b_0^I + b_1^I x \\
&[II]& ~~~ \yhat^{II} = b_0^{II} + b_1^{II} x + b_2^{II} z
\eeqn

\benum

\subquestionwithpoints{8} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
%\setcounter{enumi}{3}
\item In causal diagram A, $b_1^I x \approx b_1^{II} x \approx 0$.
\item In causal diagram B, $b_1^I x \approx b_1^{II} x \approx 0$.
\item In causal diagram C, $b_1^I x \approx b_1^{II} x \approx 0$.
\item In causal diagram B, $b_1^I x \neq b_1^{II} x \approx 0$.
\item In causal diagram C, $b_1^I x \neq b_1^{II} x \approx 0$.
\item In all causal diagrams, model [II] performs better out of sample.
\end{enumerate}

The following statements are true about the most correct interpretation of $b_1^I$ for causal diagram B for two observations, O1 and O2.

\begin{enumerate}[(a)]
\setcounter{enumi}{6}
\item O1 and O2 must be sampled in same way as the observations in $\mathbb{D}$ and O2's $x$ measurement is one unit higher than O1's $x$ unit, then the $y$ value of O2 will be on average $b_1^I$ higher than O1 assuming the linear model is true.
\item O1 and O2 must be sampled in same way as the observations in $\mathbb{D}$ and O1 and O2 have the same $x$ and then I manipulate the $x$ value of O2 to be one higher than O1, then the $y$ value of O2 will be on average $b_1^I$ higher than O1 assuming the linear model is true.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] In lab 10 we worked with a data from an accounting firm. The tables were bills, payments and discounts. The bills for one customer and the discounts table is below. The \texttt{discount\_id} column references the \texttt{id} column in the discounts table.

\begin{lstlisting}
> billsA[order(discount_id)]
         id   due_date invoice_date    amount discount_id
1: 16294508 2017-06-25   2017-06-25  99476.43     7302585
2: 17129704 2017-07-29   2017-06-29  99475.01     7564949
3: 15247786 2015-09-18   2015-08-19  99484.62     8218876
4: 17096637 2016-02-02   2016-01-03  99668.94     8806662
5: 17222576 2016-09-15   2016-08-16  99476.93     8806662
6: 16797375 2017-01-29   2016-12-30 116487.16     8806662
> discounts[order(id)]
         id num_days pct_off days_until_discount
 1: 6098612       20      NA                  NA
 2: 6386294      120      NA                  NA
 3: 6609438       NA       1                   7
 4: 7197225       60      NA                  NA
 5: 7302585       NA      NA                  NA
 6: 7708050       NA      NA                  NA
 7: 7833213       10      NA                   5
 8: 8295837       14      NA                  10
 9: 8496508       10       2                  15
10: 8583519        0       2                  NA
11: 8610918        1      NA                  NA
12: 8784190       NA      NA                  30
13: 8806662       30      NA                  NA
14: 8951244       NA      NA                  NA
15: 8988984       60      NA                  NA
\end{lstlisting}
\vspace{-1cm}
\benum

\subquestionwithpoints{12} Each of the following are short for \qu{If we were to do a \line(1,0){50}~of billsA and discounts, we would create a table with \line(1,0){50}}. Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item left join / 8 columns.
\item left join / 9 columns.
\item left join / 6 rows.
\item left join / 2 rows.
\item right join / 17 rows.
\item right join / 15 rows.
\item inner join / 2 rows.
\item inner join / 4 rows.
\item inner join / 21 rows.
\item full join / 2 rows.
\item full join / 21 rows.
\item full join / $>$21 rows.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Here is a dataset about weather:

\begin{figure}[htp]
\centering
\includegraphics[width=6in]{weather_data.png}
\end{figure}

\benum

\subquestionwithpoints{10} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item If listwise deletion would be used, there would be $n = 5,337$ observations left.
\item If \texttt{wind\_gust} were dropped, there would be less imputations that would have to be made.
\item It would be good practice to drop the observations that have missingness in \texttt{temp}, \texttt{dewp} and \texttt{humid}.
\item If \texttt{wind\_gust} were the response variable, then $n = 5,337$.
\item If \texttt{wind\_gust} were the response variable, and listwise deletion was used, then $n \geq 2141$.
\item If imputation via \texttt{missForest} were used on the \texttt{weather} data frame, the resulting data frame would have $n = 26,115$ observations. \\~\\
If each of these columns contained measurements that were features in a model and we were to both impute and record missingness, the resulting $\X$ matrix in $\mathbb{D}$ would have \line(1,0){20}~columns:
\item 13
\item 16
\item 20
\item 26
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Let $p + 1 = 501$ of features that are all standardized and $n > p + 1$. Let $f(\x) = \x\bbeta$, the first entry of $\x$ is one always, let $\delta$ be a non-trivial amount of homoskedastic noise and $\bbeta = \bracks{\beta_0~\beta_1~\ldots~\beta_{500}}^\top = \bracks{-5~5~4~3~2~1~\zerovec_{495}}^\top$. Consider the usual definition of SSE as the sum of squared error, $\mathcal{H} = \braces{\w^\top \x~:~ \w \in \reals^{p + 1}}$ and all the following algorithms that produce an estimate of $\bbeta$ via:

\beqn
\mathcal{A}_1 &:& \b^{\mathcal{A}_1} = \argmin_{\w \in \reals^{p + 1}}\braces{SSE} \\
\mathcal{A}_2 &:& \b^{\mathcal{A}_2} = \argmin_{\w \in \reals^{p + 1}}\braces{SSE + \sum_{j=1}^{p + 1} |w_j|} \\
\mathcal{A}_3 &:& \b^{\mathcal{A}_3} = \argmin_{\w \in \reals^{p + 1}}\braces{SSE + \sum_{j=1}^{p + 1} w_j^2} \\
\mathcal{A}_4 &:& \b^{\mathcal{A}_4} = \argmin_{\w \in \reals^{p + 1}}\braces{SSE + \half \sum_{j=1}^{p + 1} |w_j| + \half\sum_{j=1}^{p + 1} w_j^2}
\eeqn

\benum

\subquestionwithpoints{12} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item $\b^{\mathcal{A}_1} = \inverse{\X^\top \X + \I}\X^\top \y$.
\item $\b^{\mathcal{A}_2} = \inverse{\X^\top \X + \I}\X^\top \y$.
\item $\b^{\mathcal{A}_3} = \inverse{\X^\top \X + \I}\X^\top \y$.
\item The OLS algorithm will fail in this setting.
\item $\mathcal{A}_4$ is elastic net regression with $\alpha = \half$.
\item Out of sample, $\mathcal{A}_2, \mathcal{A}_3$ and $\mathcal{A}_4$ will have higher predictive performance than $\mathcal{A}_1$.
\item $\mathcal{A}_2$ likely produces an estimate of $\bbeta$ with more zero entries than $\mathcal{A}_4$.
\item $\mathcal{A}_3$ likely produces an estimate of $\bbeta$ with more zero entries than $\mathcal{A}_4$.
\item $\mathcal{A}_2$ is the most straightforward algorithm that could be used to locate variables that affect the response.
\item $\norm{\bracks{b^{\mathcal{A}_1}_6, b^{\mathcal{A}_1}_7, \ldots, b^{\mathcal{A}_1}_{500}}^\top} < \norm{\bracks{b^{\mathcal{A}_2}_6, b^{\mathcal{A}_2}_7, \ldots, b^{\mathcal{A}_2}_{500}}^\top}$
\item $\norm{\bracks{b^{\mathcal{A}_2}_6, b^{\mathcal{A}_2}_7, \ldots, b^{\mathcal{A}_2}_{500}}^\top} < \norm{\bracks{b^{\mathcal{A}_3}_6, b^{\mathcal{A}_3}_7, \ldots, b^{\mathcal{A}_3}_{500}}^\top}$
\item $\norm{\bracks{b^{\mathcal{A}_4}_6, b^{\mathcal{A}_4}_7, \ldots, b^{\mathcal{A}_4}_{500}}^\top} < \norm{\bracks{b^{\mathcal{A}_1}_6, b^{\mathcal{A}_1}_7, \ldots, b^{\mathcal{A}_1}_{500}}^\top}$
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Assume the response is a real number. In class we studied the following three decompositions of MSE in the modeling context where $\delta$ was realized from a mean-centered r.v. with variance $\sigsq$ independent of the value of $\x$:

\beqn
&[I]& ~~MSE(\x_*) = \sigsq + (f(\x_*) - g(\x_*))^2 \\
&[II]& ~~MSE(\x_*) = \sigsq + \bias{G(\x_*)}^2 + \var{G(\x_*)} \\
&[III]& ~~MSE = \sigsq + \expesub{\mathcal{X}}{\bias{G(\x_*)}^2} + \expesub{\mathcal{X}}{\var{G(\x_*)}} \\
\eeqn

\benum
\subquestionwithpoints{17} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item In I and II, the new observation $\x_*$ was considered drawn from a random universe.
\item In I and II, the training set $\mathbb{D}$ was considered drawn from a random universe.
\item In II and III, the training set $\mathbb{D}$ was considered drawn from a random universe.
\item In II, the responses $\y$ were considered drawn from a random universe.
\item In III, the responses $\y$ were considered drawn from a random universe.
\item In II, the design matrix $\X$ was considered drawn from a random universe.
\item In III, the design matrix $\X$ was considered drawn from a random universe.\\~\\
In general...
\item the $\bias{G(\x_*)}$ term is low for OLS.
\item the $\bias{G(\x_*)}$ term is low for CART.
\item the $\bias{G(\x_*)}$ term is low for a bag of CART models.
\item the $\bias{G(\x_*)}$ term is low for RF.
\item the $\var{G(\x_*)}$ term is low for OLS.
\item the $\var{G(\x_*)}$ term is low for CART.
\item the $\var{G(\x_*)}$ term is low for bag of CART models.
\item the $\var{G(\x_*)}$ term is low for RF.
\item the $\var{G(\x_*)}$ term is lower for CART than it is for a bag of CART models.
\item the $\var{G(\x_*)}$ term is lower for RF than it is for a bag of CART models.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Consider the following RF model output:

\begin{lstlisting}
YARF v1.0 for classification
Missing data feature ON.
500 trees, training data n = 2000 and p = 99
Model construction completed within 0.08 minutes.
OOB results on all observations as a confusion matrix:
           predicted 0 predicted 1 model errors
actual 0      1425.000      77.000        0.051
actual 1       192.000     306.000        0.386
use errors       0.119       0.201        0.134

\end{lstlisting}
\vspace{-1cm}
\benum
%    Accuracy: 86.55%

\subquestionwithpoints{20} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item $\mathcal{Y} = \braces{0,1}$.
\item The features that are important in this model are simple to ascertain.
\item The best estimate of this model's accuracy in-sample is less than 86.6\%.
\item The best estimate of this model's accuracy in-sample is more than 86.6\%.
\item The best estimate of this model's accuracy oos is 86.6\%.
\item This RF model will be more accurate oos if it was rebuilt with $M > 500$.
\item This RF model will be less accurate oos if it was rebuilt with $M > 500$.
\item This RF model will be more accurate if $p_{\text{try}} = 99$.
\item When predicting in the future, if there were 100 samples where $y=1$, this model would predict incorrectly 38.6\% of the time.
\item When predicting in the future, if there were 100 samples where $y=1$, this model would predict incorrectly 20.1\% of the time.
\item When predicting in the future, if there were 100 samples where $\hat{y}=1$, this model would predict incorrectly 38.6\% of the time.
\item When predicting in the future, if there were 100 samples where $\hat{y}=1$, this model would predict incorrectly 20.1\% of the time.
\item The oos precision of this model is 79.9\%.
\item The oos recall of this model is 79.9\%.
\item The oos FDR estimate of this model is 11.9\%.
\item The oos FOR estimate of this model is 11.9\%.
\item If the cost of the false positive is greater than the cost of a false negative, this would be a good model to use in the real world.
\item The AUC for this model can be calculated given the output above.
\item The OOB results were calculated on about 2/3 of the total number of observations, $n=2000$.
\item The predictions from each of the $M=500$ constituent trees in this model are positive correlated.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Let $\mathcal{Y} = \braces{0,1}$ and employ the following algorithm on training data $\mathbb{D}$:

\beqn
\mathcal{A} &:& \b = \argmin_{\w \in \reals^{p + 1}}\braces{\prod_{i=1}^n \tothepow{\oneover{1 + e^{-\w^\top \x_i}}}{y_i} \tothepow{\oneover{1 + e^{\w^\top \x_i}}}{1 - y_i}}
\eeqn

\benum

\subquestionwithpoints{13} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item This algorithm has a closed-form solution for $\b$ as a function of $\X$ and $\y$.
\item $R^2$ is an interpretable metric in this model.
\item This algorithm has hyperparameter(s) that need to be specified before running.
\item This algorithm is called \qu{logistic regression}.
\item The $\b$ that is returned by this algorithm may have numerical error.
\item This algorithm makes statistical assumptions about the training data.
\item The $\b$ returned by this algorithm allows us to compute probability estimates of $\cprob{Y_* = 1}{\x_*} = \b\x_*$.
\item If $\b\x_{*_1} > \b\x_{*_2}$ that implies that the model's estimate for the probability of the response being one for observation $\x_{*_1}$ is higher than for observations $\x_{*_2}$.\\

Consider the following output of $\mathcal{A}$ on the training data $\mathbb{D}$:

\begin{lstlisting}
> b
         (Intercept)            x1                   x2
         -10.0175573            0.5971474            0.3090818
\end{lstlisting}

\item If $x_* = \bracks{0~0}$ then the probability estimate of $Y_* = 1$ will be very low.
\item Using the above output, one can compute the Brier score for training data $\mathbb{D}$.
\item Using the above output, one can compute an ROC curve for this model.
\item $g_0(\x) = 0$ for this model.
\item $g_0(\x) = \ybar$ for this model.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Consider the boston housing data that has response as the median home value and $n = 506$ and $p = 13$. We fit a regression tree model fit setting $N_0 = 200$. Here is the tree visualized:

\begin{figure}[htp]
\centering
\includegraphics[width=9in]{yarf_mod_tree_1.png}
\end{figure}
\vspace{-1.2cm}
\benum

\subquestionwithpoints{12} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item This tree model can be written as a linear model.
\item This tree model has 5 nodes.
\item This tree model has 5 root nodes.
\item This tree model splits $\mathcal{X}$ into rectangular prisms (i.e. dimension 3).
\item This tree model is overfit.
\item If you label the leaves 1,2 \ldots, L from left to right in the above, and you must split further on one of them, then splitting on leaf \#1 is most likely to yield a better tree model.
\item An OLS model will be more accurate out of sample than this model.
\item The most important feature in the data for predicting the response is \texttt{lstat}.
\item The model of the form $\yhat = b_0 + b_1 \u$ where is $\u$ restricted totr be one of the features in this dataset has highest $R^2$ when $\u$ is the variable \texttt{rm}.
\item The model of the form $\yhat = b_0 + b_1 \u$ where $\u$ is restricted to be dummy has highest $R^2$ when $\u$ is the computed via the function $\indic{rm \leq 6.94}$.
\item If \texttt{rm} = 7, the tree would predict 37.34 for $y$ regardless of the values of the other 12 features.
\item If \texttt{rm} = 6, the tree would predict $<$37.34 for $y$ regardless of the values of the other 12 features.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem [11min] Let $p=1$, $n > 100 + 1$, $d \in \braces{10, 11, \ldots, 100}$, $\mathcal{A}$ = OLS and consider the following different complexity sets that are used within the algorithm:

\beqn
\mathcal{H}_1 &=& \braces{w_0 + w_1 x_1 ~:~w_0, w_1 \in \reals}\\
\mathcal{H}_2 &=& \braces{w_0 + w_1 \natlog{x_1} ~:~w_0, w_1 \in \reals}\\
\mathcal{H}_3 &=& \braces{w_0 + w_1 x_1 + w_2 x^2 ~:~w_0, w_1, w_2 \in \reals}\\
\mathcal{H}_4 &=& \braces{w_0 + w_1 x_1 + w_2 x^2 + \ldots + w_d x^d ~:~w_0, w_1, \ldots, w_d \in \reals}\\
\eeqn

\vspace{-1cm}
\benum

\subquestionwithpoints{17} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item The lowest RMSE will be from $\mathcal{H}_4$.
\item The lowest oos RMSE will be from $\mathcal{H}_4$.
\item The degrees of freedom in $\mathcal{H}_4$ equals $d$.
\item $b_1$ is the same in both $\mathcal{H}_1$ and $\mathcal{H}_3$.
\item If the response was logged, $b_1$ would be the same in both $\mathcal{H}_1$ and $\mathcal{H}_2$.
\item The design matrix for $\mathcal{H}_2$ looks like $\X = \bracks{\onevec_n, \bracks{\natlog{x_1}~\natlog{x_2} ~ \ldots ~ \natlog{x_n}}^\top}$.
\item The design matrix for $\mathcal{H}_3$ looks like $\X = \bracks{\onevec_n, \bracks{x_1^2~x_2^2 ~ \ldots ~ x_n^2}^\top}$.
\item In $\mathcal{H}_4$, if $d = 100$, then $R^2 = 100\%$.
\item In $\mathcal{H}_4$, if $d$ was set to be large, predictions for $\x_* \in \mathcal{X}$ will likely be more accurate than $\x_* \notin \mathcal{X}$.
\item In $\mathcal{H}_4$, if $d$ was set to be large, predictions for $\x_* \notin \mathcal{X}$ will likely be worse than if $d$ were set to be small.
\item In $\mathcal{H}_4$, if $d$ was varied across its range specified in the problem, then you can trace out the in-sample vs. oos complexity curve and locate the optimal model as a function of the hyperparameter $d$.
\item Assuming you do the procedure in the previous question and locate the optimal model given by $d = d^\star$, the oos $s_e$ from that procedure for the optimal model would be biased upwards.
\item The oos $s_e$ from that procedure for the optimal model would be biased downwards.\\

When comparing two mutually observed observations (A) and (B) sampled in the same fashion as those in $\mathbb{D}$, when the x measurement of (A) is \line(1,0){40}~ than the $x$ measurement of (B) then (A) is predicted to have a \line(1,0){40}~ from the (B)'s assuming the model is true. The blanks would be filled in by...

\item In $\mathcal{H}_1$, \qu{one unit larger} / \qu{response $y$ that differs by $b_1$}.

\item In $\mathcal{H}_2$, \qu{one unit larger} / \qu{response $y$ that differs by $b_1$}.
\item In $\mathcal{H}_2$, \qu{10\% larger} / \qu{response $y$ that differs by $0.1 b_1$}.
\item In $\mathcal{H}_2$, \qu{10\% larger} / \qu{response $y$ 10\% larger}.
\end{enumerate}
\eenum\instr\pagebreak

%%%%%%%%%%%%%%%%%%%%%%%%

\problem This question is about the concept of model validation and the strategy we discussed in class. Let's say we divide scramble the rows of $\mathbb{D}$ then create a partition

\beqn
\mathbb{D} = \fivevec{\mathbb{D}_{\text{train}}}{\text{------}}{\mathbb{D}_{\text{select}}}{\text{------}}{\mathbb{D}_{\text{test}}}
\eeqn

\noindent in a 8:1:1 ratio train : select : test (in number of rows). We wish to select the best model $g_{m_*}$ out of $M$ candidate models $g_1, g_2, \ldots, g_M$ using the model selection procedure from class.

\benum
\subquestionwithpoints{12} Record the letter(s) of all the following that are \textbf{true}.

\begin{enumerate}[(a)]
\item The three sets above could be permuted and the model selection procedure would still be valid.
\item The three sets above could be permuted and the model selection procedure would yield the same $g_{m_*}$.
\item The test set has 10\% of the total dataset.
\item The select set has 11.11\% of the total dataset.
\item When selecting models, the training data has 80\% of the data used for exclusively the selection step of the model selection procedure.
\item If we were to use only inner CV, then there would be a total of $K = 10$ folds.
\item If we were to use only outer CV, then there would be a total of $K = 10$ folds.
\item If we were to not use any CV, then we would get an estimate of the future performance of one specific $g_{m_*}$.
\item If we were to use only inner CV, then we would get an estimate of the future performance of one specific $g_{m_*}$.
\item If we were to use only outer CV, then we would get an estimate of the future performance of one specific $g_{m_*}$.
\item If we were to use both inner and outer CV, then we would get an estimate of the future performance of one specific $g_{m_*}$.
\item If $\mathbb{D}_{\text{train}}$ was reduced in size, then we increase misspecification error among all $M$ models during the model selection procedure.
\end{enumerate}
\eenum\instr\pagebreak

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%


\subsection{Nominal and real prices}

If previously one fish trades for two chunks of meat, we would expect the shell price for fish to be twice as high as the shell price for meat. Relative prices between all goods could be maintained for any price for meat. If all prices doubled, one fish would still get you two chunks of meat.

Nominal prices are those that are actually seen in markets, real prices refer to these prices adjusted for the price level. For example all prices doubling would double all nominal prices but affect no real prices. Doubling only the nominal price of fish would increase the real price of fish, and decrease the real price of other goods.


\lesson{2}{Sep 09 2021 Thu (08:20:00)}{Measuring Matter}{Unit 1}

\subsubsection*{Measurements and Units}

All measurements are made of \bf{2} parts:

\begin{itemize}
	\item A number.
	\item A unit.
\end{itemize}

\begin{definition}[Matter]
	Anything that takes up space, whether big or small, is \bf{Matter}.
\end{definition}

\begin{definition}[Volume]
	In science, the amount of space an object occupies is called \bf{volume}
\end{definition}

\begin{definition}[Mass]
	\bf{Mass} is a measure of the amount of matter in an object. This means, the more \bf{Matter} an object has, the object's \bf{Mass} increases. \\
	Scientists measure the \bf{Mass} of an object using the base \bf{Unit Gram}.
\end{definition}

\subsubsection*{Systems of Measurements}

The \bf{U.S. Customary System (English System)} consists of measurements we use every day to describe:

\begin{itemize}
	\item How much we have.
	\item What size we want.
	\item How far we want to go.
	\item Etc \ldots
\end{itemize}

The \bf{United States} is the one of three countries in the \bf{whole world} that use this system.

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|l| }
			\hline
			Type & U.S. Customary System & Metric System \\
			\hline
			\multirow{3}{4em}{} Length      & inch, foot, mile 		& meter 			\\
								Mass/Weight & ounce, pound, ton 	& gram 			    \\
								Volume 		& pint, quart, gallon 	& liter 		    \\
								Temperature	& degrees Fahrenheit 	& degrees Celsius   \\
								\hline
		\end{tabular}

		\caption{The \bf{U.S. Customary System (English System)}}
		\label{table:english-system}
	\end{table}
\end{center}

The \bf{Metric System} is a system of measuring units based on the power of
$10$. It's the preferred system used in science measurement because so many
countries already use metrics as their standard measurement system.

One reason for the widespread use of the \bf{Metric System} is that it
seems to simplify the number of measurements. Instead of multiple units for the
same measurements, such as:

\begin{itemize}
	\item Inches.
	\item Feet.
	\item Miles.
\end{itemize}

for length, the \bf{Metric System} uses only one word for the length, the
\bf{Meter}.

For larger or smaller measurements, metric prefixes are used
rather than changing the unit of measure or using fractions, like quarter of an
inch. A prefix works with any unit of measure in the \bf{Metric System}:

\begin{itemize}
	\item Meter.
	\item Gram.
	\item Liter.
\end{itemize}

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|c|l| }
			\hline
			Prefix & Symbol & Multiplier & Example \\
			\hline
			\multirow{3}{4em}{} Kilo-   & k     & $1,000$   & Kilometer (km) 	\\
								Hecto- 	& h     & $100$ 	& Hectometer (hm)   \\
								Deca- 	& da 	& $10$ 		& Decameter (dam)   \\
								Unit	&  	    & $1$ 		& Meter (m) 		\\
								Deci-	& d 	& $0.1$ 	& Decimeter (dm)    \\
								Centi-	& c 	& $0.01$ 	& Centimeter (cm)   \\
								Milli-	& m 	& $0.001$   & Millimeter (mm)   \\
								\hline
		\end{tabular}

		\caption{The \bf{Metric System Prefix References}}
		\label{table:metric-system}
	\end{table}
\end{center}

The \bf{International System of Measurement of Units} is the primary system
used by scientists and engineers. It will also be used in \bf{Chemistry}.
In this system, scientists have specified what units should be used as the
seven base units of measurement from which all other units are derived.

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|l| }
			\hline
			Quantity (Symbol) & Name of Unit (Symbol) \\
			\hline
			\multirow{3}{4em}{} Length (l)                      & 	Meter (m) 		\\
								Mass (m)                        & 	Kilogram (kg)   \\
								Time (t) 					    & 	Second (s) 		\\
								Electric Current (i) 			& 	Ampere (A) 		\\
								Thermodynamic Temperature (T) 	& 	Kelvin (k) 		\\
								Amount of Substance (n) 		& 	Mole (mol) 		\\
								Luminous Intensity ($l_v$) 		& 	Candela (cd) 	\\
								\hline
		\end{tabular}

		\caption{The \bf{Metric System Prefix References}}
		\label{table:metric-system}
	\end{table}
\end{center}

\subsubsection*{Converte between or within Measurement Systems}

\begin{definition}[Dimensional Analysis] Analyzing the relationships between
	different quantities, identify their units of measurement, and convert their
	quantities for equal comparison.

	\bf{Dimensional Analysis} uses a ratio, or a conversion factor to change
	units of measurement while maintaining the value of measurement.

	In a \bf{Conversion Factor}, the \bf{numerator} and \bf{denominator} equal the same amount, just in different units.

	For example, to convert $3$ miles into feet, you would multiply $3$ miles by
	a conversion factor created from the units of miles and feet. Because $1$
	mile equals $5,280$ feet, two conversion factors can be created

    \begin{align}
    	1 = \frac{1 \rm{ mile}}{5,280 \rm{ feet}} \rm{ or } 1 = \frac{5,280 \rm{ feet}}{1 \rm{ mile}}
    \end{align}

	Multiplying any number, such as $3$ miles, by either of these fractions does
	not change the number's overall value.
\end{definition}

\subsubsection*{Using the conversion factor for converting derived units}

\begin{marginfigure}
	\begin{tikzpicture}[every edge quotes/.append style={auto, text=blue}]
		\pgfmathsetmacro{\cubex}{3}
		\pgfmathsetmacro{\cubey}{3}
		\pgfmathsetmacro{\cubez}{3}
		\draw [draw=blue, every edge/.append style={draw=blue, densely dashed, opacity=.5}, fill=magenta]
			(0,0,0) coordinate (o) -- ++(-\cubex,0,0) coordinate (a) -- ++(0,-\cubey,0) coordinate (b) edge coordinate [pos=1] (g) ++(0,0,-\cubez)  -- ++(\cubex,0,0) coordinate (c) -- cycle
			(o) -- ++(0,0,-\cubez) coordinate (d) -- ++(0,-\cubey,0) coordinate (e) edge (g) -- (c) -- cycle
			(o) -- (a) -- ++(0,0,-\cubez) coordinate (f) edge (g) -- (d) -- cycle;
		\path [every edge/.append style={draw=blue, |-|}]
			(b) +(0,-5pt) coordinate (b1) edge ["3cm"'] (b1 -| c)
			(b) +(-5pt,0) coordinate (b2) edge ["3cm"] (b2 |- a)
			(c) +(3.5pt,-3.5pt) coordinate (c2) edge ["3cm"'] ([xshift=3.5pt,yshift=-3.5pt]e)
			;
	\end{tikzpicture}
	\caption{Cube that's $3 \times 3 \times 3$}
	\label{fig:3-3-3-cube}
\end{marginfigure}

\begin{definition}[Derived Units]
	Some measurements you are familiar with, such as speed, contain more than one \bf{Unit}.

	A very common \bf{Derived Unit} is \bf{Volume}. For liquids and gasses, \bf{Volume} tends to be measured in liters. But, for regular-shaped solids, measurements and calculations can determine their \bf{Volume}. To calculate the \bf{Volume} of the cube in Figure~\ref{fig:3-3-3-cube}, just raise the side measurement to the third power.

	\begin{align}
		V &= (3 cm)^3 \\
		  &= (3 \times 3 \times 3) cm \\
		  &= 27 cm^3
	\end{align}
\end{definition}

Now, let's learn how to convert \bf{Derived Units}:

\newpage

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|l| }
			\hline
			Length & Weight & Liquid Capacity \\
			\hline
			\multirow{3}{4em}{} 1 foot = 12 inches      & 1 pound = 16 ounces 	& 1 cup = 8 fluid ounces    \\
								1 yard = 3 feet 		& 1 ton = 2,000 pounds 	& 1 pint = 2 cups           \\
								1 mile = 5,280 feet 	& 						& 1 quart = 2 pints         \\
								1 mile = 1,760 yards	& 						& 1 gallon = 4 quarts       \\
								\hline
		\end{tabular}

		\caption{U.S. Customary Units of Measure}
		\label{table:U.S.-customary-units-of-measure}
	\end{table}
\end{center}

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|l| }
			\hline
			Length & Weight/Mass & Liquid Capacity \\
			\hline
			\multirow{3}{4em}{} 1 foot = 2.54 centimeters   & 1 pound = 0.454 kilograms & 1 gallon = 3.785 liters           \\
							    1 meter = 39.37 inches 	    & 1 ton = 2.2 pounds 		& 1 liter = 0.264 gallons  	 	    \\
								1 mile = 1.609 kilometers 	& 1 ounce = 28.35 grams		& 1 liter = 1,000 cubic centimeters \\
								1 kilometer = 0.6214 miles	& 							& 1 milliliter = 1 cubic centimeter \\
													\hline
		\end{tabular}

		\caption{Metric Units of Measure}
		\label{table:metric-units-of-measure}
	\end{table}
\end{center}

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|c|l| }
			\hline
			Prefix & Symbol & Multiplier & Example \\
			\hline
			\multirow{3}{4em}{} Kilo- 	& k     & $1,000$   & Kilometer (km) 	\\
								Hecto- 	& h 	& $100$ 	& Hectometer (hm) \\
								Deca- 	& da	& $10$ 		& Decameter (dam) \\
								Unit	&  		& $1$ 		& Meter (m) 			\\
								Deci-	& d 	& $0.1$ 	& Decimeter (dm) 	\\
								Centi-	& c 	& $0.01$ 	& Centimeter (cm) \\
								Milli-	& m 	& $0.001$   & Millimeter (mm) \\
								\hline
		\end{tabular}

		\caption{The \bf{Metric System Prefix References}}
		\label{table:metric-system}
	\end{table}
\end{center}

\begin{example}[Fractional Unit]
	If you start with a number that has a
	factional unit, there may be more steps to the \bf{Dimensional Analysis}.
	It doesn't have to be harder, you still want to make sure that you cancel out
	all the units except the units you want in your final answer. Look at the
	following problem to see how to change the numerator and/or denominator of a
	fractional unit.

	In this problem, you've been asked to convert from the fractional unit
	kilometers per hour to the unit centimeters per second. This means that
	you'll need to convert the numerator and denominator to get the final answer.

	When you have a given that has a fractional unit, it can be helpful to write
	the measurement as a fraction with an one in the denominator:

    \begin{align}
    	\frac{0.36 km}{1 h}
    \end{align}

	This makes it more obvious that there are two numbers that need to convert:
	$0.36 km$ and $1 h$.

	Once you know what needs to be converted, you can determine those
	relationships that will help you go from kilometers to centimeters and hours
	to seconds.

	Since you need to change both the numerator and the denominator, you can
	start with the numerator first. The one hour is still in the denominator, but
	you can ignore it until you get kilometers converted to centimeters:

    \begin{align}
    	\frac{0.36 \rm{ \sout{km}}}{1 h} \times \frac{1,000 \rm{ \sout{m}}}{1 \rm{ \sout{km}}} \times \frac{100 cm}{1 \rm{ \sout{m}}}
    \end{align}

	Notice that the units that need to be canceled ($km$ and $m$ ) are all
	diagonal from each other. That leaves centimeters as the only unit that does
	not cancel. It is in the numerator, where it needs to be for the final
	answer.

	Now that we've changed the numerator from the unit $km$ to $cm$ on top, you
	can continue the problem by changing the denominator from $hr$ to $sec$:

    \begin{align}
    	\frac{36,000 cm}{1 \rm{ \sout{h}}} \times \frac{1 \rm{ \sout{h}}}{60 \rm{ \sout{min}}} \times \frac{1 \rm{ \sout{min}}}{60 sec}
    \end{align}

	Notice that you need to put hours on the top of the next conversion in order
	to cancel out with ours in the denominator from $hr$ to $sec$

	Notice that you need to put hours on the top of the next conversion in order
	to cancel out with hours in the denominator of the given. In the next step,
	the minutes unit is canceled to leave seconds in the denominator -- where it
	needs to be for the final answer:

	\begin{align}
		\frac{36,000 cm}{1 \rm{ \sout{h}}} \times \frac{1 \rm{ \sout{h}}}{60 \rm{ \sout{min}}} \times \frac{1 \rm{ \sout{min}}}{60 sec} &= 10 \frac{cm}{s} \\
		                                                                              &= 0.36 \frac{km}{h} \\
		                                                                              &= 10 \frac{cm}{s}
	\end{align}
\end{example}

\begin{example}[Powered Unit] Some units are raised to a power.
	\bf{Area} and \bf{Volume} of solids are the common ones.

	Area is measured in squared length, such as the square meter, $m^2$, while
	\bf{Volume} is measured in cubic length, such as the cubic meter, $m^3$.
	Because one cubic centimeter equals $1$ milliliter, that conversion is quite
	simple. However, there are times you must convert from one cubic meter to a
	cubic centimeter. Let's look at an example:

	Convert the \bf{Volume} $5.0 m^3$ to the unit $cm^3$.

	Just like any unit conversion, we look at the starting and ending units, then
	think about what we know that can help us convert the units. You haven't
	learned an equivalent relationship  between cubic metres and cubic
	centimeters, but you know the relationship between the meters and
	centimeters:

    \begin{align}
    	1 m = 100 cm
    \end{align}

	We simply use the relationship we already know and cube the entire thing.
	This makes the fraction a three dimensional \bf{Volume} measurement
	instead of a one dimensional length measurement:

    \begin{align}
    	\left( \frac{100 cm}{1 m}\right)^3
    \end{align}

	After we do that, we can setup the problem, by starting with the given
	measurement and adding the relationship we already know.

	The units $m^3$ and $m$ cannot cancel out because they are not exactly the same. The entire conversion factor must be cubed in order to make the units match:

	\begin{align}
		5.0 m^3 \times \frac{100 cm}{1 m} &= 5.0 m^3 \times \left( \frac{100 cm}{1 m} \right)^3 \\
		                                  &= 5.0 m^3 \times \left( \frac{1,000,000 cm^3}{1 m^3} \right)
	\end{align}

	Now that the cubic meter units are able to cancel, it's time to calculate the answer. Because the entire fraction is being cubed, the numbers and units must all be cubed:

	\begin{align}
		5.0 m^3 \times \frac{1,000,000 cm^3}{1 m^3} \times \frac{5.0 \times 1,000,000 cm^3}{1} = 5,000,000 cm^3 \rm{ or } 5 \times 10^6 cm^3
	\end{align}

	Therefore, $5.0 m^3$ is equivalent to $5 \times 10^6 cm^3$.
\end{example}

\newpage


\section{Baseline}
\label{sec:Baseline}
In this section, we analyze the results obtained from applying our methodologies on synthetic data and establish their efficacy.

\subsection{Ground truth: Task $1_a$}
\label{sec:Ground truth: Task $1_a$}
\graphicspath{{pictures/task1a/}}
For task1\textsubscript{a}, we use $\Omega_1$ as our synthetic data and calculate the 10 most negatively correlated and 10 most positively correlated features with $Death$.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a1.png}
  \caption{Correlations between Death and other features for the set $\Omega_1$.}
    \label{fig:1a1}
\end{figure}
In Figure \ref{fig:1a1} we look at the correlation between $Death$ and some of our features. As expected, vaccines are the most negatively correlated with $Death$, especially $Vaccine3$, which seems to be the most efficient. On the other hand, $CovidPositive$ appears most predictive of death, followed by some symptoms.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a2.png}
  \caption{Auto-correlations among features for the set $\Omega_1$.}
  \label{fig:1a2}
\end{figure}

In Figure \ref{fig:1a2}, we observe the auto-correlations between features. We notice a high correlation between $CovidPositive$ and some symptoms, such as $No\_Taste/Smell$ and $Pneumonia$. In addition, there is a high auto-correlation between $Income$ and $Age$. For this reason, we consider only the $CovidPositive$ population and remove the $Income$ variable from this data to obtain all independent explanatory variables, which is a necessary assumption for our chosen $Logistic$ $Regression$ model.
%Reference this section in the results, because this happens in the real data

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a3.png}
  \caption{Correlations between $Death$ and selected features for the set $\Omega_1$.}
  \label{fig:1a3}
\end{figure}

At this point, we observe the new \textbf{Figure \ref{fig:1a3}} containing a distinct correlation-rank. Indeed, there is a significant difference from \textbf{Figure \ref{fig:1a1}}, because previously the feature $CovidPositive$ brought some auto-correlated symptoms to the top-correlated features with $Death$. This is to say that, since $CovidPositive$ is highly correlated to $Death$ and some symptoms are auto-correlated to $CovidPositive$, then these symptoms seemed to also be correlated with $Death$. However, this logic contradicts the underlying process to generate the synthetic data and is not desirable. When we analyze the data within the $CovidPositive$ population, we can then notice that these specific symptoms do not explain $Death$, but only $CovidPositive$. Under this analysis, the auto-correlated symptoms features disappear from the top-ranked and are replaced by other variables that actually explain $Death$ according to our design of the synthetic data.

Our results here establish that methodology\textsubscript{1} can reliably identify features which affect patient outcomes either positively or negatively. The method reliably and correctly identifies the known positive relationship between the $Age$, $Gene_1$, $Hypertension$, $Gene_2$, $Diabetes$ features with $Death$ as we defined in the synthetic data generation.

%Methodology 2 section for this part
Following methodology\textsubscript{1}, we then apply methodology\textsubscript{2} on $\Omega_1$.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a7.png}
  \caption{Conditional Probability of Death in $\Omega_1$ without comorbidities.}
  \label{fig:1a7}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a8.png}
  \caption{Conditional Probability of Death in $\Omega_1$ with comorbidities.}
  \label{fig:1a8}
\end{figure}

In \textbf{Figures \ref{fig:1a7}} and \textbf{\ref{fig:1a8}} we observe the conditional probability when comorbidities are not present and present, respectively. The conditional probability of death is observed to be higher when $Diabetes$ and $Hypertension$ are present, as well as with $Gene_1$ and $Gene_2$. In the latter case, the effect appears less pronounced, however in our synthetic data it is the combined presence of both genes that increases the probability of death, so individually their impact on the chance of death is less noticeable in this methodology. Nonetheless, the features which cause death in our synthetic data are correctly identified and the increased effect on death is clearly shown over the 1000 probability estimates.

%Write Methodology 3 section under here for this section
Finally we apply methodology\textsubscript{3} to $\Omega_1$.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a9.png}
  \caption{Coefficient Values of Logistic Regression on $\Omega_1$, after randomized grid-search 500-fold CV.}
  \label{fig:1a9}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1a10.png}
  \caption{95\% Confidence Interval for Logistic Regression Coefficient Values.}
  \label{fig:1a10}
\end{figure}

As described in Section \ref{sec:methodologies}, we train a $Logistic$ $regression$ classifier to predict $Death$. The best performing model scored an accuracy of $0.82$, and we then observed the coefficient values of the model. In \textbf{Figure \ref{fig:1a9}}, we show the 5 largest positive and 5 largest negative coefficient values for the classifier. The vaccines have by far the largest negative coefficient value in predicting $Death$, as expected. On the other side, $Hypertension$, $Gene_2$, $Gene_1$, and $Diabetes$ are the most useful features for the model to predict $Death$. We can then conclude that as with the first two methodologies, methodology\textsubscript{3} accurately identifies the features which predict death and our overall framework for analyzing the real data is robust.

%End of this section
\subsection{Ground truth: Task $1_b, 1_c$}
\graphicspath{{pictures/task1b/}}
In this subsection we use another synthetic dataset $\Omega_2$, in which we imposed a correlation between each vaccine and a specific side effect.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1b1.png}
  \caption{Correlations between Death and selected features for the set $\Omega_2$.}
    \label{fig:1b1}
\end{figure}

Similarly to \textbf{Figure \ref{fig:1a1}}, in \textbf{Figure \ref{fig:1b1}} we observe the positive and negative correlations between features and death. These two figures are very similar, with the exception that side-effects from the vaccines now appear among negatively correlated features. These results are consistent with the generation of $\Omega_2$ as described in Section \ref{sec:synthetic_data}.

%Write Methodology 2 for this task below here (shorter than before since it's already described in the previous subsection)

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1b5.png}
  \caption{Conditional Probability of Death in $\Omega_2$ without comorbidities.}
  \label{fig:1b5}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1b6.png}
  \caption{Conditional Probability of Death in $\Omega_2$ with comorbidities.}
  \label{fig:1b6}
\end{figure}

\textbf{Figures \ref{fig:1b5}} and \textbf{\ref{fig:1b6}} again show the effect of the vaccines in reducing the likelihood of death, as defined for $\Omega_2$. We then also investigate the conditional probability of symptoms given vaccines.

\graphicspath{{pictures/task1c/}}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1c3.png}
  \caption{Conditional Probability of Symptoms in $\Omega_2$ without comorbidities.}
  \label{fig:1c3}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{1c4.png}
  \caption{Conditional Probability of Symptoms in $\Omega_2$ with comorbidities.}
  \label{fig:1c4}
\end{figure}

These plots show the side-effect we defined for two of the vaccines (omitting $Vaccine3$ for purposes of space) in $\Omega_2$. The chance of having a $Headache$ is accurately shown to be affected by $Vaccine2$, and likewise the relationship between $Blood\_Clot$ and $Vaccine1$.

%Write Methodology 3 for this task below here (shorter than before since it's already described in the previous subsection)

Lastly, we analyze $\Omega_2$ with our third methodology. As in Task $1a$\textsubscript{\ref{sec:Ground truth: Task $1_a$}}, the model is able to learn the correct features to predict $Death$, which are the same as before because we are training the model to predict the same outcome. We can then conclude that our methods analyze the $\Omega_2$ synthetic data appropriately.

\subsection{Ground truth: Task$_2$}
\graphicspath{{pictures/task2/}}
For task$_2$, we use $\Omega_3$ as our synthetic data. In this subsection we are interested in the effect of treatments on alleviating symptoms and preventing deaths.

%Methodology 1

\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_1.png}
  \caption{Correlations between $Death$ and other features for $\Omega_3$.}
    \label{fig:2_1}
\end{figure}

In \textbf{Figure \ref{fig:2_1}} we examine the top negative and positive correlations with $Death$. As expected, among the top negative features we find the two treatments while in the positive features we observe $Age$, $Hypertension$, $Gene_1$, $Gene_2$ and $Diabetes$. We defined the data such that both treatments are effective, with $Treatment 1$ being more effective (thus both are highly negatively correlated to death). Additionally, $Treatment 1$ is associated with a side effect of $Headache$ and $Treatment 2$ is associated with a side effect of $Fever$ and these features are thereby negatively correlated with $Death$ due to their auto-correlation with the treatments.

%Methodology 2
\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_5_top.png}
  \caption{Conditional Probability of Death in $\Omega_3$ without comorbidities.}
    \label{fig:2_5}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_6_top.png}
  \caption{Conditional Probability of Death in $\Omega_3$ with comorbidities.}
    \label{fig:2_6}
\end{figure}

Methodology 2 as applied to $\Omega_3$ shows the influence of treatment on the chance of death. When treatments are not applied, the chance of death is above $14\%$ as in \textbf{Figure \ref{fig:2_5}}, both treatments are given, the probability of death is reduced to $0.01$ as in \textbf{Figure \ref{fig:2_6}}. Each treatment also individually reduces the chance of death. \textbf{Figure \ref{fig:2_6_after}} shows the probability of symptoms after treatment has been applied. Treatments predict each other's side effects because it is possible for a patient to be treated with both.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_6_after.png}
  \caption{Conditional Probability of Symptoms Given Treatments for $\Omega_3$.}
    \label{fig:2_6_after}
\end{figure}

%Methodology 3

Our final test on synthetic data is applying methodology\textsubscript{3} to $\Omega_3$. The methodology here returns the coefficient values for a $Logistic$ $regression$ model predicting death given treatment.

\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_11.png}
  \caption{Coefficient Values of Logistic Regression on $\Omega_3$, after randomized grid-search 500-fold CV.}
    \label{fig:2_11}
\end{figure}

\begin{figure}[H]
  \includegraphics[width=\linewidth]{2_12.png}
  \caption{95\% Confidence Interval for Logistic Regression Coefficient Values.}
    \label{fig:2_12}
\end{figure}

In \tetbf{Figure \ref{fig:2_11}} we show the top positive and negative coefficient values for this model. The best model accuracy for this data was $0.86$. These coefficients accurately show that treatments are highly predictive of survival in $\Omega_3$. \textbf{Figure \ref{fig:2_12}} shows the confidence interval for the coefficient values, showing that the model training procedure results in a useful range of values that allow the correct conclusion to be drawn that treatments are effective.

Therefore, given all of the experiments performed on the three synthetic datasets, we are able to determine that each of our methodologies are capable of inferring the correct conclusions to explaining the synthetic data and match how we created this data.