%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Wenneker Article
% LaTeX Template
% Version 2.0 (28/2/17)
%
% This template was downloaded from:
% http://www.LaTeXTemplates.com
%
% Authors:
% Vel (vel@LaTeXTemplates.com)
% Frits Wenneker
%
% License:
% CC BY-NC-SA 3.0 (http://creativecommons.org/licenses/by-nc-sa/3.0/)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%----------------------------------------------------------------------------------------
%	PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
%----------------------------------------------------------------------------------------

\documentclass[10pt, a4paper, twocolumn]{article} % 10pt font size (11 and 12 also possible), A4 paper (letterpaper for US letter) and two column layout (remove for one column)

\input{structure.tex} % Specifies the document structure and loads requires packages

%----------------------------------------------------------------------------------------
%	ARTICLE INFORMATION
%----------------------------------------------------------------------------------------

\title{Pricing Asian Options on Commodities with GARCH Model} % The article title

\author{
	\authorstyle{Lue Shen\textsuperscript{1}, Leheng Chen\textsuperscript{1}, Chu Song\textsuperscript{1} and Jing Qian\textsuperscript{1}} % Authors
	\newline\newline % Space before institutions
	\textsuperscript{1}\institution{Department of Mathematics, the Hong Kong University of Science and Technology, Hong Kong SAR, China}\\ % Institution 1
	}

% Example of a one line author/institution relationship
%\author{\newauthor{John Marston} \newinstitution{Universidad Nacional Autónoma de México, Mexico City, Mexico}}

\date{May 14th, 2019} % Add a date here if you would like one to appear underneath the title block, use \today for the current date, leave empty for no date

%----------------------------------------------------------------------------------------

\begin{document}

\maketitle % Print the title

\thispagestyle{firstpage} % Apply the page style for the first page (no headers and footers)

%----------------------------------------------------------------------------------------
%	ABSTRACT
%----------------------------------------------------------------------------------------

\lettrineabstract{This report studies log return time series for 16 commodities, including energy, agriculture and metals. Basic time series analysis, i.e. statistics and correlation analysis, are conducted for all commodity series. Binomial tree method, Monte Carlo simulations with log-normal assumptions and Monte Carlo simulations with GJR-GARCH model are introduced to price Asian options on crude oil, ethanol, natural gas, gold and silver. Multiple conclusions are derived based on the pricing results. Future work is also discussed at the end of this report.}

%----------------------------------------------------------------------------------------
%	ARTICLE CONTENTS
%----------------------------------------------------------------------------------------

\section{Introduction}

Asian option, also called average value option, is essentially an exotic option, whose payoff depends on the path of the underlying price. As presented in equation \ref{eq:1}, the payoff is determined by the strike price and the average underlying price (either arithmetic or geometric) during the option’s life at the maturity. This averaging feature makes it popular among derivative traders, because the payoff will not be affected dramatically from unpredictable oscillations caused by the underlying’s market performance close to maturity, and these kinds of options also provide a nice hedging choice to companies exposed to average prices, say oil production firms and farm producers. Most common underlying for Asian option are commodities and foreign exchange rate. In this report, Asian options on commodities with high liquidity will be utilized to construct numerical examples and examine proposed pricing methods.

\begin{equation} \label{eq:1}
\begin{aligned}
C_T &= (A_T(t,T) - K)^+
\\
P_T &= (K - A_T(t,T))^+
\end{aligned}
\end{equation}

Since their invention in late seventies, several pricing methods have been proposed, and generally speaking, they can be classified into three categories: semi-analytical, approximation and Monte Carlo. The semi-analytical pricing approach assumes the underlying price following a geometric Brownian motion (GBM). Taking logarithms, the average value of the GBM can be expressed in terms of the sum of normal random variables. Using fast Fourier transform (FFT) and convolutions, the final payoff can be calculated in an analytical way. \citep{carverhill1990average} Benhamou applied the same method for discrete Asian options on underlying with non-normal returns in 2000. \citep{benhamou2000fast} Approximation methods approximate the real distribution of the average underlying price at the maturity with tractable ones, such as Edgeworth series expansion \citep{turnbull1991quick} and lognormal distributions \citep{levy1992pricing}. The Monte Carlo approach is a combination of stochastic models and Monte Carlo simulations. Kemna and Vorst proposed a Monte Carlo simulation scheme for the arithmetic average, as well as a variance reduction technique in their paper \citep{kemna1990pricing}.

This report introduces an innovative Monte Carlo scheme to price a discretely monitored Asian options (average value options) on commodities by assuming that log returns of the underlying price follow GJR-GARCH model. Other non-GARCH models, i.e. binomial tree model and Monte Carlo simulations with constant volatility, are also implemented as a comparison to the proposed pricing method.

\section{Theory}

\subsection{Commodities}

A commodity, such as crude oil or wheat, is a basic good used in commerce that is interchangeable with other commodities of the same type. Commodities can be classified into a few categories according to their usages and values, say raw materials, basic resources, agricultural, and mining products, etc. The price of a commodity good is typically determined as a function of its market as a whole: established physical commodities possess actively traded spot and derivative markets.

Commodity futures are agreements to buy or sell a raw material at a specific date in the future at a particular price. The intrinsic value of one commodity future, at any time t during its life, can be expressed by equation \ref{eq:3}. Since its intrinsic value is a linear function of its underlying price and the value of the contract is zero, we assume that the future price is also a linear function of the underlying price with a coefficient close to 1 and it contains all the volatility and underlying price information for us to price their options. Due to data source limitations, prices of commodities are not available to us, hence we study their future prices as an alternative data source for underlying prices, regardless of its minor delays.

\begin{equation} \label{eq:3}
F_t = S_t - f_t
\end{equation}

In this project, 16 commodities from 3 broad categories, namely energy, agriculture and metals, are selected for analysis. For each of them, price data of the most liquid/major future contract is collected for analysis. Details as follow:

\textbf{Energy:}
\begin{itemize}
\item Crude Oil: WTI Financial Futures
\item Natural Gas: Henry Hub Natural Gas Futures
\item Refined Products: RBOB Gasonline Futures
\item Biofuels: Chicago Ethanol (Platts) Futures
\item Coal: Coal (API2) CIF ARA (ARGUS-McCloskey) Futures
\end{itemize}

\textbf{Agriculture:}
\begin{itemize}
\item Corns: Corns Futures
\item Wheats: Chicago SRW Wheat Futures
\item Soybean: Soybean Futures
\item Soybean Meal: Soybean Meal Futures
\item Soybean Oil: Soybean Oil Futures
\item Livestock: Live Cattle Futures
\item Livestock: Lean Hog Futures
\end{itemize}

\textbf{Metals:}
\begin{itemize}
\item Gold: Gold Futures
\item Silver: Silver Futures
\item Platinum: Platinum Futures
\item Copper: Copper Futures
\end{itemize}

\subsection{Asian Options}

Asian option, also named average value option, is one of the most popular path dependent options among derivative traders. Its payoff is determined by the average underlying price over some pre-set period of time. The average price of the underlying asset can either determine the underlying settlement price or the option strike price. This is different from the case of usual European options and American options, whose payoff depends on underlying price at the exercise date. Hence Asian option is a typical form of exotic options. These exotic features provide risk reduction of market manipulation of the underlying instrument close to maturity \citep{kemna1990pricing}. On top of that, the cost of Asian options is usually much lower than European or American options with same strike and maturity.

Suppose $S_t$ is the underlying price at time $t$, $A_T(0, T)$ is the arithmetic average underlying price at maturity $T$, then we have

\[ A_T(0,T) = \frac 1T \sum_{t=0}^T S_t \]

in discrete form, or in continuous form like this

\[ A_T(0,T) = \frac 1T \int_0^T S_t dt\]

For geometric average underlying price, say $\tilde{A}_T(0, T)$, can be expressed as
\[ \tilde{A}_T(0, T) = (\prod_{t=0}^T S_t )^{\frac 1T}\]

in discrete form, or in continuous form like this

\[ \tilde{A}_T(0, T) = \exp (\frac 1T \int_0^T \log S_t dt) \]

For a European Asian call or put option with strike price $K$, its payoff becomes

\[ C_T = (A_T(0,T) - K)^+\]

or

\[ P_T = (K - A_T(0,T))^+\]

In this report, only arithmetic average underlying price in discrete form is taken into considerations for final payoff. We use strike prices and maturities from 5 commodity option contracts to construct numerical examples, say European Asian call/put options, for proposed pricing models. Considering different specifications, there are 319 option contracts in total, with maturity varying from 5 days to 5 years. Their contract specifications, together with latest settlement prices, are collected as reference. Here is the list:

\begin{itemize}
\item WTI Crude Oil Asian Option
\item Chicago Ethanol (Platts) Asian Option
\item Gold American Option
\item Silver American Option
\item Natural Gas European Option
\end{itemize}

\subsection{Forward Risk Free Interest Rates}
In any pricing model based on numerical methods, the risk free rate always plays an important part and will affect the final pricing results significantly. Since all of the interested options as well as their underlyings are traded in US exchanges,  US treasury yield curve on the day we collect price data will be regarded as risk free rate in numerical models. To implement risk free rate and simulate movements of underlying prices, the forward curve can be constructed as the following:

\begin{equation} \label{eq:4}
\begin{aligned}
f(t_i, t_{i+1}) &= \frac {\log (e^{r_{t_{i+1}}t_{i+1}} / e^ {r_{t_i}t_i)}} {t_{i+1} - t_i}
\\
&= \frac {r_{t_{i+1}}t_{i+1} - r_{t_i}t_i}{t_{i+1} - t_i}
\end{aligned}
\end{equation}

In equation \ref{eq:4}, $r_{t_i}$ is the yield of a treasury instrument maturing at $t_i$, and it can be regarded as the risk free rate for $(0,t_i)$. $f(t_i, t_{i+1})$ is the forward rate in $(t_i, t_{i+1})$ derived by two treasury instruments with adjacent maturity dates.

\subsection{Asian Option Pricing Models}

\subsubsection{Binomial Tree Model}

The binomial tree model uses tree structure to simulate the path of the underlying asset over the whole life of Asian option and applies the forward shooting grid method to avoid the exponential growth of average price. In this case, we add extra grids to record the possible average price and restrict the number of possible average price to be linear growth. To update the average value when the asset value jumps, we use gird function to determine the updated average price.

Under Black-Scholes world, we assume that the volatility of asset value is constant. For the binomial tree, let the jump factor $ u, d $ to be $ u=\exp(\sigma\sqrt{\Delta t}), d=\exp(-\sigma\sqrt{\Delta t}) $ where $ \sigma $ is constant volatility, and $ \Delta t $ is the time step.

Denote $ S^n_j $ and $ A^n_k $ to be the asset value jumping upward $ j $ times and average price with index $ k $ at $ n $-th time level, respectively. To restrict the possible values for $ F $ to a certain set of predetermined values, we limit the number of averaging values to some multiple of the number of values assumed by the asset price: Assume the coefficient to be $ 1 / \rho \in \mathbb{N} $. For a given time step $ \Delta t $, we let the asset value $ S^n_j $ and the average value $ A^n_k $ to be:
\begin{equation}
\begin{split}
&S^n_j= S_0 e^{j \Delta W}, \ \ A^n_k = S_0 e^{k \Delta Y}, \\ &\Delta W = \sigma\sqrt{\Delta t},\ \Delta Y = \rho\Delta W j, k \in\mathbb{N}
\end{split}
\end{equation}

Then consider at $ (n, j) $ node, for a upward jump from $ (n, j) $ to $ (n + 1, j + 1) $, the asset price will change from $ S^n_j $ to $ S^{n + 1}_{j + 1} $. Let $ A^{n + 1}_{k^+(j)} $ to be the updated value changing from $ A^n_k $ by the upward move. Then by usual computation of updating average value, we have:
\begin{equation}\label{arith: avg}
A^{n + 1}_{k^\pm(j)} = \frac{(n + 1) A^n_k + S^{n +1}_{j \pm 1}}{n + 2}
\end{equation}
Note that $ A^{n + 1}_{k^{\pm}(j)} $ do not coincide with $ A^{n + 1}_{k'} = S_0 e^{k' \Delta Y} $ for some $ k' \in \mathbb{N} $ in general, indicating that $ k^\pm(j) $ may not be integer. Recall $ A^{n + 1}_{k^{\pm}(j)} = S_0 e^{k^{\pm}(j) \Delta Y} $ and $ S^{n + 1}_{j \pm 1} = S_0 e^{(j \ pm 1) \Delta W} $, by \eqref{arith: avg}, we equate the two parts and have the grid function:
\begin{equation}\label{arith: grid func}
k^{\pm}(j) = \frac{\ln\frac{(n + 1) \exp(k \Delta Y) + \exp((j \pm 1) \Delta W)}{n + 2}}{\Delta Y}
\end{equation}

Let $ K' $ denote a subset of integer s.t. $ A^{n + 1}_{k'} \leq A^{n + 1}_{k^{\pm}(j)} $ Define $ k^{\pm}_{floor} $ s.t. $ A^{n + 1}_{k^{\pm}_{floor}} = \max_{k' \in K'} S^{n + 1}_{k'} $. In other words, $ k^{\pm}_{floor} = \lfloor k^{\pm}(j) \rfloor $, then $ k^\pm_{ceil} = k^\pm_{floor} + 1 $. Define $ k_{floor} = \lfloor k \rfloor $ and $ k_{ceil} = k_{floor} + 1 $. At $ n $th step, we have $ A_t \in [-S^n_n, S^n_n] $ and thus $ -n/\rho \leq k \leq n/\rho $. This helps us to restrict the size of $ k $. For a $ k \leq | S^n_n | $, we have $ A^n_k \in [A^n_{floor}, A^n_{ceil}] $ and $ -n/\rho \leq k_{floor} < k_{ceil} \leq n/\rho $ at $ n $th time step.

To solve the problem that that $ k^\pm(j) $ may not be integer, we define the linear interpolation formula that will be used in our FSG method.

Let $ c^n_{j, l} $ denote the numerical approximation to the Asian call value at $ (n, j) $ node with the averaging state variable assuming the value $ A^n_l $. Similar notations for $ c^n_{j, l_{floor}} $ and $ c^n_{j, l_{ceil}} $. For $ j \in \mathbb{R} / \mathbb{N} $, the $ c^n_{j, l} $ will be approximated by linear interpolation:
\begin{equation}\label{arith: interpolation}
c^n_{j, l} = c^n_{j, l_{floor}} + \varepsilon_l (c^n_{j, l_{ceil}} - c^n_{j, l_{floor}})
\end{equation}
where $ \varepsilon_l $ is the fraction of one step $ \Delta Y $ between $ \ln A^n_{l_{ceil}} $ and $ A^n_{l_{floor}} $:
\begin{equation}
\varepsilon_l = \frac{\ln\frac{A^n_l}{A^n_{l_{floor}}}}{\Delta Y}, A^n_l =  A^n_{l_{floor}} e^{\varepsilon_l \Delta Y}
\end{equation}

Finally we have the pricing formula under a binomial tree schemes is given by:
\begin{equation}\label{arith: back induction}
\begin{split}
c^n_{j, k} &= \frac{1}{R} \left[ p c^{n + 1}_{j + 1, k^+(j)} + (1 - p) c^{n + 1}_{j - 1, k^-(j)}\right] \\
&\approx \frac{1}{R} \{ p \left[ \varepsilon_{k^+(j)} c^{n + 1}_{j + 1, k^+_{ceil}} + (1 - \varepsilon_{k^+(j)}) c^{n + 1}_{j + 1, k^+_{floor}} \right] \\
&+ (1 - p) \left[ \varepsilon_{k^-(j)} c^{n + 1}_{j - 1, k^-_{ceil}} + (1 - \varepsilon_{k^-(j)}) c^{n + 1}_{j - 1, k^-_{floor}} \right]\} \\
&n = N - 1, \cdots, 0, j = -n, -n + 2, \cdots, n, \\
&k \in \mathbb{N} \cap [-\frac{n}{p}, \frac{n}{p}]
\end{split}
\end{equation}
where the risk neutral probability of jump upward is :
\begin{equation}
p=\frac{R-d}{u-d}, \ \ R = e^{r_t (T - t)}
\end{equation}
where $ r_t $ is the forward rate at time $ t $, Finally, the terminal payoff of put and call are:
\begin{equation}\label{arith: terminal}
\begin{split}
&p^N_{j, k} = \max(K - S_0 e^{k \Delta Y}, 0) \\
&c^N_{j, k} = \max(S_0 e^{k \Delta Y} - K, 0), \\
&j = -N, -N + 2, \cdots, N
\end{split}
\end{equation}

\subsubsection{Monte Carlo Simulations with Constant Volatility}

In this method, the underlying commodity price $S_t$ is assumed to follow the stochastic process below:

\begin{equation} \label {eq:5}
dS_t = (r_t - \frac 12 \sigma^2)S_t dt + \sigma S_t dW_t
\end{equation}

where $W_t$ is a standard Wiener process, $r_t$ is the risk free interest rate at time $t$, $\sigma$ is a constant, representing the volatility of the underlying price's log return.

With US treasury yield curve, the forward the rate at time $t$ can be used to represent $r_t$ for simulations, and its calculation method is covered in section 2.3 already. With historical future prices of the commodity's major future contract, the volatility $\sigma$ can be calculated as

\begin{equation} \label{eq:6}
\begin{aligned}
r_i &= \log S_{t_i} - \log S_{t_{i-1}}
\\
\sigma &= \sqrt {\frac 1{N-1} \sum_{i=1}^N (r_i - \bar r)^2}
\end{aligned}
\end{equation}

Given a commodity start price $S_0$, the underlying price movements can be simulated by the dynamics of its log returns $\{r_i\}$:

\begin{equation} \label{eq:mcpricing}
\begin{aligned}
dS_t &= (r_t - \frac 12 \sigma^2)S_t dt + \sigma S_t dW_t
\\
d\log S_t &= (r_t - \frac 12 \sigma^2) dt + \sigma dW_t
\\
r_i &= r_{i-1} + (r_t - \frac 12 \sigma^2) dt + \sigma dW_t
\\
S_{t_i} &=S_{t_{i-1}}\exp(r_i \Delta t) = S_0\exp(\sum_{k=0}^i r_k \Delta t)
\end{aligned}
\end{equation}

The average price $A_T(0,T)$ at the maturity is essentially

\[ A_T(0,T) = \frac 1N \sum_{i=0}^N S_{t_i}\]

where $t_0 = 0$, $t_N = T$.

Without loss of generality, Asian call option is used to illustrate the Monte Carlo scheme, and the put option can be easily derived thereafter. The Asian call option payoff becomes

\[ V_T = (A_T(0,T) - K)^+\]

With $n$ paths of simulated underling prices, the fair payoff at the maturity can be expressed as an average of the payoff generated from each path.

\[\bar V_T = \frac 1n \sum_{i=1}^n V_T^i \]

Multiplying with corresponding discount factor, the value of the contract at present can be derived.

\begin{equation}
\begin{aligned}
df(0,T) &= \exp (-\sum_{i=0}^{N-1} r_{t_i}\Delta t)
\\
\bar V_0 &= df(0,T) \bar V_T
\end{aligned}
\end{equation}

where $r_{t_i}$ is the forward risk free interest rate at $t_i$, and $\Delta t = 1/252$.

Since the underlying is only traded on business days, BUS/BUS date convention is implemented in the simulation scheme.

\subsubsection{Monte Carlo Simulations with GJR-GARCH model}

As a comparison to the constant volatility MC scheme, GJR-GARCH model defines the log return of underlying price as a GARCH process. For a given time series $\{ S_t\}$,

\begin{equation}
\begin{aligned}
r_t &= \log S_t - \log S_{t-1}
\\
r_t &= \mu +\epsilon_t
\\
\epsilon_t &= \sigma_t Z_t
\end{aligned}
\end{equation}

where $Z_t$ follows a standard normal distribution and

\begin{equation}
\begin{aligned}
\sigma_t^2 &= \omega + \sum_i {\alpha_i \epsilon_{t-i}^2} + \sum_j {\beta_j \sigma_{t-j}^2}  \\
&+ \sum_k {\gamma_k \epsilon_{t-k}^2 I_{\epsilon_{t-k} < 0}}
\end{aligned}
\end{equation}

This model will illustrate the volatility behavior of the time series, and since the volatility is the source engine to generate simulated price paths, we assume that a more subtle volatility model will contribute to more accurate pricing results.

In this report, for each commodity, the best model configuration, say the number of $\alpha$, $\beta$, $\gamma$ and whether using zero mean, is selected based on the following steps.

\textbf{Step one: zero mean vs constant mean}

Constant mean model will be considered at the first place. If the p-value of $\mu$ is larger than 0.05, then the mean is considered as insignificant, and it will be removed to refit the model again as a consequence.

\textbf {Step two: Ljung-Box test}

The fitted model will be plugged into historical data, and the series of $\{ Z_t\}$ is derived. If $\{ Z_t\}$ passes Ljung-Box test at a significance level of 0.05, then it proves that the model well explains the historical time series. Otherwise, the model will be rejected.

\textbf {Step three: BIC}

Bayesian information criterion (BIC) is implemented as a criterion for model selection, instead of Akaike information criterion (AIC). This is because BIC has more punishments on number of parameters when the data set is large comparing AIC.

\begin{equation}
\begin{aligned}
AIC &= 2k - 2\ln (\hat L)
\\
BIC &= \ln(n) k - 2\ln (\hat L)
\end{aligned}
\end{equation}

where $\hat L$ is the maximized value of the likelihood function of the model, $k$ is the number of parameters of the model to estimate, and $n$ is the sample size.

If one model has a less BIC and less number of parameters than the other one, it will be selected as the temporary best model. Alternatively, if BIC of one model is $5\%$ less than the other one, it will also win the selection.

With the best fitted GJR-GARCH model, the volatility of future log returns will be predicted. A series of $\{ \sigma_0(t)\}$ can be derived, since

\[ \sigma_0(t) = E_0[\sigma(t)]\]

and then a path of underlying price can be derived from the generated log return series ${r_t}$, where

\[r_t = \mu + \sigma_0(t) Z_t\]

The rest pricing procedure is the same as the constant volatility scheme.

\section{Process}

\subsection{Data Source}

There are two types of data involved in this project, say commodity future price and option specifications. All of commodity future price data are retrieved from investing.com, a global financial portal owned by Fusion Media Limited. The time span starts from 2000-01-03 and ends at 2019-03-22. However, some futures, like silver and corn, may not have such a long price history. In this case, we take as earlier as we can, and all the future prices are available after year 2009. On the other hand, the option contract specifications and corresponding latest settle prices are recorded manually from CME group website, and there are 319*2=638 options in total (319 call put pairs).

\subsection{Data Preprocess}

The commodity prices are retrieved one by one, hence to further analyze or utilize them, one must concat price series into one data table with respect to dates. The concatenation generates plenty of missing values due to the difference in trading day conventions. For example, futures of silver, platinum and US soybean oil can be traded on Sundays, and all other future contracts will have missing prices on those days. Besides, the time span also varies from one commodity to another, which produces missing values for those with shorter time span before their first date point.

Two preprocessing rules are developed, one for time series analysis while the other for pricing.

\textbf {Preprocessing for time series analysis:}

When two time series are selected for correlation analysis or collinearity analysis, only date points when both have data will be kept and all the date points with missing values will be removed.

\textbf {Preprocessing for pricing:}

Since Asian options only have one underlying, every historical time series will be handled independently. Thus date points with missing values will be dropped in this case.

For option specification data, the currency unit for strike price is different from each other, some are using USD cent, some are 0.001 USD. A currency divisor dictionary is specially created, and the strike price will all be converted to USD before pricing.

\subsection{Correlation Analysis}

Correlation analysis is a method of statistical evaluation used to study the strength of a relationship between two, numerically measured, continuous variables. In this step, the correlation coefficients among future prices and price log returns are calculated respectively by equation \ref{eq:correlation}. Heat maps are used to reflect their relationships. The results are shown in section 4.1.

\begin{equation} \label{eq:correlation}
\begin{aligned}
\rho_{xy} &= \frac {\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)}{(n-1)s_xs_y}
\\
&=\frac {\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_{i=1}^n(x_i - \bar x)^2 \sum_{i=1}^n(y_i - \bar y)^2}}
\end{aligned}
\end{equation}

\subsection{Non-GARCH Model Pricing}

This part illustrates the process of Non-GARCH model pricing.

\subsubsection{Binomial Tree Method}

We use binomial tree model to price the current Asian option value. We first calculate the historical volatility of log return. Then we compute the forward rate based the U.S. treasury yield curve. We will use the historical volatility and forward rate as the volatility and risk free rate of the model. For choosing the time step, we consider the total number of business days from the spot date to maturity date. If the time length reaches over 1 year, we restrict the time steps to 252, so that the computational speed could be faster. Since the underlying asset is futures and each futures contract contains more than one unit of goods, hence we choose the price the option that the underlying asset contains 1 unit good.

\subsubsection{Monte Carlo Simulations}

After data preprocessing, the option specifications are stored in one data table, and the underlying table is stored in the other. To price one option with Monte Carlo simulations, the option specifications are retrieved as strike price $K$, start date $t$, maturity date $T$, and underlying name.

Using the underlying name, the fixed volatility $\sigma$ can be derived from historical underlying price data, as mentioned in section 2. The forward risk free interest rates are also calculated from treasury yield curve before simulations. The step size of Monte Carlo simulation is one business date, so the number of steps can be derived from Julian date difference of $t$ and $T$.

For every simulation, the underlying price path can be generated by following equation \ref{eq:mcpricing}. Then the average price at the maturity, say $A_T(0,T)$ can be calculated based on the prices along the path. And the final discounted final payoff can be derived correspondingly, the average value of which will be the value priced by Monte Carlo simulation method.

\subsection{GARCH Model Pricing}

Before pricing the option, the best GARCH model needs to be selected from 2000 possible models ($2\times10\times10\times10$). There are 10 possible values, say from 0 to 9, for each of $\alpha$, $\beta$ and $\gamma$, and the model can be either zero mean or constant mean. The details about model selection criterion is mentioned in section 2.4.3. With the best fitted GARCH model, predictions for mean of volatilities can be derived from the model. Since the step size is one business day, the number of predictions equal to the number of business days from commercing date  and the maturity of the option.

The rest pricing procedure are the same as those of Monte Carlo simulations mentioned in last section, except for the simulation scheme for underlying prices, which follows in section 2.4.3.

\section{Results}

\subsection{Correlation Results}

The heap maps are shown in the appendices.

Figure \ref{priceCorrelation} is the correlation coefficient heat map of future prices. It is clear that there are many strong positive linear relationships between some contracts, such as crude oil and gasoline or silver and platinum or soybean products. There is almost no strong negative linear relationships, but two commodities, say natural gas and live cattle, are less correlated to all other commodities. Except for those commodities made from same materials and those with competitive relationships, most of correlation coefficients are very high, say close to right color in heat map. These present randomness of correlations in commodity prices.

By contrast, figure \ref{logRtCorrelation} shows the correlation coefficient heat map of future log returns. Strong correlations are only presented between commodities within the same category, especially in agriculture or metals, and the log returns in same category are positively correlated.

\subsection{Model results}

\begin{table}[!ht]
\centering
\small
\begin{tabular}{ccccc}
\toprule
Underlying & $ (p, o, q) $ & $ Q(20) $ & p-value & BIC \\
\midrule
Crude Oil WTI & (1,1,1) & 13.69 & 0.84 & 21129 \\
Ethanol & (1,1,1) & 39.91 & 0.005 & 13715 \\
Gold & (1,0,1) & 27.53 & 0.12 & 14403  \\
Natural gas & (1,0,1) & 23.48 & 0.2 & 24608 \\
Silver & (1,0,5) & 29.42 & 0.07 & 12151 \\
\bottomrule
\end{tabular}
\caption{Model result: GJR model for each underlying futures}
\label{model result}
\end{table}

Table \ref{model result} shows the model we fit based on BIC criteria. The p-value of Ljung-Box test shows that the GJR-GARCH model for Crude Oil, Gold, Natural gas and Silver is adequate but the model for Ethanol is not. The order of first four GJR-GARCH models are all lest than 1, indicates that the selected model is simple. The order of the last model(Silver) is (1, 0, 5), indicating that the volatility of Silver contract has long memory compared to other underlying futures.

\paragraph{Convergence of the Monte Carlo simulation}

Figure \ref{oil conv}, \ref{eth conv}, \ref{silver conv}, \ref{gas conv} shows the sample mean of the  call option payoff on Monte-Carlo simulation, The x-axis shows the simulation times and y-axis shows the estimated option value. These figures shows that the option value convergences to a single value and hence the option value from the Monte-Carlo simulation is rational. The prediction of the volatility is shown in appendix.

For the three pricing model(GARCH-MC, MC, BT), we use the following criterion to evaluate the performance of the model:

\paragraph{ARE}

We use ARE to evaluate the difference between market price and fair price derived from the model. The formula of ARE is:
\begin{equation}\label{are}
ARE = \frac{1}{N} \sum^N_{j = 1} \frac{\vert V^{model}_j - V^{market}_j \vert}{V^{market}_j} \times 100
\end{equation}
where $ N $ is the total number of the samples, $ V^{model}_j $ and $ V^{market}_j $ are the fair price from the model and the market price of the $ j $-th sample, respectively. The results are shown in table \ref{are stat} \citep{zhu2015model}.

Compared to market price, GARCH-MC model outperforms to other model when the underlying assets are Ethanol and Silver, and call option on Gold, put option on Crude Oil. The MC model outperform on call option on Crude Oil, put option on Natural gas. The BT model outperforms only on call option on Natural gas. Therefore, the fair price form GARCH-MC model is the closets price to the market price.

\paragraph{Fair price of European option}

Instead of market price, the European option price calculated from BS model can also be used as criterion. The analytical value calculated from the BS formula is also a fair price under BS world. Since Asian option uses the average price of the whole path to compute the payoff while European option only consider the asset price on maturity, the risk of holding one unit of European option should be larger than holding one unit of Asian option. Hence the fair price of the Asian option should be smaller than the European option value with the same specification of options.

Denote $ N_s $ as the total number that the fair price of Asian option is smaller then the corresponding European option with the same specification and $ N $ as the total number of samples. We use the proportion $ PS = N_s / N $ as the statistic to evaluate the model, the result is:
\begin{table}[!ht]
\small
\centering
\begin{tabular}{cccc}
Models & $ PS_{call} $ & $ PS_{put} $ \\
GARCH-MC & 1 & 1 \\
MC & 1 & 1 \\
BT & 1 & 1 \\
\end{tabular}
\caption{Proportion of number of Asian option value smaller than European option value with same specification}
\label{PS}
\end{table}

From table \ref{PS}, we can conclude that all the value from our model are rational.

\section{Discussions}

\subsection{Data Limitations}
Commodity futures are the assessment of raw materials trading in open market. Future prices are essentially estimates for the value of future commodities, instead of the spot price. The difference between two prices, say future price and spot price, will be affected by delivery time, risk-free rate, storage cost and convenience yield. Therefore, it will bring inaccuracies into the pricing process when using futures price not the commodity spot prices.

Besides, only WTI Crude Oil Asian option and Chicago Ethanol (Platts) Asian option are Asian options and all the rest options are either American or European options, specifications of which are used to construct numerical examples for the proposed pricing models. In future studies, more traded Asian options settings should be collected to examine these models.

\subsection{Model Limitations}
Chicago Ethanol (Platinum) futures has many duplicate prices on adjacent dates in the early stage, resulting in zero log returns. Moreover, there is no fitted GJR-GARCH model for it, and GJR-GARCH(1,1,1) is used to fit its time series of log returns. However, ARIMA(0,0,1) can sufficiently explain the time series, we may consider applying ARIMA GARCH to fit the historical data and embedding into Monte Carlo scheme in further studies.

\subsection{Monte Carlo Limitations}

In Monte Carlo GARCH method, although Monte Carlo simulations are executed 10000 times, the standard deviation of payoffs generated is relatively large compared to constant volatility Monte Carlo method. This indicates that the number of paths might not be enough. Hence in GARCH Monte Carlo simulations, the final payoff could be a partial solution, and it may take million paths to obtain an output with lower variance. On top of that, the payoff variance from the proposed GARCH Monte Carlo simulation might underestimate the real variance, especially when the number of simulated path is insufficient. In this case, probability bounds analysis (PBA) can be implemented as an examination for partial information. Using PBA, the sparse simulations will provide far-apart upper and lower bounds, and this will well estimate the risk in pricing results.

\subsection{Advanced GARCH Models}

Affine GARCH model is described as the following:

\begin{equation}
\begin{aligned}
r_t &= r - \frac 12 \sigma_t^2 + \sigma_t Z_t
\\
\sigma_t^2 &= \omega + \alpha_1 (Z_{t-1} -\lambda \sigma_{t-1})^2+\beta_1 \sigma_{t-1}^2
\end{aligned}
\end{equation}

According to Lorenzo, a semi-analytical formula for geometric Asian options can be provided when the underlying follows an affine GARCH process and the extreme asymmetry of this kind of models with non-normal innovations will price more accurately for options with very short time to maturity\citep{lorenzo2011pricing}. Hence, affine GARCH might provides a better pricing result if it is implemented to Monte Carlo simulation scheme.

\section{Conclusions}

In this report, 16 commodity future prices are studied, and 5 options of them are priced by three pricing methods, say binomial tree model, Monte Carlo simulations with constant volatility, and Monte Carlo simulations with GJR-GARCH volatility. Strong sector correlations are discovered in the 16 time series of commodity log returns. According to pricing result analysis, the ARE criterion shows that GJR-GARCH model is the most appropriate pricing model in three models implemented in this report. However, there are also a few limitations in this report, say the data limitations and Monte Carlo convergence limitations. There are also other advanced GARCH models, say affine GARCH, to examine in the future studies.

%----------------------------------------------------------------------------------------
%	BIBLIOGRAPHY
%----------------------------------------------------------------------------------------
\printbibliography[title={References}] % Print the bibliography, section title in curly brackets

%----------------------------------------------------------------------------------------
\onecolumn
\newpage
\section{Appendices}
\subsection{Figures}
\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{priceCorrelation.jpg} % Figure image
	\caption{Correlation Analysis of Commodity Prices} % Figure caption
	\label{priceCorrelation} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure} [!ht]
	\includegraphics[width=\linewidth]{logRtCorrelation.jpg} % Figure image
	\caption{Correlation Analysis of Commodity Log Returns} % Figure caption
	\label{logRtCorrelation} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure} [!ht]
	\includegraphics[width=\linewidth]{Curdeoilmcconvergence1.png} % Figure image
	\caption{Convergence Plot for a Crude Oil Asian Option} % Figure caption
	\label{oil conv} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Ethanolmcconvergence1.png} % Figure image
	\caption{Convergence Plot for an Ethanol Option} % Figure caption
	\label{eth conv} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Naturalgasmcconvergence1.png} % Figure image
	\caption{Convergence Plot for a Natural Gas Asian Option} % Figure caption
	\label{gas conv} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Silvermcconvergence1.png} % Figure image
	\caption{Convergence Plot for a Silver Asian Option} % Figure caption
	\label{silver conv} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{CrudeOilvol.png} % Figure image
	\caption{Volatility Prediction on Crude Oil} % Figure caption
	\label{oil pred} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Ethanolvol.png} % Figure image
	\caption{Volatility Prediction on Ethanol} % Figure caption
	\label{eth pred} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Goldvol.png} % Figure image
	\caption{Volatility Prediction on Gold} % Figure caption
	\label{gold pred} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Naturalgasvol.png} % Figure image
	\caption{Volatility Prediction on Natural Gas} % Figure caption
	\label{gas pred} % Label for referencing with \ref{bear}
\end{figure}

\begin{figure}[!ht]
	\includegraphics[width=\linewidth]{Silvervol.png} % Figure image
	\caption{Volatility Prediction on Silver} % Figure caption
	\label{silver pred} % Label for referencing with \ref{bear}
\end{figure}

\clearpage

\subsection{Tables}
\begin{table}[!ht]
	\centering
	\footnotesize
	\begin{tabular}{lcccccccccc}
		\toprule
		\multirow{2}{*}{Models}  & \multicolumn{2}{c}{Crude Oil WTI} & \multicolumn{2}{c}{Ethanol} & \multicolumn{2}{c}{Gold} & \multicolumn{2}{c}{Silver} & \multicolumn{2}{c}{Natural gas} \\
		& put & call & put & call & put & call & put & call & put & call  \\
		\midrule
		GARCH-MC & 46.679 & 304.798 & 10.757 & 25.258 & 72.799 & 84.586 & 17.455 & 26.597 & 215.232 & 324.858 \\
		MC & 83.121 & 122.238 & 56.035 & 73.732 & 52.874 & 91.129 & 64.379 & 89.397 & 64.891 & 84.962 \\
		BT & 402.088 & 819.106 & 45.110 & 62.466 & 106.801 & 149.124 & 83.969 & 107.557 & 264.242 & 72.697 \\
		\bottomrule
	\end{tabular}
	\caption{ARE for the Asian put ans call option for each underlying futures.}
	\label{are stat}
\end{table}

\clearpage
\subsection{Scripts}

\subsubsection{corr.py}
\lstinputlisting{corr.py}

\subsubsection{binomialTreePrice.py}
\lstinputlisting{binomialTreePricer.py}

\subsubsection{biTreePriceSimulation.py}
\lstinputlisting{biTreePriceSimulation.py}

\subsubsection{garchPricer.py}
\lstinputlisting{garchPricer.py}

\end{document}


\documentclass[preprint,superscriptaddress]{revtex4-2}

%%% linking references
\usepackage{hyperref}
\hypersetup{
  colorlinks=true,
  linkcolor=blue,
  urlcolor=cyan,
}

%%% physics and math packages
\usepackage{physics} % general physics package
\usepackage{amssymb} % math fonts and symbols
\usepackage{bm} % for making math symbols bold
\usepackage{braket} % for nice brackets (e.g. |X>)
\usepackage{mathtools} % for \coloneqq

%%% shorthands to use inside math environments
\renewcommand{\t}{\text} % text in math mode
\newcommand{\f}[2]{\dfrac{#1}{#2}} % shorthand for fractions
\newcommand{\p}[1]{\left(#1\right)} % parenthesis
\renewcommand{\sp}[1]{\left[#1\right]} % square parenthesis
\renewcommand{\set}[1]{\left\{#1\right\}} % square parenthesis
\renewcommand{\c}{\cdot} % inner product
\renewcommand{\d}{\text{d}} % integration measure, e.g. d^3 x
\renewcommand{\v}{\bm} % bold symbols (e.g. for vectors)
\newcommand{\uv}[1]{\hat{\bm #1}} % unit vectors, or vector operators
\newcommand{\bk}{\Braket}
\renewcommand{\ket}{\Ket}
\renewcommand{\bra}{\Bra}

\newcommand{\B}{\mathcal{B}}
\newcommand{\E}{\mathcal{E}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\J}{\mathcal{J}}
\renewcommand{\L}{\mathcal{L}}
\renewcommand{\O}{\mathcal{O}}
\renewcommand{\P}{\mathcal{P}}
\newcommand{\Q}{\mathcal{Q}}
\renewcommand{\S}{\mathcal{S}}

\newcommand{\z}{\text{z}}
\newcommand{\x}{\text{x}}
\newcommand{\y}{\text{y}}

\newcommand{\up}{\uparrow}
\newcommand{\dn}{\downarrow}

% remove extra white space inside enumerate environments
\usepackage{enumitem}
\setlist[enumerate]{leftmargin=*}

\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex}

% for colored text
\usepackage[dvipsnames]{xcolor}
\newcommand{\blue}[1]{{\color{blue} #1}}
\newcommand{\red}[1]{{\color{red} #1}}
\newcommand{\green}[1]{{\color{ForestGreen} #1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

Dear Editors of Physical Review Research,

Please find attached our revised manuscript, ``Engineering spin squeezing in a 3D optical lattice with interacting spin-orbit-coupled fermions'', for consideration for publication in Physical Review Research.
Below, we address the recommendations and criticisms made by past referees in the peer-review process for Physical Review X, and detail the corresponding changes that we have made to our manuscript.

We were pleased to find that the referee reports were in unanimous agreement about the quality and content of the manuscript, calling it ``well written'' and saying that ``it offers a pathway towards pushing the record level of precision achievable for atomic clocks''.

The most important feedback from previous referees concerned the potential of our manuscript to attract broader interest outside the optical lattice clock community.
We have addressed these concerns in our responses to the referees, as well as corresponding changes to the manuscript.
In short, we believe that developing realistic proposal to achieve a genuine quantum advantage in a world-class sensor unavoidably requires focusing on a specific platform.
Our work is therefore well placed in the context of a broad scientific effort to a achieve a quantum advantage in emergent technologies.
We hope that with these clarifications, our paper can be accepted for publication in Physical Review Research.

Excerpts from the referee reports are written in \blue{blue}, excerpts from the previous version of our manuscript are written in \red{red}, and excerpts from the revised version of our manuscript are written in \green{green}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{First referee}

\blue{The authors replied to my previous report with exhaustive and
  mostly convincing explanations.}

\blue{In the revised version of the manuscript the novelty of their
  results is now clearly explained, which strengthens the impact of
  the work.}

\blue{The reply to my specific questions, e.g.~on the importance of
  the underlying trapping potential and robustness of the achievable
  spin squeezing in a realistic setting, is quite detailed and
  accompanied by significant extensions of the supplementary
  materials. The new version of the manuscript now contains important
  details on the practical implementation of the suggested protocol,
  while keeping an appropriate format for PRX. In light of the
  authors' reply and of the revisions made, I can now fully recommend
  the manuscript for publication.}

We thank the Referee for their positive appreciation of our work, and for the recommendation for publication in PRX.
We appreciate their comment that the revisions we made to our manuscript throughout the review process with PRX ``clearly explained the novelty of our results'' and ``strengthen[ed] the impact of our work''.

We hope that the editors of Physical Review Research agree with this Referee's assessment.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Second referee}
\label{sec:R2}

\begin{enumerate}
\item Concerning:
  \label{pt:R2.scope}

  \blue{The proposal details a protocol to use the weakly interacting,
    weakly spin-orbit coupled regime of fermions in an optical lattice
    to generate spin squeezing for increased precision in the
    state-of-the-art optical lattice clock.  The individual components
    are achievable with current technologies and have all been
    experimentally demonstrated, with many theoretical and
    experimental advancements from the groups led by the senior
    authors, A. M. Rey and J. Ye. The manuscript is in general
    well-written and accessible to the motivated reader, with
    comprehensive appendices.}

  \blue{This work represents current efforts to enhance measurement
    sensitivity through non-classical states and promises considerable
    levels of spin squeezing for the system under consideration. In
    particular, because it focuses so closely on the current
    state-of-the-art optical lattice clock, it offers a pathway
    towards pushing the record level of precision achievable for
    atomic clocks.}

  \blue{At the same time, because the work focuses so closely on this
    particular lattice clock, the experimentalists most interested in
    this work are likely two of the authors and their colleagues. This
    is my biggest concern about the impact of this work: while it does
    open up one new research opportunity, at the immediate level this
    scope seems limited. I ask that the authors address this point in
    full and the more detailed points below.}

  We appreciate the recognition that our manuscript is ``well-written
  and accessible to the motivated reader, with comprehensive
  appendices''.

  Indeed, our work focuses on current state-of-the-art optical lattice
  clocks, since it aims to push current record levels of precision
  achievable with classical sensors.  Without identifying a concrete
  system with world-class sensing capabilities, our effort would
  reduce to a mere proof-of-principle concept which, despite having
  some merit, would lack any practical utility.  Focusing on a
  specific platform is unavoidable for any {\it realistic} and {\it
    practical} proposal to advance the frontier of metrology.  Our
  work therefore fits well into the context of a worldwide scientific
  effort to achieve a quantum advantage in emergent technologies.

  To emphasize the impact of our work in this new paradigm of enhanced
  quantum metrology with correlated quantum states, and thereby to
  further illustrate the relevance of our proposal to a broader
  scientific community, we have rewritten the last paragraph of our
  introduction (left column of page 2), which previously read:

  \red{The intriguing and non-trivial observation reported here is
    that weakly interacting fermions in a lattice with dynamics
    described by the Hubbard model --- which thus obey fermionic
    quantum statistics and interact via purely contact collisions ---
    can feature all-to-all collective spin interactions that naturally
    generate spin-squeezed states in a manner robust to realistic
    experimental imperfections. In the spirit of quantum simulation,
    our work illustrates that short-range interacting systems can be
    used to emulate dynamical magnetic behaviors found in arrays of
    long-range interacting systems (such as arrays of trapped ions[36]
    or cavity QED systems[37, 38] with interactions mediated by
    phonons or photons, respectively), thus enhancing their potential
    as a metrological resource. Although surprising, this result is in
    line with the recent understandings in quantum research that
    explore how the interplay between well understood simple phenomena
    can give rise to emergent new behaviors. In this direction, our
    work emphasizes that the precise control of interactions, which
    becomes possible in 3D optical lattice systems with quantized
    motional degrees of freedom in all three directions, is a key for
    the implementation of the protocols studied here, which may enable
    new quantum technologies.}

  and now reads:

  \green{Despite an abundance of proof-of-principle experiments with
    entangled states[24, 39], so far only the remarkable example of
    LIGO[40, 41] has demonstrated a quantum advantage in a
    state-of-the-art quantum sensing or measurement system.  The new
    generation of 3D optical lattice systems have fully quantized
    motional degrees of freedom[4], allowing for precise control of
    collisional interactions.  We demonstrate how these interactions
    can naturally give rise to metrologically useful correlated
    many-body fermionic states, opening a path to not only generate
    entanglement, but also harness it to achieve a quantum advantage
    in a world-class sensor.  Such an advance will ultimately deliver
    gains to real-world applications including timekeeping,
    navigation, communication, and our understanding of the
    fundamental laws of nature[42].}

  Similarly, we have changed the following sentence in the
  Conclusions:

  \red{To our knowledge, this is the first proposal to use atomic
    collisions for entanglement generation in state-of-the art atomic
    clocks, opening a path for the practical improvement of
    world-leading quantum sensors using correlated many-body fermionic
    states.}

  to:

  \green{To our knowledge, this is the first proposal to use quantum
    correlations in a many-body fermionic system to push
    state-of-the-art quantum sensors beyond the independent-particle
    regime, thereby achieving a genuine quantum advantage.}

  We hope that these changes highlight the broader scope and relevance
  of our work.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{Abstract (L: line)}

  \blue{1. L15: The statement ``This capability exemplifies a new
    paradigm of using driven non-equilibrium systems to overcome
    current limitations in quantum metrology...'' goes unaddressed in
    the main text. The manuscript could be improved if this larger
    impact and its context were made clear, perhaps in the last
    paragraph of the introduction or in the conclusion.}

  We thank the referee for pointing out that our manuscript could be
  improved by following through on the subject of applying driven
  non-equilibrium systems to quantum metrology.  Accordingly, we have
  added the following text to the introduction, preceding the last
  paragraph (left column of page 2):

  \green{This capability mirrors efforts in other settings, such as NV
    centers[36, 37] and trapped ions[38], to enhance quantum metrology
    through the use of driven non-equilibrium phenomena.}

  As the subject of driven non-equilibrium does not play a central
  role in our manuscript, we have also rewritten the last sentence in
  the abstract to de-emphasize this point somewhat, replacing the
  text:

  \red{This capability exemplifies a new paradigm of using driven
    non-equilibrium systems to overcome current limitations in quantum
    metrology, allowing OLCs to enter a new regime of enhanced sensing
    with correlated quantum states.}

  by:

  \green{This capability allows OLCs to enter a new era of quantum
    enhanced sensing using correlated quantum states of driven
    non-equilibrium systems.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{Introduction (P: paragraph, L: line, skipping equations)}

  \blue{1. P4L11: While the orientation of the clock laser and optical
    lattice depth were parameters that were varied to examine the
    resulting optimum level of squeezing, strictly speaking these do
    not need to be actively controlled (beyond pointing/intensity
    stabilization) to generate spin squeezing itself; they just need
    to be set to a single value -- unless I am missing
    something. Perhaps this could be reworded.}

  Indeed, our squeezing protocol does not require {\it active} control
  over clock laser orientation and optical lattice depth.  To clarify
  this point, we have changed the text (page 2, top left paragraph):

  \red{To generate spin squeezing, our protocol only requires control
    over...}

  to:

  \green{To generate spin squeezing, our protocol only requires the
    capability to fix...}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning comments about the last paragraph of our introduction
  prior to revision:

  \blue{2. P5L4: The formulation ``...interact via purely contact
    collisions'' is a bit odd. Do you mean, ``solely interact via
    contact collisions''?}

  \blue{3. P5L8: The sentence ``In the spirit of quantum
    simulation... potential as a metrological resource'' seems a bit
    odd to me. Is the point that systems with short-range interactions
    can simulate long-range interacting systems, or that systems with
    short-range interactions can be used in metrology (because of
    effective long-range interactions)? Currently it reads as though
    the second point is being made, in which case I would remove ``In
    the spirit of quantum simulation'' and the parenthetical remark
    because metrology seems to be the main focus. Otherwise I would
    remove ``thus enhancing their potential as a metrological
    resource.''}

  \blue{4. P5L20: ``...which becomes possible in 3D optical lattice
    systems with quantized motional degrees of freedom in all three
    directions...''  As opposed to 3D optical lattices \_without\_
    quantized motional degrees of freedom in all three directions? The
    authors likely mean something along the lines of ``as a result of
    quantized motional degrees of freedom...''}

  These points are no longer applicable, as we have removed the
  relevant paragraph from our manuscript entirely, as detailed in
  point \ref{pt:R2.scope} of our response above.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{From the Fermi-Hubbard model to one-axis and two-axis
    twisting}

  \blue{1. P9L11: In writing ``...up through U/J=8'', it sounds as if
    there is disagreement beyond U/J=8. To prevent this confusion, I
    would reword the statement.}

  To clarify this point, we have changed the text:

  \red{... up through $U/J=8$.}

  to:

  \green{... up through (and exceeding) $U/J=8$.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{2. P9L17: In the sentence beginning with ``Note that we choose
    filling f=5/6...'' it may be good to explicitly state that the
    optimum squeezing occurs at half-filling f=1.}

  To clarify this point, we have changed the sentence:

  \red{Note that we chose filling $f=5/6$ to demonstrate that our
    protocol should work even in this highly hole-doped case; in
    practice, optimized experiments are capable of achieving fillings
    closer to $f=1$.}

  to:

  \green{Note that we chose filling $f=5/6$ to demonstrate that our
    protocol should work, albeit sub-optimally, even in this highly
    hole-doped case; in practice, optimized experiments are capable of
    achieving fillings closer to the optimal $f=1$.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{3. FIG 2: In (b.ii), is it possible to use the same vertical
    axis range as (a.ii)? This would facilitate comparison between the
    two fillings.}

  We have modified figure 2 of the main text accordingly.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{Experimental preparation of a squeezed state and practical
    considerations}

  \blue{1. P1: Rewording this paragraph could be a good opportunity to
    frame the current work beyond the single scope of the
    state-of-the-art OLC.}

  We thank the referee for identifying an opportunity for us to frame
  our work in a broader context.  Accordingly, we have replaced the
  first two sentences of Section III:

  \red{Here we discuss specific implementations of the above protocols
    in the state-of-the-art 3D $^{87}$Sr optical lattice clock (OLC).
    This system has demonstrated the capability to load a quantum
    degenerate gas into a 3D lattice at the ``magic wavelength''
    ($\lambda_{\t{lattice}}=2a\approx813$ nm) for which both the
    ground ($^1S_0,\dn$) and first excited ($^3P_0,\up$) electronic
    states of the atoms experience the same optical potential[4].}

  by:

  \green{Thus far, we have largely considered the general preparation
    of spin-squeezed states with the Fermi-Hubbard model.  Here, we
    discuss the specific implementation of the above protocols in the
    state-of-the-art 3D $^{87}$Sr optical lattice clock (OLC).  If
    successful, such an implementation would (to our knowledge) for
    the first time break through the proof-of-principle stage of spin
    squeezing efforts, and achieve a genuine metrological enhancement
    of a world-class quantum sensor.}

  \green{As required for our protocol, 3D $^{87}$Sr OLC has
    demonstrated the capability to load a quantum degenerate gas into
    a 3D lattice at the ``magic wavelength''
    ($\lambda_{\t{lattice}}=2a\approx813$ nm) for which both the
    ground ($^1S_0,\dn$) and first excited ($^3P_0,\up$) electronic
    states (i.e.~the ``clock states'') of the atoms experience the
    same optical potential[4].}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{2. FIG 3: It may be helpful to the reader to add a dotted line
    to (b) and (c) indicating the boundary where the optical squeezing
    times for OAT and TAT are equal, to emphasize how TAT is faster
    for N$\sim>$500 atoms as noted in the caption.}

  We have added a dotted line to figure 3 marking the boundary at
  which optimal squeezing times for OAT and TAT are equal.
  Accordingly, we have modified the last sentence in the caption of
  figure 3, from:

  \red{... TAT ... achieves this squeezing faster for $N\gtrsim500$
    atoms.}

  to:

  \green{... TAT ... achieves optimal squeezing faster for
    $N\gtrsim400$ atoms, as denoted by a dotted line in panels ({\bf
      b}, {\bf c}).}

  Note that the change from \red{$N\gtrsim500$} to
  \green{$N\gtrsim400$} reflects the boundary between faster optimal
  squeezing under OAT vs.~TAT more accurately.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{3. P4L20: When citing the level of spin squeezing attainable
    with your protocol, including some context or comparison may be
    helpful for the non-expert to evaluate the impact of your
    protocol.}

  We thank the referee for realizing that our manuscript would benefit
  from putting the amount of spin squeezing we report into context
  with other results in the literature.  We have consequently split
  off the following text in Section III:

  \red{The results in Fig.~4 show that squeezing via OAT saturates
    with system size around $N\approx10^3$ ($\ell\approx30$), while
    TAT allows for continued squeezing gains through $N=10^4$
    ($\ell=100$).  Even with decoherence, our protocol may
    realistically generate $\sim10$--$14$ dB of spin squeezing in
    $\sim1$ second with $\sim10^2$--$10^4$ atoms in a 2D section of
    the lattice, which is compatible with the atom numbers and
    interrogation times of state-of-the-art optical lattice clocks[4,
    5].}

  into a separate paragraph that elaborates on levels of spin
  squeezing attained in the literature:

  \green{The results in Fig.~4 show that squeezing via OAT saturates
    with system size around $N\approx10^3$ ($\ell\approx30$), while
    TAT allows for continued squeezing gains through $N=10^4$
    ($\ell=100$).  Even with decoherence, our protocol may
    realistically generate $\sim10$--$14$ dB of spin squeezing in
    $\sim1$ second with $\sim10^2$--$10^4$ atoms in a 2D section of
    the lattice, which is compatible with the atom numbers and
    interrogation times of state-of-the-art optical lattice clocks[4,
    5].  This amount of spin squeezing exceeds those reported in the
    ground-state nuclear spin sublevels of a state-of-the-art
    ${}^{171}$Yb OLC ($\sim6.5$ dB)[55].  While the latter protocol
    might be used to transfer spin squeezing to the electronic clock
    state, to date there has been no demonstration of spin squeezing
    in an optical clock transition.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{4. FIG 4: For this figure, as well as for figures 2 and 3, it
    may be helpful to use different colors and colorbars, one
    color(bar) for squeezing and one for time, kept consistent
    throughout.}

  At the request of the referee, we have made the color maps used in
  figures 2, 3, and 4 all distinct.  We have done so by changing the
  color map in figure 3, so as to keep a larger number of perceptually
  distinct colors in figure 4 (which makes it easier to match a color
  to a numerical value by eye).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{5. FIG 4 caption: It would be helpful to note that the maximum
    value of the colorbar is about 20 dB less than the x-range maximum
    in figure 3a.}

  At the request of the referee, we have added the following sentence
  to the caption of figure 4:

  \green{The decoherence considered in this work also limits maximally
    achievable squeezing to $\sim20$ dB less than the decoherence-free
    maxima shown in Fig.~3.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{6. P5L3: Please quantify what you mean by ``shallow
    lattices'', both here and at the end of Appendix H.}

  This comment concerns words ``shallow lattices'' in the first
  sentence of the text:

  \red{In addition to light scattering, $p$-wave losses from inelastic
    ${}^3 P_0$ collisions[17, 19, 50] can also degrade the maximum
    achievable spin squeezing, particularly in shallow lattices.  More
    details on $p$-wave losses are discussed in Appendix H, where we
    show that operating at lattices deeper than $8 E_{\t{R}}$ may be
    necessary to suppress the impact of inelastic collisions on spin
    squeezing, at the cost of slightly increasing light scattering.}

  The purpose of the first sentence, which contains the words
  ``shallow lattices'', was not to quantify relevant lattice depths,
  but rather to communicate the fact that $p$-wave losses generally
  get worse with increasing lattice depth.  The {\it following}
  sentence then quantifies the implications of $p$-wave losses for
  operating lattice depths.  Our wording previously made this
  structure unclear, so we have changed the above sentences to read:

  \green{In addition to light scattering, $p$-wave losses from
    inelastic ${}^3 P_0$ collisions[17, 19, 56], can also degrade the
    maximum achievable spin squeezing, which becomes more pronounced
    for shallower lattices.  More details on $p$-wave losses are
    discussed in Appendix H, where we show that operating at lattice
    depths $V_0\gtrsim 7 E_{\t{R}}$ may be necessary to suppress the
    impact of inelastic collisions on spin squeezing, at the cost of
    slightly increasing light scattering.}

  Note that the change from \red{$8E_{\t{R}}$} to \green{$7E_{\t{R}}$}
  better reflects our findings in Appendix H, which determined that
  useful squeezing is still achievable in the presence of with
  $p$-wave losses with $U/J\gtrsim6$, at which $V_0\gtrsim7E_{\t{R}}$.

  For similar reasons, we have changed the following text in Appendix
  H:

  \red{As can be seen, the decay rate $\gamma$ is largely suppressed
    as $V_0$ increases but is not completely negligible at shallow
    lattices.  To quantify the effect ...}

  to:

  \green{The decay rate $\gamma$ is suppressed exponentially with
    increasing lattice depth $V_0$.  To quantify the effect ...}

  and we have changed the text (in the last paragraph of Appendix H):

  \red{For shallow lattices, the effect of inelastic collisions can
    limit the spin squeezing to $\sim10$ dB.}

  to:

  \green{For shallow lattices ($V_0\lesssim7E_{\t{R}}$), the effect of
    inelastic collisions can limit the spin squeezing to $\sim10$ dB.}

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{External consultant}

\begin{enumerate}
\item Concerning:

  \blue{The present manuscript is a theory proposal motivated very
    much by the existing, and highly optimized setup of a Sr lattice
    clock in the Ye lab at JILA. There is no question that this is a
    world leading setup. To push the accuracy further, one question is
    how -- in this existing setup -- one can generate spin squeezed
    states. I note that spin squeezing is probably relevant for clocks
    involving a relatively small number of atoms (like present trapped
    ion experiments), but for lattice clocks with a very large number
    the relevance is a question to ask. Let me also mention that spin
    squeezing has been demonstrated with cavities etc. (Thompson at
    JILA, Vuletic MIT and others), but its actual usefulness in an
    actual leading atomic clock still awaits demonstration, as far as
    I know.}

  \blue{What this paper does is to start from Hubbard models, and
    design One-Axis-Twist Hamiltonians (there is a huge literature on
    this) from the Hubbard Hamiltonian naturally appearing in the
    lattice clock in the limit of shallow lattices. This is probably a
    limit they find it necessary to work in because of other
    constraints. Their idea is to get an infinite-range interaction
    required for One-Axis-Twisting by inducing an energy gap so that
    Hubbard model reduces to the One-Axis-Twist
    Hamiltonian. Conceptual ideas like this is what I have seen
    before, e.g.~in
    \url{https://journals.aps.org/pra/abstract/10.1103/PhysRevA.80.032311},
    though in a different experimental context.}

  As the consultant says, spin squeezing has indeed been demonstrated
  in other experimental platforms, and despite numerous proposals such
  as
  \href{https://journals.aps.org/pra/abstract/10.1103/PhysRevA.80.032311}{PhysRevA.80.032311},
  the ``actual usefulness'' of spin squeezing still awaits
  demonstration.  In our view, this point highlights the relevance and
  importance of our work, which goes beyond proof-of-principle
  demonstrations that spin squeezing might be used to achieve some
  technological advantage.  Specifically, our work outlines the first
  (to our knowledge) {\it realistic} proposal to generate {\it useful}
  spin squeezing that would push the state-of-the-art in a world-class
  sensor.  To emphasize this point, we have rewritten the last
  paragraph of our introduction (as detailed in point
  \ref{pt:R2.scope} of Section \ref{sec:R2} above).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item Concerning:

  \blue{My conclusions are:}

  \blue{Pushing atomic clocks, and lattice clocks in particular, to a
    next level of accuracy, and exploring the use of spin squeezing,
    and in particular its use in an actual precision experiment, is
    highly relevant.}

  \blue{The present proposal might be a good way to go in the Jun Ye
    lab, and would thus have quite an impact if successful.}

  \blue{However, this paper is quite specialized to the JILA setup,
    and the question is if these are ideas which are conceptually new,
    and applicable in a broader context. Here my answer would be
    slightly mixed: using Spins Orbit Coupling to open gaps in the
    spectrum to make infinite range interactions is an interesting
    idea. But is it enough?}

  \blue{Of course, if they can demonstrate the existence of squeezing
    based on these ideas in a real experiment at JILA on whatever
    level, I would be unreservedly impressed and enthusiastic. As a
    theory paper alone, it might be a bit too specialized. It is the
    old question and dilemma for theorists: publish a theory paper by
    itself, or wait until experimentalists actually do it -- which
    might take a long time.}

  We appreciate the recognition that our proposal is ``highly
  relevant'', with the potential to have ``quite an impact''.  In
  light of this recognition, and the admission that the consultant
  would be ``unreservedly impressed and enthusiastic'' with a
  successful experimental implementation of our proposal, with all due
  respect we must express firm disagreement with the consultant's
  present takeaway from the ``old question and dilemma'' of whether
  theorists should publish their ideas, or wait until these ideas are
  implemented experimentally.  Due to the scarcity of realistic
  proposals for a practical quantum advantage in world-class sensors,
  we believe that our proposal and methods are worthwhile
  disseminating to a broad scientific audience that can scrutinize our
  ideas, and build on them to develop their own (possibly modified or
  improved) implementations.  We hope that the editors agree with our
  view on the importance to widely disseminate not only experimental
  results, but also theoretical proposals that guide experiments in
  ways that actually advance state-of-the-art sensors and pave the
  ground for the practical development and application of new
  technologies.

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Miscellaneous changes}

\begin{enumerate}
\item To clarify our high-level summary of the mechanism for spin
  squeezing, we have added the following sentence to the summary of
  our results in the introduction (second to last paragraph of Section
  I, top left of page 2 in the revised text):

  \green{Interactions thereby prolong inter-particle spin coherence
    through a spin-locking effect, which additionally transforms the
    dephasing effect of SOC into a collective spin squeezing process.}

\item To add structure to our manuscript, we have changed the title of
  Section II from ``\red{From the Fermi-Hubbard model to one-axis and
    two-axis twisting}'' to ``\green{Spin squeezing with the
    Fermi-Hubbard model}'', and we have added the sub-section headings
  ``\green{Model validity}'' (II.A) and ``\green{Two-axis twisting}''
  (II.B).

\item We have shortened the title of Section III, from
  ``\red{Experimental preparation of a squeezed state and practical
    considerations}'' to ``\green{Experimental implementation and
    practical considerations}''.

\item To clarify some notation, we have changed the last sentence
  before the conclusion, from:

  \red{Such a pulse can transfer $\ket{g, I=-7/2}$ to
    $\ket{e, I=-9/2}$ without affecting $\ket{g, I=-9/2}$.}

  to:

  \green{Such a pulse can transfer $\ket{g,-7/2}$ to $\ket{e,-9/2}$
    without affecting $\ket{g,-9/2}$, where $g$ and $e$ respectively
    denote the ground and excited (electronic) clock states.}

\item We have added the following text to the beginning of the caption
  to figure H1:

  \green{{\bf $p$-wave loss rates}.}
\end{enumerate}

\end{document}


%GiG
\documentclass{beamer}
\usetheme{Copenhagen}
\setbeamertemplate{navigation symbols}{}
\setbeamertemplate{headline}{}
\DeclareMathOperator*{\argmax}{arg\,max}

\usepackage{hyperref}

\definecolor{azure}{rgb}{0.0, 0.5, 1.0}
%\newcommand{\tblue}[1]{\textcolor{blue}{#1}}
\newcommand{\tblue}[1]{{\Large {\textcolor{azure}{#1}}}}
\newcommand{\thblue}[1]{{\Huge {\textcolor{azure}{#1}}}}
\newcommand{\hred}[1]{{\textcolor{red}{#1}}}
\newcommand{\furl}[1]{{\footnote{\url{#1}}}}

\newcommand{\mypause}{\pause}
%\newcommand{\mypause}{}

\title[Saravanan Thirumuruganathan]
{Lecture 6: $k$-Nearest Neighbors}

\author[CSE 5334]
{Instructor: Saravanan Thirumuruganathan}

\date[]

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

%\begin{frame}{Outline}
%  \tableofcontents
%  % You might wish to add the option [pausesections]
%\end{frame}

\section{Outline}

\begin{frame}
\frametitle {Outline}
\begin{enumerate}
\item Introduction to Classification
\item $k$-NN (Nearest Neighbor) Classifier
\end{enumerate}
\end{frame}

\begin{frame}{In-Class Quizzes}
\begin{itemize}
\item {\Large {\bf URL:}} {\LARGE \bf \url{http://m.socrative.com/}}
\item {\Large {\bf Room Name:} {\LARGE \bf 4f2bb99e}}
\end{itemize}
\end{frame}

\section{Data Mining}
\begin{frame}{}
    \begin{center}
        \thblue{Introduction to Classification}
    \end{center}
\end{frame}

\begin{frame}{Major Tasks in Data Mining}
    \begin{itemize}
        \item Predictive methods
            \begin{itemize}
                \item Given some training data, build a model and use it to predict some variables of interest for unseen data
            \end{itemize}
        \item Descriptive methods
            \begin{itemize}
                \item Given some data, identify some significant, novel and useful patterns in the data that are interpretable by humans
            \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Data Mining Tasks}
    \begin{itemize}
        \item Classification, Regression: Predictive
        \item Clustering, Association Rule mining: Descriptive
    \end{itemize}
\end{frame}

\begin{frame}{Types of Learning}
    \begin{itemize}
        \item Supervised Learning
        \item Unsupervised Learning
        \item Reinforcement Learning
    \end{itemize}
\end{frame}

\begin{frame}{Supervised Learning}
    \begin{itemize}
        \item {\bf Dataset:}
        \begin{itemize}
            \item \hred{Training} (labeled) data: $D = \{ (x_i, y_i) \}$
            \item $x_i \in \mathbb{R}^d$
            \item \hred{Test} (unlabeled) data: $x_0 \in \mathbb{R}^d$
        \end{itemize}
        \item {\bf Tasks:}
        \begin{itemize}
            \item Classification: $y_i \in \{1, 2, \ldots, C\}$
            \item Regression: $y_i \in \mathbb{R}$
        \end{itemize}
        \item {\bf Objective:} Given $x_0$, predict $y_0$
        \item {\bf Supervised} learning as $y_i$ was given during training
    \end{itemize}
\end{frame}

\begin{frame}{Unsupervised Learning}
    \begin{itemize}
        \item Given: dataset $D = \{x_i\}$
        \item Objective: Find interesting patterns without explicit supervision
        \item {\bf Tasks:}
        \begin{itemize}
            \item Clustering
            \item Outlier detection
            \item Dimensionality reduction
            \item Many more
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Reinforcement Learning}
    \begin{itemize}
        \item Training ``agents'' to take actions to maximize rewards
        \item Reinforcement is given via action-reward
        \item Objective: Find out what is the optimal action {\bf a} to take when in state {\bf x}, in order to maximize long-term reward
        \item {\bf Examples:} Learning correct answers from score, self-driving cars, learning to fly helicopters autonomously, learning to play games
    \end{itemize}
\end{frame}

\begin{frame}{Classification Methods}
    \begin{itemize}
        \item {\bf Model based:} Build a (simple) model from the training data and use it to predict unseen data
        \item {\bf Memory based:} Keep in memory all training data and use it to predict unseen data
    \end{itemize}
\end{frame}

\begin{frame}{Classification Models}

    Some of the methods we will discuss in the class:
    \begin{itemize}
        \item Tree based: Decision and Regression trees
        \item Instance based: Nearest Neighbor
        \item Bayesian and Naive Bayes
        \item Neural Networks and Deep Learning
        \item Support Vector Machines
    \end{itemize}
\end{frame}

\begin{frame}{Binary and Multi-Class Classification}
    \begin{itemize}
        \item $C=2$: Predict which of the two classes for the unseen record
        \begin{itemize}
            \item Spam or Ham for emails
            \item Benign or malignant for tumours
        \end{itemize}
        \item $C > 2$: Multi-Class classification - predict the right class.
        \begin{itemize}
            \item Categorize mail as important, social, unimportant
            \item Identify color of eyes
            \item Identify wine type from features
        \end{itemize}
        \item Multi-class classification is often much more harder
    \end{itemize}
\end{frame}

\begin{frame}{Trade-offs}
    \begin{itemize}
        \item Prediction accuracy versus interpretability
        \item Good fit versus over-fit or under-fit
        \item Parsimony versus black-box
    \end{itemize}
\end{frame}

\begin{frame}{Classification Design Cycle\furl{http://www.cs.sun.ac.za/~kroon/courses/machine_learning/lecture2/kNN-intro_to_ML.pdf}}
    \begin{enumerate}
        \item Collect data and labels (the real effort)
        \item Choose features (the real ingenuity)
        \item Pick a classifier (some ingenuity)
        \item Train the classifier (some knobs, fairly mechanical)
        \item Evaluate the classifier (needs care)
    \end{enumerate}
\end{frame}

\section{$k$-NN Classifier}
\begin{frame}{}
    \begin{center}
        \thblue{$k$-NN Classifier}
    \end{center}
\end{frame}

\begin{frame}{Instance based Classifiers}
    \begin{itemize}
        \item Store ALL the training data
        \item Use the training data to predict class label for a new record
        \item Common Examples:
        \begin{itemize}
            \item Rote-Learner: Memorize entire training data, predict value if the new record matches some training data
            \item Nearest Neighbor: Use $k$ points closest to new record to perform classification
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Nearest Neighbor Methods}
    \begin{itemize}
        \item Non-parametric, model-free approaches
        \item Formalized in 1960s
        \item Simple to understand and implement
    \end{itemize}
\end{frame}

\begin{frame}{Why $k$-NN}
    \begin{itemize}
        \item One of the top-10 Data Mining algorithms\furl{http://www.cs.umd.edu/~samir/498/10Algorithms-08.pdf}
        \item {\bf $1$-NN Error bounds:}
        \begin{itemize}
            \item When number of training data $n$ tends to $\infty$ in a $C$-Class problem then the $1$-NN error rate (1NNER)
                is bounded by $$BER \leq 1NNER \leq BER \times \left( 2 - \frac{C}{C-1} \times BER \right)$$
            \item $1$-NN Error rate is at most twice that of BER
        \end{itemize}
        \item Asymptotically Consistent: With infinite training data and large enough $k$, $k$-NN approaches the best possible classifier (Bayes Optimal)
    \end{itemize}
\end{frame}

\begin{frame}{$k$-Nearest Neighbor}
    \begin{itemize}
        \item Distance Metric: To compute the similarities between records
        \item $k$: How many neighbors to look at?
        \item A weighting function (optional)
        \item Decision strategy: Often simple majority voting
    \end{itemize}
\end{frame}

\begin{frame}{$k$-NN Algorithm}
    \begin{enumerate}
        \item Compute the test point's distance from each training point
        \item Sort the distances in ascending (or descending) order
        \item Use the sorted distances to select the $k$ nearest neighbors
        \item Use majority rule (for classification) or averaging (for regression)
    \end{enumerate}
\end{frame}

\begin{frame}{$1$-NN Example\furl{http://www.lkozma.net/knn2.pdf}}
    \begin{center}
        \includegraphics[scale=0.75]{1NN.png}
    \end{center}
\end{frame}

\begin{frame}{$k$-NN Example\furl{http://www.lkozma.net/knn2.pdf}}
    \begin{center}
        \includegraphics[scale=0.75]{kNN.png}
    \end{center}
\end{frame}

\begin{frame}{Distance Metric}
    \begin{itemize}
        \item Used to compute similarity between entities
        \item If all values are numeric, Euclidean measure is often used
    \end{itemize}
\end{frame}
\begin{frame}{Voronoi Cells in 2D\furl{http://www.lkozma.net/knn2.pdf}}
    \begin{center}
        \includegraphics[scale=0.5]{voronoi2D.png}
    \end{center}
\end{frame}
\begin{frame}{Distance Metric}
    \begin{center}
        \includegraphics[scale=0.3]{distanceMetric.png}
    \end{center}
\end{frame}

\begin{frame}{Feature Normalization}
    \begin{itemize}
        \item Features should be on the {\bf same} scale
        \item Example: if one feature has its values in millimeters and another has in centimeters, we would need to normalize
        \item Common way: Center and Normalize to get $0$ mean and unit variance $$z_i = \frac{x_i - \overline{x_i}}{\sigma} $$
    \end{itemize}
\end{frame}

\begin{frame}{Finding Optimal $k$}
    \begin{itemize}
        \item Often $k$-NN has lower error rate than $1$-NN
        \item But the error does not monotonically decrease
        \item Picking $k$: Cross validation
    \end{itemize}
\end{frame}

\begin{frame}{Impact of $k$\furl{http://courses.cs.tamu.edu/rgutier/cs790_w02/l8.pdf}}
    \begin{center}
        \includegraphics[scale=0.3]{varingK.png}
    \end{center}
\end{frame}
\begin{frame}{Impact of $k$\footnote{ISLR}}
    \begin{center}
        \includegraphics[scale=0.3]{knn2D.png}
    \end{center}
\end{frame}
\begin{frame}{Impact of $k$\footnote{ISLR}}
    \begin{center}
        \includegraphics[scale=0.3]{knnK10.png}
    \end{center}
\end{frame}
\begin{frame}{Impact of $k$\footnote{ISLR}}
    \begin{center}
        \includegraphics[scale=0.3]{knnK1K100.png}
    \end{center}
\end{frame}

\begin{frame}{Impact of $k$\furl{http://www.cs.cornell.edu/courses/CS4758/2013sp/materials/cs4758-knn-lectureslides.pdf}}
    \begin{itemize}
        \item Small $k$
            \begin{itemize}
                \item Creates many small regions for each class
                \item May lead to non-smooth decision boundaries and overfit
                \item Leads to higher variance (i.e. classifier is less stable)
            \end{itemize}
        \item Large $k$
            \begin{itemize}
                \item Creates fewer larger regions
                \item Usually leads to smoother decision boundaries (although, too smooth boundaries might underfit)
                \item Leads to higher bias (i.e. classifier is less precise)
            \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Weighted $k$-NN}
    \begin{itemize}
        \item Often you might want to use some weights
        \item Typically to give higher weights to points nearby than to points that are farther
        \item One possibility: $\frac{1}{dist^2}$ (i.e. inverse of squared distance)
        \item Alternatively, give more weight to similarity on important features $$dist(x_i, x_j) = \sum_{k=1}^{d} w_k \, dist(x_{ik}, x_{jk})$$
    \end{itemize}
\end{frame}

\begin{frame}{Computational Complexity}
    \begin{itemize}
        \item $O(nd)$ where $n$ is training set size and $d$ is the number of dimensions
        \item VERY expensive, computationally
        \item Often, special data structures such as Voronoi diagrams, KD-trees are used to speed things up.
    \end{itemize}
\end{frame}

\begin{frame}{Other Things to Watch Out}
    \begin{itemize}
        \item Missing data (features) will cause problems
        \item Sensitive to class outliers
        \item Sensitive to irrelevant features (so ensure feature engineering and normalization are done first)
    \end{itemize}
\end{frame}

\section{Summary}
\begin{frame}{Summary}

\tblue{Major Concepts:}
\begin{itemize}
    \item Major data mining tasks
    \item Classification basics
    \item $k$-NN, variants - pros and cons
\end{itemize}
\end{frame}

%\begin{frame}{Slide Material References}

%\begin{itemize}
%    \item
%\end{itemize}
%\end{frame}

\end{document}