% Title:
% 	CheatSheet
% -------------------------------------------------
% Description:
% 	Template for cheat sheets.
%
% Creator: Tommy O.

% -------------------------------------------------
% Package imports
% -------------------------------------------------
\documentclass[11pt, a4paper]{article}
\usepackage[utf8]{inputenc} % Input encoding
\usepackage[english]{babel} % Set language to english
\usepackage{graphicx} % For importing graphics
\usepackage{amsthm, amsfonts, amssymb, amssymb} % All the AMS packages
\usepackage{mathtools} % Fixes a few AMS bugs
\usepackage[expansion=false]{microtype} % Fixes to make typography better
\usepackage{hyperref} % For \href{URL}{text}
\usepackage{fancyhdr} % For fancy headers
\usepackage[sharp]{easylist} % Easy nested lists
\usepackage{parskip} % Web-like paragraphs
\usepackage{multicol} % For multiple columns
\usepackage{tikz-cd} % For diagrams
\usepackage{listings} % To include source-code
\usepackage[margin = 1.2cm]{geometry} % May be used to set margins
\usepackage{nicefrac} % Enables \nicefrac{nom}{denom}
%\usepackage[sc]{mathpazo}% A nice font, alternative to CM

% -------------------------------------------------
% Package setup
% -------------------------------------------------
\newcommand{\Title}{Cheat Sheet SUBJECT}
\newcommand{\Subtitle}{SUBTITLE}
\newcommand{\Author}{Tommy O}
\newcommand{\listSpace}{-0.2em}% Global list space

% Shortcuts for sets in mathematics
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}

% Change the section command to inluce lines, save space, etc.
\usepackage{titlesec}
\titlespacing\paragraph{0pt}{6pt plus 4pt minus 2pt}{8pt plus 4pt minus 2pt}
\titleformat{\section}
{\normalfont\Large\bfseries}{\thesection}{0em}{$\vartriangleright$ }[{\titlerule[1.2pt]}]

% -------------------------------------------------
% Document start
% -------------------------------------------------
\begin{document}
	\pagestyle{empty}
\noindent
\begin{minipage}[t]{.75\textwidth}
	\raggedright
	{\huge \scshape \Title} \\
	{\small \itshape \Subtitle \hspace{1ex}-- \Author \hspace{1ex}-- {\footnotesize Last edit: \today}}
\end{minipage}%
\begin{minipage}[t]{.25\textwidth}
	\raggedleft
	\includegraphics[width=2.5cm]{figs/logo.pdf}
\end{minipage}
\vspace{0.1em}
\hrule\hrule

% -------------------------------------------------
% Document content start
% -------------------------------------------------
\section*{An introductory section}
Plug the series expansion into the equation, collect powers of $\epsilon$ and
set to 0.
Let $A(T)$ and $B(T)$ be constants of the slow time.
Set coefficients of forcing terms to zero to avoid resonance.
Find $A(0)$ and $A(0)$ using $0 = x(0, \epsilon) = x_0 (0, 0) + \epsilon x_1 (0, 0) + \dots$. This implies that all $x_i(0,0) = 0$.
\begin{multicols}{2}
\section*{Phase plane ($\mathbb{R}^2$)}
\begin{easylist}[itemize]
	\ListProperties(Space=\listSpace, Space*=\listSpace)
	# A good website for plotting phase planes is \url{http://comp.uark.edu/~aeb019/pplane.html}.
	# Quantitative behavior with exact formulas is often unattainable,
	so we settle for qualitative behavior.
	# \textbf{Nulliclines} are curves where either $\dot{x} = 0$ or $\dot{y} =
	0$.
\end{easylist}

\section*{Limit cycles(closed orbits)}
\begin{easylist}[itemize]
	\ListProperties(Space=\listSpace, Space*=\listSpace)
	# \textbf{Ruling out closed orbits}:
	## If $\dot{\mathbf{x}} = - \nabla V$ (\textbf{gradient system}) there are no closed orbits.
	## If there exists a \textbf{Liapunov function} there is no closed orbit.
	A Liapunov function has:
	\begin{easylist}[enumerate]
		\ListProperties(Space=\listSpace, Space*=\listSpace)
		### $V(\mathbf{x}) > 0$ for all $\mathbf{x} \neq \mathbf{x}^*$(pos.def).
		### $\dot{V}(\mathbf{x}) < 0$ for all $\mathbf{x}$ (downhill flow)
	\end{easylist}
\end{easylist}
\begin{easylist}[itemize]
	\ListProperties(Space=\listSpace, Space*=\listSpace)
	## \textbf{Dulac's criterion} states that if $\nabla \cdot \left(g \dot{\mathbf{x}}\right)$
	has one sign there are no closed orbits. This is from the div. theorem:
	\begin{equation*}
		\iint \nabla \cdot \mathbf{F} \ dA = \oint \mathbf{F} \cdot\hat{\mathbf{n}} \ dr
	\end{equation*}

	# \textbf{Finding closed orbits}:
	## The \textbf{Poincaré-Bendixson theorem} states that if one can construct a trapping
	region then $\Omega$ must have a limit cycle.
	\begin{center}
		\includegraphics[width=3.5cm]{figs/trapping}
	\end{center}

	# \textbf{Relaxation oscillations} operate on two time scales; a slow
	buildup and a fast release.
\end{easylist}

\section*{References}
\small
\begin{easylist}[itemize]
	\ListProperties(Space=\listSpace, Space*=\listSpace)
	# Book number one.
	# Book number two.
\end{easylist}
\end{multicols}

\end{document}


\clearpage
\subsection{Linked List} % (fold)
\label{sub:linked_list}

Pointers and dynamic memory allocation make it possible to store values in new and interesting ways. One way of structuring this data is to dynamically allocate each value, and link these together using pointers. An illustration of this is shown in \fref{fig:linked-list}.

Linked lists have the advantage of being very fast to perform insert and delete actions, when compared with arrays. The disadvantage is an increase in storage size to keep all the pointers, and the fact you must loop through the nodes to access any value in the list.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.90\textwidth]{./topics/dynamic-memory/diagrams/LinkedList}
   \caption{Illustration of a linked list in memory}
   \label{fig:linked-list}
\end{figure}

\begin{figure}[p]
  \pseudocode{lst:iterate-linked-list}{Pseudocode for looping through a linked list}{topics/dynamic-memory/concepts/list-iterate.txt}
\end{figure}

\mynote{
\begin{itemize}
  \item A Linked List is a \textbf{term} given to a certain way data can be structured in memory.
  \item A Linked List has \textbf{Nodes}, the equivalent of the elements of an array.
  \item Each \textbf{Node}, has some data and a pointer to the \textbf{Next} element in the list.
  \item The \textbf{Last} element in the list has \textbf{Nothing} as its next node.
  \item To access a Node in the list you must loop through from the first node until you reach the node you are after.
  \item You can insert and delete elements by changing the links in the list.
  \item If the grey node in \fref{fig:linked-list} is being \emph{inserted} then the previous node must be adjusted to point to it, and it to point to the next element of the list.
  \item If the grey node in \fref{fig:linked-list} is being \emph{deleted} then the previous node changes its link to skip that node and point to the next node in the list.
  \item The pseudocode in \lref{lst:iterate-linked-list} shows the standard way of applying an action to each node of a Linked List.
\end{itemize}
}

% subsection linked_list (end)


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Homework 1}
\date{September 12, 2014}
\author{Jon Allen}
\maketitle
2.4 a(d), b, d, f, g*

2.5 c,d,i*
\renewcommand{\labelenumi}{2.\arabic{enumi}}
\renewcommand{\labelenumii}{\Alph{enumii}.}
\renewcommand{\labelenumiii}{(\alph{enumiii})}
\begin{enumerate}
\setcounter{enumi}{3}
\item
  \begin{enumerate}
  %2.4A
  \item
  In each of the following, compute the limit. Then, using $\varepsilon=10^{-6}$, find an integer $N$ that satisfies the limit definition.
    \begin{enumerate}
    \setcounter{enumiii}{3}
    \item
    $\displaystyle \lim_{n\to\infty}\frac{n^2+2n+1}{2n^2-n+2}$

    \begin{align*}
      \lim_{n\to\infty}\frac{n^2+2n+1}{2n^2-n+2}&=\lim_{n\to\infty}\frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-\frac{1}{2}n+1}\\
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-\frac{1}{2}n+1+\frac{5}{2}n}&\le
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-\frac{1}{2}n+1}\le
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-\frac{1}{2}n+1-\frac{3}{2}n}\\
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2+2n+1}&\le
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-\frac{1}{2}n+1}\le
      \frac{1}{2}\cdot\frac{n^2+2n+1}{n^2-2n+1}\\
      \frac{1}{2}&\le
      \frac{n^2+2n+1}{2n^2-n+2}\le
      \frac{1}{2}\cdot\frac{(n+1)^2}{(n-1)^2}\\
      \left\lvert\frac{1}{2}\cdot\frac{(n+1)^2}{(n-1)^2}-\frac{1}{2}\right\rvert&=\frac{1}{2}\cdot\frac{(n+1)^2-(n-1)^2}{(n-1)^2}\\
      \left\lvert\frac{1}{2}\cdot\frac{(n+1)^2}{(n-1)^2}-\frac{1}{2}\right\rvert&=\frac{1}{2}\cdot\frac{(n+1)^2-(n-1)^2}{(n-1)^2}\\
      &=\frac{1}{2}\cdot\frac{n^2+2n+1-(n^2-2n+1)}{(n-1)^2}\\
      &=\frac{1}{2}\cdot\frac{4n}{(n-1)^2}=\frac{1}{2}\cdot\frac{4n}{n^2-2n+1}\\
      &=\frac{1}{2}\cdot\frac{1}{\frac{n}{4}-\frac{1}{2}+\frac{1}{4n}}=\frac{1}{\frac{n}{2}-1+\frac{1}{2n}}\\
      1&<\frac{n}{2}-1+\frac{1}{2n}\quad\forall n\ge 4\\
      \frac{N}{2}-1+\frac{1}{2N}&>\frac{1}{2}\cdot10^k\\
      N-2+\frac{1}{N}&>10^k\\
      (10^k+2)-2+\frac{1}{10^k+2}&>10^k\\
      \intertext{We choose $N=10^k+2$ and $\varepsilon=2\cdot10^-k$}
      \left\lvert\frac{1}{2}\cdot\frac{(n+1)^2}{(n-1)^2}-\frac{1}{2}\right\rvert&\le\frac{1}{\frac{10^k+2}{2}-1+\frac{1}{2(10^k+2)}}<2\cdot10^-k\\
      \lim_{n\to\infty}\frac{(n+1)^2}{2(n-1)^2}&=\frac{1}{2}
    \end{align*}
    Using the squeeze theorem we can conclude that $\displaystyle \lim_{n\to\infty}\frac{n^2+2n+1}{2n^2-n+2}=\frac{1}{2}$

    Now we find an appropriate value of $N$ for $\varepsilon=10^{-6}$
    \begin{align*}
      \left\lvert\frac{n^2+2n+1}{2n^2-n+2}-\frac{1}{2}\right\rvert&=
      \left\lvert\frac{n^2+2n+1}{2(n^2-\frac{1}{2}n+1)}-\frac{n^2-\frac{1}{2}n+1}{2(n^2-\frac{1}{2}n+1)}\right\rvert\\
      &=\left\lvert\frac{\frac{3}{2}n}{2(n^2-\frac{1}{2}n+1)}\right\rvert=\frac{3n}{4[n(n-\frac{1}{2})+1]}\\
      \frac{3}{4}\cdot\frac{1}{n-\frac{1}{2}+\frac{1}{n}}&<\frac{1}{10^6}\\
      n-\frac{1}{2}+\frac{1}{n}>\frac{3}{4}10^6\\
      (\frac{3}{4}10^6+1)-\frac{1}{2}+\frac{1}{\frac{3}{4}10^6+1}>\frac{3}{4}10^6
    \end{align*}
    Looks like a good value for $N$ is $\frac{3}{4}10^6+1$ or 750001.
    \end{enumerate}
  %2.4B
  \item
  Show that $\displaystyle \lim_{n\to\infty}\sin\frac{n\pi}{2}$ does not exist using the definition of limit.

  Note:
  \begin{gather*}
    \sin\frac{(n+4)\pi}{2}=\sin\left(\frac{n\pi}{2}+2\pi\right)=\sin\frac{n\pi}{2}\cos 2\pi+\cos\frac{n\pi}{2}\sin 2\pi=\sin\frac{n\pi}{2}\\
    \sin\frac{(4k+a)\pi}{2}=\sin\left(2\pi k+\frac{a\pi}{2}\right)=\sin(2\pi k)\cos\frac{a\pi}{2}+\cos(2\pi k)\sin\frac{a\pi}{2}=\sin\frac{a\pi}{2}
  \end{gather*}
  We only need to look at four cases: $n=0, n=1, n=2, n=3$. I forget whether we defined $\mathbb{N}$ to include zero on that first day, but lets just include it here today. So the four values that $\left\lvert\sin\frac{n\pi}{2}-\sin\frac{(n+1)\pi}{2}\right\rvert$ can be are:
  \begin{align*}
    n=0&\to
    \left\lvert\sin 0-\sin\frac{\pi}{2}\right\rvert=1\\
    n=1&\to
    \left\lvert\sin\frac{\pi}{2}-\sin \pi\right\rvert=1\\
    n=2&\to
    \left\lvert\sin\pi-\sin\frac{3\pi}{2}\right\rvert=1\\
    n=3&\to
    \left\lvert\sin\frac{3\pi}{2}-\sin2\pi\right\rvert=1\\
  \end{align*}
  Now we notice that
  \[\left\lvert a_n-L\right\rvert+\left\lvert a_{n+1}-L\right\rvert\ge \left\lvert(a_n-L)-(a_{n+1}-L)\right\rvert=\left\lvert a_n-a_{n+1}\right\rvert=1\]

  Lets choose $\varepsilon=\frac{1}{2}$. Then $|a_n-L|<\frac{1}{2}$
  \begin{align*}
    \left\lvert a_n-L\right\rvert+\left\lvert a_{n+1}-L\right\rvert&<\frac{1}{2}+\left\lvert a_{n+1}-L\right\rvert\\
    \left\lvert a_n-a_{n+1}\right\rvert&<\frac{1}{2}+\left\lvert a_{n+1}-L\right\rvert\\
    1&<\frac{1}{2}+\left\lvert a_{n+1}-L\right\rvert\\
    \frac{1}{2}&<\left\lvert a_{n+1}-L\right\rvert\\
  \end{align*}
  Which is a problem because if $|a_n-L|<\frac{1}{2}$ then since $n+1>n\ge N$ we should have $|a_{n+1}-L|<\frac{1}{2}$ not the other way around. We must not have a limit. $\Box$
  \setcounter{enumii}{3}
  %2.4D
  \item
  Prove that if $\displaystyle L=\lim_{n\to\infty}a_n$, then $\displaystyle L=\lim_{n\to\infty}a_{2n}$ and $\displaystyle L=\lim_{n\to\infty}a_{n^2}$.
  \begin{align*}
    \left\lvert a_n-L\right\rvert&<\varepsilon\quad\forall n\ge N
  \end{align*}
    because $2n\ge n\quad\forall n\in\mathbb{N}$ we have $2n\ge n\ge N$ and therefore $\left\lvert a_{2n}-L\right\rvert<\varepsilon$. The argument is exactly the same for $n^2$ because $n^2\ge n\ge N$
  \setcounter{enumii}{5}
  %2.4F
  \item
  Define a sequence $\left(a_n\right)_{n=1}^\infty$ such that $\displaystyle \lim_{n\to\infty}a_{n^2}$ exists but $\displaystyle \lim_{n\to\infty}a_n$ does not exist.

  \[a_n=\left\lfloor\frac{\lfloor\sqrt{n}\rfloor}{\sqrt{n}}\right\rfloor\]
  %2.4G
  \item
  Suppose that $\displaystyle \lim_{n\to\infty}a_n=L$ and $L\ne0$. Prove there is some $N$ such that $a_n\ne0$ for all $n\ge N$.
  \subsubsection*{proof}
    We know that there is some $N$ such that $|a_n-L|<\varepsilon$ for all $0<\varepsilon, n\ge N$. This is equivalent to $L-\varepsilon<a_n<L+\varepsilon$.
  We have two cases. $L>0$ and $L<0$. If $L>0$ then we choose $\varepsilon=L$ and $0=L-L<a_n<L+L$. Because $0<a_n$ it is safe to say $a_n\ne0$ I think. If $L<0$ then we choos $\varepsilon=-L$ which leads to $L+L<a_n<L-L=0$. Now again, because $a_n<0$ we can say $a_n\ne0$. $\Box$
  \end{enumerate}
\item
  \begin{enumerate}
  \setcounter{enumii}{2}
  %2.5C
  \item
  If $\displaystyle \lim_{n\to\infty}a_n=L>0$, prove that $\displaystyle \lim_{n\to\infty}\sqrt{a_n}=\sqrt{L}$. Be sure to discuss the issue of when $\sqrt{a_n}$ makes sense. HINT: Express $|\sqrt{a_n}-\sqrt{L}|$ in terms of $|a_n-L|$
  \subsubsection*{proof}
  We must specify that $a_n\ge0$. This is after all \emph{real} analysis.
  \begin{align*}
    \left\lvert a_n-L\right\rvert&<\varepsilon\\
    \left\lvert\sqrt{a_n}^2-\sqrt{L}^2\right\rvert&<\varepsilon\\
    \left\lvert(\sqrt{a_n}+\sqrt{L})(\sqrt{a_n}-\sqrt{L)}\right\rvert&<\varepsilon\\
    \sqrt{a_n}+\sqrt{L}>\sqrt{L}&>0\\
    (\sqrt{L})\left\lvert(\sqrt{a_n}-\sqrt{L)}\right\rvert<(\sqrt{a_n}+\sqrt{L})\left\lvert(\sqrt{a_n}-\sqrt{L)}\right\rvert&<\varepsilon\\
    \left\lvert(\sqrt{a_n}-\sqrt{L)}\right\rvert&<\frac{\varepsilon}{\sqrt{L}}\\
  \end{align*}
  Now we can write an arbitrary $\gamma>0,\gamma\in\mathbb{R}$ as $\frac{\varepsilon}{\sqrt{L}}$ where $\epsilon>0,\epsilon\in\mathbb{R}$ and so we have the inequality $\left\lvert(\sqrt{a_n}-\sqrt{L)}\right\rvert<\gamma$ which fits the definition of a limit and proves that $\displaystyle \lim_{n\to\infty}\sqrt{a_n}=\sqrt{L}$. $\Box$
  %2.5D
  \item
  Let $(a_n)_{n=1}^\infty$ and $(b_n)_{n=1}^\infty$ be two sequences of real numbers such that $|a_n-b_n|<\frac{1}{n}$. Suppose that $\displaystyle L=\lim_{n\to\infty}a_n$ exists. Show that $(b_n)_{n=1}^\infty$ converges to $L$ also.
  \begin{align*}
    b_n-\frac{1}{n}&<a_n<b_n+\frac{1}{n}\\
    -\frac{1}{n}-a_n&<-b_n<\frac{1}{n}-a_n\\
    \frac{1}{n}+a_n&>b_n>a_n-\frac{1}{n}\\
    \lim_{n\to\infty}\left(\frac{1}{n}+a_n\right)&=0+L\\
    \lim_{n\to\infty}\left(a_n-\frac{1}{n}\right)&=L-0\\
    \lim_{n\to\infty}b_n&=L
  \end{align*}
  Using theorem 2.5.2 and the squeeze theorem. $\Box$
  \setcounter{enumii}{8}
  %2.5I
  \item
  Suppose that $\displaystyle \lim_{n\to\infty} a_n=L$. Show that $\displaystyle \lim_{n\to\infty}\frac{a_1+a_2+\dots+a_n}{n}=L$.
  \subsubsection*{proof}
  \end{enumerate}
\end{enumerate}
\end{document}


\documentclass[11pt,a4paper]{article}
\usepackage{a4wide}
\usepackage{color}
\usepackage{graphicx}
% Default margins are too wide all the way around. I reset them here
%\setlength{\topmargin}{-.5in}
%\setlength{\textheight}{9in}
%\setlength{\oddsidemargin}{.125in}
%\setlength{\textwidth}{175mm}
\setlength{\parindent}{0cm}
\begin{document}
\title{COSC3500\\Serial Implementation Report}
\author{Anthony Vaccaro\\University of Queensland}

%\today
\maketitle
\begin{abstract}
This report introduces the concept of modelling forest fires with cellular
automata, and discusses an implementation of said model.
\end{abstract}

\newpage
\section{The Forest-Fire Model}

The Forest-Fire model is a cellular automaton-based model of a wild fire
spreading through a forest of trees or other plants. The model has a few
variants but typically follows these rules:

\begin{itemize}
\item An empty cell becomes a tree with probability p.
\item A tree becomes a burning tree with probability f.
\item A tree becomes a burning tree if its neighbours are burning.
\item A burning tree becomes an empty cell.
\end{itemize}
Where variables p and f can be adjusted to produce different patterns.

These simple rules can produce complex patterns and have lead to some research
into the area of modelling forest fires.

\newpage
\section{Algorithm used to represent model}

The algorithm chosen for this report and corresponding program is based on the
rules given above, with a few minor adjustments:

\begin{itemize}
\item Burning trees take l timesteps to become empty.
\item Trees with burning neighbours have probability c to catch fire.
\end{itemize}

This gives provides the simulation with four main variables to adjust - p, f,
l, and c.

\includegraphics{tex/flowchart}

\newpage
\section{Implementation details}

The implementation was chosen to be written in C, based on the student's past
experience with programming. The implementation did not require any complex
programming techniques, although a few key points are worth discussing.

As much of this simulation relies on probabilities, the use of a random number
generator requires some thought. the \texttt{random()} function of the C
standard library was selected due to its ease of use and availability, but as
it is only a pseudo-random number generator, perhaps this could be changed in
later efforts. There are websites that provide "true randomness" based on
atmospheric noise\cite{random}, and perhaps data from this service could be
used and compared against the output of \texttt{random()}.

Exporting the output of the program was done via the use of
libpng\cite{libpng}, which can easily convert 2d arrays into pictures. Sample
code was used from a third party website in order to assist with this
process\cite{png}. After each timestep was completed, the grid of cells was
exported to an image, with pixels being coloured based on their status.

\newpage
\section{Verification of program's output correctness}

The implementation currently uses \texttt{random()} for its random number
generation, and this can be made deterministic by using a predefined seed with
the \textt{srandom()} function. This will be used to compare the results of
later optimisations and multiprocessing techniques.

\newpage
\section{Algorithm's performance while scaling with forest size}

As the algorithm operates on neighbours, which remain constant with an
increasing grid size, performance should scale linearly with number of cells in
the grid.

\begin{verbatim}

$ time ./forest 100 0 100
real    0m1.179s
user    0m0.433s
sys     0m0.046s

$ time ./forest 200 0 100
real    0m2.524s
user    0m1.802s
sys     0m0.113s

$ time ./forest 300 0 100

real    0m5.395s
user    0m4.500s
sys     0m0.172s

\end{verbatim}

\newpage
\section{Further Considerations}

\newpage

\begin{thebibliography}{99}

\bibitem{png}
"Write a PNG file using C and libpng", 2010 Ben Bullock
http://www.lemoda.net/c/write-png/

\bibitem{libpng}
"libpng Home Page", 2013 Greg Roelofs
http://www.libpng.org/pub/png/libpng.html

\bibitem{random}
"RANDOM.ORG - True Random Number Service", 2012 RANDOM.ORG
http://www.random.org/

\end{thebibliography}
\end{document}


\chapter{Reinforcement Learning}
\label{chapter2}
\thispagestyle{empty}

\begin{quotation}
{\footnotesize
\noindent{\emph{All models are wrong, but some are useful. \\
} }
\begin{flushright}
George E. P. Box
\end{flushright}
}
\end{quotation}
\vspace{0.5cm}

\noindent In this chapter, we present the basics of the Reinforcement Learning framework needed for contextualize the work of the following chapters. We will first introduce the Markov Decision Process model, \cref{sec:MDP}, then we present the main approaches for the \textit{exact} solution of Markov Decision Processes, namely \textit{Linear Programming}, \cref{sec:lin_prog}, and \textit{Dynamic Programming}, \cref{sec:dyn_prog}. In \cref{sec:policy-search} we present \textit{Policy Search} algorithms, that represents an approximate solution to the Markov Decision Processes problem. At the end of this chapter we will consider some notable extensions to the Markov Decision Process framework, \cref{sec:mdp-ext}.
\section{Markov Decision Processes} \label{sec:MDP}
% Informal introduction to MDP
A Markov Decision Process \citep{puterman2014markov} (MDP) is a formal framework for modelling sequential decision making problems. The decision maker is usually called \textit{agent}.
 In MDPs an agent interacts with an \textit{environment} through \textit{actions} and receives a \textit{reward} based on the action and on the current state of the environment. The goal of the agent is to maximize the cumulative sum, possibly discounted, of rewards in a given \textit{horizon} (possibly infinite). The task the agent has to learn is defined through the rewards it receives.
 \subsection{Formal Model}
 %  Formal introduction to MDP
In this work we consider finite-time MDPs in which time is divided in discrete steps. At each time step $t = 0,1,...,H$ the agent receives a representation $s_t$ of the state of the environment, $s_t \in \mathcal{S}$, takes an action, $a_t \in \mathcal{A}$, receives a reward, $r_t \in \mathbb{R}$, and lands in the next state $s_{t+1}$ according to the environment dynamics $P: \mathcal{S} \times \mathcal{A} \rightarrow \Delta(\mathcal{S})$. Here $H$ denotes the horizon length, $H \in \mathbb{R}^+ \cup \{+\infty \} $.\newline
Formally an MDP is a tuple $\langle\mathcal{S}, \mathcal{A}, R, \gamma, P, \mu\rangle$, where:
\begin{itemize}
\item $\mathcal{S}$ is  the \textit{state space} (discrete or continuous);
\item $\mathcal{A}$ is the \textit{action space}  (discrete or continuous);
\item $R : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$ is the \textit{reward function};
\item $\gamma \in [0,1]$ is the \textit{discount factor} for future rewards;
\item $P: \mathcal{S} \times \mathcal{A} \rightarrow \Delta (\mathcal{S})$ is the \textit{transition function}, where $P(\cdot | s, a)$ is the probability distribution over the next states given that the agent executes action $a$ in state $s$;
\item $\mu \in \Delta(\mathcal{S}) $ is the initial state distribution.
\end{itemize}

Formally the reward function is defined over $\mathcal{S} \times \mathcal{A} \times \mathcal{S} $, so a reward realization is \newline $R(s_t,a_t,s_{t+1})$. Usually we forget the dependence on the state $s_{t+1}$ by taking the expectation over the next states according to the environment dynamics:
\begin{equation}
	r(s_t,a_t) = \underset{s_{t+1} \sim P(\cdot | s_t, a_t)}{\mathbb{E}} \left[R(s_t,a_t,s_{t+1})\right] .
\end{equation}
A trajectory is sequence $\tau=\langle s_t, a_t, R_t\rangle_{t=0,...,H}$ in which $s_t$ represent the current state, $a_t$ the action taken, $R_t$ the reward received at time step $t$.
\begin{figure}[tb!]
\centering
\includegraphics[width=0.8\textwidth]{pictures/agentenv}
\caption{\small Agent Environment interface, from \citep{sutton_introduction}.}
\end{figure}

 \subsection{Transition Model}
 % talk about transition model properties
The transition function must satisfy two properties in order for the problem to be an MDP:
\begin{description}
\item \textit{Markovian}: the current state and action are a sufficient statistic to define the probability over the next states;%Formally the Markovian property states that:
%\begin{equation}
%P(\cdot | s_t, a_t) = P( \cdot | s_t,a_t, s_{t-1}, a_{t-1} ..., s_0, a_0)
%\end{equation}.
\item \textit{Stationarity}: the environment dynamics does not change over time. %, this means that, if $s_{t_1} = s_{t_2}$ and $a_{t_1} = a_{t_2}$, then:
%\begin{equation}
%P(\cdot | s_{t_1}, a_{t_1}) = P( \cdot | s_{t_2},a_{t_2})
%\end{equation}.
\end{description}
These usually are assumptions made in order to simplify the problem. The Markovian property can be ensured by adding enough information to the state. The Stationarity property can be ensured in a similar way, if a task is non stationary it can be translated in a stationary task by adding the time component  to the state description.

 \subsection{Policy} \label{sec:policy}
 % talk about policy properties
The agent interacts with the environment by means of a \textit{policy} that defines its \textit{behaviour}. At each decision epoch a decision maker takes action according to its behaviour $b$. In the most general case the action $a_t$ depends on the whole history from time 0 to time $t$. We denote the set of histories with $\mathbb{H}_t$. A \textit{stochastic} behaviour $b_t$ can be defined as:
\begin{equation}
	b_t : \mathbb{H}_t \rightarrow \Delta(\mathcal{A}) \, .
\end{equation}
A behaviour is said to be \textit{Markovian} if the distribution over actions depends only on the current state:
\begin{equation}
	b_t : \mathcal{S} \rightarrow \Delta(\mathcal{A}) \, .
\end{equation}
A policy is \textit{stationary} if the distribution over actions does not depends on the time.
In this work we will denote a \textit{stationary} \textit{Markovian} policy with $\pi$:
\begin{equation}
	\pi : \mathcal{S} \rightarrow \Delta (\mathcal{A}) \, .
\end{equation}
It is useful to consider \textit{parameterized} policies, where $\boldsymbol{\theta} \in \mathbb{R}^d$ is the vector of policy parameters. To denote a parameterized policy we use the notation $\pi_{\boldsymbol{\theta}}$ and we omit the subscript when it is clear from the context that the  considered policy is parameterized. The usage of a parameterization implicitly defines a set of policy in which we are interested, we denote the policy family with $\Pi$ and we have $\pi \in \Pi$.
Common policy parameterizations are:
\begin{description}
\item \textit{Linear} (deterministic), the resulting action is a linear combination of the state features $\phi(s)$:
\begin{equation}
\pi(s) = \boldsymbol{\theta}^T \boldsymbol{\phi}(s) .
\end{equation}
\item \textit{Gaussian}, the resulting action has a gaussian distribution, in which the mean and the variance depend on state features:
\begin{equation}
\pi( \cdot | s) \sim \mathcal{N} ( m( \boldsymbol{\phi}(s)), v(\boldsymbol{\phi}(s)) ) .
\end{equation}
The Gaussian parameterization is useful for continuous action spaces.
\item \textit{Boltzmann}, used for discrete action spaces, the resulting action is a soft-max acting on the weighted state features:
\begin{equation}
\pi(a_i | s) = \frac{e^{\boldsymbol{\theta}_i^T \boldsymbol{\phi}(s)}}{\sum_j e^{\boldsymbol{\theta}_j^T \boldsymbol{\phi}(s)}} \, ,
\end{equation}
where $\boldsymbol{\theta}_i$ is the set of parameters associated to action $a_i$.
\end{description}
In all of these parametrizations the state features might be non-linear features depending on some parameters, e.g. coming from a neural network; radial basis function (RBF) features, tile coding features.

%%%%%%%%%%%%%%%%%% STATE DISTRIBUTION %%%%%%%%%%%%%%%%%%%
\subsection{State distribution} \label{sec:state-distr}
Behaving accordingly to a policy in an MDP induces a distribution over the state space. Here we recall the formulation of the $\gamma$-discounted future state occupancy \citep{pol_grad_func_approx}:
\begin{equation}
	\hat{d}_{\mu,\gamma}^\pi(s) = \sum_{t=0}^{+\infty}\gamma^t \mathbb{P}(s_t = s | \mu, P, \pi) \, .
\end{equation}
The previous formula defines $\hat{d}_\mu^\pi$ of a state $s$ as the sum of the discounted probability of being in $s$ in a time step $t$ given the policy, the initial state distribution and the transition model.
Formally $\hat{d}_\mu^\pi$ is not a distribution as it does not sum up to 1. We normalize it obtaining the $\gamma$ discounted state distribution:
\begin{equation}
	d_{\mu,\gamma}^\pi(s) = (1 - \gamma) \hat{d}_\mu^\pi(s) \, .
\end{equation}
%%%%%%%%%%%%%%%%%% STATE KERNEL %%%%%%%%%%%%%%%%%%%
\subsection{State Kernel}
The transition model and the policy naturally define the \textit{state kernel}, that has a primary role in our work. Formally the state kernel is a function $P^\pi : \mathcal{S} \rightarrow \Delta(\mathcal{S})$. It defines, for each state, a probability measures over the set of states. It consider jointly the effect of the transition model and the policy.
The state kernel is obtained by marginalizing the transition model over the actions:
\begin{equation}
	P^\pi (s' | s) = \int_\mathcal{A} \pi(a | s) P(s' | s, a) \mathrm{d}a \, .
\end{equation}
%%%%%%%%%%%%%%%%%%%%%%%%%% GOALS and REWARD %%%%%%%%%%%%%%%%%%%
 \subsection{Goal and Rewards}
 % talk about performance objectives, like average reward setting, full return etc
 In Reinforcement Learning the agent's goal is to maximize the total amount of rewards it receives. This is based on the reward hypothesis \citep{sutton_introduction}:

\begin{quotation}
	 \emph{That all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward).}
\end{quotation}

 Even if a scalar reward signal might seem limiting, this concept has proven to be very flexible and applicable to a wide range of tasks. For example, in locomotion tasks the reward can be defined to be proportional to the amount of distance traveled from the previous step to the current step. In making a robot reaching a goal the reward can be defined as -1 for every step until the agent finds the goal, encouraging the agent to reach the goal as fast as possible.
 It is very important in RL that the reward is given in such a way that it describes what is the goal of the task, it should not describe how to do something. However, in some cases the reward definition might be not trivial. Think for example to a reward for a driving task. In the case of driving it is not trivial to define what does it mean to drive in a good way. \newline
RL tasks can be divided in \textit{episodic} and \textit{continuing}. In episodic tasks we have that $H$, the horizon length, is finite, while in continuing tasks $H = +\infty$.
The \textit{return}, $G_t$ quantifies the agent performance. For \textit{episodic} tasks the return from time step $t$ can be defined as the sum of rewards received until the end of the episode:
\begin{equation}
G_t = \sum_{k=t}^H r_k	\, .
\end{equation}
It is easy to see that for \textit{continuing tasks} this formulation of return diverges since we have an infinite sum of rewards.
To deal with \textit{continuing tasks} we need to introduce the notion of \textit{discounting}. The \textit{discount factor} $\gamma \in [0,1]$ quantifies the present values of future payoffs.
We introduce the \textit{discounted return} as the cumulative, discounted, sum of rewards until the end of the episode:
\begin{equation}
G_t = \sum_{k=0}^H \gamma^k r_{t+k+1} \, .
\end{equation}
The discount factor, besides having the interpretation mentioned before, can be seen as the probability that the process continues for another step. From the agent point of view, if the discount factor is near to zero greedy actions are profitable since future rewards do not have high values. If the discount factor is near to 1 the agent is \textit{far-sighted}, it is possible for him to sacrifice an action related to a good immediate reward now for a higher reward in the future steps.
We indicate with $G(\tau)$ the return associated with the trajectory $\tau$.
\subsection{Policy and Value Functions}
Policy evaluation is the process of quantifying how good a policy is, it is a key step in almost all RL algorithms. The performance of a policy is defined as the expected value of the return under state and action distribution:
\begin{equation}
J^\pi = \underset{\begin{subarray}{c}
	s_0 \sim \mu \\
	a_t \sim \pi(\cdot | s_t) \\
	s_{t+1} \sim P(\cdot | s_t, a_t)
\end{subarray}
}{\mathbb{E}}\left[ \sum_{t=0}^H \gamma^t R(s_t,a_t, s_{t+1})\right] \, .
\end{equation}
The simple idea formalized above is that policy $\pi_1$ it is better then policy $\pi_2$ if, on average, it collects more rewards.
Solving an MDP means to find the policy $\pi^*$ such that:
\begin{equation}
\pi^* \in \arg \max_{\pi} J^{\pi} \, .
\label{argmaxJ}
\end{equation}
When we have a parametrized policy we can cast the problem to the point of view of policy parameters. We have that $J$ depends on the policy parameters:

\begin{align}
J(\boldsymbol{\theta}) &= \underset{
\begin{subarray}{c}
s_0 \sim \mu \\
	a_t \sim \pi_\theta(\cdot | s_t) \\
	s_{t+1} \sim P(\cdot | s_t, a_t)
\end{subarray}
}{\mathbb{E}}\left[ \sum_{t=0}^H \gamma^{t} R(s_t, a_t, s_{t+1}) \right] . \\
\boldsymbol{\theta}^* & \in \arg \max_{\boldsymbol{\theta}} J(\boldsymbol{\theta}) \, .
\end{align}
The \textit{Value function} (or state-value function) provides a utility measure to a state following a policy. The value of a state $s$ under policy $\pi$, denoted as $v^\pi(s)$, is the expected  discounted return starting from $s$ following the behaviour prescribed by $\pi$:
\begin{equation}
v^\pi(s) = \underset{\pi}{\mathbb{E}} \left[ G_t | s_t = s \right] \, .
\end{equation}
In this way the value function embeds long-term information.
Similarly, we define the value of a state-action pair $(s,a)$ under policy $\pi$ as the expected discounted return starting from $s$, executing action $a$ and following the behaviour $\pi$. This utility measure is denoted as \textit{Q-function} or \textit{action-value function}:
 \begin{equation}
q^\pi(s,a) = \underset{\pi}{\mathbb{E}} \left[ G_t | s_t = s, a_t = a \right] \, .
\end{equation}

 The \textit{value function} is not suitable for control as it does not provide information on which action to take. The \textit{Q-function} provides a utility to all actions in a particular state.
 Value function and Q-function are clearly strictly related, the former is obtained by averaging the latter over the action distribution defined by the policy:

 \begin{equation}
v^\pi(s) = \underset{a \sim \pi(\cdot | s)}{\mathbb{E}} \left[ q^\pi(s,a) \right] \, .
\end{equation}

Using the state value function it is possible to re-write the performance of a policy:
\begin{equation}
J^\pi = \underset{s \sim \mu}{\mathbb{E}} \left[ v^\pi(s) \right] \, .
\end{equation}

The value function has also a recursive formulation, denoted as \textit{Bellman expectation operator for $v^\pi$} \citep{Bellman:1957}:
 \begin{equation}
v^\pi(s) = \underset{\begin{subarray}{c}
   a \sim \pi(\cdot | s) \\
   s'\sim P(\cdot | s,a)
   \end{subarray}}
   {\mathbb{E}} \left[R(s,a,s') + \gamma v^\pi (s') \right] \, .
\end{equation}
We can express in a recursive manner also the Q-function, the associated operator is denoted as \textit{Bellman expectation operator for $q^\pi$}:
 \begin{equation}
q^\pi(s,a) = \underset{\begin{subarray}{c}
   a' \sim \pi(\cdot | s') \\
   s' \sim P(\cdot | s,a)
   \end{subarray}}
   {\mathbb{E}} \left[R(s,a,s') + \gamma q^\pi(s',a')\right] \, .
\end{equation}

The Bellman expectation operators have several properties (see \ref{sec:opt-cond}). The operators are mainly used in iterative policy evaluation (see \ref{sec:pi}). We give the formal definition of the Bellman Expectation operators.
%%%%%%%%%%%%%%%%%%% BELLMAN EXPECTATION OPERATOR %%%%%%%%%%%%%%%%%%%%%%
\begin{definition}[Bellman Expectation Operator for $v^\pi$]
	The Bellman expectation operator for $v^\pi$ is defined as $T^{\pi} : \mathbb{R}^{| \mathcal{S} |} \rightarrow \mathbb{R}^{| \mathcal{S} | }$, maps state-value functions to state-value functions:
	\begin{equation}
		(T^{\pi} v)(s) = \underset{\begin{subarray}{c}a \sim \pi(s) \\ s' \sim P(\cdot | s,a) \end{subarray}}{\mathbb{E}} \left[ R(s,a,s') + \gamma v(s') \right] \, .
	\end{equation}
\end{definition} \label{def:bellman_exp_op_for_v}
\begin{definition}[Bellman Expectation Operator for $q^{\pi}$]
	The Bellman expectation operator for $q^{\pi}$ is defined as $T^{\pi} : \mathbb{R}^{| \mathcal{S} | \times | \mathcal{A} |} \rightarrow \mathbb{R}^{| \mathcal{S} | \times | \mathcal{A} |}$, maps Q-functions to Q-functions:
	\begin{equation}
		(T^{\pi} q)(s,a) =\underset{\begin{subarray}{c} a' \sim \pi(s') \\ s' \sim P(\cdot | s,a) \end{subarray}}{
			\mathbb{E}} \left[ R(s,a,s') + \gamma q(s',a') \right] \, .
	\end{equation}
\end{definition} \label{def:bellman_exp_op_for_q}

%%%%%%%%%%%%%%%% OPTIMALITY CONDITIONS %%%%%%%%%%%%%%%%%%5
\subsection{Optimality Conditions}\label{sec:opt-cond}
Besides \cref{argmaxJ} we can define another partial ordering over policies, the ordering induced by \textit{value functions}.
\begin{definition}
\label{optimal_policy}
	Policy $\pi$ is better or equal ($\succeq$) policy $\pi'$ if its expected return is greater or equal to that of $\pi'$ \textit{for all states}:
	\begin{equation}
		\pi \succeq \pi' \iff v^\pi(s) \geq v^{\pi'}(s), \; \forall s \in \mathcal{S} \, .
	\end{equation}
\end{definition}

Following the previous definition an \textit{optimal policy} is a policy that is better or equal to all other policies in all states. This optimality condition is stricter than the one presented in (\ref{argmaxJ}).
From the definition of optimal policy the \textit{optimal state-value function} and the \textit{optimal Q-function} follows:
\begin{align}
	v^{*}(s) &= \max_\pi v^\pi(s) \\
	q^{*}(s,a) &= \max_\pi q^\pi(s,a) \, .
\end{align}
There is always at least an optimal (deterministic) policy maximizing the state-value function in every state \citep{Puterman:1994:MDP:528623} and all optimal policies share the same \textit{optimal state value-function}.
The knowledge of the \textit{optimal} Q-function makes it possible to find the \textit{optimal} deterministic policy selecting in every state the action yielding the highest $q$-value:
\begin{equation}
	\pi^*(s) \in \arg \max_a q^{*}(s,a) \, .
\end{equation}

This holds if we have complete freedom in the selection of the policy (i.e. when $\Pi$ is the set of all Markovian policies). However for practical algorithms we need to restrict our policy space (e.g. parametric policy space), in these cases it is convenient to use the definition of optimal policy in (\ref{argmaxJ}).
\newline
The optimal state-value function and optimal Q-function satisfy the \textit{Bellman optimality equations}:
\begin{align}
	v^*(s) &= \max_a q^*(s,a) \label{opt_v} \\
		   &= \max_a \underset{s' \sim P(\cdot | s,a)}{
			\mathbb{E}} \left[ R(s,a,s') + \gamma v^*(s') \right] \, . \\
	q^*(s,a) &= \underset{s' \sim P(\cdot | s,a)}{
			\mathbb{E}} \left[ R(s,a,s') + \gamma \max_{a'} q^*(s',a') \right] \label{opt_q} \, .
\end{align}
At an intuitive level the \textit{Bellman optimality equation} for $v^*$ expresses the fact that the optimal state-value function must equal the expected return for the best action in that state. The \textit{Bellman optimality equation} for $q^*$, similarly, expresses the fact that the optimal Q-function must equal the immediate reward plus the discounted return of the best action in the next state according to the environment dynamic. \newline
The right handside of \cref{opt_v} it's defined as \textit{Bellman Optimality Operator}.

%%%%%%%%%%%%%%%%%%% BELLMAN OPTIMALITY OPERATOR %%%%%%%%%%%%%%%%%%%%%%
\begin{definition}[Bellman Optimality Operator for $v^*$]
	The Bellman optimality operator for $v^*$ is defined as $T^* : \mathbb{R}^{| \mathcal{S} |} \rightarrow \mathbb{R}^{| \mathcal{S} | }$, maps state-value functions to state-value functions:
	\begin{equation}
		(T^* v)(s) = \max_a \underset{s' \sim P(\cdot | s,a)}{\mathbb{E}} \left[ R(s,a,s') + \gamma v(s') \right] \, .
	\end{equation}
\end{definition} \label{def:bellman_opt_op_for_v}
\begin{definition}[Bellman Optimality Operator for $q^*$]
	The Bellman optimality operator for $q^*$ is defined as $T^* : \mathbb{R}^{| \mathcal{S} | \times | \mathcal{A} |} \rightarrow \mathbb{R}^{| \mathcal{S} | \times | \mathcal{A} |}$, maps Q-functions to Q-functions:
	\begin{equation}
		(T^* q)(s,a) =\underset{s' \sim P(\cdot | s,a)}{
			\mathbb{E}} \left[ R(s,a,s') + \gamma \max_{a'} q(s',a') \right] \, .
	\end{equation}
\end{definition} \label{def:bellman_exp_op_for_v}
The Bellman operators (\ref{def:bellman_exp_op_for_v}, \ref{def:bellman_exp_op_for_q}, \ref{def:bellman_opt_op_for_v}, \ref{def:bellman_exp_op_for_q}) are characterized by the following properties, where we will use $T$ to denote both the expectation operator and the optimality operator:
\begin{itemize}
	\item Monotonicity: \begin{equation}
			f_1 \leq f_2 \implies T f_1 \leq T f_2 \, ;
		\end{equation}
	\item Max-Norm contraction:
			\begin{equation}
			\norm{T f_1 - T f_1}_\infty \leq \gamma \norm{f_1 - f_2}_\infty \, , \forall f_1, f_2 \, ;
		\end{equation}
	\item $v^*$ is the \textit{unique fixed point} of $T^*$. $v^\pi$ is the \textit{unique fixed point} of $T^{\pi}$.
	\item Convergence:
	 \begin{equation}
			\lim_{k \rightarrow \infty} (T^*)^k f = v^* \, , \forall f \in \mathbb{R}^{| \mathcal{S} |} \,
		\end{equation}
		\begin{equation}
			\lim_{k \rightarrow \infty} (T^{\pi})^k f = v^\pi \, , \forall f \in \mathbb{R}^{| \mathcal{S} |} \, .
		\end{equation}
\end{itemize}
Solving \cref{opt_v} and \cref{opt_q} leads us directly to the MDP solution. Unfortunately these equations are non-linear because of the presence of the $\max$ operator and there is no closed form solution for the general case.
There exists many iterative solution methods trying to solve an approximation of the Bellman optimality equations that are discussed in the following sections.

\section{Linear Programming} \label{sec:lin_prog}
The problem of computing an optimal policy for an infinite horizon finite-states MDP can be formulated as linear program (LP) \citep{depenoux_probabilistic_1963}. In this section we will go further in the LP formulation since our algorithm extends on this.
The basic idea follows from \cref{optimal_policy}: we want to find the value function maximizing the value of each state weighted by the initial state distribution subject to a feasibility constraint:
\begin{align*}
	\underset{v}{\text{minimize}} & \; \sum_{s \in \mathcal{S}} \mu(s) v(s) \\
	\text{subject to} &  \; v(s) \geq r(s,a) + \gamma \sum_{s' \in \mathcal{S}} P(s' | s,a) v(s') \, , \forall s \in \mathcal{S}, \, \forall a \in \mathcal{A} \, .
\end{align*}
Where we have $|\mathcal{S}|$ variables and $|\mathcal{S}| |\mathcal{A}|$ constraints. Notice that the maximization role is taken by the constraints, while we need to minimize since otherwise an optimal solution would have infinite values for all variables $v$.
\begin{theorem}[Linear Programming Solution]
$v^*$ is the solution of the above linear program.
\end{theorem} \label{lpth}

It is possible to prove the above theorem using the properties of the Bellman optimality operators, the interested reader can refer to \cref{LPproof}.
It is also interesting to analyze the Dual Linear Program:

\begin{align*}
	\underset{p}{\text{maximize}} & \; \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} d(s,a)r(s,a) \\
	\text{subject to} &  \; \sum_{a \in \mathcal{A}} d(s,a) = \mu(s) + \gamma \sum_{s' \in \mathcal{S}} \sum_{a' \in \mathcal{A}} d(s',a') P(s | s', a'), \; \forall s \in \mathcal{S} \\
	& d(s,a) \geq 0, \;  \forall s \in \mathcal{S}, \; \forall a \in \mathcal{S} \, .
\end{align*}

In the dual program $d(s,a)$ is the discounted state action distribution \citep{pol_grad_func_approx}:
\begin{equation}
	d(s,a) = \sum_t^\infty \gamma^t \mathbb{P}(s_t = s, a_t = a) \, .
\end{equation}

The objective of the dual is the usual objective: maximize the expected discounted sum of rewards.
The constraints are needed for ensuring that $p$ is a valid distribution.
From the dual program it is possible to extract the optimal policy:
\begin{equation}
	\pi^*(s) \in \arg \max_a d(s,a)\, .
\end{equation}
For each state $s$ we have that $d(s,a^*) = 1$ if action $a^*$ is optimal in state $s$. When there are multiple optimal actions we have a split of the probability between these actions.
The LP formulation is an exact way of solving the MDPs, unfortunately this formulation becomes impractical when the states space grows too much.

\section{Dynamic Programming} \label{sec:dyn_prog}
Dynamic programming \citep{Bellman:1957} (DP) is by far the most common way for solving MDPs when the dynamics and the reward function are known. DP takes into account the sequential or temporal structure of the problem. It is a method for solving complex problems by breaking them into subproblems.
A problem, in order to be solved by DP techniques, must have two properties: \begin{enumerate*} \item Optimal Substructure, \item Overlapping Subproblems \end{enumerate*}. The optimal substructure property means that the principle of optimality applies and the optimal solution can be decomposed into subproblems. The overlapping subproblems property means that subproblems recur many times and the solutions can be stored and reused.
In MDPs both properties are satisfied. The \textbf{Bellman equation} gives the recursive decomposition while the \textbf{state-value function} caches and reuses previous solutions.

\begin{figure}[tb]
\centering
\includegraphics{pictures/value_iteration}
\caption{\small Policy Iteration, from \citep{sutton_introduction}.}
\end{figure}

\subsection{Policy Iteration} \label{sec:pi}
Policy Iteration \citep{howard:dp} is a two stage algorithm: Policy Evaluation (PE) and Policy Improvement (PI).
The first stage (Policy Evaluation) aims at evaluating the current policy through the application of the Bellman Expectation Operator. The PE phase is needed since we want to perform an update of the current policy. The update direction can be computed using the action-value function in the following way. Suppose we know $v^\pi(s')$ for a particular $s'$. To improve the current policy it is enough to compute $q^\pi(s',a)$ for each $a \in \mathcal{A}$ and then to compare it with $v^\pi(s')$. If exists an action $a'$ such that $q(s',a') > v^\pi(s')$, then we can \textit{improve} the current policy using the following update rule:
\begin{equation}
\pi'(s) =
	\left\{
	\begin{array}{ll}
		\pi(s)  & \mbox{if } s \neq s' \\
		\underset{a}{\arg \max} \, q^\pi(s,a) & \mbox{if } s = s'
	\end{array}
\right .
\label{eq:greedy1}
\end{equation}
Basically this update rule states that the new policy is the same as the old policy except for states in which there exists an action $a$ such that the action-value function $q^\pi$ is greater than the value function $v^\pi$.
The justification for the previous statement comes from the \textit{Policy Improvement Theorem}.
\begin{theorem}[Policy Improvement Theorem]
	Let $\pi$ and $\pi'$ be a pair of policies such that:
	\begin{equation}
		q^\pi(s, \pi'(s)) \geq v^\pi(s) \, , \forall s \in \mathcal{S} \, ,
	\end{equation}
	Then the policy $\pi'$ must be as good or better than $\pi$.
	\label{th:policy_improvement}
\end{theorem}

Given the previous theorem we can expand the update rule (\ref{eq:greedy1}) by computing the \textit{greedy policy} in all states:
\begin{equation}
\pi'(s) \in \underset{a}{\arg \max} \, q^\pi(s,a) 	\, .
	\label{eq:greedy2}
\end{equation}

This update rule meets the condition of \cref{th:policy_improvement}, so the updated policy is as good as, or better than the old one.
When the new policy is as good as the old, $v^{\pi'} = v^\pi$, we know, thanks to the optimality operator property, that $v^\pi = v^*$, we have obtained the optimal policy.
Now we can formalize the Policy Iteration Algorithm, see \cref{alg:policy-iteration}.
All the previous ideas can be easily extended to the case of stochastic policies.

\begin{algorithm}[tb]
  \caption{Policy Iteration
    \label{alg:policy-iteration}}
  \begin{algorithmic}[1]
  \Require{$\mathcal{S}, \mathcal{A}, P, R$}
  \Ensure{$\pi \approx \pi^*$}
  \State Initialize $v^\pi$ and $\pi$ randomly
  \Repeat
  \State $v^\pi$ $\leftarrow$ Evaluate policy $\pi$ by applying $T^{\pi}$
  \State $\pi$ $\leftarrow$  Improve policy $\pi$
  \Until{no improvement found} \\
  \Return{Policy $\pi$}
  \end{algorithmic}
\end{algorithm}

\subsection{Value Iteration}
The Value Iteration (VI) algorithm performs evaluation and improvement in the same step. While Policy Iteration performs a search in the space of policies, Value Iteration searches in the space of value functions, calculating the policy only in the last step.
Value Iteration is obtained simply by turning the Bellman Optimality Operator into an update rule:
\begin{equation}
		v_{k+1}(s) = \max_a \underset{s' \sim P(\cdot | s,a)}{\mathbb{E}} \left[ R(s,a,s') + \gamma v_{k}(s') \right] \, , \forall s \in \mathcal{S} \, .
\end{equation}
Value Iteration algorithm is reported in \cref{alg:value-iteration}. \newline
Both PI and VI converge to an optimal policy for discounted finite MDPs.

\begin{algorithm}[tb]
  \caption{Value Iteration
    \label{alg:value-iteration}}
  \begin{algorithmic}[1]
  \Require{$\mathcal{S}, \mathcal{A}, P, R$}
  \Ensure{$\pi \approx \pi^*$}
  \State Initialize $v$ randomly
  \Repeat
  \State $v$ $\leftarrow$ Apply Bellman Optimality operator $T^*v$
  \Until{no changes in the value function} \\
  \Return{Policy $\pi$:
  \begin{equation}
  	\pi(s) = \underset{a}{\arg \max} \underset{s' \sim P(\cdot | s,a)}{\mathbb{E}} \left[ R(s,a,s') + \gamma v(s') \right] \, , \forall s \in \mathcal{S} \, .
  \end{equation}
  }
  \end{algorithmic}
\end{algorithm}
\section{Policy Search}\label{sec:policy-search}
Policy search (PS) methods explore directly the policy space. In the PS framework the RL problem is formalized as:
\begin{equation}
	\pi^* \in \arg \max_{\pi \in \Pi} J^\pi \, .
\end{equation}
Among PS methods it is worth mentioning policy gradient methods \citep{pol_grad_func_approx, PETERS2008682}. Policy gradient methods consider a set of parameterized policies (see \cref{sec:policy}) $\pi_{\boldsymbol{\theta}}(a|s)$ where $\boldsymbol{\theta} \in \mathbb{R}^d$ is the vector of policy parameters. In this cases the space of policies is represented by $\Pi = \left\{ \pi_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \Theta \subseteq \mathbb{R}^d \right\}$.  A standard requirement is that the policy must be differentiable in its parameters.
A common technique to maximize the performance of a policy is to perform \textit{gradient ascent} over the policy parameters:
\begin{equation}
	\boldsymbol{\theta}^{k+1} = \boldsymbol{\theta}^k + \alpha \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}^k) \, ,
	\label{eq:theta-update}
\end{equation}
where $\alpha \geq 0$ is the \textit{learning rate}, also called \textit{step size}. The quantity $\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}^k)$ is the gradient of the performance of the policy and it can be computed through the policy gradient theorem \citep{pol_grad_func_approx}.
\begin{theorem}[Policy Gradient Theorem] For any MDP:
\begin{equation}
	\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}) = \int_\mathcal{S} d_{\mu, \pi_{\boldsymbol{\theta}}}(s)\int_\mathcal{A} \nabla_{\boldsymbol{\theta}}\pi_{\boldsymbol{\theta}}(a | s) q^{\pi_{\boldsymbol{\theta}}}(s,a) da ds \,.
\label{eq:pol-grad}
\end{equation}
\end{theorem}
In real cases since the state distribution is not known it is not possible to compute analytically the gradient of the performance measure. We need to resort to a sample approximation of it.
Gradient ascent method guarantees convergence at least to a local optimum even if in practice local optima can be avoided using a stochastic approximation to the gradient or policies belonging to high dimensional spaces.

\subsubsection{Gradient Estimation}
REINFORCE method \citep{reinforce} builds on the fact that the outer integral in \cref{eq:pol-grad} is the analytical expression of the expected value under the state distribution induced by the policy $\pi$. We can rewrite the gradient in the following way (where we use $\pi$ for $\pi_{\boldsymbol{\theta}}$):
\begin{align}
	\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}) &= \underset{s \sim d_{\mu,\pi}(\cdot)}{\mathbb{E}} \left[ \int_\mathcal{A} \nabla \pi(a | s) q^\pi (s,a) ds \right] \\
	& = \underset{\begin{subarray}{c}
					s \sim d_{\mu,\pi}(\cdot) \\
					a \sim \pi(\cdot |s )
				  \end{subarray}}
		{\mathbb{E}} \left[ \nabla \log \pi(a | s) q^\pi(s,a) \right], \tag*{log trick: $\nabla f = f \nabla \log f$}
\end{align}
The estimation of the action value function can be performed in a straightforward way from the sum of discounted rewards: $\widehat{q}^\pi \approx \sum_{k=0}^H \gamma^k r(s_k,a_k) $
Summarizing, we can obtain an approximation of the gradient through the following estimator:
\begin{equation}
	\widehat{\nabla_\theta J(\boldsymbol{\theta})}_{RF} = \langle \left(\sum_{k=0}^H \nabla_\theta \log \pi(a_k, s_k) \right) \left( \sum_{k=0}^H \gamma^k r(s_k,a_k) \right) \rangle_N \, ,
\end{equation}
where $N$ is the batch size, the number of collected trajectories, $\langle \cdot \rangle_N$ denotes the sample mean over $N$ trajectories and $H$ is the horizon length. \newline
REINFORCE gradient estimation suffers of high variance, that grows at least cubically with the length of the horizon and quadratically with the magnitude of the reward. \newline
G(PO)MDP \citep{gpomdp} employs a better approximation with lower variance exploiting the observation that future actions do not depend on past rewards (if the policy does not change within an episode) leading to the following estimation:
\begin{equation}
	\widehat{\nabla_\theta J(\boldsymbol{\theta})}_{G(PO)MDP} = \langle \sum_{l=0}^H \left( \sum_{k=l}^H \nabla_\theta \log \pi(a_k | s_k) \right) \left( \gamma^l r(s_l,a_l) \right) \rangle_N \, .
\end{equation}
\subsubsection{Natural Policy Gradient}
As presented in \citep{NIPS2012_4576}, the general form of Policy Gradient updates is:
\begin{equation}
	\boldsymbol{\theta}^{k+1} = \boldsymbol{\theta}^k + \alpha \mathbf{G}(\boldsymbol{\theta}^k)^{-1}\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}^k) \, ,
\end{equation}
 where $\mathbf{G}(\boldsymbol{\theta})$ is a positive definite matrix. The matrix $\mathbf{G}(\boldsymbol{\theta})$ defines the metric used to measure vectors in the parameter space. The parameter update in Equation (\ref{eq:theta-update}) can be derived from this formulation by using $\mathbf{G}(\boldsymbol{\theta}^k)=\boldsymbol{I}$, that is the Euclidean norm. This assumption does not take into account that the space parameterized is actually a Riemannian manifold. As \citep{amari_1998} suggested it is better to define a metric based not on the choice of coordinates but rather on the manifold (i.e. the surface) that these coordinates parameterize.
 Natural Gradient \citep{kakade_natural_2002} exploits this structure and uses as metric the Fisher Information Matrix of the trajectory distribution and due to the Markovian structure of the dynamics it is given by:
\begin{equation}
	\mathbf{G}(\boldsymbol{\theta}) = \underset{\begin{subarray}{c}
	s \sim d_{\mu,\gamma}^\pi(\cdot) \\
	a \sim \pi_{\boldsymbol{\theta}}(\cdot | s)
	\end{subarray}
	}{\mathbb{E}} \left[ \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}} (a |s) \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}} (a |s)^T \right] \, .
\end{equation}
Convergence to a local maximum is guaranteed; by choosing a more direct path to the optimal solution in parameter space, the natural gradient has faster convergence and avoids premature convergence of steepest ascent gradient; the natural policy gradient can be shown to be covariant, i.e., independent of the parameterization of the policy; it requires fewer data points for a good gradient estimate.

\clearpage

%%%%%%%%%%%%%%%%%% MDP EXTENSIONS %%%%%%%%%%%%%%
\section{MDP extensions}\label{sec:mdp-ext}
The standard MDP framework has been extended in the literature in order to cover many real cases scenarios. Here we report some notable extensions that can be related and compared to the framework considered in this work.

\subsection{MDP with imprecise probability}
MDP with imprecise probability (MDPIP) \citep{harmanec_generalizing_2002} covers the case in which it is not easy (or even possible) to define a precise probability measure for a given transition $P(\cdot | s, a)$. In this cases the probability parameters are imprecise and therefore the transition model cannot be defined by means of a probability distribution but it must be defined through a set of probability distribution. These sets of distribution are known as transition credal sets $\mathcal{K}(\cdot | s,a)$.\newline
In this framework it is common to use game theoretic approaches maximizing the lowest expected reward with respect to the probability parameters, known as $\Gamma$-maximin criterion. In this cases the optimal value function is:
\begin{equation}
	v^*(s) = \max_{a \in \mathcal{A}} \left\{ \min_{P(\cdot | s,a) \in \mathcal{K}(\cdot | s,a)} r(s,a) + \gamma \sum_{s' \in \mathcal{S}} P(s' | s,a)v^*(s') \right\} \, .
\end{equation}

\subsection{Bounded-parameter Markov decision processes}
Bounded-parameter MDPs \citep{givan_bounded-parameter_2000} can be used to represent variation or uncertainty concerning the parameters of sequential decision problems in cases where no prior probabilities on the parameter values are available. BMDPs form an efficiently solvable special case of the class of MDPIPs.
In this context interval value functions are introduced as a natural extension of traditional value functions. An interval value function assigns a closed real interval to each state, representing the fact that the value of that state falls within that interval. An interval value function can be used to bound the performance of a policy over the set of exact MDPs associated with a given bounded-parameter MDP.
\subsection{Non stationary MDPs}
Non stationary MDPs \citep{bowerman_1974} are used to model scenarios in which the transition dynamic and the reward function are non stationary, i.e. they change over time. We can model this scenario with a different transition and reward functions for each timestep:
\begin{align}
	&P(s' | s, a, t) = P^t(s' | s,a) \\
	&R(s,a,s',t) = R^t(s,a,s') \, .
\end{align}
A usual assumption made in this family of MDPs is that there is a correlation between contniguous time frames. In this way the agent can use recent experience and forget past experience. In \citep{nonstatmdp} an optimality condition for non stationary MDPs has been defined. Moreover, \citep{GARCIA2000304}, \citep{Cheevaprawatdomrong:2007} and \citep{nonstatmdp2} focused on how to find an optimal policy for non stationary MDPs in an effective way.


\documentclass[12pt, a4paper]{article}
\usepackage{geometry}
\geometry{verbose,tmargin=2cm,bmargin=2cm,lmargin=3cm,rmargin=3cm}
\usepackage{verbatim}
\usepackage[normalem]{ulem}
\usepackage{colortbl}
\usepackage{graphicx}
\usepackage{algorithm,algpseudocode}
\usepackage[colorlinks,citecolor=red,urlcolor=blue,bookmarks=false,hypertexnames=true]{hyperref}
\usepackage{url}

\makeatother
\date{}
\begin{document}

\title{\Large 16-833: Robot Localization and Mapping, Spring 2021\\
\textbf{Homework 2 - SLAM using Extended Kalman Filter (EKF-SLAM)}}
\maketitle
\begin{flushright}
\textbf{\uline{Due: Wednesday Mar 17, 11:59pm, 2021}}
\par\end{flushright}

Your homework should be submitted as a\textbf{ typeset PDF file} along
with a\textbf{ }folder\textbf{ }including\textbf{ code} \textbf{only
(no data)}. The PDF must be submitted on \textbf{Gradescope}, and
code submitted on \textbf{Canvas}. In your implementation, please
only fill in the functions marked by $\mathtt{TODO}$, and play with parameters, and keep other parts unchanged. If you have questions, post them on Piazza or come to office
hours. Please do not post solutions or codes on Piazza.

This homework must be done \textbf{individually}, and plagiarism will
be taken seriously. You are free to discuss and troubleshoot with
others, but the code and writeup must be your own. Note that you should
list the name and Andrew ID of each student you have discussed with
on the first page of your PDF file.

\section{Theory (40 points)}

In this part we are going to guide you through some steps of EKF-SLAM
in math. This will be helpful in your implementation in the next problem.

A robot is moving on the 2D ground plane. In each time step $t$,
the robot is controlled to move forward (the $x$-direction of the
robot's coordinates) $d_{t}$ meters, and then rotate $\alpha_{t}$
radian. The pose of the robot in the global coordinates at time $t$
is written as a vector $\mathbf{p}_{t}=\left[\begin{array}{ccc}
x_{t} & y_{t} & \theta_{t}\end{array}\right]^{\top}$, where $x_{t}$ and $y_{t}$ are the 2D coordinates of the robot's
position, and $\theta_{t}$ is the robot's orientation.

\begin{enumerate}
\item Based on the assumption that there is no noise or error in the
control system, predict the next pose $\mathbf{p}_{t+1}$ as a nonlinear
function of the current pose $\mathbf{p}_{t}$ and the control inputs
$d_{t}$ and $\alpha_{t}$. (5 points)

\item However, in reality there are some errors when the robot moves
due to the mechanism and the terrain. Assume the errors follow Gaussian
distributions: $e_{x}\sim\mathcal{N}\left(0,\sigma_{x}^{2}\right)$
in $x$-direction, $e_{y}\sim\mathcal{N}\left(0,\sigma_{y}^{2}\right)$
in $y$-direction, and $e_{\alpha}\sim\mathcal{N}\left(0,\sigma_{\alpha}^{2}\right)$
in rotation respectively (all in robot's coordinates). For details, please see Fig.~\ref{fig:1b}.
Now if the uncertainty of the robot's pose at time $t$ can be represented as
a 3-dimensional Gaussian distribution $\mathcal{N}\left(0,\Sigma_{t}\right)$,
what is the predicted uncertainty of the robot at time $t+1$? Please
express it as a Gaussian distribution with zero mean. (5 points)
\begin{figure}
    \centering
    \includegraphics[width=0.99\textwidth]{1b.png}
    \caption{0) Robot state; 1) robot moves $d_t$ along its x-axis;
    2) due to noise it shifted $e_x$ and $e_y$ along its x and y axises respectively;
    3) robot rotates $\alpha_t$ after shifting;
    4) due to noise the rotation is disturbed by $e_\alpha$.}
    \label{fig:1b}
\end{figure}

\item Consider a landmark $l$ being observed by the robot at time $t$
with a laser sensor which gives a measurement of the bearing angle
$\beta$ (in the interval $(-\pi,\pi]$) and the range $r$, with
noise $n_{\beta}\sim\mathcal{N}\left(0,\sigma_{\beta}^{2}\right)$
and $n_{r}\sim\mathcal{N}\left(0,\sigma_{r}^{2}\right)$ respectively.
Write down the estimated position $\left(l_{x},l_{y}\right)$ of landmark
$l$ in global coordinates as a function of $\mathbf{p}_{t}$, $\beta$,
$r$, and the noise terms. (5 points)

\item According to 1.3, if we now know that $l$ is at $\left(l_{x},l_{y}\right)$
in the global coordinates, please predict the measurement of bearing
and range based on $l_{x}$, $l_{y}$, $\mathbf{p}_{t}$, and the
noise terms (use functions $\mathit{np.arctan2}(.)$ and $\mathit{warp2pi}(.)$
that warps an arbitrary angular value into the range $(-\pi,\pi]$
if needed). (5 points)

\item An important step in EKF-SLAM is to find the measurement Jacobian
$H_{p}$ with respect to the robot pose. Please apply the results
in (d) to derive the analytical form of $H_{p}$ (Note: $H_{p}$ should
be a $2\times3$ matrix represented by $\mathbf{p}_{t}$ and $l$).
(10 points)

\item For each measurement of bearing and range, we also need a measurement
Jacobian $H_{l}$ with respect to its corresponding landmark $l$.
Please again derive the analytical form of $H_{l}$ (Note: $H_{l}$
should be a $2\times2$ matrix represented by $\mathbf{p}_{t}$ and
$l$). Why do we not need to calculate the measurement Jacobian with
respect to other landmarks except for itself (Hint: based on what
assumption)? (10 points)
\end{enumerate}

Note in this section, in addition to conventional linearization with additive noise (in the form of $x_{t+1} = g(x_t, u_t) + \epsilon_t$), you may encounter non-additive noise (in the form of $x_{t+1} = g(x_t, u_t, \epsilon_t))$. \href{https://en.wikipedia.org/wiki/Extended_Kalman_filter#Non-additive_noise_formulation_and_equations}{Derivation of the linearization} for this formulation can be helpful.
Also, all the noise reading ($\epsilon, n$) can be regarded as infinitismal. Second and higher order terms comprised by their products can be ignored.

\section{Implementation and Evaluation (45 points)}

In this part you will implement your own 2D EKF-SLAM solver in Python.
The robot in problem 1 starts with observing the surrounding environment
and measuring some landmarks, then executes a control input to move.
The measurement and control steps are repeated several times. For
simplicity, we assume the robot observes the same set of landmarks
in the same order at each pose. We want to use EKF-SLAM to recover
the trajectory of the robot and the positions of the landmarks from
the control input and measurements.

\begin{enumerate}
\item Find the data file $\mathtt{data/data.txt}$ that contains the
control inputs and measurements. Open the file to take a look. The
data is in the format of measurements$=\left[\begin{array}{ccccc}
\beta_{1} & r_{1} & \beta_{2} & r_{2} & \cdots\end{array}\right]$, where $\beta_{i}$, $r_{i}$ correspond to landmark $l_{i}$, and control$=\left[\begin{array}{cc}
    d & \alpha\end{array}\right]$ (refer to problem 1 for the notations). The
lines are in sequential time order.

What is the fixed number
of landmarks being observed over the entire sequence? (5 points)

\item The $\mathtt{code/}$ folder contains the Python code file
$\mathtt{ekf\_slam.py}$. Follow the comments and fill in the functions $\mathtt{warp2pi},~\mathtt{init\_landmarks}, ~\mathtt{predict}, ~\mathtt{update}$ to enable the system. The visualization will show the trajectory and the landmarks with their covariances in the shape of 3-$\sigma$ ellises. It will be updated sequentially, and you may process the next step by clicking your mouse or hitting any key.

Attach a clear figure of your
visualization in the write up once all the steps are finished. (25 points)

\item In the output figure, the magenta and blue ellipses represent
the predicted and updated uncertainties of the robot's position (orientation
not shown here) at each time respectively. Also, the red and green
ellipses represent the initial and all the updated uncertainties of
the landmarks, respectively.

Describe how EKF-SLAM improves the estimation
of both the trajectory and the map by comparing the uncertainty ellipses.
(5 points)

\item Now let's evaluate our output map. Suppose we have the ground
truth positions of all the landmarks:

$\left[\begin{array}{cccc}
l_{1}^{\top} & l_{2}^{\top} & \cdots & l_{k}^{\top}\end{array}\right]=\left[\begin{array}{cccccccccccc}
3 & 6 & 3 & 12 & 7 & 8 & 7 & 14 & 11 & 6 & 11 & 12\end{array}\right]$.

Plot the ground truth positions of the landmarks in the output figure
(code already provided in function $\mathtt{evaluate}$)
and attach it below. Is each of them inside the smallest corresponding
ellipse? What does that mean?

Compute and report the \emph{Euclidean} and
\emph{Mahalanobis} distances of each landmark estimation with respect
to the ground truth. What do the numbers tell you? (10 points)
\end{enumerate}

\section{Discussion (15 points)}

\begin{enumerate}
\item Explain why the zero terms in the initial landmark covariance
matrix become non-zero in the final state covariance matrix (print
out the final $P$ matrix to check it). Additionally, when setting
the initial value for the full covariance matrix $P$ (line 201 in
$\mathtt{ekf\_slam.py}$) an assumption is made regarding certain cross-covariances
that is not necessarily correct. Can you point out what that is? (5
points)

\item Play with the parameters (line 163-167). Modify each parameter~$\sigma_{x}$,
$\sigma_{y}$, $\sigma_{\alpha}$$\sigma_{b}$, $\sigma_{r}$ at a
time and fix the others. In particular, you should set each parameter
to be 10 times larger/smaller to the original parameters to discuss
how each parameters influence the results. Attach figures for
better explanation. (10 points)

\item With the same set of landmarks being observed all the time, the
EKF-SLAM system runs in constant time in each iteration. However,
for real-world problems, the total number of landmarks can grow unbounded
if the robot keeps exploring new environments. This will slow down
the EKF-SLAM system a lot and become a crucial problem for real-time
applications. What changes can we make to the EKF-SLAM framework to
achieve constant computation time in each cycle or at least speed
up the process (list as many possible solutions as you can think of)?
(Bonus 5 points)
\end{enumerate}

\section{Code Submission}

Upload your $\mathtt{ekf\_slam.py}$.
Please do not upload the $\mathtt{data/}$ folder or any other data
or image.
\end{document}


\documentclass[12pt]{article}
\input{physics1}
\begin{document}

\section*{NYU Physics I---Problem Set 4}

Due Thursday 2018 October 04 at the beginning of lecture.

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
A normal, healthy NYU student climbs 10 stories of stairs at a
reasonable, steady pace. Compute the potential-energy gain of the
student. Compute also the kinetic energy of the student during this
climb. You will have to estimate the mass of the student, the pace of
stair climbing, and the height of a story of a building. Make
reasonable and explicit assumptions. Give a simple explanation for why
it is reasonable that one energy is far larger than the other.

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Let's extend the problem that Prof Hogg started in lecture:
Consider a car driving around a circular banked turn at
constant altitude with the bank set at $15\,\deg$ to the horizontal,
but this time with the car traveling at $55\,\mph$.

\textsl{(a)} Draw the free-body diagrams and compute the magnitude of
the normal force in the case that the car is going around the curve at
exactly the speed such that no lateral friction force is required at
all, and the car neither loses nor gains altitude (it goes in a
perfect, level, circle). That is, draw the free-body diagram for the
frictionless, perfect-driving case.

\textsl{(b)} What is the radius of curvature $R$ of the turn in this
case?

\textsl{(c)} Now imagine that the car came into this same turn---with
the same bank angle and same radius of turn $R$---but at
$70\,\mph$. That is, a speeder. And, further, assume that there is
plenty of static frictional force for the car to rely on. (Note, it is
\emph{static} friction that keeps a car on the road, not sliding
friction!). What is the magnitude of the frictional force required to
keep the car driving in the level, circular turn? Be very careful with
your coordinate system and your diagrams.

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Here is a non-trivial machine that delivers a kind of mechanical
advantage:
\\ \rule{0.33\textwidth}{0pt}
\resizebox{0.33\textwidth}{!}{\includegraphics{../mp/tackle_blocks.pdf}}

\textsl{(a)}~Draw free-body diagrams for all the masses and pulleys in
this mechanism. Assume that the strings and pulleys are light and
frictionless, so that the strings are perfect tension-transmitters.

\textsl{(b)} What is the kinematic constraint---the relationship
between the accelerations of block 1 and block 2? Be very careful with
this one. Treat all the strings as inextensible for simplicity.

\textsl{(c)} Find the tensions in all three strings and the
accelerations of the two blocks.

\textsl{(d)}~Now set the mass $m_2$ to the value it must have if the
system is to be perfectly ``balanced''; that is, for there to be no
net acceleration of either block.  In this situation, if block $m_1$
is lowered a small distance $h$, what is the net change in potential
energy, accounting for the displacements of both blocks?

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Gasoline and olive oil are both substances with great chemical energy
content per unit mass.

\textsl{(a)} In the case of gasoline, the chemical energy is mainly in
carbon bonds.  If you assume that gasoline is \emph{entirely} carbon
atoms, and each one releases $4\,\eV$ of energy when it is combusted, how
much energy per unit mass is there in gasoline?  Get an answer in MJ
per kg and compare to what you find on \textit{Wikipedia}.  How far off
are our assumptions?

\textsl{(b)} Now convert your answer to kcal per g and compare it to
what is written on the ``Nutrition Facts'' label on an olive oil
bottle.  How close are you?  It should be close, (Prof Hogg thinks), because
biofuel is made from things like olive oil!

\paragraph{Extra Problem (will not be graded for credit):}%
\textsl{(a)} Assume that a car moving at speed
$v=75\,\mi\,\h^{-1}$ encounters an air resistance force of
$\frac{1}{2}\,\rho\,A\,v^2$, where $\rho$ is the density of air and $A$ is the
cross-sectional area of the car, about $2.5\,\m^2$ How much work does it
take to move the car $30\,\mi$ at this speed?

\textsl{(b)} If a car with these properties was \emph{perfectly
  efficient}, how many miles per gallon would it get? You need to use a value for
the energy density in gasoline.

\textsl{(c)} What does this
make you think about the future of cars that are \emph{far more
  efficient} than cars we have today? For example, could we have cars
that are 100 times more efficient than today's cars? If so, what
would they look like?

\end{document}


\documentclass[12pt]{article}
\usepackage[usenames]{color} %used for font color
\usepackage{amssymb} %maths
\usepackage{amsmath} %maths
\usepackage[utf8]{inputenc} %useful to type directly diacritic characters
\usepackage{graphicx}
\graphicspath{ {./} }
\author{Tianshuang (Ethan) Qiu}
\begin{document}
\title{Homework 1}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\prob}{\mathbb{P}}

\maketitle
\newpage

\section{1}
\newpage
\section{2}
\newpage
f
\newpage

\section{3}
\newpage
\section{Ross 4.7}

\subsection{a}

Since the infimum is a member of the set of lowerbounds for any set, and the
supremum is a member of the set of upperbounds, we have $\inf S \leq  s(\forall s \in S)$,
and $\sup S \geq s(\forall s \in S)$. Therefore by the ordered field axiom, $\inf S \leq \sup S$
for any set.
\newline
Thus we have proven $\inf T \leq \sup T$ and $\inf S \leq \sup S$. Now we show that
$\inf T \leq \inf S$.
\newline
Suppose that the above statement is false, so $\inf S > \inf T $. Then consider k =
$(\inf S + \inf T) /2$. Since $\inf S > \inf T , k < \inf S$. From the definition of
infimum, $k < s(\forall s \in S)$. Furthermore, since $k > \inf T $, $\exists t \in T$ such that
$t > k$. However, $S \subseteq T$ , so every element of S is in T. It is impossible for an
element to exist in T but not in S. Therefore our assumption is incorrect.
\newline
We conclude that $\inf T \leq \inf S$.
We can repeat the same argument symmetrically for the supremum and show
that $\sup T \geq \sup S$. Then by the ordered field axioms we can arrive at the
conclusion $\inf T \leq \inf S \leq \sup S \leq \sup T$ .
Q.E.D.

\subsection{b}
Let $x = \sup S, y = \sup T, Z = S \cup T $. Furthermore, let $x \geq y$ (switch $S, T$ if
$x < y$). To show that $x = \sup Z$, we need to show that $\forall z \in Z, x \geq z$, and
that x is the minimum of all upperbounds of $Z$.
\newline
To show the first part, we assume that the statement is false. $\exists z \in Z s.t. z >
x$. Since $Z$ is the union of $S$ and $T$ , all the elements inside must be from $S$
or $T$ . $x = \sup S, y = \sup T ,$ and since $x \geq y$, $x$ is an upperbound for both
S and T . This is a contradiction to our assumption that $\exists z \in Zs.t.z > x.$
Therefore our assumption is incorrect and $x$ must be an upperbound for $Z$.
\vspace{\baselineskip}
\newline
For the second part, we also assume that it is false. $\exists a < x, s.t.a \geq z(\forall z \in Z)$.
Consider $b_0 = \frac{a+x}{2}$
Firstly we can see that $b_0 > a$ since $a < x$.
Secondly,
if $b_0 \in S$, our proof is complete since $b_0 > a$ and $b_0 \in S$, therefore $b_0 \in Z$.
We have found an element in the superset that is greater than our assumed
supremum. This is a contradiction.
\newline
If $b_0 \notin S$, then it must be smaller than the infimum of $S$ since $S$ is a subset of $\R$. In this case $b_0 < s \forall s \in S$. Once again we have found an element in the superset that is greater than the supremum, a contradiction.
\newline
From both contradictions we can see that $\sup (S \cup T) = \max \{\sup S, \sup T\}$
\newline
Q.E.D.
\newpage

\section{Ross 4.8}
\subsection{a}
We pick an arbitrary $s \in S$. According to the specifications of the problem, this $s \leq t \forall t \in T$. Therefore this $s$ is a lower bound of the set $T$, it is bounded below.
\newline
We can repeat the same logic symetrically and pick $t \in T$ to show that it is greater than or equal to every element of $S$. So $S$ is bounded above.

\subsection{b}
We will prove this via contradiction. Assume that the statement is false, $\sup S > \inf T$. By the definition of the supremum, $\exists s \in S s.t. s > \inf T$. If T has no supremum or if the supremum is greater than or equal to $s$, $\exists t \in T s.t. t < s$. Otherwise, $s \geq t \forall t \in T$. Either way, we have found an element in each set that contradicts the prerequisites of this problem.
\newline
Our assumption is false and $\sup S \leq \inf T$.

\subsection{c}
Let $S = T = \{ 0 \}$. Since they are the same set, it satisfies that $s \leq t \forall s \in S$ and $t \in T$. $S \cap T = \{0\}$, a non-empty set.

\subsection{d}
Let $S = {s \in \R | 0 \leq s < 5}, T = {t \in \R | 5 < t < 10}$. This satisfies that $s \leq t \forall s \in S and t \in T$ and $\sup S = \inf T$. However, since the ends at 5 for the two sets are open, they have no overlap. $S \cap T = \{\}$

\newpage

\section{Ross 4.11}
For this problem we simply need to replace the 1 in the denseness proof with an arbitrary n.
\newline
Let $a,b \in \R, a < b, c \in \N$. By the Archimedean property there exists $n \in \N$ such that $n(b-a)>c$. Therefore $bn - an > c$. Furthermore, by the same property there is an integer $k$ such that $k > \max{|an|, |bn|}$. Therefore $-k < an < bn < k$.
\newline
Then consider the set $J ={j \in \Z, -k \leq j \leq k}, K = {k \in K, k > an}$. This set is a subset of integers, bounded above and below, and non empty (contains at least k). Let $m_1 = \min K$. Then $-k < an < m_1$. Since $m_1 > -k$, $m_1$ is in J. $an > m_1-1$ by our choice of $m_1$.  $m_1-1 \leq an$, $m_1 \leq an+1 \leq bn$. Therefore $an < m_1 < bn$. We can simply let $m_2 = m_1 + 1$. Since $bn - an > c$, we can keep adding 1 to $m_1$ until $c-1$. Furthermore, we can pick $c$ to be arbitrarily large, so we can add 1 arbitrarily many times. Therefore it is infinite.
\newline
Q.E.D.

\newpage
\section{Q7}

\subsection{a}
We assume that this is false, so $r^2 < 2$ or $r^2 > 2$. For the former we can let $x^2 = \frac{2+r^2}{2}$. This $x^2$ is greater than $r^2$ and less than $2$, so $x$ must be greater than $r$ by problem 3. By the denseness of rationals, there must be a rational between $x$ and $\sqrt 2$. This rational is in the set and greater than the supremum. It is a contradiction, so $r^2 \geq 2$.
\newline
If $r^2>2$, we can simply change our argument above symetrically. Let $y^2 = \frac{2+r^2}{2}$. This $y^2$ is greater than 2 but less than $r^2$. By problem 3 $y$ must be smaller than $r$. Since $y^2$ is greater than 2, it is greater than every element in $S$. We have found a smaller upperbound than the supremum. This is a contradiction, therefore $r^2 = 2$.

\subsection{b}
In this set, consider $a = 3$. $a > s \forall s \in S$. Since s is bounded above, it must have an supremum by the Completeness Axiom. As we have proven above, $r^2 = 2$ must exist.

\subsection{c}
To prove this, we need to demonstrate that $r^2 = 2, r \notin \Q$.
\newline
Assume that $r \in \Q$ and that $r = \frac{p}{q}| p,q \in \N, \gcd(p,q)$. Since $r^2 = 2$, $2 = \frac{p^2}{q^2}$
$$p^2 = 2 q^2$$
From this we can see that $p^2$ is even since it is equal to 2 times $q^2$. For $p^2$ to be even, $p$ must also be even. So we can write $p = 2k( k \in \Z)$, $r = \frac{2k}{q}$
$$r^2 = 2 = \frac{4k^2}{q^2}$$
$$2q^2 = 4k^2$$
$$q^2 = 2k^2$$
Here we see that $q^2$ is also even, so $q$ must be even. However, we have assumed that $\gcd(p,q)=1$. This is a contradiction, so $r \notin \Q$.
\newline
Therefore the Completeness Axiom does not hold for $\Q$.

\end{document}


\documentclass{wg21}

\usepackage{xcolor}
\usepackage{soul}
\usepackage{ulem}
\usepackage{fullpage}
\usepackage{parskip}
\usepackage{csquotes}
\usepackage{listings}
\usepackage{minted}
\usepackage{enumitem}

\lstdefinestyle{base}{
  language=c++,
  breaklines=false,
  basicstyle=\ttfamily\color{black},
  moredelim=**[is][\color{green!50!black}]{@}{@},
  escapeinside={(*@}{@*)}
}

\newcommand{\cc}[1]{\mintinline{c++}{#1}}
\newminted[cpp]{c++}{}

\title{Making \cc{std::forward_list} constexpr}
\docnumber{P1929R0}
\audience{LEWGI}
\author{Alexander Zaitsev}{zamazan4ik@tut.by, zamazan4ik@gmail.com}

\begin{document}
\maketitle

\section{Revision history}
\begin{itemize}
  \item R0 -- Initial draft
\end{itemize}

\section{Abstract}
\cc{std::forward_list} is not currently \cc{constexpr} friendly. With the loosening
of requirements on \cc{constexpr} in \cite{P0784R1} and related papers, we
can now make \cc{std::forward_list} \cc{constexpr}, and we should in order to support
the \cc{constexpr} reflection effort (and other evident use cases).

\section{Motivation}
\cc{std::forward_list} is not so widely-used standard container as \cc{std::vector} or \cc{std::string}. But there is no reason to keep \cc{std::forward_list} in non-\cc{constexpr} state since one of the main directions of C++ evolution is compile-time programming. And we want to use in compile-time as much as possible from STL. And this paper makes \cc{std::forward_list} available in compile-time.

\section{Proposed wording}
We basically mark all the member and non-member functions of \cc{std::forward_list} \cc{constexpr}.

Direction to the editor: please apply \cc{constexpr} to all of \cc{std::forward_list},
including any additions that might be missing from this paper.

In \textbf{[support.limits.general]}, add the new feature test macro
\cc{__cpp_lib_constexpr_forward_list} with the corresponding value for header
\cc{<forward_list>} to Table 36 \textbf{[tab:support.ft]}.

Change in \textbf{[forwardlist.syn] 22.3.4}:
\begin{quote}
\begin{codeblock}
#include <initializer_list>

namespace std {
  // 22.3.10, class template \tcode{forward_list}
  template<class T, class Allocator = allocator<T>> class forward_list;

  template<class T, class Allocator>
    @\added{constexpr}@ bool operator==(const forward_list<T, Allocator>& x, const forward_list<T, Allocator>& y);
  template<class T, class Allocator>
    @\added{constexpr}@ synth-three-way-result<T> operator<=>(const forward_list<T, Allocator>& x, const forward_list<T, Allocator>& y);

  template<class T, class Allocator>
    @\added{constexpr}@ void swap(forward_list<T, Allocator>& x, forward_list<T, Allocator>& y)
      noexcept(noexcept(x.swap(y)));

  template<class T, class Allocator, class U>
    @\added{constexpr}@ void erase(forward_list<T, Allocator>& c, const U& value);
  template<class T, class Allocator, class Predicate>
    @\added{constexpr}@ void erase_if(forward_list<T, Allocator>& c, Predicate pred);

  [...]
}
\end{codeblock}
\end{quote}

Add after \textbf{[forwardlist.overview] 22.3.9.1/2}:
\begin{quote}
\added{The types \texttt{iterator} and \texttt{const_iterator} meet the
constexpr iterator requirements ([iterator.requirements.general]).}
\end{quote}

Change in \textbf{[forwardlist.overview] 22.3.9.1}:
\begin{quote}
\begin{codeblock}
namespace std {
  template<class T, class Allocator = allocator<T>>
  class forward_list {
  public:
    // types
    using value_type             = T;
    using allocator_type         = Allocator;
    using pointer                = typename allocator_traits<Allocator>::pointer;
    using const_pointer          = typename allocator_traits<Allocator>::const_pointer;
    using reference              = value_type&;
    using const_reference        = const value_type&;
    using size_type              = @\impdef@; // see 22.2
    using difference_type        = @\impdef@; // see 22.2
    using iterator               = @\impdef@; // see 22.2
    using const_iterator         = @\impdef@; // see 22.2

    // 22.3.9.2, construct/copy/destroy
    @\added{constexpr}@ forward_list() : forward_list(Allocator()) { }
    @\added{constexpr}@ explicit forward_list(const Allocator&);
    @\added{constexpr}@ explicit forward_list(size_type n, const Allocator& = Allocator());
    @\added{constexpr}@ forward_list(size_type n, const T& value, const Allocator& = Allocator());
    template<class InputIterator>
      @\added{constexpr}@ forward_list(InputIterator first, InputIterator last, const Allocator& = Allocator());
    @\added{constexpr}@ forward_list(const forward_list& x);
    @\added{constexpr}@ forward_list(forward_list&& x);
    @\added{constexpr}@ forward_list(const forward_list& x, const Allocator&);
    @\added{constexpr}@ forward_list(forward_list&& x, const Allocator&);
    @\added{constexpr}@ forward_list(initializer_list<T>, const Allocator& = Allocator());
    @\added{constexpr}@ ~forward_list();
    @\added{constexpr}@ forward_list& operator=(const forward_list& x);
    @\added{constexpr}@ forward_list& operator=(forward_list&& x)
      noexcept(allocator_traits<Allocator>::is_always_equal::value);
    @\added{constexpr}@ forward_list& operator=(initializer_list<T>);
    template<class InputIterator>
      @\added{constexpr}@ void assign(InputIterator first, InputIterator last);
    @\added{constexpr}@ void assign(size_type n, const T& u);
    @\added{constexpr}@ void assign(initializer_list<T>);
    @\added{constexpr}@ allocator_type get_allocator() const noexcept;

    // 22.3.9.3, iterators
    @\added{constexpr}@ iterator               before_begin() noexcept;
    @\added{constexpr}@ const_iterator         before_begin() const noexcept;
    @\added{constexpr}@ iterator               begin() noexcept;
    @\added{constexpr}@ const_iterator         begin() const noexcept;
    @\added{constexpr}@ iterator               end() noexcept;
    @\added{constexpr}@ const_iterator         end() const noexcept;

    @\added{constexpr}@ const_iterator         cbegin() const noexcept;
    @\added{constexpr}@ const_iterator         cbefore_begin() const noexcept;
    @\added{constexpr}@ const_iterator         cend() const noexcept;

    // capacity
    [[nodiscard]] @\added{constexpr}@ bool empty() const noexcept;
    @\added{constexpr}@ size_type max_size() const noexcept;

    // 22.3.9.4, element access
    @\added{constexpr}@ reference       front();
    @\added{constexpr}@ const_reference front() const;

    // 22.3.9.5, modifiers
    template<class... Args> @\added{constexpr}@ reference emplace_front(Args&&... args);
    @\added{constexpr}@ void push_front(const T& x);
    @\added{constexpr}@ void push_front(T&& x);
    @\added{constexpr}@ void pop_front();

    template<class... Args> @\added{constexpr}@ iterator emplace_after(const_iterator position, Args&&... args);
    @\added{constexpr}@ iterator insert_after(const_iterator position, const T& x);
    @\added{constexpr}@ iterator insert_after(const_iterator position, T&& x);

    @\added{constexpr}@ iterator insert_after(const_iterator position, size_type n, const T& x);
    template<class InputIterator>
    @\added{constexpr}@ iterator insert_after(const_iterator position, InputIterator first, InputIterator last);
    @\added{constexpr}@ iterator insert_after(const_iterator position, initializer_list<T> il);

    @\added{constexpr}@ iterator erase_after(const_iterator position);
    @\added{constexpr}@ iterator erase_after(const_iterator first, const_iterator last);
    @\added{constexpr}@ void     swap(forward_list&)
      noexcept(allocator_traits<Allocator>::is_always_equal::value);

    @\added{constexpr}@ void resize(size_type sz);
    @\added{constexpr}@ void resize(size_type sz, const value_type& c);
    @\added{constexpr}@ void     clear() noexcept;

    // 22.3.9.6, forward_list operations
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list& x);
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x);
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list& x, const_iterator i);
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x, const_iterator i);
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list& x, const_iterator first, const_iterator last);
    @\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x, const_iterator first, const_iterator last);

    @\added{constexpr}@ size_type remove(const T& value);
    template<class Predicate> @\added{constexpr}@ size_type remove_if(Predicate pred);

    @\added{constexpr}@ size_type unique();
    template<class BinaryPredicate>
    @\added{constexpr}@ size_type unique(BinaryPredicate binary_pred);

    @\added{constexpr}@ void merge(forward_list& x);
    @\added{constexpr}@ void merge(forward_list&& x);
    template<class Compare> @\added{constexpr}@ void merge(forward_list& x, Compare comp);
    template<class Compare> @\added{constexpr}@ void merge(forward_list&& x, Compare comp);

    @\added{constexpr}@ void sort();
    template<class Compare> @\added{constexpr}@ void sort(Compare comp);

    @\added{constexpr}@ void reverse() noexcept;
  };

  template<class InputIterator,
           class Allocator = allocator<@\textit{iter-value-type<InputIterator>}@>>
    forward_list(InputIterator, InputIterator, Allocator = Allocator())
      -> forward_list<@\textit{iter-value-type<InputIterator>}@, Allocator>;

  // swap
  template<class T, class Allocator>
    @\added{constexpr}@ void swap(forward_list<T, Allocator>& x, forward_list<T, Allocator>& y)
      noexcept(noexcept(x.swap(y)));
}
\end{codeblock}%
\end{quote}

Change in \textbf{[forwardlist.cons] 22.3.9.2}:
\begin{quote}
\begin{itemdecl}
@\added{constexpr}@ explicit forward_list(const Allocator&);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ explicit forward_list(size_type n, const Allocator& = Allocator());
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ forward_list(size_type n, const T& value, const Allocator& = Allocator());
\end{itemdecl}
[...]
\begin{itemdecl}
template<class InputIterator>
  @\added{constexpr}@ forward_list(InputIterator first, InputIterator last,
                             const Allocator& = Allocator());
\end{itemdecl}
[...]
\end{quote}

Change in \textbf{[forwardlist.capacity] 22.3.9.3}:
\begin{quote}
\begin{itemdecl}
@\added{constexpr}@ iterator before_begin() noexcept;
@\added{constexpr}@ const_iterator before_begin() const noexcept;
@\added{constexpr}@ const_iterator cbefore_begin() const noexcept;
\end{itemdecl}
[...]
\end{quote}

Change in \textbf{[forwardlist.access] 22.3.9.4}:
\begin{quote}
\begin{itemdecl}
@\added{constexpr}@ reference front();
@\added{constexpr}@ const_reference front() const;
\end{itemdecl}
[...]
\end{quote}

Change in \textbf{[forwardlist.modifiers] 22.3.9.5}:
\begin{quote}
\begin{itemdecl}
template<class... Args> @\added{constexpr}@ reference emplace_front(Args&&... args);

@\added{constexpr}@ void push_front(const T& x);
@\added{constexpr}@ void push_front(T&& x);

@\added{constexpr}@ void pop_front();

@\added{constexpr}@ iterator insert_after(const_iterator position, const T& x);
@\added{constexpr}@ iterator insert_after(const_iterator position, T&& x);
@\added{constexpr}@ iterator insert_after(const_iterator position, size_type n, const T& x);
template<class InputIterator>
  @\added{constexpr}@ iterator insert_after(const_iterator position, InputIterator first, InputIterator last);
@\added{constexpr}@ iterator insert_after(const_iterator position, initializer_list<T>);

template<class... Args> @\added{constexpr}@ iterator emplace_after(const_iterator position, Args&&... args);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ iterator erase_after(const_iterator position);
@\added{constexpr}@ iterator erase_after(const_iterator first, const_iterator last);

@\added{constexpr}@ void resize(size_type sz);

@\added{constexpr}@ void resize(size_type sz, const value_type& c);

@\added{constexpr}@ void clear() noexcept;
\end{itemdecl}
\end{quote}

Change in \textbf{[forwardlist.ops] 22.3.9.6}:
\begin{quote}
\begin{itemdecl}
@\added{constexpr}@ void splice_after(const_iterator position, forward_list& x);
@\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x);
[...]
@\added{constexpr}@ void splice_after(const_iterator position, forward_list& x, const_iterator i);
@\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x, const_iterator i);
[...]
@\added{constexpr}@ void splice_after(const_iterator position, forward_list& x, const_iterator first,
const_iterator last);
@\added{constexpr}@ void splice_after(const_iterator position, forward_list&& x, const_iterator first,
const_iterator last);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ size_type remove(const T& value);
template<class Predicate> @\added{constexpr}@ size_type remove_if(Predicate pred);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ size_type unique();
template<class BinaryPredicate> @\added{constexpr}@ size_type unique(BinaryPredicate binary_pred);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ void merge(forward_list& x);
@\added{constexpr}@ void merge(forward_list&& x);
template<class Compare> @\added{constexpr}@ void merge(forward_list& x, Compare comp);
template<class Compare> @\added{constexpr}@ void merge(forward_list&& x, Compare comp);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ void sort();
template<class Compare> @\added{constexpr}@ void sort(Compare comp);
\end{itemdecl}
[...]
\begin{itemdecl}
@\added{constexpr}@ void reverse() noexcept;
\end{itemdecl}
[...]
\end{quote}

Change in \textbf{[forwardlist.erasure] 22.3.9.7}:
\begin{quote}
\begin{itemdecl}
template<class T, class Allocator, class U>
  @\added{constexpr}@ void erase(forward_list<T, Allocator>& c, const U& value);

template<class T, class Allocator, class Predicate>
  @\added{constexpr}@ void erase_if(forward_list<T, Allocator>& c, Predicate pred);
\end{itemdecl}
\end{quote}

\section{Implementation}
Possible implementation can be found here: \href{https://github.com/ZaMaZaN4iK/llvm-project/tree/feature/forward_list_constexpr}{LLVM fork}. Notice that when proposal was written constexpr destructors were not supported in Clang. Also in this implementation isn't used \cc{operator<=>} - bunch of old operators used instead (just because libcxx at the moment doesn't use \cc{operator<=>} for \cc{std::forward_list}).

\section{References}
\renewcommand{\section}[2]{}%
\begin{thebibliography}{9}

\bibitem[P0784R1]{P0784R1}
  Multiple authors,
  \emph{Standard containers and constexpr}\newline
  \url{http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0784r1.html}

\end{thebibliography}

\end{document}


\section{Introduction}
\subsection{Exercise 1.1: Self-Play}
\subsubsection*{Q}
Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case? Would it learn a different policy for selecting moves?

\subsubsection*{A}
\begin{itemize}
    \item Would learn a different policy than playing a fixed opponent since the opponent would also be changing in this case.
    \item May not be able to learn an optimal strategy as the opponent keeps changing also.
    \item Could get stuck in loops.
    \item Policy could remain static since on average they would draw each iteration.
\end{itemize}

\subsection{Exercise 1.2: Symmetries}
\subsubsection*{Q}
Many tic-tac-toe positions appear different but are really the same because of symmetries. How might we amend the learning process described above to take advantage of this? In what ways would this change improve the learning process? Now think again. Suppose the opponent did not take advantage of symmetries. In that case, should we? Is it true, then, that symmetrically equivalent positions should necessarily have the same value?

\subsubsection*{A}
\begin{itemize}
    \item We could label the states as unique up to symmetries so that our search space is smaller, this way we will get a better estimate of optimal play.
    \item If we are playing an opponent who does not take symmetries into account when they are playing then we should not label the states as the same since the opponent is part of the environment and the environment is not the same in those states.
\end{itemize}

\subsection{Exercise 1.3: Greedy Play}
\subsubsection*{Q}
Suppose the reinforcement learning player was greedy, that is, it always played the move that brought it to the position that it rated the best. Might it learn to play better, or worse, than a nongreedy player? What problems might occur

\subsubsection*{A}
\begin{itemize}
    \item The greedy player will not explore, so will in general perform worse than the non-greedy player
    \item If the greedy player had a perfect estimate of the value of states then this would be fine.
\end{itemize}

\subsection{Exercise 1.4: Learning from Exploration}
\subsubsection*{Q}
Suppose learning updates occurred after all moves, including exploratory moves. If the step-size parameter is appropriately reduced over time (but not the tendency to explore), then the state values would converge to a set of probabilities. What are the two sets of probabilities computed when we do, and when we do not, learn from exploratory moves? Assuming that we do continue to make exploratory moves, which set of probabilities might be better to learn? Which would result in more wins?
\subsubsection*{A}
I think that an estimate for the probability of the state producing a win should be based on the optimal moves from that state.
\begin{itemize}
    \item The one in which we only record the optimal moves is the probability of our optimal agent winning. If we include exploration then this is the probability of the training agent winning.
    \item Better to learn the probability of winning with no exploration since this is how the agent will perform in real time play.
    \item Updating from optimal moves only will increase probability of winning.
\end{itemize}

\subsection{Exercise 1.5: Other Improvements}
\subsubsection*{Q}
Can you think of other ways to improve the reinforcement learning player? Can you think of any better way to solve the tic-tac-toe problem as posed?

\subsubsection*{A}
I'm not too sure here...
\begin{itemize}
    \item We could rank the draws as better than the losses.
    \item We might like to try running multiple iterations of games before updating our weights as this might give a better estimate.
\end{itemize}