\documentclass[12pt,letterpaper]{article}
\usepackage[latin1]{inputenc}
\usepackage{amsmath, amsfonts, amssymb} % for equations
\usepackage{graphicx} % for inserting figures

% the above packages are the "base"

\usepackage{hyperref} % enable links within pdf
\hypersetup{colorlinks = true, linkcolor = black, urlcolor = blue}

\usepackage{float} % to fix the position of figures using [H]

% in-text citation styles
\usepackage[sort]{natbib}
\bibpunct[; ]{(}{)}{;}{a}{,}{;}

%  You "comment out" lines with the % symbol.

\title{\textbf{This will be my awesome title.}\\An Intro to \LaTeX}

\author{} % insert your name here to have it appear

%\date{} % uncomment to have the date appear

\begin{document}

\maketitle

\tableofcontents

\pagebreak

\section{Introduction}
\label{sec:intro} % labels allow automatic referencing

I think it's fair to say that most biologists switch from using word processors (like Microsoft Word or Apple Pages) to \LaTeX\ when they start writing papers containing more than just one or two equations.
I have to admit though, that after using both \LaTeX\ and Pages for a few years (depending on the type of manuscript I was writing), I've now switched entirely to writing manuscripts in \LaTeX\ (when collaborators allow).
It took me a while to get the hang of it, but now I prefer it even when there are no equations involved.

Note that your \LaTeX\ file is code that gets compiled.  Also note that (almost) all the lecture notes for
this class were written in \LaTeX\, so take a look at a few of their \texttt{tex} files to learn a few additional
tricks.

\section{Basics}

\subsection{Document structure}

What isn't visible here in this pdf document is that the code for every \LaTeX\ document begins with a preamble that sets things up.
In the preamble you define the type of document it is, load necessary
packages, define any additional functions you might like to have, and provide your author name and the title of the paper.
Your actual text is then written after beginning your document using \verb+\begin{document}+ and is followed at the very end by \verb+\end{document}+.\footnote{Many 	other functions use the same  begin $\ldots$ end structure.}

You ``comment out'' lines with the \% symbol.
Also, if you look at the raw \texttt{.tex} file behind this pdf, you'll notice that I wrote each sentence on its own line (i.e. with hard-returns between each).
That's not necessary, but allows \texttt{Git} to distinguish changes on a sentence-specific rather than whole paragraph basis.
A common alternative is to setup your editor such that all lines are limited to 72 or 80 characters.

\subsection{Basic equations}

The fundamental principle of calculus entails
\begin{equation}
	\label{eqn:calculus}
	\lim_{{\Delta t} \to 0}\frac{f(a+\Delta t)-f(a)}{\Delta t}.
\end{equation}
Although the typical way of writing a derivative is $\frac{dx}{dt}$, some fields also write it as $\dot{x}$.

\subsection{Lists}

\noindent % if you don't want a particular section to auto-indent
Calculus rocks because it can be used to represent all of the following:
\begin{itemize}
	\item first item
	\item second item
\end{itemize}
It also rocks because
\begin{enumerate}
	\item first item
	\item second item
\end{enumerate}

\subsection{Tables}

One could also organize all the things calculus can be used for in a table:
\begin{center}
\begin{tabular}{lc} % {lc} means the 1st column is left-aligned while the 2nd is centered
		\hline
		\hline
		Reason & Explanation\\  % the \\ is  a "return/next line"
		\hline
			1 & blah blah blah \\
			2 & blah blah blah \\
		\hline
		\hline
	\label{tab:mytable}
\end{tabular}
\end{center}

\subsection{Within-text referencing}

I can easily reference the section (sect.\ref{sec:intro}), equation (eqn. \ref{eqn:calculus}) and table (Table \ref{tab:mytable}).
Their numbers will auto-generate, which makes it easy to move them around in your paper and adhere to
a journal's stylistic preferences.

\subsection*{My un-numbered subsection}

Sections and subsection are numbered by default, but that can be overwritten for a given section, or globally using \verb+\setcounter{secnumdepth}{0}+ in the preamble.

\section{More equations}

\subsection*{Using align}

The Lotka-Voltera equations are given by
\begin{align}
\frac{dx}{dt} 	&=	\alpha x - \beta xy \\
\frac{dy}{dt}	&=	\gamma xy - \delta y
\end{align}

Often it's useful to typeset the steps of derivations.
Pay it forward to folks who are trying to learn these methods, and to yourself when you can't remember the details but have to lecture on it in 5 minutes.
In these cases you don't need to number each line.

\begin{align}
N(10) 	&= 	\lambda N(9) 		\\ 		\notag
		&= 	\lambda^2 N(8) 	\\		\notag
		&=	\lambda^3 N(7) 	\\		\notag
		& \dots				\\		\notag
		&=	\lambda^{10} N(0)	\\		\notag
\end{align}

\section{Inserting figures}

Note that the position of figures is auto-determined (e.g., Fig.~\ref{fig:logo})!
You can force the position of figures using the \texttt{float} package and then the [H] option for your figure (e.g., Fig.~\ref{fig:miktex}).

% Figures are included like this:
\begin{figure}
	\centering
	\includegraphics[width=0.2\linewidth]{figs/LaTeX_logo.png}
	\caption{This is the \LaTeX~logo.}
	\label{fig:logo}
\end{figure}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.8\linewidth]{figs/miktex.png}
	\caption{Why use \LaTeX\ (source: \url{http://www.pinteric.com/miktex.html})}
	\label{fig:miktex}
\end{figure}

\section{Inserting external tables}

It's relatively easy to use the R \texttt{Hmisc} package to generate \LaTeX\ tables (see
\texttt{../R/ExportTable.R}) and then import them into your document using \texttt{input}.
Again, just like figures, their placement in the document is auto-determined.
If you provided a caption to the tables when generating their tex files in R, it's easy to reference them (Table \ref{tab:data} and \ref{tab:coefs}).

\input{tables/data}
\input{tables/estcoefs}

\section{References}

The \texttt{natbib} package is great for citing references.
Reformatting for a different journal is as easy as changing the arguments of a function.
How to cite references and include them in a bibliography is demonstrated in the accompanying
\texttt{manuscript.tex} template in the parent folder of these lecture notes.

\begin{figure}
	\centering
	\includegraphics[width=1\linewidth]{figs/phd022702s.png}
	\caption{Reading is fundamental (source: \url{http://phdcomics.com/comics/archive.php?comicid=286})}
	\label{fig:phdrefs}
\end{figure}

\end{document}


% !TEX TS-program = pdflatex
% !TEX encoding = UTF-8 Unicode

% This is a simple template for a LaTeX document using the "article" class.
% See "book", "report", "letter" for other types of document.

\documentclass[11pt]{article} % use larger type; default would be 10pt

\usepackage[utf8]{inputenc} % set input encoding (not needed with XeLaTeX)

%%% Examples of Article customizations
% These packages are optional, depending whether you want the features they provide.
% See the LaTeX Companion or other references for full information.

%%% PAGE DIMENSIONS
\usepackage{geometry} % to change the page dimensions
\geometry{a4paper} % or letterpaper (US) or a5paper or....
% \geometry{margin=2in} % for example, change the margins to 2 inches all round
% \geometry{landscape} % set up the page for landscape
%   read geometry.pdf for detailed page layout information

\usepackage{graphicx} % support the \includegraphics command and options

% \usepackage[parfill]{parskip} % Activate to begin paragraphs with an empty line rather than an indent

%%% PACKAGES
\usepackage{booktabs} % for much better looking tables
\usepackage{array} % for better arrays (eg matrices) in maths
\usepackage{paralist} % very flexible & customisable lists (eg. enumerate/itemize, etc.)
\usepackage{verbatim} % adds environment for commenting out blocks of text & for better verbatim
\usepackage{subfig} % make it possible to include more than one captioned figure/table in a single float
% These packages are all incorporated in the memoir class to one degree or another...

%%% HEADERS & FOOTERS
\usepackage{fancyhdr} % This should be set AFTER setting up the page geometry
\pagestyle{fancy} % options: empty , plain , fancy
\renewcommand{\headrulewidth}{0pt} % customise the layout...
\lhead{}\chead{}\rhead{}
\lfoot{}\cfoot{\thepage}\rfoot{}

%%% SECTION TITLE APPEARANCE
\usepackage{sectsty}
\allsectionsfont{\sffamily\mdseries\upshape} % (See the fntguide.pdf for font help)
% (This matches ConTeXt defaults)

\usepackage{amsmath}

%%% ToC (table of contents) APPEARANCE
\usepackage[nottoc,notlof,notlot]{tocbibind} % Put the bibliography in the ToC
\usepackage[titles,subfigure]{tocloft} % Alter the style of the Table of Contents
\renewcommand{\cftsecfont}{\rmfamily\mdseries\upshape}
\renewcommand{\cftsecpagefont}{\rmfamily\mdseries\upshape} % No bold!

%%% END Article customizations

%%% The "real" document content comes below...

\title{Tsvstat}
\author{wozgonon}
\date{} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

\section{Introduction}

The {\it tsvtstat} tool can be used to generate a table of summary statistics given a table or stream of tab separated numeric observations.

\section{Statistics}

The formulas used to generate statistics are compatible with those used by common spreadsheets.

\subsection{Mean}

\begin{align*}
\bar{x}&=\sum \frac{x_i}{n}  \\
\implies  \sum x_i &= n\bar{x} & \text{(1)}
\end{align*}

\subsection{Sample Variance and Standard Deviation}

\begin{align*}
s^2&= \frac{1}{n-1}\sum_1^n (x_i-\bar{x})^2 & \text{Sample Variance}\\
              &=\frac{1}{n-1}\sum_1^n (x_i^2 - 2\bar{x} x_i +\bar{x}^2) \\
              &=\frac{1}{n-1}(\sum_1^n x_i^2 - 2\bar{x} \sum x_i +n\bar{x}^2) \\
              &=\frac{1}{n-1}(\sum_1^n x_i^2 - 2n\bar{x}^2 +n\bar{x}^2) & \text{substitute (1)}\\
              &=\frac{1}{n-1}\sum_1^n x_i^2 - n\bar{x}^2) \\
s &= \sqrt{s^2} & \text{Sample standard deviation} \\
\end{align*}

\subsection{Skewness}
\begin{align*}
skew &= \frac{n}{(n-1)(n-2)}\sum\frac{(x,-\bar{x})^3}{s^3} \\
        &= \frac{n}{(n-1)(n-2)}\frac{\sum x_i^3 - 3\bar{x}\sum x_i^2 + 2n\bar{x}^3}{s^3} & \text{subst. (3)}\\
\\
\sum_1^n {(x_i - \bar{x})^3} &= \sum_1^n (x_i^2 - 2x_i\bar{x} + \bar{x}^2)(x_i - \bar{x}) \\
                                &= \sum_1^n (x_i^3 - 2x_i^2\bar{x} + x_i\bar{x}^2 - \bar{x} x_i^2 + 2x_i\bar{x}^2 - \bar{x}^3) \\
                                &= \sum_1^n (x_i^3 - 3x_i^2\bar{x} + 3x_i\bar{x}^2 - \bar{x}^3) & \text{(2)}\\
                                &= \sum_1^n x_i^3 - 3\bar{x}\sum x_i^2 + 3\bar{x}^2\sum x_i - n\bar{x}^3 \\
                                &= \sum_1^n x_i^3 - 3\bar{x}\sum x_i^2 + 3n\bar{x}^3 - n\bar{x}^3 & \text{by (1)} \\
                                &= \sum_1^n x_i^3 - 3\bar{x}\sum x_i^2 + 2n\bar{x}^3 & \text{(3)} \\
\end{align*}

\subsection{Kurtosis}
\begin{align*}
kurt &= \frac{n(n+1)}{(n-1)(n-2)(n-3)}\sum_1^n\frac{(x,-\bar{x})^4}{s^4} - \frac{3(n-1)^2}{(n-2)(n-3)}\\
      &= \frac{n(n+1)}{(n-1)(n-2)(n-3)}\frac{\sum_1^n  x_i^4 - 4\bar{x}\sum_1^n x_i^3 + 6\bar{x}^2\sum_1^n x_i^2-  3n\bar{x}^4}{s^4} & \text{substitute (4)}\\
      & - \frac{3(n-1)^2}{(n-2)(n-3)} \\
\\
\text{excess\ kurtosis} &= kurt -3
\\
\sum_1^n {(x_i - \bar{x})^4} &= \sum_1^n  (x_i - \bar{x})^3(x_i - \bar{x}) \\
&=  \sum_1^n  (x_i^3 - 3x_i^2\bar{x} + 3x_i\bar{x}^2 - \bar{x}^3)(x_i - \bar{x}) & \text{substitute (2)} \\
&= \sum_1^n  x_i(x_i^3 - 3x_i^2\bar{x} + 3x_i\bar{x}^2 - \bar{x}^3) - \bar{x}(x_i^3 - 3x_i^2\bar{x} + 3x_i\bar{x}^2 - \bar{x}^3) \\
&= \sum_1^n  (x_i^4 - 3x_i^3\bar{x} + 3x_i^2\bar{x}^2 -  x_i\bar{x}^3) - (\bar{x}x_i^3 - 3x_i^2\bar{x}^2 + 3x_i\bar{x}^3 - \bar{x}^4) \\
&= \sum_1^n  (x_i^4 - 4x_i^3\bar{x} + 6x_i^2\bar{x}^2 -  4x_i\bar{x}^3 + \bar{x}^4) \\
&= \sum_1^n  x_i^4 - 4\bar{x}\sum_1^n x_i^3 + 6\bar{x}^2\sum_1^n x_i^2-  4\bar{x}^3 \sum_1^n x_i + n\bar{x}^4) \\
&= \sum_1^n  x_i^4 - 4\bar{x}\sum_1^n x_i^3 + 6\bar{x}^2\sum_1^n x_i^2-  4n\bar{x}^4 + n\bar{x}^4 \\
&= \sum_1^n  x_i^4 - 4\bar{x}\sum_1^n x_i^3 + 6\bar{x}^2\sum_1^n x_i^2-  3n\bar{x}^4 & \text{(4)}\\
\end{align*}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS240-2016S: Programming in C
% Copyright 2016 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS240-2016S
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

The following code snippets contain compilation errors.
Identify the errors and fix them so that the function properly performs the task indicated.

\begin{enumerate}[label=\textbf{(\alph*)}]

\item Function to convert a string to integer:
\begin{lstlisting}
/**
 * Updates a given pointer to an integer with a numeric value
 * that corresponds to a given string. Returns 0 if conversion
 * is successful and -1 if not.
 */
int str2int(char *str, int *num)
{
	char ch;
	int i = 0;
	int sign = 1;
	num = 0;
	if (str[i] == '-') {
		sign = -1;
		i++;
	}
	while ((ch = str[i++]) != '\0') {
		if (!isdigit(ch))
			return -1;
		num = num * 10 + ch - '0';
	}
	num *= sign;
	return 0;
}
\end{lstlisting}

\item Function to populate an array with random one-digit numbers:
\begin{lstlisting}
/**
 * Populates a given array with random 1-digit numbers.
 */
void populate(int *arr, int size)
{
	int i;
	srand(time(NULL));
	for (i = 0; i < size; i++)
		arr[i] = rand();
}
\end{lstlisting}

\item Function to check if a given string is palindrome:
\begin{lstlisting}
/**
 * Checks whether a given string is palindrome or not.
 * Returns 1 if palindrome is detected and 0 if not.
 */
int is_palindrome(char *str)
{
	int i;
	int len = 0;
	while (str[len++] != '\n');
	for (i = 0; i < len / 2; i++)
		if (str[i] != str[len - i - 2])
			return 0;
	return 1;
}
\end{lstlisting}

\end{enumerate}


\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{times}
\usepackage{listings}
\input{chapel_listing}

\pagestyle{empty}

\begin{document}
\lstset{language=chapel}

\section*{Histogram Exercise}

\section{Serial Histogram}
Using histogram.chpl as a starting point, fill in the function
\begin{chapel}
computeHistogram(X: [] real, Y: [] int)
\end{chapel}
This function takes two arguments.  \chpl{X} is an array of random
numbers defined over \chpl{1..numNumbers}.  \chpl{Y} is an empty array
that should contain the histogram data.  It is defined over
\chpl{1..numBuckets}.

The output should look something like:
\begin{quote}
\begin{footnotesize}
\begin{verbatim}
> a.out
Running Histogram Example
 Number of Random Numbers =        8
 Number of Buckets        =       10

Random Numbers

0.585924 0.59617 0.491852 0.788843 0.354106 0.031778 0.469706 0.4702

Histogram computed in 1e-06 seconds

     X
     XX
 X  XXX X
+----------+
 Raw Data
  0.00 - 0.10: 1
  0.10 - 0.20: 0
  0.20 - 0.30: 0
  0.30 - 0.40: 1
  0.40 - 0.50: 3
  0.50 - 0.60: 2
  0.60 - 0.70: 0
  0.70 - 0.80: 1
  0.80 - 0.90: 0
  0.90 - 1.00: 0
\end{verbatim}
\end{footnotesize}
\end{quote}

To change the problem size, set the configuration variable on the
command line as follows:
\begin{quote}
\begin{footnotesize}
\begin{verbatim}
> a.out --numNumbers=1000
\end{verbatim}
\end{footnotesize}
\end{quote}
You can also change the number of buckets via \chpl{numBuckets} and
you can avoid printing the random numbers by setting
\chpl{printRandomNumbers} to \chpl{false}.  Try the following run:
\begin{quote}
\begin{footnotesize}
\begin{verbatim}
> a.out --printRandomNumbers=false --numNumbers=1000000
\end{verbatim}
\end{footnotesize}
\end{quote}

\section{Parallel Histogram}
Now add another configuration constant with the line
\begin{chapel}
config const numThreads: int = 4;
\end{chapel}
Then use \chpl{numThreads} in a function that computes the histogram
with that many threads.  Hint: use a \chpl{coforall} statement to
create the tasks and a sync variable to ensure that the code is
race-free.

Does your parallel code run faster than your serial code?  What about
at larger problem sizes?

\section{Normal Histogram}
Now add another array of random numbers and compute the histogram of
the element-wise averages.  You can store the averages in the first
array for simplicity.

\end{document}


\documentclass{article}

\usepackage{graphicx}
\usepackage{subcaption}

\begin{document}
	\section{ XOR Gate}
An XOR (exclusive OR) gate acts in the same way as the exclusive OR logical connector.
It gives a true output (result of 1) if one, and only one, of the inputs to the gate is true (1), i.e either or but not both

	\begin{figure}[h!]
		\centering
		\begin{subfigure}[b]{0.4\linewidth}
			\includegraphics[width=\linewidth]{XOR.png}
		\end{subfigure}

			\begin{subfigure}[b]{0.4\linewidth}
		\includegraphics[width=\linewidth]{XOR table.jpg}
	\end{subfigure}

	\end{figure}
\end{document}


\documentclass[aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\usepackage{amsmath,amsfonts,amsthm,bm}
\usepackage{color}
\usepackage{minted}
\usepackage{graphicx} % Allows including images
\usepackage{booktabs} % Allows the use of \toprule, \midrule and \bottomrule in tables
\usepackage{tikz}
\usepackage[version=3]{mhchem}
\usepackage{pgfplots}
\pgfplotsset{compat=1.16}
\setminted{fontsize=\scriptsize}

\hypersetup{
    colorlinks=true,
    linkcolor=red,
    filecolor=magenta,
    urlcolor=red,
}

\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\let \vec \mathbf

\mode<presentation> {
    \usetheme{CambridgeUS}
    %\setbeamertemplate{footline} % To remove the footer line in all slides uncomment this line
    \setbeamertemplate{footline}[page number] % To replace the footer line in all slides with a simple slide count uncomment this line
    \setbeamertemplate{navigation symbols}{} % To remove the navigation symbols from the bottom of all slides uncomment this line
}

\title[Neural Networks]{Neural Networks}

\author{Shyue Ping Ong}
\institute[UCSD]{University of California, San Diego\\
\medskip
}
\date{NANO281} % Date, can be changed to a custom date

\begin{document}

\begin{frame}
    \titlepage % Print the title page as the first slide
\end{frame}

\begin{frame}{Overview}
    \tableofcontents
\end{frame}

\section{Preliminaries}

\begin{frame}{Preliminaries}
    \begin{itemize}
        \item Neural networks/deep learning has gotten a lot of hype in recent years.
        \item In many areas, they have outperformed many traditional ML methodologies.
    \begin{figure}
        \centering
        \includegraphics[width=0.45\textwidth]{figures/imagenet.pdf}
        \includegraphics[width=0.45\textwidth]{figures/alphago.jpg}
    \end{figure}
    \end{itemize}

\end{frame}

\section{Neural Networks}

\begin{frame}{Artificial Neural Network}
\begin{itemize}
        \item An artificial neural network (ANN) is a learning algorithm that is (very) loosely based on the structure of the brain.
        \item Somewhat vaguely, you will also hear the terms ``multi-layer perceptron'' (MLP) used in place of ANN.
        \begin{block}{Universal Approximation Theorem\cite{csajiApproximationArtificialNeural2001}}
        A feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions under mild assumptions on the activation function.
        \end{block}
\end{itemize}
\begin{figure}
        \centering
        \includegraphics[width=0.4\textwidth]{figures/neuron.jpg}
    \end{figure}
\end{frame}

\begin{frame}{Neural Networks}
    \input{figures/neural_network_tikz}
\end{frame}

\begin{frame}{Neuron}
\begin{figure}
\begin{tikzpicture}
[roundnode/.style={circle, draw=black, very thick, minimum size=7mm}
]
\node at (0, 0) (X1) {$X_1$};
\node at (0, 1) (X2) {$X_2$};
\node at (0, 2) (X3) {$X_2$};
\node[roundnode] at (2, 1) (C) {};
\draw[->] (X1) -- (C);
\draw[->] (X2) -- (C);
\draw[->] (X3) -- (C);
\node at (4, 1) (Y) {$Y_m$};
\draw[->] (C) -- (Y);
\end{tikzpicture}
\end{figure}
\begin{equation*}
    Y_m = \sigma(\alpha_{0m} + \alpha_{m}^T \vec{X})
\end{equation*}
    \begin{itemize}
        \item Output of each neuron is a linear function of the inputs.
        \item In hidden layers, the output is passed through an \textit{activation function} $\sigma$.
        \item Common choices of $\sigma$ include the rectified linear unit (RELU), sigmoid function, softplus, etc.
    \end{itemize}
\end{frame}

\begin{frame}{Activation Functions}
\begin{figure}
    \centering
    \begin{tikzpicture}[scale=0.7]
        \begin{axis}[
            axis lines=middle,
            inner axis line style={-stealth},
            xlabel={$x$},
            ylabel={\large $y$},
            ytick={0,1,...,2},
            xtick={-2,-1,...,2},
            y=2cm,
            x=2cm,
            x tick label style={below},
            y tick label style={left},
            x label style={at={(ticklabel* cs:1.01)},anchor=west},
            y label style={at={(ticklabel* cs:1.05)},anchor=south},
            ymin=0,
            ymax=2,
            xmin=-2,
            xmax=2,
            legend style={at={(0,0.85)},anchor=west}
        ]
        \addplot[color=red,very thick] coordinates {(-2, 0) (0, 0) (2, 2)};
        \addlegendentry{relu}
        \addplot[color=green,very thick,domain=-2:2,samples=100] {ln(1+exp(x))};
        \addlegendentry{softplus}
        \node at (axis cs:-1,0.7) {$y=\ln(1+e^x)$};
        \node at (axis cs:1.3,0.4) {$y=\left\{ \begin{array}{ll}
            x & , x \geq 0  \\
            0 & , x < 0
        \end{array} \right.$};
        \end{axis}
    \end{tikzpicture}
    \begin{tikzpicture}[scale=0.7]
    \begin{axis}[
    axis lines=middle,
    inner axis line style={-stealth},
    xlabel={$x$},
    ylabel={\large $y$},
    ytick={0,0.5,1},
    xtick={-10,-5, 0,...,10},
    x label style={at={(ticklabel* cs:1.01)},anchor=west},
    y label style={at={(ticklabel* cs:1.05)},anchor=south},
    ymin=0,
    ymax=1,
    xmin=-10,
    xmax=10,
    legend style={at={(0,0.8)},anchor=west}
]
\addplot[color=red,very thick,domain=-10:10,samples=100] {1/(1+exp(-0.5*x))};
\addlegendentry{$s=0.5$}
\addplot[color=green,very thick,domain=-10:10,samples=100] {1/(1+exp(-x))};
\addlegendentry{$s=1$}
\addplot[color=blue,very thick,domain=-10:10,samples=100] {1/(1+exp(-2*x))};
\addlegendentry{$s=2$}
\node at (axis cs:5,0.5) {$y=\frac{1}{1+e^{-sx}}$};
\end{axis}
    \end{tikzpicture}
\end{figure}
\end{frame}

\begin{frame}{Fitting a Neural Network}
    \begin{itemize}
        \item Model \textit{weights} ($\alpha$ in the linear functions) are fitted by \textit{back-propogation}, basically a form of gradient descent.
        \item Loss functions: squared error for regression, squared error or cross entropy for classification.
        \item To avoid overfitting, regularization (similar to ridge regression) is typically applied. E.g., \textit{weight decay}:
        \begin{equation*}
            J = \sum \alpha^2
        \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{Parameter Decisions for Neural Networks}
    \begin{itemize}
        \item Number of hidden units and layers: generally error on the side of having too many hidden units than too few - flexibility is needed to capture non-linearities in the data.
        \item Extra weights can be shrunk to zero with appropriate regularization.
        \item Learning rate is a key parameter in fitting ANNs as well as other models. The learning rate controls the rate of gradient descent. A high learning rate reduces training time, but also decreases accuracy. State-of-the-art is to use an adaptive learning rate.
        \item The error function is non-convex and contains multiple minima, i.e., final solution obtained depends on initial weights. Typical approach is to start with a number of random starting configurations and choose solution with lowest penalized error.
        \item Alternative is to average predictions over a number of ANN, i.e., ensemble models.
    \end{itemize}
\end{frame}

\begin{frame}{Software packages for Neural Networks}
    \begin{itemize}
        \item Scikit-learn has a basic implementation of a multi-layer perceptron. For this course, we will use this to avoid overloading students with different software packages.
        \item However, you should be aware that if you are into serious work with ANNs, alternative software are available that offer more features and efficiency. Note that these packages are not limited to ANNs, but ANNs are the most common use case.
        \begin{itemize}
            \item \href{https://www.tensorflow.org/}{Tensorflow}. Open-source package by Google. Very powerful and efficient, but a very steep learning curve. Highly recommend that you use the Keras package (already integrated into TF2) which provides a high level API to it similar to scikit-learn in some ways.
            \item \href{https://pytorch.org/}{Pytorch}. Open-source package by Facebook. Equally powerful as TF but also steep learning curve.
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}[fragile]{ANNs in Scikit-learn and TF2.0}
\inputminted{python}{example_sklearn_tf_neural_network.py}
\end{frame}

\begin{frame}{Beyond the Feedforward Neural Network}
    \begin{itemize}
        \item ANNs can be an entire course in itself. Here, we cover some basic variations.
        \begin{description}
        \item[Convolutional NNs] Uses covolutional operations to achieve translational invariance. Especially important in image processing applications. In modeling crystals with periodic boundary conditions, this can also be a useful feature.
        \item[Recurrent NNs] Connections between nodes form a directed graph along a temporal sequence. Exhibits temporal dynamic behavior and can handle variable length sequences of inputs. Applications: handwriting recognition or speech recognition.
        \end{description}

    \end{itemize}
\end{frame}

\section{Applications}

\begin{frame}{Example Application: NN for interatomic potentials}
    \begin{itemize}
        \item Atom-centered symmetry functions (ACSF)\cite{behlerAtomcenteredSymmetryFunctions2011} to represent the atomic local environments.
        \begin{eqnarray*}
            G_{i}^{\rm atom, \rm rad} & = & \sum_{j\neq i}^{N_{\rm atom}} e^{-\eta(R_{ij}-R_{s})^{2}} \cdot f_{c}(R_{ij}), \\
            G_{i}^{\rm atom, \rm ang} & = & 2^{1-\zeta}\sum_{j,k \neq i}^{N_{\rm atom}} (1 + \lambda \cos \theta_{ijk})^{\zeta} \cdot e^{-\eta^{\prime}(R_{ij}^{2}+R_{ik}^{2}+R_{jk}^{2})}\cdot\\
            & & f_{c}(R_{ij}) \cdot f_{c}(R_{ik}) \cdot f_{c}(R_{jk}),
        \end{eqnarray*}
        where $R_{ij}$ is the distance between atom $i$ and neighbor $j$; $\eta$ is the width of the Gaussian; $R_{s}$ is the position shift over all neighboring atoms; $\zeta$ controls the angular resolution. $f_{c}(R_{ij})$ is a cutoff function.
        \item Fully connected ANNs describe the PES with respect to ACSFs.\cite{behlerHighDimensionalNeuralNetwork}
        \item Has been developed for Si, \ce{TiO2}, water, metal-organic frameworks, ZnO, \ce{Li3PO4}, etc.
    \end{itemize}
\end{frame}

\begin{frame}{Example Application: NNP for Si}
    \begin{figure}
        \centering
        \includegraphics[width=0.45\textwidth]{figures/nnp.pdf}
        \includegraphics[width=0.45\textwidth]{figures/nnp-si.pdf}
        \caption{Left: Schematic of NNP architecture. Right: Comparison of radial distribution functions from MD simulations using various interatomic potentials for Si.\cite{behlerGeneralizedNeuralNetworkRepresentation2007}}
        \label{fig:my_label}
    \end{figure}
\end{frame}

\begin{frame}{Example Application: NNP-computed phase diagram for amorphous Li-Si}
\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth]{figures/li-si-nnp.pdf}
    \includegraphics[width=0.5\textwidth]{figures/li-si-nnp2.pdf}
    \caption{From ref \cite{artrithConstructingFirstprinciplesPhase2018}}
\end{figure}
\end{frame}

\begin{frame}{Example application: ANNs to predict crystal stability}
\begin{figure}
    \centering
    \includegraphics[width=0.35\textwidth]{figures/dnn-crystal-stability.png}
    \includegraphics[width=0.35\textwidth]{figures/dnn-crystal-stability2.png}
    \caption{ANNs predict formation energies of garnets and perovskite crystals from Pauling electronegativity and ionic radii.\cite{yeDeepNeuralNetworks2018}}
\end{figure}
\end{frame}

\begin{frame}{Example application: CNNs for determination of crystal symmetry from electron diffraction}
\begin{figure}
    \centering
    \includegraphics[width=0.4\textwidth]{figures/science-cnn-electron-diffraction.jpg}
    \includegraphics[width=0.25\textwidth]{figures/science-cnn-electron-diffraction2.jpg}
    \caption{CNNs to determine crystal symmetry from electron diffraction patterns with $>$ 90\% accuracy.\cite{kaufmannCrystalSymmetryDetermination2020}}
\end{figure}
\end{frame}

\begin{frame}[allowframebreaks]{Bibliography}
    \bibliographystyle{unsrt}
    \bibliography{refs}
\end{frame}

\begin{frame}
    \Huge{\centerline{The End}}
\end{frame}

\end{document}