\documentclass[]{interact}

\usepackage{epstopdf} % To incorporate .eps illustrations using PDFLaTeX, etc.
\usepackage[caption=false]{subfig} % Support for small, `sub' figures and tables
%\usepackage[nolists,tablesfirst]{endfloat}% To `separate' figures and tables from text if required
%\usepackage[doublespacing]{setspace}% To produce a `double spaced' document if required
%\setlength\parindent{24pt}% To increase paragraph indentation when line spacing is doubled

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{float}

\usepackage[numbers,sort&compress]{natbib} % Citation support using natbib.sty
\bibpunct[, ]{[}{]}{,}{n}{,}{,} % Citation support using natbib.sty
\renewcommand\bibfont{\fontsize{10}{12}\selectfont} % Bibliography support using natbib.sty
\makeatletter % @ becomes a letter
\def\NAT@def@citea{\def\@citea{\NAT@separator}} % Suppress spaces between citations using natbib.sty
\makeatother % @ becomes a symbol again

\theoremstyle{plain} % Theorem-like structures provided by amsthm.sty
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{proposition}[theorem]{Proposition}

\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}

\theoremstyle{remark}
\newtheorem{remark}{Remark}
\newtheorem{notation}{Notation}

\begin{document}

\articletype{} % Specify the article type or omit as appropriate

\title{Comparing the Performance Between Genetic and Clonal Selection Algorithm with Trivial Self-Driving Mechanism}

\author{
\name{Yejun Chang (Andrew)}
\affil{Henry M. Gunn High School}
}

\maketitle

\begin{abstract}
The rise in popularity of machine learning incited engineers to ponder ways that machines can self-learn without explicit programming of instructions. The paper compares the efficiency of genetic and clonal selection algorithm in a simple problem of navigating a car around the track. By assessing the efficiency, this paper analyzes how clonal selection algorithm can aid with machine learning problems and the inspirations we can take from the human immune system. A total of five different selection mechanisms were tested on a track with various widths and curves, and clonal selection algorithm was determined to be faster on average than genetic algorithms.
\end{abstract}

\begin{keywords}
artificial immune system; clonal selection algorithm; evolutionary algorithm; genetic algorithm; neural network; self driving;
\end{keywords}

\section{Introduction}

Humans are continuously inspired by the natural world, and computer algorithms are no exception to this everlasting trend. The rise of big data and smart systems in recent years has led software engineers to seek efficient machine learning algorithms, and many have caught interest in developing algorithms around biological ideas. Neural networks are currently at the center of this practice in order to emulate the unpredictable nature of a human brain for computers. One area that has been increasing in popularity in recent years is \textit{artificial immune system}. Inspired by the way our body fights the invading pathogens, artificial immune system emulates such process and aims solve complex machine learning problems.

In this paper, we will review the foundations of genetic and clonal selection algorithms and test how each performs in a machine learning problem of navigating a car around the track. Genetic algorithm will be divided into its own sub-categories by giving three variations to its selection operator. The goal is to assess the efficiency of each algorithms, analyze the benefits and drawbacks, and understand their differences and implications. Our analysis will be centered around the clonal selection algorithm and how artificial immune systems can fair in the field of machine learning.

\section{Background}

\subsection{Genetic Algorithm}
Genetic algorithm is a searching or an optimization algorithm inspired by Charles Darwin's biological evolutionary theory based on natural selection where the fittest individual survives and passes on the genes to its offspring. Similarly, genes are encoded accordingly to a particular problem in a genetic algorithm. Often structured with an array-like nucleic acids in a DNA strand, these genes determine an individual's behavior and thus its fitness. Those with higher fitness are then selected to create a subsequent generation with higher average fitness value. During this process, genetic recombination and mutation are performed to ensure variation in the population. By repeating these steps over multiple generations, the population can converge to a desired solution. A population is considered to have converged if at least 95\% of the population share alike genes. The pseudocode follows the following general structure:

\hfill
\begin{algorithmic}[1]
    \Function{Genetic Algorithm}{}
	\State Randomly generate $initialPopulation$
    \State $currentPopulation$ $\leftarrow$ $initialPopulation$
    \While{$\neg StopCondition()$}
        \For{$p_{i}$ $\in$ $currentPopulation$}
            \State $CalculateFitness(p_{i})$
        \EndFor
        \State Sort $currentPopulation$ high to low by fitness
        \State $intermediatePopulation$ $\leftarrow$ $Selection(currentPopulation)$
        \State $newPopulation$ $\leftarrow$ $Recombination(intermediatePopulation)$
	    \State $Mutation(newPopulation)$
	    \State Increment generation count
	    \State $currentPopulation$ $\leftarrow$ $Recombination(newPopulation)$
	\EndWhile
	\EndFunction
\end{algorithmic}

\subsubsection{Roulette Wheel Selection}
Genetic algorithms can be varied by altering their selection operators. The first of the three variations explored in this paper is roulette wheel selection. The nomenclature is derived from the idea of spinning a roulette wheel and randomly selecting an individual. The selection mechanism is highly dependent on an individual's fitness value because the chance of selection for reproduction is much higher for ones with better fitness. The trivial method of roulette wheel selection is to take the sum of all fitness values in a sorted population---let's call this $S$. A random number $X$ in the interval $[0, S]$ and prefix sums of fitness values are generated, and an individual with the least prefix sum greater than $X$ is returned and placed in the intermediate population subsequently. \cite{Obi98} This repeats until the intermediate population is filled to its desired size. This process effectively generates the sector of a roulette wheel that each individual covers.

\subsubsection{Rank Selection}
The second of the three variations explored is rank selection. Instead of assigning chance of selection based on a fitness value, a fixed value of selection chance is apportioned to an individual according to its rank by fitness. For instance, an individual with highest fitness value will always be assigned 40\% selection chance. second place will be fixed at 30\%, third place at 20\%, and fourth place at 10\%. The apportionment of percent values is completely dependent on one's discretion.

\subsubsection{Remainder Stochastic Sampling}
The last of the three variations explored is remainder stochastic sampling. For each individual where the ratio of its fitness to the population average fitness is greater than 1, the floor of that ratio indicates the number of its copies placed in the intermediate population. Then, every individuals in the current population has a chance to place an additional copy in the intermediate population by a probability of the aforementioned ratio. \cite{Bus01}

\subsection{Clonal Selection Algorithm}
Frank Burnet's clonal selection theory roughly models the human immunological process as the following: a B cell proliferates into a clone after being activated by the binding of an antigen to a specific matching receptor on its surface. Some clonal cells differentiate into plasma cells, which are short-lived cells that secrete antibody against the antigen. Others form memory cells, which are longer-lived and which, by proliferating rapidly, help to mount an effective defense upon a second exposure to the antigen. \cite{CloND} This secondary immune response is much faster, incorporating antibodies with up to a 10,000 fold increase in binding affinity. \cite{UoA00} Memory cells undergo random somatic hypermutations or receptor editing in B cells, which can lead to such increase in binding affinity. Clonal selection of those better cells lead to faster immune response, while non-binding cells will die via apoptosis.

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/clonal-selection-principle.png}
    \caption{A diagram illustrating the clonal selection process. \cite{Dai07}}
    \label{fig:csp}
\end{figure}

The clonal selection algorithm is inspired by the clonal selection process presented above. While genetic algorithm relies on the evolutionary characteristic of sexual reproduction, clonal selection algorithm relies on the evolutionary characteristic of our immune system. A few best-fit individuals will proliferate, creating the clonal population. This clonal population undergoes hypermutation or receptor editing and the fitness is evaluated again. A few best-fit individuals will be chosen again subsequently to be put in the intermediate population. Until the size of the intermediate population is equal to the desired population size, top individuals from original current population will fill up the remaining intermediate population. This ensures the idea of elitism that best individuals are always preserved without getting lost to mutations. Additionally, a couple random new individuals may occasionally be introduced to the new population.

The main difference between somatic hypermutation and receptor editing is the degree of variation that each brings to a population. Whereas somatic hypermutation is an accumulation of point mutations in B cells, receptor editing is more analgous to chromosomal mutations that involves a long segment of DNA. Consequently, hypermutation usually brings smaller changes to an individual's affinity while receptor editing usually larger changes to the affinity. Hypermutation uses the method of \textit{hillclimbing} to find the solution. In a curve that relates gene expressions and affinity, hillclimbing will either move up or down by a small amount in the curve. The accumulation of hillclimbing will eventually lead to the finding of the hill's local maximum. Receptor editing meanwhile allows us to jump from one hill to another by bringing larger changes, enabling the search for the global maximum. See figure below.

\begin{figure}[H]
    \centering
    \includegraphics[width=200px]{img/affinity-curve.png}
    \caption{Antigen-binding site-affinity curve. Somatic hypermutation allows the individual at point A to search for the local maximum A' by slowly moving up the hill. Recepting editing allows A to jump to points B or C at different hills and search for the global maximum C'. \cite{Cas01}}
    \label{fig:affinity}
\end{figure}

\subsection{Artificial Neural Network}
Artificial neural network, or ANN, simulates mutual interactions between neurons of an animal's brain to solve complex calculations. Neurons are biological cells present in human brains, and they are composed of parts such as the nucleus, cell body, dendrites, axons, and synaptic terminals. Dendrites receive signals from other neurons, axon conveys signals through the neuron, and synaptic terminals release neurotransmitters and convey them to next neuron in the biological neural network.

\begin{figure}[H]
    \centering
    \includegraphics[height=150px]{img/biological-neuron.png}
    \caption{An image of a motor neuron. Inputs are received through the dendrites, and outputs are released by the axon terminals.}
    \label{fig:biological-neuron}
\end{figure}

An artificial neuron is inspired by such interaction of dendrites, an axon, and axon terminals in a biological neuron. Similar to how dendrites receive electric signals and transmit the signal through the axon once the membrane electric potential reaches its critical threshold, an artificial neuron takes in numerical inputs and uses weights and bias to calculate a sum. This sum is ``activated" by some activation function and outputs a final value. Typically, an artificial neural network connects multiple of these artificial neurons and comprises different layers. The three major layers of an artificial neural network are input, hidden, and output layers. Hidden layers can be a set of multiple layers that serve different functions of a problem.

\subsection{The Self-Driving Mechanism}

The simple self-driving car mechanism will take five numerical inputs from the car's raycast sensors that detect a wall. The angle of detection are the five inputs to the input layer, and they are combined with a feed-forward $5 \times 4 \times 3 \times 2$ neural network that determines the car's instantaneous tangential and radial acceleration. The neural network's weights are the genes of our genetic and clonal selection algorithm, whose fitness is eventually determined by the sigmoid function that  recombines the weights into a fitness value between 0 and 1.\footnote{The unity project used in this paper is a modified version of Samuel Arzt's Applying\_EANNs github repository.}

\begin{figure}[H]
\centering
\subfloat[An illustration of car raycast physics in action.]{%
\resizebox*{5cm}{!}{\includegraphics{img/raycast.png}}}\hspace{15pt}
\subfloat[An image of track used.]{%
\resizebox*{7.5cm}{!}{\includegraphics{img/track.png}}}
\label{self-driving}
\caption{Implementation of trivial self-driving methods.}
\end{figure}

\section{Pseudocodes}

\subsection{Roulette Wheel Selection}

\begin{algorithmic}[1]
    \Function{Roulette Wheel Selection}{$currentPopulation$}
    \State $totalSum \leftarrow 0$
    \For{$p_i \in curentPopulation$}
        \State $totalSum \leftarrow totalSum + p_{i}.fitness$
    \EndFor
    \While{$inntermediatePopulation.size < maxPopulationSize$}
        \State Generate $randomDouble$ between 0 and $totalSum$
        \State $prefixSum \leftarrow 0$
        \For{$p_i \in currentPopulation$}
            \State $prefixSum \leftarrow prefixSum + p_i.fitness$
            \If {$randomDouble < prefixSum$}
                \State Add $p_i$ to intermediatePopulation
                \State \textbf{Break}
            \EndIf
        \EndFor
    \EndWhile
    \State \textbf{Return} $intermediatePopulation$
	\EndFunction
\end{algorithmic}

\subsection{Rank Selection}

\begin{algorithmic}[1]
    \Function{Rank Selection}{$currentPopulation$}
        \For{$p_i \in currentPopulation$}
            \State Generate $randomDouble$ between 0 and 1
            \If {$randomDouble < p_i.selectionPercentage$}
                \State Add $p_i$ to intermediatePopulation
                \State \textbf{Break}
            \EndIf
        \EndFor
    \State \textbf{Return} $intermediatePopulation$
	\EndFunction
\end{algorithmic}

\subsection{Remainder Stochastic Sampling}

\begin{algorithmic}[1]
    \Function{Ramainder Stochastic Sampling}{$currentPopulation$}
    \State $averageFitness \leftarrow Sum(currentPopulation) \div populationSize$
    \For{$p_i \in currentPopulation$}
        \State $ratio \leftarrow p_i.fitness \div averageFitness$
        \If{$ratio < 1$}
        \State \textbf{Break}
        \Else
        \State Add $Floor(ratio)$ copies of $p_i$ to $intermediatePopulation$
        \EndIf
    \EndFor
    \For{$p_i \in currentPopulation$}
        \State $ratio \leftarrow p_i.fitness \div averageFitness$
        \State $remainder \leftarrow ratio - Floor(ratio)$
        \State Generate $randomDouble$ between 0 and 1
        \If {$randomDouble < remainder$}
        \State Add $p_i$ to $intermediatePopulation$
        \EndIf
    \EndFor
    \State \textbf{Return} $intermediatePopulation$
	\EndFunction
\end{algorithmic}

\subsection{Clonal Selection \cite{BroND}}

\begin{algorithmic}[1]
    \Function{Clonal Selection Algorithm}{$currentPopulation$}
    \While{$\neg StopCondition()$}
        \For{$p_{i} \in currentPopulation$}
            \State $CalculateFitness(p_{i})$
        \EndFor
        \State Select top few individuals and place in $selectPopulation$
        \For{$p_i \in selectPopulation$}
            \State $clonalPopulation \leftarrow Clone(p_i, numberOfClones)$
        \EndFor
        \For{$p_i \in clonalPopulation$}
            \State $Mutate(p_i, mutationType)$
            \State $CalculateFitness(p_i)$
        \EndFor
        \State Fill $clonalPopulation$ select individuals from $currenPopulation$
        \State Add random individuals to $clonalPopulation$ as desired
        \State $currentPopulation \leftarrow clonalPopulation$
        \State Increment generation count
    \EndWhile
	\EndFunction
\end{algorithmic}

\section{Experiment Design}
A car navigates around the track on roulette wheel selection, rank selection, remainder stochastic sampling, clonal selection with hypermutation, or clonal selection with receptor editing. Each trial is repeated five times. The first three trials for each algorithm records maximum and average fitness of every individuals in a population of each generation. After performing two additional trials, average generation count of the five trials is calculated.

To make the experiment as fair as possible, random generation for initial population is fixed by seeding the random number generator. This allows us to compare how each algorithm evolves at a different pace from equal starting point. Mutation rates are carefully tuned to ensure that the algorithms do not converge too fast, which can lead to finding the local maximum but no global maximum. It is important as well that mutation rates are not too high so that it does not end up becoming a virtual random search. For this problem, it was tuned to a 5\% mutation rate with amount by which numbers may mutate set at 2\%. The population size was set at 30.

\section{Data}

\subsection{Genetic Algorithms: Number of Generations Taken to Complete the Track}
\begin{table}[H]
\centering
\begin{tabular}{c|c|c|c}
Trial   & Roulette Wheel & Rank Selection & Remainder Stochastic Sampling \\ \hline
1       & 22             & 11             & 10                   \\
2       & 38             & 25             & 36                   \\
3       & 32             & 7              & 26                   \\
4       & 21             & 20             & 21                   \\
5       & 18             & 24             & 32                   \\ \hline
Average & 26.2           & 17.4           & 25
\end{tabular}
\end{table}

\subsection{Clonal Selection Algorithm: Number of Generations Taken to Complete the Track}
\begin{table}[H]
\centering
\begin{tabular}{c|c|c}
Trial   & Hypermutation & Receptor Editing \\ \hline
1       & 10            & 17               \\
2       & 29            & 18               \\
3       & 9             & 16               \\
4       & 8             & 16               \\
5       & 22            & 18               \\ \hline
Average & 15.6          & 17
\end{tabular}
\end{table}

\subsection{Genetic Algorithm: Graphs Relating Maximum and Average Fitness Over Generations}

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/Roulette Wheel Selection_ Generation vs Fitness.png}
    \label{fig:roulette-graph}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/Rank Selection_ Generation vs Fitness.png}
    \label{fig:rank-graph}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/Remainder Stochastic Sampling_ Generation vs Fitness.png}
    \label{fig:remainder-graph}
\end{figure}

\subsection{Clonal Selection Algorithm: Graphs Relating Maximum and Average Fitness Over Generations}

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/Hypermutation Clonal Selection_ Generation vs Fitness.png}
    \label{fig:hypermutation-graph}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=300px]{img/Receptor Editing Clonal Selection_ Generation vs Fitness.png}
    \label{fig:receptor-graph}
\end{figure}

\section{Analysis}

Table 5.1 and 5.2 indicate that rank selection and clonal selection with hypermutation and receptor editing had an average of 17.4, 15.6, and 17 generations, respectively. These averages are significantly less than that of roulette wheel selection and remainder stochastic sampling, which were 26.2 and 25 generations, respectively. One possible explanation for this phenomenon is that top individuals in rank selection and clonal selection have fixed probability of selection whereas roulette wheel selection and remainder stochastic sampling have probability dependent on the fitness of every individual in a population. In other words, selection is absolute in rank and clonal selection and relative in roulette wheel and remainder stochastic sampling. The least-fit individuals in a population are often values random and futile for evolution, and they can lower the chance of selection for top individuals and reapportion that chance to the selection of lesser-fit individuals. Meanwhile, rank selection has a fixed chance of selection for the best-fit individual, which was set at 40\% in this experiment. Clonal selection always selects a certain number of best-fit individuals. This fixed chance independent of lesser-fit individuals' influence ensures elitism and allows the population to evolve from the best ones.

\subsection{Limitations}
One thing to note is that the graphs indicate the evolution of the best-fit individual plateauing around 0.86 until it suddenly discovers an individual of fitness 1.0, which means that the car has completed the track. It was observed that the cars were continuously struggling to successfully maneuver through the sharp curve near the end of the track, which was due to the failure to slow down near the curve and make the turn without collisions. It is rather odd that the raycast physics and the neural network failed to adjust the direction and magnitude of acceleration according to distance and angles. All of the algorithms evolved a gene that reaches the plateau fairly quickly in less than ten generations. It seems more of a matter of suddenly finding a gene that can maneuver through the curve. However, the average fitness shows a steady increase over each generations, which indicates that population is evolving as a whole every time.

\subsection{Somatic Hypermutation vs Receptor Editing}
The average number of generations of each method were similar at 15.6 and 17 generations. However, trials involving hypermutation has a more sporadic set of data while receptor editing has a data set with very low deviation hovering around 17. The best trial for hypermutation took merely 8 generations, but the worst trial took 29. This indicates that receptor editing is more precise and hypermutation is more random. Receptor editing may also hold advantage in a scenario where the fitness of best-individual plateaus because it can quickly jump to the new hills (see Section 2.2) and find a new maximum.

\section{Conclusion and Implication}

\subsection{Benefits of Clonal Selection Algorithm}
The most obvious benefit of an evolutionary algorithm is that it can solve problems that traditional sequential algorithms cannot solve. While standard algorithms may take years and years to solve a NP-complete problem, clonal selection algorithm can quickly discover an approximate solution using nifty evolutionary method. For self-driving, it is nearly impossible to code an algorithm for millions of cases that vehicles may encounter on road. The roads are extremely unpredictable; the cars do not know where and how the curves will turn or when something will suddenly appear. Computer vision can aid this process, but for simple navigation purposes, it will be extremely difficult to code a non-evolutionary algorithm. Clonal selection algorithm was also notably faster than roulette wheel selection and remainder stochastic sampling, which demonstrates the potential of improvements that can be made in this field. Furthermore, its application has immense potential in machine learning as it mimics how our immune system learns through the encounters with pathogens and evolve to ensure the strength of our secondary immune response.

\subsection{Drawbacks of Clonal Selection Algorithm}
The fundamental problem is the heuristic nature of the algorithm. It may take a very long time to discover a solution or in extremely unlucky circumstances, it may not be able to discover one. This problem becomes more apparent when there is a plateau in the generation-fitness curve because finding a solution then becomes a problem of which individual can mutate into a better individual faster. However, this issue is rather trivial because evolution stems from the idea of natural selection where the best-fit individual survives. The best-fit individual sort of acts as a ``pivot" that prevents the algorithm from turning into a random search, and the drawback is further minimized by ensuring elitism. However, this also requires the meticulous tuning of numbers such as mutation or crossover rates so that the algorithm can converge efficiently.

Most of the drawbacks stem from having suboptimal number of hidden layers, mutation rates, population size, and so on. These values are called hyperparameters because they are set by a user prior to an experiment, unlike parameters such as neural network weights that are learned during the machine learning process. Through hyperparameter optimization techniques, we can minimize the loss function and derive an optimal tuple of hyperparameters for the problem.

\begin{thebibliography}{99}

\bibitem{BroND}
Brownlee, J. (n.d.). Clonal selection algorithm. Retrieved March 17, 2020, from Clever Algorithms: Nature-Inspired Programming Recipes website: http://www.cleveralgor ithms.com/nature-inspired/immune/clonal\_selection\_algorithm.html

\bibitem{Bus01}
Busetti, F. (2001, May). Genetic algorithms overview. Retrieved from https://ww w.researchgate.net/profile/Franco\_Busetti/publication/2383112\_Genetic\_Algorithms\_Over view/links/5c0e66e592851c39ebe262ca/Genetic-Algorithms-Overview.pdf

\bibitem{Cas01}
Castro, L., \& Zuben, F. (2001, February). The clonal selection algorithm with engineering applications. Retrieved from https://www.researchgate.net/publication/24686 77\_The\_Clonal\_Selection\_Algorithm\_with\_Engineering\_Applications

\bibitem{CloND}
Clonal selection theory [Image]. (n.d.). Retrieved from https://www.britannica.com/ science/clonal-selection-theory/images-videos

\bibitem{Dai07}
Dai, H., Jianchen, Z., \& Zheng, T. (2007, December). An improved clonal selection algorithm and its application to traveling salesman problems [Image]. Retrieved from https://www.researchgate.net/figure/The-clonal-selection-principle\_fig1\_31492211

\bibitem{Cas00}
De Castro, Leandro \& José, Fernando \& von Zuben, Antonio Augusto. (2000). Artificial Immune Systems: Part I-Basic Theory and Applications.

\bibitem{Obi98}
Obitko, M. (1998). Selection. Retrieved March 17, 2020, from Introduction to Genetic Algorithms website: https://obitko.com/tutorials/genetic-algorithms/selection.php

\bibitem{UoA00}
University of Arizona. (2000, November 10). Immunology problem set. Retrieved March 17, 2020, from The Biology Project website: http://www.biology.arizona.edu/immunology /tutorials/immunology/09t.html

\end{thebibliography}

\end{document}


\documentclass[../notes.tex]{subfiles}

\pagestyle{main}
\renewcommand{\chaptermark}[1]{\markboth{\chaptername\ \thechapter\ (#1)}{}}
\setcounter{chapter}{32}

\begin{document}

\chapter{The Nature and Propagation of Light}
\section{Light Wave Terminology and Basics}
\begin{itemize}
    \item \marginnote{8/10:}\textbf{Geometric optics}: The study of situations in which EM radiation interacts with objects (possibly with holes) such that $\lambda<<\text{size of obstacles, holes}$.
    \item \textbf{Physical optics}: The study of situations in which EM radiation interacts with objects (possibly with holes) such that $\lambda\approx\text{size of obstacles, holes}$.
    \item \textbf{Ray}: An imaginary line, perpendicular to the wave fronts, that indicates the direction of propagation.
    \item Any time a wave hits a medium, you get reflection and transmission.
    \item \textbf{Huygens principle}: All points on a wavefront act as point sources of spherical wavelets. After a time $\Delta t$, the new position of the wavefronts is the surface of tangency of the wavelets.
    \begin{figure}[h!]
        \centering
        \begin{tikzpicture}[
            every node/.append style={black}
        ]
            \footnotesize
            \draw [blx,thick] (0,0) -- ++(0,3);
            \draw [blx,semithick,densely dashed]
                (0,0.5) circle (4mm)
                (0,1.5) circle (4mm)
                (0,2.5) circle (4mm)
                (0,0.5) -- node[below,xshift=-2pt]{$r$} ++(135:0.4)
                (0,1.5) -- node[below,xshift=-2pt]{$r$} ++(135:0.4)
                (0,2.5) -- node[below,xshift=-2pt]{$r$} ++(135:0.4)
            ;
            \fill [blx]
                (0,0.5) circle (1.5pt)
                (0,1.5) circle (1.5pt)
                (0,2.5) circle (1.5pt)
            ;

            \draw [blx,thick,dash dot] (0.4,-0.5) -- ++(0,4);
            \node [align=center] at (1.5,3) {new\\wavefront}
                edge [out=-90,in=0,->,shorten >=1pt] (0.4,2)
            ;
        \end{tikzpicture}
        \caption{Huygens principle.}
        \label{fig:HuygensPrinciple}
    \end{figure}
    \begin{itemize}
        \item The radius $r$ of the wavelets, in terms of the time $\Delta t$ from their creation, is $r=c\Delta t$.
    \end{itemize}
    \item Light rays hitting a surface (see Figure \ref{fig:reflectingWaves}).
    \begin{figure}[h!]
        \centering
        \begin{subfigure}[b]{0.45\linewidth}
            \centering
            \begin{tikzpicture}
                \footnotesize
                \draw [semithick] (0,0) -- (5,0) coordinate (s) node[right]{Surface};
                \draw [dashed] (3.4,0) coordinate (P) -- ++(0,0.9) coordinate (Q);

                \draw [blx,thick] (0.2,0.8) -- ++(2.6,1.3);
                \draw [blx,thick] (0.45,0.3) -- ++(2.6,1.3);
                \draw [blx,thick] (1.1,0) coordinate (a) -- ++(2.2,1.1) coordinate (b);

                \draw [rex,thick,postaction={decorate},decoration={markings,mark=at position 0.45 with \arrowreversed{latex}}] (3.4,0) -- ++(-1.2,2.4) coordinate (R);

                \pic [draw,angle eccentricity=1.5,pic text={$\theta_1$},pic text options={yshift=-0.5pt}] {angle=s--a--b};
                \pic [draw,angle eccentricity=1.5,pic text={$\theta_1$},pic text options={xshift=0.5pt}] {angle=Q--P--R};
            \end{tikzpicture}
            \caption{Waves hit a surface.}
            \label{fig:reflectingWavesa}
        \end{subfigure}
        \begin{subfigure}[b]{0.45\linewidth}
            \centering
            \begin{tikzpicture}[
                every node/.append style={black}
            ]
                \footnotesize
                \draw [semithick] (0,0) coordinate (S) -- (5,0) coordinate (s) node[right]{Surface};

                \draw [blx,thick]
                    (1.1,0) coordinate (a) -- ++(4.4,2.2) coordinate (b)
                    (a) node[below left]{$A$} -- node[left]{$r$} ++(67.5:0.9) node[above right]{$B$}
                    (3.1,0) node[below right]{$D$} -- node[right]{$r$} ++(112.5:0.9) node[above left]{$C$}
                ;
                \draw [blx,thick,densely dashed]
                    (a) ++(9mm,0) arc[start angle=0,end angle=180,radius=9mm] ++(9mm,0)
                    ++(2,1) circle (9mm)
                    ++(2,1) circle (9mm)
                ;
                \fill [blx]
                    (a) circle (1.5pt)
                    ++(2,1) circle (1.5pt)
                    ++(2,1) circle (1.5pt)
                ;
                \draw [blx,thick,dash dot]
                    (3.1,0) -- ++(3,1.5)
                    (3.1,0) coordinate (A) -- ++(-2.4,1.2) coordinate (B)
                ;

                \pic [draw,angle eccentricity=1.5,pic text={$\theta_1$},pic text options={yshift=-0.5pt}] {angle=s--a--b};
                \pic [draw,angle eccentricity=1.5,pic text={$\theta_2$},pic text options={yshift=-0.5pt}] {angle=B--A--S};
            \end{tikzpicture}
            \caption{Invoking Huygens principle.}
            \label{fig:reflectingWavesb}
        \end{subfigure}\\[1em]
        \begin{subfigure}[b]{0.45\linewidth}
            \centering
            \begin{tikzpicture}
                \footnotesize
                \begin{scope}
                    \draw [blx,thick]
                        (0,0) coordinate (a) node[below left,black]{$A$}
                        -- node[left,black]{$c\Delta t$} ++(67.5:0.9) coordinate (b) node[above,black]{$B$}
                        -- (2,0) coordinate (c) node[below right,black]{$D$}
                        -- node{$|$} cycle
                    ;
                    \pic [draw,angle eccentricity=1.5,pic text={$\theta_2$},pic text options={yshift=-0.5pt}] {angle=b--c--a};
                    \pic [draw,angle radius=2mm] {right angle=a--b--c};
                \end{scope}
                \begin{scope}[xscale=-1,xshift=-6cm]
                    \draw [blx,thick]
                        (0,0) coordinate (A) node[below right,black]{$D$}
                        -- node[right,black]{$c\Delta t$} ++(67.5:0.9) coordinate (B) node[above,black]{$C$}
                        -- (2,0) coordinate (C) node[below left,black]{$A$}
                        -- node{$|$} cycle
                    ;
                    \pic [draw,angle eccentricity=1.5,pic text={$\theta_1$},pic text options={yshift=-0.5pt}] {angle=A--C--B};
                    \pic [draw,angle radius=2mm] {right angle=A--B--C};
                \end{scope}
            \end{tikzpicture}
            \caption{Proving identical triangles.}
            \label{fig:reflectingWavesc}
        \end{subfigure}
        \caption{Properties of reflecting waves.}
        \label{fig:reflectingWaves}
    \end{figure}
    \begin{itemize}
        \item Since the triangles containing the \textbf{angle of incidence} and the \textbf{angle of reflection} ($\triangle ACD$ and $\triangle ABD$, respectively) share $\overline{AD}$, $r$, and a right angle (see Figure \ref{fig:reflectingWavesc}), we have that they are identical.
        \item Thus, $\theta_1=\theta_2$.
        \item Since light rays have a constant phase offset (specifically, $\ang{90}$) from light waves, it follows that light rays also reflect off of surfaces with their original angle of incidence.
    \end{itemize}
    \item \textbf{Angle of incidence}: The angle with which wavefronts hit a surface, or the angle a light ray makes with a normal to a surface.
    \begin{itemize}
        \item The quantity $\theta_1$ in Figure \ref{fig:reflectingWaves}.
    \end{itemize}
    \item \textbf{Angle of reflection}: The angle with which the reflected wavefront intersects with a surface.
    \begin{itemize}
        \item The quantity $\theta_2$ in Figure \ref{fig:reflectingWavesb}.
    \end{itemize}
    \item \textbf{Law of reflection}: The principle that $\theta_1=\theta_2$.
\end{itemize}

\section{Reflection}
\begin{itemize}
    \item Corner reflector:
    \begin{figure}[h!]
        \centering
        \begin{tikzpicture}
            \footnotesize
            \draw [semithick] (0,2) coordinate (m1) node[above]{$M$} -- (0,0) coordinate (m2) -- (5,0) coordinate (m3) node[right]{$M$};
            \draw [very thin,dashed]
                (-0.8,0.9) -- ++(1.6,0) coordinate (n1)
                (1.8,-0.8) -- ++(0,1.6) coordinate (n2)
            ;

            \draw [
                rex,thick,postaction={decorate},
                decoration={
                    markings,
                    mark=at position 0.1  with \arrow{latex},
                    mark=at position 0.45 with \arrow{latex},
                    mark=at position 0.9  with \arrow{latex}
                }
            ]
                (2,1.9) coordinate (A)
                -- ++(-2,-1) coordinate (B)
                -- ++(1.8,-0.9) coordinate (C)
                -- ++(3,1.5) coordinate (D)
            ;

            \pic [draw,angle radius=2mm] {right angle=m1--m2--m3};
            \pic [draw,angle eccentricity=1.3,pic text={$\theta$}] {angle=n1--B--A};
            \pic [draw,angle eccentricity=1.3,pic text={$\theta$}] {angle=C--B--n1};
            \pic [draw,angle eccentricity=1.3,pic text={$\theta$}] {angle=B--C--m2};
            \pic [draw,angle eccentricity=1.3,pic text={$\ang{90}$\,-\,$\theta$},pic text options={xshift=-1mm,yshift=1mm}] {angle=n2--C--B};
            \pic [draw,angle eccentricity=1.3,pic text={$\ang{90}$\,-\,$\theta$},pic text options={xshift=1mm,yshift=1mm}] {angle=D--C--n2};
            \pic [draw,angle eccentricity=1.3,pic text={$\theta$}] {angle=m3--C--D};
        \end{tikzpicture}
        \caption{Corner reflector.}
        \label{fig:cornerReflector}
    \end{figure}
    \begin{itemize}
        \item The law of reflection implies that the ray entering at an angle $\theta$ will exit at the same angle, just displaced a bit.
    \end{itemize}
    \item If you don't want radar to detect planes:
    \begin{itemize}
        \item Eliminate right angles (they're corner reflectors).
        \item Make the surface something that absorbs radiation (hi tech ceramic).
    \end{itemize}
    \item Mirrors create images; tie to projective geometry.
    \begin{itemize}
        \item Seeing an object in the mirror is equivalent to seeing it as far behind the mirror as the thing is in front of the mirror.
        \item This is \textbf{ray tracing}!
    \end{itemize}
    \item \textbf{Virtual image}: An image that is created by light rays that don't really exist, i.e., projected light rays.
    \item Mirrors don't reverse left and right; they reverse front and back.
\end{itemize}

\section{Refraction}
\begin{itemize}
    \item Light waves and glass.
    \begin{itemize}
        \item Light waves enter glass, shake the atoms therein, and then those atoms emit their own light.
        \item However, the light emitted by the atoms goes in all directions.
        \item When you sum up all the secondary sources in a horrible integral, you end up with an effective wave propagating to the right, but at a speed less than the speed of light.
    \end{itemize}
    \item \textbf{Index of refraction}: The quotient of the speed of light in a vacuum and the speed of light in a particular medium. \emph{Denoted by} $\bm{n}$.
    \begin{equation*}
        n = \frac{c}{v}
    \end{equation*}
    \begin{itemize}
        \item Since light can never travel faster than the speed of light, $n\geq 1$.
        \item Some common $n$ values:
        \begin{itemize}
            \item $n_\text{water}\approx 1.33$.
            \item $n_\text{glass}\approx 1.5$.
            \item $n_\text{diamond}\approx 2.5$.
            \item $n_\text{air}\approx 1.003\approx 1$.
        \end{itemize}
    \end{itemize}
    \item In materials, $v$ changes and $f$ remains constant, so $\lambda$ changes.
    \begin{itemize}
        \item $c=f\lambda$ and $v=f\lambda'$ imply that
        \begin{equation*}
            \lambda' = \lambda\cdot\frac{v}{c} = \frac{\lambda}{n}
        \end{equation*}
    \end{itemize}
    \item In a surface, the wavefront gets bent.
    \begin{figure}[h!]
        \centering
        \begin{tikzpicture}[
            every node/.append style={black}
        ]
            \footnotesize
            \draw [semithick] (0,0) coordinate (S) -- (5,0) coordinate (s) node[right]{S};

            \draw [blx,thick]
                (1.1,0) coordinate (a) -- ++(4.4,2.2) coordinate (b)
            ;
            \draw [blx,thick,densely dashed]
                (a) ++(4.5mm,0) arc[start angle=0,end angle=-180,radius=4.5mm] ++(4.5mm,0)
                ++(2,1) circle (9mm)
                ++(2,1) circle (9mm)
            ;
            \fill [blx]
                (a) circle (1.5pt)
                ++(2,1) circle (1.5pt)
                ++(2,1) circle (1.5pt)
            ;
            \draw [blx,thick,dash dot]
                (3.1,0) -- ++(3,1.5)
                (3.1,0) coordinate (A) -- ++(-2.4,-0.6) coordinate (B)
            ;

            \pic [draw,angle eccentricity=1.5,pic text={$\theta_1$},pic text options={yshift=-0.5pt}] {angle=s--a--b};
            \pic [draw,angle radius=1.1cm,angle eccentricity=1.2,pic text={$\theta_2$},pic text options={yshift=0.2pt}] {angle=S--A--B};
        \end{tikzpicture}
        \caption{Properties of refracting waves.}
        \label{fig:refractingWaves}
    \end{figure}
    \begin{itemize}
        \item Similarly, the ray gets bent.
        \item When looking from air into a different surface, $\theta_2<\theta_1$.
        \item Explains why when you reach for something in water, it appears closer and in a different spot --- you're reaching for the virtual image!
    \end{itemize}
    \item \textbf{Angle of refraction}: The angle with which the refracted wavefront intersects with a surface.
    \begin{itemize}
        \item The quantity $\theta_2$ in Figure \ref{fig:refractingWaves}.
    \end{itemize}
    \item \textbf{Snell's law}: The formula $n_1\sin\theta_1=n_2\sin\theta_2$. \emph{Also known as} \textbf{law of refraction}.
    \item \textbf{Critical angle}: The angle of incidence such that light will be refracted at $\ang{90}$.
    \begin{itemize}
        \item If $\theta_\text{inc}>\theta_\text{crit}$, you only get reflection!
        \item From Snell's Law,
        \begin{equation*}
            \theta_\text{crit} = \sin^{-1}(n_2/n_1)
        \end{equation*}
    \end{itemize}
    \item \textbf{Total Internal Reflection}: Conditions such that all light is reflected and none is refracted. \emph{Also known as} \textbf{TIR}.
    \begin{itemize}
        \item Only a possibility when the light wave is moving to a medium with a lower index of refraction.
    \end{itemize}
    \item If you have a glass rod and you put light in at one end at an angle greater than the critical angle, it will be trapped and can only come out at the other end.
    \begin{itemize}
        \item Total internal reflection allows us to redirect light however we want.
        \item Light "flowing" through this construction is analogous to water flowing through a pipe.
        \item As long as the angle we bend the light pipe at isn't too sharp, it will stay trapped in the light pipe.
        \item In a well-designed light pipe, you will loose very little intensity.
        \item This is the principal behind fiber optics.
        \item You lose current in a wire due to resistance, but you don't lose much intensity in a light pipe.
    \end{itemize}
\end{itemize}

\section{Office Hours (Pandey)}
\begin{itemize}
    \item \marginnote{8/11:}Can you explain the upwards and downwards displacement currents in Figure \ref{fig:planarWavefront}?
    \begin{itemize}
        \item Not really.
    \end{itemize}
\end{itemize}

\section{Dispersion}
\begin{itemize}
    \item \marginnote{8/12:}The way that atoms shake in a material with light passing through it depends on the frequency of the light.
    \item Thus, $n=n(f)$.
    \begin{itemize}
        \item For example, $n_\text{blue}\neq n_\text{red}$.
    \end{itemize}
    \item In general, $n$ increases as $f$ increases (or $\lambda$ decreases).
    \begin{itemize}
        \item Thus, $n_\text{blue}>n_\text{red}$, for example.
    \end{itemize}
    \item Thus, if you have light made up of all kinds of different colors, every color will travel through the material at a different speed.
    \begin{figure}[h!]
        \centering
        \begin{tikzpicture}[
            every node/.style={black}
        ]
            \footnotesize
            \draw [yellow,thick] (-120:1) -- ++(-2,-0.5) node[left]{white};
            \draw [red,thick] (-120:1) -- (-60:0.9) -- ++(2,-0.3) node[right]{red};
            \draw [blue,thick] (-120:1) -- (-60:1.1) -- ++(2,-0.7) node[right]{blue};

            \draw [thick] (0,0) -- (-60:2) -- (-120:2) -- cycle;
        \end{tikzpicture}
        \caption{A prism dispersing light.}
        \label{fig:prism}
    \end{figure}
    \begin{itemize}
        \item This is the principle behind a \textbf{prism}.
    \end{itemize}
    \item \textbf{Dispersion}: The dependence of the index of refraction on the frequency of light.
    \item When light passes through water droplets in the sky, it gets refracted by different amounts, depending on the color of the light. This is what creates a rainbow!
    \item \textbf{Fermat's Principle}: Light follows the path that takes the least time.
    \item Light traveling between $A$ and $B$ via a mirror.
    \begin{itemize}
        \item In this case, least time implies shortest distance.
        \item And indeed, the path with the shortest distance via the mirror is the one where $\theta_1=\theta_2$, as can be proven with optimization and calculus.
    \end{itemize}
    \item Light traveling between $A$ and $B$ via a change of medium.
    \begin{itemize}
        \item In this case, shortest distance \emph{does not} imply least time.
        \item If $n_1<n_2$, you can travel faster in $n_1$ then you can in $n_2$, so you want to minimize the time you spend in $n_2$ without going too far out of your way.
        \item It follows that the path with the least time will be the one given by Snell's law.
    \end{itemize}
    \item Why does light take the path of least time?
    \begin{itemize}
        \item Fermat says, "because it does."
        \item Quantum physics has a more bizarre reason for this.
    \end{itemize}
    \item How does light know which path to take?
    \begin{itemize}
        \item It doesn't --- it tries all paths and only one succeeds.
    \end{itemize}
\end{itemize}

\section{Polarization}
\begin{itemize}
    \item \marginnote{8/13:}Midterm on Tuesday.
    \begin{itemize}
        \item 80 minutes, more questions.
        \item HW 1-3.
        \item Chapter 15-16, 33-34.
    \end{itemize}
    \item \textbf{Polarized} (wave) \textbf{in the $\bm{xy}$-plane}: A wave where both the medium and displacement are entirely contained in the $xy$-plane. \emph{Also known as} \textbf{vertically polarized} (wave).
    \item If you shake a string up and down, you will create a polarized wave in the $xy$-plane.
    \item If you shake electrons up and down, you will create a vertically polarized EM wave, with $\vec{E}_\text{trans}$ entirely contained in the vertical plane.
    \begin{itemize}
        \item If you face said wave parallel to the propagation, you only see movement up and down.
    \end{itemize}
    \item A metal comb running vertically blocks vertically polarized microwaves since the microwaves lose energy creating currents in the metal.
    \begin{itemize}
        \item Works with microwaves since they're long, but wouldn't with visible light.
        \item However, a polaroid does work with the visible spectrum.
    \end{itemize}
    \item A polaroid filter is composed of long, stretched out organic molecules analogous to the teeth on the metal comb.
    \begin{itemize}
        \item If the \textbf{transmission axis} is vertical, all vertical waves get through.
        \item If it is horizontal, they all get blocked.
        \item If it is rotated at some angle $\phi$ from the vertical, decompose the radial vector into parallel and perpendicular components.
        \begin{itemize}
            \item The perpendicular component is totally blocked, and the parallel component is totally transmitted.
            \item It follows that $\vec{E}=\vec{E}_\parallel=\vec{E}_\text{polarized}\cos\phi$, so the wave amplitude of vertically polarized light decreases by $\cos\phi$.
        \end{itemize}
    \end{itemize}
    \item Since $I\propto A^2$, we get the following.
    \item \textbf{Law of Malus}: The following formula, where $I_2$ is the intensity of light that gets past the polaroid filter and $I_1$ is the initial intensity.
    \begin{equation*}
        I_2 = I_1\cos^2\phi
    \end{equation*}
    \item For unpolarized light, $I_\text{trans}=I_\text{unpol}\cos^2\phi$, but for light at every angle $\phi$.
    \begin{itemize}
        \item Thus, if we average $\cos^2\phi$ over all $\phi$, we get that
        \begin{equation*}
            I_\text{trans} = \frac{1}{2}I_\text{unpol}
        \end{equation*}
        \item Note that the above result is for a \textbf{perfect filter}. In reality, stacking filters causes some additional loss of intensity.
    \end{itemize}
    \item If you put a two filters on top of each other at perpendicular angles, it will entirely eliminate the intensity.
    \begin{itemize}
        \item At some angle $\phi$ in between, you'll have a variable loss of intensity.
    \end{itemize}
    \item If you put a third filter at an angle between two perpendicular filters, you'll regain some intensity.
\end{itemize}

\end{document}


\section{Requirements}
No formal background in programming is necessary. A completed undergraduate course in mathematics
(linear algebra, basic calculus, combinatorics and logic) would help.

You should bring a laptop on which latest version of Anaconda is installed.
This can be downloaded for free https://www.anaconda.com/distribution/
Make sure that you download the file for Python 3.7 and for your own Operating System.

Without the Anaconda software, you will not be able to follow the class.


\documentclass{article}

\usepackage{graphicx,tikz}
\usepackage{fullpage}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{algorithm}
\usepackage{algorithmicx}
\usepackage{algpseudocode}
\usepackage{booktabs}
%\usepackage{color}

%\renewcommand{\baselinestretch}{0.95}
%\setlength{\textfloatsep}{0.1cm}
%\setlength{\abovecaptionskip}{0.1cm}

\begin{document}

\title{\Large\bf New Sampling-based Approximation Algorithms for Frequent Itemsets Mining}
\author{Shiyu Ji, Kun Wan\\ \{shiyu, kun\}@cs.ucsb.edu}
\date{}
\maketitle

\newtheorem{definition}{Definition}
\theoremstyle{definition}
\newtheorem{theorem}{Theorem}
\theoremstyle{plain}
\newtheorem{lemma}{Lemma}
\theoremstyle{plain}
\newtheorem{corollary}{Corollary}
\theoremstyle{plain}

\section{Problem Statement}
\subsection{Introduction}
Frequent Itemsets (FI) mining has been popular in research recently \cite{AIS93, HCX07, RU15}. The goal of FI mining is to find out the items that most frequently appear in the observed transactions, e.g., the researchers who are the most prolific in writing papers with others, the patterns that appear frequently in a long piece of genetic code, etc.

In the era of big data, to compute the exact frequencies can be very time consuming. Thus in many cases approximated values are also acceptable \cite{RU15}.
For FI mining in large scaled transactional dataset, we often take samplings on the transactions, and compute the frequencies of the itemsets among the sampled transactions as approximated results of their true frequencies among all the transactions. Usually the sampling size is much less than the scale of all the transactions, and the approximations can achieve acceptable precision. Thus FI approximation can be useful in practice.

The state-of-art progressive sampling based FI approximation algorithms \cite{RU15} need an upper bound of the approximation error for the worst case, i.e., the maximum error the algorithm can generate among all the items. The algorithms keep taking new samples until the upper bound is less than the acceptable threshold. Hence how to bound the maximum error as tightly as possible is an interesting problem. The current bounds use some results of Rademacher average in statistical learning theory \cite{Vap98,Vap13,BBL04,BBL05}.
However, we find that based on the idea given by \cite{BBL04}, we can develop a new upper bound \emph{without} Rademacher average.
We also find that this new bound is asymptotically tighter than the existing bounds, i.e., given the chosen samples, as the allowed error probability approaches zero, the new bound is roughly only half of the existing ones. This implies that by using the new bound, a progressive sampling based FI approximation algorithm can reach the guaranteed accuracy with much fewer samples. We also notice that there is no parameter in the new bound that needs to be progressively computed. Hence the sample size that will guarantee the worst error can be precomputed.

Based on the similar idea, we also consider the top-$k$ FI mining problem, which seeks for the $k$ most frequent itemsets in the observed ones. We need to decide when the sampling should stop. The number of the samples transactions is enough if the worst-case error upper bound is less than the frequency gap between the $k$-th and the $(k+1)$-th most frequent itemsets. Hence we propose a progressive approximation algorithm to address the top-$k$ FI mining problem.

{\bf Our Plan}. We plan to give a new worst-case error upper bound that is asymptotically tighter than the state-of-art bounds, and propose an approximation algorithm which can guarantee the worst-case error upper bound with precomputed sample size. We will also give a progressive sampling algorithm to find the top-$k$ most frequent itemsets. We will need experiments on real data sets to evaluate the performance of our algorithms, and compare our results with the state-of-art approximation algorithms.

\subsection{Related Works}
\label{sec:rw}
Sampling-based frequent itemset approximation has been studied extensively by the researchers. The first works on this problem used heuristic methods to progressively approximate the frequencies \cite{CHS02,CCY05,Parth02}. There were no guarantee on the worst-case error upper bound. To fix this, Riondato and Upfal were the first to propose FI approximation algorithms that could guarantee the worst error bounds by using results of Vapnik-Chervonenkis (VC) dimension \cite{RU12,RU14} and Rademacher average \cite{RU15}. Note that in statistical learning theory VC dimension and Rademacher average are usually used to address the worst-case error upper bound for \emph{infinite case}, i.e., the number of possible functions in the learning model is infinite \cite{BBL04}. However in the case of FI mining, since there are only \emph{finite} itemsets, it is possible to develop bounds without VC dimension or Rademacher average \cite{BBL04}. In this paper we apply this idea on FI mining problem. Riondato et al. also considered using parallelism in FI mining \cite{RDF12}, which is an orthogonal topic to sampling-based FI approximation.

In practice we are often only interested in the most frequent itemsets. Thus top-$k$ FI mining is a popular research topic with many research works \cite{PRU10,SW02,RU15,RV14}.

\section{Data Sets}
For consistency, we choose FIMI'03 data repository \cite{GZ04}, the real-world data set from \cite{RU15} (the data repository is available at \texttt{http://fimi.ua.ac.be/data/}). The item and transaction data sizes of the FIMI datasets are given in Table \ref{tab:data}. We will use these data sets to evaluate the samples sizes and worst case errors of our algorithms, and compare our results with the state-of-art algorithms.

\begin{table}[!t]
\centering
\begin{tabular}{l | r r}
\specialrule{1pt}{1pt}{1pt}
Name & No. of Transactions & No. of items \\
\hline
accidents & 340183 & 468 \\
chess & 3196 & 75 \\
connect & 67557 & 129 \\
kosarak & 990002 & 41270 \\
mushroom & 8124 & 119 \\
pumsb & 49046 & 7116 \\
pumsb star & 49046 &  7116 \\
retail  & 88162 & 16470 \\
\specialrule{1pt}{1pt}{1pt}

\end{tabular}
\caption{Dataset characteristics}
\label{tab:data}
\end{table}

\section{Deliverables}
We plan to deliver the documents as follows:
\begin{itemize}
\item A report or paper about our new bounds and algorithms, and experiment results.
\item Source code of our experiments on real data sets.
\end{itemize}

\section{Milestones}
As our plan, we list the possible milestones in the future as follows:
\begin{itemize}
\item Establish a new worst-case error upper bound and do the proof work.
\item Propose sampling-based approximation algorithms, i.e., estimating frequency for each itemset, approximating the top-$k$ frequent itemsets, etc.
\item Evaluate our algorithms with real-world data sets, and compare our results with the existing works like \cite{RU15}.
\end{itemize}

%\bibliographystyle{./IEEEtran}
\bibliographystyle{plain}
\bibliography{./prop}

\end{document}


\documentclass{article}

%% preamble
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage{color}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{cancel}

\topmargin 0pt
\advance \topmargin by -\headheight
\advance \topmargin by -\headsep
\textheight 9.in
\oddsidemargin 0pt
\evensidemargin \oddsidemargin
\marginparwidth 0.5in
\textwidth 6.5in
\newcommand{\myhrule}{ \begin{center}\rule{.9\linewidth}{.25mm}\end{center} }
\definecolor{darkgray}{rgb}{0.95,0.95,0.95}
\definecolor{heavygray}{rgb}{0.05,0.05,0.05}
\definecolor{foo}{rgb}{.8,0,.8}
\newcommand{\pad}{\vspace{8pt}\noindent}
\newcommand{\red}[1]{{\color{red}#1\color{black}}}
\newcommand{\myhref}[2]{\href{#1}{\color{foo}\underline{#2}\color{black}}}

\newcommand{\Mod}[1]{\ \mathrm{mod}\ #1}

\begin{document}

\title{CMDA 3634 Spring 2018 Homework 05}

%% change this to your name
\author{Vincent Zhao}
\vspace{-64pt}\maketitle
\begin{center}\underline{You must complete the following task by 5pm on Tuesday 04/24/18.}\end{center}
Your write up for this homework should be presented in a {\LaTeX} formatted PDF document. You may copy the \LaTeX{} used to prepare this report as follows

\begin{enumerate}
\item Click on this  \myhref{https://www.sharelatex.com/read/cyvqcnwmfnsc}{link}
\item Click on Menu/Copy Project.
\item Modify the HW05.tex document to respond to the following questions.
\item Remember: click the Recompile button to rebuild the document when you have made edits.
\item Remember: Change the author.

\end{enumerate}

\pad \emph{Each student} must individually upload the following files to the CMDA 3634 Canvas page at \myhref{https://canvas.vt.edu}{https://canvas.vt.edu}

\begin{enumerate}
\item \verb|firstnameLastnameHW05.tex| {\LaTeX} file.
\item Any figure files to be included by \verb|firstnameLastnameHW05.tex| file.
\item \verb|firstnameLastnameHW05.pdf| PDF file.
\end{enumerate}

\noindent In addition, all source code must be submitted to an online git repository as follows:
\begin{enumerate}
    \item While on the webpage for your git repository, go to Settings $\rightarrow$ Collaborators.
    \item Add nchalmer@vt.edu and zjiaqi@vt.edu as collaborators.
    \item Make a folder named HW05 in you repository and store all relevant source files to this assignment in this folder. Ensure your assignment files compile with {\verb make }.
\end{enumerate}

\pad You must complete this assignment on your own.

\vspace{16pt}
\begin{center}
\underline{\bf 100 points will be awarded for a successful completion.}
\vspace{8pt}\underline{\bf Extra credit will be awarded as appropriate.}
\end{center}

\newpage

\section*{ElGamal Public-key Cyptography}

Through the previous assignments, we've manged to develop a functional program which performs encryption and decryption of string messages using an $n$ bit ElGamal cryptographic system. We can set up the cryptographic system by finding the values of the public key $(p,g,h)$. We can cast characters in strings into elements of $\mathbb{Z}_p$ and encrypt them. Finally, we can decrypt elements of $\mathbb{Z}_p$ and cast them back to characters if we know the secret key $x$.

In the case where we imagined that we do not have access to the secret key $x$, we have used MPI and OpenMP to crack our ElGamal encryption in parallel using many CPU cores. Even with that level of parallelism, it is easy to see that our program can use $n$ bit encryption where $n$ is large enough that we still cannot find the secret key in a practical amount of time.

In this final assignment, let's turn our previous monolithic programs into something a bit more useful by separating the basic operations into their own programs which run and save their results in files. Once we've accomplished that, let's tackle our final hurdle and use GPU acceleration to enable us to break even the strongest encryption our program can produce, namely $n=31$ bit cryptographic systems.

\pad {\bf Part 1: Setup, Encrypt, and Decrypt}

In the HW05 folder of the instructor's github the {\verb makefile } has been altered so that four separate programs will be compiled when you type {\verb make } or {\verb make } {\verb all }. The names of the programs are {\verb setup }, {\verb encrypt }, {\verb decrypt }, and {\verb cudaDecrypt }. They can be compiled separately by typing {\verb make } {\verb setup }, {\verb make } {\verb encrypt }, etc. Note that the function {\verb cudaDecrypt } requires CUDA's {\verb nvcc } compiler, and therefore the New River cluster, to compile.

\vspace*{1em}
\noindent{\bf Q1}(10 points) The program in {\verb setup.c } is meant to perform the ElGamal cryptosystem setup routines and write the public key information to a file named {\verb public_key.txt }. Complete the program so that, after querying the user for a bit length $n$, it write the bit length $n$, prime $p$, generator $g$, and number $h$, to a file named {\verb public_key.txt }. For example, after running {\verb setup }, the file {\verb public_key.txt } could look something like,
\begin{verbatim}
    25
    30080879
    28192225
    28973487
\end{verbatim}
Which means $n=25$, $p=30080879$, $g=28192225$, and $h=28973487$.
\vspace*{1em}

\noindent{\bf ANSWER:} \\
\textit{See attached code on Canvas and GitHub.}

\vspace*{1em}
\noindent{\bf Q2}(20 points) The program in {\verb encrypt.c } is meant to input a message string from the user and use the public key information in the file {\verb public_key.txt } to encrypt the message. Edit the program so that it after reading the message as a string from the user the program reads the public key information from {\verb public_key.txt } and encrypts the message. Afterwards, write the the number of cyphertext pairs, and each cyphertext pair $(\hat{m},a)$, into a file called {\verb message.txt }. The contents of {\verb message.txt } should look something like,
\begin{verbatim}
    14
    28395666 9350445
    3562739 5794092
    10156865 16818778
    2858293 6363325
    29107964 21484289
    3274137 24856781
    19487449 15592980
    18341691 30012651
    15930624 18293246
    11025926 2833780
    3862999 23130298
    16935347 23161724
    21230787 9009798
    25866655 2102881
\end{verbatim}
which indicates that there are 14 cyphertext entries, $(\hat{m},a)_1 = (28395666,9350445)$, $(\hat{m},a)_2 = (3562739,$ $5794092)$, etc.
\vspace*{1em}

\noindent{\bf ANSWER:} \\
\textit{See attached code on Canvas and GitHub.}

\vspace*{1em}
\noindent{\bf Q3.1}(20 points) The program in {\verb decrypt.c } is meant to read the contents of {\verb public_key.txt } and {\verb message.txt } and decrypt the message. After reading the files, the program will ask the user for the secret key. Complete the program so that if the user inputs $x=0$ or an incorrect secret key, the program will attempt to break the encryption by searching for the correct secret key $x$. Once the correct secret key is found, decrypt the contents of {\verb message.txt } and print the message to the terminal.

\vspace*{1em}

\noindent{\bf ANSWER:} \\
\textit{See attached code on Canvas and GitHub.}

\vspace*{1em}
\noindent{\bf Q3.2}(5 points) Using the {\verb decrypt } function or one of your programs from previous assignments, give a rough estimate of how long you would expect a serial code to break an $n=31$ bit encryption. Be sure to explain how you arrived at your estimate.
\vspace*{1em}

\noindent{\bf ANSWER:} \\
Using the decrypt function on an $n=31$ bit encryption means brute-forcing a possible $2^{31}$ values for the secret key. I tried to crack an $n=31$ bit encryption on my own computer and left my program running for over an hour with no results to show. Testing the values that approach it, I saw that the runtime increased at an exponential rate from $n=20$ to $n=27$, with $n=27$ taking 1098 seconds to run (or 18 minutes). The throughput also decreased as the number of bits increased, with $n=27$ having a throughput of $90000$ values per second. A very rough estimate for how long the serial code would take to crack an $n=31$ bit encryption is $\frac{2^{31}}{90000} = 23,860$ seconds, or $6.6$ hours.

\pad {\bf Part 2: GPU Accelerated Decryption}
In this part, we will use Nvidia's CUDA programming language to add GPU acceleration to the {\verb decrypt } function. In the HW05 folder is the CUDA file {\verb cudaDecrypt.cu }. Begin this section by copying the contents of the {\verb main } function in your complete {\verb decrypt.c } file to to {\verb cudaDecrypt.cu }

\vspace*{1em}
\noindent{\bf Q4.1}(10 points) Examine the loop where we search for the secret key. Describe in words how/why we could perform this section of the program in a GPU kernel. What would the kernel require as input? How much output will the kernel produce, and how do we access it on the host? How much memory must we allocate on the device? How many threads per block would be reasonable? Why?
\vspace*{1em}

\noindent{\bf ANSWER:} \\
The loop for finding the secret key takes on a brute force approach and tries every value from $0$ to $p - 1$ until it finds the correct value. For large values of $p$, this approach will take a very long time to complete. We use a GPU kernel for this section of code in order to speed up the process.

First we allocate memory on both the host (the local CPU) and the device (the remote GPU). Then the data is initialized on the host. We transfer the data from the host over to the device, and then execute our kernels on the GPU which will run our program in parallel. Once we get results, we transfer that data over to the host and finish our program.

GPU kernels can help us save time by finding the result faster (since it can run in parallel). The kernel needs two pieces of information to execute: first the number of thread-blocks in the grid, and second the number of threads in each thread-block. The output can be accessed on the respective variables on the host after copying the memory from the device.

\vspace*{1em}
\noindent{\bf Q4.2}(35 points) Edit the {\verb cudaDecrypt.cu } file to implement the kernel you described in Q4.1, i.e. add a GPU kernel function which can find the secret key $x$. If any {\verb __device__ } functions are required add them to {\verb cudaDecrypt.cu } as well (you may need to rename them to not conflict with functions in {\verb functions.c }). Be sure to {\verb cudaMalloc } any arrays the kernel will need, and use {\verb cudaMemcpy } to transfer memory between host and device.
\vspace*{1em}

\noindent{\bf ANSWER:} \\
\textit{See attached code on Canvas and GitHub.}

\vspace*{1em}
\noindent{\bf Q4.3}(5 points) Experiment with decrypting several different $n$-bit encryptions. In general, how much faster (in time and throughput) is the GPU accelerated version of your decryption program? Assuming you could scale perfectly in parallel with MPI or OpenMP, how many CPU cores would you need in order to achieve the same performance as one of the P100 GPUs on the New River cluster?

\vspace*{1em}
\noindent{\bf Bonus:}(20 points) In the HW05 folder of the instructor's github there is a file named {\verb bonus_public_key.txt } and {\verb bonus_message.txt }. Decrypt the message and include a screenshot of the output here.
\vspace*{1em}

\noindent{\bf ANSWER:} \\
Bonus message is attached below! Really sad I missed out on the meeting!

    \begin{center}
        \includegraphics[width=0.8\textwidth]{bonus_message.PNG}
    \end{center}

%\myhrule

%\begin{thebibliography}{9}
%\bibitem{gol} Wikipedia -- description of the Game of Life
%\url{https://en.wikipedia.org/wiki/Conway's_Game_of_Life}

%\end{thebibliography}

\end{document}


\documentclass[24pt, a4]{article}
% To compile latex inside of vim
% ! clear; pdlatex name_of_file.tex or "%"

% \usepackage[utf8]{inputenc}
\usepackage[parfill]{parskip}
\usepackage{listings}
\usepackage{geometry}
\usepackage{multicol}

\title{Blind 75 Coding Questions}
\author{Mustafa Muhammad}
\date{27th December 2021}

\begin{document}

\maketitle

\begin{multicols}{2}
\textbf{Array}
\begin{enumerate}
    \item{Two Sum}
    \item{Best Time to Buy and Sell Stock}
    \item{Contains Duplicate}
    \item{Product of Array Except Self}
    \item{Maximum Subarray}
    \item{Maximum Product Subarray}
    \item{Find Minimum in Rotated Sorted Array}
    \item{Search in Rotated Sorted Array}
    \item{3 Sum}
    \item{Container With Most Water}
\end{enumerate}

\columnbreak
\textbf{Binary}
\begin{enumerate}
    \item{Sum of Two Integers}
    \item{Number of 1 Bits}
    \item{Counting Bits}
    \item{Missing Number}
    \item{Reverse Bits}
\end{enumerate}

\end{multicols}

\begin{multicols}{2}

\textbf{Dynamic Programming}
\begin{enumerate}
    \item{Climbing Stairs}
    \item{Coin Change}
    \item{Longest Increasing Subsequence}
    \item{Longest Common Subsequence}
    \item{Word Break Problem}
    \item{Combination Sum}
    \item{House Robber}
    \item{House Robber II}
    \item{Decode Ways}
    \item{Unique Paths}
    \item{Jump Game}
\end{enumerate}

\columnbreak

\textbf{Graph}
\begin{enumerate}
    \item{Clone Graph}
    \item{Course Schedule}
    \item{Pacific Atlantic Water Flow}
    \item{Number of Islands}
    \item{Longest Consecutive Sequence}
    \item{Alien Dictionary (Leetcode Premium)}
    \item{Graph Valid Tree (Leetcode Premium)}
    \item{Number of Connected Components in an Undirected Graph (Leetcode Premium)}
\end{enumerate}

\end{multicols}

\begin{multicols}{2}

\textbf{Interval}
\begin{enumerate}
    \item{Insert Interval}
    \item{Merge Intervals}
    \item{Non-overlapping Intervals}
    \item{Meeting Rooms (Leetcode Premium)}
    \item{Meeting Rooms II (Leetcode Premium)}
\end{enumerate}

\columnbreak

\textbf{LinkedList}
\begin{enumerate}
    \item{Reverse a Linked List}
    \item{Detect Cycle in a Linked List}
    \item{Merge Two Sorted Lists}
    \item{Merge K Sorted Lists}
    \item{Remove Nth Node From End Of List}
    \item{Reorder List}
\end{enumerate}

\end{multicols}

\begin{multicols}{2}

\textbf{Matrix}
\begin{enumerate}
    \item{Set Matrix Zeroes}
    \item{Spiral Matrix}
    \item{Rotate Image}
    \item{Word Search}
\end{enumerate}

\textbf{String}
\begin{enumerate}
    \item{Longest Substring Without Repeating Characters}
    \item{Longest Repeating Character Replacement}
    \item{Minimum Window Substring}
    \item{Valid Anagram}
    \item{Group Anagrams}
    \item{Valid Parentheses}
    \item{Valid Palindrome}
    \item{Longest Palindromic Substring}
    \item{Palindromic Substrings}
    \item{Encode and Decode Strings (Leetcode Premium)}
\end{enumerate}

\columnbreak

\textbf{Tree}
\begin{enumerate}
    \item{Maximum Depth of Binary Tree}
    \item{Same Tree}
    \item{Invert/Flip Binary Tree}
    \item{Binary Tree Maximum Path Sum}
    \item{Binary Tree Level Order Traversal}
    \item{Serialize and Deserialize Binary Tree}
    \item{Subtree of Another Tree}
    \item{Construct Binary Tree from Preorder and Inorder Traversal}
    \item{Validate Binary Search Tree}
    \item{Kth Smallest Element in a BST}
    \item{Lowest Common Ancestor of BST}
    \item{Implement Trie (Prefix Tree)}
    \item{Add and Search Word}
    \item{Word Search II}
\end{enumerate}

\end{multicols}

\textbf{Heap}
\begin{enumerate}
    \item{Merge K Sorted Lists}
    \item{Top K Frequent Elements}
    \item{Find Median from Data Stream}
\end{enumerate}

\newpage
\section{Two Sum}
Return index of two elements in the array that add upto the target value.
\begin{lstlisting}
class Solution:
    def twoSum(self, nums: List[int], target: int) -> List[int]:
        hash_map = {}

        for idx, num in enumerate(nums):
            res = target - num
            if res not in hash_map:
                hash_map[num] = idx
            else:
                return idx, hash_map[res]

        return -1, -1
\end{lstlisting}

\section{Best Time To Buy and Sell Stock}
Return the maximum profit that can be made using the price values provided.
\begin{lstlisting}
class Solution:
    def maxProfit(self, prices: List[int]) -> int:
        profit = 0
        buying_price = float('inf')
        for i in range(0, len(prices)):
            buying_price = min(prices[i], buying_price)
            profit = max(profit, prices[i] - buying_price)
        return profit
\end{lstlisting}

\section{Contains Duplicate}
Return boolean value if there are duplicate elements in the array.
\begin{lstlisting}
class Solution:
    def containsDuplicate(self, nums: List[int]) -> bool:
        dup = set()
        for i in range(len(nums)):
            if nums[i] not in dup:
                dup.add(nums[i])
            else:
                return True
        return False
\end{lstlisting}

\section{Product of Array Except Self}
Input: nums = [1,2,3,4]
Output: [24,12,8,6]

Input: nums = [-1,1,0,-3,3]
Output: [0,0,9,0,0]
\begin{lstlisting}
class Solution:
    def productExceptSelf(self, nums: List[int]) -> List[int]:
        res = [1] * len(nums)

        prefix = 1
        for i in range(0, len(nums)):
            res[i] *= prefix
            prefix *= nums[i]

        postfix = 1
        for j in range(len(nums)-1, -1, -1):
            res[j] *= postfix
            postfix *= nums[j]

        return res
\end{lstlisting}
\section{Maximum Subarray}
Trick to solving is if the current sum is less than zero we set the current sum to zero.
\begin{lstlisting}
class Solution:
    def maxSubArray(self, nums: List[int]) -> int:
        #Kadane's algorithm repeated work
        max_sum = nums[0]
        curr_sum = 0
        for i in range(0, len(nums)):
            if curr_sum < 0:
                curr_sum = 0
            curr_sum += nums[i]
            max_sum = max(curr_sum , max_sum)
        return max_sum
\end{lstlisting}
\section{Maximum Product Subarray}
Technique to solving is to maintain current minimum and current maximum and utilize both to get the maximum answer.
\begin{lstlisting}
class Solution:
    def maxProduct(self, nums: List[int]) -> int:
        #if the max subarray is just one value
        result = max(nums)
        curr_min = 1
        curr_max = 1
        for n in nums:
            temp = curr_max
            curr_max = max(curr_max*n, curr_min*n, n)
            curr_min = min(curr_min*n, temp*n, n)
            result = max(result, curr_max)
        return result
\end{lstlisting}
\section{Container with most water}
Two pointer approach and move in the direction of greater height.
\begin{lstlisting}
class Solution:
    def maxArea(self, height: List[int]) -> int:
        right = 0
        left = len(height)-1
        max_area = 0
        while right < left:
            min_height = min(height[right], height[left])
            max_area = max(max_area, min_height * (left-right))
            if height[right] < height[left]:
                right+=1
            else:
                left -= 1
        return max_area
\end{lstlisting}

\section{Find minimum in rotated sorted array}
\begin{lstlisting}
class Solution:
    def findMin(self, nums: List[int]) -> int:
        # binary search
        left = 0
        right = len(nums)-1
        while left <= right:
            mid = int(left + (right-left)/2)

            next = (mid+1)%len(nums)
            prev = (mid-1+len(nums))%len(nums)

            if nums[mid] <= nums[prev] and nums[mid] <= nums[next]:
                return nums[mid]
            elif nums[0] <= nums[mid]:
                left = mid + 1
            elif nums[mid] <= nums[-1]:
                right = mid -1
        return nums[0]
\end{lstlisting}

\section{Search in rotated sorted array}
\begin{lstlisting}
class Solution:
    def search(self, nums: List[int], target: int) -> int:

        def find_min_index(nums):
            left = 0
            right = len(nums)-1

            while left <= right:
                mid = int(left + (right-left)/2)

                next = (mid+1)%len(nums)
                prev = (mid-1+len(nums))%len(nums)

                if nums[mid] <= nums[prev] and nums[mid] <= nums[next]:
                    return mid

                elif nums[0] <= nums[mid]:
                    left = mid + 1
                elif nums[mid] <= nums[-1]:
                    right = mid -1

            return -1

        def binary_search(nums, target):
            left = 0
            right = len(nums)-1

            while left <= right:
                mid = int(left + (right-left)/2)

                if nums[mid] == target:
                    return mid

                elif nums[mid] < target:
                    left = mid + 1

                elif nums[mid] > target:
                    right = mid -1

            return -1

        ind = find_min_index(nums)

        arr1 = nums[0:ind]
        arr2 = nums[ind:]

        val = binary_search(arr1, target)
        val2 = binary_search(arr2, target)

        return val if val2 == -1 else val2+len(arr1)
\end{lstlisting}
\section{3 Sum}
Sort the array and run two sum II for each unique element in the loop.
\begin{lstlisting}
class Solution:
    def threeSum(self, nums: List[int]) -> List[List[int]]:
        res = []
        nums = sorted(nums)
        for i in range(0, len(nums)):
            if i > 0 and nums[i] == nums[i-1]:
                continue
            left = i+1
            right = len(nums)-1

            while left < right:
                summed = nums[left] + nums[right] + nums[i]
                if summed == 0:
                    res.append((nums[left], nums[right], nums[i]))
                    left += 1
                    while nums[left] == nums[left -1] and left < right:
                        left += 1

                elif summed < 0:
                    left += 1

                elif summed > 0:
                    right -=1
        return res
\end{lstlisting}
\section{Container with most water}
\begin{lstlisting}
class Solution:
    def maxArea(self, height: List[int]) -> int:
        left = 0
        right = len(height)-1
        max_area = -1
        while left < right:
            h = min(height[left], height[right])
            area = (right - left) * h
            max_area = max(max_area, area)
            if height[left] < height[right]:
                left += 1
            elif height[right] < height[left]:
                right -=1
            elif height[right] == height[left]:
                left+=1
        return max_area
\end{lstlisting}

\section{Binary - Sum of two integers}
We make use of xor a=1, b=0, ans=1. a=1, b=1, ans=0

When we have two one's we have a carry. If a and b are 1, we have a carry. a\&b left shift 1

If we dont have a carry we are done.

Code doesnt work in python since the assumption that we make is that the length of integers is 32 bits. However python allows for an infinite length of integers. Java does not.
\begin{lstlisting}
class Solution {
    public int getSum(int a, int b) {
        while( b!= 0){
            int temp = (a &b) <<1;
            a = a ^ b;
            b = temp;
        }
        return a;
    }
}
\end{lstlisting}

\section{Number of one bits}

Count the number of 1's in bits.
\begin{lstlisting}
class Solution:
    def hammingWeight(self, n: int) -> int:
        total = 0
        while n != 0:
            total += n %2
            n = n >> 1
        return total
\end{lstlisting}

\section{Counting Bits}
Dynamic Programming Solution
\begin{lstlisting}
class Solution:
    def countBits(self, n: int) -> List[int]:
        dp = [0]*(n+1)
        offset = 1

        for i in range(1, n+1):
            if offset * 2 == i:
                offset = i

            dp[i] = 1 + dp[i - offset]

        return dp
\end{lstlisting}

\section{Missing Number}
\begin{lstlisting}
class Solution:
    def missingNumber(self, nums: List[int]) -> int:
        xor1 = 0
        xor2 = nums[0]
        for i in range(1, len(nums)+1):
            xor1 ^= i
        for j in range(1, len(nums)):
            xor2 ^= nums[j]

        return xor1 ^ xor2
\end{lstlisting}

\section{Reverse Bits}
\begin{lstlisting}
class Solution:
    def reverseBits(self, n: int) -> int:
        res = 0
        for i in range(0, 32):
            res = res << 1
            if n % 2 == 1:
                res += 1

            n = n >> 1
        return res
\end{lstlisting}

\section{DP - Climbing stairs}
You are climbing a staircase. It takes n steps to reach the top.

Each time you can either climb 1 or 2 steps. In how many distinct ways can you climb to the top?
\begin{lstlisting}
class Solution:
    def __init__(self):
        self.map = {}

    def climbStairs(self, n: int) -> int:
        if n == 0:
            return 0
        if n == 1:
            return 1
        if n == 2:
            return 2

        if n in self.map:
            return self.map[n]

        self.map[n] = self.climbStairs(n-1) + self.climbStairs(n-2)

        return self.map[n]
\end{lstlisting}

\section{Coin Change}
\begin{lstlisting}
class Solution:
    def coinChange(self, coins: List[int], amount: int) -> int:
        memo = {}

        def unbounded(coins, i, target):
            key = (i, target)

            if key in memo:
                return memo[key]

            if target == 0:
                return 0

            if i<=0 and target > 0:
                return float('inf')

            if coins[i-1] > target:
                memo[key] = unbounded(coins, i-1, target)
                return memo[key]

            memo[key] = min(1+unbounded(coins, i, target-coins[i-1]),
                           unbounded(coins, i-1, target))

            return memo[key]

        res = unbounded(coins, len(coins), amount)

        if res == float('inf'):
            return -1

        return res
\end{lstlisting}

\section{Longest Increasing Subsequence}
Recursive memoized solution.

Best to explore using nested loops as well.
\begin{lstlisting}
class Solution:
    def lengthOfLIS(self, nums: List[int]) -> int:
        memo = {}

        def lcs(nums, prev, current):
            key = (prev, current)

            if key in memo:
                return memo[key]

            if len(nums) == current:
                return 0

            c1 = 0

            if (prev == -1) or (nums[prev] < nums[current]):
                c1 = 1+lcs(nums, current, current+1)

            c2 = lcs(nums, prev, current+1)

            memo[key] = max(c1, c2)

            return memo[key]

        return lcs(nums, -1, 0)
\end{lstlisting}

\section{Longest Common Subsequence}
\begin{lstlisting}
class Solution:
    def longestCommonSubsequence(self, text1: str, text2: str) -> int:
        memo = {}

        def lcs(s1, s2, i, j):
            key = (i, j)

            if key in memo:
                return memo[key]

            if i<=0 or j<=0:
                return 0

            if s1[i-1] == s2[j-1]:
                memo[key] = 1+lcs(s1, s2, i-1, j-1)
                return memo[key]

            memo[key] = max(lcs(s1, s2, i, j-1), lcs(s1, s2, i-1, j))

            return memo[key]

        return lcs(text1, text2, len(text1), len(text2))
\end{lstlisting}

\section{Word Break}
\begin{lstlisting}
class Solution:
    def wordBreak(self, s: str, wordDict: List[str]) -> bool:
        dp = [False] * (len(s)+1)
        dp[len(s)] = True

        for i in range(len(s), -1, -1):
            for w in wordDict:
                if i+len(w) <= len(s) and s[i : i + len(w)] == w:
                    dp[i] = dp[i+len(w)]

                if dp[i]:
                    break

        return dp[0]
\end{lstlisting}

Recursive DP
\begin{lstlisting}
class Solution:
    def wordBreak(self, s: str, wordDict: List[str]) -> bool:
        memo = {}

        def wb(s, wordDict):
            if s in memo:
                return memo[s]

            if len(s) == 0:
                memo[""] = True
                return True

            for word in wordDict:
                if s.startswith(word) and len(s) >= len(word):
                    if wb(s[len(word):], wordDict):
                        return True

            memo[s] = False

            return False

        return wb(s, wordDict)
\end{lstlisting}

\section{Combination Sum}

Given an array of distinct integers candidates and a target integer target, return a list of all unique combinations of candidates where the chosen numbers sum to target. You may return the combinations in any order.

Time Complexity = 2\^T (Where T is the target value, as the height of the decision tree is determined by the target value.)

\begin{lstlisting}
class Solution:
    def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
        res = []

        def dfs(i, curr, total):
            if total == target:
                res.append(curr.copy())
                return

            if i>= len(candidates) or total > target:
                return

            curr.append(candidates[i])
            dfs(i, curr, total+candidates[i])
            curr.pop()
            dfs(i+1, curr, total)

            return

        dfs(0, [], 0)
        return res
\end{lstlisting}

\section{House Robbers}

Find the recurrence, code up the recursion.

Recurrence = max(arr[0] + dp(arr[2:]), dp(arr[1:]))

\begin{lstlisting}
class Solution:
    def rob(self, nums: List[int]) -> int:
        # recurrence
        #rob = max(nums[0] + rob(nums[2:]), rob(nums[1:]))
        memo  = {}

        def dp(nums, n):
            if n >= len(nums):
                return 0

            if n in memo:
                return memo[n]

            memo[n] = max(nums[n] + dp(nums, n+2), dp(nums, n+1))

            return memo[n]

        return dp(nums, 0)
\end{lstlisting}

\section{House Robber II}

This time the first and last house are connected so we split the array into two parts.

We run our house robber I function on arr[:-1] and arr[1:]

\begin{lstlisting}
class Solution:
    def rob(self, nums: List[int]) -> int:
        if len(nums) == 1:
            return nums[0]

        memo = {}

        def dp(nums, n):
            if n >= len(nums):
                return 0
            if n in memo:
                return memo[n]

            memo[n] = max(nums[n] + dp(nums, n+2), dp(nums, n+1))

            return memo[n]

        arr1 = dp(nums[:-1], 0)

        memo = {}

        arr2 =  dp(nums[1:], 0)

        max_ret = max(arr1, arr2)

        return max_ret
\end{lstlisting}

\section{Decode Ways}

Input: s = "12"
Output: 2
Explanation: "12" could be decoded as "AB" (1 2) or "L" (12).

Input: s = "226"
Output: 3
Explanation: "226" could be decoded as "BZ" (2 26), "VF" (22 6), or "BBF" (2 2 6).

Input: s = "06"
Output: 0
Explanation: "06" cannot be mapped to "F" because of the leading zero ("6" is different from "06").

\begin{lstlisting}
class Solution:
    def numDecodings(self, s: str) -> int:
        #11106
        #possibilities = AAJF or KJF

        memo = {}

        def dp(i):
            if i in memo:
                return memo[i]
            if i >= len(s):
                return 1
            if s[i] == "0":
                return 0
            res = dp(i+1)

            if i+1 < len(s) and (s[i] == "1" or s[i] == "2" and s[i+1] in "0123456"):
                res += dp(i+2)

            memo[i] = res

            return memo[i]

        return dp(0)
\end{lstlisting}

\section{Unique Paths}

Time limit exceeded solution using DFS.
\begin{lstlisting}
class Solution:
    def uniquePaths(self, m: int, n: int) -> int:
        matrix = [[0]*n for i in range(m)]

        matrix[m-1][n-1] = 1

        visited = set()

        count = [0]

        def dfs(matrix, i, j):
            if i >= len(matrix) or j >= len(matrix[0]):
                return

            if matrix[i][j] == 1:
                count[0] += 1
                return

            if (i, j) in visited:
                return

            visited.add((i, j))

            dfs(matrix, i+1, j)
            dfs(matrix, i, j+1)

            visited.remove((i, j))

            return

        dfs(matrix, 0, 0)

        return count[0]
\end{lstlisting}

Dynamic Programming solution.

\begin{lstlisting}
class Solution:
    def uniquePaths(self, m: int, n: int) -> int:
        row = [1] * n

        for i in range(m-1):
            new_row = [1]*n
            for j in range(n-2, -1, -1):
                new_row[j] = new_row[j+1] + row[j]

            row = new_row

        return row[0]
\end{lstlisting}

\section{Jump Game}
Greedy method, just keep moving the goal post to the left if we can reach the goal.

\begin{lstlisting}
class Solution:
    def canJump(self, nums: List[int]) -> bool:
        goal = len(nums)-1

        for i in range(len(nums)-1, -1, -1):
            if i+nums[i] >= goal:
                goal = i

        return goal == 0
\end{lstlisting}

\section{LinkedList Reverse}
Drawing a diagram always helps. Try to trace in the middle of the list.
\begin{lstlisting}
class Solution:
    def reverseList(self, head: ListNode) -> ListNode:
        current = head

        prev = None

        while current != None:
            next = current.next
            current.next = prev
            prev = current
            current = next

        return prev
\end{lstlisting}

\section{Detect Cycle in LinkedList}

\begin{lstlisting}
class Solution:
    def hasCycle(self, head: ListNode) -> bool:
        slow_pointer = head
        fast_pointer = head

        while fast_pointer and fast_pointer.next:
            fast_pointer = fast_pointer.next.next
            slow_pointer = slow_pointer.next

            if fast_pointer == slow_pointer:
                return True

        return False
\end{lstlisting}

\section{Merge K sorted Lists}
Input: lists = [[1,4,5],[1,3,4],[2,6]]

Output: [1,1,2,3,4,4,5,6]

Explanation: The linked-lists are:
[
  1->4->5,
  1->3->4,
  2->6
]

merging them into one sorted list:

1-1-2-3-4-4-5-6
\begin{lstlisting}
import heapq

class Solution:
    def mergeKLists(self, lists: List[Optional[ListNode]]) -> Optional[ListNode]:
        heap = []

        for l in lists:
            while l != None:
                heapq.heappush(heap, l.val)
                l = l.next

        ans = ListNode()

        head = ans

        while len(heap) > 0:
            ans.next = ListNode(heapq.heappop(heap))
            ans = ans.next

        return head.next
\end{lstlisting}

\section{Remove Nth element from end of list}

\begin{lstlisting}
class Solution:
    def removeNthFromEnd(self, head: ListNode, n: int) -> ListNode:
        count = 0

        temp = head

        while temp != None:
            temp = temp.next
            count += 1

        if count == n:
            return head.next

        position = count - n -1

        temp2 = head

        ans = temp2

        for i in range(0, position):
            temp2 = temp2.next

        temp2.next = temp2.next.next

        return ans
\end{lstlisting}

\end{document}


\documentclass[9pt, a4paper, oneside, reqno]{amsart}

\usepackage[final]{pdfpages}
\usepackage{wrapfig}

\usepackage{enumitem}
\usepackage{parskip}
\usepackage{fancyhdr}
\usepackage{color}
\usepackage{multicol}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}

\newcommand{\hint}[1]{\footnote{\raggedleft\rotatebox{180}{Hint: #1\hfill}}}

\pagestyle{fancy}

\newlist{questions}{enumerate}{1}
\setlist[questions, 1]{label = \bf Q.\arabic*., itemsep=1em}

\lhead{\scshape Apurva Nakade}
\rhead{\scshape Honors Single Variable Calculus}
\renewcommand*{\thepage}{\small\arabic{page}}
\title{Problem Set 13}

\begin{document}

\maketitle
\thispagestyle{fancy}

\section*{Part 1 - Nowhere Differentiable Function}
In this problem set we'll construct a function $ f_\infty(x)$ which is continuous everywhere but differentiable nowhere(!) using only the techniques you've learnt in this course.

\begin{questions}
	\item
	Define the function $ \{ x \}$ as
	\begin{align*}
		\{x \} & = \mbox{the distance of $x$ from the nearest integer }
	\end{align*}
	Let $ n$ denote a positive integer. Define the function
	\begin{align*}
		f_n(x)
		 & =  \{x\} + \dfrac{\{ 2 x\}}{2} + \dots + \dfrac{\{ 2^n x\}}{2^{n}} \\
		 & = \sum \limits_{i=0}^{n} \dfrac{\{ 2^i x\}}{2^{i}}
	\end{align*}
 Draw the graph of the functions $ \{x\}$, $ \dfrac{\{ 2 x\}}{2}$, $ \dfrac{\{ 2^n x\}}{2^{n}}$, and $ f_1(x)$.

	\item Let $ x \in [0,1]$. Determine the relation between
	\begin{enumerate}
		\item $ \{x\}$ and $ \{x+1\}$
		\item $ \{x\}$ and $ \{1-x\}$
		\item $ \{x\}$ and $ \{x+1/2 \}$
		\item $ \{x\}$ and $ \{2x\}$
	\end{enumerate}

	\item Using properties of continuous functions argue that $ \{ x \}$ is a continuous function, and hence so is $ f_n(x)$.

	\item What are the points at which $ f_n(x)$ is not differentiable? Assuming that it makes sense to take limits of functions, what do you think are the points at which the function $ \lim \limits_{n\rightarrow \infty}f_n(x)$ is not differentiable. (Proof not needed.)

	\item Using your favorite test for series convergence, show that for every real number $ x$ the series $\sum \limits_{i=0}^{\infty} \dfrac{\{ 2^i x\}}{2^{i}}$ converges.
\end{questions}

Hence we can \textbf{define} a new function
\begin{align*}
	f_\infty(x) & = \sum \limits_{i=0}^{\infty} \dfrac{\{ 2^i x\}}{2^{i}}
\end{align*}
\begin{questions}[resume]
	\item Show that $ |f_\infty(x) - f_n(x)| \le 2^{-n}$ for all $ x$.

	\item Using the triangle inequality
	\begin{align*}
		|f_\infty(x) - f_\infty(a)|
		 & \le |f_\infty(x) - f_n(x)|              \\
		 & \qquad + |f_n(x) - f_n(a)|              \\
		 & \qquad  \qquad + |f_n(a) - f_\infty(a)|
	\end{align*}
	and the $ \epsilon-\delta$ definition of continuity, prove that $ f_\infty(x)$ is a continuous function.
\end{questions}
In order to show that $ f_\infty(x)$ is not differentiable anywhere it's helpful to use binary expansions of numbers.

Let $ x$ be a real number with binary expansion $ x = m + 0.a_1 a_2 a_3 \dots$ where $ m$ is an integer and each $ a_i = 0$ or $ 1$.
\begin{questions}[resume]
	\item Show that the number whose binary expansion consists of all one's $ 0.111 \dots 1 \dots $ is equal to 1.\hint{This is a geometric series.} What is the corresponding statement for decimal expansions?

	Because of this we can assume that there are no \emph{trailing 1's} in the binary expansion of any number.

	\item
	\begin{enumerate}
		\item Find $ \{ x \}$ in terms of the binary expansion of $ x$.
		\item Find $ \dfrac{\{ 2^i x\}}{2^{i}}$ in terms of the binary expansion of $ x$.
		\item If $ x$ has a finite binary expansion $ x = m + 0.a_1 a_2 \cdots a_n$, what is $ f_\infty(x)$?
	\end{enumerate}
\end{questions}

Let $ b_n$ be the position of the $ n^{th}$ zero after the decimal point in the binary expansion of $ x$.
	For example,
	\begin{quote}
		{if $ x = 1011.1010101\dots$ then $ b_n =2n$,}\\
	{if $ x = 0.1011011011\dots$ then $ b_n =3n-1$, etc.}
\end{quote}
\begin{questions}[resume]
	\item
	\begin{enumerate}
		\item Show that
		\begin{align*}
			\dfrac{\{ 2^i(x + 2^{-b_n})\}}{2^{i}} - \dfrac{\{ 2^i x\}}{2^{i}}
			&=
			\begin{cases}
				-2^{-b_n} & \mbox { if }i < b_n - 1 \\
				0 & \mbox { if }i \ge b_n
			\end{cases}
		\end{align*}
		\item Show that $ \dfrac{f_\infty(x + 2^{-b_n}) - f_\infty(x)}{2^{-b_n}} < -(b_n - 1) + 2^{-b_n+1}$.
	\end{enumerate}
	\item Show that $ \lim \limits_{n \rightarrow \infty} \dfrac{f_\infty(x + 2^{-b_n}) - f_\infty(x)}{2^{-b_n}}$ does not exist. Conclude that $ f(x)$ is not differentiable at $ x$.
\end{questions}

\newpage
\section*{Part 2 - Trigonometry and Complex Numbers}{
	Complex numbers are numbers of the form $ a+i.b$ where $ a,b$ are real numbers and $ i^2 = -1$. Complex numbers are very useful in calculus, especially for finding integrals, because of the following \textbf{Euler's identity}
	\begin{align*}
		e^{i \theta} = \cos \theta + i \sin \theta
	\end{align*}
}
\begin{questions}[resume]
	\item Verify Euler's identity using the Taylor series for $ e^x$, $ \sin x$ and $ \cos x$.
	\item Using the fact that $ e^{i(a+b)} = e^{ia}.e^{ib}$ compute the formulae for  $ \sin(a+b)$, $ \cos(a+b)$, $ \sin 2x$, $ \cos 2x$, $ \sin 3x$, and $ \cos 3x$.
	\item Show that
	\begin{align*}
		\cos x = \dfrac{e^{ix} + e^{-ix}}{2} &  & \sin x = \dfrac{e^{ix} - e^{-ix}}{2i}
	\end{align*}
	(Compare these with the formulae for $ \sinh x$ and $ \cosh x$.)
	\item Use these formulae to compute the following integrals\hint{You might need to use the identity $ \dfrac{1}{a+ib} = \dfrac{a-ib}{a^2 + b^2}$}
	\begin{enumerate}
		\item $ e^{ax} \sin bx$
		\item $ e^{ax} \cos bx$
		\item $ \cos^2 x$
		\item $ \sin x \cos 4x$
	\end{enumerate}
\end{questions}

\end{document}