\documentclass[11pt]{article}

\usepackage{xcolor}
\usepackage{amsmath}
\begin{document}

\newcommand{\jrs}[1]{\textcolor{blue}{[#1]}}

\section*{Thesis Corrections}

In addition to the correction of minor typos, I have addressed thee examiner comments as described in the following sections.

\section{Addressing comments from examiner 1}
[preliminary recommendation]

\begin{enumerate}
\item At the beginning of Chapter 2, it is not clearly stated why optimization algorithm is related to STA or quantum optimal control. It is clearly explained later in Section 2.1. It would be better to briefly explain it at the beginning.

\jrs{I have moved this discussion to the start of the chapter}

\item In Chapter 4, the Lyaponov exponent is computed for vortex dynamics. It is a bit unusual that the Lyaponov exponent can be negative. To my knowledge, it is always non-negative. It would be nice to offer some explanation.

\jrs{I have added a paragraph explaining our calculation of the Lyapunov exponent, which was ambiguous in the original text.
In particular, I have modified the full Lyapunov exponent to be $\Lambda$, and introduced the time-dependent Lyapunov exponent $\lambda(t)$, which was the actual quantity calculated in this work.
I have modified the text as such:}

\jrs{...
\begin{equation}
\Lambda = \lim_{t\to\infty}\frac{1}{t}\text{ln}\frac{||\delta\textbf{P}(t)||}{||\delta\textbf{P}(0)||}
\label{eqn:lyap_full}
\end{equation}
where $||\cdot||$ denotes the Euclidean norm.}

\jrs{
Because this system is finite, the value of the Lyapunov exponent from Equation~\eqref{eqn:lyap_full} will tend to 0.
For this reason, rather than performing the full calculation of the Lyapunov exponent, $\Lambda$, I will instead show the exponent as a function of time, here notated as
\begin{equation}
\lambda(t) = \frac{1}{t}\text{ln}\frac{||\delta\textbf{P}(t)||}{||\delta\textbf{P}(0)||}.
\label{eqn:lyap}
\end{equation}
\noindent This value can characterize chaotic dynamics in finite-sized systems, and will be referred to as the time-dependent Lyapunov exponent for the remainder of this work.
}

\item The explanation of Nelder--Mead method is not clear. For example, there is no explanation of heights.

\jrs{I have added the following: For this method, one is attempting to find the minimum value in some $n$-dimensional space.
In Nelder and Mead's original notation, a ``height'' is defined as the value of the cost function with provided input parameters, often depicted as the an elevation out of the plane.}

\item There are many typos, here are a few examples:

Page 6: under Eq.(1.1) partial/partial x should be partial/partial r; $V_0$ could be any potential, not just trapping potential Page 7: second line: Which should be which Page 28: partial A/partial i should be something else

\jrs{These have all been addressed. I have also read through the thesis thoroughly and addressed every inconsistency I could find.}

\end{enumerate}
[comments after the oral examination]

Mr. James Schloss gave a very good presentation in his oral examination, which is very well organized and logically clear. In the closed-door examination, James was able to answer most of the questions raised by examiners. When he was able to answer questions, his answers were coherent and convincing; when he was not able to answer, he would quickly concede that he did not know the answer.

My overall impression is that Mr. James Schloss is solid in physics and very strong in computer programming, in particular with GPU. The amount of work that he has put into his thesis work is remarkable. I recommend that he pass the oral examination with minor revisions to his thesis. In the revision, Mr. James Schloss should address the following issues:

\begin{enumerate}
\item Justification of his calculation of Lyaponov exponent;

\jrs{This has been addressed as described above}

\item Why four vortecies are needed to see the chaotic motion;

\jrs{This statement was made based on the work of Aref and Pomphrey (\textit{Proc. R. Soc. Lond. A} 380, 359(1982)), who showed that the dynamics of 4 \textit{co-rotating} can display non-integrable dynamics.
In fact, it was shown that for a harmonic trap, chaos can be induced with as few as 3 vortices (2 co-rotating with 1 anti-vortex, example: \textit{Chaos} 24, 024410(2014)).
With this in mind, I have added the following statement (with appropriate citations in the text):}

\jrs{
When analyzed with a reduced Hamiltonian approach, harmonically trapped BECs seem to exhibit chaotic effects with as few as three vortices, two co-rotating vortices and an anti-vortex rotating in the other direction.
In this approach, the system can be seen as having three degrees of freedom with two integrals of motion, the energy and angular momentum.
This can be further reduced to two degrees of freedom with appropriate canonical transformations and using the angular momentum as a parameter.
For this reason, it seems that three point-vortices is the minimum number necessary for chaotic dynamics.}

\end{enumerate}

\section{Addressing comments from examiner 2}

[Preliminary recommendation]

This thesis represents an excellent body of work, combining both cutting-edge computational development with applications to interesting ultracold atomic physics. The GPUE package contains complex and well-motivated optimisations for the analysis of vortices in degenerate Bose gases. These dynamics require high spatial and temporal resolution, and require computing resources to be carefully optimised. The proposals to generate NOON states and nonstandard phase imprinting via artificial magnetic fields are very interesting, and clearly explained. I recommend this thesis be awarded a PhD. The candidate should consider minor revisions according to the comments below.

Chapter 1: In the beginning, the benefits of SSFM are compared with RK, Euler and Crank-Nicholson methods, which I found odd. Runge-Kutta, Euler, Crank-Nicholson are all timestep methods, and can be used happily in conjunction with Fourier-based derivative calculations. Runge-Kutta and Euler are also explicit, and technically parallelisable in exactly the same way as SSFM. I am interested in the claim that there are extra difficulties with parallelising RK methods, as referenced. Compared to those methods, one key advantage of SSFM is that the sub-units in each space are often solvable to all order (exponential solutions in time), this can lead to better stability properties.

Runge-Kutta can be superior for many problems, however. Importantly, Runge-Kutta can also be executed using the all-order correct exponential solutions within each space for each step. This can give high order stability with the other benefits of SSFM. This is done in the XMDS2 package, and parallelised for some cases in the xSDPE package. I would be interested to hear more details on the specific advantages of the SSFM over (for example) RK4IP in a GPU context. Is the issue essentially one of memory?

\jrs{This is correct. I have modified this statement to account for this: \\
For example, the SSFM relies on embarrassingly parallel element-wise matrix multiplications and Fast Fourier Transform (FFT) routines that have been optimized for parallel and distributed systems, but it also requires less memory than Runge-Kutta, which requires multiple arrays of the size of the wavefunction in memory.}

In chapter 2, I know that local and global searching algorithms are often called 'optimal control', but I believe this is confusing given the extensive literature on optimal control that finds provable optimal solutions. Most of the algorithms discussed in machine learning are certainly \textit{optimising} the control scheme, but technically without exhaustive search, they do not find the provably optimal control. It might be better to talk in terms of optimising without claims of 'optimal control'.

On p45: the fact that optimisations over rotation outperform optimisations over both rotation and barrier height shows that the results are definitely not optimal, and that the infidelity landscape is not densely searched. This is not a criticism of the method - high dimensional searches are rarely safely dense - but it does point to the benefit of distinction in the nomenclature.

\jrs{I acknowledge this distinction and have changed all instances of ``optimal control'' to either ``optimization'' or ``quantum optimal control,'' which are more accurate descriptions.}

In eq1.44 and below, need to designate lambda as a vector of parameters to use the later vector notation. Using a curl for the magnetic field analogy then requires lambda to be a spatial degree of freedom.

\jrs{all $\lambda$s have been bolded to signify they are vectors.}

Minor typos:

\begin{itemize}
\item In eq.1.9, I think the $c_{n}$s should still be included. They are less important due to the renormalisation, but they are still required for the equation to be correct (most trivially at dt=0).

\item eq 1.31 $J \rightarrow j$
\item cyllindrically $\rightarrow$ cylindrically
\item "follow the magnetic fields lines" $\rightarrow$ "follow the magnetic field lines"
\item "In addition GPU memory considerations" $\rightarrow$ "In addition to GPU memory considerations"
\item "parallel reduction has exists in GPUE" $\rightarrow$ "parallel reduction exists in GPUE"
\item eq 3.7 is still a template of the copy
\item "With an appropriate distributed transpose methods" $\rightarrow$ "With an appropriate distributed transpose method"
\item "inhomogeniously" $\rightarrow$ "inhomogenously"
\item "various quantum phenomenon that were computational intractable" $\rightarrow$ "various quantum phenomena that were computationally intractable"

\end{itemize}

\jrs{These have all been addressed}

[comments after the oral examination]

James presentation was excellent. His performance on the oral examination was satisfactory. We all agreed that James had demonstrated sufficient mastery of his subject, and had produced more than enough original work to be awarded his doctorate.

He was asked a number of questions, some regarding technical mathematical elements of his work, some on implementation details, and several on the algorithmic choices demonstrated in his computational work. Most of his answers were entirely satisfactory, although there were a couple of points that we felt would be best clarified in the thesis before the final version was submitted. They were:

\begin{enumerate}
\item The nature and motivations of the specific Lyapunov calculations done on the group of four vortices should be expanded upon. The definition of the Lyapunov exponent should be correct.

\jrs{This was addressed in the previous section}
\item Experimental considerations for constructing the toroidal potential around the fibre should be discussed. James mentioned in the examination that he chose parameters compatible with fibre trapping. These details should be included.

\jrs{I have renamed the text and added necessary citations to read: \\
...where the frequencies in the $\hat r$ and $\hat z$ directions are chosen to be $\omega_r = \omega_z = 7071$Hz to match typical experimental conditions in fiber trapping and is roughly consistent with the known trapping frequencies for toroidal traps}

\item The underlying mechanism behind the artificial magnetic fields seems very close to the one that makes light shifts, yet the effective term in the Hamiltonian is different. The reason for the difference should be discussed.

\jrs{I have added the following (with appropriate citations): \\
As an important note, the fields typically used for fiber trapping have very large detunings, as small detunings lead to higher scattering rates and losses.
These values will lead to insignificant gauge fields.}
\end{enumerate}
\end{document}


% Filename: slide04.tex
% This code is part of 'Cursos CAMECC: Introducao ao LaTeX para o Curso 29 - Licenciatura em Matematica'
%
% Description: This file correspond to the slides of using for cap04@camecc_lic.tex of the course.
%
% Created: 07.06.12 11:30:10 AM
% Last Change: 07.06.12 11:30:30 AM
%
% Authors:
% - Raniere Silva, r.gaia.cs@gmail.com
%
% Organization: CAMECC - Centro Academico dos Estudantes do IMECC
%
% Copyright (c) 2012, Raniere Silva. All rights reserved.
%
% This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
%
% This work is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
%
\documentclass[12pt]{beamer}
\usetheme{CambridgeUS}
\let\Tiny=\tiny  % Redefine at least \Tiny for avoid warning
\input{use_pack.tex}

% Para os slides com codigos do Beamer.
\newenvironment{slide}[1]
{\begin{frame}[fragile,environment=slide]
    #1}
{\end{frame}}

\begin{document}
\title[Introdu\c{c}\~{a}o ao LaTeX - 04/04]{Cursos CAMECC \\ Introdu\c{c}\~{a}o ao LaTeX para o Curso 29 \\ Licenciatura em Matem\'{a}tica \\ Aula 04}
\author{Raniere Silva}
\institute[CAMECC]{Centro Acad\^{e}mico dos Estudantes do IMECC}
\begin{frame}
    \titlepage
\end{frame}

\begin{frame}
    \frametitle{Licen\c{c}a}
    Este trabalho foi licenciado com a Licen\c{c}a Creative Commons Atribui\c{c}\~{a}o - CompartilhaIgual 3.0 N\~{a}o Adaptada. Para ver uma c\'{o}pia desta licen\c{c}a, visite \url{http://creativecommons.org/licenses/by-sa/3.0/} ou envie um pedido por carta para Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
    %
    %Com base na obra dispon\'{i}vel em \url{https://github.com/r-gaia-cs/latex_with_vim/}.
    %
    %Podem estar dispon\'{i}veis permiss\~{o}es adicionais ao \^{a}mbito desta licen\c{c}a em \url{https://github.com/r-gaia-cs/latex_with_vim/}.
    %
    \begin{center}
        \includegraphics[keepaspectratio=true]{../../figures/cc-by-sa.png}
    \end{center}
\end{frame}

\begin{frame}
    \frametitle{Sum\'{a}rio}
    \tableofcontents
\end{frame}

\section{Obtendo Ajuda}
\begin{frame}[fragile]
    \frametitle{RFM!!!}
    No terminal, utilize
    \begin{code}
texdoc tikz
    \end{code}
    para o manual do TikZ e
    \begin{code}
texdoc beamer
    \end{code}
    para o manual do Beamer.
\end{frame}

\section{TikZ}
\begin{frame}[fragile]
    \frametitle{Algumas formas}
    \begin{code}
\documentclass{article}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
    \draw (0,0) -- (1,0);
    \draw (0,-1) -- (1,0) -- (2,-1);
    \draw (-2,-2) rectangle (0,-3);
    \fill (2,-2) circle (1);
\end{tikzpicture}
\end{document}
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Sorvete}
    \begin{code}
\documentclass{article}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
    \draw[fill=orange, rotate=45] (0,1) rectangle (4,1.5);
    \draw[fill=red!70] (0,1) circle (2);
    \draw[fill=brown] (-2,0) -- (0,-4) -- (2,0);
\end{tikzpicture}
\end{document}
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \frametitle{\^{A}ngulos}
    \begin{code}
\documentclass{article}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
    \draw[->] (0,0) -- (1,0) node[below right]{A} -- (2.5,0);
    \draw[->] (0,0) -- (0,2.5);
    \draw[dotted] (-2,-2) grid (2,2);
    \draw[blue] (0,0) circle (1);
    % (60:1) significa 60 graus e raio 1.
    \draw (0,0) node[below left]{O} -- (60:1) node[above right]{B};
    \draw[red] (.2,0) arc (0:60:0.2);
\end{tikzpicture}
\end{document}
    \end{code}
\end{frame}

\begin{frame}[fragile]
    \frametitle{Gr\'{a}ficos de Fun\c{c}\~{o}es}
    \begin{code}
\begin{tikzpicture}[domain=0:4]
    \draw[very thin,color=gray] (-0.1,-1.1) grid (3.9,3.9);
    \draw[->] (-0.2,0) -- (4.2,0) node[right] {$x$};
    \draw[->] (0,-1.2) -- (0,4.2) node[above] {$f(x)$};
    \draw[color=red] plot (\x,\x) node[right] {$f(x) =x$};
    \draw[color=blue] plot (\x,{sin(\x r)}) node[right] {$f(x) = \sin x$};
    \draw[color=orange] plot (\x,{0.05*exp(\x)}) node[right] {$f(x) = \frac{1}{20} e^x$};
\end{tikzpicture}
    \end{code}
    O exemplo acima foi retirado de ``The TikZ and PGF Packages - Manual for version 2.10'' de Till Tantau.
\end{frame}

\section{Beamer}
\begin{slide}
    \frametitle{Hello}
    \begin{code}
\documentclass{beamer}
\begin{document}
\begin{frame}
    Hello.
\end{frame}
\end{document}
    \end{code}
\end{slide}

\begin{slide}
    \frametitle{T\'{i}tulo}
    \begin{code}
\documentclass{beamer}
\begin{document}
\begin{frame}
    \frametitle{Hello}
    Me chamo \ldots
\end{frame}
\end{document}
    \end{code}
\end{slide}

\begin{slide}
    \frametitle{Tema}
    \begin{code}
\documentclass{beamer}
% Mais temas em http://www.hartwork.org/beamer-theme-matrix/
\usetheme{CambridgeUS}
\begin{document}
\begin{frame}
    \frametitle{Hello}
    Me chamo \ldots
\end{frame}
\end{document}
    \end{code}
\end{slide}

\begin{slide}
    \frametitle{``Anima\c{c}\~{o}es''}
    \begin{code}
\documentclass{beamer}
\usepackage{tikz}
\begin{document}
\begin{frame}
\begin{tikzpicture}
    \draw[fill=orange, rotate=45] (0,1) rectangle (4,1.5); \pause
    \draw[fill=red!70] (0,1) circle (2); \pause
    \draw[fill=brown] (-2,0) -- (0,-4) -- (2,0);
\end{tikzpicture}
\end{frame}
\end{document}
    \end{code}
\end{slide}

\begin{frame}
    \frametitle{Fim}
    \begin{center}
        Perguntas.
    \end{center}
\end{frame}
\end{document}


\chapter{\proj Image Compression}
\label{ch_comp}
\index{compression}
% \chapterhead{Programs}
\markright{Image Compression}

\section{Introduction}
Image compression can be necessitated for different reasons, which may
in practice motivate different compression strategies. Cases which can be
easily distinguished include: quicklook, very large images (e.g.\ the
all-sky atlas, ALADIN,
with links to cataloged information on tens of millions of astronomical
objects \cite{compress:bartlett96}); fast access to
large pictorial databases; and Web-based and other types of data transmission
(a number of references to past work on progressive image transmission
can be found in \cite{compress:hannaford93}).

Following the kind of images and the application needs, different strategies
can be used:
\begin{enumerate}
\baselineskip=0.4truecm
\item Lossy compression: in this case the compression ratio is relatively
low ($< 5$).
\item Compression without visual loss. This means that one cannot see the
difference between the original image and the decompressed one. Generally,
compression ratios between 10 and 20 can be obtained.
\item Good  quality compression: the decompressed image does not contain any
artifact, but some information is lost. Compression ratios up to 40 can be
obtained in this case.
\item Fixed compression ratio: for some technical reasons, one may decide to
compress all images with a compression ratio higher than a given value,
whatever the effect on the decompressed image quality.
\item Signal/noise separation: if noise is present in the data, noise
modeling can allow very high compression ratios just by including
filtering in wavelet space during the compression.
\end{enumerate}
Following the image type, and the selected strategy, the optimal compression
method may differ. A major
interest in using a multiresolution framework is to get, in a natural way, the
possibility for progressive information transfer.

According to Shannon's theorem, the number of bits we need to code
an image $I$ without distorsion (losslessly) is given by its entropy $H$.
If the image (with $N$ pixels) is coded with $L$ intensity levels, each level having a
probability $p_i$ to appear, the entropy $H$ is
\begin{eqnarray}
H(I) = \sum_{i=1}^L - p_i \log_2 p_i
\end{eqnarray}
The probabilities $p_i$ can be easily derived from the image histogram.
The compression ratio is given by:
\begin{eqnarray}
{\cal{C}}(I) = \frac{\mbox{number of bits per pixel in the raw data}}{H(I)}
\end{eqnarray}
and the distortion is measured by
\begin{eqnarray}
R = \parallel I - \tilde I \parallel =  \sum_{k=1}^N (I_k - \tilde I_k)^2
\end{eqnarray}
where $\tilde I$ is the decompressed image.

A Huffman, or an arithmetic, coder is generally used to transform the set
of integer values into the  new set of values, in a reversible way.

Compression methods try to use the redundancy contained in the raw data
in order to reduce the number of bits. The main efficient methods belong to
 the transform
coding family, where the image is first transformed into another set of data
where the information is more compact (i.e.\ the entropy of the new set is
lower than the original image entropy). The typical steps are:
\begin{enumerate}
\item transform the image (for example using a discrete cosine transform, or
a wavelet transform),
\item quantize the obtained coefficients, and
\item code the values by a Huffman or an arithmetic coder.
\end{enumerate}
The first and third points are reversible, while the second is not.
The distortion will depend on the way the coefficients are quantized. We
may want to minimize the distortion with the minimum of bits, and a
trade-off will then be necessary in order to also have ``acceptable''
quality.  ``Acceptable'' is evidently subjective, and will depend
on the application. Sometimes, any loss is unacceptable, and the price
to pay for this
is a very low compression ratio (often between one and two).

\newpage
\section{Compression Methods}
We review briefly here different methods:
\begin{itemize}
\baselineskip=0.4truecm
\item{HCOMPRESS: }
{\tt HCOMPRESS} \cite{compress:white92} was developed at
 Space Telescope Science Institute (STScI),
Baltimore, and is commonly
used to distribute archive images from the Digital Sky Survey.
It is based on the Haar wavelet transform. The algorithm consists of
\begin{enumerate}
\item applying a Haar transform to the data,
\item quantizing the wavelet coefficients linearly as integer values,
\item applying a quadtree to the quantized values, and
\item using a Huffman coder.
\end{enumerate}

\item{HCOMPRESS + iterative decompression: }
Iterative decompression was proposed \cite{compress:bijaoui96}
to decompress files which
were compressed using HCOMPRESS. The idea is to consider the decompression
problem as a restoration problem, and to add constraints on the solution in
order to reduce the artifacts.

\item{FITSPRESS: }
{\tt FITSPRESS} \cite{compress:press92} is based on a wavelet transform, using
Daubechies-4 filters.
It was developed at the Center for Astrophysics, Harvard.

\item{JPEG: }
JPEG is the standard video compression software
for single frame images \cite{compress:furht95}.
It decorrelates pixel coefficients within 8 by 8 pixel blocks using the
Discrete Cosine
Transform and uniform quantization.

\item{Wavelet: }
Bi-orthogonal wavelet transform. The 7/9 filter is one of the best filters.
\cite{wave:antonini92}

\item{Fractal: }
The image is decomposed by block, and each block is represented by a fractal.
See \cite{compress:fisher94} for more explanation.

\item{Mathematical morphology: }
This method is based on mathematical morphology \cite{compress:huang91_1} (erosion and dilation).
It consists of detecting structures above a given level, the level being
equal to the background plus three times the noise standard deviation.
Then, all structures are compressed using erosion and dilation, and
additionally a quadtree and Huffman coding. This method is based on object
detection, and leads to a high compression ratio if the image does not contain
a lot of information, as is often the case in astronomy.

\item{Pyramidal median transform: }
The principle of this compression method is to select the
information we want to keep,
by using the pyramidal median transform, and to code this information without
 any loss. Thus
the first phase searches for the minimum set of quantized multiresolution
coefficients which produce an image of ``high quality''. The quality is
evidently subjective, and we will define by this term an image such as
the following:
\begin{itemize}
\baselineskip=0.4truecm
\item there is no visual artifact in the decompressed image,
\item the residual (original image -- decompressed image) does not
contain any structure.
\end{itemize}
Lost information cannot be recovered, so if we do not accept any loss,
we have to compress what we take as
noise too, and the compression ratio will be low (3 or 4 only).
\end{itemize}

The PMT is similar to the mathematical morphology method in the sense that
both try to understand what is in the image, and to compress only what is
considered as significant. The PMT uses a multiresolution approach, which
allows more powerful separation between signal and noise. Both methods have
been developed for astronomical image compression. They certainly can also
been used for medical, or biological images.

Concerning computation time for
such methods, it is in general very competitive,
of the order of a few seconds for a 1024 $\times$
1024 integer (16 bits) image
(machine Sun Ultra-Enterprise, 250 MHz and 1 processor). Only the methods
producing a fixed compression ratio take more time, because they need to
iterate in order to obtain a good compression ratio.

The \proj compression program allows us to use any of the previously
described strategies. User parameters define the chosen compression ratio,
the noise model, etc. For
large images, compression can be performed by blocks, and block size is a
parameter. This allows us to decompress a given block (or area) at a given
resolution, without having to read the whole file. The file format has been
specified in order to gain  direct access to a given block at a given
resolution using the C command ``fseek''. Available methods are
bi-orthogonal wavelet transform, Haar and Feauveau wavelet transform,
PMT, mathematical morphology.

%======================================================================

\section{Large Image Visualization Environment: LIVE}

With new technology developments, images furnished by detectors are larger
and larger. For example, current astronomical projects plan to deal with images
of sizes larger than 8000 by 8000 pixels (VLT: 8k $\times$ 8k, MegaCam
16k $\times$ 16k, etc.). A digitized mammogram film may lead to images
of about 5k $\times$ 5k.
Analysis of such images is obviously not easy, but the main problem is clearly
that of archiving and  network access to the data by users.

In order to visualize an image in a reasonable amount of time, the transmission
is based on two concepts:
\begin{itemize}
\item data compression
\item progressive decompression
\end{itemize}
The concept of progressive transmission has appeared in recent years. It
consists
of visualizing quickly a low resolution image, and then
increasing the quality
with the arrival of new bits. Wavelet based methods become very attractive
for this,
because they integrate in a natural way this multiresolution concept. A coarse
resolution image can be directly obtained without
having to decompress the whole file.

But with very large images, a third concept is necessary, which is
the region of
interest. Indeed, images are becoming so large it is impossible to
display them in a
normal window (typically of size 512 $\times$ 512), and we need to
have the ability to
focus on a given area of the image at a given resolution. To move from one
area to another, or to increase the resolution of a part of the area is a
user task, and is a new active element of the decompression. The goal of LIVE
is to furnish this third concept.

An alternative to LIVE for extracting a region of the image would be to
let the server extract the selected area, compress it, and send it to the user.
This solution is simpler and does not need any development, but presents
several drawbacks:
\begin{itemize}
\item Server load: the server must decompress the full size image and
re-compress the selected area on each user request.
\item Transfer speed: indeed, to improve the resolution of a
256 $\times$ 256 image to
a 512 $\times$
512 image, the number of bits using the LIVE strategy is of the order of 10
to 100 less than if the full 512 $\times$ 512 is transferred. The
lower the compression ratio is, the more the LIVE strategy becomes important.
\end{itemize}

The principle of LIVE is to use the previously described technology, and
to add the following functionalities:
\begin{itemize}
\item Full image display at a very low resolution.
\item Image navigation: the user can go up (the quality of an area of the
image is improved) or down (return to the previous image).
%Then the new image
%represents only a quarter of the previous one.
Going up or down in resolution implies a four-fold increase or decrease
  in the size of what is viewed.
\end{itemize}

\begin{figure}[htb]
\centerline{
\hbox{
\psfig{figure=fig_live.ps,bbllx=2.5cm,bblly=7cm,bburx=18cm,bbury=23cm,height=15cm,width=15cm,clip=}
}}
\caption{Example of large image, compressed by block, and represented at five
resolution levels. At each resolution level, the visualization window is
superimposed at a given position. At low resolution, the window covers the
whole image, while at the full resolution level, it covers only one block.}
\label{fig_live}
\end{figure}

In order to illustrate this concept, see Figure~\ref{fig_live}.
A large image (say 4000 $\times$ 4000),
which is compressed by blocks (8 $\times$ 8, each block having a
size of 500 $\times$ 500),
is represented at five resolution levels.
The visualization window (of size 256 $\times$ 256 in our example)
covers the whole
image at the lowest resolution level (image size 250 $\times$ 250),
but only one block
at the full resolution (in fact between one and four,
depending on the
position in the image). The LIVE concept consists of moving the visualization
window into this pyramidal structure, without having to load into memory the
large image. The image is first visualized at low resolution, and the user
can indicate (using the mouse) which part of the visualized subimage he
wishes to zoom on. At each step, only wavelet coefficients of the
corresponding blocks
and of the new resolution level are decompressed.

An IDL program is available (XLIVE) in \proj which allows manipulation and
visualization of large images. Future versions of \proj will also
include the LIVE
concept in Java, allowing use over the network in client-server mode.

% =========================== PROG ===========================
\clearpage
\newpage

\section{Compression/Decompression Programs}
Three image compression programs are available:
\begin{itemize}
\item {\em mr\_comp}: lossy image compression by several methods. This
program is well adapted to the compression of images contaminated by noise.
The pyramidal median transform (default method in {\em mr\_comp})
is very efficient for astronomical image compression.
\item {\em wcomp}: lossy image compression by the bi-orthogonal wavelet
transform (7/9 filters), and non-uniform quantization. This program
is similar to {\em mr\_comp} with specific options (i.e. ``-N -m 5 -s 1'').
It also allows the user to evaluate the PSNR between the original image
and the decompressed one.
\item {\em mr\_lcomp}: lossless image compression using the lifting scheme
method.
\end{itemize}
These programs furnishes a ``.MRC'' file format, and the decompression
can be performed using the {\em mr\_decomp} program.

\subsection{Image Compression: wcomp}
\index{wcomp}
Program {\em wcomp} compresses an
image by the bi-orthogonal wavelet transform.

For very large images, the compression can be carried out by block
using the ``-C''
option.  Then, each block is considered as an independent image, and
compressed. Only one block need be in memory for this approach. Another
advantage is the fact that a block can be decompressed alone, without
decompression of the whole image.
{\bf
\begin{center}
 USAGE: wcomp option image\_in [compressed\_file\_out]
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-g QuantifParam]}  \\
 Quantization parameter.
\item {\bf [-n number\_of\_scales]}  \\
Number of scales used in the multiresolution transform. Default is 6.
% \item {\bf [-s NSigma]}  \\
% Thresholding at NSigma * SigmaNoise. Default is 3.
\item {\bf [-E]} \\
 Evaluation mode. Once the compression is finished,
 decompression is performed, and the PSNR between the
 original and decompressed image is calculated.
\item {\bf [-C BlockSize] } \\
Compress by block. {\em BlockSize} is the size of each block.
Default is not to compress blockwise.
\item {\bf [-v]} \\
Verbose. Default is no verbose output.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\item wcomp image.d \\
Compress the image, and store the result ``image.d.MRC"
\item wcomp -g 30 image.d \\
Ditto, but increase the compression rate.
\item mr\_comp -E image.d comp.MRC\\
Compress the image, store the result in ``comp.MRC'', and calculate
the PSNR between the original and the decompressed image.
\item mr\_comp -C 512 image.d \\
Compress the image by block of size 512x512.
\end{itemize}

\subsection{Noisy image Compression: mr\_comp}
\index{mr\_comp}

Program {\em mr\_comp} compresses an
image by the pyramidal median transform (PMT)
\cite{starck:sta96_2,starck:sta96_3,starck:mur98_2,starck:mur96_1},
by mathematical morphology \cite{compress:huang91_1}, or by wavelet
transforms.

The PMT method uses noise modeling,
assuming Gaussian, Poisson, or Gaussian + Poisson noise. By default, the
noise is not compressed, and the decompressed image will look like a
filtered image. Option ``-s'' fixes the detection level, for the separation
 between signal and noise, and option ``-q'' specifies the quality we want
 when the detected multiresolution coefficients are coded.
 If the user wants to  keep the noise, two options allow him or her
to do this.
  Option ``-l'' for lossless compression (see also the
{\em mr\_lcomp} program),
and
option ``-r'' when an error on the noise reconstruction is acceptable. For
the second case, option ``-e'' fixes the quality of the noise reconstruction.
By default, all operations are done using float. By using the ``-O'' option,
all operations are done using integer (the program runs faster),
 and the input image is first converted into an integer image.
If the input image is coded with
floating values, some information may be lost during this conversion.
If no output file name is given, the program uses the input file name, and
adds to it the suffix ``.MRC''.
% On VMS machines, the output file name must always be given.

For the mathematical morphology method, only  ``-O'', ``-S'' and ``-D''
options are valid, ``-S'', ``-D'' allowing us to change
the structuring element and its size.

For other compression methods, options are the same as for PMT, but ``-O''
is not valid.

If the data does not contain any noise, option ``-N" must be set. In this case,
the compression ratio cannot be determined from the significant coefficients.
Then, the parameter ``-q'' can be used for increasing the compression ratio,
or the compression ratio can also be fixed, using the ``-R'' option. This
option implies that we need to iterate in order to get an optimal distortion
for the given compression ratio. For this reason the ``-R''
option requires more
computation time.

Depending on the kind of data, {\em mr\_comp} allows different strategies
to be taken. For astronomical and medical
images, which contain noise and diffuse
objects, the PMT transform gives good results. For an image without any noise
and containing edges, the bi-orthogonal wavelet transform (option -m 5)
is better. For  lossless compression (i.e.\ output decompressed image =
input image), the min-max transform can also be considered.

For very large images, the compression can be carried out by block
using the ``-C'' option (see previous section).

{\bf
\begin{center}
 USAGE: mr\_comp option image\_in [compressed\_file\_out]
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-m Compression\_Method] }
\begin{enumerate}
\baselineskip=0.4truecm
\item Compression by the pyramidal median transform
\item Compression by mathematical morphology
\item Compression by the Haar transform
\item Compression by the min-max transform (or G transform)
\item Compression by the Mallat-Daubechies transform
\item Compression by the Feauveau transform
\item Compression by mixed WT and PMT method
\end{enumerate}
Default is Compression by the pyramidal median transform.
\item {\bf [-g SigmaNoise]}  \\
SigmaNoise = Gaussian noise standard deviation. Default is automatically estimated.
\item {\bf [-p]} \\
Poisson Noise. Default is no Poisson component (hence just Gaussian).
\item {\bf [-c gain,sigma,mean]}  \\
See section~\ref{sect_support}.
\item {\bf [-n number\_of\_scales]}  \\
Number of scales used in the multiresolution transform. Default is 6.
\item {\bf [-s NSigma]}  \\
Thresholding at NSigma * SigmaNoise. Default is 3.
\item {\bf [-r]} \\
If -r option is given, the noise is compressed
with a step of sigma\_noise/2. By default, the noise is not conserved.
\item {\bf [-k]} \\
Keep isolated pixels in the support at the first scale. Default is not to do
this.
If the point spread function is defined on only one pixel,
this option should be set.
\item {\bf [-l]} \\
Save the noise (for lossless compression). Default is not to do this.
\item {\bf [-q SignalQuantif]} \\
The signal is quantized by Sq = S/(SignalQuantif*Sigma\_Noise).
By default, SignalQuantif = 1.5
\item {\bf [-e NoiseQuantif]} \\
The signal is quantized by Nq = S/(NoiseQuantif*Sigma\_Noise).
By default, NoiseQuantif = 0.5.
\item {\bf [-f]} \\
Keep all the header in the case of FITS format files. Default is not to do
this.
\item {\bf [-b bad\_pixel\_value]} \\
All pixels with this value will be considered as bad pixels,
and not used for the noise modeling.
\item {\bf [-O]} \\
Optimization. If set, the program works with integer instead of float,
and the execution time is faster.
\item {\bf [-B]} \\
Optimization without BSCALE (rescaling) operation in the case of FITS images.
\item {\bf [-P]} \\
Keep only positive coefficients.
\item {\bf [-W]} \\
Set the Median window size equal to 5. Default is 3.
\item {\bf [-S]} \\
Use a square structuring element. (Only for mathematical
morphology compression).
 Default structuring element is a circle.
\item {\bf [-D Dim]} \\
 Dimension of the structuring element (Only for mathematical
morphology compression).
 Default is 5.
\item {\bf [-R Compression\_Ratio]} \\
 Fixes the compression ratio. Default is not to do this.
\item {\bf [-C BlockSize] } \\
Compress by block. {\em BlockSize} is the size of each block.
Default is No.
\item {\bf [-i NbrIter]} \\
Method 3 (using Haar transform) can be improved by iterating. {\em NbrIter}
fixes the number of iterations. Default is not to do this.
\item {\bf [-N]} \\
Do not use noise modeling.
\item {\bf [-v]} \\
Verbose. Default is no verbose output.

\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\item mr\_comp image.d \\
Compress the image, and store the result ``image.d.MRC"
\item mr\_comp -f image.fits icompress\\
Compress the image, and store the result ``icompress.MRC". The
complete FITS header is saved in the compressed file.
\item mr\_comp -m 5 -R 30 image.d \\
Compress the image with a compression ratio of 30, using a bi-orthogonal
wavelet transform.
\item mr\_comp -m 5 -q 30 image.d \\
Compress the image, fixing the quantization parameter, and
using a bi-orthogonal
wavelet transform.
\item mr\_comp -O -m 4 -N image.d \\
Apply optimized lossless compression.
\end{itemize}

\newpage
\subsection{Compression: mr\_lcomp}
\index{mr\_lcomp}
Program {\em mr\_lcomp} compress an image by an integer
wavelet transform via the lifting scheme. The integer wavelet
coefficients are
coded without losing any information. This leads to a lossless compression
method (if the input image contains integer values!).  Five
transforms are available.

{\bf \begin{center}
 USAGE: mr\_lcomp option image\_in [compressed\_file\_out]
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-m Compression\_Method] }
\begin{enumerate}
\baselineskip=0.4truecm
\item  Lifting scheme: median prediction.
\item  Lifting scheme: integer Haar WT.
\item  Lifting scheme: integer CDF WT    .
\item  Lifting scheme: integer (4,2) interpolating transform.
\item  Lifting scheme: integer 7/9 WT.
\end{enumerate}
Default is Lifting scheme: integer Haar WT.
\item {\bf [-n number\_of\_scales]}  \\
Number of scales used in the multiresolution transform. Default is 6.
\item {\bf [-f]} \\
Keep all the header in the case of FITS format files. Default is not to do
this.
\item {\bf [-B]} \\
Optimization without BSCALE (rescaling) operation in the case of FITS images.
\item {\bf [-v]} \\
Verbose. Default is no verbose output.
\item {\bf [-C BlockSize] } \\
Compress by block. {\em BlockSize} is the size of each block.
Default is No.
\end{itemize}

\newpage
\subsection{Compression strategies}

\subsubsection*{Compression method}
If the images contain isotropic features (the case of
medical, biomedical, and astronomical images), the PMT (with {\em mr\_comp})
transform
is a good choice. If the image contains edges, a bi-orthogonal transform should
be used (option ``-m 5'' with {\em mr\_comp}, and ``-m 2'' or ``-m 3'' with
{\em mr\_lcomp}).

\subsubsection*{Lossless compression}
For lossless compress, {\em mr\_lcomp} is normally a better choice than
{\em mr\_comp}. However, {\em mr\_comp} separates signal from noise, which
can be in many cases useful. The Haar
lifting scheme method has also an advantage,
which is that a pixel in the decompressed image at a lower resolution
corresponds to the average of the pixel values in a square defined by
the resolution level. This allows the user to make correct scientific
measurement without having to decompress the whole image. For this reason,
the Haar lifting scheme compression method would be a good candidate for
a new data format.

\subsubsection*{Noise model}
By default, {\em mr\_comp} tries to separate the noise from the signal. If
the image is not noisy, this operation should not be carried out, and the
option ``-N'' must be set.

\subsubsection*{Compression ratio}
To obtain a fixed compression ratio, the option ``-R x'' (x being
the compression
ratio value) has to be given. The price to be paid
is that the compression takes
more time because it iterates until the correct compression ratio is obtained.

\subsubsection*{Large images}
For very large images, the ``-C'' option implies that block compression
will be performed. The image is separated into N $\times$ M blocks and
each block is
compressed separately. Blocks are directly read from  disk, so the
compression takes more time, but requires less memory space. A block can
be decompressed individually and at any resolution level, a
 property used by the IDL XLIVE program.

\newpage
\subsection{Decompression: mr\_decomp}
\index{mr\_decomp}
Program {\em mr\_decomp} decompresses a file compressed with {\em mr\_comp}
or {\em mr\_lcomp}.
We are not always interested in decompressing the image at full resolution.
The option ``-r'' allows an image at lower resolution to be extracted from
the compressed file, and produces in this way
a smaller image than the original. Even if the compressed file contains
the noise of the input image, the noise will not be decompressed.
Note that for lower resolution decompression, only the necessary part of the
file is read and decompressed (and so the decompression is particularly
efficient). This is possible because of the pyramidal compression method.
The option ``-g'' recreates a noise map, and adds it to the decompressed image.
The final image is evidently not closer to the original one, but the appearance
will be very similar.

{\bf
\begin{center}
 USAGE: mr\_decomp option CompressedFileName OutputFileName
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-B BlockNbr]} \\
Decompress only one block. {\em BlockNbr} is the block number to decompress.
Default is no.
\item {\bf [-r resolution]} \\
{\em resolution} must be $ \geq 0$ and $<$ number of scales of the transform.
By default, the image is reconstructed at full resolution with its noise
if this exists in the compressed file.
\item {\bf [-t output\_type]} \\
If the input image was a FITS format image, the image
output type can be fixed
by the user to ``i'' for integer, or ``s'' for short.
By default, the output type
is the same as the type of the original image.
\item {\bf [-g]} \\
Add simulated noise to the decompressed image with the same properties
as in the original image. This preserves appearances.
\item {\bf [-v verbose]} \\
Default is no.
\item {\bf [-I IterRecNbr]} \\
Use an iterative reconstruction. Only used with orthogonal transforms.
Default is no iteration.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\item mr\_comp ngc2997.fits \\
Compress the image, and store the result ``ngc2997.fits.MRC"
\item mr\_decomp ngc2997.fit.MRC decngc.fits\\
Decompress the file, and store the result ``decngc.fits".
\item cat ngc2997.fits.MRC $|$ mr\_decomp - - $>$ decngc.fits \\
Unix command, doing the same job.
\item mr\_decomp -r 1 ngc2997.fit.MRC  decngc.fits \\
Decompress the file at a resolution lower. The decompressed image
has  size 128 $\times$ 128 when the original one had a size of 256 $\times$
256.
\item mr\_decomp  -r 3 -i ngc2997.fit.MRC decngc.fits \\
Same as before, but the  decompressed image has size 32 $\times$ 32.
\end{itemize}

\newpage
\subsection{Decompression: mr\_upresol}
\index{mr\_upresol}
From an image which has been compressed by {\em mr\_comp} or {\em mr\_lcomp},
we can extract a low resolution image. The principle of {\em mr\_upresol}
is to improve the resolution, by a factor two, of the low resolution image.
Only wavelet coefficients of the
new scale are extracted from the compressed file (MRC format).
This program is used by the IDL widget program XLIVE.

If the image was compressed by block (``-C'' option using mr\_comp or
mr\_lcomp), then {\em mr\_upresol} allows us to focus on a region of the
image. The coordinates of the center of the region are given either by the
``-x'' and ``-y'' options, or by ``-X'' and ``-Y'' options. In the first
case, the coordinates must begin in the pixel units of the low resolution image,
while in the second case, units are those of the full resolution image.
The minimum size of the region to visualize is given by the option
``-W''. Only blocks covering the selected region will be decompressed.

 Two files are created. The new image file, and an information file (suffix
``.inf'') which is an ascii file and  which contains information about  the
decompressed image. It contains the three following lines:
\begin{verbatim}
L_Nl  L_Nc  Nl  Nc
Resol BlockSize KeepResi
FirstBlockNl LastBlockNl  FirstBlockNc  LastBlockNc
\end{verbatim}
where L\_Nl $\times$ L\_Nc is
the size of the full resolution image,  Nl $\times$ Nc is the size of
the decompressed image, Resol its resolution level, BlockSize is the block size
used for the compression,  KeepResi indicates whether the noise map is stored
in the compressed file, and FirstBlockNl,LastBlockNl,FirstBlockNc, LastBlockNc
indicates which blocks are decompressed.
{\bf
\begin{center}
USAGE: mr\_upresol options CompressedFileName OutputFileName
\end{center}}
where options are:
\begin{itemize}
\baselineskip=0.4truecm
\item {\bf [-t output\_type]} \\
If the input image was a FITS format image, the image
output type can be fixed  by the user to ``i'' for integer, or ``s''
for short.
By default, the output type  is the same as the type of the original image.
\item {\bf [-I IterRecNbr]} \\
Use an iterative reconstruction. Only used with orthogonal transforms.
Default is no iteration.
\item {\bf [-l LowResol\_Image\_FileName]} \\
Image at a lower resolution.
\item {\bf [-x PosX]} \\
Center pixel position (x) of the region to zoom
in the input low resolution image. \\
Default value is 0.
\item {\bf [-y PosY]} \\
Center pixel position (y) of the region to zoom
in the input low resolution image. \\
Default value is 0.
\item {\bf [-X PosX]} \\
Center pixel position (x) of the region to zoom
in the full resolution image (real pixel coordinate). Default value is 0.
\item {\bf [-Y PosY]} \\
 Center pixel position (y) of the region to zoom
 in the full resolution image (real pixel coordinate).
 Default value is 0.
\item {\bf  [-W WindowSize]} \\
 Window size. Default is 256.
\item {\bf [-v verbose]} \\
Default is no.
\end{itemize}
\subsubsection*{Examples:}
\begin{itemize}
\baselineskip=0.4truecm
\item mr\_lcomp -n 4 ngc2997.fits ngc2997.fits.MRC \\
mr\_upresol ngc2997.fits.MRC resol3.fits \\
mr\_upresol -l resol3.fits ngc2997.fits.MRC resol2.fits\\
mr\_upresol -l resol2.fits ngc2997.fits.MRC resol1.fits\\
mr\_upresol -l resol1.fits ngc2997.fits.MRC resol0.fits\\
Compress first image, decompress it resolution by resolution.
\item mr\_lcomp -n 4 -C 512 ngc2997.fits ngc2997.fits.MRC \\
mr\_upresol -X 200 -Y 250 -W 400 -l resol3.fits ngc2997.fits.MRC resol2.fits\\
mr\_upresol -X 200 -Y 250 -W 400  -l resol2.fits ngc2997.fits.MRC resol1.fits\\
mr\_upresol -X 200 -Y 250 -W 400  -l resol1.fits ngc2997.fits.MRC resol0.fits\\
Compress first image by block of 512. An image is created for each resolution
level, which contains the region around the pixels of coordinates  (200,250).
The ``-W'' options specifies the size of the region of interest (here 400 pixels).
At each level, only blocks covering this surface area are decompressed.
\end{itemize}


\documentclass[a4paper,twoside,10pt]{report}

%% Language %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage[USenglish]{babel} %francais, polish, spanish, ...
\usepackage[T1]{fontenc}
\usepackage[ansinew]{inputenc}

\usepackage{lmodern} %Type1-font for non-english texts and characters
\usepackage{mathtools}
%% Packages for Graphics & Figures %%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx} %%For loading graphic files
\usepackage{subfig} %%Subfigures inside a figure
\usepackage{geometry}

\usepackage{clrscode3e}

%\usepackage{pst-all} %%PSTricks - not useable with pdfLaTeX

%% Math Packages %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{amssymb}

%% Page Margins etc %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\setlength{\textheight}{24cm}
\setlength{\textwidth}{16cm}
\setlength{\topmargin}{-1cm}
\setlength{\oddsidemargin}{0pt}
\setlength{\evensidemargin}{0pt}

%%%%%% Misc %%%%%%
\newcommand{\blank}[1]{\hspace*{#1}\linebreak[0]}
\DeclarePairedDelimiter{\ceil}{\lceil}{\rceil}

\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}
\newcommand{\abs}[1]{\lvert#1\lvert}
\newcommand{\marginline}{\noindent\makebox[\linewidth][r]{\rule{\textwidth}{1pt}}}
\newcommand{\exer}[1]{\noindent{\Large\textsc{Question #1}} \\ \marginline}
\newcommand{\exernoline}[1]{\textcolor{Dandelion}{\LARGE{Exercise #1}}}
\newcommand{\note}[1]{\textcolor{OrangeRed}{\bf{#1}}} % errors and/or other things to consider
\newcommand{\lemma}[2]{\noindent \textit{\textbf{Lemma #1}}. #2}
\renewcommand{\qedsymbol}{\blacklozenge}

% New Environments %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newsavebox{\selvestebox}
\newenvironment{prob}
  {\newcommand\colboxcolor{FFFFFF}%
   \begin{lrbox}{\selvestebox}%
   \begin{minipage}{\dimexpr\columnwidth-2\fboxsep\relax}}
  {\end{minipage}\end{lrbox}%
   \begin{center}
   \colorbox[HTML]{\colboxcolor}{\usebox{\selvestebox}}
   \end{center}}

\newenvironment{proof*}{\begin{proof}[\textit{\textbf{Proof}}]}{\\ \end{proof}}
\newenvironment{soln}{\begin{proof}[\textit{\textbf{Solution}}]}{\\ \end{proof}}

\newenvironment{exercise}[1]{\exer{#1}}{}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \[\begin{tikzpicture}
% \draw[step=0.5cm,color=gray] (-1.5,-1.5) grid (1, 1);
% \begin{pgfonlayer}{myback}
% \fhighlight{m-1-1}{m-3-3}
% \fhighlight[green!30]{m-3-9}{m-5-13}
% \end{pgfonlayer}
% \end{tikzpicture}\]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\begin{exercise}{1b}
%\begin{soln}0
Consider the following algorithm:
\begin{codebox}
\Procname{\(\proc{Update-MST}(V, E, w, T, e_0, w_0)\)}
\li \If $e_{0} \in T$
    \Then
    \li \Comment Suppose $e_{0} = (u_{0}, v_{0})$
    \li \For $v \in V$
        \Do
        \li Augment $v$ to keep track of a representative vertex.
        \End
    \li Run DFS on $T - \{e_{0}\}$ starting from $u_{0}$, where for each encountered vertex $u$, set $u$'s representative to $u_{0}$
    \li Run DFS on $T - \{e_{0}\}$ starting from $v_{0}$, where for each encountered vertex $v$, set $v$'s representative to $v_{0}$
    \li best-weight = $\infty$
    \li best-node = NIL
    \li \For $e \in E - \{e_{0}\}$
        \Do
        \li \Comment suppose $e = (u, v)$
        \li \If representative($u$) $\neq$  representative($v$) and $w(e) <$ best-weight
            \Then
            \li best-weight $\leftarrow w(e)$
            \li best-node $\leftarrow e$
            \End
        \End
    \li \If best-weight = $\infty$
        \Then
        \li \Return $\const{NIL}$
    \li \Else
        \li \Return $T - \{e_0\} \cup$ best-node
        \End
\li \Else
    \li  \Return $T$
    \End
\end{codebox}

\section{Runtime Analysis}
If the removed edge $e_{0}$ is not in $T$, then $T$ is still a valid MST for $G$, and the algorithm returns $T$ which is $O(1)$. Now let's consider the case where
$e_{0} \in T$.\\
\begin{enumerate}
\item Augmenting all vertices in $V$ to keep track of a
representative involves allocating $O(n)$ memory, which is takes $O(n)$ time.
\item Since $T$ is an MST, removing an edge $e_{0}$ from $T$ creates two disjoint trees, say $T_{1}$ and $T_{2}$, such that $|T_{1}| + |T_{2}| = T - 1$. Then running DFS on $T_{1}$ and $T_{2}$ and setting representatives (which is possible to implement in constant time) on lines 7-8 has a total runtime of $O(n)$.
\item Each loop of the for loop starting on line 9 takes a constant number of operations (accessing the representatives, comparing values, and setting values), which means that the entire for loop is $O(m)$
\end{enumerate}
Hence, the total worst case runtime is $O(n+m)$ which in practice is faster than $O(m\log^{*}n)$.

\section{Proof of Correctness}

We limit our attention to the case in which $e_0 \in T$.

If there is no edge in $E$ which connects the disjoint trees created by the deletion of $e_0$ in $T$ ($V_1$ and $V_2$ in the algorithm), then removing $e_0$ also causes a disconnection in $G$ since by definition this means that the edges in $E$ but not in $T$ do not connect $V_1$ and $V_2$. In this case, it is impossible to construct a spanning tree, and so there is no solution, which is what is returned in the algorithm in this case (it returns NIL if it iterates through the edges and cannot find an edge connecting $V_1$ and $V_2$).

%Lemma 1: If there is no edge between the disconnection of %$T$ given by $(C_1, C_2)$ except $e_0$, then removing %$e_0$ from $E$ disconnects $G$.

Removing $e_0$ from $T$ yields two connected components $C_1$ and $C_2$, since $T$ was a minimum spanning tree (property). The algorithm simply finds the minimum cost edge $e$ between the two connected components, and adds this to $T - \{e_0\}$ as its result. Suppose, for the sake of contradiction, that the resulting tree $T_1 := T - \{e_0\} \cup \{e\}$ is not a minimum spanning tree. Then there must exist some other tree $T'$ (which is minimum) such that $w(T') < w(T)$. Consider the following cases:
\begin{enumerate}
\item \underline{Case 1: $e \notin T'$} \\
Then there must be some other edge $e'$ which connects a vertex from one of the edges in $C_1$, call it $u'$, to a vertex from one of the edges in $C_2$, call it $v'$ (otherwise there would be a disconnection and $T'$ would not be spanning). Now, by adding $e_{0}$ and removing $e'$ from $T'$, we obtain a tree $T'' \in E$ such that $w(T'') = w(T') - w(e') + w(e_0)$. We know that $w(T') = w(T) - w(e_0) + w(e)$ and, since based on the algorithm $e$ is cost edge connecting a vertex from one of the edges in $C_1$ to a vertex from one of the edges in $C_2$, we also know that $w(e) \leq w(e')$. Then $w(T'') = w(T) - w(e_0) + w(e) - w(e') + w(e_0)$ where $w(e) - w(e') \leq 0$, which means that $w(T'') \leq w(T)$, leading to a contradiction since $T$ is an MST for $G$.
\item \underline{Case 2: $e \in T'$} \\
Then at least one of the subtrees in $T'$ corresponding to $C_1$ and $C_2$ (i.e. the subtrees that contain the same vertices as in $C_1$ and $C_2$), denote them as $C_1', C_2'$ have a weight less than its counterpart. Suppose without loss of generality that $w(C_1)' < w(C_1)$. Then $w(C_1' \cup C_2 \cup e_0) < w(C_1 \cup C_2 \cup e_0) = w(T)$ and we have a contradiction since this gives a tree $T'' \subseteq E$ with a weight lower than $T$ which is an MST for $G$.
\end{enumerate}
Since either case leads to a contradiction, there does not exist a tree $T'$ where $w(T') < w(T)$. Hence, T is an MST.
%\end{soln}
\end{exercise}
\end{document}

CS/CSC373/files of other terms/CSC373 2009-2019/2015/assignments/assignment 1/my work/a1q1b.tex


\documentclass{article}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{hyperref}

\hypersetup{
  colorlinks, linkcolor=red
}

\begin{document}

\title{CS838 Lab 2}
\author{Sek Cheong}
\maketitle

%\begin{abstract}
%The abstract text goes here.
%\end{abstract}

\section{Introduction}
The purpose of this experiment was to design a neural network with single hidden layer to predict protein secondary structures. The data set we used in this experiment was the \href{https://archive.ics.uci.edu/ml/datasets/Molecular+Biology+(Protein+Secondary+Structure)}{Molecular Biology (Protein Secondary Structure) Data Set} obtained from the UC Irvine Machine Learning Repository. We used convolution with the window of 17 amino for the input train the network with the middle amino secondary structure. The overall accuracy of our network was about $62\%$ obtained in less than 5 epochs on average.

\section{The Data Set}
The data set from UC Irvine was composed of two files one for training and one for testing. We combine these two files into a single file. The combined file was then split into training set, tuning set, and test set.  The test set was a collection of every 5th example in combined file. The tune set was a collection of every 6th example in the combined file. The remaining examples in the combined file were used as the training set.

\section{The Experiment}
We used all the examples in the training set for each epoch in training of the network. The early stop criteria was set on the difference of accuracy on the tuning set of previous epoch and the current epoch. When the difference is greater or equal than $0.001$ we stop the training.

\begin{equation}
\epsilon = accuracy(tune,epoch-1) - accuracy(tune, epoch) >= 0.001
\end{equation}

We tested the network with various number of hidden units and learning rate. We also tested the network unregularized and regularized with learning rate, momentum, and weight decay. Following were our findings.

\subsection{Unregularized}
We used the following setting for our network for this experiment:

\begin{equation*}
HU=9, \eta=0.01, \epsilon=0.001, \alpha=0, \lambda=0, MaxEpoch=100
\end{equation*}

Amount the experiments the unregularized network (with other setting being equal) gave as the lowest accuracy. We were only able to obtain on average $56.7\%$ accuracy.

\subsection{Learning Rate}
The learning rate $\eta$ contribute greatly to the overall accuracy of the network prediction. We experimented $\eta$ from $0.00050$ to $0.09$. The following graph shows the $\eta$ vs. accuracy:

\begin{center}
 \includegraphics[width=3.0in]{learningrate}
\end{center}

To make the comparison meaningful, we used the same setting as in 3.1, we only change the value of $\eta$ for each run. As you can see from the graph, setting $\eta$ too small or too large could adversely affect network accuracy. In out case the optimal $\eta$ was somewhere between $0.008$ and $0.009$.

\subsection{Number of Hidden Units}
The number of hidden units also plays a significant role in the network accuracy. If the network have to few hidden units the network will not have the desired capacity to be trained to predict the give sample distribution. If the network has too many hidden units overfitting is likely to occur.

\begin{center}
 \includegraphics[width=3.0in]{hiddenunits}
\end{center}

In out case the optimal number of hidden units was between $7$ and $10$ units. When the number of hidden units increase the network accuracy plateaued and steadily dropping lower.

\subsection{Momentum}
The momentum $\alpha$ also has a significant affect on the accuracy of the network. The appropriate momentum range is $0<\alpha<1$. The empirical result shows that larger momentum with smaller $\eta$ gives yields pretty good result.

\begin{center}
 \includegraphics[width=3.0in]{momentum}
\end{center}

In out experiment we set the momentum somewhere between $0.75$ to $0.95$ yield the best result.

\subsection{Weight Decay}
The weight decay $\lambda$ improves the accuracy to some degree. The result suggested that once $\lambda$ reached $0.00006$ the accuracy was in steady decline.

\begin{center}
 \includegraphics[width=3.0in]{weightdecay}
\end{center}

%\begin{figure}
%    \centering
%    \includegraphics[width=2.0in]{learningrate}
%    \caption{Simulation Results}
%    \label{simulationfigure}
%\end{figure}

The weight decay was supposed to improve the accuracy by regularize the weight complexity of the network. When combined with the momentum $\alpha$ it did improve accuracy but not by much, however, the training time was noticeably reduced.

\section{Conclusion}
Our neural network performed pretty well and could achieve accuracy around $62\%$ consistently. The number of hidden units could significant affect the performance of the network. Too few hidden units, the network accuracy would suffer and too many hidden units would not improve network accuracy or even reduce the accuracy. The training time is in polynomial time proportion to the number of hidden units. Therefore, it is important to choose the right amount of hidden units to get the best balanced result. Our best setting is as following:
\begin{equation*}
HU=9, \eta=0.009, \epsilon=0.001, \alpha=0.90, \lambda=0.0006,  MaxEpoch=150
\end{equation*}
A network with properly tuned learning rate, momentum, and weight decay could significantly out perform networks without any regularization. The weight decay did not seem to be very useful in out experiment.

\end{document}


%        File: TDDD14-lecture-notex.tex
%     Created: Mon Apr 04 10:00  2016 C
% Last Change: Mon Apr 04 10:00  2016 C
%
\documentclass[a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{float}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{amssymb}
\usepackage{mathtools}
\newtheorem{definition}{Definition}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}
\newtheorem{example}{Example}
\title{Lecture Notes\\TDDD14}
\author{Pontus Persson}
\date{VT-16}

\begin{document}
\maketitle
\tableofcontents
\marginpar{Lecture 1\\2016-04-04}
\section{Introduction}
Course questions:
\begin{itemize}
    \item What is a computer?
    \item What is their computational power?
    \item Programming languages
    \item Compilers
\end{itemize}
Some answers:
\begin{itemize}
    \item Different computational models leads to different computational powers
    \item Yes, there are problems which a computer can not solve
\end{itemize}
\subsection{Automata}
\textbf{Course backbone:} Automata\\
We use them to define formal languages and more importantly, computational models.
\\Many applications:
\begin{itemize}
    \item Artificial intelligence
    \item Vending machines
    \item Traffic lights, turnstile
    \item Video games
\end{itemize}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{turnstile.png}
    \caption{Turnstile Automata}
    \label{fig:turnstile}
\end{figure}
We treat mainly three types of automata:
\begin{enumerate}
    \item Finite memory (finite automata, FAs, e.g. turnstile)
    \item Infinite memory with restricted access (push-down automata, PDAs)
    \item Infinite memory without restricted access.
\end{enumerate}
\subsection{Some terminology}
Should be familiar with:
\begin{itemize}
    \item Sets and operations with sets
    \item Functions
    \item Relations
    \item Cartesian product
    \item Some sense of order and partial orders
    \item Equivalence relation
    \item Graphs and trees
    \item Binary and decimal repr. of numbers
\end{itemize}
\section{Basic definitions}
\begin{definition}
    A \textbf{decision problem} is a function with a one bit output: ``yes'' or ``no''.\\
    To specify a decision problem, we need two things:
    \begin{itemize}
        \item The set A of possible inputs
        \item The subset $B\subseteq A$ of ``yes'' instances
    \end{itemize}
\end{definition}
\begin{example}
    Given n, is n odd? Is a decision problem\\
    $A=\mathbb{N}$, $B=\left\{ n\in\mathbb{N} \middle| n\equiv 1 (mod 2) \right\}$\\
\end{example}
\begin{example}
    Given a,b. What is the greatest common divisor? Is not a decision problem.
\end{example}
In this course: an input is always a string of some alphabet.
\begin{definition}
    An \textbf{alphabet} is any finite set.\\
    We call the elements of this set symbols or letters. An alphabet is typically
    denoted with $\Sigma, \Gamma$. Symbols are often denoted with $a,b,c,\ldots$
\end{definition}
\begin{definition}
    A \textbf{string} over $\Sigma$ is any finite-length sequence of elements of $\Sigma$.\\
    We usually denote strings x,y,z,w,u,v,\ldots
\end{definition}
\begin{example}
    If $\Sigma=\left\{ a,b \right\}$, then ``aabab'' is a string over $\Sigma$ of length 5.
\end{example}

\begin{definition}
    The \textbf{length} of a string x is the number of symbols in x.\\
    Denoted by $|x|$.
\end{definition}
\begin{definition}
    The \textbf{null string} of \textbf{empty string} is denoted by $\varepsilon$.\\
    $|\varepsilon|=0$
\end{definition}
\begin{definition}
    Some terminology.
    \begin{itemize}
        \item $a^n$ A string of a's of length n ($a^5=aaaaa$, $a^0=^{def}\varepsilon$)
        \item The set of all string over $\Sigma$ is denoted by $\Sigma^*$.
        \item $\emptyset^*=\left\{ \varepsilon \right\}$
        \item If $\Sigma$ is non-empty then, $\Sigma^*$ is an infinite set of
            finite-length strings.
        \item Note: $\emptyset,\left\{ \varepsilon \right\}, \varepsilon$ are three
            \textbf{different} things.
    \end{itemize}
\end{definition}
\begin{definition}
    \textbf{Concatenation} takes two strings and x,y and creates a new string xy
            by putting them together end to end. xy is called the concatenation of x and y.\\
            \textbf{Note:} $xy \neq yx$ in general.
            \begin{itemize}
                \item Concatenation is associative: $(xy)z=x(yz)$
                \item $\varepsilon$ is an identity for concatenation $\varepsilon x=x\varepsilon = x$
                \item $|xy|=|x|+|y|$
                \item $a^ma^n=a^{m+n}$
            \end{itemize}
\end{definition}
\begin{definition}
    We write $x^n$ for the string obtained by concatenating n copies of x.
\end{definition}
\begin{definition}
    If $a\in\Sigma$ and $x\in\Sigma^*$ we write $\#a(x)$ for the number
            of a's in x
\end{definition}
\begin{definition}
    A \textbf{prefix} of a string x is an initial substring of x.
\end{definition}
\begin{definition}
    A \textbf{suffix} of a string x is an ending substring of x.
\end{definition}
\begin{definition}
    The \textbf{complement} in $\Sigma^*$, $A\subseteq\Sigma^*$ is denoted by
    $~A=\left\{ x\in\Sigma^* \middle| x\notin A \right\}$, we denote it by $\hat{A}$.
\end{definition}
\begin{definition}
    Set concatenation $A,B\in\Sigma^*$, $AB=\left\{ xy \middle | x\in A, y\in B \right\}$
\end{definition}
\begin{definition}
    Any subset $A\subseteq \Sigma^*$ is called a \textbf{language}.
\end{definition}
\section{States and Transitions}
\marginpar{Lecture 2\\2016-04-06}
Intuitively, a \textbf{state} is a snapshot of a system (automata). A state gives
all relevant information necessary to determine how the system will evolve from
that point on.
\\\textbf{Transitions} are changes of state, they can happen spontaneously, or in response
to external inputs.
\\A system that consists of only finitely many states and transitions among
them is called a \textbf{finite state system}.
\\We will model these by a model called \textbf{finite automaton}.
\subsection{Deterministic finite automaton (DFA)}
Formally a \textbf{deterministic finite automaton} (DFA) is a structure.
\begin{align*}
    M=(Q,\Sigma,\delta,s,F)
\end{align*}
Where:
\begin{itemize}
    \item $Q$ is a finite set of states
    \item $\Sigma$ is a finite set, the input alphabet
    \item $\delta : Q\times\Sigma\rightarrow Q$ is the transition function
    \item $s\in Q$ the start state
    \item $F\subseteq Q$ is the set of final states (accept states)
\end{itemize}

\begin{example}
    3.1 in the book.\\
    a)\\
\begin{align*}
    Q&=\left\{ 0,1,2,3 \right\}\\
    \Sigma&=\left\{ a,b \right\}\\
    \delta(0,a)&=1\\
    \delta(1,a)&=2\\
    \delta(2,a)&=\delta(3,a)=3\\
    \delta(q,b)&=q,\mbox{, }q\in Q\\
    s&=0\\
    F&=\left\{ 3 \right\}
\end{align*}
b) (table representation)
\begin{table}[H]
    \centering
    \begin{tabular}{l|l l}
        & a & b\\\hline
        0 & 1 & 0\\
        1 & 2 & 1 \\
        2 & 3 & 2 \\
        3F & 3 & 3
    \end{tabular}
    \caption{Table representation}
\end{table}
c) (transition diagram)
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{transition-diagram.png}
    \caption{Transition diagram}
\end{figure}
This automaton excepts all strings/languages containing at least three ``a''s.\\
\end{example}
\subsubsection{State acceptance}
We define a function
\begin{align*}
    \hat{\delta}&: Q\times\Sigma^*\rightarrow Q
\end{align*}
by induction on the length of the string x:
\begin{align*}
    \hat{\delta}(q,\varepsilon)&=q\\
    \hat{\delta}(q,xa)&=\delta(\hat{\delta}(q,x),a)
\end{align*}
$\hat{\delta}$ is the multi-step version of $\delta$. $\delta$ and $\hat{\delta}$
agree on string of length one:
\begin{align*}
    \hat{\delta}(q,a)&= \hat{\delta}(q, \varepsilon a)\\
    &=\delta(\hat{\delta}(q,\varepsilon),a)\\
    &=\delta(q,a)
\end{align*}
A string $x$ is said to be accepted by the automaton $M$ if:
\begin{align*}
    \hat{\delta}(s,x)&\in F
\end{align*}
and rejected by automaton $M$ if:
\begin{align*}
    \hat{\delta}(s,x)\notin F
\end{align*}
where $s$ is the start state and $F$ is the set of final states of $M$.
\subsubsection{Accepted languages}
The set or language accepted by $M$ is the set of all strings accepted by $M$
, and is denoted $L(M)$.
\begin{align*}
    L(M)=\left\{ x\in\Sigma^*\middle | \hat{\delta}(s,x)\in F \right\}
\end{align*}
A language $A\subseteq \Sigma^*$ is said to be \textbf{regular} if $A=L(M)$
for some finite automaton $M$.\\
\textbf{NOTE:} The language accepted by the finite automaton from the example
above is regular.
\begin{example}
    3.2 in the book.\\
    \begin{align*}
        L_2&=\left\{ xaaay \middle | x,y\in \left\{ a,b \right\}^* \right\}\\
        &=\left\{ x\in\left\{ a,b \right\}^* \middle | x
        \mbox{ contains a substring of three consecutive a's} \right\}
    \end{align*}\\
    \begin{table}[H]
        \centering
        \begin{tabular}{r|l l}
            & a & a
            \\\hline
            $\rightarrow$0 & 1 & 0\\
            1 & 2 & 0\\
            2 & 2 & 0\\
            3F & 3 & F
        \end{tabular}
    \end{table}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.9\textwidth]{example3-2.png}
    \end{figure}
\end{example}
\begin{example}
    \begin{align*}
        L_3=\left\{ x\in\left\{ 0,1 \right\}\middle | \mbox{x represents a multiple of three in binary} \right\}
    \end{align*}
    Here is an automaton that accepts $L_3$:
    \begin{table}[H]
        \centering
        \begin{tabular}{r|l l}
            & 0 & 1\\\hline
            $\rightarrow$0F & 0 & 1\\
            1 & 2 & 0\\
            2 & 1 & 2
        \end{tabular}
    \end{table}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.9\textwidth]{example4.png}
    \end{figure}
    \textbf{Observations:}
    \begin{enumerate}
        \item append 0: multiply by 2
        \item append 1: multiply by 2 and add 1
    \end{enumerate}
    If the number represented by the string that is scanned so far is:
    \begin{itemize}
        \item $0 (mod 3)$
        \item $1 (mod 3)$
        \item $2 (mod 3)$
    \end{itemize}
    it will be in state 0, 1, 2 respectively.\\
    Property from Number Theory:
    \begin{align*}
        x,y,z &\in \mathbb{Z}\\
        \mbox{if } \\
        x&\equiv y (mod z)\\
        \mbox{then } \\
        ax+b &\equiv ay+b (mod x),\forall a,b\in\mathbb{Z}
    \end{align*}
Show that the transition will be true for all combinations of states and inputs.
\end{example}
\subsection{Determinism}
Determinism: At any step during the execution it is completely determined
what the system does next (depending on current state and input).
\subsection{Non-determinism}
Non-determinism: During the execution it is not always determined what to do
next, that means that a choice has to be made (a choice that does not depend
on current state nor input).
\subsection{Non-deterministic automaton (NFA)}
In a non-deterministic automaton the next state is not necessarily uniquely
determined by current state and input. There may be one, many, or even zero
next states. NFAs may have several start states.
\\An NFA works as a DFA, but in addition, whenever the next state is not uniquely
determined, it ``guesses'' what next state to assume.
\\Obviously, different guesses may lead to different computation paths and outcomes
(accept vs. reject).
\begin{figure}[H]
    \centering
    \includegraphics[width=0.6\textwidth]{nfa.png}
    \caption{An example of an NFA}
\end{figure}
\subsubsection{NFA acceptance}
An NFA is said to accept an input $x$ if at least one possible computation
path of input $x$ starting from at least one of the start states leads to a
final state.
\begin{example}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.9\textwidth]{nfa2.png}
        \caption{State diagram for example 7}
    \end{figure}
    Is an NFA because of the choice in state 1 and getting stuck in state 6 if
    there is input left.\\
    Input: 010111 can be accepted.\\
    Input: 100000 always rejected.
\end{example}
Can NFAs accept more languages than DFAs?\\
\textbf{NO!}\\
We will show how to construct for any NFA an equivalent DFA in the sense of
accepted language.
\subsubsection{NFA to DFA conversion}
\begin{example}
    5.1 in the book\\
    \begin{align*}
        A=\left\{ x\in\left\{ 0,1 \right\}^* \middle |
        \mbox{the second symbol from the right is 1} \right\}
    \end{align*}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.6\textwidth]{example5-1-1.png}
        \caption{NFA example}
    \end{figure}
    \begin{table}[H]
        \centering
        \begin{tabular}{r|l l}
            DFA & 0 & 1 \\\hline
            $\emptyset$ & $\emptyset$ & $\emptyset$\\
            $\left\{ q \right\}$ & $\left\{ p \right\}$ & $\left\{ p,q \right\}$\\
            $\left\{ p \right\}$ & $\left\{ r \right\}$ & $\left\{ r \right\}$\\
            $\left\{ r \right\}$ & $\emptyset$ & $\emptyset$\\
            $\left\{ p,q \right\}$ & $\left\{ p,r \right\}$ & $\left\{ p,q,r \right\}$\\
            $\left\{ p,r \right\}$ & $\left\{ p \right\}$ & $\left\{ p,q \right\}$\\
            $\left\{ r,q \right\}$ & $\left\{ r \right\}$ & $\left\{ r \right\}$\\
            $\left\{ p,q,r \right\}$ & $\left\{ p,r \right\}$ & $\left\{ p,q,r \right\}$
        \end{tabular}
        \caption{All possible subsets we could be in}
    \end{table}
    Cross off all states not reachable when starting from the start state.
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.7\textwidth]{example-5-2.png}
        \caption{DFA equivalent to previous NFA}
    \end{figure}
\end{example}
    \subsubsection{Formal definition of NFAs}
    An NFA is a five-tuble:
    \begin{align*}
        N&=\left( Q,\Sigma,\Delta,S,F \right)
    \end{align*}
    where:
    \begin{itemize}
        \item $S\subseteq Q$ is the set of start sets
        \item $\Delta: Q\time\Sigma^*\rightarrow 2^Q$
        \item All the rest as for the DFA definition
    \end{itemize}
    \textbf{Note:} $2^Q=\left\{ A\middle | A\subseteq Q \right\}$
        $\Delta(p,a)$ gives the set of all states that $N$ is allowed
        to move to from $p$ in one step ? input symbol $a$. The set $\Delta(p,a)$
        can be empty.\\
        $\Delta$ is called the transition function of $N$.
        \\We extent $\Delta$ to $\hat{\Delta}$ to the multistep version of $\Delta$.
        \begin{align*}
            \hat{\Delta}&:2^Q\times \Sigma^* \rightarrow 2^Q\\
            \hat{\Delta}(A,\varepsilon)&=A\\
            \hat{\Delta}(A,xa)&=\cup_{q\in\hat{\Delta}(A,x)}\Delta(q,a)
        \end{align*}
        For $A\subseteq Q$ and $x\in\Sigma^*$, $\hat{\Delta}(A,x)$ is the set
        of all states reachable under the input string $x$ from some state in $A$.
        \\\textbf{Note:} for $a\in\Sigma$
        \begin{align*}
            \hat{\Delta}(A,a)&=\cup_{p\in\hat{Delta}(A,\varepsilon)}\Delta(p,a)\\
            &=\cup_{p\in A}\Delta(p,a)
        \end{align*}
        The automaton $N$ is said to accept $x\in\Sigma^*$ if:
        \begin{align*}
            \hat{\Delta}(S,x)\cap F \neq \emptyset
        \end{align*}
        $L(N)$ is the set of all strings accepted by $N$.
        \begin{align*}
            L(N)=\left\{ x\in\Sigma^* \middle | \mbox{N accepts x} \right\}
        \end{align*}
        \textbf{Note:} Any DFA is equivalent to an NFA where:
        $\Delta(p,a) = \left\{ \delta(p,a) \right\}$.
        \begin{lemma}
            For any $x,y\in\Sigma^*$ and $A\subseteq Q$: $\hat{\Delta}(A,xy)=\hat{\Delta}(\hat{\Delta}(A,x),y)$
        \end{lemma}
        \subsubsection{The subset construction}
        Let
        \begin{align*}
            N=(Q_N,\Sigma,\Delta_N,S_N,F_N)
        \end{align*}
        be an arbitrary NFA.\\
        Let $M$ be the following DFA:
        \begin{align*}
            M=(Q_M, \Sigma, \delta_M, s_M, F_M)
        \end{align*}
        where:
        \begin{itemize}
            \item $Q_M = 2^Q$
            \item $\delta_M(A,a)=\hat{\Delta}_N(A,a)$
            \item $s_M=S_N$
            \item $F_M=\left\{ A\subseteq Q_N \middle | A \cap F_N \neq \emptyset \right\}$
        \end{itemize}

        \begin{lemma}
            For any $A\subseteq Q_N$ and $x\in\Sigma^*$
            \begin{align*}
                \hat{\delta}_M(A,x) = \hat{\Delta}_N(A,x)
            \end{align*}
            Proof via induction on the length of $x$.
        \end{lemma}

        \begin{theorem}
            The automata $M$ and $N$ accept the same language
            \begin{align*}
                x&\in L(M)\\
                \Leftrightarrow &\hat{\delta}_M(s_M,x)\in F_M\\
                \Leftrightarrow &\hat{\Delta}_N(S_M,x)\in F_M\\
                \Leftrightarrow &\hat{\Delta}_N(S_N,x)\in F_M\\
                \Leftrightarrow &\hat{\Delta}_N(S_N,x)\cap F_N \neq \emptyset\\
                \Leftrightarrow &x\in L(M)
            \end{align*}
        \end{theorem}
        \marginpar{Lecture 4\\2016-04-13}
        \subsection{$\varepsilon$-transition}
        We can have transitions without reading an input symbol.
        \begin{figure}[H]
            \centering
            \includegraphics{epsilon.png}
            \caption{$\varepsilon$-transition}
            \label{fig:epstrans}
        \end{figure}
        \begin{definition}
            $NFA_\varepsilon$\\
            $N_\varepsilon = \left( Q,\Sigma, \Delta, S, F \right)$\\
            as NFA, but $\Delta: Q\times (\Sigma\cup \left\{ \varepsilon \right\}
            \rightarrow 2^Q$\\
            Note: $\Sigma$ never contains $\varepsilon$!
        \end{definition}

        For $A\subseteq Q$\\
        $C_\varepsilon(A)$ = all states reachable from some $q\in A$ via an
        $\varepsilon$-transition.
        \begin{definition}
            \begin{enumerate}
                \item $A\subseteq C_\varepsilon(A)$
                \item if $q\in C_\varepsilon(A)$, then $\Delta(q,\varepsilon)
                    \subseteq C_\varepsilon(A)$
                \item No other states are in $C_\varepsilon(A)$
            \end{enumerate}
        \end{definition}
        \begin{definition}
            \begin{align*}
            \hat{\Delta}&: 2^Q\times \Sigma^* \rightarrow 2^Q\\
            \hat{\Delta}(A,\varepsilon)&=C_\varepsilon(A)\\
            \hat{\Delta}(A,xa)&=C_\varepsilon(\cup_{q\in\hat{\Delta}(A,x)}\Delta
            (q,a)
        \end{align*}
        Note:
        \begin{align*}
            \hat{\Delta}(A,a)=C_\varepsilon(\cup_{q\in C_\varepsilon(A)}\Delta(q,a)
        \end{align*}
        \end{definition}
        \subsubsection{DFA from $\varepsilon$-NFA}
        From $N_\varepsilon = \left( Q_{N_c}, \Sigma, \Delta_{N_c}, S_{N_c}, F_{N_c} \right)$
        we construct the DFA $M=\left(Q_M, \Sigma, \delta_M, s_m, F_m\right)$.\\
        where:
        \begin{align*}
            Q_M&=2^{Q_{N_c}}\\
            s_M&=C_\varepsilon(S_{N_c}\\
            \delta_M(A,a)&=\hat{\Delta}(A,a)\\
            F_M&=\left\{ A\subseteq Q_{N_c}\middle | A\cap F_{N_c} \neq \emptyset \right\}
        \end{align*}
        You can prove via induction that $\hat{\delta}(s_M,x)=\hat{\Delta}(S_{N_c},x)$
        and then you show $L(N_c)=L(M).$.
        \begin{figure}[H]
            \centering
            \includegraphics{nfaeps.png}
            \caption{$NFA_\varepsilon$}
            \label{fig:nfaeps}
        \end{figure}
        \begin{table}[H]
            \centering
            \begin{tabular}{r|l l|l|l| l l}
                & a & b & $\varepsilon$ & $C_\varepsilon$ & a & b\\\hline
                SF & $\emptyset$ & $\emptyset$ & 1 & $\left\{ 1 \right\}$ & $\left\{ s,1 \right\}$ & $\left\{  1,2,s\right\}$\\
                1 & 1 & 2 & $\emptyset$ & $\left\{ 1 \right\}$ &$\left\{ 1 \right\}$ & $\left\{ 1,2,S \right\}$\\
                2 & $\emptyset$ & $\emptyset$ & s & $\left\{ 2,s,1 \right\}$ & $\left\{ 1 \right\}$ & $\left\{ 1,2,s \right\}$
            \end{tabular}
        \end{table}
        \begin{figure}[H]
            \centering
            \includegraphics[width=0.8\textwidth]{dnanoeps.png}
            \caption{DFA from $NFA_\varepsilon$}
        \end{figure}
\section{Regular expressions}
\begin{definition}
    Given an alphabet $\Sigma$, the following are \textbf{regular expressions}:
    \begin{itemize}
        \item the empty set, denoted $\emptyset$
        \item the singleton set that contains the empty string
            $\left\{ \varepsilon \right\}$, denoted by $\varepsilon$
        \item Any singleton set that contains a symbol from $\Sigma$
    \end{itemize}
\end{definition}
Let $X,Y$ be two regular expressions. The following operations are used to
produce more complex regular expressions:
\begin{itemize}
    \item concatenation, denoted $XY$.
    \item alternation (set unioun), denoted $X+Y$
    \item Kleene star, denoted $X^*$
\end{itemize}
Parentheses are allowed to reduce ambiguity. To reduce the use of parenthesis,
the Kleene star has highest priority, then concatenation and last alternation.
\textbf{Examples:}\\
$\Sigma = \left\{ a,b \right\}$
\begin{enumerate}
    \item $ab+a=\left\{ ab,a \right\}$
    \item $\left( a+b \right)^*=\left\{ \varepsilon,a,b,aa,ab,ba,bb,aaa,aab,\ldots \right\}$
    \item $\left(a+\varepsilon\right)b^*=\left\{ \varepsilon,a,b,ab,abb,abbb,bb,bbb,\ldots \right\}$
\end{enumerate}
Alteration and concatenation are associative, therefore parenthesis can be
omitted in the following cases:
\begin{itemize}
    \item $(a+b)+c=a+b+c$
    \item $(ab)c=a(bc)=abc$
\end{itemize}
Alternation is also commutative, concatenation is not.\\
Some transformation laws. Let $\alpha,\beta,\gamma$ be regular expressions.
\begin{align}
    \alpha+\emptyset &= \alpha\\
    \alpha+\alpha&=\alpha\\
    \varepsilon \alpha &= \alpha\\
    \alpha(\beta + \gamma) &= \alpha\beta + \alpha\gamma\\
    (\alpha+\beta)\gamma &= \alpha\gamma + \beta\gamma\\
    \varepsilon + \alpha\alpha^* &= \alpha^*\\
    (\alpha\beta)^*\alpha &= \alpha(\beta\alpha)^*\\
    (\alpha^*\beta)^* &= (\alpha+\beta)^* = \alpha^*(\beta\alpha^*)^*\\
    (\varepsilon+\alpha)^* &= \alpha^*\\
    \alpha\alpha^* &= \alpha^*\alpha
\end{align}
\subsection{Transform regular expression to $NFA_\varepsilon$}
\begin{itemize}
    \item $\emptyset\rightarrow \emptyset$
    \item $\varepsilon \rightarrow $ one state that is both start and final
    \item $a\rightarrow$ a start state and a final state connected by input $a$
\end{itemize}
\subsection{From regex to $NFA_\varepsilon$}
\subsubsection{Concatenation}
Concatenation of $X,Y$. Let $M_x$ ans $M_y$ be automata accepting $X$ and $Y$, respectively.\\
Let $F_x$ be the final states of $M_x$ and $S_y$ be the start states of $M_y$.
Introduce a new state $s,$ and introduce $\varepsilon$-transitions from every
state in $F_x$ to $s$ and from $s$ to every state in $S_y$. Make the states
in $F_x$ non-final states. The new automata is called $M_{XY}$.

\subsubsection{Alternation}
Alternation of $X,Y$. Simply define $M_{X+Y}$ as $M_X$ $M_Y$ ``together''
(is the sense that they are still disconnected). Same automata but, disconnected.

\subsubsection{Kleene star}
$X^*$. Introduce a new final start state s. $\varepsilon$-transitions
from s to all start states of $M_X$ and $\varepsilon$-transitions from all
final state of $M_X$ to s.\\
\textbf{Example:} $L=\left\{a^nb \middle | n\geq 0 \right\}= \left\{ b,ab,aab,aaab,\ldots \right\}$\\
$L^*=\left\{ w\in \left\{ a,b \right\}^* \middle | \mbox{w ends with b or }w=\varepsilon \right\}$
\subsection{From DFA to regexp}
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{dfa2regex.png}
    \caption{DFA}
\end{figure}
Associate a language to each state.\\
$A_i=L(q_i)=\left\{ x\in \Sigma^* \middle | \hat{\delta}(q_i,x)\in F \right\}=$
strings that move M from state $q_i$ to some final state.\\
Idea: express these languages in terms of each other, using regular expression-operations.
\begin{align}
    A_1&=aA_1+bA_2+\varepsilon\\
    A_2&=bA_2+aA_3\\
    A_3&=aA_1+bA_2
\end{align}
Then solve the equations! In particular we are interested in $A_1=L(M)$.
\\How to solve it? \textbf{Ardm's Theory}: \\Let $A,B$ languages, $\varepsilon \notin A$.
The equation $X=AX+B$ has the unique solution $X=A^*B$.
Eliminate $A_3$ using (13):
\begin{align*}
    A_2 &= bA_2 + a(aA_1+bA_2)\\
    &= bA_2 + aaA+1 + abA+2\\
    &= (ab+b)A_2+aaA_1\\
    \Leftrightarrow A_2 &= (ab+b)^*aaA_1\\
    \Rightarrow A_1 &= aA_1 + b(ab+b)^*aaA_1+\varepsilon \\
    &= (a+b(ab+b)^*aa)A_1+\varepsilon\\
    &= (a+b(ab+b)^*aa)^*
\end{align*}
\section{Equivalence relations}
Given $~\subseteq X\times X$ is an equivalence-relation if for any $x,y,x\in X$:
\begin{enumerate}
    \item $x\sim x$ (reflexive)
    \item $x\sim y \Rightarrow y\sim x$ (symmetric)
    \item $x\sim y$ and $y\sim z$ then $x\sim y$
\end{enumerate}
The equ-class of an $x\in X$ is the set $\left\{  y\in X \middle | y\sim x\right\}$.
It is denoted either by $\left[ x \right]_\sim$ or $x/\sim$.
\\The set of all equ-classes is denoted by $X/\sim$.\\
The number of equ-classes is called the index of $\sim$.

\section{DFA state minimization}
\begin{enumerate}
    \item Delete inaccessible states
    \item Collapse ``equivalent'' states
\end{enumerate}
Two states are equivalent when they have the same associated languages.\\
$p\in Q$, $L(p)=\left\{ x\in \Sigma^* \middle | \hat{\delta}(p,x) \in F \right\}$.
\subsection{Quotient construction}
$p \approx q \Leftrightarrow \forall x\in \Sigma^* (\hat{\delta}(p,x)\in F \Leftrightarrow \hat{\delta}(q,x) \in F) \Leftrightarrow L(p)=L(q)$.
\\\textbf{Note:} $\approx$ is an equ-relation.
\begin{lemma}
    $\approx$ is even a convergence relation with respect to $\delta$. That is,
    when $p\approx q$, then $\delta(p,a) \approx \delta(q,a) \forall a\in \Sigma$
\end{lemma}
Let $M=(Q,\Sigma, \delta, s, F$ be a DFA. Define $M/\approx = (Q', \Sigma, \delta', s', F')$.
\\Where:
\begin{align*}
    Q'&=\left\{ \left[ p \right]\middle | p \in Q \right\}\\
    \delta'(\left[ p \right], a) &= \left[ \delta(p,a) \right]\\
    s' &= \left[ s \right]\\
    F' &= \left\{ \left[ p \right] \middle | p\in F \right\}
\end{align*}

\begin{lemma}
    $p\in F$ iff $\left[ p \right]\in F'$.
\end{lemma}
\begin{lemma}
    $\hat{\delta}'(\left[ p \right],x)=\left[ \hat{\delta}(p,x) \right] \forall x\in \Sigma^*$
\end{lemma}
\begin{theorem}
        $L(M/\approx)=L(M)$\\
        \textbf{Proof:}
        \begin{align*}
            x\in L(M/\approx) &\Leftrightarrow \hat{\delta}'(s',x)\in F'\\
            &\Leftrightarrow \hat{\delta}'(\left[ s \right],x)\in F' \\
            &\Leftrightarrow \left[ \hat{\delta}(s,x) \right] \in F'\\
            &\Leftrightarrow \hat{\delta}(s,x) \in F \\
            &\Leftrightarrow x\in L(M)
        \end{align*}
    \end{theorem}
    \subsection{Minimization example}
    How to minimize a DFA $M$? Detect inaccessible states, then construct $M/\approx$.
    \\Ho to determine $\approx$?\\
    Stepwise identify pairs if states that are definitely not equivalent.\\
    if $p  \not \approx q$, then there is a distinguishing string x,
    \begin{align*}
        \hat{\delta}(p,x)\in F \land \hat{\delta}(x,q) \notin F
    \end{align*}
    This observation can be extended to a correctness proof of the following algorithm
    to determine $\approx$.
    \begin{enumerate}
        \item Write down a table of all pars $\left\{ p,q \right\}$, initially unmarked
        \item Mark any $\left\{ p,q \right\}$ s.t. $p\in F \land q\not \in F$
            or $p\not\in F \land q\in F$
        \item Repeat until no more changes occur:
            \begin{itemize}
                \item If there exists an unmarked pair $\left\{ p,q \right\}$ s.t.
                    $\left\{ \delta(p,a), \delta(q,a) \right\}$ is marked
                    for some $a\in \Sigma$, then mark $\left\{ p,q \right\}$
            \end{itemize}
        \item When done, we have $p\approx q$ iff $\left\{ p,q \right\}$ is unmarked
    \end{enumerate}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.9\textwidth]{unminimized.png}
        \caption{Unminimized DFA}
    \end{figure}
    \begin{table}[H]
        \centering
        \begin{tabular}{|l|l|l|l|l|l|}
            \hline
            0\\\hline
            x& 1\\\hline
            x& &2 \\\hline
            & x & x & 3 \\\hline
            & x & x & 4 \\\hline
            x & x1 & x & x & x & 5\\\hline
        \end{tabular}
        \caption{after step 1 and 2}
    \end{table}
    \begin{figure}[H]
        \centering
        \includegraphics[width=0.9\textwidth]{minimized.png}
        \caption{minimized}
    \end{figure}
    \subsection{Myhill-Nerode Theorem}
    Let $L\subseteq \Sigma^*$ a language.\\
    Then L is regular if $\equiv_L$ is of infinite index.\\
    A DFA $M$ induces an equ-relation on $\Sigma^*$:
    \begin{align*}
        x\equiv_M y \Leftrightarrow \hat{\delta}(s,y)
    \end{align*}
    That is, $x \equiv_L$ if M goes on $x$ to the same state as on $y$.
    \\\textbf{Note:} $\equiv_M$ is an equ-relation. Every equ-class corresponds
    to a state of $M$.\\
    In the book $\equiv_M$ is called Myhill-Nerode relation.\\
    Also a language $L$ induces an equ-relation on $\Sigma^*$:
    \begin{align*}
        x\equiv_L y \Leftrightarrow \forall z\in \Sigma^*: xz\in L \Leftrightarrow yz\in L
    \end{align*}
    \subsubsection{Sketch of proof}
    ``$\Rightarrow$'' be L regular . Then there is a DFA M for L. We observe:
    \begin{itemize}
        \item whenever $x \equiv_M y$, then $x\equiv_L y$\\
            Therefore, any $\equiv_M$-equ-class is a subset of exactly on $\equiv_L$-equ-class.
        \item Since any $\equiv_L$-equ-class is a subset of $\Sigma^*$ and since
            $\equiv_M$ partitions $\Sigma^*$, any number of $\equiv_L$-equ-class
            must be in some $\equiv_M$-equ-class.
    \end{itemize}
    We conclude from these points that the number of $\equiv_L$-equ-classes can not be greater
    than the number of $\equiv_M$-equ-classes, which is the number of states of $M$,
    which is finite. $\Rightarrow \equiv_L$ is of finite index.\\
    ``$\Leftarrow$''. Let the number of $\equiv_L$-equ-classes be finite. We construct
    a DFA whose states ``are'' the $\equiv_L$-equ-clasees.
    \begin{itemize}
        \item start state: $\left[ \varepsilon \right]$
        \item a state $\left[ x \right]$ is final iff $\left[ x \right]$ contains
            some string from L
        \item $\delta(\left[ x \right], a) = \left[ xa \right]$
    \end{itemize}
    \begin{enumerate}
        \item Show that $\delta$ is well-defined\\
            (that is, $\delta(\left[ x \right],a)=\left[ ya \right] \forall y\in \left[ x \right]$
        \item Show that the DFA accepts exactly L
    \end{enumerate}
    \section{Slutenhetsegenskaper (closure properties)}
    $L_1,L_2$ reguljära språk. Är $L_1\cup L_2$ reguljärt? Is it true for all
    regular languages? In that case regular languages are closes under union.
    \begin{itemize}
        \item L regular $\Rightarrow$ $\sim L$ regular? $\sim L =$ complement\\
            Given an DFA A for L. Read x and ends in an end state $\Rightarrow$ $x\in L$\\
            New DFA $A'$ for $\sim L$ have the inverted end and non-end state compared to A.\\
            $\sim L$ is regular.
        \item $L_1,L_2$ regular $\Rightarrow$ $L_1 \cup L_2$ regular?\\
            Yes: $L_1=L(r_1),L_2=L(r_2),L_1\cup L_2 = L(r_1+r_2)$
        \item $L_1,L_2$ regular $\Rightarrow$ $L_1 \cap L_2$ regular? Yes!\\
            $L_1:a*b$ $L_2:ab*$ Build DFA where each state symbolizes being in a state in $L_1$ and $L_2$.\\
            Only states which contains end states from both languages are ok.\\
            $L_1\cap L_2: ab$
        \item Alternatively: $L_1\cap L_2 = \sim (\sim L_1 \cup \sim L_2)$
        \item Regular languages are closed under $\sim,\cup,\cap$ and under concatination and $*$.
    \end{itemize}

    \section{Homomorfismer}
    A homomorphism is a function $h:\Sigma^*\rightarrow \Gamma^*$.\\
    $h(xy)=h(h)h(y)$, $h(\varepsilon)=\varepsilon$\\
    It is enough to define $h:\Sigma \Rightarrow \Gamma^*$.\\
    \begin{itemize}
        \item $A\subseteq \Sigma^*:h(A)=\left\{ h(x) \middle | x\in A \right\}\subseteq \Gamma^*$
        \item $B\subseteq \Gamma^*:h^{-1}(B)=\left\{ x \middle | h(x) \in B \right\} \subseteq \Sigma^*$
    \end{itemize}
    Example: $\Sigma = \left\{ a,b,c,d,e \right\}$, $\Gamma = \left\{ p,q \right\}$.\\
    $h(a)=pp,h(b)=pq,h(c)=qq,h(d)=ppp,h(e)=qqq$.\\
    $A=\left\{ abc \right\},h(A)=\left\{ pppqqq \right\}$\\
    $B=\left\{ pppqqq \right\},h^{-1}(B)=\left\{ abc,de \right\}$\\
    L regular $\Rightarrow$ h,$h^{-1}$ regular.\\
    Regular languages are closed under homomorphism and inverse homomorphism.
    \section{Pumpinglemmat}
    \marginpar{Lecture 6\\2016-04-20}

\end{document}