\documentclass[
    12pt, % Default font size, values between 10pt-12pt are allowed
    %spanish, % Uncomment for Spanish
]{fphw}

% Template-specific packages
\usepackage[utf8]{inputenc} % Required for inputting international characters
\usepackage[T1]{fontenc} % Output font encoding for international characters
\usepackage{fontspec,unicode-math} % Required for using utf8 characters in math mode
\usepackage{parskip}  % To add extra space between paragraphs
% \usepackage{mathpazo} % Use the Palatino font
\usepackage{graphicx} % Required for including images
\usepackage{booktabs} % Better horizontal rules in tables
\usepackage{hyperref} % For links (both internal and external)
\usepackage{listings} % Required for insertion of code
\usepackage{enumerate}% To modify the enumerate environment
\usepackage{cleveref} % Better \ref command -> \cref
\usepackage{import}   % This 4 packages and the command allow importing pdf
\usepackage{xifthen}  % figures generated with inkscape
\usepackage{pdfpages} % Source: https://castel.dev/post/lecture-notes-2/
\usepackage{mathtools}% Mathematical environments
\usepackage{relsize}  % \smaller and \mathsmaller commands
\usepackage{physics}  % A lot of good commands for mathematics
\usepackage[binary-units]{siunitx}  % To write magnitudes with units
\usepackage{transparent}
\newcommand{\incfig}[1]{%
    \def\svgwidth{0.95\columnwidth}
    \small
        \import{./images/}{#1.pdf_tex}
}

\setlength{\parindent}{15pt}

%----------------------------------------------------------------------------------------
%    ASSIGNMENT INFORMATION
%----------------------------------------------------------------------------------------

\title{Assignment 1 \\ Optimization of a sequential application} % Assignment title

\author{Emilio Domínguez Sánchez} % Student name

\date{October 18th, 2020} % Due date

\institute{University of Murcia \\ Faculty of Informatics} % Institute or school name

\class{Arquitectura y Organización de Computadores} % Course or class name

\professor{Dr. José Manuel García Carrasco} % Professor or teacher in charge of the assignment

%----------------------------------------------------------------------------------------
%    Definitions
%----------------------------------------------------------------------------------------

\lstset{
    basicstyle=\footnotesize,        % the size of the fonts that are used for the code
    frame=L,
    captionpos=t
}
\newcommand{\tech}{\texttt}
\newcommand{\gcc}{\textit{gcc}}
\newcommand{\clang}{\textit{clang}}

\DeclareMathOperator{\performance}{Performance}
\DeclareMathOperator{\bandwidth}{Bandwidth}
\DeclareSIUnit\flop{FLOP}
\DeclareSIUnit[per-mode=fraction]\fps{\flop\per\s}
\DeclareSIUnit[per-mode=fraction]\gfps{\giga\flop\per\s}
\DeclareSIUnit[per-mode=fraction]\gBps{\giga\byte\per\s}
\DeclareSIUnit\op{operation}
\DeclareSIUnit\ins{instruction}
\DeclareSIUnit\cycle{cycle}
\DeclareSIUnit\core{core}
\DeclareSIUnit\socket{socket}
\DeclareSIUnit\node{node}
\DeclareSIUnit\query{query}
\DeclareSIUnit\channel{channel}
\DeclareSIUnit\controller{controller}

\begin{document}

\maketitle % Output the assignment title, created automatically using the information in the custom commands above

%----------------------------------------------------------------------------------------
%    ASSIGNMENT CONTENT
%----------------------------------------------------------------------------------------

\section*{Instructions for the student}

    You are asked to write a document where you
explain breefly the positive aspects of the assignment,
as well as the negatives and
anything that you may have missed in it.

\noindent
In addition, you should choose a different hardware from that of the laboratory
and answer the following questions.

\noindent
The length of the document shall be between one and two pages.

\section*{Questions}

\begin{enumerate}
    \item Give an overview of the chosen hardware.
We are interested in the CPU's and memory's microarchitecture.

    \item What is the theoretical peak performance obtainable for
floating point operations(in \si{\gfps})?

    \item What is the theoretical peak memory bandwidth?

    \item Measure the peak performance for floating point operations
using the benchmarks \tech{helloflops1} and \tech{helloflops2}.
Reason about the result obtained.

    \item Measure the peak memory bandwidth using the benchmark \tech{copy}.
Reason about the result obtained.

    \item Find out the latency of the floating point sum
using the set of benchmarks \tech{sumaflopsX}.
Can you determine how many functional units are capable of performing
floating point sums for each cpu core?
\end{enumerate}

\section*{Compilation options}

You can use the following commands to compile the source files

\begin{lstlisting}
gcc -Wall -O3 -march=native helloflops1.c -o helloflops1
gcc -Wall -O3 -march=native -fopenmp helloflops2.c -o helloflops2
gcc -Wall -O3 -march=native -fopenmp hellomem.c -o hellomem
for i in $(seq 1 16); do
    gcc -Wall -O3 -march=native sumaflops$i.c -o sumaflops$i;
done
\end{lstlisting}

\noindent
The option \tech{-march=native} specifies that the target architecture is
that of the underlying machine.
\tech{-O3} activates the most aggresive optimizations, including vectorization.
And \tech{-fopenmp} enables the handling of \textit{OpenMP} directives.

%----------------------------------------------------------------------------------------

\section{Hardware}

    The target of the study is my home desktop computer.

\subsection{CPU}

    The command \tech{cat /proc/cpuinfo} shows that the cpu is an
\textit{Intel(R) Core(TM) i7-4790K}
that runs at a base frequency of $\SI{4.00}{\GHz}$.

\noindent
An internet search\footnotemark reveals the technical data about the processor.
The most important information is the cpu architecture.
The \textit{Intel(R) Core(TM) i7-4790K} is a desktop processor of the
\textit{Haswell} family.

\footnotetext{Some of the information presented here is extracted from
\url{https://en.wikichip.org/wiki/intel/microarchitectures/haswell_(client)}
and Intel's official datasheets.}

\subsubsection{The Haswell family}

    We will measure the performance for single precision floating point operations
($\si{\flop}$).
In this section we will document the features that can contribute to
the peak performance.

\begin{itemize}
    \item First, how many floating points per computer operation can be performed?

    Haswell cpus can perform traditional floating point operations as well as
    \textit{fused multiply-add}(FMA) operations,
    which allow to perform a multiplication and an addition,
    and some other variants of FMA. %TODO citation needed
    Both types of operations can be used for scalar inputs and for vectors.

    \begin{center}
    \begin{tabular}{c|c}
        \textbf{Options} & \textbf{Factor} \\
        \hline \hline
        Standard operations & $\times 1$ \\
        FMA(and alike) operations & $\times 2$ \\
    \end{tabular}
    \end{center}

    \item How many operations per instruction can be performed?

    Haswell cpus support vectorization through \textit{Streaming SIMD Extensions}(SSE)
    and \textit{Advanced Vector Extensions}(AVX).
    The former allow operations on $\SI{128}{\bit}$ vectors while the latter
    works with vectors of $\SI{256}{\bit}$.

    \begin{center}
    \begin{tabular}{c|c}
        \textbf{Options} & \textbf{Factor} \\
        \hline \hline
        Old SSE, SSE2, SSE3 and SSE4 encoding & $\times 4$ \\ %TODO wikipedia says AVX2 means sse=256
        AVX, AVX2 & $\times 8$ \\
    \end{tabular}
    \end{center}

    \item How many instructions can be executed per cycle and core?

    To answer this question, we need to take a look at the pipeline of the processor.
    Haswell cpus incorporate two functional units(per core) capable of performing
    both simple and FMA operations.

    \begin{center}
    \begin{tabular}{c|c}
        \textbf{Options} & \textbf{Factor} \\
        \hline \hline
        Floating point operations & $\times 2$ \\
    \end{tabular}
    \end{center}

    \item What is the frequency of the processor?

    The base frequency of the \textit{4790-k} is $\SI{4}{\GHz}$ while
    the turbo frequency is $\SI{4.4}{\GHz}$.
    Intel does not specify different frequency for vector operations in their datasheets,
    even though the base frequency of vector operations could be lower\footnotemark.

    \begin{center}
    \begin{tabular}{c|c}
        \textbf{Options} & \textbf{Factor} \\
        \hline \hline
        Single processor at base frequency & $\SI{4.00}{\GHz}$ \\
        Single processor at turbo frequency & $\SI{4.40}{\GHz}$ \\
        Multiple processors at base frequency & $\SI{4.00}{\GHz}$
    \end{tabular}
    \end{center}

    \footnotetext{This \href{https://community.intel.com/t5/Intel-ISA-Extensions/AVX-Base-and-Turbo-Frequencies-on-non-E5-CPUs/td-p/1024392}{thread}
    on an Intel forum poses that question and the answers seem to indicate that
    the frequencys are the same for all types of operations.}

    \item How many cores are available?

    The processor comes with $4$ cores.
    Intel's simultaneous multi-threading(\textit{hyperthreading}) allows $2$ threads
    to execute in each of those cores,
    but the number of execution units remain the same,
    meaning we shall not take that into account.
    However, in real applications and one of the benchmarks that we are going to see,
    when there are dependencies between the data,
    simultaneous multi-threading can help the processor make use of execution units
    that would otherwise remain unused due to data hazards. %TODO referencia al doc del av

    \begin{center}
    \begin{tabular}{c|c}
        \textbf{Options} & \textbf{Factor} \\
        \hline \hline
        Core & $\SI{4.00}{\GHz}$ \\
    \end{tabular}
    \end{center}

\end{itemize}

\subsection{Memory}

    We can read information about the installed memory in Linux through the command
\tech{lshw -class memory}(\cref{lst:lshw-mem}),
which we need to execute with root privileges.
Our desktop computer is equipped with two \textit{4GiB DIMM DDR3} slots
in separated channels.

\begin{lstlisting}[caption=Excerpt of the output of \tech{lshw -class memory}, label=lst:lshw-mem]
  *-memory
       description: System Memory
       physical id: 42
       slot: System board or motherboard
       size: 8GiB
     *-bank:0
          description: DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
          product: F3-12800CL9-4GBXL
          vendor: Fujitsu
          physical id: 0
          serial: 00000000
          slot: DIMM_A1
          size: 4GiB
          width: 64 bits
          clock: 1600MHz (0.6ns)
     *-bank:1
          description: DIMM [empty]
          product: [Empty]
          vendor: [Empty]
          physical id: 1
          serial: [Empty]
          slot: DIMM_A2
     *-bank:2
          description: DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
          product: F3-12800CL9-4GBXL
          vendor: Fujitsu
          physical id: 2
          serial: 00000000
          slot: DIMM_B1
          size: 4GiB
          width: 64 bits
          clock: 1600MHz (0.6ns)
     *-bank:3
          description: DIMM [empty]
          product: [Empty]
          vendor: [Empty]
          physical id: 3
          serial: [Empty]
          slot: DIMM_B2
\end{lstlisting}

\section{Theoretical performance}

\subsection{Floating point operations performance}

    The CPU peak performance can be found as

\begin{equation}\label{eq:cpu-performance}
\begin{multlined}
    \performance_{\text{peak}} \qty(\si{\gfps}) = \\
    \mathsmaller{
    \qty(\si[per-mode=fraction]{\flop\per\op}) \times
    \qty(\si[per-mode=fraction]{\op\per\ins}) \times
    \qty(\si[per-mode=fraction]{\ins\per\cycle\per\core}) \times
    \qty(\si[per-mode=fraction]{\giga\cycle\per\s}) \times
    \qty(\si[per-mode=fraction]{\core\per\socket}) \times
    \qty(\si[per-mode=fraction]{\socket}).
    }
\end{multlined}
\end{equation}

\noindent
Substituting with the values that would give a higher throughput of operations
in \cref{eq:cpu-performance} gives

\begin{equation}\label{eq:cpu-performance-val}
\begin{multlined}
    \performance_{\text{peak}} = \\
    \SI[per-mode=fraction]{2}{\flop\per\op} \times
    \SI[per-mode=fraction]{8}{\op\per\ins} \times
    \SI[per-mode=fraction]{2}{\ins\per\cycle\per\core} \times
    \SI[per-mode=fraction]{4.00}{\giga\cycle\per\s} \times
    \SI[per-mode=fraction]{4}{\core\per\socket} \times
    \SI[per-mode=fraction]{1}{\socket} = \\
    \SI{512}{\gfps}.
\end{multlined}
\end{equation}

\subsection{Memory bandwidth}

    The memory bandwidth can be found as

\begin{equation}\label{eq:mem-bandwidth}
    \bandwidth_{\text{peak}} \qty(\si{\gBps}) =
    \mathsmaller{
    \qty(\si[per-mode=fraction]{\byte\per\query\per\channel}) \times
    \qty(\si[per-mode=fraction]{\giga\query\per\s}) \times
    \qty(\si[per-mode=fraction]{\channel\per\controller}) \times
    \qty\Big(\si[per-mode=fraction]{\controller}).
    }
\end{equation}

\noindent
Substituting in \cref{eq:mem-bandwidth} the desktop's parameters gives

\begin{equation}\label{eq:mem-bandwidth-val}
    \bandwidth_{\text{peak}} =
    \mathsmaller{
    \SI[per-mode=fraction]{8}{\kilo\byte\per\query\per\channel} \times
    \SI[per-mode=fraction]{1.600}{\giga\query\per\s} \times
    \SI[per-mode=fraction]{2}{\channel\per\controller} \times
    \SI[per-mode=fraction]{1}{\controller} =
    \SI{25.6}{\gBps}.
    }
\end{equation}

\section{Actual performance}

    All of the benchmarks are based on files shared by our professor.
Nonetheless, I have modified some of them to obtain specific results.

\subsection{Floating point operations benchmarks}

    The first benchmark is called \tech{helloflops1.c}(\cref{lst:helloflops1})
and is basically a big loop where we perform a multiplication and an adittion
which can be compiled as a FMA instruction.
The lack of data hazards allows for vectorization.

\begin{lstlisting}[
    caption=Excerpt of file \tech{helloflops1.c},
    language=C,
    label=lst:helloflops1,
]
tstart = dtime();
// loop many times to really get lots of calculations
for (j = 0; j < MAXFLOPS_ITERS; j++) {
  //
  // scale 1st array and add in the 2nd array
  // example usage - y = mx + b;
  //
  for (k = 0; k < LOOP_COUNT; k++) {
    fa[k] = a * fa[k] + fb[k];
  }
}
tstop = dtime();
\end{lstlisting}

    The second benchmark is called \tech{helloflops2.c}(\cref{lst:helloflops2})
and performs the same operations as the first benchmark,
but spawns multiple threads that work on different portions of the arrays.

\begin{lstlisting}[
    caption=Excerpt of file \tech{helloflops2.c},
    language=C,
    label=lst:helloflops2,
]
tstart = dtime();
// scale the calculation across threads requested
// need to set environment variables OMP_NUM_THREADS and KMP_AFFINITY
#pragma omp parallel for private(j, k)
for (i = 0; i < numthreads; i++) {
  // each thread will work it's own array section
  // calc offset into the right section
  int offset = i * LOOP_COUNT;
  // loop many times to get lots of calculations
  for (j = 0; j < MAXFLOPS_ITERS; j++) {
    // scale 1st array and add in the 2nd array
    for (k = 0; k < LOOP_COUNT; k++) {
      fa[k + offset] = a * fa[k + offset] + fb[k + offset];
    }
  }
}
tstop = dtime();
\end{lstlisting}

    The execution time for both benchmarks with different compilation flags for the first
and different number of threads for the second are the following:

\begin{lstlisting}[
    caption=Execution results for the cpu peak performance benchmarks using \gcc{}
]
gcc -Wall -O2 -march=native -mtune=native -ftree-vectorize …
    -o helloflops1-gcc helloflops1.c
gcc -Wall -O2 -march=native -mtune=native …
    -o helloflops1-novec-gcc helloflops1.c
gcc -Wall -O3 -march=native -mtune=native …
    -o helloflops1-O3-gcc helloflops1.c
gcc -Wall -O3 -march=native -mtune=native …
-fopenmp -o helloflops2-gcc helloflops2.c
for i in helloflops1-gcc helloflops1-novec-gcc …
    helloflops1-O3-gcc helloflops2-gcc  ; do \
	echo ./$i ;\
	./$i ;\
done ;\
echo ./helloflops2-gcc 8 ;\
./helloflops2-gcc 8
./helloflops1-gcc
Initializing
Starting Compute
GFlops =    128.000, Secs = 4.505,  GFlops per sec = 28.413
./helloflops1-novec-gcc
Initializing
Starting Compute
GFlops =    128.000, Secs = 29.972, GFlops per sec = 4.271
./helloflops1-O3-gcc
Initializing
Starting Compute
GFlops =    128.000, Secs = 1.265,  GFlops per sec = 101.154
./helloflops2-gcc
Initializing
Starting Compute on 4 threads
GFlops =    512.000, Secs = 2.938,  GFlops per sec = 174.240
./helloflops2-gcc 8
Initializing
Starting Compute on 8 threads
GFlops =   1024.000, Secs = 3.957,  GFlops per sec = 258.788
\end{lstlisting}

\begin{lstlisting}[
    caption=Execution results for the cpu peak performance benchmarks using \clang{}
]
clang -Wall -O2 -march=native -mtune=native …
    -o helloflops1-clang helloflops1.c
clang -Wall -O2 -march=native -mtune=native …
    -fno-tree-vectorize -o helloflops1-novec-clang helloflops1.c
clang -Wall -O3 -march=native -mtune=native …
    -o helloflops1-O3-clang helloflops1.c
clang -Wall -O3 -march=native -mtune=native …
    -fopenmp -o helloflops2-clang helloflops2.c
for i in helloflops1-clang helloflops1-novec-clang …
    helloflops1-O3-clang helloflops2-clang  ; do \
	echo ./$i ;\
	./$i ;\
done ;\
echo ./helloflops2-clang 8 ;\
./helloflops2-clang 8
./helloflops1-clang
Initializing
Starting Compute
GFlops =    128.000, Secs = 2.002,  GFlops per sec = 63.930
./helloflops1-novec-clang
Initializing
Starting Compute
GFlops =    128.000, Secs = 23.943, GFlops per sec = 5.346
./helloflops1-O3-clang
Initializing
Starting Compute
GFlops =    128.000, Secs = 2.025,  GFlops per sec = 63.209
./helloflops2-clang
Initializing
Starting Compute on 4 threads
GFlops =    512.000, Secs = 4.741,  GFlops per sec = 107.999
./helloflops2-clang 8
Initializing
Starting Compute on 8 threads
GFlops =   1024.000, Secs = 8.525,  GFlops per sec = 120.111
\end{lstlisting}

\noindent
The results clearly expose that to obtain a good performance we need to make use of
vectorization and threads.
In regards to the peak performance, we can see
that the peak performances using a single thread was
$\SI{101.154}{\gfps}$ using \gcc{} and
$\SI{63.930}{\gfps}$ using \clang{}.
The first one is close to the expected performance when we drop
the $\times 4$ multiplier offered by the four cores in \cref{eq:cpu-performance-val},
but we should also take into account that for a single core,
the cpu can operate at a turbo frequency of $\SI{4.4}{\GHz}$.
This means that the performance was lower than expected.
The same happens when using all of the cores,
where the greatest performance, $\SI{258.788}{\GHz}$,
is obtained with \gcc{} and is half the peak performance of
\cref{eq:cpu-performance-val}.

    Note however that the greatest performance is obtained by
making use of simultaneous multi-threading.
We can guess that the assembly generated for one thread
does not fill the pipeline completely.
Thus, the extra threads can benefit of unused execution units.

\subsection{Memory bandwidth benchmark}

    We performed a single memory benchmark that copyes a very big array
using muliple cores(\cref{lst:hellomem}).

\begin{lstlisting}[
    caption=Excerpt of file \tech{hellomem.c},
    language=C,
    label=lst:hellomem,
]
tstart = dtime();
// use omp to scale the test across
// the threads requested. Need to set environment
// variables OMP_NUM_THREADS and KMP_AFFINITY
for (i=0; i<BW_ITERS; i++)
{
//
// copy the arrays to/from memory (2 bw ops)
// use openmp to scale and get aggregate bw
//
#pragma omp parallel for
    for(k=0; k<BW_ARRAY_SIZE; k++)
    {
       fa[k] = fb[k];
    }
}
tstop = dtime();
\end{lstlisting}

\noindent
Again, the code was compiled using \gcc{} and \clang{}.
Both compilers performed very similarly, obtaining an $\SI{80}{\percent}$
of the theoretical peak calculated in \cref{eq:mem-bandwidth-val}.

\begin{lstlisting}[
    caption=Execution results for the memory benchmarks for both
    \gcc{} and \clang{}
]
gcc -Wall -O2 -march=native -mtune=native -ftree-vectorize -fopenmp …
    -o hellomem-gcc hellomem.c
clang -Wall -O2 -march=native -mtune=native -fopenmp …
    -o hellomem-clang hellomem.c
./hellomem-gcc
Initializing
Starting BW Test on 8 threads
Gbytes =   1024.000, Secs = 53.322, GBytes per sec = 19.204
./hellomem-clang
Initializing
Starting BW Test on 8 threads
Gbytes =   1024.000, Secs = 53.181, GBytes per sec = 19.255
\end{lstlisting}

\subsection{Floating point sums. Latency and number of functional units}

    The last benchmarks\footnotemark are a set of files that contain a loop
with accumulating variables(\cref{lst:sumaflops5}).
An accumulating variable represents a data hazard because the cpu needs to wait
for the output of the previous operation to perform the next.
Thanks to that,
it is possible to check the latency of the functional units
if we use a single variable.
Once we start adding more variables we should see the performance increase
as the cpu can starts to pipeline operations,
until we hit the maximum number of operations the cpu can pipeline in a single core,
which is equal to the number of execution units.

\footnotetext{Originally, the codes \tech{sumaflopsX.c} given by our professor
added $1.0$ to all the variables inside the for loop.
As a result, \tech{gcc} was capable of recognizing that the result for all of the
variables would be the same and perform the operations on a single variable.
I changed the code to avoid such actions, writing different operands for each sum.}

\begin{lstlisting}[
    caption=Excerpt of file \tech{sumaflops5.c},
    language=C,
    label=lst:sumaflops5,
]
tstart = dtime();
// loop many times to really get lots of calculations
for (j = 0; j < MAXFLOPS_ITERS; j++) {
  //
  // just to sum a variable a lot of times
  //
  for (k = 0; k < LOOP_COUNT; k++) {
    suma1 += 1.0f;
    suma2 += 2.0f;
    suma3 += 3.0f;
    suma4 += 4.0f;
    suma5 += 5.0f;
  }
}
tstop = dtime();
\end{lstlisting}

\noindent
The results for this tests were:

\begin{lstlisting}[
    caption=Execution results for the data hazards benchmarks using \clang{}
]
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops1-clang sumaflops1.c
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops2-clang sumaflops2.c
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops3-clang sumaflops3.c
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops4-clang sumaflops4.c
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops5-clang sumaflops5.c
clang -Wall -O3 -march=native -mtune=native …
    -o sumaflops6-clang sumaflops6.c
for i in sumaflops1-clang sumaflops2-clang sumaflops3-clang …
    sumaflops4-clang sumaflops5-clang sumaflops6-clang ; do \
	echo ./$i ;\
	./$i ;\
done
./sumaflops1-clang
Starting Compute
GFlop =      0.640, Secs = 0.437, GFlop per sec = 1.465 (GFLOPs)
./sumaflops2-clang
Starting Compute
GFlop =      1.280, Secs = 0.437, GFlop per sec = 2.929 (GFLOPs)
./sumaflops3-clang
Starting Compute
GFlop =      1.920, Secs = 0.437, GFlop per sec = 4.394 (GFLOPs)
./sumaflops4-clang
Starting Compute
GFlop =      2.560, Secs = 0.583, GFlop per sec = 4.395 (GFLOPs)
./sumaflops5-clang
Starting Compute
GFlop =      3.200, Secs = 0.729, GFlop per sec = 4.388 (GFLOPs)
./sumaflops6-clang
Starting Compute
GFlop =      3.840, Secs = 0.874, GFlop per sec = 4.393 (GFLOPs)
\end{lstlisting}

\begin{lstlisting}[
    caption=Execution results for the data hazards benchmarks using \gcc{}
]
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops1-gcc sumaflops1.c
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops2-gcc sumaflops2.c
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops3-gcc sumaflops3.c
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops4-gcc sumaflops4.c
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops5-gcc sumaflops5.c
gcc -Wall -O3 -march=native -mtune=native …
    -o sumaflops6-gcc sumaflops6.c
for i in sumaflops1-gcc sumaflops2-gcc sumaflops3-gcc …
    sumaflops4-gcc sumaflops5-gcc sumaflops6-gcc ; do \
	echo ./$i ;\
	./$i ;\
done
./sumaflops1-gcc
Starting Compute
GFlop =      0.640, Secs = 0.437, GFlop per sec = 1.464 (GFLOPs)
./sumaflops2-gcc
Starting Compute
GFlop =      1.280, Secs = 0.437, GFlop per sec = 2.929 (GFLOPs)
./sumaflops3-gcc
Starting Compute
GFlop =      1.920, Secs = 0.437, GFlop per sec = 4.391 (GFLOPs)
./sumaflops4-gcc
Starting Compute
GFlop =      2.560, Secs = 0.585, GFlop per sec = 4.378 (GFLOPs)
./sumaflops5-gcc
Starting Compute
GFlop =      3.200, Secs = 0.729, GFlop per sec = 4.391 (GFLOPs)
./sumaflops6-gcc
Starting Compute
GFlop =      3.840, Secs = 0.876, GFlop per sec = 4.383 (GFLOPs)
\end{lstlisting}

    Both compilers performed almost identically.
When only one variable is present, the obtained performance is $\SI{1.465}{\gfps}$
which is approximately one third of $\SI{4.40}{\GHz}$,
the turbo frequency when using a single core.
This agrees with the processor specification where it is stated that the latency
for the \textit{floating point add unit} is $3$ cycles.
It is also reasonable that we see a linear increase when we use $2$ or $3$ variables,
but after that the performance stays stable.
Hence, we conclude that the code is not using more than one execution unit.

    The haswell architecture posseses two floating point operations,
but only one is capable of performing floating point additions.
However, the other one is capable of performing FMA operations
(from which we could be interested only in the sum)
at the cost of a higher latency of $5$ cycles.
The reason why I believe the compilers are not making use of this resource
is that the bigger latency means that if the same variables are always dedicated
to the same functional unit,
either the \textit{floating point add unit} or the \textit{floating point FMA unit},
the latency for some variables is bigger.
For instance, for the executable \tech{sumaflops4.c},
calculating a sum via an FMA for the extra variable, the fourth,
means that each iteration of the for loop needs at least $5$ cycles to complete,
while not using it means that each iteration is completed in $4$ cycles.
However, the compilers could do better, because more performance can be obtained
if we unroll the loop $4$ iterations and
we change which variable goes to the FMA unit every iteration.
This motivated me to work a little bit more on this,
learn assembly,
and write a loop that would make use of the whole power of the processor's core.

\section{Extra: Improving upon data hazards by inspection of the assembly}

    We have seen that the benchmarks did not agree with the
theoretical performance of the computer.
We have also analyzed in the case of the data hazards that the compiler sometimes
doesn't make full use of the processor capabilities.
In this section I will prove how a better use of the functional units could have
been made by the compiler in the same circunstances,
making use of a FMA capable unit to perform additions.

\noindent
Let's start by inspecting the assembly code for the file \tech{sumaflops4.c}.

\begin{lstlisting}
.LBB1_8:                                #   Parent Loop BB1_6 Depth=1
                                        # =>  This Inner Loop Header: Depth=2
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	vaddps	%xmm0, %xmm1, %xmm1
	addl	$8, %edi
	jne	.LBB1_8
\end{lstlisting}

\noindent
By looking at the assembly we verify our previous hypothesis,
that the compiler was only using \tech{vaddss} instructions.

\noindent
If we modify the source code to include FMA operations,

\begin{lstlisting}[language=C]
for (k = 0; k < LOOP_COUNT/4; k++) {
  suma1 = fmaf(1.0f, 1.0f, suma1);
  suma2 += 2.0f;
  suma3 += 3.0f;
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 = fmaf(2.0f, 1.0f, suma2);
  suma3 += 3.0f;
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 += 2.0f;
  suma3 = fmaf(3.0f, 1.0f, suma3);
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 += 2.0f;
  suma3 += 3.0f;
  suma4 = fmaf(4.0f, 1.0f, suma4);
}
\end{lstlisting}

\noindent
we would see no difference with the up-to-date versions of
\gcc{} and \clang{}.
Both compilers optimize the multiplication in the FMA and
write assembly as if it did not exist.
With the intent of avoiding writing assembly,
I decided to modify the result computed by the program,
knowing that we could always write the same assembly with different constants.

\begin{lstlisting}[language=C]
for (k = 0; k < LOOP_COUNT/4; k++) {
  suma1 = fmaf(1.0f, 1.1f, suma1);
  suma2 += 2.0f;
  suma3 += 3.0f;
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 = fmaf(2.0f, 1.1f, suma2);
  suma3 += 3.0f;
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 += 2.0f;
  suma3 = fmaf(3.0f, 1.1f, suma3);
  suma4 += 4.0f;
  suma1 += 1.0f;
  suma2 += 2.0f;
  suma3 += 3.0f;
  suma4 = fmaf(4.0f, 1.1f, suma4);
}
\end{lstlisting}

    The assembly code produced for this last code does use \tech{vfmaddss} operations.
The measured performance for this code was $\SI{5.859}{\gfps}$,
$\frac{4}{3}$ times faster than the original code and the file \tech{sumaflops3}.
What this means is that we managed to pipeline using the second floating point unit,
obtaining more than $\SI[per-mode=fraction]{1}{\flop\per\cycle}$.

\begin{lstlisting}
.LBB1_9:                                #   Parent Loop BB1_7 Depth=1
                                        # =>  This Inner Loop Header: Depth=2
	vaddss	%xmm1, %xmm5, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vaddss	%xmm0, %xmm9, %xmm4
	vaddss	%xmm5, %xmm8, %xmm5
	vfmadd231ss	%xmm1, %xmm3, %xmm6 # xmm6 = (xmm3 * xmm1) + xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vaddss	%xmm4, %xmm9, %xmm4
	vaddss	%xmm5, %xmm8, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vfmadd231ss	%xmm1, %xmm2, %xmm7 # xmm7 = (xmm2 * xmm1) + xmm7
	vaddss	%xmm4, %xmm9, %xmm4
	vaddss	%xmm5, %xmm8, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vfmadd231ss	%xmm1, %xmm9, %xmm4 # xmm4 = (xmm9 * xmm1) + xmm4
	vaddss	%xmm1, %xmm5, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vaddss	%xmm4, %xmm9, %xmm4
	vaddss	%xmm5, %xmm8, %xmm5
	vfmadd231ss	%xmm1, %xmm3, %xmm6 # xmm6 = (xmm3 * xmm1) + xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vaddss	%xmm4, %xmm9, %xmm4
	vaddss	%xmm5, %xmm8, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vfmadd231ss	%xmm1, %xmm2, %xmm7 # xmm7 = (xmm2 * xmm1) + xmm7
	vaddss	%xmm4, %xmm9, %xmm0
	vaddss	%xmm5, %xmm8, %xmm5
	vaddss	%xmm3, %xmm6, %xmm6
	vaddss	%xmm2, %xmm7, %xmm7
	vfmadd231ss	%xmm1, %xmm9, %xmm0 # xmm0 = (xmm9 * xmm1) + xmm0
	addl	$2, %edi
	jne	.LBB1_9
\end{lstlisting}

    The next challenge would be:
can we obtain $\SI[per-mode=fraction]{2}{\flop\per\cycle}$,
which we know is the maximum for our processor?
The answer is yes, but we cannot do it for the program with $4$ variables.

\subsection{Obtaining peak performance without vectorization}

    We want to launch an \tech{add} and a \tech{fma} operation each cycle.
The \tech{add} operation has a latency of $3$ cycles,
while the \tech{fma} operation has a latency of $5$ cycles.
Therefore, the first $3$ cycles we have compute on $6$ different variables.

\begin{center}
    \begin{tabular}{l||c c}
        cycle & ADD acts on & FMA acts on \\
        \hline \hline \\
        0 & reg0 & reg3 \\
        1 & reg1 & reg4 \\
        2 & reg2 & reg5 \\
    \end{tabular}
\end{center}

\noindent
On cycle $3$, the result of the first addition is available
and we can operate with that value again,
but the result from the FMA operation is not available yet.
The same is true for the cycle $4$, when we would be able to use the register $1$,
but none of the registers $3$ to $5$.
If only we had to operate over $3$ more variables,
we could be able to launch $3$ more operations in the next $3$ cycles.

    The idea to obtain the peak performance is
to swap the variables that used the \textit{ADD unit} and are ready
with those that used the \textit{FMA unit} and are in the process of computation and
launch new FMA operations over the variables which were not used
in the previous $3$ cycles.
Note that we would perform $2$ operations on the same variable over $9$ cycles.
It would be necessary to unroll the loop to write two iterations.
The whole idea can remind us of the \textit{three-field system}\footnote{
\url{https://en.wikipedia.org/wiki/Three-field_system}
}.

\begin{table}[h]
    \centering
    \caption{Distribution of operations for the highest performance}
    \begin{tabular}{l||c c}
        cycle & ADD acts on & FMA acts on \\
        \hline \hline \\
        0 & reg0 & reg3 \\
        1 & reg1 & reg4 \\
        2 & reg2 & reg5 \\
        \\
        3 & reg6 & reg0 \\
        4 & reg7 & reg1 \\
        5 & reg8 & reg2 \\
        \\
        6 & reg3 & reg6 \\
        7 & reg4 & reg7 \\
        8 & reg5 & reg8 \\
    \end{tabular}
\end{table}

\begin{lstlisting}[
    caption=Excerpt of file \tech{sumaflops-improved9.c},
    language=C,
    label=lst:sumaflops-improved9,
]
for (k = 0; k < LOOP_COUNT/2; k++) {
  suma1 += 1.0f;
  suma2 += 2.0f;
  suma3 += 3.0f;
  suma4 = fmaf(suma4, 1.01f, 4.0f);
  suma5 = fmaf(suma5, 1.01f, 5.0f);
  suma6 = fmaf(suma6, 1.01f, 6.0f);

  suma7 += 7.0f;
  suma8 += 8.0f;
  suma9 += 9.0f;
  suma1 = fmaf(suma1, 1.01f, 1.0f);
  suma2 = fmaf(suma2, 1.01f, 2.0f);
  suma3 = fmaf(suma3, 1.01f, 3.0f);

  suma4 += 4.0f;
  suma5 += 5.0f;
  suma6 += 6.0f;
  suma7 = fmaf(suma7, 1.01f, 7.0f);
  suma8 = fmaf(suma8, 1.01f, 8.0f);
  suma9 = fmaf(suma9, 1.01f, 9.0f);
}
\end{lstlisting}

    This idea is implemented in the file
\tech{sumaflops-improved9.c}(\cref{lst:sumaflops-improved9}).
\gcc{} performed $\SI{7.835}{\gfps}$ and \clang{} did $\SI{8.182}{\gfps}$.
We obtained marks that are very close to twice the turbo frequency of the cpu
($\SI{4.4}{\GHz}$).

%----------------------------------------------------------------------------------------

\end{document}


\chapter{Periodic LCAO for solids}
\label{chap:LCAO}

\section{Introduction}

QMCPACK implements linear combination of atomic orbitals (LCAO) and Gaussian
basis sets in periodic boundary conditions. This method uses orders of
magnitude less memory than the real space spline wavefunction. While
the spline scheme enables very fast evaluation of the wavefunction, it may
require too much on-node memory for a large complex cell. The periodic
Gaussian evaluation provides a fallback that will definitely fit in
available memory, but at significantly increased computational
expense. Well designed Gaussian basis sets should be used to accurately
represent the wavefunction, typically
including both diffuse and high angular momentum functions.

The initial implementation is limited to the $\Gamma$-point using
trial wavefunctions generated by PySCF\cite{Sun2018}, but other codes such as
Crystal can be interfaced on request.

%\subsection{Single Particle Orbitals}
%
%In QMC the many-body trial wavefunction is expressed as the product of an antisymmetric part and a correlating Jastrow factor:
% \begin{equation}
%\Psi_T(\vec{R}) = \mathcal{A}(\vec{R}) \exp\left[\sum_i J_i(\vec{R})\right]
%\end{equation}
%
%Where $\Psi_T(\vec{R})$ is the trial wave function, $\vec{R}$ is a space spin coordinates, $J(\vec{R})$ the jastrow function and $\mathcal{A}(\vec{R})$  the antisymmetric wavefunction. $\mathcal{A}(\vec{R})$  is traditionally obtained from methods such as DFT, Hartree Fock, MCSCF or CI expansion.  Many trial-wavefunctions forms have been explored, but the most popular and effective general form remains the Slater Jastrow form
% \begin{equation}
%\Psi_T(\vec{R}) = \exp\left[\sum_i J_i(\vec{R})\right]\sum_k^M C_kD_k^{\uparrow}(\varphi)D_k^{\downarrow}(\varphi)
%\end{equation}
%Where $D_k^{\downarrow}(\varphi)$ is a slater determinant expressed in terms of single particle orbitals (SPO) $\varphi_i=\sum^{N_b}_l C_l ^i \Phi_l$ . The choice of SPO representation is crucial for QMC as the cost of computing $\Phi_l$ scales linearly with the number of basis functions evaluation.  The scaling grows with the system size and the total evaluation of the N SPOs scales as  $ \mathcal{O}(N)^3$ per Monte Carlo step. In the QMCPACK parallelization scheme, SPOs are stored in read only memory replicated on each node or GPU, limiting the size of the systems to the available memory per node.
%
%In real space QMC methods it is standard to use a real space b-spline scheme or a closely related method, due to the considerable speedup over plane-waves while retaining simple convergence properties. Use of atomic orbitals and Gaussians that include more physics or chemistry results in much more efficient basis sets, but gives up easy convergence properties.
%
%\subsubsection{B-splines}
%  3D tricubic B-splines provide a basis in which only
%64 elements are nonzero at any given point in space.
%The one-dimensional cubic B-spline is given by,
%\begin{equation}
%f(x) = \sum_{i'=i-1}^{i+2} b^{i'\!,3}(x)\,\,  p_{i'},
%\label{eq:SplineFunc}
%\end{equation}
%where $b^{i}(x)$ are $p_i$ the piecewise cubic polynomial basis functions
%and $i = \text{floor}(\Delta^{-1} x)$ is the index of
%the first grid point $\le x$.  Constructing a tensor product in each Cartesian
%direction, we can represent a 3D orbital as
%\begin{equation}
%  \phi_n(x,y,z) =
%  \!\!\!\!\sum_{i'=i-1}^{i+2} \!\! b_x^{i'\!,3}(x)
%  \!\!\!\!\sum_{j'=j-1}^{j+2} \!\! b_y^{j'\!,3}(y)
%  \!\!\!\!\sum_{k'=k-1}^{k+2} \!\! b_z^{k'\!,3}(z) \,\, p_{i', j', k',n}.
%\label{eq:TricubicValue}
%\end{equation}
%This allows the rapid evaluation of each orbital in constant time.
%Furthermore, this basis is systematically improvable with a single spacing
%parameter, so that accuracy is not compromised and convergence checks are simple.
%
%Trial wavefunctions for materials are commonly produced using plane wave codes such as Quantum Espresso. The conversion to real space b-splines is straightforward. Compared to directly evaluating Fourier series, b-splines are approximately one order of magnitude faster, with the speedup increasing with system size.
%
%\subsubsection{Linear Combination of Atomic Orbitals (LCAO)}

LCAO schemes use physical considerations to construct a highly
efficient basis set compared to plane waves. Typically only a few tens
of basis functions per atom are required compared to thousands of
plane-waves. Many forms of LCAO schemes exist and are being
implemented in QMCPACK. The details of the already implemented methods
will be described in the following section of the manual.

\noindent \textbf{Gaussian Trial Orbitals (GTOs):}
 The Gaussian basis functions follow a radial-angular decomposition
\begin{equation}
     \phi (\mathbf{r} )=R_{l}(r)Y_{lm}(\theta ,\phi )
\end{equation}
where $ Y_{{lm}}(\theta ,\phi )$ is a spherical harmonic, $l$ and $m$
are the angular momentum and its $z$ component, and $r,\theta ,\phi $
are spherical coordinates. In practice they are atom centered and the
$l$ expansion typically includes 1-3 additional channels compared to
the formally occupied states of the atom. e.g. 4-6 for a nickel atom with
occupied $s$, $p$, and $d$ electron shells.

The evaluation of GTOs within PBC differs slightly from evaluating
GTOs in Open Boundary Conditions (OBC).  The orbitals are evaluated at
a distance $r$ in the primitive cell (similar to OBC) and then the
contributions of the periodic images are added by evaluating the
orbital at a distance $r+T$ where T is a translation of the cell
lattice vector. This requires loops over the periodic images until the
contributions are orbitals $\Phi$. In the current implementation, the
number of periodic images is an input parameter named
\textit{PBCimages}, which takes three integers corresponding to the
number of periodic images along the supercell axes (X, Y and Z axes
for a cubic cell). By default these parameters are set to
\textit{PBCimages= 5 5 5} but they \textbf{require manual convergence
  checks}. Convergence checks can be performed by checking the total
energy convergence with respect to \textit{PBCimages}, similar to checks
performed for plane wave cutoff energy and b-spline grids. Use of
diffuse Gaussians may require these parameters to be increased, while
sharply localized Gaussians may permit a decrease. The cost of
evaluating the wavefunction increases sharply as \textit{PBCimages} is
increased. This input parameter will be replaced by a tolerance
factor and numerical screening in future.

\section{Generating and using periodic gaussian trial wavefunctions
  using PySCF}

Similar to any QMC calculation, using periodic GTOs requires the
generation of a periodic trial wavefunction. QMCPACK is currently
interfaced to PySCF which is a multipurpose electronic structure
written mainly in Python with key numerical functionality implemented
via optimized C and C++ libraries\cite{Sun2018}. Such a wavefunction
can be generated following the example for a 2x1x1 supercell,
below. Note that the current implementation and examples cover only
the use of k-points where symmetry allows real coefficients to be
used.  This allows calculation at $\Gamma$) and, e.g., some high
symmetry k-points at the Brillouin zone edges.  More general k-points
requiring complex coefficients will be supported in future releases.

\begin{lstlisting}[caption=Example PySCF input for single k-point calculation for a 2x1x1 Carbon supercell.]
#!/usr/bin/env python

import numpy
from pyscf.pbc import gto, scf, dft
from mpi4pyscf.pbc import df
from pyscf.pbc.tools.pbc import super_cell

nmp = [2, 1, 1]

cell = gto.Cell()

cell.a = '''
         3.37316115       3.37316115       0.00000000
         0.00000000       3.37316115       3.37316115
         3.37316115       0.00000000       3.37316115'''
cell.atom = '''
   C        0.00000000       0.00000000       0.00000000
   C        1.686580575      1.686580575      1.686580575
            '''
cell.basis='bfd-vtz'
cell.ecp = 'bfd'

cell.unit='B'
cell.drop_exponent=0.1

cell.verbose = 5
cell.build()

supcell = super_cell(cell, nmp)
mydf = df.FFTDF(supcell)
mydf.auxbasis = 'weigend'

mf = dft.RKS(supcell)
mf.xc = 'lda'

mf.exxdiv = 'ewald'
mf.with_df = mydf

e_scf=mf.kernel()

print 'e_scf',e_scf

kpts=[]
title="C_Diamond"
from PyscfToQmcpack import savetoqmcpack
savetoqmcpack(supcell,mf,title=title,kpts=kpts)

\end{lstlisting}

Note that the last 4 lines of the file
\begin{lstlisting}
kpts=[]
title="C_Diamond"
from PyscfToQmcpack import savetoqmcpack
savetoqmcpack(supcell,mf,title=title,kpts=kpts)
\end{lstlisting}

contains an empty list of k-points (since this is a gamma point
calculation) and a title. The title variable will be the name of the
HDF5 file where all the data needed by QMCPACK will be stored.  The
function \textit{savetoqmcpack} will be called at the end of the
calculation and will generate the HDF5 similarly to the non-periodic
PySCF calculation in section~\ref{sec:convert4qmc} (convert4qmc). The
function is distributed with QMCPACK and located in the
qmcpack/src/QMCTools directory under the name
\textit{PyscfToQmcpack.py}. In order for the script to work, you need
to specify the path to the file in your PYTHONPATH such as

\begin{lstlisting}
export PYTHONPATH=QMCPACK_PATH/src/QMCTools:$PYTHONPATH
\end{lstlisting}

When using multiple k-points, it is necessary to expand the kpoints into the equivalent supercell, adjust for the phase factor in the coefficient's value due to the translation by the lattice vector and order the molecular coefficients from each k-point according to their occupation. These operations are all automated in the \textit{savetoqmcpack()} function.\\

The following example corresponds to the same carbon system (2x1x1) however, in this case, we use a primitive simulation cell and a 2x1x1 kpoint mesh.

\begin{lstlisting}[caption=Example PySCF input for single k-point calculation for a 2x1x1 Carbon supercell.]
#!/usr/bin/env python

import numpy
from pyscf.pbc import gto, scf, dft,df
kmesh = [2, 1, 1]

cell = gto.Cell()
cell.a = '''
         3.37316115       3.37316115       0.00000000
         0.00000000       3.37316115       3.37316115
         3.37316115       0.00000000       3.37316115'''
cell.atom = '''
   C        0.00000000       0.00000000       0.00000000
   C        1.686580575      1.686580575      1.686580575
            '''
cell.basis='bfd-vtz'
cell.ecp = 'bfd'

cell.unit='B'
cell.drop_exponent=0.1

cell.verbose = 5

cell.build()

kpts = cell.make_kpts(kmesh)
kpts -= kpts[0]

mydf = df.GDF(cell,kpts)
mydf.auxbasis = 'weigend'
mf = scf.KRHF(supcell,kpts).density_fit()

mf.exxdiv = 'ewald'
mf.with_df = mydf
e_scf=mf.kernel()

title="C_Diamond-211"

from PyscfToQmcpack import savetoqmcpack
savetoqmcpack(supcell,mf,title=title,kpts=kpts,kmesh=kmesh)

\end{lstlisting}

Note the difference between the 2 input files where:\\
\begin{lstlisting}
kmesh=[2,1,1]  #k-point mesh
\end{lstlisting}

\begin{lstlisting}
kpts = cell.make_kpts(kmesh)
kpts -= kpts[0]
\end{lstlisting}
Will generate k-points centered around the $\Gamma$-point and will insure the molecular coefficients are real.\\

\begin{lstlisting}
mf = scf.KRHF(supcell,kpts).density_fit()
\end{lstlisting}
The Computational algorithm chosen in PySCF is \textit{KRHF} instead of \textit{RHF}.

Finally, to generate the HDF5 file needed by \qmcpack we call the \textit{savetoqmcpack} function\\
\begin{lstlisting}
from PyscfToQmcpack import savetoqmcpack
savetoqmcpack(supcell,mf,title=title,kpts=kpts,kmesh=kmesh)
\end{lstlisting}
In this call, we simply specify the k-point mesh used in order to force the converter to generate the desired cell. Note that if the parameter \textit{kmesh} is omitted, the converter will still try to ``guess'' it.

In order to generate QMCPACK input files, you will need to run for both cases \textit{convert4qmc} exactly as specified in section ~\ref{sec:convert4qmc};
\begin{lstlisting}
convert4qmc -pyscf C_Diamond.h5
\end{lstlisting}

This tool can be used with any option described in convert4qmc. Since
the HDF5 contains all the information needed, there is no need to
specify any other specific tag for periodicity.A supercell at
$\Gamma$-point or using multiple k-points will work without further
modification..

Running convert4qmv will generate 3 input files;\\
\begin{lstlisting}[style=XML,caption=CDiamond.structure.xml. This file contains the geometry of the system.]
<?xml version="1.0"?>
<qmcsystem>
  <simulationcell>
    <parameter name="lattice">
  6.74632230000000e+00  6.74632230000000e+00  0.00000000000000e+00
  0.00000000000000e+00  3.37316115000000e+00  3.37316115000000e+00
  3.37316115000000e+00  0.00000000000000e+00  3.37316115000000e+00
</parameter>
    <parameter name="bconds">p p p</parameter>
    <parameter name="LR_dim_cutoff">15</parameter>
  </simulationcell>
  <particleset name="ion0" size="4">
    <group name="C">
      <parameter name="charge">4</parameter>
      <parameter name="valence">4</parameter>
      <parameter name="atomicnumber">6</parameter>
    </group>
    <attrib name="position" datatype="posArray">
  0.0000000000e+00  0.0000000000e+00  0.0000000000e+00
  1.6865805750e+00  1.6865805750e+00  1.6865805750e+00
  3.3731611500e+00  3.3731611500e+00  0.0000000000e+00
  5.0597417250e+00  5.0597417250e+00  1.6865805750e+00
</attrib>
    <attrib name="ionid" datatype="stringArray">
 C C C C
</attrib>
  </particleset>
  <particleset name="e" random="yes" randomsrc="ion0">
    <group name="u" size="8">
      <parameter name="charge">-1</parameter>
    </group>
    <group name="d" size="8">
      <parameter name="charge">-1</parameter>
    </group>
  </particleset>
</qmcsystem>
  \end{lstlisting}

  As one can see, that for both examples the 2 atom primitive cell has been expanded to contain 4 atoms in a 2x1x1 carbon cell.
\begin{lstlisting}[style=XML,caption=CDiamond.wfj-Twist0.xml. This file contains the trial wavefunction.]
<?xml version="1.0"?>
<qmcsystem>
  <wavefunction name="psi0" target="e">
    <determinantset type="MolecularOrbital" name="LCAOBSet" source="ion0" transform="yes" twist="0  0  0" href="C_Diamond.h5" PBCimages="5  5  5">
      <slaterdeterminant>
        <determinant id="updet" size="8">
          <occupation mode="ground"/>
          <coefficient size="116" spindataset="0"/>
        </determinant>
        <determinant id="downdet" size="8">
          <occupation mode="ground"/>
          <coefficient size="116" spindataset="0"/>
        </determinant>
      </slaterdeterminant>
    </determinantset>
    <jastrow name="J2" type="Two-Body" function="Bspline" print="yes">
      <correlation size="10" speciesA="u" speciesB="u">
        <coefficients id="uu" type="Array"> 0 0 0 0 0 0 0 0 0 0</coefficients>
      </correlation>
      <correlation size="10" speciesA="u" speciesB="d">
        <coefficients id="ud" type="Array"> 0 0 0 0 0 0 0 0 0 0</coefficients>
      </correlation>
    </jastrow>
    <jastrow name="J1" type="One-Body" function="Bspline" source="ion0" print="yes">
      <correlation size="10" cusp="0" elementType="C">
        <coefficients id="eC" type="Array"> 0 0 0 0 0 0 0 0 0 0</coefficients>
      </correlation>
    </jastrow>
  </wavefunction>
</qmcsystem>
 \end{lstlisting}
This files contains information related to the trial wavefunction. It is identical to the input file from an Open Boundary Conditions calculation to the exception of the following tags:\\
\begin{table}[h]
\begin{center}
\begin{tabularx}{\textwidth}{l l l l l }
\hline
\multicolumn{5}{l}{*.wfj.xml specific tags} \\
\hline
%\multicolumn{2}{l}{Outputfiles}  & \multicolumn{3}{l}{}\\
   &   \bfseries tag     & \bfseries tag type & \bfseries default   & \bfseries description \\
   &   \texttt{twist             } &  3 doubles  & Gamma ( 0 0 0)& coordinate of the twist to compute\\
   &   \texttt{href             } &  string  & default& name of the HDF5 file generated by\\
   &                              &          &        &  PySCF and used for convert4qmc\\
   &   \texttt{PBCimages            } &  3 Integer   & 5 5 5  & Number of periodic images to evaluate the orbitals\\
    \hline
    \end{tabularx}
\end{center}
\end{table}

Other files containing QMC methods (such as optimization, VMC and DMC blocks) will be generated and will behave in a similar fashion regardless of the type of SPO in the trial wavefunction.


% either Uni Rostock style
\documentclass[mathserif, aspectratio=43]{intbeamer}
\setbeamertemplate{page number in head/foot}[parts]
\AtBeginPart{\makepart\setcounter{framenumber}{0}}
\addtobeamertemplate{block begin}{
\setlength{\textwidth}{0.98\textwidth}}{}

% or some beamer theme
% \documentclass[aspectratio=43]{beamer}
% \usetheme{Boadilla}
% \usecolortheme{dove}
% \setbeamertemplate{navigation symbols}{}
% \usefonttheme[onlymath]{serif}
% \addtobeamertemplate{headline}{}{\insertsection}

% required packages
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{bm}
\usepackage{trfsigns}
\usepackage{tikz}
\usepackage{comment}

\newcommand\fsd{\mathrm{d}} %der/int operator

% shorten figure caption
\addto\captionsenglish{\renewcommand{\figurename}{Fig.}}
%\addto\captionsngerman{\renewcommand{\figurename}{Abb.}}

%matplotlib colors:
\definecolor{C0}{HTML}{1f77b4}
\definecolor{C1}{HTML}{ff7f0e}
\definecolor{C2}{HTML}{2ca02c}
\definecolor{C3}{HTML}{d62728}
\definecolor{C7}{HTML}{7f7f7f}

\title[UE SigSys]{Übung \\ \huge Signal- und Systemtheorie}
\author[Schultz, INT, IEF, Uni Rostock]{Frank Schultz}
\date[SS 2021]{Sommersemester 2021}

\begin{document}
\maketitle
\section{UE 00}
% TBD

%\begin{comment}

\section{UE 01 / 2021-04-14}
%
\begin{frame}{UE 01 Schedule / Objectives}
Signals
\begin{itemize}
\item The Four Fourier Transforms (FTs)
\item Basics Fourier Series (Sinc / Rect duality)
\item Basics Fourier Transform (Sinc / Rect duality)
\item Fundamental FT Correspondences
\end{itemize}
Systems
\begin{itemize}
\item The Dirac Impulse
%\item From Dirac Impulse to the Green's Function of ODEs =
\item From Dirac Impulse to the Impulse Response of an LTI System
\end{itemize}
\end{frame}

\begin{frame}{Fourier Reihe / Fourier Series (FS)}
\begin{itemize}
\item for continuous-time, periodic signals $x(t)\in\mathbb{C}$
with signal period $T_0$, fundamental (physical) frequency $f_0=\frac{1}{T_0}$
\item in SigSys we rather use angular frequency, i.e. $\omega_0 = 2 \pi f_0 = \frac{2\pi}{T_0}$
\item analysis (i.e. get Fourier coefficients $\tilde{X}$ over $\mu\in\mathbb{Z}$)
$$\tilde{X}[\mu] =
\int\limits_{0}^{T_0} x(t) \, \e^{-\im \frac{2\pi}{T_0} \mu t}\,\mathrm{d}t =
\int\limits_{0}^{T_0} x(t) \, \e^{-\im \omega_0 \mu t}\,\mathrm{d}t$$
\item synthesis (i.e. linear combination)
$$x(t) =
\frac{1}{T_0}\sum\limits_{\mu=-\infty}^{+\infty} \tilde{X}[\mu] \,\e^{+\im \frac{2\pi}{T_0} \mu t} =
\frac{1}{T_0}\sum\limits_{\mu=-\infty}^{+\infty} \tilde{X}[\mu] \,\e^{+\im \omega_0 \mu t}
$$
\item note: $T$ and $c_k T$ in tutorial corresponds to $T_0$ and $\tilde{X}[\mu]$ here in slides
\end{itemize}
\end{frame}

\begin{frame}{Fourier Transform (FT)}
\begin{itemize}
\item for continuous-time signals $x(t)\in\mathbb{C}$
\item analysis (i.e. FT spectrum over $\omega\in\mathbb{R}$)
$$X(\im\omega) = \int\limits_{-\infty}^{+\infty} x(t) \,\e^{-\im \omega t}\,\mathrm{d}t$$
\item synthesis
$$x(t) = \frac{1}{2\pi}\int\limits_{-\infty}^{+\infty} X(\im\omega) \,\e^{+\im \omega t}\,\mathrm{d}\omega
$$
\item FT operator symbol $$x(t) \,\laplace\,  X(\im\omega)$$
\end{itemize}
\end{frame}

\begin{frame}{Discrete-Time Fourier Transform (DTFT)}
\begin{itemize}
\item for discrete-time signals $x[k]\in\mathbb{C}$, $k\in\mathbb{Z}$
\item analysis (i.e. DTFT spectrum over $\Omega\in\mathbb{R}$)
$$X(\e^{\im\Omega}) = \sum\limits_{k=-\infty}^{+\infty} x[k] \,\e^{-\im \Omega k}$$
\item synthesis
$$x[k] = \frac{1}{2\pi}\int\limits_{0}^{2\pi} X(\e^{\im\Omega}) \,\e^{+\im \Omega k}\,\mathrm{d}\Omega
$$
\end{itemize}
\end{frame}

\begin{frame}{Discrete Fourier Transform (DFT)}
%
\begin{itemize}
\item for discrete-time, periodic signals $x[k]\in\mathbb{C}$, $k\in\mathbb{Z}$
with signal period $N$ samples
\item analysis (i.e. DFT coefficients over $\mu\in\mathbb{Z}$)
$$X[\mu] = \sum\limits_{k=0}^{N-1} x[k] \,\e^{-\im \frac{2\pi}{N} \mu k} =
\sum\limits_{k=0}^{N-1} x[k] \,\e^{-\im \Omega_0 \mu k}$$
\item synthesis
$$x[k] = \frac{1}{N}\sum\limits_{\mu=0}^{N-1} X[\mu] \,\e^{+\im \frac{2\pi}{N} \mu k} =
\frac{1}{N}\sum\limits_{\mu=0}^{N-1} X[\mu] \,\e^{+\im \Omega_0 \mu k}$$
\item with fundamental DFT frequency $\Omega_0 = \frac{2\pi}{N}$
\end{itemize}
%
\end{frame}

\begin{frame}{Fourier Series for Periodic Rect Impulse}
Task 1.1
\begin{figure}
\includegraphics[width=\textwidth]{../fs/D1483A84E2_0.pdf}
\caption{1.1}
\end{figure}
\end{frame}

\begin{frame}{Fourier Series for Periodic Rect Impulse}
Task 1.1
\begin{figure}
\includegraphics[width=\textwidth]{../fs/D1483A84E2_1.pdf}
\caption{1.2}
\end{figure}
\end{frame}

\begin{frame}{Fourier Series for Periodic Rect Impulse}
Task 1.1
\begin{figure}
\includegraphics[width=\textwidth]{../fs/D1483A84E2_2.pdf}
\caption{1.3}
\end{figure}
\end{frame}

\begin{frame}{Fourier Series for Periodic Rect Impulse}
Task 1.1
\begin{figure}
\includegraphics[width=\textwidth]{../fs/D1483A84E2_3.pdf}
\caption{1.4}
\end{figure}
\end{frame}

\begin{frame}{Fourier Series for Periodic Rect Impulse}
Task 1.1
\begin{figure}
\includegraphics[width=\textwidth]{../fs/D1483A84E2_4.pdf}
\caption{1.5}
\end{figure}
\end{frame}

\begin{frame}{Fourier Transform for Single Rect Impulse}
Task 1.2
\begin{figure}
\includegraphics[width=0.7\textwidth]{../ft/8C3958BE4F_1.pdf}
\includegraphics[width=0.7\textwidth]{../ft/8C3958BE4F_0.pdf}
\caption{1.6}
\end{figure}
\end{frame}

\begin{frame}{Fourier Transform Spectrum -> Magnitude / Phase Plot}
Task 1.2
\begin{figure}
\includegraphics[width=0.7\textwidth]{../ft/8C3958BE4F_SingleCase_MagPhase.pdf}
\caption{1.7}
\end{figure}
\end{frame}

\begin{frame}{Fourier Transform Rect / Sinc Duality}
For our used definition of the rect function (Rechteckimpuls)
$$\text{rect}(t) := \begin{cases} 1 & |t| < \frac{1}{2} \\ \frac{1}{2} & |t| = \frac{1}{2} \\ 0 & |t| > \frac{1}{2} \end{cases}$$
and of the sinc function (Spaltfunktion)
$$\text{sinc}(t) := \frac{\sin(t)}{t}$$
the duality
$$\text{rect}(t) \,\laplace\,  \,\,\,\text{sinc}\left( \frac{\omega}{2} \right)$$
$$\text{sinc}(t) \,\laplace\, \pi \,\text{rect}\left(\frac{\omega}{2}\right)$$
is of fundamental importance in SigSys
\end{frame}

\begin{frame}{Fourier Transform for Right-Shifted Single Rect Impulse}
Task 1.3: Time Shift Leads to Phase Change
\begin{figure}
\includegraphics[width=0.7\textwidth]{../ft/A8A2DEE53A.pdf}
\caption{1.8}
\end{figure}
\end{frame}

\begin{frame}{Fourier Transform for Left-Shifted Single Rect Impulse}
Task 1.4: here special also time-reversed signal of Task 1.3
\begin{figure}
\includegraphics[width=0.7\textwidth]{../ft/1CFE5FE3A1.pdf}
\caption{1.9}
\end{figure}
\end{frame}

\begin{frame}{Fourier Transform Time / Phase Reversal}
%
from Task 1.3 and 1.4 we find
\begin{align*}
x(+t) \quad \laplace \quad X(+\im\omega) =& |X(\im\omega)| \, \e^{+\im\angle X(\im\omega)}\\
x(-t) \quad \laplace \quad X(-\im\omega) =& |X(\im\omega)| \, \e^{-\im\angle X(\im\omega)}
\end{align*}
%
\begin{figure}
\includegraphics[width=0.49\textwidth]{../ft/A8A2DEE53A.pdf}
\includegraphics[width=0.49\textwidth]{../ft/1CFE5FE3A1.pdf}
\caption{1.8 \& 1.9}
\end{figure}
%
\end{frame}

\begin{frame}{Fourier Transform Shift / Modulation Duality}
%
\textbf{Zeitverschiebung} um $\tau\in\mathbb{R}$ aka \textbf{Verschiebungssatz}
\begin{align*}
x(t) & \quad \laplace \quad X(\im\omega)\\
x(t - \tau) & \quad \laplace \quad X(\im\omega) \cdot \e^{-\im\omega \tau}
\end{align*}
%
\textbf{Frequenzverschiebung} um $\omega_0\in\mathbb{R}$ aka \textbf{Modulationssatz}
(see Task 1.5)
\begin{align*}
x(t) & \quad \laplace \quad X(\im\omega)\\
\label{eq:A8A2DEE53A_ModTheoremTime}
x(t) \cdot \e^{+\im\omega_0 t} & \quad \laplace \quad X(\im[\omega-\omega_0])
\end{align*}
%
Wie sind die Fourier Transformationen von
\begin{align*}
\cos(\omega_0 t) = \frac{\e^{+\im\omega_0 t}+\e^{-\im\omega_0 t}}{2}\qquad
\sin(\omega_0 t) = \frac{\e^{+\im\omega_0 t}-\e^{-\im\omega_0 t}}{2\im}\quad??
\end{align*}
\end{frame}

\begin{frame}{Dirac Impulse And Sifting Property}
Task 1.6:
%
%\begin{equation*}
%\label{eq:DiracDistrDef}
$\lim_{\xi\to\infty} \int\limits_{-\infty}^{+\infty}
\delta_\xi(t-t_0) \cdot f(t) \, \fsd t
:= \int\limits_{-\infty}^{+\infty} \delta(t-t_0) \cdot f(t) \, \fsd t \stackrel{\mathrm{def}}= f(t_0)$
%\end{equation*}
%
\begin{figure}
\includegraphics[width=0.6\textwidth]{../dirac_impulse_ct/D410BDAAE0.pdf}
\caption{1.10, top: Rect, bottom: normalized Sinc}
\end{figure}
\end{frame}

\begin{frame}{Dirac Impulse And Sifting Property}
Task 1.6

\begin{itemize}
\item we define Dirac Delta impulse by $\delta(t)$ for the following equation
%
\item
$\lim_{\xi\to\infty} \int\limits_{-\infty}^{+\infty}
\delta_\xi(t-t_0) \cdot f(t) \, \fsd t
:= \int\limits_{-\infty}^{+\infty} \delta(t-t_0) \cdot f(t) \, \fsd t \stackrel{\mathrm{def}}= f(t_0)$
%
\item again: right integral equation is not a Riemann integral, but a definition
%
\item from that definition we directly find special cases
%
$$\int\limits_{-\infty}^{+\infty} \delta(t) \cdot f(t) \, \fsd t = f(0) \qquad\text{for}\qquad t_0=0$$
%
$$\int\limits_{-\infty}^{+\infty} \delta(t) \fsd t = 1
\qquad\text{for}\qquad f(t)=1$$
%
\end{itemize}
%
\end{frame}

\begin{frame}{Fourier Transform Duality for Dirac Impulse}

\begin{figure}[h!]
\centering
%
\begin{tikzpicture}0
\def\l{-2.5}
\def\r{+2.5}
\draw[->] (-2+\l,0) -- (2+\l,0) node[right] {$t$};
\draw[->] (0+\l,0) -- (0+\l,1.5) node[above] {$x(t)=\delta(t)$};
\draw[->, C0, line width=1mm] (0+\l,0) -- (0+\l,1) node[left] {$(1)$};
\node at (0,1) {$\laplace$};
\draw[->] (-2+\r,0) -- (2+\r,0) node[right] {$\omega$};
\draw[->] (0+\r,0) -- (0+\r,1.5) node[above] {$X(\im\omega)=1$};
\draw[-, C0, line width=1mm] (-2+\r,1) -- (2+\r,1) node[right] {$1$};
\end{tikzpicture}
%
\begin{tikzpicture}0
\def\l{-2.5}
\def\r{+2.5}
\draw[->] (-2+\l,0) -- (2+\l,0) node[right] {$t$};
\draw[->] (0+\l,0) -- (0+\l,1.5) node[above] {$x(t)=1$};
\draw[-, C0, line width=1mm] (-2+\l,1) node[left] {$1$} -- (2+\l,1);
\node at (0,1) {$\laplace$};
\draw[->] (-2+\r,0) -- (2+\r,0) node[right] {$\omega$};
\draw[->] (0+\r,0) -- (0+\r,1.5) node[above] {$X(\im\omega)=2\pi\delta(\omega)$};
\draw[->, C0, line width=1mm] (0+\r,0) -- (0+\r,1) node[right] {$(2\pi)$};
\end{tikzpicture}
%
\caption{1.11}
\end{figure}

\end{frame}

\begin{frame}{Most Important Fourier Transform Properties}
%
as of now we know so far
\begin{align*}
x(t)= \delta(t) &\quad \laplace \quad X(\im\omega)=1\\
x(t)= 1 &\quad \laplace \quad X(\im\omega)=2\pi \delta(\omega)\\
x(t - \tau) & \quad \laplace \quad X(\im\omega) \cdot \e^{-\im\omega \tau}\\
x(t) \cdot \e^{+\im\omega_0 t} & \quad \laplace \quad X(\im[\omega-\omega_0])
\end{align*}
%
from these we can deduce further nice properties
\begin{align*}
\delta(t-\tau)& \quad \laplace \quad 1 \cdot \e^{-\im\omega \tau}\\
1 \cdot \e^{+\im\omega_0 t} & \quad \laplace \quad 2\pi\delta(\omega-\omega_0)
\end{align*}
%
Jetzt können wir 'lösen': Wie sind die Fourier Transformationen von
\begin{align*}
\cos(\omega_0 t) = \frac{\e^{+\im\omega_0 t}+\e^{-\im\omega_0 t}}{2}\qquad
\sin(\omega_0 t) = \frac{\e^{+\im\omega_0 t}-\e^{-\im\omega_0 t}}{2\im}\quad??
\end{align*}
%
\end{frame}

\begin{frame}{Impulse Response of System, Green's Function of ODE}
Task 1.7: Find the solution $y(t)$ for $t\geq 0$ of the ODE and given initial conditions
$$\frac{16}{25} \ddot{y}(t) + \frac{24}{25} \dot{y}(t) + y(t) = 0,
\quad y(0)=0,\quad \dot{y}(0)=\frac{25}{16}$$
The solution is known as impulse response $h(t)$ in SigSys
$$h(t) := y(t) = \frac{25}{16} \e^{-\frac{3}{4} t} \sin(t), \qquad t\geq 0$$
since the problem is equivalent with excitation $x(t) = \delta(t)$
$$A \ddot{y}(t) + B \dot{y}(t) + C y(t) = \delta(t),\qquad {y}(0_-) = 0,\qquad \dot{y}(0_-) = 0$$
%
Convolution integral using impulse response for any other $x(t \leq 0)$
$$y_p(t) = \int\limits_{0}^{t} h(t-\tau) x(\tau) \fsd \tau$$
%
\end{frame}

%\end{comment}

\section{UE 02 / 2021-04-21}
\begin{frame}{UE 02 Schedule / Objectives}

\begin{itemize}
\item Elementarsignale
\item Signaloperationen im Detail
\item Testen von Systemen auf Linearität und Zeitinavarianz (LTI)
\item Why and How To Faltung? ET Beispiele
\end{itemize}

\end{frame}

\section{UE 03 / 2021-04-28}
\begin{frame}{UE 03 Schedule / Objectives}

\begin{itemize}
\item Faltung aus UE 02
\item Intro Laplace Transformation UE 03
\item Sprungsignal, Konvergenzbereich
\item Moduliertes Sprungsignal, Konvergenzbereich
\item RC Glied mit Laplace Transformation
\end{itemize}

\end{frame}

% \section{UE 04}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 05}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 06}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 07}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 08}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 09}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 10}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 11}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 12}
% \begin{frame}{}
% %
% \end{frame}
% \section{UE 13}
% \begin{frame}{}
% %
% \end{frame}
%
\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS630: Database Management Systems
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

Consider a database schema with three relations:

\begin{terminal}
Parts (@*\underline{pid}*@:integer, pname:string, price:integer)
Suppliers (@*\underline{sid}*@:integer, sname:string, state:string, zipcode:string)
Orders (@*\underline{pid}*@:integer, @*\underline{sid}*@:integer, quantity:integer)
\end{terminal}

The description is as follows: a factory keeps a database with parts that it uses, the suppliers of those parts, and purchase orders.
Each part is uniquely identified by \texttt{pid}.
Each part has a string description \texttt{pname}, \texttt{year} of fabrication and \texttt{price} per unit.
Parts are provided by suppliers, and each supplier is uniquely identified by \texttt{sid}.
Each supplier has a name \texttt{sname}, and it operates at the given \texttt{state} and \texttt{zipcode}.
The company has a number of orders for parts, and each order contains the \texttt{pid} of the part ordered, the \texttt{sid} of the supplier from which the part is ordered, and the \texttt{quantity} ordered.
You are allowed flexibility on the exact attribute types you use for your schema, as long as they reasonably match the specification above (e.g., in terms of number types, string types).
Include the schema definition in your submission in a file called \texttt{schema.sql}.

Write Embedded SQL programs that perform the following tasks:

\begin{enumerate}[label=(\alph*)]
\item Prompt the user for a supplier ID and print the specifications (part number, part name, year and price for all parts ordered from that supplier.
Attributes of each part must be printed on a new line (10p).
\item Prompt the user for the value of a zipcode and a part ID.
Output in tabular format (one per line) the supplier ID, supplier name and the \textit{dollar amount} of the order for the given part for each supplier in the given zipcode that supplied that part.
Each line should start with a unique entry number (begin numbering with 1, continue with 2,3 and so on) which is NOT part of any schema, but uniquely identifies the output line.
If there are no matching records, print the message \textit{No Records Found} on the screen.
If the result is not empty, prompt the user for an entry number and delete the corresponding record from the Orders table (15p).
\end{enumerate}

Create two separate source files for (a) and (b), and name them \texttt{Q1a.pc} and \texttt{Q1b.pc}.

\section*{Solution to Question 1}

\lstset{language=c}
\lstset{tabsize=4}

\begin{itemize}

\item[] File \texttt{q1a.c}
\lstinputlisting[firstline=10]{
	\topDirectory/src/pc/hw04/hw04q01p01.pc
}

\item[] File \texttt{q1b.c}
\lstinputlisting[firstline=10]{
	\topDirectory/src/pc/hw04/hw04q01p02.pc
}

\end{itemize}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage[top=2cm,bottom=2cm,left=2cm,right=2cm]{geometry}
\usepackage{secdot}
\usepackage{hyperref}
\usepackage{url}
\usepackage{amsmath,amssymb}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage[usenames, dvipsnames]{color}
\usepackage{marvosym}
\hypersetup{
    colorlinks=true,
    urlcolor=ForestGreen
}
\title{Exam questions}
\author{NLA course}
\date{2016}

\begin{document}

\maketitle

Online rendering on github or nbviewer might be incorrect in some places. Therefore, please read lectures locally on your laptop.

\section*{List of questions}

\begin{enumerate}
%	\item Representation of numbers in computer. Fixed and floating point arithmetics. Loss of significance. Naive summation vs. Kahan summation.
	\item Vector norms. Forward and backward stability. Disks in different norms. First norm and compressed sensing.
	\item Matrix-matrix multiplication. Computer memory hierarchy and blocking. Standard libraries for linear algebra. Strassen algorithm (no need to remember exact formulas) and its complexity (with derivation).
	\item Matrix norms. Example of a norm that is not a matrix norm (does not satisfy submultiplicative property). Scalar product. Cauchy-Schwarz-Bunyakovsky inequality.
	\item Unitary matrices and their properties. Examples: Fourier matrix, permutation matrix, Householder reflections, Givens rotations.
	\item Matrix rank. Skeleton decomposition (with proof). Low-rank matrix storage and matrix-vector product. Instability of  matrix rank.
	\item Singular value decomposition (SVD). Proof of its existence. Null space and image of $A$ and $A^*$ in terms of singular vectors. Interpretation of SVD as a separation of variables.
	\item Eckart-Young theorem (proof for the spectral norm). Applications of SVD: latent semantic analysis, collaborative filtering in recommender systems, dense matrix compression.
	\item LU-decomposition. Criteria of its existence and uniqueness (both with proof). Connection with Gaussian elimination. Complexity of decomposing and solving linear system. The concept of pivoting.
	\item Cholesky factorization (with proof of its existence).
	\item Neumann series.  Bounds for $\|(I-A)^{-1}\|$ and $\dfrac{\|(A+\Delta A)^{-1} - A^{-1}\|}{\|A^{-1}\|}$ (both with proofs). Estimate of a perturbation of a solution in terms of condition number (with proof). Linear systems with ``consistent'' and ``inconsistent'' right-hand sides and behavior of solution error for large condition numbers.
	\item Least squares (LS) problem for over- and underdetermined linear systems. Methods of solving LS via QR, pseudoinverse, SVD, padding into a bigger system.
	\item Eigenvalue problem. Eigendecomposition. Criteria of eigendecomposition existence via algebraic and geometric multiplicities (no proof). Example of a nondiagonalizable matrix. Computing eigenvalues using characteristic equation. Is it a good idea? Gershgorin circles theorem (with proof).
	\item Power method and its convergence (with bound derivation). Application in PageRank.
	\item Schur decomposition (with proof). Normal matrices and their diagonalizability. Properties of eigenvalues of Hermitian, unitary and skew-Hermitian matrices. Variational concept for eigenvalues: Rayleigh quotient.
	\item QR decomposition. Proof of its existence. Connection with the Gram-Schmidt algorithm. QR via Cholesky decomposition.
	\item  QR via Householder reflections and Givens rotations (both with derivation). Comparison of these algorithms.
	\item QR algorithm. Convergence theorem (no proof). Accelerating convergence with shifts. Reducing complexity of QR algorithm from $\mathcal{O}(n^4)$ down to $\mathcal{O}(n^3)$.
	\item Divide and conquer algorithm for symmetric eigenvalue problems (with derivation).
	\item Jacobi method for eigenvalue problem. Its convergence (with proof).
	\item Sparse matrix arithmectics: {COO, LIL, CSR, CSC} formats. Their comparison. Blocking for increasing efficiency of sparse formats.
	\item Sparse LU decomposition, connection with graphs. Dependence of fill-in on ordering of graph nodes (with example). Nested dissection and spectral bisection algorithms.
	\item Richardson iteration. Optimal choice of parameter. Convergence estimate.
	\item Chebyshev iteration. Error bounds via Chebyshev polynomials.
	\item Solving linear system as an optimization problem: minimization of residual and energy functional. Approximation of a solution  of a linear system by a subspace, Galerkin projection. Krylov subspace and ill-posedness of natural Krylov basis.
	\item Ill-posedness of natural Krylov basis. Orthogonal basis in Krylov subspace. Arnoldi relation and its derivation. Lanczos process.
	\item Direct Lanczos method for solving linear systems. Conjugate gradient method.
	\item Convergence theory of the conjugate gradient method.
	\item MINRES method. GMRES method and its connection to Anderson acceleration. Disadvantages of GMRES for nonsymmetric systems. Idea of BiCG and BiCGStab.
	\item Preconditioning concept. Right and left preconditioners. Jacobi, Gauss-Seidel and SOR preconditioners.
	\item Incomplete LU for preconditioning. ILUt, ILU($k$), second-order ILU (ILU2).
	\item Inverse and Rayleigh quotient iterations. Speed of convergence. Convergence behavior of inexact inverse iteration.
	\item Block power method.
	\item Ritz approximation: Ritz values and vectors and their properties. Rayleigh-Ritz method.
	\item Arnoldi relation (with derivation). Lanczos and Arnoldi methods for solving partial eigenvalue problem. Their advantages and disadvantages.
	\item Convergence bound for $\lambda_{\max}$ in Lanczos method (with derivation).
	\item Preconditioned eigensolvers: PINVIT and LOBPCG methods. Finding parameters using Rayleigh-Ritz method.
	\item Jacobi-Davidson method: derivation of the Jacobi equation. Connection to the RQ iteration. Subspace acceleration.
	\item Fast Fourier Transform (FFT). Cooley-Tukey algorithm (with derivation).
	\item Continuous and discrete convolution. Eigendecomposition for circulant matrices (with proof).
	\item Product of Toeplitz matrix by vector via FFT. BTTB matrix-by-vector product via 2D FFT.
	\item The concept of displacement rank. Displacement rank of Toeplitz matrices.
	\item Matrix functions. Matrix exponential and its applications. Problem with evaluating matrix exponential and how to avoid it. Schur-Parlett algorithm. Matrix functions via Pade approximation.
	\item Sylvester and Lyapunov equations, their applications.
Solving Sylvester equation using Bartels-Stewart method.
\end{enumerate}

\end{document}


\documentclass[10pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage[export]{adjustbox}
\graphicspath{ {./images/} }
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{mhchem}
\usepackage{stmaryrd}
\usepackage{bbold}

\title{Jinchao Xu }

\author{Deep Learning Algorithms and\\
Analysis}
\date{}

\begin{document}
\maketitle

Summer 2020

\includegraphics[max width=\textwidth]{2022_01_05_20c4aca6821d56656c19g-2}

\section{Contents}
$1 \quad$ Nonlinear Models $\ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots$

1.1 Nonlinear classifiable sets $\ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots$

References $\ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots \ldots$

\includegraphics[max width=\textwidth]{2022_01_05_20c4aca6821d56656c19g-4}

\section{Nonlinear Models}
\subsection{Nonlinear classifiable sets}
In the section, we will extend the linearly separable sets to nonlinear case. A natural extension is like what kernel method does in SVM for binary case, we will introduce the so-called feature mapping.

Thus, we have the following natural extension for linearly separable by using feature mapping and original definition of linearly separable.

Definition 1 (nonlinearly separable sets). These data sets $A_{1}, A_{2}, \cdots, A_{k} \subset \mathbb{R}^{d}$ are called nonlinearly separable, if there exist a feature space $\mathbb{R}^{\tilde{d}}$ and a smooth (if it has derivatives of all orders) feature mapping
$$
\varphi: \mathbb{R}^{d} \mapsto \mathbb{R}^{d}
$$
such that
$$
\tilde{A}_{i}:=\varphi\left(A_{i}\right)=\left\{\tilde{x} \mid \tilde{x}=\varphi(x), x \in A_{i}\right\}, \quad i=1,2, \ldots, k
$$
are linearly separable.

Remark 1. 1. This definition is also consistent with the definition of linearly separable as we can just take $d=d$ and $\varphi=$ id if $A_{1}, A_{2}, \cdots, A_{k}$ are already linearly separable.

  \begin{enumerate}
    \setcounter{enumii}{2}
    \item The kernel method in SVM is mainly based on this idea for binary case $(\mathrm{k}=2)$ where they use kernel functions to approximate this $\varphi(x)$.

    \item For most commonly used deep learning models, they are all associated with a softmax mapping which means that we can interpret these deep learning models as the approximation for feature mapping $\varphi$.

  \end{enumerate}
However, softmax is not so crucial for this definition actually as we have the next equivalent result. Theorem 1. $A_{1}, A_{2}, \cdots, A_{k} \subset \mathbb{R}^{d}$ are nonlinearly separable is equivalent that there there exist a smooth classification function
$$
\psi: \mathbb{R}^{d} \mapsto \mathbb{R}^{k}
$$
such that for all $i=1: k$ and $j \neq i$
$$
\psi_{i}(x)>\psi_{j}(x), \quad \forall x \in A_{i}
$$
Proof. On the one hand, it is easy to see that if $A_{1}, A_{2}, \cdots, A_{k} \subset \mathbb{R}^{d}$ are nonlinearly separable then they we can just take
$$
\psi(x)=p(\varphi(x) ; \theta)
$$
where the $\boldsymbol{p}(y ; \theta)$ is the softmax function for linearly separable sets $\varphi\left(A_{i}\right)$ for $i=$ $1,2, \cdots, k$.

On the other hand, let assume that $\psi$ is the smooth classification functions for $A_{1}, A_{2}, \cdots, A_{k} \subset \mathbb{R}^{d} .$ We claim that, we can take $\varphi(x)=\psi(x)$ and then
$$
\varphi\left(A_{1}\right), \varphi\left(A_{2}\right), \cdots, \varphi\left(A_{k}\right) \subset \mathbb{R}^{k} \quad(\tilde{d}=k)
$$
will be linearly separable. Actually, if you take $\theta=(I, 0)$ in softmax mapping $\boldsymbol{p}(x ; \theta)$, then the monotonicity of $e^{x}$ show that for all $i=1: k$ and $j \neq i$
$$
\boldsymbol{p}_{i}(\varphi(x) ; \theta)=\frac{e^{\psi_{i}(x)}}{\sum_{i=1}^{k} e^{\psi_{i}(x)}}>\frac{e^{\psi_{j}(x)}}{\sum_{i=1}^{k} e^{\psi_{i}(x)}}=\boldsymbol{p}_{j}(\varphi(x) ; \theta), \quad \forall x \in A_{i} .
$$
]

Similarly to linearly separable sets, we have the next lemme for $k=2$.

Lemma 1. A $_{1}$ and $A_{2}$ are nonlinearly separable is equivalent that there exists a function $\varphi: \mathbb{R}^{d} \mapsto \mathbb{R}$ such that
$$
\varphi(x)>0 \quad \forall x \in A_{1} \quad \text { and } \quad \varphi(x)<0 \quad \forall x \in A_{2}
$$
Proof. Based the equivalence of nonlinearly separable sets, there exists $\psi_{1}(x)$ and $\psi_{2}(2)$ such that for all $i=1: 2$ and $j \neq i$
$$
\psi_{i}(x)>\psi_{j}(x), \quad \forall x \in A_{i}
$$
Then, we can just take
$$
\varphi(x)=\psi_{1}(x)-\psi_{2}(x)
$$
On the other hand, if there exist $\varphi(x)$, then we can construct $\psi_{1}(x)$ and $\psi_{2}(2)$ as
$$
\psi_{1}(x)=\frac{1}{2} \varphi(x) \quad \text { and } \quad \psi_{2}(x)=-\frac{2}{2} \varphi(x)
$$
Remark 2. Here we mention that, we only assume that for all $i=1: k$ and $j \neq i$ we have $\psi_{i}(x)>\psi_{j}(x), \forall x \in A_{i}$ for nonlinearly separable. We do not assume that $\psi_{i}(x) \geq 0$ or $\sum_{i=1}^{k} \psi_{i}(x)=1$, which means that $\psi(x)=\left(\begin{array}{c}\psi_{1}(x) \\ \psi_{2}(x) \\ \vdots \\ \psi_{k}(x)\end{array}\right)$ is not a discrete probability distribution over all k classes.

The previous theorem shows that softmax function is not so crucial in nonlinearly separable case. Combined with deep learning models, we have the following understanding about what deep learning models are approximating.

  \begin{enumerate}
    \item If a classification model is followed with a softmax, then the it is approximating the feature mapping $\varphi: \mathbb{R}^{d} \mapsto \mathbb{R}^{\bar{d}}$.

    \item If the classification model dose not followed by softmax, then it is approximating $\psi: \mathbb{R}^{d} \mapsto \mathbb{R}^{k}$ directly.

  \end{enumerate}
Example 1. Consider $k=2$ and $A_{1} \subset\left\{(x, y) \mid x^{2}+y^{2}<1\right\}, \quad A_{2} \subset\left\{(x, y) \mid x^{2}+y^{2}>1\right\}$, then we can have the following nonlinear feature mapping:

\includegraphics[max width=\textwidth]{2022_01_05_20c4aca6821d56656c19g-7}

Here we have the following comparison for linear and nonlinear models from the viewpoint of loss functions:

Linear case (Logistic regression):
$$
L_{\lambda}(\theta)=\sum_{j=1}^{N} \ell\left(y_{j}, p\left(x_{j} ; \theta\right)\right)+\lambda R(\|\theta\|)
$$
Nonlinear case:
$$
L_{\lambda}(\theta)=\sum_{j=1}^{N} \ell\left(y_{j}, p\left(\varphi\left(x_{j} ; \theta_{1}\right) ; \theta_{2}\right)\right)+\lambda R(\|\theta\|)
$$
Remark 3. We have the following remarks.
$$
\text { 1. } \ell(q, p)=\sum_{i=1}^{k}-q_{i} \log p_{i} \leftrightarrow \text { cross-entropy }
$$

  \begin{enumerate}
    \setcounter{enumii}{2}
    \item $p(x ; \theta)=\operatorname{softmax}(W x+b)$ where $\theta=(W, b)$

    \item $\theta=\left(\theta_{1}, \theta_{2}\right)$ for nonlinear case

    \item $\lambda R(\|\theta\|) \leftrightarrow$ regularization term

  \end{enumerate}
In general, we have the following popular nonlinear models for $\varphi(x ; \theta)$

  \begin{enumerate}
    \item Polynomials.

    \item Piecewise polynomials (finite element method).

    \item Kernel functions in SVM.

    \item Deep neural networks.

  \end{enumerate}
\section{References}
[1] L. Chen, P. Sun, and J. Xu. Optimal anisotropic meshes for minimizing interpolation errors in $l^{p}$-norm. Mathematics of Computation, $76(257): 179-204,2007 .$

[2] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016

[3] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, pages 630-645. Springer, $2016 .$

[4] M. Leshno, V.Y. Lin, A. Pinkus, and S. Schocken. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6):861-867, 1993 .

[5] A. Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, $8: 143-195,1999 .$

[6] L. R. Scott and S. Zhang. Finite element interpolation of nonsmooth functions satisfying boundary conditions. Mathematics of Computation, $54(190): 483-493$, $1990 .$

[7] J. Xu. Finite element methods. Lecture notes, 2020 .

\end{document}