%%
%% Automatically generated file from DocOnce source
%% (https://github.com/hplgit/doconce/)
%%
%%

%-------------------- begin preamble ----------------------

\documentclass[%
oneside,                 % oneside: electronic viewing, twoside: printing
final,                   % draft: marks overfull hboxes, figures with paths
10pt]{article}

\listfiles               %  print all files needed to compile this document

\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
\usepackage[table]{xcolor}
\usepackage{bm,ltablex,microtype}

\usepackage[pdftex]{graphicx}

\usepackage[T1]{fontenc}
%\usepackage[latin1]{inputenc}
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern

% Hyperlinks in PDF:
\definecolor{linkcolor}{rgb}{0,0,0.4}
\usepackage{hyperref}
\hypersetup{
    breaklinks=true,
    colorlinks=true,
    linkcolor=linkcolor,
    urlcolor=linkcolor,
    citecolor=black,
    filecolor=black,
    %filecolor=blue,
    pdfmenubar=true,
    pdftoolbar=true,
    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
    }
%\hyperbaseurl{}   % hyperlinks are relative to this root

\setcounter{tocdepth}{2}  % levels in table of contents

% --- fancyhdr package for fancy headers ---
\usepackage{fancyhdr}
\fancyhf{} % sets both header and footer to nothing
\renewcommand{\headrulewidth}{0pt}
\fancyfoot[LE,RO]{\thepage}
% Ensure copyright on titlepage (article style) and chapter pages (book style)
\fancypagestyle{plain}{
  \fancyhf{}
  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2021, "Computational Physics II FYS4411/FYS9411":"http://www.uio.no/studier/emner/matnat/fys/FYS4411/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
%  \renewcommand{\footrulewidth}{0mm}
  \renewcommand{\headrulewidth}{0mm}
}
% Ensure copyright on titlepages with \thispagestyle{empty}
\fancypagestyle{empty}{
  \fancyhf{}
  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2021, "Computational Physics II FYS4411/FYS9411":"http://www.uio.no/studier/emner/matnat/fys/FYS4411/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
  \renewcommand{\footrulewidth}{0mm}
  \renewcommand{\headrulewidth}{0mm}
}

\pagestyle{fancy}

% prevent orhpans and widows
\clubpenalty = 10000
\widowpenalty = 10000

% --- end of standard preamble for documents ---

% insert custom LaTeX commands...

\raggedbottom
\makeindex
\usepackage[totoc]{idxlayout}   % for index in the toc
\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc

%-------------------- end preamble ----------------------

\begin{document}

% matching end for #ifdef PREAMBLE

\newcommand{\exercisesection}[1]{\subsection*{#1}}

% ------------------- main content ----------------------

% ----------------- title -------------------------

\thispagestyle{empty}

\begin{center}
{\LARGE\bf
\begin{spacing}{1.25}
Project 2, Modelling time-dependent quantum mechanical many-body systems. Deadline June 1
\end{spacing}
}
\end{center}

% ----------------- author(s) -------------------------

\begin{center}
{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS4411/index-eng.html}}{Computational Physics II FYS4411/FYS9411}}
\end{center}

    \begin{center}
% List of all institutions:
\centerline{{\small Department of Physics, University of Oslo, Norway}}
\end{center}

% ----------------- end author(s) -------------------------

% --- begin date ---
\begin{center}
Apr 7, 2021
\end{center}
% --- end date ---

\vspace{1cm}

\subsection*{Introduction}

The aim of this project is to use the time-dependent Hartree-Fock method to
model electronic systems subject to an intense semi-classical laser. We will
study how these systems react when they are perturbed by an external field.

\subsection*{Theoretical background}

We start our analysis by considering time-independent Hartree-Fock theory, as
much of the tools and formalism will be the same as in the time-dependent case.
Consider the time-independent Schrödinger equation

\begin{align}
    \hat{H}\vert\Psi\rangle = E\vert\Psi\rangle,
\end{align}

where $\hat{H}$ is a time-independent Hamiltonian, $\vert\Psi\rangle$ an
eigenfunction of the Hamiltonian and $E$ the corresponding eigenenergy. In
Hartree-Fock theory we assume that the full many-body wavefunction
$\vert\Psi\rangle$ can be approximated well by a single Slater determinant. That
is,

\begin{align}
    \vert\Psi\rangle = \vert\Phi\rangle = \vert\phi_1, \dots, \phi_N\rangle,
\end{align}

where we label this Slater determinant by $\vert\Phi\rangle$. In a coordinate
representation this Slater determinant can be represented by

\begin{align}
    \Phi(x_1, \dots, x_N)
    = \langle x_1, \dots, x_N \vert \phi_1, \dots, \phi_N \rangle
    = \frac{1}{\sqrt{N!}}
    \begin{vmatrix}
        \phi_1(x_1) & \dots & \phi_1(x_N) \\
        \vdots & \ddots & \vdots \\
        \phi_N(x_1) & \dots & \phi_N(x_N)
    \end{vmatrix},
\end{align}

% TODO: Is this the correct ordering of the indices?
where the coordinates $x_i$ contain both position and spin. Hartree-Fock is a
method for finding the "best" such Slater determinant, that is, the single
Slater determinant that will yield the lowest energy for the given Hamiltonian.
We achieve this by finding the \emph{molecular orbitals} $\vert\phi_i\rangle$
that minimizes the energy using the variational method. In Hartree-Fock it is
the molecular orbitals that are the unknowns we are trying to find. More
specifically, we use \emph{atomic orbitals} $\vert\chi_{\alpha}\rangle$ as our
initial guess, and the do a basis transformation

\begin{align}
    \vert\phi_j\rangle = \sum_{\alpha} C_{\alpha j}\vert\chi_{\alpha}\rangle.
\end{align}

Thus, we are interested in finding the coefficients $C_{\alpha j}$ such that we
can transform from the known atomic orbitals $\vert\chi_{\alpha}\rangle$ to the
optimal molecular orbitals $\vert\phi_j\rangle$.

% Alternative derivation
Recall that we in time-indepedent Hartree-Fock theory approximate the true
many-body wavefunction $\vert\Psi\rangle$ by a single Slater determinant
$\vert\Phi\rangle$ containing the $N$ occupied \emph{molecular orbitals}
$\vert\phi_j\rangle$ subject to the constraint that

\begin{align}
    \langle \phi_i \vert \phi_j \rangle = \delta_{ij},
\end{align}

that is, the orbitals are orthonormal. The expectation value of the Hamiltonian
is then

\begin{align}
    \langle \Phi \vert \hat{H} \vert \Phi \rangle
    = \sum_{i} \langle \phi_i \vert \hat{h} \vert \phi_i \rangle
    + \frac{1}{2} \sum_{ij}
    \langle \phi_i \phi_j \vert \hat{u} \vert \phi_i \phi_j \rangle_{AS},
\end{align}

where the many-body Hamiltonian is given by

\begin{align}
    \hat{H} = \hat{h} + \hat{u}.
\end{align}

Here $\hat{h}$ is the one-body part of the Hamiltonian containing the kinetic
energy and an external potential. The two-body part $\hat{u}$ will contain the
two-body interactions between the particles in the system. In our case these
interactions will be the Coulomb interaction. We have labeled the
\emph{anti-symmetric two-body matrix elements} by

\begin{align}
    \langle \phi_p \phi_q \vert \hat{u} \vert \phi_r \phi_s \rangle_{AS}
    \equiv
    \langle \phi_p \phi_q \vert \hat{u} \vert \phi_r \phi_s \rangle
    - \langle \phi_p \phi_q \vert \hat{u} \vert \phi_s \phi_r \rangle.
\end{align}

By constructing the Lagrangian functional

\begin{align}
    \mathcal{L}
    &= \langle \Phi \vert \hat{H} \vert \Phi \rangle
    - \sum_{ij} \lambda_{ji} \left(
        \langle \phi_i \vert \phi_j \rangle - \delta_{ij}
    \right)
    \\
    &=
    \sum_{i} \langle \phi_i \vert \hat{h} \vert \phi_i \rangle
    + \frac{1}{2} \sum_{ij}
    \langle \phi_i \phi_j \vert \hat{u} \vert \phi_i \phi_j \rangle_{AS}
    - \sum_{ij} \lambda_{ji} \left(
        \langle \phi_i \vert \phi_j \rangle - \delta_{ij}
    \right),
\end{align}

where $\lambda_{ji}$ are the Lagrange multipliers used to optimize the
Lagrangian under the orthonormality constraint of the molecular orbitals.

We consider a system of electrons confined in a pure two-dimensional
isotropic harmonic oscillator potential, with an idealized  total Hamiltonian given by

\begin{equation}
\label{eq:finalH}
\hat{H}=\sum_{i=1}^{N} \left(  -\frac{1}{2} \nabla_i^2
+ \frac{1}{2} \omega^2r_i^2  \right)+\sum_{i<j}\frac{1}{r_{ij}},
\end{equation}
where natural units ($\hbar=c=e=m_e=1$) are used and all energies are in
so-called atomic units a.u. We will study systems of many electrons $N$ as
functions of the oscillator frequency  $\omega$ using the above Hamiltonian.
The Hamiltonian includes a standard harmonic oscillator part

\begin{equation*}
\hat{H}_0=\sum_{i=1}^{N} \left(  -\frac{1}{2} \nabla_i^2 + \frac{1}{2} \omega^2r_i^2  \right),
\end{equation*}
and the repulsive interaction between two electrons given by

\begin{equation*}
\hat{H}_1=\sum_{i<j}\frac{1}{r_{ij}},
\end{equation*}
with the distance between electrons given by $r_{ij}=\vert \bm{r}_1-\bm{r}_2\vert$. We define the
modulus of the positions of the electrons (for a given electron $i$) as $r_i = \sqrt{r_{i_x}^2+r_{i_y}^2}$.

The aim of this project is to develop a coupled cluster doubles (CCD) code, where $2p-2h$ excitations are included only.

We will start with a two-electron problem and compare our results to those of Taut, see reference [1] below.

The ansatz for the ground state  is given by
  \begin{equation*}
     \vert \Psi_0\rangle = \vert \Psi_{CC}\rangle = e^{\hat{T}} \vert
     \Phi_0\rangle = \left( \sum_{n=1}^{N} \frac{1}{n!} \hat{T}^n
     \right) \vert \Phi_0\rangle,
  \end{equation*}
  where $N$ represents the maximum number of particle-hole excitations
  and $\hat{T}$ is the cluster operator defined as
  \begin{align*}
              \hat{T} &= \hat{T}_1 + \hat{T}_2 + \ldots + \hat{T}_N
              \\ \hat{T}_n &= \left(\frac{1}{n!}\right)^2
              \sum_{\substack{ i_1,i_2,\ldots i_n \\ a_1,a_2,\ldots
                  a_n}} t_{i_1i_2\ldots i_n}^{a_1a_2\ldots a_n}
              a_{a_1}^\dagger a_{a_2}^\dagger \ldots a_{a_n}^\dagger
              a_{i_n} \ldots a_{i_2} a_{i_1}.
          \end{align*}
      The energy is given by
      \begin{equation*}
          E_{\mathrm{CC}} = \langle\Phi_0\vert \overline{H}\vert
          \Phi_0\rangle,
      \end{equation*}
      where $\overline{H}$ is a similarity transformed Hamiltonian
      \begin{align*}
          \overline{H}&= e^{-\hat{T}} \hat{H}_N e^{\hat{T}}
          \\ \hat{H}_N &= \hat{H} - \langle\Phi_0\vert \hat{H} \vert
          \Phi_0\rangle.
      \end{align*}
      The coupled cluster energy is a function of the unknown cluster
      amplitudes $t_{i_1i_2\ldots i_n}^{a_1a_2\ldots a_n}$, given by
      the solutions to the amplitude equations
      \begin{equation}\label{eq:amplitudeeq}
          0 = \langle\Phi_{i_1 \ldots i_n}^{a_1 \ldots a_n}\vert
          \overline{H}\vert \Phi_0\rangle.
      \end{equation}
In order to set up the above equations,
the similarity transformed Hamiltonian $\overline{H}$ is expanded
  using the Baker-Campbell-Hausdorff expression,
      \begin{equation}\label{eq:bch}
          \overline{H}= \hat{H}_N + \left[ \hat{H}_N, \hat{T} \right]
          + \frac{1}{2} \left[\left[ \hat{H}_N, \hat{T} \right],
            \hat{T}\right] + \ldots + \frac{1}{n!} \left[
            \ldots \left[ \hat{H}_N, \hat{T} \right], \ldots \hat{T}
            \right] +\dots
      \end{equation}
  and simplified using the connected cluster theorem
      \begin{equation*}
          \overline{H}= \hat{H}_N + \left( \hat{H}_N \hat{T}\right)_c
          + \frac{1}{2} \left( \hat{H}_N \hat{T}^2\right)_c + \dots +
          \frac{1}{n!} \left( \hat{H}_N \hat{T}^n\right)_c +\dots
      \end{equation*}
We will discuss parts of the the derivation below.

  We will now approximate the cluster operator $\hat{T}$ to include
  only $2p-2h$ correlations. This leads to the so-called CCD
  approximation, that is
  \[
  \hat{T}\approx
  \hat{T}_2=\frac{1}{4}\sum_{abij}t_{ij}^{ab}a^{\dagger}_aa^{\dagger}_ba_ja_i,
  \]
  meaning that we have
  \[
  \vert \Psi_0 \rangle \approx \vert \Psi_{CCD} \rangle =
  \exp{\left(\hat{T}_2\right)}\vert \Phi_0\rangle.
  \]
  Inserting these equations in the expression for the computation of
  the energy we have, with a Hamiltonian defined with respect to a
  general reference vacuum
  \[
  \hat{H}=\hat{H}_N+E_{\mathrm{ref}},
  \]
  with
  \[
  \hat{H}_N=\sum_{pq}\langle p \vert \hat{f} \vert q \rangle
  a^{\dagger}_pa_q + \frac{1}{4}\sum_{pqrs}\langle pq \vert \hat{v}
  \vert rs \rangle a^{\dagger}_pa^{\dagger}_qa_sa_r,
  \]
  we obtain that the energy can be written as
  \[
  \langle \Phi_0 \vert
  \exp{\left(-\hat{T}_2\right)}\hat{H}_N\exp{\left(\hat{T}_2\right)}\vert
  \Phi_0\rangle = \langle \Phi_0 \vert \hat{H}_N(1+\hat{T}_2)\vert
  \Phi_0\rangle = E_{CCD}.
  \]
  This quantity becomes
  \[
  E_{CCD}=E_{\mathrm{ref}}+\frac{1}{4}\sum_{abij}\langle ij \vert
  \hat{v} \vert ab \rangle t_{ij}^{ab},
  \]
  where the latter is the correlation energy from this level of
  approximation of coupled cluster  theory.  Similarly, the expression for the
  amplitudes reads
  \[
  \langle \Phi_{ij}^{ab} \vert
  \exp{\left(-\hat{T}_2\right)}\hat{H}_N\exp{\left(\hat{T}_2\right)}\vert
  \Phi_0\rangle = 0.
  \]
  These equations can be reduced to (after several applications of
  Wick's theorem), for all $i > j$ and all $a > b$,
  \begin{align}
  0 = \langle ab \vert \hat{v} \vert ij \rangle +
  \left(\epsilon_a+\epsilon_b-\epsilon_i-\epsilon_j\right)t_{ij}^{ab}+\frac{1}{2}\sum_{cd} \langle ab \vert \hat{v} \vert
  cd \rangle t_{ij}^{cd}+\frac{1}{2}\sum_{kl} \langle kl \vert \hat{v}
  \vert ij \rangle t_{kl}^{ab}+\hat{P}(ij\vert ab)\sum_{kc} \langle kb
  \vert \hat{v} \vert cj \rangle t_{ik}^{ac} & \nonumber
  \\ +\frac{1}{4}\sum_{klcd} \langle kl \vert \hat{v} \vert cd \rangle
  t_{ij}^{cd}t_{kl}^{ab}+\hat{P}(ij)\sum_{klcd} \langle kl \vert
  \hat{v} \vert cd \rangle t_{ik}^{ac}t_{jl}^{bd}-\frac{1}{2}\hat{P}(ij)\sum_{klcd} \langle kl \vert \hat{v} \vert
  cd \rangle t_{ik}^{dc}t_{lj}^{ab}-\frac{1}{2}\hat{P}(ab)\sum_{klcd}
  \langle kl \vert \hat{v} \vert cd \rangle t_{lk}^{ac}t_{ij}^{db},&
  \label{eq:ccd}
  \end{align}
  where we have defined
  \[
  \hat{P}\left(ab\right)= 1-\hat{P}_{ab},
  \]
  where $\hat{P}_{ab}$ interchanges two particles occupying the
  quantum numbers $a$ and $b$.  The operator $\hat{P}(ij\vert ab)$ is
  defined as
  \[
  \hat{P}(ij\vert ab) = (1-\hat{P}_{ij})(1-\hat{P}_{ab}).
  \]
  The single-particle energies $\epsilon_p$ are normally taken to be Hartree-Fock single-particle energies.
  Recall also that the unknown amplitudes $t_{ij}^{ab}$ represent
  anti-symmetrized matrix elements, meaning that they obey the same
  symmetry relations as the two-body interaction, that is
  \[
  t_{ij}^{ab}=-t_{ji}^{ab}=-t_{ij}^{ba}=t_{ji}^{ba}.
  \]
  The two-body matrix elements are also anti-symmetrized, meaning that
  \[
  \langle ab \vert \hat{v} \vert ij \rangle = -\langle ab \vert
  \hat{v} \vert ji \rangle= -\langle ba \vert \hat{v} \vert ij
  \rangle=\langle ba \vert \hat{v} \vert ji \rangle.
  \]
  The non-linear equations for the unknown amplitudes $t_{ij}^{ab}$
  are solved iteratively.

In order to develop a program, chapter 8 of the recent \href{{http://www.springer.com/us/book/9783319533353}}{Lecture Notes in Physics (volume 936)} is highly recommended as literature.
All material is available from the \href{{https://github.com/ManyBodyPhysics/LectureNotesPhysics/blob/master/doc/src/lnp.pdf}}{source site}. Example of CCD codes are available from the \href{{https://github.com/ManyBodyPhysics/LectureNotesPhysics/tree/master/Programs/Chapter8-programs/cpp/CCD}}{program site}. These can be used to benchmark your own program.

\paragraph{Project 2 a):}
We will use our Hartree-Fock basis from project 1 to define matrix elements and the single-particle energies to be used in the CCD equations. The Hartree-Fock basis defines the so-called reference energy
\begin{equation}
  E_{\mathrm{ref}} = \sum_{i\le F} \sum_{\alpha\beta}
  C^*_{i\alpha}C_{i\beta}\langle \alpha | h | \beta \rangle +
  \frac{1}{2}\sum_{ij\le F}\sum_{{\alpha\beta\gamma\delta}}
  C^*_{i\alpha}C^*_{j\beta}C_{i\gamma}C_{j\delta}\langle
  \alpha\beta|\hat{v}|\gamma\delta\rangle.
\end{equation}
You will need to transform the matrix elements from the harmonic oscillator basis to the Hartree-Fock basis.
The first step is to program
\begin{equation}
  \langle pq \vert \hat{v} \vert rs\rangle_{AS}=
  \sum_{{\alpha\beta\gamma\delta}}
  C^*_{p\alpha}C^*_{q\beta}C_{r\gamma}C_{s\delta}\langle
  \alpha\beta|\hat{v}|\gamma\delta\rangle_{AS},
\end{equation}
where the coefficients are those from the last Hartree-Fock iteration and the matrix elements are all anti-symmetrized.
You can extend your Hartree-Fock program to write out these matrix elements after the last Hartree-Fock iteration.
Make sure that your matrix elements are structured according to conserved quantum numbers, avoiding thereby the write out of many zeros.

To test that your matrix elements are set up correctly, when you read in these matrix elements in the CCD code, make sure that the reference energy from your Hartree-Fock calculations are reproduced.

\paragraph{Project 2 b):}
Set up a code which solves the CCD equation by encoding the equations as they stand, that is follow the mathematical expressions and perform the sums over all single-particle states.  Compute the energy of the two-electron systems using
all single-particle states that were needed in order to obtain the Hartree-Fock limit. Compare these with Taut's results for $\omega=1$ a.u.  Since you do not include singles you will not get the exact result. If you wish to include singles, you will able to obtain the exact results in a basis with at least ten major oscillator shells.
Perform also calculations with $N=6$, $N=12$ and $N=20$ electrons and compare with reference [2] of Pedersen et al below.

\paragraph{Project 2 c):}
The next step consists in rewriting the equations in terms of matrix-matrix multiplications and subdividing
the matrix elements and operations in terms of two-particle configuration that conserve total spin projection and projection of the orbital momentum. Rewrite also the equations in terms of so-called intermediates, as detailed
in section 8.7 of \href{{https://github.com/ManyBodyPhysics/LectureNotesPhysics/blob/master/doc/src/lnp.pdf}}{Lietz et al}.
This section gives a detailed description on how to build a coupled cluster code and is highly recommended.

Rerun your calculations for $=2$, $N=6$, $N=12$ and $N=20$ electrons using your optimal Hartree-Fock basis. Make sure your results from 2b) stay the same.

Calculate as well ground state energies for $\omega=0.5$ and $\omega=0.1$. Try to compare with eventual variational
Monte Carlo results from other students, if possible.

\paragraph{Project 2 d):}
The final step is to parallelize your CCD code using either OpenMP or MPI and do a performance analysis. Use the $N=6$ case. Make a performance analysis by timing your serial code
with and without vectorization. Perform several runs and compute an average timing analysis
with and without vectorization. Comment your results.

Compare thereafter your serial code(s)  with the speedup you get by parallelizing your code, running either OpenMP or MPI or both.
Do you get a near $100\%$ speedup with the parallel version? Comment again your results and perform timing benchmarks several times in order
to extract  an average performance time.

\paragraph{Literature.}
\begin{enumerate}
 \item M. Taut, Phys. Rev. A \textbf{48}, 3561 - 3566 (1993).

 \item M. L. Pedersen, G. Hagen, M. Hjorth-Jensen, S. Kvaal,  and F. Pederiva, Phys. Rev. B \textbf{84}, 115302 (2011)
\end{enumerate}

\noindent
\subsection*{Introduction to numerical projects}

Here follows a brief recipe and recommendation on how to write a report for each
project.

\begin{itemize}
  \item Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  \item Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  \item Include the source code of your program. Comment your program properly.

  \item If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  \item Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  \item Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  \item Try to give an interpretation of you results in your answers to  the problems.

  \item Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  \item Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
\end{itemize}

\noindent
\subsection*{Format for electronic delivery of report and programs}

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

\begin{itemize}
  \item Use canvas to hand in your projects, log in  at  \href{{http://canvas.uio.no}}{\nolinkurl{http://canvas.uio.no}} with your normal UiO username and password.

  \item Upload \textbf{only} the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  The full version of the codes should be in your github repository.

  \item In your github repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  \item Still in your github make a folder where you place your codes.

  \item In this and all later projects, you should include tests (for example unit tests) of your code(s).

  \item Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.
\end{itemize}

\noindent
Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.

% ------------------- end of main content ---------------

\end{document}


%%% PREAMBLE - Do not touch %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage[ansinew]{inputenc}
\usepackage[portuges,brazil,english]{babel}
\usepackage{model}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{color}
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\input{pics/abaco}

\cvprfinalcopy % *** Uncomment this line for the final submission
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
\ifcvprfinal\pagestyle{empty}\fi

\newcommand{\TODO}[1]{TODO: #1}
\newcommand{\CITEONE}[2]{\mbox{#1 \cite{#2}}}
\newcommand{\CITETWO}[3]{\mbox{#1 and #2 \cite{#3}}}
\newcommand{\CITEN}[2]{\mbox{#1 et al. \cite{#2}}}

%%% Report beginning %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

%%% Title and authors %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{MC886 - Assignment 1\\
Predicting social network shares using Linear Regression}
\author{Gustavo de Mello Crivelli\thanks{Undergrad student in the Institute of Computing, University of Campinas (Unicamp). \textbf{Contact}: \tt\small{gmcrivelli@gmail.com}}\\
RA 136008}

%%% Abstract %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\maketitle
\begin{abstract}
Using a dataset of over 30.000 \CITEONE{entries}{Dataset}, Python was used to build and train a predictor for the number of times a given publication will be shared in social networks. A separate dataset was then used for testing. An initial attempt with only basic linear regression was able to make predictions with an error of 3183 shares, and managed to predict 2.6\% of the test dataset with an error of 100 shares. Further optimizations using Gradient Descent, feature scaling and regularization led to a final predictor able to guess shares with an error of 2788 shares, and had 8.3\% of predictions within 100 shares of the correct value.
\end{abstract}

%%% Introduction %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Social networks are bustling hubs of information exchange. Being able to predict how far an article might go in terms of social network sharing, as well as understanding what makes a publication become viral, is invaluable for any content creator.

This report details the process of attempting to build a predictor able to do the above, using a range of Linear Regression and Gradient Descent techniques.

\subsection{Dataset}

The dataset used for this work was published by \hyperlink{www.mashable.com}{www.mashable.com}\CITEONE{}{Dataset}. Their content as the rights to reproduce it belongs to them. Hence, this dataset does not share the original content but some statistics associated with it\CITEONE{}{Portugal}.

The training dataset contains 31715 entries, each with 58 total predictive attributes and a target attribute (the number of shares). Each feature has a different value range, and some of them are discrete. All of this needs to be dealt with in order to achieve better results in prediction.

The test dataset contains 7929 entries and is similar to the training dataset in terms of attributes. It is used to validate the predictor learned while using the training dataset.

%%% Add section %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proposed Solutions}
All results are listed in the Results and Discussion section.

\subsection{Linear Regression}

The first attempt was made using basic linear regression. Given a dataset $X$ with $m$ entries, an entry $x^{(i)}$ from that dataset and a target value $y^{(i)}$, we want to find an array of coefficients $\theta$ such that, given $h_\theta(x) = \theta \cdot x$, the cost function $J(\theta) = \frac{1}{2m}\sum_{i=1}^{m}(h_\theta(x^{(i)}) - y^{(i)})^2$ is \CITEONE{minimal}{Classroom}.

For this, \CITEONE{scikit-learn}{scikit-learn} was used. No further optimizations were done at this time.

\subsection{Feature Scaling}

The second proposed solution utilized the same linear regression as above, but normalized the features in the training to keep their values in the range $[0,1]$. The test dataset was also normalized, using the same parameters found during the training set normalization.

Surprisingly, this performed with worse results than the non-normalized attempt.

\subsection{Gradient Descent}
\label{sec:gd}

The third proposed solution moved on to Gradient Descent. Let $n$ be the number of coefficients in the $\theta$ array. In order to find the minimum cost of $J(\theta)$, the coefficient array $\theta$ is iteratively updated until the function converges. This is described by $\theta_j := \theta_j - \alpha\frac{\delta}{\delta\theta_j}J(\theta)$ for $0 \leq j < n$, where $\alpha$ is a scalar representing the learning rate.

The feature scaling proposed above was also used in this and in the following proposed solutions. For this solution, 10000 iterations were made, and an initial $\theta$ was built with random values in the range $(-1000,1000)$.

Figure \ref{fig:gd1} details the variations in the mean absolute error of predicted shares as the training iterates. The high values are a good indication of overfitting to the training dataset, and this will need to be dealt with.

\begin{figure}
\begin{center}
	\includegraphics[width=0.99\columnwidth]{pics/gd1.png}
	\caption{Mean error by Gradient Descent iteration.\label{fig:gd1}}
\end{center}
\end{figure}

\subsection{Gradient Descent with Feature Dropping}
\label{sec:gdfd}

The next step was to remove superfluous features that might be contributing with noise, and leading the predictor to overfit to the training data. For this, the coefficient array found in the previous round of training was analyzed, and every feature whose corresponding coefficient had an absolute value inferior to 1000 was removed from consideration.

After this procedure, the only features still in consideration were the ones in the set $F = {6,8,11,18,22,23,24,25,26,27,28,29,30,29,41,43}$. (Please reference the \CITEONE{dataset}{Dataset} for more details.

Furthermore, the dataset itself was filtered, and entries with anomalous features were removed. For instance, all entries marked as having zero words in the content (Feature 3), were likely the result of a malformed observation, as can be easily verified by accessing the link to the publication in that entry.

The results of these changes can be seen in Figure \ref{fig:gd2}

\begin{figure}
\begin{center}
	\includegraphics[width=0.99\columnwidth]{pics/gd2.png}
	\caption{Mean error by Gradient Descent iteration, with feature dropping.\label{fig:gd2}}
\end{center}
\end{figure}

\subsection{Gradient Descent with Regularization}
\label{sec:gdr}

It was then attempted to build a more complex predictor by changing $h_\theta(x)$ to be a second-degree polynomial function. For this, a new feature array was produced for each entry, composed of an array of the features mentioned in the previous step, concatenated with an array containing the square of those features.

To minimize overfitting, a $\lambda$ regularization parameter was introduced, so that the cost function becomes $J(\theta) = \frac{1}{2m}(\sum_{i=1}^{m}(h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\sum_{j=1}^{n}\theta_{j}^{2})$. The intent of this is to penalize more complex models over simpler ones, in a way that will reinforce important features while reducing the influence of noise.

The lambda with the best results was found to be $\lambda = 1000$. The results of these changes can be seen in Figure \ref{fig:gd3}

\begin{figure}
\begin{center}
	\includegraphics[width=0.99\columnwidth]{pics/gd3.png}
	\caption{Mean error by Gradient Descent iteration, with feature dropping and regularization.\label{fig:gd3}}
\end{center}
\end{figure}

\subsection{Gradient Descent with Further Feature Dropping}
\label{sec:gdrfd}

A second attempt was made at choosing the most relevant features. Instead of only taking the features with the highest coefficients, the following algorithm was executed:

\begin{enumerate}
    \item Start with an empty set of selected features $F$.
    \item For every feature $f$ in the dataset that is still not in $F$, train and evaluate the predictor using only $F \cup \{f, f^2\}$ as features.
    \item Select the feature $f$ that resulted in the lowest mean error in Step 2. If it is lower than the previous mean error, add $\{f, f^2\}$ permanently to $F$. Otherwise, stop.
    \item Repeat from Step 2.
\end{enumerate}

Using this, a new set of features was selected, composed of features $F = {6,7,8,19,21,25,27,29,35}$. This takes care of the discrete features, as the only remaining discrete feature is Feature 25 (whether or not the article was published on a Friday). The results of this filtering can be seen in Figure \ref{fig:gd4}, and are considerably better than in the previous attempts.

\begin{figure}
\begin{center}
	\includegraphics[width=0.99\columnwidth]{pics/gd4.png}
	\caption{Mean error by Gradient Descent iteration, with further feature dropping and regularization.\label{fig:gd4}}
\end{center}
\end{figure}

\subsection{Normal Equations}

At this point, a different direction was attempted by using Normal Equations to analytically solve the regression. All of the techniques adopted regarding dataset filtering and feature processing were also applied here.

Since calculating $\theta$ in this manner requires a lot of computing power, a smaller sample of the original dataset was used. This contained the first $10000$ entries that were considered to be usable.

%%% Add section %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results and Discussion}

Table \ref{table:results} contains the results of the above experimentation.

\begin{table*}
\begin{center}
    \begin{tabular}{ | l | l | l | l |}
    \hline
     & \textbf{Mean error} & \textbf{Accuracy within 500 shares} & \textbf{Accuracy within 100 shares} \\ \hline
    \textbf{(2.1)} Linear Regression (LR) & 3183.5 & 14.6\% & 2.6\% \\ \hline
    \textbf{(2.2)} LR + Feature Scaling & 3306.2 & 13.8\% & 2.7\% \\ \hline
    \textbf{(2.3)} Gradient Descent (GD) & 3194.3 & 11.5\% & 2.2\% \\ \hline
    \textbf{(2.4)} GD + Feature Dropping (FD) & 3199.4 & 9.9\% & 2.0\% \\ \hline
    \textbf{(2.5)} GD + FD + Regularization (Reg) & 3228.2 & 9.7\% & 1.9\% \\ \hline
    \textbf{(2.6)} GD + Reg + Further FD & 2789.5 & 33.3\% & 8.3\% \\ \hline
    \textbf{(2.7)} Normal Equations & 3176.5 & 17.7\% & 3.2\% \\
    \hline
    \end{tabular}
    \caption{Comparison of the results obtained.\label{table:results}}
\end{center}
\end{table*}

Surprisingly, every technique implemented seemed to deliver progressively worse results. This may have been because of the amount of noise contained in the dataset, as is suggested by the drastic improvement in the results once the features were re-analyzed and re-selected.

The results obtained when Normal Equations were attempted indicate that it is able to reach the same results as the Gradient Descent method. However, due to its increased demand of memory and computing power, such a method is only feasible by either selecting a smaller subset of training examples, or by some other manner of training in batches that may allow the usage of all training examples (although this was not done here).

As for the learning rates applied in Gradient Descent implementations, it was found through experimentation that for implementations \hyperref[sec:gd]{2.3}, \hyperref[sec:gdfd]{2.4} and \hyperref[sec:gdr]{2.5}, the optimal learning rate was $\alpha = 0.1$, as larger values caused the training to diverge and smaller values were simply slower to converge. For implementation \hyperref[sec:gdrfd]{2.6}, the optimal learning rate was found to be $\alpha = 0.05$.

%%% Add section %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions and Future Work}
Linear regression and Gradient Descent are the most basic of Machine Learning tools. However, it is possible to increment those with more advanced techniques in order to achieve better results, as was shown by the results obtained by experimentation.

That it was possible to predict shares with a mean absolute error of 2789.5 leads to a few conclusions. First, predicting social network shares is absolutely possible, as the predictor did get better at it as more techniques were implemented.

Second, it may be that the current model is not complex enough to properly predict shares, and more advanced techniques such as Deep Learning are likely to deliver better results.

Finally, if you are going to publish something, it is probably a good idea to do it on a Friday.
%%% References %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
{\small
\bibliographystyle{unsrt}
\bibliography{refs}
}

\end{document}


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{scribe}
\usepackage{listings}

\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=cyan,
}

\LectureNumber{4}
\LectureDate{June 7, 2019}
\LectureTitle{Stein Variational Gradient Descent}

\lstset{style=mystyle}

\begin{document}
	\MakeScribeTop

%#############################################################
%#############################################################
%#############################################################
%#############################################################

Stein Variational Gradient Descent (SVGD) is a popular, non-parametric Bayesian Inference algorithm that’s been applied to Variational Inference, Reinforcement Learning, GANs, and much more. This week, we study the algorithm in its entirety, building off of last week’s work on KSD, and seeing how viewing KSD from a KL-Divergence-minimization lens induces a powerful, practical algorithm. We discuss the benefits of SVGD over other similar approximators, and look at a practical implementation of the algorithm.

\section{Table of Contents}

\begin{enumerate}
    \item Motivation
    \item Variational Inference
    \item Variational Inference Using Smooth Transforms
    \item Stein Operators and KL Divergence
    \item SVGD
    \item Benefits of SVGD
    \item Implementations
\end{enumerate}

\section{Variational Inference}

Last week, we saw an efficient test to test if two distributions, $p$ and $q$, were the same in \textit{Kernelized Stein Discrepancy}. This week, we're looking into subsequent step: given some ground truth distribution $p$, and some initial representation of $q$, how can we \textit{transform} $q$ into $p$? \\

\noindent This week, we'll be studying the Stein Variational Gradient Descent paper, which casts the above problem in the framework of \href{https://ermongroup.github.io/cs228-notes/inference/variational/}{variational} \href{https://arxiv.org/abs/1601.00670}{inference}: given some $p$, and some parameterized family of (simpler) distributions $\mathcal{Q}$, how do we find some $q^* \in \mathcal{Q}$ that approximates $p$? In this sense, variational inference casts the problem as an optimization of some "cost" function $\mathcal{J}(q)$, which is a function of the chosen distribution. \\

\noindent Alternative methods to approximate $p$, most commonly sampling-based variants of \href{https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo}{MCMC}, provide strong guarantees in infinite time, but not many in the finite sample regime. While still an exciting research area, there are some issues with scaling these methods up, and accelerating them with common hardware like GPUs, both of which are positive charateristics of variational inference.

\subsection{The KL-Divergence}

In order to frame the variational inference problem as an optimization problem, we need a method of comparing our two distributions. For this, we turn to the \href{https://timvieira.github.io/blog/post/2014/10/06/kl-divergence-as-an-objective-function/}{KL-Divergence}. This function, formally defined in the equation below, measures differences in information content between two distributions.

\begin{equation}
KL(q || p) = \mathbf{E}_q[\log q(x) - \log p(x)]
\end{equation}

\noindent When working with some unnormalized distribution $\bar p$, the optimization, which is searching for some distribution $ q^* \in \mathcal{Q}$, becomes:

\begin{equation}
KL(q || p) = \arg\min_{q \in \mathcal{Q}} \{ {\mathbf{E}_q[\log q(x)] - \mathbf{E}_q[\log \bar p(x)] + logZ} \}
\end{equation}

\noindent The normalization constant $Z$ drops out of the optimization here. It's important to note that the KL Divergence is a \textit{divergence}, not a distance: this means, in general, $KL(q || p) \neq KL(p || q)$. In fact, working out the gradients for each direction gives different benefits (and issues), but in general, we focus on the former as the latter requires evaluation on the normalization constant of $p$. Optimizing the equation above will - in theory; in practice, they will almost never find the globally optimal solution - find a distribution $q^*$ that minimizes the KL Divergence.

\section{Variational Inference Using Smooth Transforms}

Of course, after choosing to optimize the KL Divergence, we still have to choose the family of distributions. Choices often center around parameterized functions classes, or normal distributions, but for SVGD, we are going to focus on the distributions obtained by \textit{smooth transforms} from a tractable reference distribution. \\

\noindent $\mathcal{Q}$ takes the form of $z = \textbf{T}(x)$, where $\textbf{T}: \mathcal{X} \rightarrow \mathcal{X}$ is a smooth one-to-one transform, with $x \sim q_0(x)$. It can be shown (from optimal transport theory) that these are expressive distributions, that in theory can approximate almost any other distribution. \\

\noindent While we've constrained our distribution family, in practice, we \textit{also} need to worry about the class of transforms: balancing accuracy, tractability, and computational constraints, along with the injectiveness required by $\mathbf{T}$ provide a lot of requirements, many of which cannot be achieved in parallel. In the following two sections, we'll see the choices that SVGD makes to achieve these constraints.

\section{Stein Operators and KL Divergence}

The authors describe a small perturbation of the identity map: $\mathbf{T} = x + \epsilon \phi(x)$, which, given a small enough $\epsilon$, guarantees $\mathbf{T}$ to be a injective map.

\paragraph{Note:} Theorem 3.1 and Lemma 3.2 will be discussed qualitatively (with focus on their implications), as the paper's appendix does a good job of walking through the steps. \\

\noindent Theorem 3.1 relates the derivative of KLD to the negative trace of the Stein Operator, and because of Kernelized Stein Discrepancy, we can relate $\phi_{q, p}^*$ as the optimal perturbation direction (i.e steepest descent on KL) in the RKHS. Lemma 3.2 shows us that this generates a iterative procedure that allows us to start with \textit{any} $q_0$ that will, under the smooth transform assumptions above, converge to ground truth distribution $p$. In short, we apply iterative transforms, $\mathbf{T}^*_i = x + \epsilon_i \cdot \phi^*_{q_i, p}(x)$, to each distribution $q_i$ (starting from tractable $q_0$), until we converge ($\phi_{q_\infty, p}^* = 0$, which means the transform becomes the identity map).  \\

\noindent The paper includes an interpretation from a functional gradient perspective, but as that won't be important until next week, we skip directly to the practical algorithm proposed.

\section{Stein Variational Gradient Descent}

Now, we're ready to set the stage for SVGD. From the paper, we know that the optimal direction is:

\begin{equation}
    \phi^*_{q, p}(\cdot) = \mathbf{E}_{x \sim q} [k(x, \cdot)\nabla_x \log p(x) + \nabla_x k(x, \cdot)]
\end{equation}

\noindent To approximate the expectation, we average over an initial set of particles and use an empirical version of the transform (averaged over the particles) to move our particles to match our target distribution $p(x)$.  \\

\noindent The entire algorithm can be condensed into a single line, which repeats until convergence given an initial set of $n$ particles $\{ x_i \}$ and target distribution $p(x)$.

\begin{equation}
    \hat \phi^*(x_i) = \frac{1}{n} \sum^n_{j=1}[k(x_j, x_i)\nabla_{x_j} \log p(x_j) + \nabla_{x_j} k(x_j, x_i)]
\end{equation}

\noindent This is calculated individually for each particle, and then used with some step size to "move" the particle.

\subsection{Benefits of SVGD}

\begin{enumerate}
    \item With more particles (a larger  n), the approximation for each  $q_i$  gets better.
    \item With a single particle, this reduces into a MAP estimate.
    \item SVGD, unlike other methods, is expressive without requiring the inversion of a Jacobian matrix (a very expensive operation).
    \item The two terms in the above equation help introduce accuracy - the first term maximizes log probability of the samples - and diversity - the second term repulses similar particles - into the approximation.
    \item With automatic differention packages, many parts of SVGD can be parallelized and offloaded to efficient hardware implementations.
    \item The final form of the update is incredibly similar to standard gradient descent, making implementation and understanding extremely easy.
    \item (A pro, but not from this paper) SVGD has strong guarantees for particular settings, and can be seen from various different lenses (one of which we'll see next week).
\end{enumerate}

\subsection{Implementations}

Here are some implementations of SVGD in:

\begin{enumerate}
    \item \href{https://github.com/dilinwang820/Stein-Variational-Gradient-Descent}{Matlab and Python (original authors)}
    \item \href{https://github.com/activatedgeek/svgd}{Autograd (Sanyam Kapoor)}
    \item \href{https://github.com/calwoo/steins-method}{Pytorch (Calvin Woo)}
    \item \href{https://colab.research.google.com/drive/1ihKq85VQo3RBT-qaUl0D7gwLwlxB5VY7}{Jax (Bhairav Mehta)}
\end{enumerate}

%%%%%%%%%%% If you don't have citations then comment the lines below:
%
\bibliographystyle{abbrv}           % if you need a bibliography
\bibliography{mybib}                % assuming yours is named mybib.bib

%%%%%%%%%%% end of doc
\end{document}


\documentclass{article}
    % General document formatting
    \usepackage[margin=0.7in]{geometry}
    \usepackage[parfill]{parskip}
    \usepackage[utf8]{inputenc}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{tikz}
    \usepackage{fancyhdr}
    \usepackage{listings}
    \usepackage{multicol}
    \usepackage{polynom}

\pagestyle{fancy}
\fancyhf{}
\rhead{Edgar Jacob Rivera Rios - A01184125}

\begin{document}
\begin{titlepage}

    \newcommand{\HRule}{\rule{\linewidth}{0.5mm}} % Defines a new command for the horizontal lines, change thickness here

    \center % Center everything on the page

    %----------------------------------------------------------------------------------------
    %	HEADING SECTIONS
    %----------------------------------------------------------------------------------------

    \textsc{\LARGE Tecnológico de Monterrey}\\[1.5cm] % Name of your university/college
    \textsc{\Large Fundamentos de computación}\\[0.5cm] % Major heading such as course name
    %\textsc{\large Minor Heading}\\[0.5cm] % Minor heading such as course title

    %----------------------------------------------------------------------------------------
    %	TITLE SECTION
    %----------------------------------------------------------------------------------------

    \HRule \\[0.4cm]
    { \huge \bfseries Homework 7}\\[0.4cm] % Title of your document
    \HRule \\[1.5cm]

    %----------------------------------------------------------------------------------------
    %	AUTHOR SECTION
    %----------------------------------------------------------------------------------------

    \begin{minipage}{0.4\textwidth}
    \begin{flushleft} \large
    \emph{Student:}\\
    Jacob \textsc{Rivera} % Your name
    \end{flushleft}
    \end{minipage}
    ~
    \begin{minipage}{0.4\textwidth}
    \begin{flushright} \large
    \emph{Professor:} \\
    Dr. Hugo \textsc{Terashima} % Supervisor's Name
    \end{flushright}
    \end{minipage}\\[2cm]

    % If you don't want a supervisor, uncomment the two lines below and remove the section above
    %\Large \emph{Author:}\\
    %John \textsc{Smith}\\[3cm] % Your name

    %----------------------------------------------------------------------------------------
    %	DATE SECTION
    %----------------------------------------------------------------------------------------

    {\large \today}\\[2cm] % Date, change the \today to a set date if you want to be precise

    %----------------------------------------------------------------------------------------
    %	LOGO SECTION
    %----------------------------------------------------------------------------------------

    \includegraphics[width=0.4\textwidth,height=\textheight,keepaspectratio]{logo-tec-negro.png} % Include a department/university logo - this will require the graphicx package

    %----------------------------------------------------------------------------------------

    \vfill % Fill the rest of the page with whitespace

\end{titlepage}

\section{Problems}
Solve the following problems:
\begin{enumerate}
    \item Generate two 4 X 4 matrices and manually apply the Strassen’s method to multiply them. Verify that the result is correct by comparing with the one provided by the traditional method. Show the steps.
    \begin{multicols}{2}
        \begin{center}
            $A=$
            \begin{tabular}{|c c c c|}
                \hline
                1&2&3&4\\
                5&6&7&8\\
                9&10&11&12\\
                13&14&15&16\\
                \hline
            \end{tabular}\\
            $B=$
            \begin{tabular}{|c c c c|}
                \hline
                16&15&14&13\\
                12&11&10&9\\
                8&7&6&5\\
                4&3&2&1\\
                \hline
            \end{tabular}
        \end{center}
    \end{multicols}
    \begin{multicols}{2}
        \begin{center}
            $A=$
            \begin{tabular}{|c c|c c|}
                \hline
                1&2&3&4\\
                5&6&7&8\\
                \hline
                9&10&11&12\\
                13&14&15&16\\
                \hline
            \end{tabular}\\
            $B=$
            \begin{tabular}{|c c|c c|}
                \hline
                16&15&14&13\\
                12&11&10&9\\
                \hline
                8&7&6&5\\
                4&3&2&1\\
                \hline
            \end{tabular}
        \end{center}
    \end{multicols}
    \begin{multicols}{4}
        \begin{center}
            $A_1=$
            \begin{tabular}{|c c|}
                \hline
                1&2\\
                5&6\\
                \hline
            \end{tabular}\\
            $A_3=$
            \begin{tabular}{|c c|}
                \hline
                9&10\\
                13&14\\
                \hline
            \end{tabular}\\
            $A_2=$
            \begin{tabular}{|c c|}
                \hline
                3&4\\
                7&8\\
                \hline
            \end{tabular}\\
            $A_4=$
            \begin{tabular}{|c c|}
                \hline
                11&12\\
                15&16\\
                \hline
            \end{tabular}\\
            $B_1=$
            \begin{tabular}{|c c|}
                \hline
                16&15\\
                12&11\\
                \hline
            \end{tabular}\\
            $B_3=$
            \begin{tabular}{|c c|}
                \hline
                8&7\\
                4&3\\
                \hline
            \end{tabular}\\
            $B_2=$
            \begin{tabular}{|c c|}
                \hline
                14&13\\
                10&9\\
                \hline
            \end{tabular}\\
            $B_4=$
            \begin{tabular}{|c c|}
                \hline
                6&5\\
                2&1\\
                \hline
            \end{tabular}\\
        \end{center}
    \end{multicols}

    \begin{align*}
        C_1 &= A_1  B_1\\
        C_2 &= A_2  B_2\\
        C_3 &= A_3  B_3\\
        C_4 &= A_4  B_4
    \end{align*}

    \begin{multicols}{2}
        \begin{center}
            $A_1B_1$\\
            \begin{align*}
                X_1 &= (1 + 6) * (16 + 11) = 189\\
                X_2 &= 16 * (5 + 6) = 176\\
                X_3 &= 1 * (15-11) = 4 \\
                X_4 &= 6 * (12 - 16) = -24\\
                X_5 &= 11 * (1 + 2) = 33\\
                X_6 &= (5 - 1) * (16 + 15) = 124\\
                X_7 &= (2 - 6) * (12 + 11) = -92\\
            \end{align*}
            $C_1=$
            \begin{tabular}{|c c|}
                \hline
                189 + -24 - 33 + -92 &4 + 33\\
                176 + -24& 189 + 4 - 176 + 124\\
                \hline
            \end{tabular}\\
            $=$
            \begin{tabular}{|c c|}
                \hline
                40&37\\
                152&141\\
                \hline
            \end{tabular}\\
        \end{center}

        \begin{center}
            $A_2B_2$\\
            \begin{align*}
                X_1 &= (3 + 8) * (14 + 9) = 253\\
                X_2 &= 14 * (7 + 8) = 210\\
                X_3 &= 3 * (13-9) = 12 \\
                X_4 &= 8 * (10 - 14) = -32\\
                X_5 &= 9 * (3 + 4) = 63 \\
                X_6 &= (7 - 3) * (14 + 13) = 108\\
                X_7 &= (4 - 8) * (10 + 9) = -76\\
            \end{align*}
            $C_2=$
            \begin{tabular}{|c c|}
                \hline
                253 + -32 - 63 + -76 &12 + 63\\
                210 + -32& 253 + 12 - 210 + 108\\
                \hline
            \end{tabular}\\
            $=$
            \begin{tabular}{|c c|}
                \hline
                82&75\\
                178&163\\
                \hline
            \end{tabular}\\
        \end{center}
    \end{multicols}
    \pagebreak
    \begin{multicols}{2}
        \begin{center}
            $A_3B_3$\\
            \begin{align*}
                X_1 &= (3 + 8) * (14 + 9)  = 253\\
                X_2 &= 8 * (13 + 14) = 216\\
                X_3 &= 9 * (7 -3) = 36 \\
                X_4 &= 14 * (4 - 8) = -56\\
                X_5 &= 3 * (9 + 10) = 57\\
                X_6 &= (13 - 9) * (8 + 7) = 60\\
                X_7 &= (10 - 14) * (4 + 3) = -28\\
            \end{align*}
            $C_3=$
            \begin{tabular}{|c c|}
                \hline
                253 + -56 - 57 + -28 &36 + 57\\
                216 + -56& 253 + 36 - 216 + 60\\
                \hline
            \end{tabular}\\
            $=$
            \begin{tabular}{|c c|}
                \hline
                112&93\\
                160&133\\
                \hline
            \end{tabular}\\
        \end{center}

        \begin{center}
            $A_4B_4$\\
            \begin{align*}
                X_1 &= (1 + 6) * (16 + 11) = 189\\
                X_2 &= 6 * (15 + 16) = 186\\
                X_3 &= 11 * (5-1) = 44\\
                X_4 &= 16 * (2 - 6) = -64\\
                X_5 &= 1 * (11 + 12) = 23 \\
                X_6 &= (15 - 11) * (6 + 5) = 44\\
                X_7 &= (12 - 16) * (2 + 1) = -12\\
            \end{align*}
            $C_4=$
            \begin{tabular}{|c c|}
                \hline
                189 + -64 - 23 + -12 &44 + 23\\
                186 + -64& 189 + 44 - 186 + 44\\
                \hline
            \end{tabular}\\
            $=$
            \begin{tabular}{|c c|}
                \hline
                90&67\\
                122&91\\
                \hline
            \end{tabular}\\
        \end{center}
    \end{multicols}

    \item Generate a monic polynomial with $k= 4$ (that is $n= 15$) and solve it using the recursive algorithm presented in class. show the steps.
    \begin{equation*}
        x^{15}+2x^{14}+3x^{13}+4x^{12}+x^{11}+2x^{10}+3x^{9}+4x^{8}+x^{7}+2x^{6}+3x^{5}+4x^{4}+x^{3}+2x^{2}+3x+4
    \end{equation*}
    \begin{align*}
        k &= 4\\
        j &= 2^{k-1} = 8\\
        b &= a_{2^{k-1}-1} - 1 = a_7 - 1 = 0
    \end{align*}
    *\[\polylongdiv{x^{15}+2x^{14}+3x^{13}+4x^{12}+x^{11}+2x^{10}+3x^{9}+4x^{8}+x^{7}+2x^{6}+3x^{5}+4x^{4}+x^{3}+2x^{2}+3x+4}{x^8}\]

    \begin{equation*}
        p(x) = (x^8)(x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4) + (x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4)
    \end{equation*}

    \begin{equation*}
        x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4
    \end{equation*}
    \begin{align*}
        k &= 3\\
        j &= 2^{k-1} = 4\\
        b &= a_{2^{k-1}-1} - 1 = a_3 - 1 = 0
    \end{align*}
    *\[\polylongdiv{x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4}{x^4}\]

    \begin{equation*}
        p(x) = (x^8)((x^4)(x^3+ 2x^2+ 3x+ 4) + (x^3+ 2x^2+ 3x+ 4)) + (x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4)
    \end{equation*}

    \begin{equation*}
        x^3+ 2x^2+ 3x+ 4
    \end{equation*}
    \begin{align*}
        k &= 2\\
        j &= 2^{k-1} = 2\\
        b &= a_{2^{k-1}-1} - 1 = a_1 - 1 = 3
    \end{align*}
    *\[\polylongdiv{x^3+ 2x^2+ 3x+ 4}{x^2+3}\]

    \begin{equation*}
        p(x) = (x^8)((x^4)((x^2 +3)(x+ 2) + (-2)) + (x^3+ 2x^2+ 3x+ 4)) + (x^7+ 2x^6+ 3x^5+ 4x^4+x^3+ 2x^2+ 3x+ 4)
    \end{equation*}

    \begin{equation*}
        p(x) = (x^8)[(x^4)((x^2 +3)(x+ 2) + -2) + (x^2 +3)(x+ 2) + -2] + [(x^4)((x^2 +3)(x+ 2) + -2) + (x^2 +3)(x+ 2) + -2]
    \end{equation*}
    \begin{equation*}
        p(x) = (x^8)[(x^4)((x^2 +3)(x+ 2) + -2) + (x^2 +3)(x+ 2) + -2] + (x^4)((x^2 +3)(x+ 2) + -2) + (x^2 +3)(x+ 2) + -2
    \end{equation*}

\end{enumerate}
\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{PS9_Fillmore}
\author{benjaminpaynefillmore }
\date{April 2021}

\usepackage{natbib}
\usepackage{graphicx}

\begin{document}

\maketitle

\section{Introduction}
8.
# optimal lambda is 0.0222
# in sample rmse is 0.195
# out of sample rmse is 0.215

9.
# optimal lambda is 0.0000000001
# in sample rmse is 0.195
# out of sample rmse is 0.210

10.
I'd say yes.
Our in sample and out of sample rmse's are fairly close, with in sample rmse being slightly smaller. I'd say we're in a standard case and holding decently close to an acceptable level in the bias/variance tradeoff.
\end{document}


%&latex
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{graphicx,psfrag,epsf}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{url} % not crucial - just used below for the URL

%\pdfminorversion=4
% NOTE: To produce blinded version, replace "0" with "1" below.
\newcommand{\blind}{0}

% DON'T change margins - should be 1 inch all around.
\addtolength{\oddsidemargin}{-.5in}%
\addtolength{\evensidemargin}{-.5in}%
\addtolength{\textwidth}{1in}%
\addtolength{\textheight}{1.3in}%
\addtolength{\topmargin}{-.8in}%

\font\tt=rm-lmtl10
\font\itt=rm-lmtlo10
\font\btt=rm-lmtk10
\font\bitt=rm-lmtko10

\begin{document}

%\bibliographystyle{natbib}

\def\spacingset#1{\renewcommand{\baselinestretch}%
{#1}\small\normalsize} \spacingset{1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\if0\blind
{
  \title{\bf The Hedging Hyperprior}
  \author{Stephen D. Simon 1\thanks{
    The authors gratefully acknowledge \textit{please remember to list all relevant funding sources in the unblinded version}}\hspace{.2cm}\\
    Department of Biomedical and Health Informatics,\\University of Missouri-Kansas City\\
    Yu Jiang\\
    Division of Epidemiology, Biostatistics, and Environmental Health,\\University of Memphis\\
    and \\
    Byron J. Gajewski\\
    Department of Biostatistics,\\University of Kansas Medical Center}
  \maketitle
} \fi

\if1\blind
{
  \bigskip
  \bigskip
  \bigskip
  \begin{center}
    {\LARGE\bf Title}
\end{center}
  \medskip
} \fi

\bigskip
\begin{abstract}

Here's a scratch file where I want to park some complex formulas that are not needed (yet) for the regular paper.

In many settings,...

\end{abstract}

\noindent%
{\it Keywords:}  Beta-binomial model, Informative prior, Modified power prior
\vfill

\newpage
\spacingset{1.45} % DON'T change the spacing!
\section{Introduction}
\label{sec:intro}

Stephen Senn is ...

\section{A beta-binomial model with an informative prior distribution}
\label{sec:bb}
Consider a Bayesian ...

\section{The hedging hyperprior}
\label{sec:hh}

The hedging hyperprior ...

\section{Alternatives to the hedging hyperprior}
\label{sec:alt}

There are several ...

Consider the historic prior and the data as two sets of data, D0 and D. If, for example, your informative prior were  beta(4, 16), then you would create a pseudo data set with (4-1) successes and (16-1) failures. Apply this pseudo data to a flat prior ($\pi_0(\theta)$), get a posterior distribution the resembles your informative prior.

$\pi(\theta | D_0) \propto L(\theta|D_0)\pi_0(\theta)$

Then use $\pi(\theta | D_0)$ use that as your prior distribution for the new data.

$\pi(\theta | D,D_0) \propto L(\theta|D)\pi(\theta | D_0)$

What \cite{ibrahim03} suggested is that you could downweight the historical prior by raising the historical likelihood to the power.

$\pi(\theta | D_0, \tau) \propto L(\theta|D_0)^{\tau}\pi_0(\theta)$

With $\tau$=1, you get the full strength of the historical prior and with $\tau$=0 you wipe out the historic data and are left with just a flat prior. This formulation does not satisfy the likelihood principle, but you can include a fudge factor ($\int L(\theta|D_0)^{\tau}\pi_0(\theta)d\theta$) to make things work nicely.

It is fairly easy to show that hedging hyperprior in the beta-binomial model is equivalent to the modified power prior. Notice that

$L(\theta|D_0)^{\tau}=\left(\frac{(\alpha+\beta-2)!}{(\alpha-1)!(\beta-1)!}\theta^{\alpha-1}(1-\theta)^{\beta-1}\right)^\tau$

is proportional to

$\theta^{(\alpha-1)\tau}(1-\theta)^{(\beta-1)\tau}$

If you multiply this by a uniform prior for theta and apply the fudge factor, you get a $Beta(1+(\alpha-1)\tau,1+(\beta-1)\tau)$ distribution for $L(\theta|D_0)$.

The advantage to formulating the problem as the hedging hyperprior is that it makes it easy to see how to use standard Bayesian software like BUGS, JAGS, or STAN. Also, the hedging hyperprior is easier to visualize, allowing you to readily discern its behavior.

You can also use a hierarchical approach. Pretend that you have a multi-center trial with two centers. Consider the Beta(a, b) informative prior as (a+b)-2 pseudo-observations with a-1 successes and b-1 failures from one center of a multi-center trail and the n actual observations with x successes and n-x failures as a second center. Then you can model these two centers using a hierarchical model. The second center will borrow strength from the first center, but the amount of information that it borrows is limited by the center-to-center variation. In other words, you use the prior only to the extent to which the prior and the data agree with one another. When they disagree strongly, the center-to-center variation is large and the posterior distribution for the second center is mostly just based on the likelihood.

$\pi_0 \sim Beta(\alpha_0,\beta_0)$

$\pi \sim Beta(\alpha_0,\beta_0)$

$\alpha_0 \sim Pareto(1)$

$\beta_0 \sim Pareto(1)$

$D_0 = \alpha-1~successes,~\beta-1~failures.$

$\alpha-1 \sim Binom(\alpha+\beta-2, \pi0)$

$x \sim Binom(n, \pi)$

$\pi(\pi0, \pi, \alpha0, \beta0)$

$L(D_0|\alpha,\beta)= \frac{(\alpha+\beta-2)!}{(\alpha-1)!(\beta-1)!}\pi_0^{\alpha-1}(1-\pi_0)^{\beta-1}\frac{\Gamma(\alpha_0)\Gamma(\beta_0)}{\Gamma(\alpha_0+\beta_0)}\pi_0^{\alpha_0-1}(1-\pi_0)^{\beta_0-1}\frac{1}{\alpha_0^2}\frac{1}{\beta_0^2}$

beta-prime distribution: $f(x) = \frac{x^{\alpha-1} (1+x)^{-\alpha -\beta}}{B(\alpha,\beta)}$

beta distribution: $f(x)=\frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$

logistic distribution: $\frac{e^{-\frac{x-\mu}{s}}} {s\left(1+e^{-\frac{x-\mu}{s}}\right)^2}$

$g(x)=log(x);~g^{-1}(y)=exp(y)$

$\frac{e^{y(\alpha-1)}}{(1+e^y)^{\alpha+\beta}B(\alpha,\beta)}e^y$

$\frac{e^{y\alpha}}{(1+e^y)^{\alpha+\beta}B(\alpha,\beta)}$

\section{Appendix}
\label{sec:app}

\section{Conclusion}
\label{sec:conc}

The bibliography ...

\section{Bibliography}

\bibliographystyle{Chicago}

\bibliography{hedging}
\end{document}


\chapter{Introduction}
Neural networks have led to a revolution in machine learning, providing solutions to tackle more challenging and complex problems. However neural networks are prone to overfitting, which can negatively affect their generalization capabilities. Neural networks also tend to be overconfident about their predictions, even when they do provide a confidence interval. All of this can be problematic for applications such as medical diagnostics, self driving cars or finance, where wrong predictions can have dramatic outcomes.
\\
\\
To moderate this risk many approaches have been proposed, most noticeably by the use of stochastic neural networks to estimate the uncertainty of model prediction. The Bayesian paradigm provides a solid framework to analyse and develop these learning algorithms. Such Bayesian neural networks have become more attractive in the recent years with the increase in computational power of personal computers and the growing body of research on efficient sampling algorithms. One branch of such algorithms is the Markov chain Monte Carlo sampling methods, which provide clever ways of sampling to reduce computational cost. Also a growing number of programming languages and software packages, such as Stan, TensorFlow Probability and PyMC3, support Bayesian statistical analysis.
\\
\\
Another factor that makes Bayesian neural networks interesting is that they allow us to interpret predictions in terms of probability distributions, which traditional approaches do not. Further it provides a principled mechanism for practitioners to build on previous knowledge by incorporating relevant prior information into their analysis. This can be crucial for tasks with small sample sizes, since the prior information regularize results, reducing the chances of poor generalization capabilities, which is common in cases with a relatively small sample size.
\section{Problem Statement} \label{sec:problem}
This thesis aims to answer the following problem statement:
\begin{center}
    \textit{How can Markov chain Monte Carlo based Bayesian neural networks be used to perform regression and classification tasks and how is this different from using non-Bayesian feedforward neural networks?}
\end{center}
\noindent
This is done by answering the following research questions:
\begin{itemize}
    \item What is a supervised machine learning algorithm and how does it work?
    \item What is over- and underfitting and how do we prevent this in a supervised machine learning algorithm?
    \item How is gradient based optimization used in training a supervised machine learning algorithm?
    \item What is a feedforward neural network and how is it trained and regularized?
    \item What is Bayesian inference and how is it different from frequentist inference?
    \item What is Monte Carlo methods and how do they work?
    \item What is a Bayesian neural network and how does it work?
    \item What is Markov chain Monte Carlo and how can we use it to sample?
    \item What is Hamiltonian Monte Carlo and how can we use it to sample?
    \item What is No-U-Turn Hamiltonian Monte Carlo and how can we use it to sample?
    \item What role do priors play in Bayesian neural networks and how can we choose them properly?
    \item How do non-Bayesian and Bayesian neural networks perform when used for predicting house prices in Boston?
    \item How do non-Bayesian and Bayesian neural networks perform when used for predicting default probabilities on credit card loans?
\end{itemize}

\section{Method}
We approach the questions posed in section \ref{sec:problem} using the positivist paradigm. Positivism is a philosophical theory that states that genuine knowledge exists and is structured in a specific way regardless of ones perspective of it. According to this theory, information is derived from sensory experienced, that is interpreted through reason and logic, which is reflected by our mathematical approach of examining Bayesian- and non-Bayesian neural networks. We evaluate these neural networks on data, which is in accordance with a positivist view on verified data as empirical evidence.
\\
\\
This thesis is mainly a theoretical examination of Markov chain Monte Carlo based Bayesian neural networks, with a focus on efficient sampling methods for these, and how they differ from non-Bayesian neural networks. We examine this by first clarifying basic machine learning theory that is relevant for neural networks. This will cover how to train such supervised machine learning algorithms and how to regularize them to prevent over- and underfitting. Afterwards we cover numerous optimization algorithms used for training, which provide the foundations for examining Adaptive Moment Estimation (ADAM), that we use in our practical evaluation in a later section. This provides the fundamentals for examining feedforward neural networks. We do this by covering its general structures and various options for designing it. We also cover how it trains using backpropagation combined with the previous covered optimization algorithms, and we cover how to do this with regularization techniques like early stopping.
\\
\\
With neural networks examined we have defined the model to which we take a Bayesian approach in Bayesian neural networks. Such networks are covered by first examining the difference in Bayesian and frequentist (read non-Bayesian) learning. Then we examine Monte Carlo methods, which are often used in Bayesian inference models.
\\
\\
We then construct a simple Bayesian neural network, to illustrate how such a network works, and to show the motivation for introducing more sophisticated sampling techniques when sampling from such networks. This naturally leads to an examination of Markov chain Monte Carlo methods, which provides the basics for such sampling methods. We use this knowledge to examine the Hamiltonian Monte Carlo method and the extensions of this in the pursuit of the most efficient sampling method for Bayesian neural networks. This pursuit ends with the No-U-Turn Hamiltonian Monte Carlo, which we use in our practical evaluation of Bayesian neural networks in a later section.
\\
\\
Afterwards we examine the effect of prior choice and only briefly how to choose priors, as such choices often depend on the specific task and data. Instead we examine various general approaches and point to literature that supports these approaches. Finally we evaluate the Bayesian and non-Bayesian neural networks on real data. First a dataset for which we aim to predict house prices in Boston using regression and next a dataset on credit card defaults, where we try to predict default probabilities for binary classification.

\section{Research Delimitation}
To have a focused approach for answering our problem statement in section \ref{sec:problem}, we choose not to elaborate on certain subjects, as we deem them out of scope for answering the problem statement appropriately. As we focus on Markov chain Monte Carlo based Bayesian neural networks, we do not examine other methods for using Bayesian inference in neural networks such as methods based on variational inference. We also refrain from examining the choice of priors too thoroughly, and we only cover the consequences of the choice and the most general methods, since the method of choosing a prior often depends on the data, the task and the researches prior knowledge of such.
\\
\\
Non-Bayesian neural networks are examined to the extent needed in order to provide fundamental knowledge to the model for which we use Bayesian inference and to illustrate the most important differences. These differences are important to illustrate pros and cons of using the Bayesian approach to these networks. This means that we examine the most basic neural networks called feedforward neural networks and do not go into more sophisticated architecture like recurrent or convolutional neural networks. We also limit our examination to the most popular activation functions and training procedures for these networks.
\\
\\


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS240-2016S: Programming in C
% Copyright 2016 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS240-2016S
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Write a program \texttt{reverse.c} that takes an integer number as a command-line argument and prints its reverse.

Your program is expected to function as shown below.

\begin{terminal}
$ gcc reverse.c -o reverse
$ ./reverse
error: missing command line argument
$ ./reverse 12345
Reverse: 54321
$ ./reverse -54321
Reverse: -12345
$ ./reverse hello
error: not a number
$ ./reverse 1234 5678 hello
Reverse: 4321
\end{terminal}


\documentclass[11pt, letterpaper]{article}
\setlength{\parindent}{0in}
\setlength{\textheight}{8.7in}
\setlength{\textwidth}{6.8in}
\setlength{\oddsidemargin}{-0.3in}
\setlength{\evensidemargin}{0.0in}
\addtolength{\topmargin}{-1in}
\setlength{\parskip}{0.1in}

\usepackage{amsmath, amsfonts, amssymb, color}
\usepackage{bm}
\usepackage{booktabs}
\usepackage{enumerate}
\usepackage{graphicx}
\usepackage{pdfpages}
\newcommand*{\justifyheading}{\raggedleft}

\renewcommand{\baselinestretch}{1.0}

\newcommand{\bx}{{\bm x}}
\newcommand{\bX}{{\bm X}}
\newcommand{\by}{{\bm y}}
\newcommand{\bY}{{\bm Y}}
\newcommand{\bW}{{\bm W}}
\newcommand{\bG}{{\bm G}}
\newcommand{\bR}{{\bm R}}
\newcommand{\bZ}{{\bm Z}}
\newcommand{\bV}{{\bm V}}
\newcommand{\bL}{{\bm L}}
\newcommand{\bz}{{\bm z}}
\newcommand{\be}{{\bm e}}
\newcommand{\bgamma}{{\bm \gamma}}
\newcommand{\bbeta}{{\bm \beta}}
\newcommand{\balpha}{{\bm \alpha}}
\newcommand{\bSigma}{{\bm \Sigma}}
\newcommand{\bmu}{{\bm \mu}}
\newcommand{\btheta}{{\bm \theta}}
\newcommand{\bepsilon}{{\bm \epsilon}}
\newcommand{\bone}{{\bm 1}}
\newcommand{\bzero}{{\bm 0}}
\newcommand{\bC}{{\bm C}}
\newcommand{\bI}{{\bm I}}
\newcommand{\bA}{{\bm A}}
\newcommand{\bB}{{\bm B}}
\newcommand{\bQ}{{\bm Q}}
\newcommand{\bS}{{\bm S}}
\newcommand{\bD}{{\bm D}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\cU}{\mathcal{U}}
\newcommand{\cI}{\mathcal{I}}
\newcommand{\cL}{\mathcal{L}}

\newcommand{\beas}{\begin{eqnarray*}}
\newcommand{\eeas}{\end{eqnarray*}}

\newenvironment{equationarrayright}{
                          \begin{eqnarray*}
                          \begin{array}{rcll}
                         }{
                          \end{array}
                          \end{eqnarray*}
                         }
\newcommand{\bear}{\begin{equationarrayright}}
\newcommand{\eear}{\end{equationarrayright}}

\renewcommand\arraystretch{1.3}

\DeclareMathOperator*{\argmin}{arg\,min}

\title{STAT/BIOST 571: Homework 3}
\author{Philip Pham}
\date{\today}

\begin{document}

\maketitle

\section*{Problem 1: Quasilikelihood and semiparametric methods for the general linear model (14 points)}
{\em This question examines the effect of different correlation structures, designs, and sample sizes in fitting a general linear model in a quasi-likelihood and semiparametric framework.  It is also an exercise in writing code systematically; please take care to break the required programming into small tasks, and write individual functions to do each of these tasks.
Please write all code ``by hand'', using matrix algebra and simple moment-based estimators.
You may find the \texttt{mvtnorm} package helpful.

For the marginal model
\[
E(Y_{ij}|x_{ij}) = \beta_0 + \beta_1 x_{ij},
\]
consider estimation by weighted least squares, where the cluster weights are the inverse of
the estimated cluster covariance matrix.  Calculate quasi-likelihood standard errors as if your assumed form
of the covariance matrix
is known to be correct (even if, in actuality, you have assumed the wrong form of the covariance) and semi-parametric  standard errors using the sandwich estimator that accounts for clustering.
All of the notation follows the lecture notes.

Throughout, the following are true in the data-generating mechanism
\begin{itemize}
\item $\beta_0=0$, $\beta_1 = 0.5$
\item $\bY_i|\bX_i\sim N(\bX_i \bbeta, \sigma^2 \bR_i)$ with $\sigma^2=1$.
\end{itemize}
The factors that will vary are
\begin{itemize}
\item The number of clusters is 15, 30, or 60
\item The design: \begin{itemize}
\item Design I has $m_i = 4$, for all clusters. In each cluster, we see $\{x_{i1}, x_{i2},x_{i3},x_{i4}\}=\{7,10,13,16\}$
\item Design II has $m_i=3$ for all clusters.  We see equal numbers of clusters with $\{x_{i1}, x_{i2},x_{i3}\}=\{7,10,13\}, \{7,10,16\}, \{7, 13, 16\}, \textrm{ or }\{10, 13, 16\}$
\end{itemize}
\item The true covariance and the assumed covariance matrices are of the form $\sigma^2 \bR_i$:
\begin{itemize}
\item For the true covariance, consider exchangeable and exponential correlation structures, with $\rho=0.5$ or $\rho=0.9$ (distances between observations in the exponential model based on $x_{ij}$).
\item For the assumed covariance, consider these and additionally the uncorrelated homoscedastic covariance.  Any covariance parameters should be estimated  using moment-based methods.
\end{itemize}
\end{itemize}}

\begin{table}
  \tiny
\centering
\begin{tabular}{ccr|ccc|ccc|ccc}
&&&\multicolumn{3}{|c}{$\mathbf{SD(\hat\beta_1)}$}&\multicolumn{3}{|c}{$\mathbf{E(\widehat{SE}_{1,QL})}$}&\multicolumn{3}{|c}{$\mathbf{E(\widehat{SE}_{1,sand})}$}\\
&&&\multicolumn{3}{c}{Assumed Corr}&\multicolumn{3}{|c}{Assumed Corr}&\multicolumn{3}{|c}{Assumed Corr}\\
$n$ & Design & True Corr & Uncor & Exch & Expon & Uncor & Exch & Expon & Uncor & Exch & Expon \\ \hline
15 &  I & Exchangeable $\rho=0.5$ & \input{simulation_results/15-I-EXCHANGEABLE-0_5.tex} \\
15 &  I & Exchangeable $\rho=0.9$ & \input{simulation_results/15-I-EXCHANGEABLE-0_9.tex} \\
15 &  I &         Exponential $\rho=0.5$ & \input{simulation_results/15-I-EXPONENTIAL-0_5.tex} \\
15 &  I &         Exponential $\rho=0.9$ & \input{simulation_results/15-I-EXPONENTIAL-0_9.tex} \\ [1ex]
15 & II & Exchangeable $\rho=0.5$ & \input{simulation_results/15-II-EXCHANGEABLE-0_5.tex} \\
15 & II & Exchangeable $\rho=0.9$ & \input{simulation_results/15-II-EXCHANGEABLE-0_9.tex} \\
15 & II &         Exponential $\rho=0.5$ & \input{simulation_results/15-II-EXPONENTIAL-0_5.tex} \\
15 & II &         Exponential $\rho=0.9$ & \input{simulation_results/15-II-EXPONENTIAL-0_9.tex} \\ [2ex]
30 &  I & Exchangeable $\rho=0.5$ & \input{simulation_results/30-I-EXCHANGEABLE-0_5.tex} \\
30 &  I & Exchangeable $\rho=0.9$ & \input{simulation_results/30-I-EXCHANGEABLE-0_9.tex} \\
30 &  I &         Exponential $\rho=0.5$ & \input{simulation_results/30-I-EXPONENTIAL-0_5.tex} \\
30 &  I &         Exponential $\rho=0.9$ & \input{simulation_results/30-I-EXPONENTIAL-0_9.tex} \\ [1ex]
30 & II & Exchangeable $\rho=0.5$ & \input{simulation_results/30-II-EXCHANGEABLE-0_5.tex} \\
30 & II & Exchangeable $\rho=0.9$ & \input{simulation_results/30-II-EXCHANGEABLE-0_9.tex} \\
30 & II &         Exponential $\rho=0.5$ & \input{simulation_results/30-II-EXPONENTIAL-0_5.tex} \\
30 & II &         Exponential $\rho=0.9$ & \input{simulation_results/30-II-EXPONENTIAL-0_9.tex} \\ [2ex]
60 &  I & Exchangeable $\rho=0.5$ & \input{simulation_results/60-I-EXCHANGEABLE-0_5.tex} \\
60 &  I & Exchangeable $\rho=0.9$ & \input{simulation_results/60-I-EXCHANGEABLE-0_9.tex} \\
60 &  I &         Exponential $\rho=0.5$ & \input{simulation_results/60-I-EXPONENTIAL-0_5.tex} \\
60 &  I &        Exponential $\rho=0.9$ & \input{simulation_results/60-I-EXPONENTIAL-0_9.tex} \\ [1ex]
60 & II & Exchangeable $\rho=0.5$ & \input{simulation_results/60-II-EXCHANGEABLE-0_5.tex} \\
60 & II & Exchangeable $\rho=0.9$ & \input{simulation_results/60-II-EXCHANGEABLE-0_9.tex} \\
60 & II &         Exponential $\rho=0.5$ & \input{simulation_results/60-II-EXPONENTIAL-0_5.tex} \\
60 & II &         Exponential $\rho=0.9$ & \input{simulation_results/60-II-EXPONENTIAL-0_9.tex}
\end{tabular}
\caption{Table of standard deviations of the estimated slopes and
  average of model-based and sandwich-based standard error estimates}
\label{tab:p1_results}
\end{table}

\begin{description}
\item[Solution:] See the results in Table \ref{tab:p1_results}. Correct standard
  errors are found when the assumed correlation structure agrees with the true
  correlation structure when using GLS (columns 2 and 3). Ignoring the
  correlation within the clusters overestimates the standard error (first
  column). When incorrectly assuming exponential correlation, the standard error
  is overestimated. When incorrectly assuming exchangeable correlation, the
  standard error is underestimated.

  Quasi-likelihood doesn't help very much when the correlation structure is
  misspecified. The standard error estimates are almost identical to just using
  GLS. The standard errors are slightly closer to those of the sandwich
  estimator, so there is some insignificant gain.

  Comparing the sandwich estimation with GLS when the assumed correlation is
  correct, one sees that the standard error estimates are underestimated for
  smaller $n$. For $n = 60$, the estimates are very good. When the correlation
  structure is misspecified, sandwich estimation comes closest to the actual the
  standard error, especially when $n$ is large (verified numerically, results
  not shown). It follows that misspecifying the correlation structure as
  exchangeable or exponential doesn't affect the standard error of
  $\hat{\beta}_1$ very much.

  This is also the case when assuming no correlation with design I. However, when
  using design II, where not all the clusters have identical covariates,
  assuming no correlation increases the standard error of our estimator
  significantly. This effect is even more pronounced when there is more
  correlation ($\rho = 0.9$ versus $\rho = 0.5$).

  Details of the calculation are described below, and code is in the appendix.

  Let $X_i$ be the covariates for each cluster with a column of $1$s
  prepended. Let $y_i$ be the cluster responses.

  We first estimate $\beta$ with the ordinary least squares (OLS) estimator,
  $\hat{\beta}_{\mathrm{OLS}} = \left(X^\intercal X\right)^{-1}X^\intercal y$,
  where $X$ is the concatenation of the cluster covariates and $y$ is the
  concatenation of the cluster responses.

  We can get the covariances for $\hat{\beta}$ with
  \begin{align*}
    \operatorname{cov}\left(\hat{\beta}\right)
    &= \left(\sum_{i=1}^n X_i^\intercal W X_i\right)^{-1}
    \left(\sum_{i=1}^n X_i^\intercal W \hat{\Sigma} W X_i\right)
    \left(\sum_{i=1}^n X_i^\intercal W X_i\right)^{-1}.
  \end{align*}

  For the different estimation methods and assumed covariances, we vary the form
  of $\hat{\Sigma}$ and $W$, both of which are assumed to be the same for all
  clusters.

  To obtain $W$, we first estimate $\sigma^2$ with
  \begin{equation}
    \hat{\sigma}^2 = \frac{1}{\sum_{i=1}^nm_i - 2}\sum_{i=1}^n
    \left(y_{i} - X_{i}\hat{\beta}_{\mathrm{OLS}}\right)^\intercal
    \left(y_{i} - X_{i}\hat{\beta}_{\mathrm{OLS}}\right).
  \end{equation}

  Let the standardized OLS residuals for each cluster be
  $\tilde{\epsilon}_i = \frac{y_i -
    X_i\hat{\beta}_{\mathrm{OLS}}}{\hat{\sigma}}.$

  If we assume that the correlation structure is exchangeable, we estimate
  \begin{equation}
    \hat{\rho}_{\mathrm{exch}} =
    \frac{1}{\sum_{i=1}^n\sum_{j=1}^{m_i - 1} j}\sum_{i=1}^n\sum_{j=1}^{m_i - 1}\sum_{k=j + 1}^{m_i}
    \left(\tilde{\epsilon}_i\tilde{\epsilon}_i^\intercal\right)_{jk},
  \end{equation}
  so $W^{-1}_{ii} = 1$ for all $i$ and
  $W_{ij}^{-1} = \hat{\rho}_{\mathrm{exch}}$ for all $i \neq j$.

  If we assume that the correlation structure is exponential, we estimate
  \begin{equation} \hat{\rho}_{\mathrm{expon}} =
    \frac{1}{\sum_{i=1}^n\sum_{j=1}^{m_i - 1}
      j}\sum_{i=1}^n\sum_{j=1}^{m_i - 1}
    \left(\tilde{\epsilon}_i\tilde{\epsilon}_i^\intercal\right)_{j,j+1},
  \end{equation}
  so
  $W_{ij}^{-1} = \hat{\rho}_{\mathrm{expon}}^{\left\lvert j - i \right\rvert}$
  for any $i$ and $j$.

  If we don't assume any correlation structure, $W = I$.

  Now that we know $W$, we can estimate $\beta$ with
  \begin{equation}
    \hat{\beta} = \frac{1}{\sum_{i=1}^nm_i}\sum_{i=1}^nm_i\left(X_i^\intercal W X_i\right)^{-1} X_i^\intercal W y_i.
  \end{equation}

  By default, for the general linear model, we would have that
  $\hat{\Sigma} = W^{-1}$, in which case, we have that
  \begin{equation}
    \operatorname{cov}\left(\hat{\beta}\right)
    = \left(\sum_{i=1}^n X_i^\intercal W X_i\right)^{-1}.
  \end{equation}

  In the quasi-likelihood model, we have an additional dispersion factor
  $\alpha$, so $\hat{\Sigma} = \hat{\alpha}W^{-1}$, where
  \begin{equation}
    \hat{\alpha} = \frac{1}{\sum_{i=1}^n m_i - 2}
    \left(y - X\hat{\beta}\right)^\intercal\left(y - X\hat{\beta}\right),
  \end{equation}
  which results in the covariance
  \begin{equation}
    \operatorname{cov}\left(\hat{\beta}\right)
    = \hat{\alpha} \left(\sum_{i=1}^n X_i^\intercal W X_i\right)^{-1}.
  \end{equation}

  For the sandwich estimator, we use the empirical estimate
  \begin{equation}
    \hat{\Sigma} = \frac{1}{n}\sum_{i=1}^n
    \left(y_i - X_i\hat{\beta}\right)^\intercal \left(y_i - X_i\hat{\beta}\right).
  \end{equation}

\end{description}

\section*{Problem 2: Efficiency of OLS for linear models with correlated data (6 points)}
{\em Review the example on slides 2.34 -- 2.35, which can also be found on pages 60 -- 62 of the Diggle et al. textbook.  We will generalize this example by considering the mean model
\[
E(Y_{ij}) = \beta_0 + \beta_1 x_j
\]
for arbitrary $\bx = (x_1,x_2,\ldots,x_5)$ that is the same for all subjects, but which may or may not
be equal to ${\bm t}=(-2,-1,0,1,2)$ (as is the case in the original version of the example).
Note that the correlation structure is still determined based on $t$, as in the original example, but now the mean model contains $x$ rather than $t$.}
\begin{enumerate}[(a)]
{\em \item Derive a general expression for the relative efficiency of OLS compared to the optimal GLS in estimating $\beta_0$ and $\beta_1$ in this problem.  Your formula should be valid for a homoscedastic exponential covariance matrix with arbitrary
$\rho$ and for arbitrary $\bx$.  That is, derive a general version of the expressions
on the bottom of 2.44 that is valid for any choice of covariates.  Note that it is acceptable for your solution to be written using matrix notation and matrix algebra.}

\begin{description}
\item[Solution:] Let there be $n$ subjects. Each subject $i$ has $m_i$
  observations
  $x_i = \begin{pmatrix} x_{i1} &
    x_{i2} & \ldots & x_{im_i} \end{pmatrix}^\intercal$. Let $X_i$ be the
  $m_i \times 2$ covariate matrix, where $X_{ij1} = 1$ and $X_{ij2} =
  x_{ij}$. Let $X$ be the $\sum_{i=1}^n m_i \times 2$ matrix obtained by
  stacking $X_1,X_2,\ldots, X_n$.

  The true model is $Y_{ij} = \beta_0 + \beta_1x_{ij} + \epsilon_{ij}$. Let the
  obvervations between subjects be independent with each other. Let $\Sigma_i$
  denote the covariance structure for within-subject observations, that is
  $\Sigma_{ijk} = \operatorname{cov}\left(\epsilon_{ij}, \epsilon_{ik}\right)$.
  Let $\Sigma$ be the $\sum_{i=1}^n m_i \times \sum_{i=1}^n m_i$ block diagonal
  matrix
  \begin{equation}
    \Sigma =
    \begin{pmatrix}
      \Sigma_1 & & & \\
      & \Sigma_2 & & \\
      & & \ddots & \\
      & & & \Sigma_n
    \end{pmatrix}.
  \end{equation}
  Let $Y_i$ be the vector of observations for subject $i$. Let $Y$ be obtained
  by concatenating the $Y_1,Y_2,\ldots,Y_n$. If
  $\beta = \begin{pmatrix} \beta_0 & \beta_1 \end{pmatrix}^\intercal$, we can
  write $Y = X\beta + \epsilon$, where
  $\epsilon \sim \mathcal{N}\left(\mathbf{0}, \Sigma\right)$.

  The OLS estimate for is
  \begin{equation}
    \hat{\beta}_{\mathrm{OLS}} = \left(X^{\intercal}X\right)^{-1}X^\intercal Y
    = \beta + \left(X^{\intercal}X\right)^{-1}X^\intercal \epsilon,
  \end{equation}
  which has covariance
  \begin{align}
    \operatorname{cov}\left(\hat{\beta}_{\mathrm{OLS}}\right)
    &= \left(X^{\intercal}X\right)^{-1}X^\intercal
    \operatorname{cov}\left(\epsilon\right)
      X\left(X^{\intercal}X\right)^{-1} \nonumber\\
    &= \left(X^{\intercal}X\right)^{-1}X^\intercal
      \Sigma
      X\left(X^{\intercal}X\right)^{-1}.
      \label{eqn:p2_beta_hat_ols}
  \end{align}

  The optimal GLS estimate, which can be derived by maximizing likelihood is
  \begin{equation}
    \hat{\beta}_{\mathrm{GLS}} = \left(X^{\intercal}\Sigma^{-1}X\right)^{-1}X^\intercal\Sigma^{-1} Y
    = \beta + \left(X^{\intercal}\Sigma^{-1}X\right)^{-1}X^\intercal\Sigma^{-1} \epsilon,
  \end{equation}
  which has covariance
  \begin{align}
    \operatorname{cov}\left(\hat{\beta}_{\mathrm{GLS}}\right)
    &= \left(X^{\intercal}\Sigma^{-1}X\right)^{-1}X^\intercal
      \Sigma^{-1}
      \operatorname{cov}
      \left(\epsilon\right)
      \Sigma^{-1}
      X\left(X^{\intercal}\Sigma^{-1}X\right)^{-1} \nonumber\\
    &= \left(X^{\intercal}\Sigma^{-1}X\right)^{-1}.
      \label{eqn:p2_beta_hat_gls}
  \end{align}

  Much of these matrix multiplications can be written as summations:
  \begin{align*}
    X^\intercal X
    &= \sum_{i=1}^n X_i^\intercal X_i \\
    X^\intercal \Sigma X
    &= \sum_{i=1}^n X_i^\intercal \Sigma_i X_i \\
    X^\intercal \Sigma^{-1} X
    &= \sum_{i=1}^n X_i^\intercal \Sigma_i^{-1} X_i.,
  \end{align*}
  which is probably computationally faster if $n$ is very large.

  In any case, using Equations \ref{eqn:p2_beta_hat_ols} and
  \ref{eqn:p2_beta_hat_gls}, we can calculate relative efficiency
  \begin{align*}
    e\left(\hat{\beta}_0\right)
    &= \left.
      \left[\left(X^{\intercal}\Sigma^{-1}X\right)^{-1}\right]_{11}
      \middle/
      \left[\left(X^{\intercal}X\right)^{-1}X^\intercal
      \Sigma
      X\left(X^{\intercal}X\right)^{-1}\right]_{11}
      \right. \\
    e\left(\hat{\beta}_1\right)
    &= \left.
      \left[\left(X^{\intercal}\Sigma^{-1}X\right)^{-1}\right]_{22}
      \middle/
      \left[\left(X^{\intercal}X\right)^{-1}X^\intercal
      \Sigma
      X\left(X^{\intercal}X\right)^{-1}\right]_{22}
      \right.
  \end{align*}
  by taking the corresponding entries of the covariance matrix.
\end{description}
{\em \item Reproduce the lines in the table on 2.35 that pertain to $\beta_1$ for the following choices of covariate vectors
\beas
\bx&=&(-2,-1,0,1,2)\\
\bx&=&(-1,-2,0,2,1)\\
\bx&=&(0,-1,1,3,2)\\
\bx&=&(0,-1,1,5,2)
\eeas}

\begin{table}[ht!]
  \scriptsize
  \centering
  \input{p2_efficiencies.tex}
  \caption{Efficiency results comparing OLS with GLS.}
  \label{tab:p2_efficiencies}
\end{table}

\begin{description}
\item[Solution:] Results can be seen Table \ref{tab:p2_efficiencies}. Code is in the appendix.
\end{description}

{\em \item Explain the key differences between the relative efficiencies you just calculated.  Phrase your answers in a manner that will be understandable by
a quantitavely sophisticated non-statistician (e.g., an epidemiologist collaborator).
}

\begin{description}
\item[Solution:] If there's not much correlation, i.e. small $\rho$, there's not
  much efficiency gain in using GLS over OLS.

  For $\beta_1$, the efficiency gain does depend on the observed covariates,
  however. Recall that the estimated for $\beta_1$ can be written as a sum of
  weighted pairwise slopes. The weight is higher for for pairs where $x_{ij_1}$
  and $x_{ij_2}$ are further part because the effect size becomes bigger
  relative to the noise. Similarly, when the errors are positively correlated,
  more of the variance is explained away, so we are more confident about the
  slope estimate $\beta_1$.

  These two factors interact multiplicatively. Thus, when
  \[\mathbf{x} \in \left\{ (-1,-2,0,2,1), (0,-1,1,3,2), (0,-1,1,5,2) \right\},\]
  there are pairs of observations where the difference in covariates is large
  and the correlation is high. GLS is able to leverage this to produce a more
  efficient estimator. In particular when $\mathbf{x} = (0,-1,1,5,2)$, there are
  neighboring observations where the difference in covariates can be quite
  large, leading to the relatively most efficient estimator when using GLS.
\end{description}
\end{enumerate}

\section*{Appendix}

Code is attached on subsequent pages.

\clearpage

\includepdf[pages=-]{p1_glm.pdf}
\includepdf[pages=-]{ols_efficiency.pdf}

\end{document}


\documentclass{article}

\usepackage{graphicx}
\usepackage{subcaption}

\begin{document}
	\section{ Types of Logic Gates}
	Fundamental gates are AND, OR and NOT
	\begin{figure}[h!]
		\centering
		\begin{subfigure}[b]{0.4\linewidth}
			\includegraphics[width=\linewidth]{NOTANDORGATE.png}
		\end{subfigure}
	\subsection{NB}
	Derived Gates are NAND, NOR, XOR and XNOR (derived from the fundamental gates)

	Universal Gates are NAND and NOR gates (the fundamental logic gates can be realized through them).

	\end{figure}
\end{document}