\documentclass{article}

\usepackage[final]{style}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{verbatim}
\usepackage{graphicx}       % for figures
\usepackage{caption}
\usepackage{hyperref}
\usepackage{amsmath}
%\title{Lecture 12: DIMENSIONALITY REDUCTION}

\title{Lecture 12: Face Recognition \& Dimensionality Reduction}

\author{
  \textbf{Kyu seo Ahn, Jason Lin, Mandy Lu, Liam Neath, Jintian Liang, Lara Bagdasarian} \\
  Department of Computer Science\\
  Stanford University\\
  Stanford, CA 94305 \\
  \texttt{\{kyuseo, jason0, mlu355, lneath, jtliang, lbagdas\}@cs.stanford.edu} \\
}

\begin{document}

\maketitle

\section{Overview and Motivation}
\subsection{Overview}
Dimensionality reduction is a process for reducing the number of features used in an analysis or a prediction model. This enhances the performance of computer vision and machine learning-based approaches and enables us to represent the data in a more efficient way. There are several methods commonly used in dimensionality reduction. The two main methods covered in this lecture are Singular Value Decomposition (SVD) and Principal Component Analysis (PCA), both of which operate under the premise that we may express a set of features in a lower-dimensional space by finding a new coordinate system in which we may still approximately reconstruct the original feature set.

\subsection{Motivation}
Dimension reduction benefits models for a number of reasons.\\
\begin{enumerate}
\item Reduction in computational cost can be achieved. In many data sets, most of the variance can be explained by a relatively small number of input variables and their linear combinations. Focusing on these key components using dimensionality reduction, we can reduce the computational cost without losing much granularity in the data.

\item Reduce the effects of the “curse of dimensionality”. In lecture 11 we learned that as we increase the dimension of a feature space, the number of data points needed to “fill in” that space with the same density explodes exponentially. That is to say, the more dimensions used in a machine learning algorithm, the more examples are needed for learning and the longer it takes the algorithm to analyze the same number of data points. By performing dimensionality reduction, we can mitigate the effects of this “curse of dimensionality”.

\item Compress data. By reducing the dimensionality of an image, we can dramatically reduce the data storage requirements.

In such cases the computational cost per data point may be reduced by many orders of magnitude with a procedure like SVD
\end{enumerate}

\section{Singular Value Decomposition}
\subsection{Overview}
Intuitively, Singular Value Decomposition (SVD) is a procedure that allows one to represent the data in a new sub-feature space, such that the majority of variations in the data is captured; this is achieved by "rotating the axes" of the original feature space to form new axes which are linear combinations of the original axes/features (e.g. age, income, gender, etc. of a customer). These “new axes” are useful because they systematically break down the variance in the data points (how widely the data points are distributed) based on each directions contribution to the variance in the data:

% \begin{figure}[h!]
%   \caption{}
%   \centering
%   \includegraphics[width=0.5\textwidth]{Dimension_reduction_High_Level.png}
% \end{figure}
% * <kyuseoahn@gmail.com> 2017-11-07T22:40:47.042Z:
%
% >   \includegraphics[width=0.5\textwidth]{Dimension_reduction_High_Level.png}
% cant get this image to load ahh
%
% ^.

The result of this process is a ranked list of "directions" in the feature space ordered from most variance to least.  The directions along which there is greatest variance are referred to as the "principal components" (of variation in the data); by focusing on the data distribution along these dimensions, one can capture most of the information represented in the original feature space without having to deal with a high number of dimensions in the original space (but see below on the difference between feature selection and dimensionality reduction).

\subsection{Technical Details of Singular Value Decomposition}
\begin{itemize}
\item SVD represents any matrix	A as a product of three	matrices: $A = U\Sigma V^T$ where U and V are rotation matrices, and $\Sigma$ is a diagonal scaling matrix.	For example:\\
\includegraphics[scale = 0.5]{usigv_example}
\item For many readers, it may be sufficient to extract SVD values by writing: [U, S, V] = numpy.linalg.svd(A). However, the underpinnings of how SVD is computed is useful for later topics. Computers typically compute SVD by taking the following steps:
	\begin{itemize}
    	\item Compute the eigenvectors of $AA^{T}$. These vectors are the columns of U. Square root of the eigenvalues are the singular values (entries of $\Sigma$).
        \item Compute the eigenvectors of $A^{T}A$. These vectors are columns of V (or rows of $V^{T}$)
    \end{itemize}
\item Since SVD relies on eigenvector computation, which are typically fast, SVD can be performed quite quickly; even for large matrices.
\item A more detailed, geometric explanation of SVD may be found here\cite{ams}.
\end{itemize}

\subsection{Applications of Singular Value Decomposition}
\begin{itemize}
\item One the most utilized applications of SVD is the computation of matrix inverses. If an arbitrary matrix A can be decomposed by way of: $A = U\Sigma V^T$, the inverse of A may be defined as: $A^{+} = V^T \Sigma^{-1}U$. Although this inverse is an approximation, it allows one to calculate the inverses of many non-square matrices. MacAusland (2014) discusses the mathematical basis of this inverse, which is named the Moore-Penrose inverse, after its creators\cite{moore-penrose}. Unsurprisingly, a large variety of matrix problems can be solved be utilizing this approach.
\item Singular Value Decomposition can also be used to compute the Principal Components of a matrix. Principal Components are heavily utilized in various data analysis and machine learning routines, hence SVD is typically a core routine within many programs.
\end{itemize}

\section{Principal Component Analysis}

\subsection{What are Principal Components?}
Continuing with the SVD example we have above, notice that Column 1 of U gets scaled by the first value from $\Sigma$.\\

\includegraphics[scale = 0.5]{Usig}

Then, the resulting vector U$\Sigma$ gets scaled by row 1 of $V^T$ to produce a contribution to the columns of A which is denoted $A_{partial}$. Each product of (column i of U)*(value i from $\Sigma$)*(row i of $V^T$) produces a component of the final A.\\

\includegraphics[scale = 0.5]{full_pca}

In this process we are building the the matrix A as a linear combination of the columns of U. As seen above, if we use all columns of U, we rebuild A perfectly. However, in real-world data, \textbf{we can use only the first few columns of U} to get a good approximation of A. This arises due to the properties of $\Sigma$. $\Sigma$ is a diagonal matrix where the largest value is in the top left corner, and the rest of the values on the diagonal decreases as you move to the right. Thus, the first few columns of $U$ contribute the largest weight towards $A$. The first few columns of $U$ are called \textbf{principal components}. In order to reduce the dimensionality of our feature space, we may select the highest weighted $p$ principal components to serve as our new axes.\\

However, not all matrices can be easily compressed as in the previous example. Selecting only the first few columns of $U$ to use as our new axes may result in a compression in which the reconstructed feature set does not resemble the original one. That is to say, we underestimated the impact of the features we removed to our final image. They made independent contributions to the final image such that eliminating them results in a poor approximation of the original image. One way to generate compression axes which better reconstruct our original image is to use Principal Component Analysis. From a high level standpoint, we want to see if it's possible to remove dimensions that don't contribute much to the final image. We achieve this by analyzing the covariance matrix. Covariance is a measure of "spread" in a set of points around their center of mass or mean. More specifically, covariance conveys information about the dependency of two dimensions, thus measuring how much each of two dimensions varies from the mean with respect to each other. A positive value of covariance indicates that both dimensions vary together, a negative value indicates that the two dimensions vary inversely and a zero covariance indicates that the two dimensions vary independently of one another. The equation we use to express covariance is as follows:
$$covariance(X,Y)= \dfrac{\sum_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})}{n-1}$$

Furthermore, since covariance is a binary computation, we can express covariance of multiple dimensions using a covariance matrix containing elements of each pairwise covariance computation. Note that covariance is a symmetric computation, i.e., $cov(x,y)=cov(y,x)$. We can construct a covariance matrix for three dimensions, expressing it as such:$\newline$ \[cov(x,y,z) =
\begin{bmatrix}
    cov(x,x) & cov(x,y) & cov(x,z) \\
    cov(y,x) & cov(y,y) & cov(y,z)\\
    cov(z,x) & cov(z,y) & cov(z,z)
\end{bmatrix}
\]
By extension, N-dimensional data is associated with an $N\times N $ covariance matrix. As we will see in the section below, our covariance matrix will give us information about which features are duplicative of others due to high dependency. We will be able to describe our image very well without these features that the covariance matrix can reveals.

\subsection{Performing Principal Component Analysis}
Principal Component Analysis can be performed using the sklearn package:
sklearn.decomposition.PCA. However, it was alluded to earlier that SVD can be used to perform Principal Component Analysis.
To train our axes, we follow the below steps:
\begin{enumerate}
\item Format your data into a $m * n$ matrix where $m$ denotes the number of samples and $n$ represents the number of features or variables corresponding to a single sample.
\item Center the matrix $X$ by subtracting the mean across datapoints for each feature and dividing by the number of datapoints in $X$. We call this centered matrix $X_C$
\item Either apply SVD to $X_C$, in which case the ith eigenvalue is $\dfrac{\sigma_i^2}{n}$, where $\sigma_i^2$ is the ith value situated diagonally in $\sigma$, and corresponding ith eigenvector consists of the ith column of $U$, or alternatively (more computationally expensive) find the covariance matrix for the centered matrix across each of the $n$ dimensions (note that this matrix is square) and compute the eigenvalues and vectors.
\item Eliminate the eigenvalues and corresponding eigenvectors for all $\sigma_k^2$ if $\sigma_1^2>> \sigma_k^2$. Our new selection of eigenvectors comprises our new axes.
\end{enumerate}

To project a new point $T = \{t_1, t_2, \cdots, t_n\}, t_i \in \mathbb{R}^n$ into our dimensionally-reduced subspace, we perform the following procedure:
\begin{enumerate}
    \item Subtract the mean from each point $t_i$ such that our new point $t'_i = t_i-\hat{\mu}$.
    \item Project each point from $t_i \in \mathbb{R^n}$ to the lower dimensional new subspace that we found in the training procedure described previously, $Y \subseteq \mathbb{R}^k$ by using $y_i = At'_i$, where the rows of $A$ are the remaining eigenvectors:
    \[ A = \begin{bmatrix}
        \phi_i^T\\
        \phi_2^T\\
        \vdots \\
        \phi_k^T
    \end{bmatrix}\], where $\phi_k$ is the eigenvector corresponding to the eigenvalue $\sigma_k^2$.
\end{enumerate}

Detailed explanations that elucidate the reasoning behind the above steps are discussed by Moore (1981) and can be found on numerous websites online.

\subsection{Applications of Principal Components}
\begin{itemize}
\item \begin{minipage}[t]{\linewidth}
	PCA has been extensively used in image compression. Much of the 	information captured within an image matrix can be extracted using matrices of lower ranks. This allows large images to be compressed without significant loss of quality. An example of PCA based compression, using only the first 16 principal components, is shown below:\\

  \centering
  \includegraphics[width=\textwidth]{compare}
  \captionof{figure}{Right: Original Image, Left: Compressed Image}
  \label{fig:sample_figure}

\raggedright
$\,$\\
With just the first 16 principal components, an image that closely resembles the original image can be reconstructed. The relative error as a result of the dimensions used for PCA for the above image is shown below:

  \centering
  \includegraphics[width=\textwidth]{PCA_dim}
  \captionof{figure}{Relative Error as Function of PCA dimensions}
  \label{fig:sample_figure}
\end{minipage}

\item Web search engines also utilize PCA. There are billions of pages on the Internet that may have a non-trivial relation to a provided search phrase. Companies such as Google, Bing and Yahoo typically narrow the search space by only considering a small subset of this search matrix, which can be extracted using PCA\cite{search-clustering}. This is critical for timely and efficient searches, and it speaks to the power of SVD.
\end{itemize}

In essence, PCA represents data samples as weights on various components - allowing one to essentially represent the difference between samples. This can significantly reduce data redundancy and can make algorithms used in a variety of industries more efficient!

% References
\small
\bibliographystyle{plain}
\bibliography{bibliography}
\end{document}


%              %
%%            %%
%%% PREAMBLE %%%
%%            %%
%              %

% Document class
\documentclass[11pt]{beamer}

\usetheme{Boadilla}
\usecolortheme{beaver}
\useinnertheme{rectangles}

\setbeamertemplate{navigation symbols}{}

% Font
\usepackage{fontspec}
\setmainfont{Latin Modern Roman}

% Language and typography
\usepackage{polyglossia}
\setdefaultlanguage{english}
\setotherlanguage{french}

\usepackage{csquotes}

\usepackage{microtype}

% Mathematics
\usepackage{mathtools}
\usepackage{physics}

% Floats
\usepackage{float}
\usepackage{booktabs}
\usepackage{multicol}

% References
\usepackage{cleveref}

%              %
%%            %%
%%% DOCUMENT %%%
%%            %%
%              %

% Information
\title{Mathematics}
\subtitle{Professional style}
\author[A. Quenon]{Alexandre Quenon}

% Text
\begin{document}
% *** Title page *** %
\begin{frame}
	\titlepage
\end{frame}

% *** Contents *** %
\begin{frame}
	\frametitle{Overview}

	\tableofcontents
\end{frame}

% *** Tutorial *** %
%-----
\section{Useful packages}

\begin{frame}
	\frametitle{Packages for professional styling}

	Some packages very useful for mathematics and specifically for matrix computations are listed here below:
	\begin{itemize}
		\item \emph{mathtools} which is mainly an upgrade of the very well-known \emph{amsmath} package (the backbone for mathematics with \LaTeX{}),
		\item \emph{physics} which provides macros to handle efficiently mathematics professional styling.
	\end{itemize}
\end{frame}

%-----
\section{Various macros}

\begin{frame}
	\frametitle{Various macros}

	\structure{Absolute value}: \texttt{absolutevalue} command or the shorter form \texttt{abs}.
	Automatic sizing, starred version to cancel it.

	Examples:
	\begin{equation*}
		\abs{a}, \quad
		\abs{a^2}, \quad
		\abs{\frac{a}{b}}, \quad
		\abs*{\frac{a}{b}} \, \text{(starred)}
	\end{equation*}

	\structure{Norm}: \texttt{norm} command.
	Automatic sizing, starred version to cancel it.

	Examples:
	\begin{equation*}
		\norm{a}, \quad
		\abs{a^2}, \quad
		\norm{\frac{a}{b}}, \quad
		\norm*{\frac{a}{b}} \, \text{(starred)}
	\end{equation*}

	\structure{Order}: \texttt{order} command.
	Automatic sizing, starred version to cancel it.

	Examples:
	\begin{equation*}
		\order{x}, \quad
		\order{x^2}, \quad
		\order{\frac{1}{x}}, \quad
		\order*{\frac{1}{x}} \, \text{(starred)}
	\end{equation*}
\end{frame}

%-----
\section{Named functions}

\begin{frame}
	\frametitle{Named functions}

	Named functions, such as trigonometric (sinus, cosinus, etc.) or logarithmic, must be written in upright font because they are not variables.
	To do so, \emph{mathtools} defines intuitive commands and \emph{physics} extends them.
	For instance, \texttt{sin} for a sinus and \texttt{log} for logarithm, which are actually the true notations.
	An optional argument allows to pass a power to the function.

	Compare the wrong and correction notations:
	\begin{itemize}
		\item $sin(x)$ VS $\sin(x)$,
		\item $log(x)$ VS $\log[2](x)$,
		\item $Re(z)$ VS $\Re(z)$ ($\real(z)$ still accepted).
	\end{itemize}
\end{frame}

%-----
\section{Vectors}

\begin{frame}
	\frametitle{Vectors}
	\framesubtitle{Notations}

	\structure{Basic style}: to indicate that a variable is a vector and not a scalar, people usually put an arrow above the variable, especially for handwritten documents.
	Tool: \texttt{vec} command.

	Example:
	\begin{equation*}
		\vec{a}, \quad
		\norm{\vec{a}}
	\end{equation*}

	\structure{Professional style}: professional vector notation consists in putting the variable in bold font.
	Tool: \texttt{vectorbold} command, or the shorter form \texttt{vb}, from the \emph{physics} package.

	In addition, unit vectors should be written in bold font with a hat.
	Tool: \texttt{vectorunit}, or the shorter form \texttt{vu}.

	Example:
	\begin{equation*}
		\vb{a}, \quad
		\norm{\vb{a}}, \quad
		\vb{a} = a_x \vu{i} + a_y\vu{j} + a_z\vu{k}
	\end{equation*}
\end{frame}

\begin{frame}
	\frametitle{Vectors}
	\framesubtitle{Operators}

	\structure{Products}:
	\begin{itemize}
		\item inner/scalar/dot product $\rightarrow$ \texttt{dotproduct}, \texttt{vdot},
		\item cross product $\rightarrow$ \texttt{crossproduct}, \texttt{cross}, \texttt{cp}.
	\end{itemize}

	Examples:
	\begin{equation*}
		\vb{a} \vdot \vb{b}, \quad
		\vb{a} \cp \vb{b}
	\end{equation*}

	\structure{Gradient-based operators}:
	\begin{itemize}
		\item gradient/nabla $\rightarrow$ \texttt{gradient}, \texttt{grad},
		\item divergence $\rightarrow$ \texttt{divergence}, \texttt{div},
		\item curl/rotational $\rightarrow$ \texttt{curl}
		\item Laplace operator/Laplacian $\rightarrow$ \texttt{laplacian}.
	\end{itemize}

	Examples:
	\begin{equation*}
		\grad{\Psi}, \quad
		\div, \quad
		\curl, \quad
		\laplacian{\Psi}
	\end{equation*}
\end{frame}

%-----
\section{Differentials and derivatives}

\begin{frame}
	\frametitle{Differentials and derivatives}

	\structure{Differential}: the `d' symbol must be in upright font because it is not a variable.
	Tool: \texttt{differential}, or the shorter form \texttt{dd}.
	Optional argument to pass the power.

	Examples:
	\begin{equation*}
		\dd, \quad
		\dd{x}, \quad
		\dd[2]{x}, \quad
		\dd(\cos\theta)
	\end{equation*}

	\structure{Derivative}: same remark as for differential.
	Very long to write with \enquote{regular} \LaTeX{}.
	Tools: \texttt{derivative} or \texttt{dv} for \enquote{normal} derivative and \texttt{partialderivative} or \texttt{pdv} for partial derivative.
	Starred version to make it \emph{inline}.

	Examples:
	\begin{equation*}
		\dv{t}, \quad
		\dv{f}{x}, \quad
		\dv{\theta}(\cos[2](\theta)), \quad
		\pdv[n]{f}{x}, \quad
		\pdv{f}{x}{y}
	\end{equation*}
\end{frame}

\end{document}


\documentclass[a4paper,12pt]{article}

%Include Packages
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{textcomp}
\usepackage{graphicx}
%Fix margin issue
\usepackage[margin=0.8in]{geometry}
\usepackage{float}

\setcounter{tocdepth}{4}

%Top Matter
\title{Numerical Studies of Stochastic Spin Systems}
\author{Michael Conroy\\
  \\
  PHY 471 Capstone Project \\
  Spring 2014 \\
  Professor: Dr. Matthew Enjalran\\
  \date{}}

\begin{document}

\maketitle

\begin{abstract}
Analytic and numerical studies of stochastic spin systems were conducted to explore magnetic phase transitions. By applying the Monte Carlo Metropolis Algorithm (MCMA) to simulations of the classical Heisenberg Paramagnet in an external magnetic field, no phase transitions was found as expected of this non-interacting system. Conversly, MCMA simulations of the classical, isotropic, 3D Heisenberg Model explicitly show a second order phase transition at its critical temperature, $T_c \approx 1.45 K$. This is due to the cooperative effects of nearest neighbor interactions.

\end{abstract}

\pagebreak

%\tableofcontents
%\listoffigures
%\listoftables

\pagebreak

\section{Introduction}
A study of stochastic spin system was undertaken during the 2013-2014 school year for a PHY 499 Independent Study and PHY 471 Capstone Project. The fields of thermodynamics, statistical mechanics, and numerical methods were applied to compare analytic solutions to MCMA simulations of the classical Ising Paramagnet and the classical Heisenberg Paramagnet in an external magnetic field. Next, the 2D and 3D Ising Model were explored for periodic boundary conditions in preparation for simulating the classical, isotropic, 3D Heisenberg Model. The overall goal was finally met by comparing the Heisenberg Paramagnet with the critical phenomena demonstrated by the 3D Heisenberg Model.

Magnetism and magnetic systems in statistical physics provide a rich wealth of phenomena to study, especially that of critical phenomena near critical points in correlated systems. Several different thermodynamic properties such as specific heat, magnetization, and susceptibility exhibit critical behavior at critical points, and what is often visible is a sharp change in the properties of a substance. Of course, uncorrelated systems do not undergo a phase transition, but they do exhibit interesting magnetic behavior in external magnetic fields.

Comparing the correlated and uncorrelated systems illustrates the differences well. Phase transitions do not exist for non-interacting paramagnetic systems, because while paramagnets exhibit some magnetic properties in an external field, they do not show a macroscopic magnetization. The phase change such as that exhibited by the 3D Heisenberg Model exhibits a permanent macroscopic magnetization at an obvious critical point.

The studies in this paper illustrate paramagnetic magnetization of a classical Heisenberg Paramanget in an external magnetic field and the magnetic, second order phase transition of the 3D Heisenberg Model. No critical phenomena was found for the paramagnet, and a magnetic, second order phase transition was clearly evident at $\beta = 0.69$ for the 3D Heisenberg model as predicted in the literature. \cite{arxiv1, arxiv2} However, the susceptibility proved difficult to calculate.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Background and Theory}

\subsection{Statistical Mechanics}
The studied spin systems are characterized by the canonical ensemble. The probability of finding the model in any particular state is therefore proportional to the Boltzmann factor
\begin{equation}\label{eq:probability}
		P_\mu = \frac{1}{Z(\beta)}e^{-\beta E(\mu)}
\end{equation}
where $Z(\beta)$ is the partition function
	\begin{equation}\label{eq:partition}
		Z(\beta) = \sum_\mu e^{-\beta E(\mu)}
	\end{equation}
Most macroscopic thermodynamics variables of a system can be expressed by the partition function or its derivatives. The variables - energy, magnetization, susceptibility, heat capacity and absolute magnetization - are calculated, as well as the averages of those variables. The system's dependence on temperature is carried by the temperature's appearance in the probability term.

Exact solutions to the thermodynamic quantities for the Heisenberg Paramagnet were calculated and compared to MCMA simulations using the energy fluctuations form of the specific heat and magnetic fluctuations form of the susceptibility. For example, a comparison of the exact and fluctuations form of the specific heat per site:
	\begin{equation}\label{specific_heat_example}
		C =\frac{k\beta^2}{N}\frac{\partial^2\log{Z}}{\partial\beta^2} = \frac{k\beta^2}{N}(\left< E^2 \right> - \left< E \right>^2)
	\end{equation}
Since no exact solution exists for the 3D Heisenberg Model, only MCMA simulations were run using the fluctuations form of the specific heat and susceptibility. The energy and magnetization calculated in the simulations are averages over the lattice or site considered. \cite{gould}
\subsection{Magnetism and Magnetic Phase Transitions}
	\subsubsection{Para-, Ferro-, and Antiferromagnetism}
	Magnetically ordered solids separate broadly into many groups of magnetism, three of which we discuss here: ferromagnetism, antiferromagnetism, paramagnetism. In ferromagnets, the local moments are energetically preferred to be all aligned in a particular direction, the direction of spontaneous magnetization, so that the solid as a whole has a nonzero spontaneous magnetic moment. In antiferromagnets, adjacent local
moments prefer to be antialigned; paramagnets have zero spontaneous magnetization, because their moments are randomly oriented.
	\begin{figure}[H]
		\centering
		\includegraphics[scale=0.5]{ferro_para_anti}
		\caption{Schematic representations of magnetic dipole arrangements in (a) paramagnetic, (b) ferromagnetic, and (c) antiferromagnetic materials.}
		\label{fig:Types_of_Magnetism}
	\end{figure}
	\subsubsection{Magnetic Phase Transitions}
	The generatlized Hamiltonian of concern is
	\begin{equation}\label{eq:full_hamiltonian}
			H = -J\sum_{<ij>}\vec{S_i}\cdot\vec{S_j} + B\sum_i{\vec{S_i}}.
		\end{equation}
The first term is the exchange interaction and is responsible for the cooperative behavior and the possibility of a phase transition. J is the exchange energy (strength of the interaction): positive favors parallel and negative anti-parallel alignment of the spins. This term is relevant for the exploration of the 3D Heisenberg Model, where the external magnetic field $B = 0$.

The second term is the Zeeman interaction and responsible for the magnetic behavior of spins in a non-zero magnetic field. Therefore, the only influence ordering the spins is the magnetic field. In the case of the classical Ising and Heisenberg Paramgnets discussed here, $J = 0$, they do not interact, there are no cooperative effects, and hence no phase transition.

While the paramagnets display no magnetic phase transition, the 3D Heisenberg Model demonstrates a second order ferromagnetic phase transition. At a certain temperature, the critical temperature or critical point, the disordered paramagnetic Heisenberg system changes from one displaying seemingly no macroscopic magnetization into a spontaneously magnetized system -- a ferromagnetic system. The phase transition comes about purely from the cooperative behavior of the spins due to the exchange interaction.

Second order phase transitions occur when a new state of reduced symmetry develops continuously from the disordered high temperature phase. The ordered phase has a lower symmetry than the Hamiltonian -- the phenomenon of spontaneously broken symmetry. There will therefore be a number (sometimes infinite) of equivalent (e.g. equal free energy) symmetry related states. These are macroscopically different, and so thermal ﬂuctuations will not connect one to another in the thermodynamic limit. To describe the ordered state we introduce a macroscopic order parameter that describes the character and strength of the broken symmetry.

When the phase transition is happening between an ordered and a disordered phase, it is possible to construct an order parameter compatible with the symmetries of the system, vanishing in the disordered phase while being finite in the ordered phase. Although for a ferromagnetic-paramagnetic transition the choice of this order parameter is usually obvious (i.e. the magnetization), there are also systems with hidden order, where the nature of the order parameter is unknown. It is possible to define the correlation function of the order parameter. Far away from a phase transition, the correlations decay usually exponentially, defining characteristic length and time scale, $\xi$ and $\tau$, respectively. \cite{landau, newman}

At second order transitions, the order parameter vanishes continuously at $T = T_c$, when increasing the temperature. Usually, there exists an upper critical dimension $d_c$ for a given phase transition, so that the fluctuations do not play an essential role and mean field approximations. For example, the Ising Model exhibits a phase transition when $d_c \geq 2$, and the Heisenberg model exhibits a phase transition when $d_c \geq 3$. At these dimensions, the parameter fluctuations become very important.

Spontaneous breaking of symmetry is also an important signature of a phase
transition: the disordered phase typically exhibits the symmetries of the microscopic model. This manifests also in the vicinity of the global minimum of the thermodynamic potential, which usually also has large symmetry in the disordered phase. The second order transition shifts the global minimum in a direction chosen spontaneously to a point of reduced symmetry in parameter space. \cite{dissertation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Models and Methods}

\subsection{The Magnetic Models}
	\subsubsection{The Classical Paramagnet}
	The classical paramagnet is expressed by the Hamiltonian
	\begin{equation}\label{eq:zeeman}
		H = -\sum_{i=1}^N{\vec{b}\cdot\vec{S_i}},
	\end{equation}
	which is a simplified version of the Zeeman interaction. This is the sum over all sites of the dot product of the effective magnetic field $\vec{b}$ along the z-axis and each site's spin, $\vec{S_i}$. The Heisenberg Paramagnet is a continuous spin model, so $\vec{S_i} = (\sin\theta_i\cos\phi_i, \sin\theta_i\sin\phi_i, \cos\theta_i)$, where $|\vec{S_i}| = 1$.  From these equations, the Hamiltonian for the Heisenberg Paramagnet will be derived. The average magnetization per site MCMA simulation and susceptibility were not calculated.
	\subsubsection{The Heisenberg Paramagnet}
	\begin{figure}[H]
			\centering
			\includegraphics[scale=5.0]{spherical_unit_vector}
			\caption{Spherical unit vector of a Heisenberg spin.}
			\label{fig:spherical_unit_vector}
		\end{figure}
		\paragraph{Description and Analytic Solution}
		The classical Heisenberg Paramagnet spin at each site is defined by two sphereical polar angles, but if the $\vec{b}$ direction is defined as the global $z$-direction, then the Zeeman interaction of \eqref{eq:zeeman} becomes
		\begin{equation}\label{eq:heisenberg_PM_hamiltonian}
			H = -b\sum_{i=1}^N{\cos \theta_i},
		\end{equation}
		because the energy of the spin is only dependent on the polar angle $\theta_i$. The partition function for a system of $N$ independent spins in an external magnetic field is then
		\begin{equation}\label{eq:heisenberg_para_partition_fxn}
			Z_N = \left[\frac{4\pi}{\beta b}\sinh(\beta b)\right]^N = (Z_1)^N,
		\end{equation}
		and the partition function for one spin is $Z_1 = (4\pi/\beta b)\sinh(\beta b)$. Again, all thermodynamic quantities for this model are now analytically calculable.
		\paragraph{MCMA Solution Form}
		For the classical Heisenberg Paramagnet model, the average energy per site $\langle E \rangle$, the specific heat $C$, and the average magnetization per site $\langle M \rangle$ are (let $k = b = 1$ here):
		\begin{equation}\label{eq:heisenberg_PM_avg_energy}
			\frac{\langle E \rangle}{N} = -\frac{1}{N}\sum_j^N{\cos \theta_j},
		\end{equation}
		\begin{equation}\label{eq:heisenberg_PM_specific_heat}
			C = \frac{\beta^2}{N}(\left< E^2 \right> - \left< E \right>^2),
		\end{equation}
		and
		\begin{equation}\label{eq:heisenberg_PM_avg_magentization}
			m = \frac{\langle M \rangle}{N} = -\frac{1}{N}\left|\sum_j^N{\cos \theta_j}\right|.
		\end{equation}

	\subsubsection{Heisenberg Model}
			\begin{figure}[H]
			\centering
			\includegraphics[scale=0.5]{lattice}
			\caption{Paramagnetic ($T>T_c$) Heisenberg Model on a 10x10x10 Lattice.}
			\label{fig:heisenberg_lattice}
		\end{figure}
		\paragraph{Description}
The classical, isotropic, 3D Heisenberg Model on the simple cubic lattice is now considered. The spins themselves are considered in the same manner as the Heisenberg Paramagnet, but now cooperative effects are applied by setting $B = 0$ and $J = 1$ for the ferromagnetic case. The zero-field version of the model was chosen for simplicity as is usually the case for multi-dimensional spin models with interacting spins.	\cite{scribd}

Easy to state and widely applicable, the Heisenberg Model is also extremely difficult to analyze.The Heisenberg model is considered the superior to the Ising model in the case of magnetic systems, but its basis on more complicated interactions between spins makes it mathematically complex. On the other hand, the Ising model, derived from statistical physics, is much simpler and much more popular. However, the Heisenberg Model was chosen for its better approximation of physical reality.
		\paragraph{MCMA Solution Form}
The average energy per site $\langle E \rangle$, the $rms$ magnetization  $\langle M \rangle$, the specific heat $C$, and the magnetic susceptibility $\chi$ are (let $k = b = 1$ here)
	\begin{equation}
		\frac{\langle E \rangle}{N} = -\frac{J}{2N}\sum_{<ij>}^{N}\vec{S_i}\cdot\vec{S_j},
	\end{equation}
	(factor of $\frac{1}{2}$ for double counting)
	\begin{equation}
		C = k\beta^2(\left< E^2 \right> - \left< E \right>^2),
	\end{equation}
	and
	\begin{equation}
		m_{rms} = \sqrt{M_x^2 + M_y^2 + M_z^2},
	\end{equation}
	where $\displaystyle M_\alpha = \frac{1}{N}\sum_i{\vec{S_{i\alpha}}}$.

The Heisenberg system has an order parameter with multiple components and all the three components are equally important. The order parameter is invariant under global rotation and therefore we need to keep track of its magnitude alone. However, this causes an issue calculating the magnetic susceptibility, $\chi$. Thus, the standard form of $\chi$
			\begin{equation}
				\chi = \frac{\beta}{N}(\langle M^2 \rangle - \langle M \rangle^2) = \beta N(\langle m^2 \rangle - \langle m \rangle^2)
			\end{equation}
cannot be used, and the correlation function must be computed. This step was not completed for this paper due to time constraints and the inherent complexity involved with implementing the calculation.

\subsection{The Metropolis Monte Carlo Method}
	\subsubsection{Introduction}
	The field of numerical analysis is applied to intractable problems and problems without analytic solutions such as the $d \geq 3$ dimensional Ising Model and 3D Heisenberg Model. The MCMA method was utilized for the models explored in this paper. The uniform pseudo-random number generators built in to Fortran and C were used and consistently seeded to maintain random number generation among simulations written in each language.
	\subsubsection{Implementation}
A standard MCMA method was applied. The Metropolis Algorithm importance sampling was implemented. The MCMA importance sampling algorithm is now described step-by-step:
	\begin{enumerate}
		\item Choose initial state
		\item Choose a site
		\item Calculate $\Delta E$ if ``flip" the spin
		\item If $\Delta E \le 0$, accept ``flip" and go to (7), otherwise (5)
		\item RNG: $0 < r < 1$
		\item If $r < \exp(-\beta \Delta E)$, accept ``flip"
		\item Go to next site and go to (3)...
	\end{enumerate}
The spin ``flip" corresponds to choosing new, randomly generated $\theta$ and $\phi$ angle for the Heisenberg Paramagnet and 3D Heisenberg Model spins. \cite{landau}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Results and Discussion}
\subsection{Heisenberg Paramagnet}
	\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{e_n.pdf}
			\caption{Average energy per site for the Heisenberg Paramagnet.}
			\label{fig:heisenberg_PM_e_avg}
		\end{figure}

		The average energy per site MCMA simulation agrees with the exact solution. As the spins align with the external magnetic field, the energy reduces to the lowest level $\langle E \rangle = -1$.

		\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{sp.pdf}
			\caption{Specific heat for the Heisenberg Paramagnet.}
			\label{fig:heisenberg_PM_sh}
		\end{figure}

		The specific heat is indicated of the fluctuations that take place as the energy curves down and the spins align with the magnetic field. The clumping of the MCMA solution points at the top of the curve is due to critical slowing down because of the Metropolis Algorithm's issues at the critical point. There is no evidence here of a phase transition, because the lowest energy state and maximum fluctuations exist at $T = 0 K$ where the specific heat is greatest.

		\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{m_avg_PM.pdf}
			\caption{Average magnetization exact solution for the Heisenberg Paramagnet.}
			\label{fig:heisenberg_PM_m_avg}
		\end{figure}

		The average magnetization exact solution shows a peak of $\langle M \rangle = 1$ at $T = 0 K$. As expected, maximum magnetization is indicative of spin abd magnetic field alignment. There is no continuous change from zero-magnetization to a non-zero magnetization across a critical point, only a trend towards a maximum magnetization as the system cools towards $T = 0 K$. Therefore, no phase transition is evident.

\subsection{Heisenberg Model}
	\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{e_avg.pdf}
			\caption{Average energy per site for the 3D Heisenberg Model.}
			\label{fig:heisenberg_e_avg}
		\end{figure}

		The average anergy per site agrees well with expectations. The energy will trend to a minimum of $\langle E \rangle = -3$ after curving down at $T_c \approx 1.45 K$ when the spins align at the lowest energy potential. In the thermodynamic limit, the energy should trend to $\langle E \rangle = 0$, but this simulation maxes out at $T = 5 K$. A larger temperature range will show the trend towards zero. Finite size effects are evident in the steepening of the curve around the critical temperature as the lattice sizes increase. In the thermodynamic limit, the graph should curve straight down at the critical temperature.

		\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{cplot}
			\caption{Heat capacity for the 3D Heisenberg Model.}
			\label{fig:heisenberg_heat_capacity}
		\end{figure}

		The heat capacity curve also agrees well with expectations. The peak should trend to infinity and width should trend to zero at the critical temperature in the thermodynamic limit. The data clearly details this as lattice sizes grow from $L = 4$ to $L = 10$. For larger lattice sizes, the peak should grow and narrow even more, but critical slowing down issues with the Metropolis Algorithm implementation clearly demonstrate that the higher lattice dimension simulations were not run long enough at the critical point. This also implies that the higher lattice sizes may not have relaxed enough thermodynamically and increased equilibration and measurement phase runs are required.

		The non-zero magnitude of the specific heat under the critical temperature is important to note. The 3D Heisenberg Model has a continuum of ground states. Therefore, fluctuations will still occur when the spins are aligned, but the entire system will randomly reorient itself in any direction. The spins will stay aligned, but the direction they point will randomly fluctuate.

		\begin{figure}[H]
			\centering
			\includegraphics[scale=0.65]{m_avg}
			\caption{Average magnetization per site for the 3D Heisenberg Model.}
			\label{fig:heisenberg_m_avg}
		\end{figure}

		The magnetization demonstrates the same steepening at the critical point. The curve also doesn't trend towards zero in the thermodynamic limit, but not due to the chosen temperature range. The magnitude of the magnetization is considered, not the magnetization itself, so the fluctuations will always cause a positive magnetization--an overestimation. This is a direct result of calculating $m_{rms}$ and is expected.

		Clearly, a continuous phase transition occurs. A zero magnetization region, $T > T_c$, is separated from a non-zero magnetization region, $T < T_c$, around a critical point where the magnetization rises and susceptibility diverges. The order parameter defines as the magnetization for this model demonstrates  a clear ferromagnetic phase change at $T_c = 1.45 K$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conclusion}
The data matches expectations and the predictions from iterature. Magnetic behavior was shown for the Heisenberg Paramagnet, which lacked a magnetic phase change as expected. Agreeing with the literature quite well, a phase transition at the critical temperature of $T_c \approx 1.45 K$ was evidenced for the classical, isotropic, 3D Heisenberg Model.

Three issues remain. The first is how the the magnetic susceptibility of the 3D Heisenberg Model was not as straightforward to calculate as assumed. The second concern is an acceptance ratio oddity for this model's simulation where artificially high ratios do not correlate well with how the model was programmed. Instead of slightly perturbing the $\theta$ and $\phi$ angles at each MCMA update, a new, completely random angle was chosen for both. Therefore, the acceptance ratios should be small instead of on the order of $\approx0.80$ at the start of each simulation to $\approx0.50$ at the conclusion.

The final concern is how the 3D Heisenberg Model was programmed. The larger simulations took on the order of a week and a half to run, but critical slowing down still became evident. The larger simulations were therefore not run for long enough times to let the system fully relax before measurements were taken. Three and four dimensional pseudo-arrays were implemented in C, which caused extremely slow run times. Altering the code to run more efficiently with one and two dimensional arrays and implementing structures would allow for shorter run times, and therefore increased runs for the equilibration and measurement phases.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Acknowledgments}
The author would like to thank Tom Sadowski for his invaluable help with programming multidimensional arrays with pointers in C.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{thebibliography}{9}

  \bibitem{arxiv1}
  {\tt arXiv:hep-lat/9301002}

  \bibitem{arxiv2}
  {\tt arXiv:hep-lat/9209017}

  \bibitem{gould}
  Gould, H., \& Tobochnik, J. (2010).
  \emph{Statistical and thermal physics: with computer applications}.
  Princeton, N.J.: Princeton University Press.

	\bibitem{landau}
  Landau, D. P., \& Binder, K. (2009).
  \emph{A guide to Monte Carlo simulations in statistical physics (3rd ed.)}.
  Cambridge: Cambridge University Press.

  \bibitem{newman}
  Newman, M. E., \& Barkema, G. T. (1999).
  \emph{Monte Carlo methods in statistical physics}.
  Oxford: Clarendon Press.
  \bibitem{dissertation}
  Rapp, \'Akos.
  \emph{Quantum Phase Transitions in Correlated Systems}.
  http://dept.phy.bme.hu/phd/dissertations/rapp\_dissertation.pdf

  \bibitem{scribd}
  Tangirala, Sairam.
  \emph{Monte Carlo Simulation of 1D Heisenberg Model}.
  http://www.scribd.com/doc/50664110/Monte-Carlo-Simulation-of-1D-Heisenberg-Model

\end{thebibliography}

\end{document}