% % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
%References list
% 1 - wikipedia radio astronomy article https://en.wikipedia.org/wiki/Radio_telescope
% 2 - LFRA from NCRA
% 3 - Thomson's radio interferometry book
% 4 - Wikipedia interferometry article
% 5 - Born and Wolf statistical
% 6 - paper on feathering - https://casa.nrao.edu/Release3.4.0/docs/userman/UserManse30.html
% 7- Rau, Cornwell 2011 " labal urvpaper
% 8 - URV's PhD thesis : label ; urvthesis
% 9- Cornwell2008 : label cornwell2008
% 10 - CASA document log for MTMFS
% 11 - URV's talk on imaging-deconvolution
% % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\documentclass{article}
\usepackage[margin = 2cm]{geometry}
\usepackage{graphicx}
\usepackage{float}
\usepackage{hyperref}
\usepackage{subfig}
\title{A joint deconvolution algorithm to combine single dish and interferometer data for wideband multi-term imaging}
\author{Nikhil Vijay Naik\footnote{Indian Institute of Technology, Kharagpur, West Bengal, India - 721302}}
\begin{document}
\maketitle
\begin{abstract}
	Imaging in Radio Astronomy is done by the means of either Single-Dish Radio Telescopes or Interferometric Array Radiotelescopes. Image Formation in a radio interferometer deals with calculating the inverse Fourier Transform of the Visibility Function $V(u,v)$. This strategy offers excellent angular resolutions but suffers from the \textit{short-spacing} problem, where sources with large angular size and low surface brightness are not seen in an interferometer image. On the other hand, single-dish radio telescopes offer very coarse resolution, but respond very well to large angular scales. \\\\This project aims to develop and prototype an algorithm to combine simulated data from these two types of telescopes, in order to bring about a good reconstruction of the true sky image at a wide range of frequencies. Consequently, an important physical parameter, \textit{viz.} the spectrum of every source in the Field of View (FOV) is estimated from the reconstructed image.
\end{abstract}
\newpage
\tableofcontents
\newpage
\section{Introduction}
Radio Telescopes are used for creating images of the sky in the Radio Frequency regime of the Electromagnetic (EM) spectrum, which ranges from 3kHz to 300GHz \hyperref[ref1]{[1]}. Radio Telescopes belong to two distinct categories :
\begin{enumerate}
\item Single-Dish Radio Telescopes and
\item Interferometric Arrays.
\end{enumerate}
\label{section1.1}\subsection{Imaging with Single-Dish Radio Telescopes : Introduction}
Single-Dish radio telescopes, such as the Robert C. Byrd Green Bank Telescope (hereafter GBT) are designed to respond linearly to the intensity of the radiation received from a point in the sky. The strategy followed to create an image from such an instrument is as follows : \\
The antenna beam \footnote{It is the spatial response of the antenna to a source of unit intensity located in the sky.} sweeps across the region of interest. The response of each point on the sky as the beam sweeps across it is summed and put into a pixel. This is carried out for the whole Field-of-View (FOV) till the region has been fully sampled. This strategy is known colloquially as "basket-weaving", and is illustrated in \figurename{ 1}. The white circles represent beam sampling.\\
\begin{figure} [H]
\centering
\begin{minipage}[b]{.45\textwidth}
\includegraphics[width=\linewidth]{Extended_Flat_true_im_enhanced.png}
\end{minipage}%
\begin{minipage}[b]{.45\linewidth}
\includegraphics[width=\textwidth]{BasketWeaving.png}
\end{minipage}
\caption{Illustration of the true sky image (left) and imaging using single-dish "basket weaving" technique (right)}
\label{ 1}
\end{figure}
Mathematically, the basket-weaving technique is equivalent to a \textit{convolution}, where the antenna beam is \textit{convolved} with the true sky image in order to form the image :
\begin{equation}
I^{Image} = I^{Sky} * P^{Antenna}
\end{equation}
where $P^{Antenna}$ is the beam or hereafter the \textit{Antenna Power Pattern}, and '$*$' denotes the convolution operator.
\subparagraph{Properties of Single-dish images}
From the theory of diffraction, it is known \hyperref[ref2]{\hyperref[ref2]{[2]}}\hyperref[ref3]{[3]} that the highest possible resolution in an image  that is created from an aperture of diameter \textit{D} for radiation of wavelength $\lambda$ is given by
\begin{equation}
\theta \sim \frac{\lambda}{D}
\end{equation}
From this equation, it is immediately seen that the resolution for radio frequencies is rather coarse. For the GBT dish with a diameter of 100m, the resolution at $\nu\sim$ 1.5GHz is still $\sim5$ arcmin. Hence, single dish radio-telescopes are not useful for obtaining high-resolution radio images. This logically leads to the technique of interferometry for conducting high-resolution imaging.
\label{sec1.2}\subsection{Imaging with Interferometric Arrays : Introduction}
\begin{figure}
\centering
\begin{minipage}[b]{.35\textwidth}
\includegraphics[width=\linewidth]{VLA_pic.jpg}
\end{minipage}%
\begin{minipage}[b]{.35\linewidth}
\includegraphics[width=\textwidth]{GMRT_pic.jpg}
\end{minipage}
\caption{Examples of interferometers : The Very Large Array (VLA) (left) and the Giant Metrewave Radio Telescope (GMRT) (right). Image credits - NRAO and NCRA (VLA and GMRT respectively) }
% % give image credits to NRAO and NCRA.
\label{2}
\end{figure}
Interferometric arrays (\figurename{ 2}) are telescopes that have multiple elements or antennas. The radiation received at each antenna is combined using signal-processing software/hardware for creating the image - this technique is called 'Interferometry'. Each pair of antennas measures an \textit{interference fringe}, and $n$ antennas give $\frac{n(n-1)}{2}$ such fringes. Measurement of these fringes leads to the mathematical reconstruction of the \textit{Visibility Function V(u,v)}. By the Van Cittert - Zernicke theorem of optics, it is known that the Intensity of radiation in the plane\footnote{If the FOV is too large, the assumption of the sky being a plane does break down} of the sky and the visibility function are related by a two-dimensional Fourier transform \hyperref[ref2]{[2]}\hyperref[ref3]{[3]}\hyperref[ref4]{[4]}\hyperref[ref5]{[5]}.
Mathematically, we write
\begin{equation}
V(u,v) = \int\int\frac{I(l,m)}{\sqrt{1-l^2 - m^2}} e^{-2\pi i [lu+mv]} du dv
\end{equation}
Where $V(u,v)$ is the visibility function, $I(l,m)$ is the intensity in space, $l, m$ are the \textit{direction cosines} and $(u,v)$ are the \textit{Fourier-domain} coordinates. A deeper treatment of interferometry, along with its mathematics can be found in \hyperref[ref2]{[2]}\hyperref[ref3]{[3]}\hyperref[ref4]{[4]}.
\subparagraph{Properties of interferometer images}
From Equation (2), we know that the finest resolution that can be measured by an aperture of diameter D is $\lambda/D$. In case of an interferometer, $D$ is the \textit{longest baseline}, defined as the maximum separation between a pair of antennas, which is unconstrained.
This makes the technique of interferometry extremely powerful and it leads to very high-resolution images (arcsecond or sub-arcsecond) of the radio sky.
As can be seen in \figurename{ 3}, the image of the same sky as seen by a single-dish radio telescope and an interferometer can look very different in terms of resolution.
\begin{figure}[H]
\centering
\begin{minipage}[b]{.39\textwidth}
\includegraphics[width=\linewidth]{Extended_Flat_SID_img}
\end{minipage}%
\begin{minipage}[b]{.4\linewidth}
\includegraphics[width=\textwidth]{VLA_img}
\end{minipage}
\caption{Image of the true sky from \hyperref[ 1]{Figure 1}, reconstructed using a single dish telescope (left) and an interferometer (right). }
\label{3}
\end{figure}
\textit{Note the point sources are almost completely absent on the left, but the extended source in the interferometer image on the right is imaged very faintly. This observation forms an important justification for this project.}
\subsection{Motivation behind the Project and Problem Statement}
As was seen in \figurename{ 3}, a single-dish instrument provides images with very coarse resolution. An interferometer provides images with excellent resolution, however, it does not faithfully reproduce objects that have very large angular sizes and low surface brightness. Scientific goals of various kinds can be achieved by combining best of what these two classes of instruments have to offer.\\Techniques such as \textit{feathering}\hyperref[ref6]{[6]} exist to combine the two kinds of images. However, these suffer from many problems, rendering them insufficient as a reliable technique for image combination. These include :
\begin{enumerate}
\item It depends heavily on the arbitrary scaling of single-dish data, which may not always lead to physically sensible results.
\item It works only for data taken at exactly one frequency. It fails to utilise the vast potential of multi-frequency data taken with modern telescopes such as the VLA or the GMRT.
\item Wideband Imaging algorithms exist \hyperref[urvpaper]{(Rau and Cornwell 2011)}\hyperref[urvthesis]{(Rau 2011, PhD thesis)} for pure interferometer data, but due to the inherent limitations of imaging with an interferometer (called the \textit{short spacing problem}), these are not useful for sources which have large angular extent.
\end{enumerate}
\textbf{\textit{This project aims to resolve all these deficiencies associated with feathering and wideband image reconstruction, namely, the dependency on the single-dish scaling and the short-spacing problem.}}\\*\\*
\textbf{\textit{\underline{Problem Statement} : Development of a joint deconvolution algorithm to combine single dish and interferometer data for wideband multi-term imaging.}}
\section{Development of the Methodology and Algorithm}
This section breaks down the problem statement into its component sub-problems. In \hyperref[AlgoDevpt]{Section 2.4}, the individual solutions are put together to develop the final algorithm which is heuristically described.
\label{shortspacingprob}\subsection{The Short Spacing Problem}
By the Van Cittert-Zernicke theorem (\hyperref[sec1.2]{Section 1.2} and references therein), the Fourier Transform of the intensity (as a function of space) is the  \textit{complex visibility function}. Hence, the Fourier transform of the reconstructed image will give the sampled visibility function (in the discrete case). \\*\\*The short spacing problem is illustrated in \figurename{ 4}. As can be seen, the Fourier transform of the interferometer image insufficiently samples the region around $(0,0)$. This is called the \textit{central uv-hole}. On the other hand, the single-dish data samples this precise region very well (\figurename{ 4} bottom half), but lacks the data at large values of (u,v). This is overcome using the feathering technique, but it suffers from a few problems. The technique and its shortcomings are discussed in \hyperref[feathering]{Section 2.3}.
The origin of the short spacing problem is briefly described below. \\*\\*
The $(u,v)$ coordinate system is defined as follows \hyperref[ref2]{[2]}\hyperref[ref3]{[3]}\hyperref[urvpaper]{(Rau and Cornwell 2011)}\hyperref[urvthesis]{(Rau 2011, PhD thesis)}:
\\*\\*
\begin{minipage}{0.4\textwidth}
$u = \frac{b_{E-W}}{\lambda}$
\end{minipage}
\begin{minipage}{0.4\textwidth}
$v = \frac{b_{N-S}}{\lambda} $
\end{minipage}
\\*\\*
Where, $b_{E-W}$ and $b_{N-S}$ are the lengths of the baselines (\textit{i.e.} - the separation between two antennas) in the East-West and North-South direction respectively. As any two antennas must always be separated by a finite nonzero distance, the values $u$ and $v$ can never be simultaneously zero. Hence, the smallest spacings always remain un-sampled. \\*As sources with large angular extent have visibility values clustered around the origin, they are always undersampled and missed by the interferometer. This is clearly seen in \figurename{ 4}\footnote{Note that the image in \figurename{ 4} is flipped.}.
\begin{figure}
\centering
\begin{minipage}[b]{.35\textwidth}
\includegraphics[width=\linewidth]{VLAimg.png}
\includegraphics[width=\linewidth]{GBTimg.png}
\end{minipage}%
\begin{minipage}[b]{.35\linewidth}
\includegraphics[width=\textwidth]{FFTVLAimg.png}
\includegraphics[width=\textwidth]{FFTGBTimg.png}
\end{minipage}
\caption{The short spacing-problem.}
% % give image credits to NRAO and NCRA.
\label{4}
\end{figure}
The interferometer (VLA) samples large values of the \textit{(u,v)} plane but leaves out the area near $(0,0)$. On the other hand, the Single-dish telescope (GBT) does not sample large values of $(u,v)$ but densely samples the centre
\label{section2.2}\subsection{Multi-Term, Multi-Frequency Wideband Imaging}
	The description of a narrow-band multiterm multifrequency algorithm draws heavily from two sources : \hyperref[urvpaper]{Rau and Cornwell 2011} and \hyperref[urvthesis]{"Parameterized Deconvolution For Wide-Band
Radio Synthesis Imaging"} (Ph.D. dissertation of Urvashi Rau Venkata at the NRAO).
Any radio image containing information on multiple angular scales can be written as a linear sum of a \textit{sky model} comprising of $\delta$ functions, and a \textit{smoothing kernel} which is usually a tapered, inverted, truncated paraboloid (\hyperref[cornwell2008]{Cornwell 2008}). Mathematically, this amounts to
\begin{equation}
\textbf{I}^{m} = \sum_{s=0}^{N_s-1}\textbf{I}_s^{shp} * \textbf{I}_s^{sky, \delta}
\end{equation}
$\textbf{I}^{m}$ is the model sky image, $\textbf{I}_s^{shp}$ is the kernel function and $\textbf{I}_s^{sky, delta}$ is the $\delta$ function model of the sky. '*' denotes the convolution operator.
The sky changes with the observing frequency. To model this frequency dependence, the sky is modeled as a Taylor series about a centre frequency :
\begin{equation}
\textbf{I}_\nu^{m} = \sum_{t=0}^{N_t-1}w_\nu^{t}\textbf{I}_t^{sky}
\end{equation}
Moreover, the dependence of the sky on the frequency is modeled as a power-law with a curvature term :
\begin{equation}
\textbf{I}^{sky}_\nu = \textbf{I}^{sky}_{\nu_0}{(\frac{\nu}{\nu_0})}^{{I}^{sky}_\alpha + {I}^{sky}_\beta log(\frac{\nu}{\nu_0})}
\end{equation}
Next, putting together equations (4) and (5) leads to :
\begin{equation}
\textbf{I}_\nu^{m} = \sum_{s=0}^{N_s}\sum_{t=0}^{N_t}w_\nu^{t}[\textbf{I}_s^{shp} *\textbf{I}_{s_t}^{sky}]
\end{equation}
In all these equations, $w_t = (\frac{\nu - \nu_0}{\nu_0})^{t}$\\
In Equation (7), $\textbf{I}_s$ represents a collection of $\delta$-functions that describe the sky locations and amplitudes of flux components corresponding to angular scale $s$ in the image formed by the $t^{th}$ Taylor series coefficient. (\hyperref[urvpaper]{Rau and Cornwell 2011}, section 2.2).\\
Using the convention outlined in \hyperref[urvpaper]{Appendix A of Rau and Cornwell 2011}, the relation between the image \textbf{I}, the visibilities \textbf{V} can be expressed in Matrix notation as:
\begin{equation}
\textbf{V}_{n\times1}^{obs} = [\textbf{S}_{n\times m}][\textbf{F}_{m\times m}]\textbf{I}_{m\times 1}^{sky}
\end{equation}
	where $\textbf{V}_{n\times1}^{obs}$ is the vector containing the observed visibilities, $[\textbf{S}_{n\times m}]$ is the matrix that indicates sampling on the $(u,v)$ plane, $[\textbf{F}_{m\times m}]$ is the Fourier Transform operator and $\textbf{I}_{m\times 1}^{sky}$ is the pixilated image of the sky intensity. Using this convention, the equation relating visibilities to the sky intensity at a given frequency $\nu$ becomes :
\begin{equation}
\textbf{V}_{\nu}^{obs} = \sum_{s=0}^{N_s}\sum_{t=0}^{N_t}w_\nu^{t}[\textbf{S}_{\nu}][\textbf{T}_s][\textbf{F}]\textbf{I}^{sky}_{s_{t}}
\end{equation}
In this case, $[\textbf{T}_s]$ is a frequency \textit{taper function}. Further description of this function can be found in \hyperref[urvpaper]{Rau and Cornwell 2011}.
For multiple frequencies to be imaged in the same data set, we can stack the frequency weights in the matrix $[\textbf{S}_{\nu}]$. The weight function $w_\nu$ will also become a matrix in this case, $[\textbf{W}_t^{mfs}]$. Rewriting equation (9) in this form, one obtains :
\begin{equation}
\textbf{V}^{obs} = \sum_{s=0}^{N_s}\sum_{t=0}^{N_t}[\textbf{W}_t^{mfs}][\textbf{S}][\textbf{T}_s][\textbf{F}]\textbf{I}^{sky}_{s_{t}}
\end{equation}
Here, $\textbf{V}^{obs}$ is now an $nN_c\times 1$ matrix of \textit{multi-frequency} visibilities, [\textbf{S}] is the $(u,v)$ sampling function of the multi-frequency data.
\subparagraph{Comment on the structure of the matrix $[\textbf{W}_t^{mfs}]$}
$[\textbf{W}_t^{mfs}]$ is diagonal $nN_c\times nN_c$ matrix of weights. It consists of $N_c$ diagonal blocks of size $n\times n$, and contains $w_\nu^{t}$ in its blocks.
\subparagraph{Multiterm, Multi Frequency Synthesis Image formation}
The interferometer data records visibilities from various frequencies. The MTMFS imaging algorithm plots all these visibilities on a single \textit{(u,v)} grid and recreates the dirty image set $\textbf{I}^{sky}_t$ through the matrix inversion route. This dirty images will then be fed to the MTMFS deconvolver which deconvolves the dirty images using a set of pre-defined angular scales \hyperref[ref9]{[9]}. The output will be a set of Taylor images that will encapsulate the multiterm data.
\begin{figure}
\centering
\begin{minipage}[b]{.45\textwidth}
\includegraphics[width=\linewidth]{Extended_Flat_mtmfsonly_image}
\end{minipage}%
\begin{minipage}[b]{.45\linewidth}
\includegraphics[width=\textwidth]{Extended_Flat_mtmfs_only_alpha}
\end{minipage}
\caption{The reconstructed multiterm images using pure MTMFS for the sky shown in \hyperref[ 1]{Figure 1}}
% % give image credits to NRAO and NCRA.
\label{5}
\end{figure}
\subparagraph{Limitations of the MTMFS deconvolution algorithm}
The MTMFS algorithm detailed in \hyperref[urvpaper]{Rau and Cornwell 2011} uses only pure interferometer data. Due to the short-spacing problem \hyperref[shortspacingprob]{(Section 2.1)}, it does not work for sources that have low surface brightness and large angular extents. However, there are important science goals that can be achieved by extending to this technique for such sources. This can be clearly seen in \hyperref[fig5]{Figure 5}, where the zeroth-order Taylor image looks almost identical to that of \hyperref[fig:Extended_Flat_image_tt0]{Figure 8}, but the spectrum is not reproduced properly at any point on the extended clous source.
\label{section2.3}\subsection{Feathering}
Feathering is the technique that is conventionally used to combine images from single-dish instruments and interferometers. A brief description of the feathering technique is presented below.
\\*\\* The composite image formed by feathering two images is obtained by computing a weighted sum in the Fourier domain and inverting the modified data to create the image. The weight given to the interferometer image is $(1-\mathcal{F}(P^{antenna}))$, where $\mathcal{F}$ is the Fourier transform and $P^{antenna}$ is the antenna pattern defined in \hyperref[section1.1]{(Section 1.1)}. The Fourier transform of the single-dish image is multiplied by the volume ratio of the interferometer \textit{restoring beam} to the single dish antenna pattern \hyperref[ref6]{[6]}. Mathematically, this amounts to :
\begin{equation}
\mathcal{F}(\textbf{I}^{feathered}) = \frac{w_s V_s + w_I V_I}{w_s+w_I}
\end{equation}
here, $V_s$ and $V_I$ are the Fourier transforms of the single-dish and interferometer data respectively. Generally, the single-dish data is also pre-scaled by a gain parameter.
\subparagraph{Limitations of the feathering algorithm}
The feathering algorithm suffers from a few limitations, including:
\begin{enumerate}
\item It depends strongly on the single-dish gain parameter which is used to pre-scale the single-dish data before beginning the calculation. This may lead to situations where the data is scaled too much and does not reflect the physical reality.
\item It does not allow manipulation of the visibility data from the raw data file, as it takes the final images as its input which have already been deconvolved and processed. Hence one cannot handle any potential artifacts which may crop up.
\item It is highly sensitive to noise levels in the data.
\end{enumerate}
\label{AlgoDevpt}\subsection{Putting everything together : Creation of the Algorithm}
To overcome the problems detailed in \hyperref[section2.2]{Section 2.2} and \hyperref[section2.3]{Section 2.3}, a new algorithm is needed which is developed in this project. The implementation of the algorithm uses tools from the \texttt{scipy}\footnote{\texttt{https://scipy.org/}}, \texttt{numpy\footnote{\texttt{https://www.numpy.org}}} and \texttt{CASA}\footnote{CASA stands for Common Astronomy Software Application. Webpage : \texttt{https://casaguides.nrao.edu/index.php/Main\_Page}} software packages. All the programming was done in the \texttt{python} programming language, and all plots were created using either the \texttt{CASAviewer} tool or the \texttt{matplotlib} plotting package.
\subparagraph{Conventional (Cube) Radio Imaging from Interferometer Data\newline}

For data from a pure interferometer, the algorithm followed for imaging is as follows \hyperref[ref10]{[10]}: \\\\
 \textbf{Setup of the imaging procedure}
\begin{enumerate}
\item The \textit{uv cell size} in the image is decided. The size of a $uv$ - cell is decided by the width of the \textit{point spread function}\footnote{hereafter PSF}, which is the inverse Fourier transform of the $uv$ - sampling function $\textbf{[S]}$ \hyperref[section2.2]{(Section 2.2)}. Usually the cell size is chosen so that the main lobe of the PSF is Nyquist-sampled.
\item The visibility data is irregularly sampled - Hence, we need to first \textit{regrid} the image along a regular grid so that the Fast Fourier Transform (FFT) algorithm will work.
\item Some visibility points are sampled more than others - hence, \textit{weight} these visibilities appropriately to ensure that information from all the points is included in the image.
\end{enumerate}
\textbf{Joint Imaging and Deconvolution Procedure}
\begin{itemize}
\item \textbf{Major Cycle} :
\begin{enumerate}
\item The raw data is regridded and weighted appropriately.
\item The weighted, gridded data is inverse Fourier transformed to form the \textit{dirty image}.
\end{enumerate}
\item \textbf{Minor Cycle} :
\begin{enumerate}
\item The \textit{dirty image} is passed to a \textit{reconstruction} algorithm such as Hogb\"{o}m CLEAN/MTMFS \hyperref[ref2]{[2]}\hyperref[ref3]{[3]}\hyperref[urvthesis]{(Rau 2011, PhD thesis)}\hyperref[ref10]{[10]}.
\item After a number of iterations, the algorithm checks for convergence as specified in the input of the program. If the number of iterations has been reached, the cleaned image is written out.
\end{enumerate}
\item \textbf{After the Minor Cycle}
\begin{enumerate}
\item The cleaned image (the \textit{model image}) is Fourier-transformed back onto the $(u,v)$ plane.
\item This $(u,v)$ plot is \textit{degridded} and saved as a \textit{model} visibility data, which is subtracted from the original visibility file.
\item The program then \textit{\textbf{loops back to the major cycle}} using this modified visibility data. The model is updated by adding the output from each execution of the minor cycle. This looping is continued till the desired threshold limit is reached.
\end{enumerate}
\end{itemize}
The algorithm developed in this project differs in a few major ways from the strategy outlined above due to the following reasons :
\begin{enumerate}
\item Feathering does not accept multi-frequency data gridded on to the same plane, because the antenna power pattern changes with frequency. Hence, it is necessary to compute the Taylor series of Equation (5) at all the values of the frequencies present in the data, and include a header containing the antenna beam or power pattern information has to be supplied to the feathering routine, implemented in \texttt{CASA} as the \texttt{feather()} task.
\item The subtraction of the model visibilities from the available data set in the Major cycle needs separate data and not the multi-frequency one. Hence, the MTMFS deconvolver has to be accordingly modified to output the multiple model images at multiple frequencies.
\item The MTMFS imager needs multiple PSFs as part of the deconvolution process. These need to be calculated during the run of the imager setup.
\end{enumerate}
\subparagraph{Conceptual differences between the narrow-band MTMFS algorithm and the new Wideband Multiterm Combination Algorithm}
\begin{enumerate}
\item The Multiterm Wideband Combination Algorithm developed in this project does a joint deconvolution, namely, that it feathers both the single-dish data as well as the PSF with the corresponding interferometer ones. In case of a single-dish telescope, the PSF is simply the antenna beam or the power pattern.
\item In the first run of the setup, the gridded interferometer residual is first feathered with the single-dish data, then passed on to carry out the deconvolution.
\item After the run of each minor cycle, the multiterm images are converted back to the images at all the observing frequencies as a model (hereafter called a "cube" of images) .Thereafter, the residual contribution of the single dish image is calculated by smoothing these "model" images with their respective beams at their respective frequencies.
\item After the run of each major cycle, the single-dish residual images are recalculated. These are then again feathered with the interferometer residuals and written out.
\item These composite residual images are converted back into the multiterm residuals and the cycle is begun afresh.
\end{enumerate}
\textbf{\underline{A detailed description of the algorithm is presented as follows}} :
\begin{itemize}
\item The setup is broken into two parts: The setup for the 'Cube' image and the setup for the 'Multiterm' image.
\item The 'Cube' setup is run first - This creates the initial cube PSFs and the initial cube of residual images.
\item The 'MTMFS' setup is run - This creates the multiterm PSFs and the initial multiterm residuals.
\item \textbf{Initial Setup consists of the following steps}:
\begin{enumerate}
\item The PSFs from the interferometer and the single-dish image are feathered together, followed by the feathering of the single dish and interferometer residual. All the images in this step are the ones generated as 'cube' images.
\item The multiterm PSFs are then calculated from the Cube PSFs, and written out to the appropriate number of files.
\item The multiterm residuals are then calculated in the same way as the PSFs and are written out to the appropriate number of files.
\end{enumerate}
\item \textit{The imaging and deconvolution part}:\\
\textbf{While the imager has not converged}:
\begin{enumerate}
\item Run the multiterm Minor cycle using the PSFs calculated in the initial setup and the residual from the previous iteration. For the first iteration, use the residual calculated in the initial setup.
\item Once the multiterm minor cycle is done, convert the multiterm model to a cube model and write it out to a file.
\item With this model, run the next (Cube) Major cycle.
\item Calculate the single-dish residual using the smoothed model output above and feather it with the interferometer residual cube.
\item Convert this feathered cube residual into its multiterm equivalent and proceed to the next iteration.
\end{enumerate}
\end{itemize}
\subparagraph{Salient features of this algorithm}
\begin{enumerate}
\item Note that, the minor cycle employs the multiterm deconvolution strategy whereas the major cycle uses a cube Gridding/FFT path.
\item Note that, while feathering, all the residuals are always in cube form.
\item Note that, all the cube data which were used during feathering has to be converted back into its multiterm form before being passed to the minor cycle.
\end{enumerate}
\subsection{What this algorithm accomplishes}
A run of this algorithm produces the following outputs :
\begin{itemize}
\item A number of images equal to the number of terms in the Taylor series is produced.
\item Residual images and model images equaling the number of terms in the Taylor series are produced by the multiterm algorithm, plus a cube residual and model produced by the cube portion of the major cycle.
\item A spectral index map is produced by the multiterm portion of the algorithm, which shows the spectral index of the sky for each pixel in the image. Another image that holds the errors for the index estimates per pixel is also created.
\item For $n$ terms, this algorithm creates $2n-1$ multiterm PSFs and one cube PSF.
\end{itemize}
\subparagraph{Significance of each output}
\begin{enumerate}
\item The images produced as output by this algorithm are the intensities $\textbf{I}^{sky}_t$ in Equation(5).
\item The residual images are equivalent to the iterations of the Fourier transform of the visibility matrix $\textbf{V}^{obs}$ in Equation (10).
\item The spectral index map is the solution of Equation (6). The maps are basically the pixel-wise estimate of the terms $\textbf{I}_\alpha$ and $\textbf{I}_\beta$.
\item The spectral PSFs are the inverse Fourier transform of the sampling matrix $[\textbf{S}]$.
\end{enumerate}
For the sky shown in \figurename{ 1}, the output images are shown in \figurename{ 6} (Larger images are presented in the \hyperref[section3]{Results} section).
\begin{figure}
\centering
\begin{minipage}[b]{.5\textwidth}
\includegraphics[width=\linewidth]{Extended_Flat_image_tt0}
\end{minipage}%
\begin{minipage}[b]{.55\linewidth}
\includegraphics[width=\textwidth]{Extended_Flat_image_tt1}
\end{minipage}
\caption{The reconstructed multiterm images for the sky shown in \hyperref[ 1]{Figure 1}}
% % give image credits to NRAO and NCRA.
\label{7}
\end{figure}

\begin{figure}
\begin{minipage}[b]{.5\textwidth}
\includegraphics[width=\linewidth]{Extended_flat_trueim_withindices}
\end{minipage}%
\begin{minipage}[b]{.5\linewidth}
\includegraphics[width=\textwidth]{Extended_spectral_trueim_withindices}
\end{minipage}
\caption{Two different simulated "sky images" with different spectral indices}
\label{ 8}
\end{figure}
\section{Testing the algorithm : Results and Plots}
This new algorithm was tested on two different test cases, both of which derive from \hyperref[ 1]{Figure 1} : \label{section3}
\begin{enumerate}
\item A \textit{flat-spectrum} sky, where the sky does not change with frequency. In this case, the plot of the spectrum must show a value of zero for every pixel
\item A \textit{steep-spectrum} sky, where each object shown in \hyperref[ 1]{Figure 1} has a nonzero spectral index. The spectral indices are indicated on the true sky image.
\end{enumerate}
Both these "true skies" are shown in \hyperref[ 7]{Figure 7}.

\subsection{A Flat-Spectrum Sky}
Here, the reconstructed images from the Flat-spectrum sky are shown (true sky image shown in \hyperref[7]{Figure 7-left panel})
 \begin{figure}[H]
\centering
\includegraphics[width=0.5\linewidth]{Extended_Flat_image_tt0}
\caption{The zero-order Image of the Flat-spectrum sky}
\label{fig:Extended_Flat_image_tt0}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.6\linewidth]{Extended_Flat_image_tt1}
\caption{The $1^{st}$ order Image of the flat-spectrum sky}
\label{fig:Extended_Flat_image_tt1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Extended_Flat_alpha}
\caption{The pixel-wise spectral map of the flat-spectrum sky}
\label{fig:Extended_Flat_alpha}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.65\linewidth]{Extended_Flat_alpha_error}
\caption{Map showing error on the spectrum estimate for Flat-spectrum sky}
\label{fig:Extended_Flat_alpha_error}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.65\linewidth]{Extended_Flat_residual_tt0}
\caption{The zero-order residual image for Flat-spectrum sky}
\label{fig:Extended_Flat_residual_tt0}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Extended_Flat_residual_tt1}
\caption{The first-order residual for the Flat-spectrum sky}
\label{fig:Extended_Flat_residual_tt1}
\end{figure}
\subsection{A Steep-Spectrum Sky}
Here, the reconstructed images from the Steep-spectrum sky are shown (true sky image shown in \hyperref[7]{Figure 7-right panel})
\begin{figure}[H]
\centering
\includegraphics[width=0.6\linewidth]{Extended_Spectral_image_tt0}
\caption{The zero-order image of the steep-spectrum sky}
\label{fig:Extended_Spectral_image_tt0}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Extended_Spectral_image_tt1}
\caption{The first-order image of the steep-spectrum sky}
\label{fig:Extended_Spectral_image_tt1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.65\linewidth]{Extended_Spectral_alpha}
\caption{The pixel-wise spectral map of the steep-spectrum sky}
\label{fig:Extended_Spectral_alpha}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Extended_Spectral_alpha_error}
\caption{Map showing error on the spectrum estimate for Steep-spectrum sky}
\label{fig:Extended_Spectral_alpha_error}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.65\linewidth]{Extended_Spectral_residual_tt0}
\caption{The zero-order residual image of the steep-spectrum sky}
\label{fig:Extended_Spectral_residual_tt0}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{Extended_Spectral_residual_tt1}
\caption{The first-order residual image for a steep-spectrum sky}
\label{fig:Extended_Spectral_residual_tt1}
\end{figure}
\subsection{Comments and Discussion on the Results}
From the figures presented in the preceding section, it is seen that the algorithm works well for the given data set.
\begin{enumerate}
\item The algorithm for estimating the pixel-by-pixel spectral maps (figures 9 and 15) work well only at the centre of the extended cloud. This is because the spectrum in each pixel is the ratio of the terms $\textbf{I}_0$ and $\textbf{I}_1$ (Equation 5). Behaviour of these functions around the edges can be unpredictable, causing the estimate to go wrong.
\item The deconvolver has to run only within a narrow range with respect to the number of iterations, else there will either be insufficient convergence (if the number of iterations is too small) or artifacts in the spectral map (if the number of iterations is too large).
\item The single dish gain (pre-scaling) parameter also determines the performance of the algorithm, outside of a range of values. If the value is given to be too small, it can actually suppress the interferometer $(u,v)$ data near the $(0,0)$ region and cause the feathered data to become nearly zero in value. If its value is too large ($>\sim 15$), it will wash out the interferometer data and make the PSF too broad for a proper deconvolution to occur.
\end{enumerate}
\section{Summary of the Project}
The project aimed to achieve the following goals :
\begin{enumerate}
\item Solve the short-spacing problem encountered by interferometers.
\item To combine the wideband multi-frequency data from a single-dish radio telescope and interferometer.
\item To evaluate the efficacy of a joint-deconvolution algorithm where the single-dish data was inserted into the interferometer data \textit{before} the creation of an image, as compared to the traditional approach of a \textit{post-facto} combination.
\item To estimate the spectrum of extended sources by creating a wideband image.
\item To prototype and test the working of an algorithm in order to accomplish these tasks.
\item To try this algorithm on real data from an actual astronomical observation.
\end{enumerate}
All the objectives except the last one were achieved in the project. The last one can be achieved through a fairly simple extension of this work, which will be tried as and when the data from both these telescopes becomes available.
\section{Knowledge and Skills gained during this Project}
\subsection{New concepts learned during the project}
\begin{itemize}
\item \textbf{Radio Interferometry} : In-depth knowledge of the technique and mathematics of radio interferometry, including but not limited to computational implementation of 2-D Fourier transforms, Van Cittert-Zernicke theorem, $(u,v)$ sampling, synthesis imaging with radio interferometer telescopes.
\item \textbf{Radio Imaging} : Process of creation of an image from interferometer and single-dish data, role of deconvolution in synthesis imaging, multi-frequency synthesis imaging, imaging algorithms - Hogb\"{o}m CLEAN and MTMFS.
\item \textbf{Image Processing} : Concept of a pixel, image cell size, notion of linear time-invariant and linear shift-invariant systems, 2-D Fourier transforms, idea behind use of a specific kernel for detecting specific features in an image, image gridding, image weighting and its impact on the final synthesised image, combination of two images in the Fourier domain by means of a weighted average.
\item \textbf{Algorithms and Data structures} : Their role in signal/image processing, implementation of convolution and deconvolution in well-known libraries like \texttt{python's scipy} and  \texttt{numpy}, image representation as a 2-D array data structure and its manipulation in both the image domain and the Fourier domain, understanding the basic theory of the Fast-Fourier-Transform (FFT) algorithm.
\item \textbf{Optimisation and Least-Sqaures minimisation} : Understanding the condition for applicability of the least-squares optimisation method, deriving the matrix conditions for calculating the coefficients.
\end{itemize}
\subsection{Programming skills gained during this project}
\begin{itemize}
\item \textbf{Programming with} \texttt{python} : In-depth understanding and application of toolboxes in \texttt{python's scipy} and  \texttt{numpy} modules, programming in the \texttt{CASA} and \texttt{ipython} interactive IDE environment.
\item \textbf{Numerical analysis and computation with }\texttt{python} : Fourier Transforms, convolution (both in the spatial domain and Fourier-transformed space), Calculation of autocorrelation function and estimating the power spectrum from it \textit{via} the Weiner-Khinchin theorem, simulation of a simple antenna pattern and using it to create a simulated image, optimisation and least-squares minimisation - implement and demonstrate a program to fit a polynomial to a bunch of noisy data points using \texttt{python's linalg} library and comparing it with the standard tool package available in \texttt{numpy} and evaluating which is better.
\item \textbf{Data Structures with }\texttt{python} : Handling of images in python, image cubes as a stack of multiple arrays, python's \texttt{copy} module, shallow copies and deep copies in python, different kinds of variable bindings in python (bind-to-variable name \textit{vs.} creating a separate data structure), handling of python's \texttt{dictionary} data type, creating a new data structure using python.
\item \textbf{Graphics and plotting with }\texttt{python} : Use of \texttt{matplotlib} plotting package from python, plotting of 1-D, 2-D and 3-D plots using this package, interactive plotting with \texttt{matplotlib's ion()} command, plotting images and intensity maps.
\item \textbf{Familiarisation of the }\texttt{CASA}\textbf{ program}: use of \texttt{CASA's tclean()} task, image plotting with \texttt{CASA} and understanding how it implements imaging and deconvolution algorithms.
\end{itemize}
\subsection{Practical Field Training}
\begin{itemize}
\item \textbf{Training} : Visiting the site of the Very Large Array (VLA) radio telescope, understanding the functioning of the antennas and the associated hardware as well as software backends that go with it.
\item \textbf{Public Outreach Programmes} : Give two guided tours of the VLA to the public on its open day, explaining all the salient features of the facility.
\end{itemize}
\section{Acknowledgements}
This project was conducted under the National Radio Astronomy's Summer Student Research Assistantship programme, under the guidance of Dr. Urvashi Rau Venkata. This project also fulfills the compulsory summer training/internship requirements, mandated as part of the B. Tech (Hons.) course in Instrumentation Engineering, under the Dept. of Electrical Engineering, Indian Institute of Technology Kharagpur. \\Many thanks to Sanjay Bhatnagar and Kumar Golap for their help in resolving several technical issues. The author wishes to acknowlegde the great amount of help provided by Amy Mioduszewski in her role as the Coordinator of the Summer Student Program at the NRAO. The National Radio Astronomy Observatory is a facility of the National Science
Foundation operated under cooperative agreement by Associated Universities,
Inc.
\section{References and Bibliography}
\begin{enumerate}
\label{ref1}\item Wikipedia webpage on radio telescopes : \texttt{https://en.wikipedia.org/wiki/Radio\_telescope}
\label{ref2} \item "Low Frequency Radio Astronomy", \textit{ed.} Jayaram N. Chengalur, Yashwant Gupta and K.S. Dwarakanath, National Centre for Radio Astrophysics, May 2003.
\label{ref3} \item "Interferometry and Synthesis in Radio Astronomy", Thompson \textit{et. al}, 3$^{rd}$ ed., Springer International Publishing , June 2016.
\label{ref4}\item  Wikipedia webpage on interferometry : \texttt{https://en.wikipedia.org/wiki/Interferometry}
\label{ref5}\item "Statistical Optics", by M. Born and E. Wolf, Dover Publications, 1959.
\label{ref6}\item NRAO's CASAguides webpage on feathering : \\ \texttt{https://casa.nrao.edu/Release3.4.0/docs/userman/UserManse30.html}
\label{urvpaper}\item "A multi-scale multi-frequency deconvolution algorithm
for synthesis imaging in radio interferometry", U. Rau and T. J. Cornwell, \textit{Astronomy and Astrophysics} 532, A71 (June 2011).
\label{urvthesis}\item "Parameterized Deconvolution For Wide-Band Radio Synthesis Imaging", Urvashi Rau Venkata, Ph.D Thesis, New Mexico Institute of Mining and Technology (2011)
\label{ref9} \item CASA guide on the MTMFS deconvolver and \texttt{tclean()} task : \\
\texttt{https://casaguides.nrao.edu/index.php/TCLEAN\_and\_ALMA}
\label{ref10}\item Talk on Wideband Imaging and Deconvolution : \\ \texttt{https://science.nrao.edu/science/meetings/2014/\\14th-synthesis-imaging-workshop/lectures-files/SIW2014\_WidebandImaging\_UR.pdf}
\end{enumerate}
\section{Appendix}
\subsection{Further extension of this work}
This work dealt with combination of data from both the interferometer as well as the single-dish instrument. As an important spin-off, the code can be modified to do the whole procedure for either the pure single-dish image or the pure interferometer image for handling cube images (multiterm imaging algorithms exist for pure interferometers). As a brief example of this, the plots (the two images, spectrum and spectral error) from the single dish-only data is presented below :
\\
\begin{figure}[H]
\centering
\includegraphics[width=0.7\linewidth]{singledishonly_image_tt0}
\caption{The zero-order image of the sky using only the single-dish data in the Wideband Joint deconvolution algorithm}
\label{fig:singledishonly_image_tt0}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.6\linewidth]{singledishonly_image_tt1}
\caption{The first-order image of the sky using only the single-dish data in the Wideband Joint deconvolution algorithm}
\label{fig:singledishonly_image_tt1}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.6\linewidth]{singledishonly_alpha}
\caption{Pixel-wise spectrum map estimated by the algorithm, using only single-dish data.}
\label{fig:singledishonly_alpha}
\end{figure}
\begin{figure}[H]
\centering
\includegraphics[width=0.6\linewidth]{singledishonly_alphaerror}
\caption{Pixel-wise map showing error in the spectrum estimate using only single-dish data}
\label{fig:SD_alpha_error}
\end{figure}
As can be seen, this code works very well for the pure single-dish data as well for the combination in terms of estimating the spectrum. As expected, the resolution on the objects seen in the plot is very coarse as compared to the case where the interferometer visibilities were also present. This combination algorithm will prove useful to the ALMA\footnote{The Atacama Large Millimeter Array} and possibly the Square Kilometer Array (SKA) radio telescope which will come up in the near future.
\subsection{Location of the Source Code and Data Use}
The source code of this entire project (along with a few sample data sets) is open for viewing can be accessed online, upon request to either of the two persons : (The whole project will be released online at a future date)
\begin{itemize}
\item Dr. Urvashi Rau Venkata, Associate Scientist, CS, National Radio Astronomy Observatory, 1003 Lopezville Rd., Socorro, New Mexico 87801.\\\textbf{email} : \texttt{rurvashi@aoc.nrao.edu}
\item Nikhil Vijay Naik, $4^{th}$ year undergraduate student, Department of Electrical Engineering, Indian Institute of Technology, Kharagpur, West Bengal, India - 721302.\\\textbf{email} : \texttt{nikhilnaik@iitkgp.ac.in}
\end{itemize}
\end{document}


\section{Conclusion and Discussion}

In this lab, we use Stokes’ method to measure fluid viscosity.
The liquid been chosen to measure the viscosity is  the castor oil in this
experiment.

We measures Distance $S$, Time $t$, Diameter of the ball $d$, Inner diameter of
the flask $D$, Density of one ball $\rho_2$, then calculate the viscosity
coefficient not directly.

The relative uncertainty of each physical quantity is quite small, thus our
measurement is very precise and the result is acceptable.

The reason for the accuracy is the nice design of the experiment and carefulness
throughout the experiment.
There are many systematical and random errors leading to the main uncertainty,
but a lot of methods are used to reduce them.

% NOTE: bonus Q
We use 40 balls to calculate the average mass of the ball.
For each ball's mass is quite small and very hard to measure due to the
uncertainty of equipment. We choose 40 instead of 10 to maintain 4 significant
numbers.

% NOTE: bonus Q
Actually the temperature has influence on $\mu$.
The higher temperature is, the more likely the molecule will move, making the
viscosity lower.

My experiment setup is closer two the air conditioner than other people, so my
result may be slightly affect by the temperature.

% NOTE: theoretical
From a engineering website

\url{http://www.engineeringtoolbox.com/absolute-viscosity-liquids-d_1259.html}
We know that the theoretical value of the viscosity of caster oil is  $0.650
Pa/s$, but the experiment result is $0.7307Pa/s$.
$$ u_{\mu,r} = (0.7307 - 0.650)/ 0.650 \times 100\%  =  12.42 \%   $$
The reason is the AC mentioned before cooling down the temperature of my
experiment setup.
We can see here lower temperature results in higher viscosity.

% NOTE: concerns
Furthermore, there are more problems that may cause uncertainty and error in this lab.
\begin{enumerate}
\item The distance between two laser device may not be measured precisely for
  the thread is not strictly perpendicular to the level.
\item When measuring the time, human's reponse may cause some delay between
  seeing the light fading and pressing down the button on the watch.
\end{enumerate}

Some ways to improve this lab
\begin{enumerate}
\item Adjust the experimental setup carefully to the level.
\item Throw the ball in the center of the cylinder and in the same position
  every time.
\item Try hard to reduce response time.
\end{enumerate}


\documentclass[12pt]{amsart}
% The srcltx package helps with inverse dvi search
%\usepackage{srcltx}

% This package has an easy way to set the margins
\usepackage[paper=letterpaper]{geometry}
\geometry{top=1in,left=1in,right=1in,bottom=1in,headheight=14pt}

% These packages include nice commands from AMS-LaTeX
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{amssymb,amsmath,amsthm}
\usepackage{bm}
\usepackage{booktabs}
\usepackage{cancel}
\usepackage{caption}
\usepackage{changepage}
\usepackage{color}
\usepackage{comment}
\usepackage{empheq}
\usepackage[shortlabels]{enumitem}
\usepackage[mathscr]{euscript}
\usepackage{float}
\usepackage{graphicx}
\usepackage{hyperref}
%\usepackage[linktoc=none]{hyperref}
\usepackage{listings}
\usepackage{lscape}
\usepackage{multimedia}
\usepackage{setspace}
\usepackage{subfig}
\usepackage{tabularx}
\usepackage{textcomp}
\usepackage{tikz}
\usetikzlibrary{patterns}
\usetikzlibrary{scopes}
\usetikzlibrary{decorations.pathmorphing}
\usetikzlibrary{shapes,backgrounds, arrows}
\usepackage{xcolor}

%This package helps make nice headers
\usepackage{fancyhdr}
\pagestyle{fancy}
\lhead{\textsf{Note}}
\chead{\textsf{Jared McBride}}
\rhead{\textsf{\today}}
\renewcommand{\headrulewidth}{1pt}

% Make the space between lines slightly more
% generous than normal single spacing, but compensate
% so that the spacing between rows of matrices still
% looks normal.  Note that 1.1=1/.9090909...
\renewcommand{\baselinestretch}{1.1}
\renewcommand{\arraystretch}{1.1}

\renewcommand{\and}{\qquad\text{and}\qquad}
\newcommand{\ds}{\displaystyle}

\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}

\newcommand\Fontvi{\fontsize{8}{7.2}\selectfont}
\newcommand{\vs}{\vspace{2 cm}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\p}{\mathbb{P}}
\newcommand{\T}{\mathbb{T}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\C}{\mathbb{C}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\F}{\mathscr{F}}
\newcommand{\scrE}{\mathscr{E}}
\newcommand{\spa}{\text{span}}
\newcommand{\rref}{\mathrm{rref}}

\newcommand{\X}{\mathfrak{X}}
\newcommand{\ssX}{\bm{\textsf{X}}}
\newcommand{\ssx}{\bm{\textsf{x}}}
\newcommand{\fS}{\mathfrak{S}}
\newcommand{\fC}{\mathfrak{C}}

\renewcommand{\b}[1]{{\color{blue} #1}}
\usepackage[T1]{fontenc}
\usepackage{beramono}
\usepackage{listings}

\renewcommand{\and}{\qquad\text{and}\qquad }

\renewcommand{\P}{\mathbb{P}}
\newcommand{\cov}{\mathrm{cov}}
\newcommand{\var}{\mathrm{var}}
\newcommand{\abox}[1]
{
	\begin{center}
		\fbox{$\displaystyle #1$}
	\end{center}
}
%%
%% Julia definition (c) 2014 Jubobs
%%
\lstdefinelanguage{Julia}%
{morekeywords={abstract,break,case,catch,const,continue,do,else,elseif,end,export,false,for,function,immutable,import,importall,if,in,macro,module,otherwise,quote,return,switch,true,try,type,typealias,using,while},%
	sensitive=true,%
	alsoother={$},%
	morecomment=[l]\#,%
	morecomment=[n]{\#=}{=\#},%
	morestring=[s]{"}{"},%
	morestring=[m]{'}{'},%
}[keywords,comments,strings]%

\lstset{%
	language         = Julia,
	basicstyle       = \ttfamily,
	keywordstyle     = \bfseries\color{blue},
	stringstyle      = \color{magenta},
	commentstyle     = \color{olive},
	showstringspaces = false,
}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}

\title{Numerical Spectral Factorization by Kalman Recursion}
\author{Jared A. McBride}
\date{\today}

\doublespacing

\begin{document}

\begin{abstract}
	In this note I describe a method presented by Kailath et al. in \cite{sayed2001} and \cite{kailath2000} for finding a spectral factor of the $z$-spectrum of a (wide-sense stationary) stochastic process under certain conditions. The method employs a version of Kalman filtering. As such I begin with a summary of relevant results from Kalman filtering. With those results in hand I present the method for finding spectral factors and conclude with examples.
\end{abstract}

\maketitle
\tableofcontents

\section{Introduction}

Given a wide-sense stationary (WSS) stochastic process $X = \big(X_n, n \in \Z\big)$ it's $z$-spectrum $S_X(z)$ is defined by\footnote{
	A note on notation: the superscript $*$ denotes the complex conjugate transpose (Hermitian transpose) operator for matrices and just the complex conjugate for real or complex numbers. The superscripts $-*$ together is an abbreviation for the complex conjugate of the multiplicative inverse, i.e. $$z^{-*} = \frac{1}{z^*}.$$
	}
$$S_X(z) = \sum_{n=-\infty}^\infty C_X[n]z^{-n},$$
where $C_X[n] = \E X_nX_0^*$ is the autocovariance sequence of the process $X$. This function is real-valued on the unit circle since, $C_X[-n] = C_X^*[n]$, and by the Wiener-Khinchin theorem, it is, in fact, nonnegative definite on the unit circle\footnote{
	Though what follows holds for scalar processes it also holds for vector valued processes. As such, I assume the vector case. Meaning, the covariance sequence is a sequence of square $d\times d$-matrices, where $d$ is the dimension of $X_0$.}
Spectral factorization refers to a particular factorization in which
$$S_X(z) = L(z)L^*(z^{-*})$$
where $L(z)$ is \emph{minimum phase}, meaning both $L(z)$ and $L^{-1}(z) = 1/L(z)$ are analytic on and outside the unit circle. The function $L$ will be referred to as a spectral factor of $S_X$. This factorization is not unique since for any spectral factor of $S_X$ right-multiplication by a unitary matrix produces another spectral factor of $S_X$.

In this note I describe a method presented by Kailath et al. in \cite{sayed2001} and \cite[p.~336]{kailath2000} for finding a spectral factor under certain conditions. The method employs a version of Kalman filtering, more specifically the Kalman recursion for finding the innovations process for a process with a finite state-space model. The innovations for such a process will be defined below. Section \ref{sec: Kalman} provides a summary of relevant results from Kalman filtering, together with some intuition relevant to the present context. With those results in hand, Section \ref{sec: Factor} presents the method for finding spectral factors. Then, in Section \ref{sec: Example} are examples.

\section{Kalman Filtering}
\label{sec: Kalman}

The purpose of this section is to provide information from Kalman filtering theory that is pertinent to the development of the factorization algorithm to be discussed. To that end, the \textit{summum bonum} of the section is the computation of what I refer to as a \emph{weak} modeling filter for a given process. The associated \emph{weak} whitening filter also bears significance in this work and will be briefly treated.

Suppose we have a (discrete-time) stochastic process\footnote{
	From here till Section \ref{sec: Kalman stationary} $X$ need not be WSS.
}
$X = \big(X_i,\; i =  0,1,2,\dots\big)$ with a finite state-space representation of the following form:
\begin{subequations}
	\label{equ: ss}
	\begin{empheq}[left=\empheqlbrace]{align}
		\label{equ: ss State}\theta_{i+1} &= F_i\theta_i + G_iu_i \\
		\label{equ: ss Obs}X_i &= H_i\theta_i + v_i
	\end{empheq}
	where $F_i \in C^{m\times m},$ $G_i \in C^{m\times p},$ and $H_i \in C^{d\times m}$ are known matrices, and $u=(u_i)$, $v=(v_i)$, and $\theta_0$ are random variables with the following property\footnote{
		Throughout the note $$\delta_{ij} = \begin{cases} 1 &\text{if }i = j\\ 0 &\text{if }i \ne j.\end{cases}$$
		}
	\begin{equation}
	\E\begin{pmatrix} \theta_0 \\u_i \\ v_i \end{pmatrix}
	\begin{pmatrix} \theta_0 \\u_j \\ v_j\\1 \end{pmatrix}^*
	=\begin{pmatrix}
	\Pi_0 & 0                & 0              & 0 \\
	0     & Q_i\delta_{ij}   & S_i\delta_{ij} & 0 \\
	0     & S^*_i\delta_{ij} & R_i\delta_{ij} & 0
	\end{pmatrix}
	\label{equ: cond}
	\end{equation}
\end{subequations}

In the context here, I think of $X$ as a given series of\textit{ observations}, they are directly measurable, and $\theta = \big(\theta_i,\; i = 0,1,2,\dots\big)$ is the \textit{state} of the process (usually not directly measurable). Both of these series are sequences of random variables, along with $u$ and $v$. The matrices $F_i$, $G_i$, $H_i$, $\Pi_0$, $Q_i$, $S_i$ and $R_i$ are deterministic for all $i$.

\subsection{A few words about the mathematical context}
In the sequel there will be much disccusion on  linear least-mean-square estimators
A nice thing about least-mean-square estimators is an orthogonality principle, namely the estimator error is orthogonal to the predictor. What follows also affords such a principle. However, there is no Hilbert space that comfortable houses this theory. So, I will build up the context here.

The context requires the iterations of vectors of more than one size, since the observations and the state need not be the same dimension. Since it is possible to write solutions the the problems I consider in terms of covariance information a natural ``inner product'' may be defined by
\begin{equation}
\langle x,y \rangle = \E xy^*\qquad \text{for all }x\in \C^d,y\in \C^m, \qquad d,m \in \N,
\label{def: inner p}
\end{equation}
So, that $\langle x,x \rangle \in \C^{d\times d}$ is the covariance of $x$ and $\langle x,y \rangle \in \C^{d\times m}$ is the cross covariance of $x$ and $y$. We use the notation $x \perp y$ if $x$ and $y$ are uncorrelated, meaning $\langle x,x \rangle = 0 \in \C^{d\times m}$.

Because this inner product\footnote{
	If its arguments are restricted to the same space, It can be shown that $\langle \cdot, \cdot\rangle$ satisfies the axioms of an inner product, provided positivity is replaced with positive defitiveness.}
is matrix-valued, rather than adding structure to a linear space with a scalar field the underlining space must be a module of vectors, with a ring of (square) matrix-valued scalars.The natural choice of scalars are matrices, of the appropriate size. This is an important point and leads me to another definition.
Let $$\text{span}_d\{y_i,\;i=0,1,\dots,N\} := \left\{\sum_{i=0}^N A_iy_i ~\bigg|~ y_i\in \C^m,A_i \in \C^{d\times m},\quad i=0,1,\dots,N\right\}$$
I would like us to think of the span as possiblily set in a different space from the arguments.

There is a norm
\begin{equation}
\|x\|=\sqrt{\langle x,x \rangle}
\label{def: norm}
\end{equation} associated with this inner product, defined in the usual way, using the unique (Hermitian) positive-semidefinite square root of the Hermitian positive-semidefinite matrix $\langle x,x \rangle$.

\subsection{the Kalman filter}
Now, the Kalman filter allows us to find, for some $i>0$, the linear least-mean-squares estimator of the state $\theta_{i}$ given all preceding observations $X_j$ for $j = 0,1,2,\dots, i-1$.
This quantity I will denote by $\hat{\theta}_{i|i-1}$. Since in this note I will only be concerned with linear least-mean-squares estimators given the $X_i$'s the subscript $\cdot_{|i}$ will denote linear estimators with respect to and conditioned on all $X_j$ for $j \le i$.
Consistent with this convention, when I write $\hat{X}_{i|i-1}$ I mean the linear least-mean-squares estimator of $X_i$ given $X_j$ for $j < i$.

For the stake of rigor and completeness I have provided lemma which describes the orthogonality principle in this context. For the sake of brevity I have placed the proof in Appendix \ref{app: Ortho proof}.

\begin{lemma}
	\label{lem: otho p}
	Given the inner product defined under (\ref{def: inner p}) and the norm induced by it, under (\ref{def: norm}), let $x\in C^d$ and $y_i \in \C^m$ for $i=0,\dots,N$ be random variables. If $\hat x$ is the linear least-mean-squares estimator of $x$ given $y$, meaning
	$$\E(x-\hat x)(x-\hat x)^* = \|x-\hat x\|^2 = \inf_{x'\in \text{\emph{span}}_d\{y_i,\;\forall i\}} \|x-x'\|^2$$
	then $(x-\hat x) \perp y_i$ for $i=0,1,\dots,N$.
\end{lemma}

We seek a least-mean-square estimator of $\theta_{i+1}$ redistricted to a linear combination of observations, $\{X_j,\;j < i\}$. In this context we seek $\hat\theta_{i+1|i}$ such that,
\begin{align}
	\hat\theta_{i+1|i} &\in\mathcal{L}_i =: \text{span}\{X_i,\;i=0,1,\dots,N\}\\
	\label{equ: cond proj}\hat\theta_{i+1|i} &= \displaystyle\argmin_{\hat\theta \in \mathcal{L}_i}\|\theta_{i+1} - \hat\theta\|^2
\end{align}
Borrowing from Hilbert space theory, we consider building an orthogonal basis for $\mathcal{L}_i$, $e = \{e_j,\;j = 1,2,\dots,i\}$ and write
\begin{equation}
\hat{\theta}_{i+1|i} = \sum_{j=0}^i \langle \theta_{i+1}, e_j \rangle R^{-1}_{e,j} e_j\label{equ: projection}
\end{equation}
where $R_{e,i} = \langle e_i, e_i \rangle$. We are not in a Hilbert space, so the claim that the estimator under (\ref{equ: projection}) satisfies (\ref{equ: cond proj}) will need to be verified. But first, let us build this orthogonal set in a way similar to the Grahm-Schmidt process, but without normalizing. Put
$$e_i = X_i - \hat{X}_{i|i-1}\quad \text{for }i>0\qquad\text{and}\qquad e_0 = X_0$$
The $\{e_i\}$ have the property $\mathcal{L}_i = \text{span}\{e_j,~j\le i\}=\mathcal{L}_i$ for any $i\ge 0$, and $\langle e_i,e_j \rangle = R_{e,i}\delta_{ij}$ by Lemma \ref{lem: otho p}.

\vspace{3cm}

This stochastic process we will refer to as the \textit{innovations} associated with $X$. Observe that because they are defined as the residual of least squares estimates over a progressively increasing space  the innovations are an orthogonal set. And we having $\langle e_i,e_j \rangle = I\delta_{ij}$. There are other orthogonality relations I would generally like to highlight. First noting that $\theta_i \in \text{span}\{\theta_0, u_j,\; j<i\}$, then by use of (\ref{equ: ss Obs}) and the definition of $e_i$ one can show the following.
\begin{subequations}
	\label{equ: inner p}
\begin{align}
\langle \theta_i, u_j \rangle = 0 &\and \langle \theta_i, v_j \rangle = 0 \qquad\;\;\;\text{for}\qquad i\le j;\\
\label{equ: inner p b}\langle X_i, u_j \rangle = 0 &\and \langle X_i, v_j \rangle = 0 \qquad\;\;\text{for}\qquad i < j;\\
\langle X_i, u_j \rangle = S^*_i &\and \langle X_i, v_j \rangle = R_i \qquad\text{for}\qquad i = j;\\
\langle e_i, u_j \rangle = 0 &\and \langle e_i, v_j \rangle = 0 \qquad\;\;\;\text{for}\qquad i < j;\\
\langle e_i, u_j \rangle = S^*_i &\and \langle e_i, v_j \rangle = R_i \qquad\;\text{for}\qquad i = j.
\end{align}
\end{subequations}

\subsection{Kalman filter recursions for the innovations}

Kalman developed a recursion to cheaply find the innovations process associated with the observations, which we will derive here.
I like the way Kialath put it, ``It turns out that the recursive construction of the innovations combines nicely with the recursive evolution of the state variables to give a recursion for the innovations...''\cite[p.~312]{kailath2000}.

We seek a recursion for $\{e_i\}$ but start with finding a recursion for $\hat{\theta}_{i|i-1}$ using (\ref{equ: projection}),
\begin{align*}
\hat{\theta}_{i+1|i} &= \sum_{j=0}^i \langle \theta_{i+1}, e_j \rangle R^{-1}_{e,j} e_j\\
&= \left(\sum_{j=0}^{i-1} \langle \theta_{i+1}, e_j \rangle R^{-1}_{e,j} e_j\right) + \langle \theta_{i+1}, e_i \rangle R^{-1}_{e,i} e_i\\
& = \hat{\theta}_{i+1|i-1} + \langle \theta_{i+1}, e_i \rangle R^{-1}_{e,i} e_i
\end{align*}
The first term of the last line undermines the recursion in $\hat{\theta}_{j|j-1}$; however, we can remedy this by conditioning equation (\ref{equ: ss State}) on $\mathcal{E}_{i-1}$ to get
$$\hat{\theta}_{i+1|i-1} = F_i\hat{\theta}_{i|i-1} +G_i\hat{u}_{i|i-1} = F_i\hat{\theta}_{i|i-1} + 0$$
by (\ref{equ: inner p b}). This provides
$$\hat{\theta}_{i+1|i} = F_i\hat{\theta}_{i|i-1} + \langle \theta_{i+1}, e_i \rangle R^{-1}_{e,i} e_i$$
The next thing to do is to make sure we can compute $e_i$ from the $\hat{\theta}_{i|i-1}$'s. If we condition equation (\ref{equ: ss Obs}) on $\mathcal{E}_{i-1}$ then
\begin{equation}
\hat{X}_{i|i-1} = H_i\hat{\theta}_{i|i-1} + \hat{v}_{i|i-1} = H_i\hat{\theta}_{i|i-1} + 0\label{equ: X hat}
\end{equation}
also by (\ref{equ: inner p b}). So, that
$$e_i = X_i - \hat{X}_{i|i-1} = X_i - H_i\hat{\theta}_{i|i-1}.$$
From this point I will use the abbreviated notation $\hat{\theta}_i = \hat{\theta}_{i|i-1}$.

Putting this together, we get a recursion for the innovations
\begin{subequations}
	\begin{empheq}[left=\empheqlbrace]{align}
		\label{equ: theta hat} \hat{\theta}_{i+1} &= F_i\hat{\theta}_{i} + K_ie_i, \quad \hat{\theta}_0 = 0\\
		\label{equ: innov} e_i &= X_i - H_i \hat{\theta}_{i}.
	\end{empheq}
\end{subequations}
where $K_i = \langle \theta_{i+1}, e_i \rangle R^{-1}_{e,i}$.

Here then is the recursion we sought, which recursively generates the innovations associated with the process $X$. For our interests, and for many others, it would be useful if the $K_i$'s and the $R_{e,i}$'s could be computed off-line, that is, independent of the observations. As it turns out this is possible and I will demonstrate how this may be accomplished.

Let $\tilde{\theta}_i = \theta_i - \hat{\theta}_{i}$ and $P_{i} = \langle \tilde{\theta}_{i},\tilde{\theta}_{i} \rangle$. It turns out that this quantity is very useful in formulating an off-line recursion. It is easy to see by subtracting (\ref{equ: X hat}) from (\ref{equ: ss Obs}) that $e_i = H_i\tilde{\theta}_{i} - v_i$, and this is enough to put $R_{e,i}$ in terms of $P_i$,
because $\hat{\theta}_{i}$ is orthogonal to $v_i$ by (\ref{equ: inner p b}), since $\hat{\theta}_{i}\in \mathfrak{E}_{i-1}$\footnote{
	This means $\hat{\theta}$ is measurable with respect to $\mathfrak{E}_{i-1}$, which is independent of $v_i$.}
, we get
$$R_{e,i} = H_iP_iH^*_i + R_i.$$
To get $K_i$ in terms of only $P_i$ and $R_{e,i}$, we start with $K_i = \langle \theta_{i+1}, e_i \rangle R_{e,i}^{-1}$ and observe that by (\ref{equ: ss Obs}) $\langle X_{i+1}, e_i \rangle = F_i\langle X_{i}, e_i \rangle  + G_i\langle u_i, e_i \rangle$. Reflecting on both inner products in turn provides
\begin{align*}
\langle X_{i}, e_i \rangle &= \langle X_{i}, \tilde{X}_i \rangle H^*_i + \langle X_{i}, v_i \rangle \\
&= \langle \hat{X}_{i} + \tilde{X}_i, \tilde{X}_i \rangle H^*_i + 0 \\
&= P_i H^*_i
\end{align*}
and $\langle u_{i}, e_i \rangle = S_i$, as we have seen. Substituting, we able to find that
$$K_i = (F_iP_iH_i^* + G_iS_i) R_{e,i}^{-1},$$
as desired.
Now, with $R_{e,i}$ and $K_i$ in place a recursion for $P_i$ can be established by
combining equations (\ref{equ: ss State}) and (\ref{equ: theta hat}) and taking the covariance of both sides, a somewhat long calculation results in the discrete Lyapunov recursion,
\begin{align}
\label{equ: DL} P_{i+1} &= F_iP_{i}F_i^* + G_iQ_iG_i^* - K_iR_{e,i}K^*_i,\qquad i\ge 0
\end{align}
which we initialize by $$P_{0} = \langle \hat{\theta}_{0}, \hat{\theta}_{0} \rangle = \langle \theta_0, \theta_0 \rangle = \Pi_0.$$

We have just have just found a recursion for the $K_i$ and $R_{e,i}$, which are necessary in the computation of $e_i$, which is independent of observations. It also provides a recursion for the error covariance of the state estimate (which is useful in its own right).  The discussion so far can be summarized into a single theorem
\begin{theorem}
	\label{thm: innov}
	Consider the standard statespace model
	$$\left\{\begin{array}{ r l}
	\theta_{i+1} &= F_i\theta_i + G_iu_i \\
	X_i &= H_i\theta_i + v_i
	\end{array}\right. $$
	for $i\ge 0$, and satisfying (\ref{equ: cond}). The innovations process of $X$ can be recursively computed using the equations
	\begin{subequations}
		\label{equ: innovations}
		\begin{empheq}[left=\empheqlbrace]{align}
			e_i &= X_i - H_i\hat{\theta}_i, \qquad i \ge 0,\\
			\hat{\theta}_{i+1} &= F_i\hat{\theta}_i + K_{p,i}e_i,\qquad \hat{\theta}_0 = 0,
		\end{empheq}
	\end{subequations}
	where
	\begin{subequations}
		\label{equ: Kal}
		\begin{align}
			\label{equ: Kal Pi}  P_{i+1} &= F_iP_{i}F_i^* + G_iQ_iG_i^* - K_iR_{e,i}K^*_i,\qquad P_0 = \Pi_0 \\
			\label{equ: Kal Rei} R_{e,i} &= H_iP_iH^*_i + R_i \\
			\label{equ: Kal Ki}  K_i &= (F_iP_iH_i^* + G_iS_i) R_{e,i}^{-1}
		\end{align}
	\end{subequations}
	from (\ref{equ: cond}). Here, $P_i = \langle \tilde{\theta}_i,\tilde{\theta}_i \rangle$ where $\tilde{\theta}_i = \theta_i - \hat{\theta}_i$.
\end{theorem}

Observe that the discrete Lyanov recursion (\ref{equ: DL}) can now be written as a discrete Riccati recursion,
\begin{equation}
	P_{i+1} = F_iP_{i}F_i^* + G_iQ_iG_i^* - (F_iP_iH_i^* + G_iS_i)(H_iP_iH^*_i + R_i)^{-1}(H_iP_iF_i^* + S^*_iG^*_i).
	\label{equ: recatti recursion}
\end{equation}

\subsection{The Stationary Case and the Modeling Filter}
\label{sec: Kalman stationary}

So far in this section, all we have assumed about $X$ is that it is a discrete-time stochastic process that has a finite state-space representation defined by (\ref{equ: ss}). But now it is time to assume conditions native to the problem at hand, which will be delineated shortly. These assumptions are afforded by the stationarity of the processes I work with.  The plan is to realize stationarity and adjust the innovations process so as to produce both stationary innovations and stationary state estimators.

Here are the further assumptions I impose on the process $X$,
\begin{itemize}
	\item the state-space representation for $X$ is time-invariant, meaning, all the parameters are constant in time. So,
	$$F_i = F,\quad G_i=G, \quad H_i = H,\quad Q_i=Q, \quad S_i=S \quad\text{and}\quad R_i = R\qquad \text{for }i\ge 0$$
	additionally I impose that $F$ be stable (meaning its eigenvalues lie strict in the unit circle) and $R$ be strictly positive-definite, $R>0$, also
	\item $\theta_i$ is in  steady state, meaning $\Pi_i = \Pi$ for $i\ge 0$ where is $\Pi$ is the positive-semi-definite solution to the Lyaponov equation.
	\begin{equation}
	\Pi = F\Pi F^* + GQG^*.
	\label{equ: state Lyapunov}
	\end{equation}
	 Since $F$ is stable this has a unique solution.
\end{itemize}

A consequence of these assumptions in that $X$ is wide-sense stationary, and it's model can be written as
\begin{subequations}
	\label{equ: ss ss}
	\begin{empheq}[left=\empheqlbrace]{align}
	\label{equ: ss ss State}\theta_{i+1} &= F\theta_i + Gu_i \\
	\label{equ: ss ss Obs}X_i &= H\theta_i + v_i
	\end{empheq}
	where
	\begin{equation}
	\E\begin{pmatrix} \theta_0 \\u_i \\ v_i \end{pmatrix}
	\begin{pmatrix} \theta_0 \\u_j \\ v_j\\1 \end{pmatrix}^*
	=\begin{pmatrix}
	\Pi & 0                & 0              & 0 \\
	0     & Q\delta_{ij}   & S\delta_{ij} & 0 \\
	0     & S^*\delta_{ij} & R\delta_{ij} & 0
	\end{pmatrix}
	\label{equ: cond ss}
	\end{equation}
\end{subequations}
It can be show by direct computation that covariance function of $X$, $R_X(i-j)$, can be written as follows \cite[p.~267]{kailath2000}
\begin{equation}
R_X(i-j) = \begin{cases}
	HF^{i-j-1}N & i>j, \\
	R + H\Pi H^* & i = j, \\
	N^*(F^*)^{j-i-1}H^* & i < j,
\end{cases}
\label{equ: autocov X}
\end{equation}
where $N= F\Pi H^* + GS$.

Under these conditions we now apply Theorem \ref{thm: innov} and find that the innovations sequence associated with $X$ has the following state-space representation
\begin{subequations}
	\label{equ: innovations stationary}
	\begin{align}
	e_i &= X_i - H\hat{\theta}_i, \qquad i \ge 0,\\
	\hat{\theta}_{i+1} &= F\hat{\theta}_{i} + K_{p,i}e_i,\qquad \hat{\theta}_0 = 0,
	\end{align}
\end{subequations}
	where now
	\begin{subequations}
	\label{equ: Kal stationary}
	\begin{align}
	\label{equ: Kal stationary Pi}  P_{i+1} &= FP_{i}F^* + GQG^* - K_iR_{e,i}K^*_i,\qquad P_0 = \Pi \\
	\label{equ: Kal stationary Rei} R_{e,i} &= HP_iH^* + R \\
	\label{equ: Kal stationary Ki}  K_i &= (FP_iH^* + GS) R_{e,i}^{-1}
	\end{align}
\end{subequations}

Rearranging (\ref{equ: innovations stationary}) slightly we can write
\begin{subequations}
	\label{equ: model}
	\begin{empheq}[left=\empheqlbrace]{align}
		\hat{\theta}_{i+1} &= F\hat{\theta}_{i} + K_{p,i}e_i,\qquad \hat{\theta}_0 = 0,\\
		X_i &= H\hat{\theta}_i + e_i, \qquad i \ge 0.
	\end{empheq}
\end{subequations}
Which provides an alternative way of modeling the process $X$, this time driven by the innovations process $e$. Said differently (\ref{equ: model}) defines a model $\mathcal{L}$ which maps $e$ to $X$.

Observe that $\mathcal{L}$, clearly causal, is also causally invertable since by algebra we can write,
\begin{subequations}
	\label{equ: white}
	\begin{empheq}[left=\empheqlbrace]{align}
		\hat{\theta}_{i+1} &= F_p\hat{\theta}_{i} + K_{p,i}X_i,\qquad \hat{\theta}_0 = 0,\\
		e_i &= - H\hat{\theta}_i + X_i, \qquad i \ge 0.
	\end{empheq}
\end{subequations}
This contrasts with the original model which took $(\theta_0, u,v)$ to $X$\footnote{
	One thing it has done was remove the additional input variable, the initial state}
, such is causally but not generally causally invertible.

Now, what can be said about the output $\mathcal{L}w$, for $w = (w_i,\; i = 0, 1, 2, \dots)$ a noise sequence with $\langle w_i, w_j \rangle = R_{e,i}\delta_{ij}$? Is it a stationary process with the same $z$-spectrum as $X$? The output in general is not stationary, since the model, found under (\ref{equ: model}), represents a time-variant system. This begs the question, does this model converge, in time, to a time-invariant model? It turns out that the system parameters $P_i$, $R_{e,i}$, and $K_i$ all converge as $i\rightarrow \infty$. Which suggests a time-invariant causal system that, when fed in a stationary input, will certainly produce a stationary output. A \textit{modeling filter} for $X$ is causal, linear system which constructs the process $X$ by passing a white-noise process through the filter. This is what we are seeking.

The convergence of $P_i$ has been well studied \cite[Sec.~14.5]{kailath2000} and there are many conditions one can check to ensure that the limit of the recursion converges to the stabilizing solution of the appropriate Racatti equation necessary to provide the causal and casually invertible modeling filter, namily the discrete algebraic Racatti equation (DARE), given by
\begin{align}
P &= FPF^* + GQG^* - (FPH^* + GS)(HPH^* + R)^{-1}(HPF^* + S^*G^*)\\
 &= FPF^* + GQG^* - KR_eK^*
\label{equ: recatti equation}
\end{align}
Since this theory is more problem dependent I will defer the matter of convergence to when we have a specific problem.

So, assume that
$$\lim_{i\rightarrow \infty} P_i = P$$
So that $P$ in the unique solution to
\begin{equation}
	P = FPF^* + GQG^* - KR_eK^*
	\label{equ: racatti}
\end{equation}
where $\lim R_{e,i} = R_e = HPH+R$ and $\lim K_i =  K = (FPH^* + GS)R^{-1}_e$, and that $P$ is such that $F_p = F - KH$ is stable. Let us, then, consider the system $$L: w \mapsto Y $$ where $w = (w_i,\; i \ge -\infty)$ with $ \langle w_i,w_j \rangle = R_e\delta_{ij}$ and $Y = (Y_i,\; i \ge -\infty)$\footnote{Notice the slight-of-hand, we now start our processes in the remote past and have them be bilaterally infinite. This is to obviate the need for the state variable to be initialized as an input in order for them to have a variance that obeys the Lyapunov equation below for each finite time} given by
\begin{subequations}
	\label{equ: modeling}
	\begin{empheq}[left=\empheqlbrace]{align}
		\eta_{i+1} &= F\eta_{i} + Kw_i,\qquad \eta_{-\infty} = 0\\
		Y_i &= H\eta_i+ w_i, \qquad i \ge -\infty.
	\end{empheq}
Since $\eta_i$ was initialized in the remote past (independent of $w_i$) we have $\langle \eta_i, w_i \rangle = 0$ for $i> -\infty$ and $\langle \eta_i, \eta_i \rangle = \Sigma$ for $i> -\infty$
where
\begin{align}
\Sigma =F\Sigma F^* +KR_eK^*.
\end{align}
\end{subequations}

We want to check that the autocovariance of $\{Y_i\}$ is the same as for $\{X_i\}$ which is given in (\ref{equ: autocov X}). This can be checked directly. For $i>0$
\begin{align*}
R_Y[i] &= \langle Y_i, Y_0 \rangle \\
	&= \langle H\eta_i+w_i, H\eta_0 + w_0 \rangle \\
	&= \langle HF\eta_{i-1}+HKw_{i-1} + w_i, H\eta_0 + w_0 \rangle \\
	&= \langle HF^2\eta_{i-2}+HFKw_{i-2} + HKw_{i-1} + w_i, H\eta_0 + w_0 \rangle \\
	&= \langle HF^i\eta_{0}+HF^{i-1}Kw_{0} + \dots + HFKw_{i-2} + HKw_{i-1} + w_i, H\eta_0 + w_0 \rangle \\
	&= \langle HF^i\eta_{0}, H\eta_0\rangle + \langle HF^{i-1}Kw_{0} + \dots + HFKw_{i-2} + HKw_{i-1} + w_i, H\eta_0 \rangle \\
	&\qquad +\langle HF^i\eta_{0}, w_0\rangle + \langle HF^{i-1}Kw_{0} + \dots + HFKw_{i-2} + HKw_{i-1} + w_i, w_0 \rangle\\
	&= HF^i\Sigma H^* + 0 + 0 + HF^{i-1}KR_e\\
	&= HF^i\Sigma H^* + HF^{i-1}(FPH^* + GS)\\
	&= HF^{i-1}\big(F(\Sigma + P)H^* + GS)\big),
\end{align*}
a similar calculation for $i<0$ yields
\begin{align*}
R_Y[i] = \big(F(\Sigma + P)H^* + GS)\big)^*(F^*)^{i-1}H^* = R^*_Y[-i],
\end{align*}
and for $i=0$
\begin{align*}
	 R_Y(0) &= \langle Y_0, Y_0 \rangle \\
	 &= \langle H\eta_0+w_0, H\eta_0 + w_0 \rangle \\
	 &= H\Sigma H^* + R_e \\
	 &= H\Sigma H^* + HPH^* + R\\
	 &= H(\Sigma + P)H^* + R
\end{align*}
Now what is $\Sigma + P$, by there definitions
\begin{align*}
\Sigma + P &= F\Sigma F^* + KR_eK^* + FPF^* + GQG^* - KR_eK^*\\
&= F(\Sigma + P)F^* + GQG^*.
\end{align*}
Meaning that $\Sigma + P$ solves the same discrete Lyapunov equation that $\Pi$ solved in (\ref{equ: state Lyapunov}). We therefore conclude $$\Pi = \Sigma + P$$ and that
\begin{equation}
R_Y[i] = \begin{cases}
HF^{i-1}N & i>0, \\
R + H\Pi H^* & i = 0, \\
N^*(F^*)^{-i-1}H^* & i < 0,
\end{cases} \qquad= R_X[i]
\label{equ: autocov Y}
\end{equation}
recalling that  $N= F\Pi H^* + GS$, as in equation (\ref{equ: autocov X}). This demonstrates that the causal LTI system $L$ provides a model filter for produce process with the same autocovariance sequence as $X$ given any white noise sequence with covariance $R_e\delta_{ij}$.

Furthermore, observe that $L$ (given by (\ref{equ: modeling})) is causal and causally invertible, since, by simple algebra again, $L^{-1}$ can be written as
\begin{subequations}
	\label{equ: whitening}
	\begin{empheq}[left=\empheqlbrace]{align}
	\eta_{i+1} &= F_p\eta_{i} + KY_i,\qquad \eta_{-\infty} = 0,\\
	w_i &= - H\eta_i + Y_i, \qquad i \ge -\infty.
	\end{empheq}
	with $\langle \eta_0, Y_i \rangle = 0$ for $i\ge 0$ and now $\eta_0$ is chosen such that $\langle \eta_0, \eta_0 \rangle = \Sigma_p$
	where
	\begin{align}
	\Sigma_p =F_p\Sigma_p F_p^* +KR_Y[0]K^*
	\end{align}
\end{subequations}
where $F_p = F - KH$ which is stable since $P$ was a stabilizing solution to the Recatti equation given under (\ref{equ: racatti}). This then a whitening filter since.

I now summarize the above discussion into a theorem
\begin{theorem}
	\label{thm: model}
	Let $X = (X_i;\;i > 0)$ be a stationary process with a time-invariant state-space representation of the form given in (\ref{equ: ss ss}). Then, the LTI system $L$ given by (\ref{equ: modeling}) is causal and causally invertable, and constructs a stationary process with the same autocovariance as $X$ by passing in any white-noise process with variance $R_e$.
\end{theorem}

\section{Spectral Factorization by Kalman Filtering}
\label{sec: Factor}

In this section I explain how the above results from Kalman filtering may be employed in finding an approximate spectral factor of a given power spectrum ($z$-spectrum).

Given a $z$-spectrum $S_X(z)$ of a WSS discrete-time process $X_n\in \C^d$ for $n>-\infty$, by definition, $$S_X(z) = \sum_{n=-\infty}^{\infty} R_X[n]z^{-n},$$
where $$R_X[n] = \cov(X_{k+n},X_{k}) = \E X_{k+n}X_{k}^* = \langle X_{k+n},X_{k}\rangle$$ is the covariance. Now, if the decay of the covariance is sufficiently fast it is reasonable to truncate $S_X(z)$ to a Laurent polynomial
$$\tilde{S}_X(z) = \sum_{n=-m}^{m} R_X[n]z^{-n}.$$
However, this isn't always the best way to approximate the spectrum. It is usual to apply some windowing function to the autocovariances. The Parzen windowing function $\Lambda_m$, given below, for some $m>0$ is an example.
$$\Lambda_m[n] = \begin{cases}
1-6\left(\frac{n}{m}\right)^2+6\left(\frac{|n|}{m}\right)^3, & |n|\le m/2 \\
2\left(1-\frac{|n|}{m}\right)^3, & m/2 < |n| \le m \\
0, & |n| > m
\end{cases}.$$
The Fourier transform $\widehat{\lambda}_m(\omega)$ of the Parzen windowing function is
$$\widehat{\Lambda}_m(\omega) = \frac{3}{4}m\left(\frac{\sin \pi \omega m/2}{\pi \omega m/2}\right)^4,\qquad -\infty \le \omega \le \infty$$
Let $R_\text{win}[n] = \Lambda_m[n]R_X[n]$ and
$$S_\text{win}(z) = \sum_{n=-m}^{m} R_\text{win}[n]z^{-n}.$$
The function $S_\text{win}(z)$ is generally a better approximation of $S_{X}(z)$ than $\tilde{S}_{X}(z).$

Now that we have approximated the $z$-spectrum we seek to factor by a Laurent polynomial, we can construct a process $Y = \big(Y_i,\; i = 0,1,2, \dots)$ with a finite, time-invariant, state-space model whose $z$-spectrum is exactly $S_\text{win}(z)$, that is $S_{Y}(z) = S_\text{win}(z)$ \cite[p.~488]{sayed2001}. We then take the state-space model and using Kalman, compute the modeling filter.

The process $Y$ is constructed using the following finite, time-invariant, state-space model:
\begin{subequations}
	\label{equ: lti ss Y}
	\begin{empheq}[left = \empheqlbrace]{align}
		\theta_{i+1} &= F\theta_i + Gv_i \\
		Y_i &= H\theta_i + u_i
	\end{empheq}
	Where\footnote{
		Recall $d$ is the dimension of the observation vectors and $m$ is the degree of the Laurent polynomial approximation of the given $z$-spectrum}
	\begin{equation}
		\E \begin{pmatrix} \theta_0 \\ v_i \\ u_i \end{pmatrix}\begin{pmatrix}\theta_0 \\ v_j \\ u_j \\ 1 \end{pmatrix}^* =
		\begin{pmatrix} \Pi & 0 & 0 & 0\\
		0 & R\delta_{ij} & S\delta_{ij} & 0 \\
		0 & S^*\delta_{ij} & Q\delta_{ij} & 0\end{pmatrix}.
	\end{equation}
\end{subequations}
The parameters $F$, $H$, and $N$ (think of the $N$ from equation (\ref{equ: autocov X}), are defined explicitly by
$$
F = \begin{pmatrix}
0 &   &        &        &   \\
I & 0 &        &        &   \\
& I & 0      &        &   \\
&   & \ddots & \ddots &   \\
&   &        & I      & 0
\end{pmatrix}\in \C^{md\times md},$$
$$
H = \begin{pmatrix}
0 & \dots  & 0       & I
\end{pmatrix}\in \C^{d\times md},
$$
and
$$N = \begin{pmatrix} R_\text{win}[m] \\ R_\text{win}[m-1] \\ \vdots \\ R_\text{win}[1] \end{pmatrix}\in \C^{md\times d}.$$
The remaining quantities are less important almost auxiliary. With the exception of $\Pi$, they have no bearing on the covariance data of the process (recall equation (\ref{equ: autocov X})). The existence of $\Pi$ is guaranteed by the fact that $F$ is stable. It's value never needs to be determined, as we will see. For completeness if one were interested in actually realizing the process $Y$ fro this statespace form the remaining variables must satisfy the following conditions
$$\Pi = F\Pi F^* + GQG^*$$
$$GS = N - F\Pi H^*$$
$$R = R_\text{win}[0] - H\Pi H^*$$

Now, provided of all the above can be satisfied it can be show that $Y$ is stationary and its autocovariance is
\begin{align*}
	C_{Y}[i] &= \begin{cases}
		HF^{i-1}N & i > 0 \\
		H\Pi H^* + R & i = 0 \\
		NF^{*(-i-1)}H^* & i < 0
	\end{cases} \\
	&=\begin{cases}
		0 & i > m \\
		R_\text{win}[i]& m \ge i \ge -m \\
		0 & i < -m
	\end{cases}
\end{align*}
So, that $S_Y(z) = S_{\text{win}}(z) \approx S_X(z)$ as desired.

In $Y$, we have a stationary process with a time-invariant state-space representation, given under (\ref{equ: lti ss Y}). This representation induces We now invoke Theorem \ref{thm: model}, and assert that the system $M: w = (w_i,\;i> -\infty) \mapsto Y' = (Y'_i,\; i>-\infty)$ given by
\begin{subequations}
	\label{equ: model Y}
	\begin{empheq}[left=\empheqlbrace]{align}
	\eta_{i+1} &= F\eta_{i} + Kw_i,\qquad \eta_{-\infty} = 0\\
	Y'_i &= H\eta_i+ w_i, \qquad i \ge -\infty,
	\end{empheq}
\end{subequations}
is causal and causally invertible and constructs a stationary process $Y'$ with the same autocovariance of $Y$ by passing in any white-noise process with variance $R_e$. This system $M$ constitutes a modeling filter for $Y$.
Observe that from System (\ref{equ: model Y}) it can be shown that
$$Y'_i = \sum_{j=1}^{m} HF^{j-1}Kw_{i-j} + w_i = \sum_{j = 0}^{m} K^{\#}_iw_{i-j}$$
where $$K^{\#}_i = \begin{cases}
K_{m-i+1} & 1\le i \le m\\
I & i = 0\end{cases}$$
The notation views $K \in C^{d\times md}$ as a column vector with matrix entries, $(d\times d)$-blocks. So in words, when $K\in C^{d\times md}$ is viewed as a block column vector with $d\times d$-blocks, $K^\#$ is the block-reverse of $K$ appened on top by the $d\times d$ Identity matrix.

And so defines an $MA(m)$ (model) process.

\section{An Example}
\label{sec: Example}

\appendix
\section{Orthogonality Principle}
\label{app: Ortho proof}

\section{CKMS}

CKMS, named for Chandrasekhar, Kailath, Morf, and Sidhu who in various ways and times contributed to its developement, is a efficient algorithm for computing the Kalman filter recursion for innovations. We will begin by observing who the Kalman filter may be used to find the spectral factors of a rational power spectrum (and approximate that of a nonrational power spectrum) and then from there give details about the CKMS recurssion.

\bibliographystyle{unsrt}
\bibliography{../Notes}
\end{document}