\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\include{pythonlisting}
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage[ruled,vlined]{algorithm2e}

\usepackage{hyperref}

\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{url}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}

\title{Exploring numerical calculations with CalcNet}

\author{\IEEEauthorblockN{Ashish Rana}
\IEEEauthorblockA{ Thapar Institute of Engg. and Tech. India\\
arana\_be15@thapar.edu}
\and
\IEEEauthorblockN{Avleen Malhi}
\IEEEauthorblockA{Aalto University Finland\\
avleen.malhi@aalto.fi}
}

\maketitle

\begin{abstract}

Neural networks are not great generalizers outside their training range i.e. they are good at capturing bias but might miss the overall concept. Important issues with neural networks is that when testing data goes outside training range they fail to predict accurate results. Hence, they loose the ability to generalize a concept. For systematic numeric exploration neural accumulators (NAC) and neural arithmetic logic unit(NALU) are proposed which performs excellent for simple arithmetic operations. But, major limitation with these units is that they can't handle complex mathematical operations \& equations. For example, NALU can predict accurate results for multiplication operation but not for factorial function which is essentially composition of multiplication operations only. It is unable to comprehend pattern behind an expression when composition of operations are involved.
Hence, we propose a new neural network structure effectively which takes in complex compositional mathematical operations and produces best possible results with small NALU based neural networks as its pluggable modules which evaluates these expression at unitary level in a bottom-up manner. We call this effective neural network as CalcNet, as it helps in predicting accurate calculations for complex numerical expressions even for values that are out of training range. As part of our study we applied this network on numerically approximating complex equations, evaluating biquadratic equations and tested reusability of these modules. We arrived at far better generalizations for complex arithmetic extrapolation tasks as compare to both only NALU layer based neural networks and simple feed forward neural networks. Also, we achieved even better results for our golden ratio based modified NAC and NALU structures for both interpolating and extrapolating tasks in all evaluation experiments.Finally, from reusability standpoint this model
demonstrate strong invariance for making predictions on different tasks.

\end{abstract}

\section{Introduction}

Generalization of concepts is the fundamental component while defining intelligence in a system  \cite{b1} \cite{b2}. While, neural networks can be applied on numerical quantities they are not associated with generalization capabilities \cite{b3, b4}. We define generalization capabilities as the ability to understand complex numerical expressions that identifies compositional patterns.
Meaning that a neural network should understand all the operations in an expression and predict high accuracy results for unseen and out of training range data.
Failure to understand these numeric generalizations is clearly visible in Table \ref{tab:table1} when test range lies outside the training range even for simpler target functions. This highlights hard memory commit behavior of the biases learnt by neural networks instead of learning abstract concepts. For mitigating this problem of memorizing behavior in neural networks, special numerically biased modules known as Neural accumulators (NAC) and neural arithmetic logic units (NALU) \cite{b5} were introduced. These units have performed exceptionally well for generalization on simple numerical operators and have achieved state-of-the-art results. But, again these units fail to make generalizations for complex mathematical equations as these units only learns the simple mathematical operations only with their weights being biased specially towards -1, 0, 1. In Figure \ref{fig1} and its weight matrices analysis, we see addition and subtraction being modelled with such numerical biases. Only meaning, biased values around -1, 0, 1 gets better results for numeric computations. Also, Figure \ref{fig2} highlights this biased nature for NAC units which helps this unit to model these functions and for NALU also, scaling up and down mechanisms for multiplication and division operations is highlighted ever increasing approximate curve specified for larger values in Figure \ref{fig2}.

\begin{figure}[!h]
\centering
\includegraphics[width=0.18\textwidth]{_assets/NeuralMLP.png}
\caption{Simple MLP with W$_1$, W$_2$ weight matrices with Y as output for X$_1$, X$_2$ as inputs. For modelling addition and subtraction operations follow along with below mentioned W$_1$, W$_2$ values. No bias vector \textit{b} assumed for this MLP and only linear activations considered.}
\label{fig1}
\end{figure}

\begin{itemize}
\item For modelling addition operation Y=X$_1$+X$_2$, W$_1$ and W$_2$ can have possible following arrangement of values.
\[
\begin{bmatrix}
    1  &  0      \\
    1  &  1      \\
    0  &  1
\end{bmatrix}
,
\begin{bmatrix}
    1        \\
    0  		 \\
    1
\end{bmatrix}
\]

\item For modelling subtraction operation Y=X$_1$-X$_2$, W$_1$ and W$_2$ can again be created with only [-1, 0, 1] values with slightly  different arrangement. Hence, restricting weights to these values added numerical bias which is better for arithmetic calculation related tasks.
\[
\begin{bmatrix}
    1  &  0      \\
    1  &  1      \\
    0  &  1
\end{bmatrix}
,
\begin{bmatrix}
    1        \\
    0  		 \\
    -1
\end{bmatrix}
\]

\end{itemize}

\begin{figure}[!h]
\centering
\includegraphics[width=0.50\textwidth]{_assets/NAC&NALU.png}
\caption{\textbf{\textit{Left: }}NAC is biased with learning [1, 0, 1] values i.e. it either accumulate or deduct instead of scaling, perfect for addition \& subtraction operations.
\textbf{\textit{Right: }}Approximate curve for NALU focuses on scaling up and down nature along with superimposition of [1, 0, 1] bias from NAC which is relatively better for multiply, divison \& power functions. }
\label{fig2}
\end{figure}

NALU units can't handle negative targets as exponential space only gives positive values which adds some limitations to it, shown in Figure \ref{fig2}. Along, with that these units can't breakdown common complex functions like factorial, exponential etc. which are essentially formed from repeated multiplication operations. Hence, we need an architecture that can comprehend such functions and breakdown their complexity into simplest most unary or binary operations along with considering sign of targets.

In this paper, we introduce a new neural network architecture pipeline which is effectively feed forward in nature that can perform complex calculations by utilizing NAC and NALU MLPs as its fundamental unit and a parser algorithm for breaking down numeric calculations into their lower most granularity operations. It achieves state of the art results for numerical computations and manipulations for complex arithmetic expressions with our golden ratio based NAC and NALU units. Hence, creating neural networks that truly understands numerical manipulations which can have feasible practical applications. Like, quantum mechanics equations involving uncertainity because of duality in particles for example, heisenberg uncertainty equation that giving approximated position or momentum of a particle. The proposed network can simulate this approximation behavior with the formulated algorithm. Finally, we applied and tested our effective neural network architecture on biquadratic formula equations. This experiment also tested robustness and invariance of this network as trained models from different numerical equation's data were reutilized.

\section{Related Work}

Very deep neural networks have performed extremely well for computer vision tasks \cite{b6} and reducing computations for such networks \cite{b9}. But, basic quantitative manipulation \& reasoning is also very important. These conceptions by our brain is extensively studied task in \cite{b1, b2, b10}.

For encouraging further systematic numerical extrapolation, a new architecture known as a neural arithmetic logic unit (NALU) is proposed which helps in representing numerical quantities in the form of linear activations allowing manipulations by employing primitive arithmetic operators which are being restrained by learned gates \cite{b5}. The NALU enhanced neural networks are able to count objects in images, execute computer code, translate numerical language into real-valued scalars, perform arithmetic over images of numbers, track time. It obtains consequential generalizations within and outside the training range of the numerical values allowing magnitude orders beyond numerical training ranges compared to conventional architectures. Another neural networks based assessment method was proposed using NALU where weight determination is done through training \cite{b13}. The obtained scores from pseudo-subjective assessment are being compared with the actual values stored in the database. Normally, neural network (NN) based prediction tool's accuracy needs to be tested beyond the NN weight training set. Nonetheless, insistent residual errors were further down-scaled by employing particle swarm optimization which is NN weights post-processing for improving the accuracy. This assessment has considerable applications for long term evolution (LTE) network operators Quality-of-Experience-aware network optimization.

Neural networks have successfully learned to represent numerical quantities and manipulate them with proper learning biases. But, still this behavior doesn't highlight systematic generalizations \cite{b3, b4}. However, in nature beings and animal species have shown numerical extrapolation capabilities \cite{b10, b11} implying that numerical manipulation is ecologically advantageous. Hence, creating networks with true numerical manipulation power showcase intelligence for developing smart systems.
Our solution models and manipulate mathematical symbols with semantic sense of mathematical grammar used while parsing the numeric expression.
Currently, our approach uses precedence rules to evaluate the mathematical expressions with respect to different operators. We haven't explored training neural networks for learning precedence logic for mathematical operators used as a part of this study.

\section{CalcNet}

Neural networks have shown great deal of success from computer vision \cite{b6, b10}, text and audio analysis \cite{b7}, to reinforcement learning \cite{b8}. But, real world problems involving numerical analysis where theory and applied science merges is still quite unexplored with neural networks. Exploring networks learning numerical manipulation based on sequence learning is not the aim of this paper. Instead, our work focuses on increasing generalization capabilities i.e. better predictions by the network for extrapolating results onto unseen solution space as well for observing underlying structure of governing-equations \cite{b12}. For this we propose a simple feed forward effective neural network architecture that have the ability to parse complex mathematical expressions with NAC and NALU multilayer perceptrons(MLPs) as it fundamental units for making predictions. This network is having NAC and NALU layers which handles each operator from given complex mathematical equation separately.
The expression evaluation follows the order of precedence rule while parsing as defined in regular mathematics for these operators. After each module has evaluated the higher precedence sub-expression their output goes in as input to find output of another lower precedence order sub-expression. This process goes on till the complete expression is evaluated as highlighted in CalcNet infix expression evaluation algorithm stated below. The net error produced by this model is the summation of error produced by each of it's modules. Intuitively stating, the errors get traversed onto other neural network modules and gets accumulated producing even more wrong predictions in later stages of expression evaluation. Here, Figure \ref{fig3} highlights the workings of this effective neural architecture, explains error propagation and presents the equivalent functioning architecture of this network.
We have also introduced modifications to NAC and NALU units and have achieved even better results for simple arithmetic operations and complex equation solving tasks by replacing expontential value($\textbf{\textit{e}}$) in \textit{sigmoid} and \textit{tanh} with golden ratio($\phi$). These variants are represented as  G-NAC and G-NALU respectively where \textit{G} signifies their golden ratio base instead of an exponential one.

\begin{figure}[!h]
\centering
\includegraphics[width=0.51\textwidth]{_assets/CalcNet.png}
\caption{\textbf{\textit{Top-Left: }}NAC/NALU based MLP module for binary operator.
\textbf{\textit{Top-Right: }}NAC/NALU based MLP module for unary operator.
\textbf{\textit{Bottom-Left: }} For f(x)=a+b/c*$\sqrt{d}$-e+f\textsuperscript{2} the CalcNet algorithm  proceed in the specified manner with errors propagating in additive fashion.
\textbf{\textit{Bottom-Right: }} Shows abstract blackbox representation with any number of inputs \& operators from which output \& error information can be utilized.
 }
\label{fig3}
\end{figure}

\begin{algorithm}
\SetAlgoLined
\textbf{Inputs:} String S$_i$, with operands and operators for \textbf{f(x)}. \\
\textbf{x$_i$:} x$_1$, x$_2$,...., x$_n$, where each \textit{x$_i$} is a variable.\\
\textbf{op$_i$:} op$_1$, op$_2$,...., op$_n$, where each \textit{op$_i$} is a variable.\\
\textbf{Pre-requisites:} \\
\textbf{M\textsubscript{NAC}:} NAC modules pre-trained for each accumulating/deducting operands, \\
\textbf{M\textsubscript{NALU}:} NALU modules pre-trained for each scaling operand \\
\textbf{Initialization:} \textbf{S\textsubscript{vr}:} variable operand stack, \textbf{S\textsubscript{op}:} operator stack \\

\KwResult{Predicted Outcome for \textbf{f(x)}}

 \While{ reading(\textit{char} from S$_i$ of \textbf{f(x)}) != None}{

\uIf{\textit{char} == operand}{
    \textit{push}(S\textsubscript{vr}, \textit{char})
  }
  \uElseIf{\textit{char} == operator}{
      \While{ precedOrder(top(S\textsubscript{op})) $\ge$ precedOrder(\textit{char}) }{
	  op$_i$ = pop(S\textsubscript{op}) \\
	  v$_a$ = pop(S\textsubscript{vr}) \\
	  v$_b$ = pop(S\textsubscript{vr})  \\
      // \textit{selectModule} function select ANN NAC/NALU model based on operator \\
      t$_i$ = predictCalcNet(v$_a$, v$_b$, selectModule(op)) \\
      \textit{push}(S\textsubscript{vr}, t$_i$) \\
      }
      \textit{push}(S\textsubscript{op}, t$_i$) \\

  }
  \uElseIf{\textit{char} == '('}{

  	  \While{ \textit{char} == ')' }{

	  op$_i$ = pop(S\textsubscript{op}) \\
	  v$_a$ = pop(S\textsubscript{vr}) \\
	  v$_b$ = pop(S\textsubscript{vr})  \\
      t$_i$ = predictCalcNet(v$_a$, v$_b$, selectModule(op)) \\
      \textit{push}(S\textsubscript{vr}, t$_i$) \\

      }
  }
  \Else{
	return \textbf{\textit{Parse Error}}
  }

 }
 // When no more characters are left for reading. \\
  \While{empty(pop(S\textsubscript{op})) != false}{
      op$_i$ = pop(S\textsubscript{op}) \\
 	  v$_a$ = pop(S\textsubscript{vr}) \\
	  v$_b$ = pop(S\textsubscript{vr})  \\
      t$_i$ = predictCalcNet(v$_a$, v$_b$, selectModule(op)) \\
      \textit{push}(S\textsubscript{vr}, t$_i$) \\
  }

  \textbf{\textit{return }top(S\textsubscript{vr})}

 \caption{CalcNet's Algorithm Pipeline}
\end{algorithm}

Neural accumulators (NACs) \cite{b5} supports accumulation numerically both in additive and subtractive manner. It is special type of linear layer with transformation matrix \textbf{W} being continuous and differentiable for gradient descent algorithm. Also, experiments show they exhibit better numeric bias than linear layers \cite{b5}. W = $\tanh$($\hat{W}$)$\odot$$\sigma$($\hat{M}$)  consists of elements in [-1, 1] with bias close to  −1, 0, and 1. See Figure \ref{fig2} for ideation of this concept for following equations of NAC:  a = Wx, W = $\tanh$($\hat{W}$)$\odot$$\sigma$($\hat{M}$) where \textit{$\hat{M}$}, \textit{$\hat{W}$}, are learning parameters \& \textit{W} is transformation matrix.

Also, for mathematical operations like multiply, power and division we use neural arithmetic logic units (NALUs) \cite{b5}. As visible from Figure \ref{fig2} these units are good at scaling up and down the given provided inputs. It uses weighted sum of two sub-cells, one for addition or subtraction and another of multiply, division or power functions. These units shows that neural accumulators (NACs) when extended, can learn scaling operations with gate-controlled sub-operations. See Figure \ref{fig2} for ideation of this concept with following equations for NALU: y = g$\odot$a + (1-g)$\odot$m; m = $\exp$W($\log$( $\mid$x$\mid$ + $\epsilon$)), g = $\sigma$(Gx) where \textit{m} is subcell that operates in \textit{log} space and \textit{g} is learned gate, both contains learning parameters. These units acts as fundamental tool for our numerical quantity manipulating neural network with their flat plateau like surfaces around the values -1, 0, 1 and sharp slopes to reach these values.

Numerical analysis is vast topic with lots of fields involving convergence \& series acceleration, discretization for converting continuous functions into discrete values, interpolation \& extrapolation and optimization problems, stochastic quantum equation evaluation. Our work is applicable on all approximation problems as an alternative to evaluation of complex mathematical operations i.e. we use approximating neural networks to get stochastic estimations.

\section{Experiments}

Experiments in this paper first starts with exploring performance of golden ratio based variants of NAC \& NALU modules specifically G-NAC \& G-NALU as defined earlier. We then experiment with complex arithmetic expressions testing the numeric reasoning and extrapolation capabilities of CalcNet. Finally, we used the trained modules from previous experiment to test reusability of CalcNet by evaluating all the roots of biquadratic equations from the derived formulas.

\footnote{ Experiment codes available at given github repostory url: \url{http://github.com/ashishrana160796/calcnet-experiments}}

\subsection{Golden Ratio based variants of NAC \& NALU}

Surface plots for NAC highlights clear bias towards the value -1, 0, 1 and for NALU along with these biases scaling up \& down surfaces are also superimposed. But, these sharp slopes for converging weights towards the before mentioned values might be leading to some issues with gradient propagation during the training of model with its threshold step function like logic as per our hypothesis. As per our hypothesis we do desire such biases but with regularization on sharp threshold surfaces, see Figure \ref{fig4}. Hence, to examine this we conducted experiments with modified non-linear functions having golden ratio($\phi$) as the base value instead of standard exponent($\textbf{\textit{e}}$) base for \textit{sigmoid} and \textit{tanh} functions in equations for NAC and NALU. As for comparison with vanilla NAC unit this golden ratio based NAC has less slope for more smoother transfer and update of gradient values, similar argument can be extended for NALU units, as shown in Figure \ref{fig4}.

\begin{figure}[!h]
\centering
\includegraphics[width=0.50\textwidth]{_assets/GoldenNAC&NALU.png}
\caption{Golden Ratio($\phi$) based NAC \& NALU units with smoother slopes as compared to Vanilla NAC \& NALU units and lesser plateau saturation of values around [-1, 0, 1] unlike in Figure \ref{fig2}. We define our newly created units as G-NAC and G-NALU respectively where \textit{G} signifies their golden ratio base instead of an exponential one.}
\label{fig4}
\end{figure}

For this experiment, we altered the \textit{backend} script in keras source code and added our modified sigmoid \& tanh functions. We used \textit{nadam} optimizer and mean square error(\textit{MSE}) as an accuracy measure for a direct comparison with \cite{b5}. We trained NAC/NALU based model shown in Figure \ref{fig5} for 100 epoch each for 0.01, 0.001, 0.0001 learning rates respectively. The results achieved are shown in Table \ref{tab:table1} for multiple arithmetic operations. For training data, we have used 2\textsuperscript{16} data points from randomly generated distribution within the range [-10,10] for NAC and NALU modules. We haven't used negative target values for training NALUs as exponentiation operation always give positive results. Hence, signs are handled separately for multiplication, division and power operation and only magnitude is considered for training. CalcNet handles signs separately in its parsing phase instead of working with negative target values for producing better results. For interpolation task, the test data is within same range only and for extrapolation tasks we have tested our trained models on data values that are upto 5 times the training range.

\begin{figure}[!h]
\centering
\includegraphics[width=0.24\textwidth]{_assets/model_plot.png}
\caption{MLP model with NAC/NALU units as hidden and output layers for learning static arithmetic tasks.}
\label{fig5}
\end{figure}

\bgroup
\def\arraystretch{1.5}
\begin{table}[h!]
  \begin{center}
    \caption{\textbf{Result summary for static arithmetic tasks}}
    \label{tab:table1}
    \begin{tabular}{|c|c|c|c|c|}

    \hline
      \textbf{Tasks} & \textbf{NAC} & \textbf{NALU} & \textbf{G-NAC} & \textbf{G-NALU}\\
    \hline
      \multicolumn{5}{c}{\textbf{Interpolation Task: Mean Square Error Test}} \\
    \hline
      a+b & 3.601e-07 & 3.442e-05 & \textbf{1.591e-07} & 3.256e-05 \\
    \hline
      a-b & 9.065e-05 & 7.827e-04 &  \textbf{6.767e-05} & 4.033e-04\\
    \hline
      a*b & 5.2594 & 0.628 & 4.913 & \textbf{0.308} \\
    \hline
      a/b & 8.745 & 2.147 & 6.241 & \textbf{1.837} \\
    \hline
      a\textsuperscript{2} & 5.201 & 1.189 & 4.837 & \textbf{1.097}\\
    \hline
      $\sqrt{a}$ & 6.674 & 0.999 & 6.137 & \textbf{0.998}\\

    \hline
      \multicolumn{5}{c}{\textbf{Extrapolation Task: Mean Square Error Test}} \\
    \hline
      a+b & 7.511e-06 & 9.679e-05 & \textbf{3.764e-06} & 7.809e-05 \\
    \hline
      a-b & 1.177e-04 & 9.152e-04 & \textbf{0.523e-04} & 5.148e-04 \\
    \hline
      a*b & 10.559 & 1.327 & 7.263 & \textbf{1.093} \\
    \hline
      a/b & 11.350 & 2.181 & 7.438 & \textbf{1.977} \\
    \hline
      a\textsuperscript{2} & 19.3114 & 3.246 & 16.139 & \textbf{2.830}\\
    \hline
      \textbf{$\sqrt{a}$} & 13.342 & 2.748 & 12.794 & \textbf{1.431}\\
    \hline
    \end{tabular}
  \end{center}
\end{table}
\egroup

Our golden ratio based NAC and NALU have clearly outperformed their respective standard counterparts for every static arithmetic tasks. Golden ratio based NALU and NAC clearly highlights scope of structural improvements in existing units. Hence, our hypothesis of thresholding issues with gradient propagation for producing changes in weights does get confirmed with this experiment. Also, have used these modified units specified as G-NAC \& G-NALU for further experimentation.

\subsection{Evaluating complex expressions}

In this static arithmetic expression task we evaluated the given function \textit{f(x)} stated as, f(x)=a+b/c*$\sqrt{d}$-e+f\textsuperscript{2}. Simple MLP and also NAC/NALU based MLP won't be able to model this function for extrapolation tasks. MLP will only focus memorization of function behavior in given range and for NAC/NALU based MLP this would be too complicated numerical bias to learn. Hence, we feed this equation in to CalcNet for training where our model uses associativity and precedence rules to evaluate intermediate expressions until complete result is derived as shown below. Also, intermediate expression predictions are also stored for evaluating net ensemble error produced and relative error increase for each intermediate arithmetic operation.

\begin{itemize}
\item \textbf{Intermediate Evaluation Steps Followed:} \\
f(x) = a+b/c*$\sqrt{d}$-e+f\textsuperscript{2} \\
f(x) = a+b/c*T1-e+f\textsuperscript{2}, where T1 = $\sqrt{d}$ \\
f(x) = a+b/c*T1-e+T2, where T2 = f\textsuperscript{2} \\
f(x) = a+T3*T1-e+T2, where T3 = b/c \\
f(x) = a+T4-e+T2, where T4 = T3*T1 \\
f(x) = T5-e+T2, where T5 = a+T4 \\
f(x) = T6+T2, where T6 = T5-e \\
f(x) = T7, where T7 = T6+T2 \\

Now, for all the T\textit{i} values NAC/NALU MLP modules are trained to form an effective larger neural network which leverages associativity \& precedence grammar to form semantic relations between the intermediate expressions.
\end{itemize}

Training parameters and procedure are kept the same as earlier experiment with testing range again going upto 5 times. But, now both training and testing range are scaled by a factor of two. For training simple MLP network for comparison with CalcNet we have used linear layer as activation and for NAC/NALU based MLP same structure as Figure \ref{fig5} is used. Here, along with training the modules for each arithmetic operation they are directed to be reused for biquadratic root evaluation experiment for evaluating roots of biquadratic equation.

\begin{itemize}
\item \textbf{Relative Error:}

Relative error metric provides a quantified measure of increase in mean square error(MSE) during prediction of outputs for particular arithmetic operation with respect to net mean square eror(MSE) for predicting the outputs.

In statistics it is formulated as:\\
\[Accuracy =\frac{\text{MSE for particular arithmetic operation}}{\text{Total MSE for all arithmetic operations}}\]
\end{itemize}

\bgroup
\def\arraystretch{1.5}
\begin{table}[h!]
  \begin{center}
    \caption{\textbf{Relative Errors for CalcNet Variants}}
    \label{tab:table2}
    \begin{tabular}{|c|c|c|c|}

    \hline
      \textbf{Temporary Variable} & \textbf{Evaluated Expression} & \textbf{CalcNet} & \textbf{G-CalcNet}\\
    \hline
      \multicolumn{4}{c}{\textbf{Interpolation: Relative Error for Intermediate Steps}} \\

    \hline
	  T1  & $\sqrt{d}$ & \textbf{0.202} & 0.232 \\
	\hline
	  T2 & f\textsuperscript{2}	& \textbf{0.234} & 0.255 \\
	\hline
	  T3 & b/c & 0.435  & \textbf{0.428}\\
	\hline
	  T4 & T3*T1  & 0.127  & \textbf{0.085}\\
	\hline
	  T5 & a+T4 &  7.188e-08  & \textbf{3.695e-08}\\
	 \hline
	  T6 & T5-e  &  4.120e-06  & \textbf{4.104e-06}\\
	 \hline
	  T7 & T6+T2 &  7.204e-08  & \textbf{3.711e-08}\\
	 \hline

      \multicolumn{4}{c}{\textbf{Extrapolation: Relative Error for Intermediate Steps}} \\

    \hline
	  T1  & $\sqrt{d}$ & 0.289 & \textbf{0.195}\\
	\hline
	  T2 & f\textsuperscript{2}	& \textbf{0.341} & 0.392 \\
	\hline
	  T3 & b/c & \textbf{0.229} & 0.265 \\
	\hline
	  T4 & T3*T1  & 0.140  & \textbf{0.149}\\
	\hline
	  T5 & a+T4 &  7.891e-07 & \textbf{5.12e-07}\\
	 \hline
	  T6 & T5-e  &  1.86e-06  & \textbf{7.12e-07}\\
	 \hline
	  T7 & T6+T2 &  7.89e-07  & \textbf{5.12e-07}\\

    \hline
    \end{tabular}
  \end{center}
\end{table}
\egroup

\bgroup
\def\arraystretch{1.5}
\begin{table}[h!]
  \begin{center}
    \caption{\textbf{Result summarization for \textit{F(x)} evaluation}}
    \label{tab:table3}
    \begin{tabular}{|c|c|c|}

    \hline
      \textbf{Model} & \textbf{Interpolation Task} & \textbf{Extrapolation Task}\\
    \hline
      \textbf{CalcNet} & 5.011 & 9.517 \\
    \hline
      \textbf{G-CalcNet} & \textbf{4.305} & \textbf{7.347} \\
    \hline
      \textbf{NALU-MLP} & 11.045 & 21.738 \\
    \hline
      \textbf{NAC-MLP} & 12.438 & 24.079 \\
    \hline
      \textbf{MLP} & 15.129 & 34.724 \\

    \hline
    \end{tabular}
  \end{center}
\end{table}
\egroup

From result summarization Table \ref{tab:table3} we can clearly see our model has clearly outperformed simple MLPs and NAC/NALU MLPs for both extrapolation as well interpolation results also. Mean square error(MSE) defined in these columns for CalcNet is summation of all the errors of intermediate steps. Also, our G-NAC/G-NALU based CalcNet specified as G-CalcNet outperformed the regular CalcNet. Our results clearly shows that our models holds a semantic sense in numeric expression evaluation where as other highlights only memorization behavior. From Table \ref{tab:table2}, relative increase in error is largest for power operation followed by division for both the variants of CalcNet. Hence, even our model pipeline holds the limitations with power and division operations evaluation.

\subsection{Evaluating biquadratic equations}

With NAC/NALU modules trained from previous experiment for CalcNet. We aim to apply our findings and results in solving real world equations. Also, we comment on the reusability of these modules in our defined model pipeline. Solutions or roots to biquadratic equations like their predecessor simpler form quadratic equations can be evaluated with formulas but these are relatively complex than evaluating simple quadratic equations. Below, we derive lucid version of formulae for $\alpha x^{4}+\beta x^{3}+\gamma x^{2}+\delta x+\varepsilon =0$ biquadratic equation.

\begin{itemize}

\item Assuming general quartic equation of the form: $\alpha x^{4}+\beta x^{3}+\gamma x^{2}+\delta x+\varepsilon =0, \textbf{(1)}$, where x gets substituted by (y - $\beta$/4$\alpha$ ). \\
The equation is reduced to: $y^{4}+Ay^{2}+By+C=0$, \textbf{(2)}\\
with $A=\frac{\gamma }{\alpha }-\frac{3\beta ^{2}}{8\alpha ^{2}}$, \\
\\
$B=\frac{\delta }{\alpha }-\frac{\beta \gamma }{2\alpha ^{2}}+\frac{\beta^3 }{8\alpha^3 }$, \\
\\
$C=\frac{\varepsilon }{\alpha }-\frac{\beta \delta }{4\alpha ^{2}}+\frac{\beta ^{2}\gamma}{16\alpha ^{3}}-\frac{3\beta ^{4}}{256\alpha ^{4}}$. \\
After adding and subtracting $2sy^{2}+s^{2}$ to the LHS of (2) \& rearranging terms, we get: $\underset{\left( y^{2}+s\right) ^{2}}{\underbrace{y^{4}+2sy^{2}+s^{2}}}-\left[ \left( 2s-A\right) y^{2}-By+s^{2}-C\right] =0,  \textbf{(2a)}$
\\
We factorize the quadratic polynomial in \textit{2a}: $\left(2s-A\right) y^{2}-By+s^{2}-C=\left(2s-A\right)(y-y_+)(y-y_-)$ and make $y_+=y_-$ , which imposes a constraint on \textit{s}.
\\
Finally, we get: $\left( y^{2}+s+\sqrt{2s-A}y-\frac{B}{2\sqrt{2s-A}}\right)  \\
 \left( y^{2}+s-\sqrt{2s-A}y+\frac{B}{2\sqrt{2s-A}}\right)=0, \textbf{(3)}$ where \textit{s} satisfies the resolvent cubic equation as follow: $8s^{3}-4As^{2}-8Cs+\left( 4AC-B^{2}\right)=0, \textbf{(4)}$ Hence, the solution of \textbf{(2)} are solutions of \textbf{(3)}. Given by: \\

$y_{1}=-\frac{1}{2}\sqrt{2s-A}+\frac{1}{2}\sqrt{-2s-A+\frac{2B}{\sqrt{2s-A}}}$ \\
$y_{2}=-\frac{1}{2}\sqrt{2s-A}-\frac{1}{2}\sqrt{-2s-A+\frac{2B}{\sqrt{2s-A}}}$ \\
$y_{3}=\frac{1}{2}\sqrt{2s-A}+\frac{1}{2}\sqrt{-2s-A-\frac{2B}{\sqrt{2s-A}}}$ \\
$y_{4}=\frac{1}{2}\sqrt{2s-A}-\frac{1}{2}\sqrt{-2s-A-\frac{2B}{\sqrt{2s-A}}}$ \\

From this, we can get solutions for original equation. Given by: $x_{k}=y_{k}-\frac{\beta }{4\alpha },  k=1,2,3,4$

\end{itemize}

Now, with our CalcNet models we predict root for quartic equation for all \textbf{y\textit{$_i$}} values and evaluate accuracy of our models with MSE value comparison. We created test set of 2$\textsuperscript{16}$ entries with values ranging upto 5 times. Test dataset is equidistributed with values from interpolated and extrapolated values with respect to previous training range. We used previous NAC/NALU and G-NAC/G-NALU trained modules for making predictions on this testset to measure extensibility and robustness of CalcNet.

\bgroup
\def\arraystretch{1.5}
\begin{table}[h!]
  \begin{center}
    \caption{\textbf{Mean Square Error and Standard Deviations for predicting $Y_i$ values for (3)}}
    \label{tab:table4}
    \begin{tabular}{|c|c|c|}

    \hline
      \textbf{Root $Y_i$ values} & \textbf{CalcNet} & \textbf{G-CalcNet}\\
    \hline
      $Y_1$ & 11.324$\pm$0.53 & \textbf{8.176$\pm$0.51} \\
    \hline
      $Y_2$ & 11.861$\pm$0.58 & \textbf{8.792$\pm$0.57} \\
    \hline
      $Y_3$ & 11.506$\pm$0.56 & \textbf{8.633$\pm$0.55} \\
    \hline
      $Y_4$ & 11.137$\pm$0.62 & \textbf{8.989$\pm$0.60} \\

    \hline
    \end{tabular}
  \end{center}
\end{table}
\egroup

G-CalcNet outperforms vanilla CalcNet in this experiment also again reconfirming our hypothesis. We achieved a good MSE error of \textbf{8.176} percent in this experiment with very low standard deviation \textbf{0.51}, refer Table \ref{tab:table4}. Hence, our model definitely is robust and can be reused again on multiple different equation with fundamental operations being the same. With the above specified accuracy and very low standard deviations, this model can be definitely used in approximation calculation methods to achieve faster results.

\section{Conclusion and Future Scope}

We were able to demonstrate higher accuracies for both interpolation and extrapolation tasks on complex arithmetic equations. Also, our modified golden ratio based G-NACs and G-NALUs structure produced better results for arithmetic operations. The algorithm governing this neural network is applicable for symbols obeying any associative grammar and precedence order. Hence, an extensible neural network module for newer domains involving new operators can be trained and re-utilized again as explained in this work.
Exploration on stochastic equation solving for quantum physics is definitely an area for future work. Further, we will explore simplification of expressions by detecting identities and theorems within the numeric expressions.

\begin{thebibliography}{00}

\bibitem{b1}  Dehaene, Stanislas. "The number sense: How the mind creates mathematics." OUP USA, 2011.
\bibitem{b2} Gallistel, C. Randy. "Finding numbers in the brain." Philosophical Transactions of the Royal Society B: Biological Sciences 373.1740 (2018): 20170119.
\bibitem{b3}Fodor, Jerry A., and Zenon W. Pylyshyn. "Connectionism and cognitive architecture: A critical analysis." Cognition 28.1-2 (1988): 3-71.
\bibitem{b4}Marcus, Gary F. "The algebraic mind: Integrating connectionism and cognitive science." MIT press, 2018.
\bibitem{b5} Trask, Andrew, et al. "Neural arithmetic logic units." Advances in Neural Information Processing Systems. 2018.

\bibitem{b6} He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

\bibitem{b7} Lee, Younggun, et al. "Voice Imitating Text-to-Speech Neural Networks." arXiv preprint arXiv:1806.00927 (2018).

\bibitem{b8} Silver, David, et al. "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm." arXiv preprint arXiv:1712.01815 (2017)

\bibitem{b9} He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.

\bibitem{b10} Gelman, Rochel, C. Randy Gallistel. "The child’s understanding of number." Harvard, 1978.

\bibitem{b11} Piazza, Manuela, et al. "Tuning curves for approximate numerosity in the human intraparietal sulcus." Neuron, 44:547–555, 2004.

\bibitem{b12} Brunton, Steven, et al. "Discovering governing equations from data by sparse identification of nonlinear dynamical systems." Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.

\bibitem{b13}
Zheng, Kan, Xiaoli Zhang, Qiang Zheng, Wei Xiang, and Lajos Hanzo. "Quality-of-experience assessment and its application to video services in LTE networks." IEEE Wireless Communications 22, no. 1 (2015): 70-78.

\end{thebibliography}

\vspace{12pt}
\end{document}


% Instructions: you don't need to change anything in the macros, but
% feel free to define new commands as you wish. Starting from the main
% body, change the specs (e.g., your
% name). use \begin{solution} \end{solution} environment to write your
% solutions. Don't forget to list your collaborators.

\documentclass[12pt,addpoints,answers]{exam}
%============Macros==================%
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage[margin=1in]{geometry}
%--------------Cosmetic----------------%
\usepackage{mathtools}
\usepackage{hyperref}
\usepackage{fullpage}
\usepackage{microtype}
\usepackage{xspace}
\usepackage[svgnames]{xcolor}
\usepackage[sc]{mathpazo}
\usepackage{enumitem}
\setlist[enumerate]{itemsep=1pt,topsep=2pt}
\setlist[itemize]{itemsep=1pt,topsep=2pt}
%----------Header--------------------%
\def\course{CSCE629 Analysis of Algorithms}
\def\term{Texas A\&M U, Fall 2019}
\def\prof{Lecturer: Fang Song}
\newcommand{\handout}[5]{
   \renewcommand{\thepage}{\arabic{page}}
   \begin{center}
   \framebox{
      \vbox{
    \hbox to 5.78in { \hfill \large{\course} \hfill }
       \vspace{2mm}
       \hbox to 5.78in { {\Large \hfill \textbf{#5}  \hfill} }
       \vspace{2mm}
       \hbox to 5.78in { \term \hfill \emph{#2}}
       \hbox to 5.78in { {#3 \hfill \emph{#4}}}
      }
   }
   \end{center}
   \vspace*{4mm}
}
\newcommand{\hw}[4]{\handout{#1}{#2}{#3}{#4}{Homework #1}}

%-----defs and commands-----%
%\def\mG{[\textbf{G}]\xspace}
\def\veps{\varepsilon}
\newcommand{\bit}{\{0,1\}}
\newcommand{\negl}{\text{negl}}
\newcommand{\corr}[1]{{\color{blue}{#1}}}
\newcommand{\alg}[1]{\textsf{#1}}
%=======Main document==============%
\begin{document}

%----Specs: change accordingly-----%
\newif\ifstudent % comment out false
 \studenttrue
% \studentfalse

\def\hwnum{9}
\def\issuedate{11/01/19}
\def\duedate{10am, 11/08/19} %
\def\yourname{your name} % put your name here
%------------------------------%

\ifstudent
\hw{\hwnum}{\issuedate}{Student: \yourname}{Due: \duedate}%
\else
\hw{\hwnum}{\issuedate}{\prof}{Due: \duedate}%
\fi

\noindent \textbf{Instructions.}

\begin{itemize}
\item Typeset your submission by \LaTeX, and submit in PDF
  format. Your solutions will be graded on \emph{correctness} and
  \emph{clarity}. You should only submit work that you believe to be
  correct, and you will get significantly more partial credit if you
  clearly identify the gap(s) in your solution. You may opt for the
  ``I’ll take 15\%'' option.
\item You may collaborate with others on this problem set.  However,
  you must \textbf{{write up your own solutions}} and \textbf{{list
      your collaborators and any external sources}} for each
  problem. Be ready to explain your solutions orally to a course staff
  if asked.
\item For problems that require you to provide an algorithm, you must
  give a precise description of the algorithm, together with a proof
  of correctness and an analysis of its running time. You may use
  algorithms from class as subroutines. You may also use any facts
  that we proved in class or from the book.
\item \textbf{If you describe a Greedy algorithm, you will get no
    credit without a formal proof of correctness, even if your
    algorithm is correct.}
\end{itemize}

\noindent This assignment contains \numquestions\ questions,
\numpages\ pages for the total of \numpoints \ points and
\numbonuspoints\ bonus points. A random subset of the problems will be
graded. \medskip

\paragraph{Exercises. Do not turn in.}
\begin{questions}
  \question (Max bipartite matching)

  \begin{parts}
    \part Give a linear-programming formulation of the bipartite
    maximum matching problem. The input is a bipartite graph
    $G = (U \cup V; E)$, where $E \subseteq U \times V$; the output is
    the largest matching in $G$. Your linear program should have one
    variable for each edge.

    \part Now dualize the linear program from part (a). What do the
    dual variables represent? What does the objective function
    represent?  What problem is this!?
  \end{parts}
\end{questions}
\newpage
\paragraph{Problems to turn in.}

\begin{questions}

\question[15] (Vertex cover) A \emph{vertex cover} of an undirected
  graph $G = (V, E)$ is a subset of the vertices which touches every
  edge—that is, a subset $S \subseteq V$ such that for each edge
  ${u,v} \in E$, one or both of $u,v$ are in $S$.  Describe and
  analyze an algorithm, as efficient as you can, to find a minimum
  vertex cover in a bipartite graph.

  \newpage

  \question (Updating max flow) You are given a flow network
  $G = (V,E)$ with source $s$ and sink $t$, and integer capacities.
  \begin{parts}
    \part[10] Suppose that you are given a max flow in $G$. Now we
    increase the capacity of a single edge $(u,v)\in E$ by 1. Given an
    $O(m+n)$-time algorithm to update the max flow.
    \part[10] Now suppose all edges have unit capacity and you are
    given a parameter $k$. The goal is to delete $k$ edges so as to
    reduce the maximum $s-t$ flow in $G$ as much as possible. In other
    words, you should find a set of edges $F\subseteq E$ so that
    $|F|=k$ and the maximum $s-t$ flow in $G'=(V,E-F)$ is as small as
    possible subject to this. Describe and analyze a polynomial-time
    algorithm to solve this problem.
  \end{parts}

  \newpage

  \question[15] (Filling classrooms) Faced with the threat of brutally
  severe budget cuts, Potemkin University has decided to hire actors
  to sit in classes as ``students'', to ensure that every class they
  offer is completely full. Because actors are expensive, the
  university wants to hire as few of them as possible.

  Building on their previous leadership experience at the now-defunct
  Sham-Poobanana University, the administrators at Potemkin have given
  you a directed acyclic graph $G = (V,E)$, whose vertices represent
  classes, and where each edge $i\to j$ indicates that the same
  ``student'' can attend class $i$ and then later attend class $j$. In
  addition, you are also given an array $cap[1,\ldots, V]$ listing the
  maximum number of ``students'' who can take each class. Describe an
  analyze an algorithm to compute the minimum number of
  ``students''that would allow every class to be filled to capacity.

\end{questions}
\end{document}


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage[T1]{fontenc}
\usepackage[USenglish]{babel}
\usepackage[nodayofweek,level]{datetime}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tikz}
\usetikzlibrary{intersections,arrows.meta}
\usepackage{pgfplots}
\usepackage[scr]{rsfso}
\usepackage{array}
\usepackage{centernot}
\usepackage{stackengine}
\usepackage{graphics,graphicx}
\usepackage{pstricks,pst-node,pst-tree}
\usetikzlibrary{arrows, automata}
\hbadness=10001 %gets rid of "\hfill underfull" warning
\stackMath

\usepackage{tikz,pgfplots}

\pgfplotsset{compat=1.10}
% Uncomment to use "fillbetween" function
% otherwise leave commented because
% the red highlighting is annoying
%\usepgfplotslibrary{fillbetween}

% Change for each new hw week
\newcommand{\dueDate}{\formatdate{8}{12}{2017}} % day/month/year
\newcommand{\hwNum}{10}

\begin{document}

\selectlanguage{USenglish}
%------------------------ Title Code ------------------------
\title{Homework: Week \hwNum}
\author{Joseph Ismailyan}
\date{}
\maketitle
\begin{flushleft}
Math 100 \\
Due: \dueDate \\
Professor Boltje \\
MWF 9:20a-10:25a
\end{flushleft}

%------------------------ Begin Page 1 ------------------------
\begin{minipage}[t]{0.40\textwidth}

\section*{Section 12.1}
\subsection*{8.}
The set $ f=\{(x,y)\in\mathbb{Z}\times\mathbb{Z}:x+3y=4\} $ is not a function because $f(x)=\frac{4-x}{3}$ which is not always an element of the integers. For example, if $ x = 2 $ then $ f(x)\in\mathbb{Q} $.

\subsection*{10.}
The set $ f=\{(x^3,x):x\in\mathbb{R}\} $ is function because every element $ x $ maps to exactly one element $ f(x) $. It also passes the horizontal line test.

\end{minipage}
% Creates vertical line
\hfill\vline\hfill
\begin{minipage}[t]{0.45\textwidth}

\section*{Section 12.2}
\subsection*{14.}
\textbf{Problem:} Is the  function $ \theta:\mathscr{P}(\mathbb{Z})\rightarrow\mathscr{P}(\mathbb{Z}) $ defined as $ \theta(X)=\overline{X} $ injective? Surjective? Bijective?
\\\\ Injective: Suppose $ X,Y\in\mathscr{P}(\mathbb{Z}) $ and $ \theta(X)=\theta(Y) $. Then $ \overline{X}=\overline{Y} $, so  $ \overline{\overline{X}}=\overline{\overline{Y}} \rightarrow X=Y $.
\\\\Surjective: Suppose $ Y\in\mathscr{P}(\mathbb{Z}) $. Then $ \overline{Y}\in\mathscr{P}(\mathbb{Z}) $ and $ \theta(\overline{Y})=\overline{\overline{Y}}=Y $.
\\\\Bijective: Since the function is both injective and surjective, then it is bijective.

\end{minipage}
\pagebreak

%------------------------ End Page 1 --------------------------

%------------------------ Begin Page 2 ------------------------

\begin{minipage}[t]{0.40\textwidth}

\subsection*{16.} Surjective: No functions are surjective because there are more elements in the codomain than in the domain, so there will always be element in the codomain which are left unmapped.
\\\\Bijective: Since the function is not surjective, it cannot be bijective, so no functions are bijective.
\\\\Injective: There are $ 7 $ options for the first element in the domain, 6 for the second, and so on. The last element in the domain has 3 options, so there will always be 2 left over. This means there are $ 7! $ different combinations and $ 2! $ will be left out. So, there are $ \frac{7!}{2!} $ total functions that are injective.

\section*{Section 12.3}
\subsection*{4.}
\textit{Proof:} Consider a square with side length of 1 unit. If we want to place 4 points inside of the square as far apart from each other as possible, logically we would place one at each corner of the square. This incidentally breaks the square of side length 1 unit into 4 smaller square each with side length $ \frac{1}{2} $ unit. Using the Pythagorean theorem, we find that the distance from one corner of the smaller square to the opposite corner is $ \frac{\sqrt{2}}{2} $. So we have 4 smaller squares and 5 points, so by the Pigeon Hole Principle, there will be at least one pair of points that are within $ \frac{\sqrt{2}}{2} $ of each other.

\end{minipage}
% Creates vertical line
\hfill\vline\hfill
\begin{minipage}[t]{0.45\textwidth}

\section*{Section 12.4}
\subsection*{4.}
\textbf{Problem:} Suppose $ A=\{a,b,c\} $. Let $ f:A\rightarrow A $ be the function $ f=\{(a,c),(b,c),(c,c)\} $ and let $ g:A\rightarrow A $ be the function $ f=\{(a,a),(b,b),(c,a)\} $. Find $ g\circ f$ and $ f\circ g $.
\\\textbf{Answer: } $ g\circ f =\{(a,a),(b,a),(c,a)\}$ and $ f\circ g =\{(a,c),(b,c),(c,c)\}$.

\section*{Section 12.5}
\subsection*{8.}
Injective: Suppose $ X,Y\in\mathscr{P}(\mathbb{Z}) $ and $ \theta(X)=\theta(Y) $. Then $ \overline{X}=\overline{Y} $, so  $ \overline{\overline{X}}=\overline{\overline{Y}} \rightarrow X=Y $.
\\\\Surjective: Suppose $ Y\in\mathscr{P}(\mathbb{Z}) $. Then $ \overline{Y}\in\mathscr{P}(\mathbb{Z}) $ and $ \theta(\overline{Y})=\overline{\overline{Y}}=Y $.
\\\\Bijective: Since the function is both injective and surjective, then it is bijective.
\\ So the function is bijective. The inverse of $ \theta(X) $ is $ \theta^{-1} (X)=\overline{X}$, the complement of the complement is the set $ X $, so $ \theta = \theta^{-1} $.

\end{minipage}
\pagebreak

%------------------------ End Page 2 --------------------------

%------------------------ Begin Page 3 ------------------------

\begin{minipage}[t]{0.40\textwidth}

\section*{Section 12.6}
\subsection*{10.}
\textbf{Problem:} Given $ f:A\rightarrow B $ and subsets $ Y,Z\subseteq B $, prove $ f^{-1}(Y\cap Z)=f^{-1}(Y)\cap f^{-1}(Z) $.
\\\texttt{Proof:} Since we must show that two sets are equal, we must show that they're subsets of each other. So first, for no particular reason, we'll show that $ f^{-1}(Y\cap Z)\subseteq f^{-1}(Y)\cap f^{-1}(Z) $. Suppose $ a\in Y\cap Z $, so $ f(a)\in Y $ and $ f(a)\in Z $. Then $ a\in f^{-1}(Y) $ and $ a\in f^{-1}(Z) $, so $ a\in f^{-1}(Y)\quad\cap\quad a\in f^{-1}(Z) \rightarrow f^{-1}(Y\cap Z)\subseteq f^{-1}(Y)\cap f^{-1}(Z) $. Now we must show that $ f^{-1}(Y)\cap f^{-1}(Z)\subseteq f^{-1}(Y\cap Z) $. Suppose $ a\in f^{-1}(Y)\cap f^{-1}(Z) $ so $a\in f^{-1}(Y) $ and $ a\in f^{-1}(Z) $. Then $ f(a)\in Y $ and $ f(a)\in Z $, so $ f(a)\in Y\cap Z $, then $ a\in f^{-1}(Y\cap Z) $, so $ f^{-1}(Y)\cap f^{-1}(Z)\subseteq f^{-1}(Z)$. Therefore $ f^{-1}(Y\cap Z)=f^{-1}(Y)\cap f^{-1}(Z) $.
\end{minipage}
% Creates vertical line
\hfill\vline\hfill
\begin{minipage}[t]{0.45\textwidth}

\subsection*{14.}
\textbf{Problem:} Let $ f:A\rightarrow B $ be a function and $ Y\subseteq B $. Prove or disprove $ f^{-1}(f(f^{-1}(Y))) =f^{-1}(Y) $. Since we must show that two sets are equal, we must show that they're subsets of each other. Suppose $ y\in f^{-1}(f(f^{-1}(Y))) $. Then $ f^{-1}(y)\in f(f^{-1}(Y))\rightarrow f(f^{-1}(y))\in f^{-1}(Y) \rightarrow (f^{-1}\circ f)(y)\in f^{-1}(Y)$. Then $ y\in f^{-1}(f(f^{-1}(Y)))\implies y\in f^{-1}(Y) $. So $ f^{-1}(f(f^{-1}(Y)))\subseteq f^{-1}(Y) $. Next, suppose $ y\in f^{-1}(Y) $, then $ f(y)\in f(f^{-1}(Y))\rightarrow f(f^{-1}(y))\in f^{-1}(f(f^{-1}(Y))) \rightarrow (f^{-1}\circ f)(y)\in f^{-1}(f(f^{-1}(Y)))\rightarrow y\in f^{-1}(f(f^{-1}(Y)))$. So $y\in f^{-1}(Y) \implies y\in f^{-1}(f(f^{-1}(Y))) $. Then $ f^{-1}(Y)\subseteq f^{-1}(f(f^{-1}(Y))) $. Thus $ f^{-1}(f(f^{-1}(Y))) =f^{-1}(Y) $.

\end{minipage}

%------------------------ End Page 4 --------------------------

%------------------------ Begin Page 5 ------------------------

%------------------------ End Page 5 ------------------------
\end{document}


\documentclass[12pt, a4paper]{article} % book, report, article, letter, slides
                                       % letterpaper/a4paper, 10pt/11pt/12pt, twocolumn/twoside/landscape/draft

%%%%%%%%%%%%%%%% PACKAGES %%%%%%%%%%%%%%%%%%%%%

\usepackage[utf8]{inputenc} % encoding

\usepackage[english]{babel} % use special characters and also translates some elements within the document.

\usepackage{amsmath}        % Math
\usepackage{amssymb}        % Math, extended collection
\usepackage{amsthm}         % Math, \newtheorem, \proof, etc
\newtheorem{theorem}{Theorem}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}

\usepackage{hyperref}       % Hyperlinks \url{url} or \href{url}{name}

\usepackage{parskip}        % \par starts on left (not idented)

\usepackage{abstract}       % Abstract

\usepackage{tocbibind}      % Adds the bibliography to the table of contents (automatically)

\usepackage{graphicx}       % Images
\graphicspath{ {./images/} }

% \usepackage[document]{ragged2e}  % Left-aligned (whole document)
% \begin{...} ... \end{...}   flushleft, flushright, center

%%%%%%%%%%%%%%%% GRAPHICS %%%%%%%%%%%%%%%%%%%%%

\usepackage{tikz}
\usetikzlibrary{arrows}

\tikzset{
  treenode/.style = {align=center, inner sep=0pt, text centered,
    font=\sffamily},
  arn_n/.style = {treenode, circle, white, font=\sffamily\bfseries, draw=black,
    fill=black, text width=1.5em},% arbre rouge noir, noeud noir
  arn_r/.style = {treenode, circle, red, draw=red,
    text width=1.5em, very thick},% arbre rouge noir, noeud rouge
  arn_x/.style = {treenode, rectangle, draw=black,
    minimum width=0.5em, minimum height=0.5em}% arbre rouge noir, nil
}

%%%%%%%%%%%%%%%% CODE %%%%%%%%%%%%%%%%%%%%%

\usepackage{minted}         % Code listing
% \mint{html}|
Something here
|
% \inputminted{octave}{BitXorMatrix.m}

%\begin{listing}[H]
  %\begin{minted}[xleftmargin=20pt,linenos,bgcolor=codegray]{haskell}
  %\end{minted}
  %\caption{Example of a listing.}
  %\label{lst:example} % You can reference it by \ref{lst:example}
%\end{listing}

\newcommand{\code}[1]{\texttt{#1}} % Define \code{foo.hs} environment

%%%%%%%%%%%%%%%% COLOURS %%%%%%%%%%%%%%%%%%%%%

\usepackage{xcolor}         % Colours \definecolor, \color{codegray}
\definecolor{codegray}{rgb}{0.9, 0.9, 0.9}
% \color{codegray} ... ...
% \textcolor{red}{easily}

%%%%%%%%%%%%%%%% CONFIG %%%%%%%%%%%%%%%%%%%%%

\renewcommand{\absnamepos}{flushleft}
\setlength{\absleftindent}{0pt}
\setlength{\absrightindent}{0pt}

%%%%%%%%%%%%%%%% GLOSSARIES %%%%%%%%%%%%%%%%%%%%%

%\usepackage{glossaries}

%\makeglossaries % before entries

%\newglossaryentry{latex}{
    %name=latex,
    %description={Is a mark up language specially suited
    %for scientific documents}
%}

% Referene to a glossary \gls{latex}
% Print glossaries \printglossaries

%%%%%%%%%%%%%%%% HEADER %%%%%%%%%%%%%%%%%%%%%

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{Arnau Abella - MIRI}
\lhead{ADS - First Delivery}
\rfoot{Page \thepage}

%%%%%%%%%%%%%%%% TITLE %%%%%%%%%%%%%%%%%%%%%

\title{Red-black Trees: A Pure Functional Implementation}
\author{Arnau Abella}
\date{\today}

%%%%%%%%%%%%%%%% DOCUMENT %%%%%%%%%%%%%%%%%%%%%

\begin{document}

\maketitle

\section{Introduction}\label{s:introduction}

Although binary search trees work very well on random or unordered data, they perform very poorly on ordered data, for which any individual operation migh take up to $\mathcal{O}(n)$ time.

The solution to this problem is to keep each tree approximately balanced. Then no individual operation takes more than $\mathcal{O}(log n)$ time. There are several implementations of self-balancing binary search trees such as \textit{2-3 tree, AA tree, AVL tree, B-tree, Treap, etc}.

This delivery is about one of the most popular families of self-balancing binary search trees, the \textit{red-black trees} \cite{gs78}.

A \textbf{red-black tree} is a binary search tree in which every node is colored either red or black.

\begin{figure}[b]
  \includegraphics[scale=0.5]{rbt}
  \centering
  \caption{Example of a red-black tree.}
  \label{fig:rbt}
\end{figure}

Every red-black tree satisfy the following two balance invariants:

\begin{enumerate}
  \item No red node has a red child. \label{inv1}
  \item Every path from the root to an empty node contains the same number of black nodes. \label{inv2}
\end{enumerate}

Taken together, these two invariants guarantee that the longest possible path in a red-black tree, one with alternating black and red nodes, is no more than twice as long as the sorthest possible, one with black nodes only.

\begin{theorem}\label{t:balance}
  The maximum depth of a node in a red-black tree of size $n$ is at most $2\lfloor \log (n+1) \rfloor$
\end{theorem}

\begin{proof}\label{p:balance}
  The maximum number of black nodes in the longest path from the root to a leaf is restricted by the number of nodes in the shortest past (inv. \ref{inv2}).

  Assume that the shortest path has depth $k$. Then, there are $k+1$ nodes in a path of depth $k$.

  From the definition of the shortest path, there is a full and complete binary subtree of depth $k$ (that includes the shortest path). Suposse that the complete binary subtree has $n$ nodes. So, by the definition of a complete binary tree, $n = 2^{k+1} -1$. Hence, we can compute the number of nodes in the shortest by using the previous equation.

  \begin{align*}
    n     &= n^{k+1}-1 \\
    k + 1 &= \log_2 (n+1)
  \end{align*}

  The number f nodes in the shortest path is $\log_2 (n+1)$. So, the largest path can have, at most, $\log_2 (n+1)$ black nodes (inv. \ref{inv2}).

  Let's make the largest path with $\log_2 (n+1)$ black nodes. We want to put as many red nodes as possible because we are limited in the number of black nodes. But, because of the inv. \ref{inv1}, we need to alternate between black and red nodes.

  If the root is black, there will be as many red nodes as black ones. The depth $k$ of the largest path is the following:

  \begin{align*}
    k &= \#red \cdot \#black \\
    k &= \log_2 (n+1) \cdot \log_2 (n+1) \\
    k &= 2\lfloor \log_2 (n+1) \rfloor \\
    k &= \mathcal{O}(\log n)
  \end{align*}

\end{proof}

\section{Implementation}\label{s:implementation}

The implementation proposed in this first delivery for the \textbf{red-black tree} is a persistent representation written in \textbf{haskell} \cite{haskell}, an advanced, lazy, purely functional programming language, and based on the implementation from \textit{C. Okasaki} written in \textit{Standard ML} \cite{oka98}.

The proposed implementation has some tweaks $wrt.$ the implementation from \textit{C. Okasaki} that improve insertion time. Also, the implementation includes an auxiliar method to construct the \textit{red-black tree} in $\mathcal{O}(n)$ instead of $\mathcal{O}(n \log n)$.

The source code, that can be found in the \nameref{s:appendix-a}.

Let's have a look at the most interesting parts of the code. First, we will start with the data definition at listing \ref{lst:constructor}, which is pretty straightforward. I included some useful classes instances such as \code{functor, foldable and traversable}. The attentive readers will notice the exclamation marks in front of the data constructors. This is a language extension called \textit{BangPatterns} that allows changing how an expression is evaluated in haskell.

\begin{listing}[H]
  \begin{minted}[xleftmargin=20pt,linenos]{haskell}
data Color = R -- ^ Red
           | B -- ^ Black
  deriving (Show, Enum)

data RedBlackTree a
  = Bin Color !(RedBlackTree a) !a !(RedBlackTree a)
  | Tip
  deriving (Functor, Foldable, Traversable)
  \end{minted}
  \caption{Red-black tree data types}
  \label{lst:constructor}
\end{listing}

\newpage

The next method we are going to have a look into detail is \code{insert}. Insert is the hardest method to implement because it must not violate any of the two invariants after the insertion.

To not violate invariant \ref{inv2} we are going to always insert a \textbf{red} node but this may have broken the invariant \ref{inv1} if the parent was already red so we need to rebalance the tree, recursively.

Figure \ref{fig:violations} displays all possible configurations where the invariant 1 is violated.

\begin{figure}
\centering
    \begin{tikzpicture}[->,>=stealth',level/.style={sibling distance = 5cm/#1,level distance = 1.5cm}]
    \node [arn_n] {z}
        child{ node [arn_r] {y}
                child{ node [arn_r] {x}
                  child{ node [arn_x] {} edge from parent node[above left] {$a$}}
                  child{ node [arn_x] {} edge from parent node[above right] {$b$}}
                }
                child{ node [arn_x] {} edge from parent node[above right] {$c$}}
        }
        child{ node [arn_x] {} edge from parent node[above right] {$d$}}
    ;
    \end{tikzpicture}
    \begin{tikzpicture}[->,>=stealth',level/.style={sibling distance = 5cm/#1,level distance = 1.5cm}]
    \node [arn_n] {z}
        child{ node [arn_r] {x}
                child{ node [arn_x] {} edge from parent node[above left] {$a$}}
                child{ node [arn_r] {y}
                  child{ node [arn_x] {} edge from parent node[above left] {$b$}}
                  child{ node [arn_x] {} edge from parent node[above right] {$c$}}
                }
        }
        child{ node [arn_x] {} edge from parent node[above right] {$d$}}
    ;
    \end{tikzpicture}
    \begin{tikzpicture}[->,>=stealth',level/.style={sibling distance = 5cm/#1,level distance = 1.5cm}]
    \node [arn_n] {x}
        child{ node [arn_x] {} edge from parent node[above left] {$a$}}
        child{ node [arn_r] {y}
                child{ node [arn_x] {} edge from parent node[above left] {$b$}}
                child{ node [arn_r] {z}
                  child{ node [arn_x] {} edge from parent node[above left] {$c$}}
                  child{ node [arn_x] {} edge from parent node[above right] {$d$}}
                }
        }
    ;
    \end{tikzpicture}
    \begin{tikzpicture}[->,>=stealth',level/.style={sibling distance = 5cm/#1,level distance = 1.5cm}]
    \node [arn_n] {x}
        child{ node [arn_x] {} edge from parent node[above left] {$a$}}
        child{ node [arn_r] {z}
                child{ node [arn_r] {y}
                  child{ node [arn_x] {} edge from parent node[above left] {$b$}}
                  child{ node [arn_x] {} edge from parent node[above right] {$c$}}
                }
                child{ node [arn_x] {} edge from parent node[above right] {$d$}}
        }
    ;
    \end{tikzpicture}
\caption{Eliminating red nodes with red parents: before} \label{fig:violations}
\end{figure}

Figure \ref{fig:rebalance} displays the resulting subtree after the rebalance.

\begin{figure}
\centering
    \begin{tikzpicture}[->,>=stealth',level/.style={sibling distance = 5cm/#1,level distance = 1.5cm}]
    \node [arn_r] {y}
        child{ node [arn_n] {x}
          child{ node [arn_x] {} edge from parent node[above left] {$a$}}
          child{ node [arn_x] {} edge from parent node[above right] {$b$}}
        }
        child{ node [arn_n] {z}
          child{ node [arn_x] {} edge from parent node[above left] {$c$}}
          child{ node [arn_x] {} edge from parent node[above right] {$d$}}
        }
    ;
    \end{tikzpicture}
\caption{Eliminating red nodes with red parents: after} \label{fig:rebalance}
\end{figure}

Here is the code of \code{insert}. You may notice that \code{balance} method is split in two different cases. This is an optimization because some cases are not possible depending on the branch you came for.

\begin{listing}[H]
  \begin{minted}[xleftmargin=20pt,linenos]{haskell}
insert
  :: (Ord a)
  => a -> RedBlackTree a -> RedBlackTree a
insert x tree =
  let (Bin _ a y b) = ins tree
  in Bin B a y b
    where
      ins               Tip  = Bin R Tip x Tip
      ins node@(Bin c a y b) =
        if      x < y   then lbalance (Bin c (ins a) y b)
        else if x > y   then rbalance (Bin c a y (ins b))
        else                 node

lbalance :: RedBlackTree a -> RedBlackTree a
lbalance (Bin B (Bin R (Bin R a x b) y c) z d) =
    Bin R (Bin B a x b) y (Bin B c z d)
lbalance (Bin B (Bin R a x (Bin R b y c)) z d) =
    Bin R (Bin B a x b) y (Bin B c z d)
lbalance tree                                  =
    tree

rbalance :: RedBlackTree a -> RedBlackTree a
rbalance (Bin B a x (Bin R b y (Bin R c z d))) =
    Bin R (Bin B a x b) y (Bin B c z d)
rbalance (Bin B a x (Bin R (Bin R b y c) z d)) =
    Bin R (Bin B a x b) y (Bin B c z d)
rbalance tree                                  =
    tree
  \end{minted}
  \caption{Red-black tree insert method}
  \label{lst:insert}
\end{listing}

Finally, the listing \ref{lst:fromOrdList} contains the code for the list constructor of a red-black tree.

The code is quite advance but it is basically taking advantage of the fact that we know the position of the node in the given tree, because the precondition of this method is that the given list is ordered, so we can remove the cost of searching the position of the $i$-th for each $i$ element inserted.

The total cost in time and space of this algorithm is $\mathcal{O}(n)$.

\begin{listing}[H]
  \begin{minted}[xleftmargin=20pt,linenos]{haskell}
fromOrdList :: [a] -> RedBlackTree a
fromOrdList xs' =
  toTree (Tip, ins ([], xs'))
    where
      balance' [(R, v1, t1)] = [(B, v1, t1)]
      balance' ((R, v1, t1):(R, v2, t2):(B, v3, t3):xs) =
        (B, v1, t1):(balance' ((R, v2, (Bin B t3 v3 t2)):xs))
      balance' xs = xs

      ins (ts, [])   = ts
      ins (ts, x:xs) = ins ( balance' ((R, x, Tip):ts), xs)

      toTree (t, [])                  =
        t
      toTree (t, ((color, v, t'):ts)) =
        toTree ((Bin color t' v t), ts)
  \end{minted}
  \caption{Red-black tree fromOrdList method}
  \label{lst:fromOrdList}
\end{listing}

\subsection{Documentation}\label{ss:documentation}

One cool feature of \textbf{cabal} \cite{cabal}, one of the most used building tool in Haskell, is that it can generate documentation \textit{"for free"} of your source code. Underneath, Cabal is using \textbf{haddock} \cite{haddock}, that can automatically generate documentation from annotated Haskell source code.

Haddock can generate documentation for a myriad of formats including \textit{html} and \textit{latex}. The documentation can be generated by \mint{bash}|$ cabal haddock|Here you can find a screenshot from the generated documentation from {\fontfamily{qcr}\selectfont RedBlackTree.hs}

\begin{figure}[H]
  \includegraphics[width=\textwidth]{haddock1}
  \centering
\end{figure}

\section{Experimentation}\label{s:experimentation}

Theoretically, the maximum depth of the tree is $2\lfloor \log (n+1) \rfloor$. Let's test if that fact holds in practice.

Before testing the maximum depth property of the red-black trees, we need to test the correctes of our code.

For this purpose, I created the following tests at \code{test/Chapter3/RedBlackTreeSpec.hs}

\begin{listing}[H]
  \begin{minted}[xleftmargin=1pt,linenos]{haskell}
spec :: Spec
spec = do
  describe "Red-black tree" $ do
    it "checkInvariants should check if
          invariants are fulfilled" $ do
      RBT.checkInvariants rbt1 `shouldBe` True
      RBT.checkInvariants bad  `shouldBe` False
      RBT.checkInvariants bad2 `shouldBe` False

    it "insert should preserve invariants" $ do
      let rbt = foldr RBT.insert
                      RBT.empty
                      ([1..10000] :: [Int])
      RBT.checkInvariants rbt `shouldBe` True

    it "fromOrdList should preserve invariants" $ do
      let rbt = RBT.fromOrdList [1..10000] :: RedBlackTree Int
      RBT.checkInvariants rbt `shouldBe` True

    it "toOrdList should return an ordered list of the
          elements of the tree" $ do
      let rbt = foldr RBT.insert
                      RBT.empty
                      ([3,5,2,1,8,9,5,4] :: [Int])
      RBT.toOrdList rbt `shouldBe` [1,2,3,4,5,8,9]

    it "toOrdList . fromOrdList == id" $ do
      let xs = [1..1000] :: [Int]
          f = RBT.toOrdList . RBT.fromOrdList
      f xs `shouldBe` xs

    prop "maxDepth = 2*floor[log (n + 1)]" $ do
      property $
        \(n :: Positive Int) ->
           let size = 1000
               rbt = RBT.fromOrdList [1..size]
           in RBT.maxDepth rbt
                <= 2*floor (logBase 2.0 (fromIntegral size + 1.0))
  \end{minted}
  \caption{Red-black tree fromOrdList method}
  \label{lst:fromOrdList}
\end{listing}

In order to prove the maximum depth property of the red-black trees we are going to implement the following experiment:

\begin{enumerate}
  \item Generate $n$ randomly-built red-black trees of size $m$, where $n$ and $m$ are large enough.
  \item Compute the height of the $n$ red-black trees.
  \item Compute the max, mean and standard deviation of the depth of each red-black tree.
  \item Compare it to the theoretically maximum heigth.
\end{enumerate}

The haskell code to generate the randomly-built red-black tree of size $m$ uses a property-based testing library called QuickCheck \cite{quickcheck} that allows the user to generate a huge amount of uniformly distributed random data.

\begin{listing}[H]
  \begin{minted}[xleftmargin=10pt,linenos]{haskell}

-- | A generator for values of type 'RedBlackTree' of the given size.
genRBT :: (Arbitrary a, Ord a)
       => Int -> Gen (RedBlackTree a)
genRBT = fmap (fromList . unUnique) . genUniqueList

newtype UniqueList a = UniqueList { unUnique :: [a] }
    deriving Show

-- | 90 % of samples are randomly distributed elements
--   10 % are sorted.
genUniqueList :: (Arbitrary a, Ord a)
              => Int -> Gen (UniqueList a)
genUniqueList n =
  frequency [ (9, genUniqueList' n arbitrary)
            , (1, (UniqueList . unSorted) <$>
                    genUniqueSortedList n arbitrary)
            ]

genUniqueList' :: (Eq a) => Int -> Gen a -> Gen (UniqueList a)
genUniqueList' n gen =
  UniqueList <$> vectorOf n gen `suchThat` isUnique

newtype UniqueSortedList a =
    UniqueSortedList { unSorted :: [a] }
  deriving Show

genUniqueSortedList :: (Ord a)
                    => Int -> Gen a -> Gen (UniqueSortedList a)
genUniqueSortedList n gen =
  UniqueSortedList . List.sort . unUnique <$> genUniqueList' n gen

isUnique :: Eq a => [a] -> Bool
isUnique x = List.nub x == x

  \end{minted}
  \caption{QuickCheck generators.}
  \label{lst:quickcheck}
\end{listing}

The code of the experiment is the following

\begin{listing}[H]
  \begin{minted}[xleftmargin=10pt,linenos]{haskell}
-- 1.- Generate a random vector of [n-m, n+m] elements
-- 2.- Transform it into a red-black tree.
-- 3.- Compute the maximum length
-- 4.- Get the statistics
-- 5.- Output them on the stdout
runExperiment
  :: Int -- ^ Number of samples
  -> Int -- ^ Size of the samples
  -> IO ()
runExperiment n size = do
  samples <- generate $ replicateM n (RBT.genRBT @Int size)
  let (Just max', mean, std) =
    L.fold ((,,) <$> L.maximum <*> L.mean <*> L.std) $
                        fmap (fromIntegral . RBT.maxDepth) samples
  report n size max' mean std

  \end{minted}
  \caption{Experiment of the theoretical depth of a red-black tree.}
  \label{lst:experiment}
\end{listing}

\begin{listing}[H]
  \begin{minted}[xleftmargin=1pt,linenos]{haskell}
# samples: 100
tree size: 100

max depth(max) : 8.0
max depth(mean): 7.720000000000001
max depth(std) : 0.448998886412873

max depth  (theoretical): 12
perfectly balanced depth: 6
  \end{minted}
  \caption{Output of the experiment for 100 samples of size 100.}
  \label{lst:output}
\end{listing}

From the output of the experiment, we conclude that the theorical maximum depth holds for an empirical red-black tree. Furthemore, the height is near to a perfectly balanced binary tree.

\section{Conclusion}\label{s:conclusion}

We have coded a simple, yet efficient purely functional implementation of red-black trees in Haskell and have proved that the maximum theoretical depth holds in practice by conducting an empirical experiment over randomly generated red-black trees.

\section{Curiosity}\label{s:curiosity}

One of the reasons this implementation is so much  simpler than typical presentations of red-black trees (e.g., Chapter 14 of \cite{clr90}) is that it uses subtly different  rebalancing  transformations.  Imperative implementations typically split the four dangerous cases considered here into eight cases, according to the color  of the sibling  of the red node with a red child.  Knowing the color of the red parent's  sibling allows the transformations  to use fewer  assignments in some cases and to terminate rebalancing early in others.

However, in a functional setting, where we are copying the nodes in question  anyway, we cannot reduce the number  of assignments in this fashion, nor can we terminate copying early, so there is no point is using the more complicated transformations.

%%%%%%%%%%%%%%%% Appendix %%%%%%%%%%%%%%%%%%%%%

\newpage

\section*{Appendix A}\label{s:appendix-a}

\subsection*{RedBlackTree.hs}\label{s:redblacktree}

\inputminted{haskell}{../../Chapter3/RedBlackTree.hs}

%%%%%%%%%%%%%%%% BIBLIOGRAPHY %%%%%%%%%%%%%%%%%%%%%

\bibliographystyle{alpha}
\bibliography{refs}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}