\documentclass[twoside,11pt]{article}

% Any additional packages needed should be included after jmlr2e.
% Note that jmlr2e.sty includes epsfig, amssymb, natbib and graphicx,
% and defines many common macros, such as 'proof' and 'example'.
%
% It also sets the bibliographystyle to plainnat; for more information on
% natbib citation styles, see the natbib documentation, a copy of which
% is archived at http://www.jmlr.org/format/natbib.pdf

% Available options for package jmlr2e are:
%
%   - abbrvbib : use abbrvnat for the bibliography style
%   - nohyperref : do not load the hyperref package
%   - preprint : remove JMLR specific information from the template,
%         useful for example for posting to preprint servers.
%
% Example of using the package with custom options:
%
% \usepackage[abbrvbib, preprint]{jmlr2e}

\usepackage{jmlr2e}
\usepackage{booktabs} % for professional tables
\usepackage{amsmath,amsfonts,amssymb,mathtools}
\usepackage{algorithm}
\usepackage{algpseudocode}

% Definitions of handy macros can go here

\newcommand{\dataset}{{\cal D}}
\newcommand{\fracpartial}[2]{\frac{\partial #1}{\partial  #2}}

% Heading arguments are {volume}{year}{pages}{date submitted}{date published}{paper id}{author-full-names}

\jmlrheading{1}{2000}{1-48}{4/00}{10/00}{meila00a}{Marina Meil\u{a} and Michael I. Jordan}

% Short headings should be running head and authors last names

\ShortHeadings{Online Scalable RNNs}{Javed, Shah, Sutton, and White}
\firstpageno{1}

\newcommand{\etal}{\textit{et al}.}
\newcommand{\ie}{\textit{i}.\textit{e}., }
\newcommand{\eg}{\textit{e}.\textit{g}. }
\newcommand{\algoname}{\text{Master-User}}
\newcommand{\archi}{\text{Col-NN}}

\def\a{\alpha}
\def\b{\beta}
\def\g{\gamma}
\def\d{\delta}
\def\dd{\mbox{d}}
\def\ve{\varepsilon}
\def\e{\epsilon}
\def\n{{\bf n}}
\def\r{\rho}
\def\t{\theta}
\def\q{{\bf q}}
\def\k{{\bf k}}
\def\x{{\bf x}}
\def\y{{\bf y}}
\def\l{\lambda}
\def\o{\omega}
\def\p{{\bf p}}
\def\s{\sigma}
\def\vp{\varphi}
\def\D{\Delta}
\def\E{{\mathbb E}}
\def\K{{\bf K}}
\def\L{{\bf L}}
\def\Pt{\tilde{P}}
\def\R{{\mathbb R}}
\def\S{{\bf S}}
\def\U{{\bf U}}
\def\W{{\bf W}}

\usepackage{todonotes}
\newcounter{todocounter}
\newcommand{\ps}[1]
{\stepcounter{todocounter}
	\todo[color=green!40,author=Khurram, inline]{\thetodocounter: #1}
}

\newcommand{\khurram}[1]{\stepcounter{todocounter}
	{\color{blue!90} Khurram: \thetodocounter: #1}}

\begin{document}

\title{Unbiased Scalable Online Recurrent Learning}
%\title{Recurrent Neural Network Architectures \\ for Scalable Online Learning }
%\title{Scalable Online Recurrent Learning }
%\author{\name Marina Meil\u{a} \email mmp@stat.washington.edu \\
     %  \addr Department of Statistics\\
     %  University of Washington\\
     %  Seattle, WA 98195-4322, USA
    %   \AND
    %   \name Michael I.\ Jordan \email jordan@cs.berkeley.edu \\
    %   \addr Division of Computer Science and Department of Statistics\\
    %   University of California\\
    %   Berkeley, CA 94720-1776, USA}
%
\editor{Kevin Murphy and Bernhard Sch{\"o}lkopf}

\maketitle

\begin{abstract}%   <- trailing '%' for backward compatibility of .sty file
Online scalable recurrent learning is challenging. Two popular gradient-based methods for recurrent learning are BPTT, and RTRL. BPTT looks at the complete sequence before computing gradients, and is unsuitable for online updates. RTRL can do online updates, but generally scales poorly with an increase in the number of parameters. In this paper, we propose two constraints that make RTRL scalable. We show that by either decomposing the network into independent modules, or learning a recurrent network incrementally and selectively, we can make RTRL scale linearly with the number of parameters without introducing bias, or increasing the variance of the gradient estimates. The two approaches result in different algorithms, that can be combined. We show the strengths and weaknesses of the proposed algorithms on two online prediction problems.
\end{abstract}

\begin{keywords}
 Agent-state construction, online learning, scalable learning, recurrent learning, representation learning
\end{keywords}

\section{Introduction}
Learning by interacting with the world is a powerful framework for building systems that can autonomously achieve goals in complex worlds. A key ingredient for building autonomous systems is agent-state construction---learning a compact representation of the history of interactions that helps in predicting and controlling the future. One solution for state construction is to use differentiable recurrent neural networks (RNNs).

State construction using RNNs requires structural credit assignment---identifying how to change network parameters to improve predictions. In RNNs, a parameter can influence  a prediction made many steps in the future. As a result, effective credit assignment requires tracking the influence of parameters on future predictions. Two popular algorithms for gradient-based structural credit assignment in recurrent neural networks are Back-propagation through time (BPTT)~(Werbos, 1988; Robinson and Fallside, 1987)  and real-time recurrent learning (RTRL)~(Williams and Zipser~1989).

BPTT and RTRL are not suitable for online state construciton for large problems. BPTT requires storing all past activations for estimating the gradient. Additionally, it requires computation proportional to the length of the sequence seen so far. This makes BPTT unsuitable for online learning on long or never-ending streams of data. RTRL can estimate the gradient on the go, and does not require more computation as the length of the sequence grows. However, RTRL scales pooly with an increase in number of parameters of the RNN. Both BPTT and RTRL can be approximated, to make them more suitable for online learning.

Existing approximations to gradient-based learning approximate the estimate of the gradient, but keep the function class of the network the same. These approximations either introduce bias, which can result in poor performance or even divergence, or increase the variance of the estimate, making learning extremely slow. In this work, we propose a different strategy; instead of introducing bias in the estimate of the gradient, or increase the variance of the estimate, we instead limit the function class of the RNNs in a way that allows us to compute unbiased low-variance gradient estimate in a scalable way. Limiting the function class also introduces bias, however since learning is still done using the correct gradient, our method is less susceptible to instablity in learning.

\section{Background}
\subsection{Recurrent learning}
\begin{equation}
y(t) = \sum_{k=0}^{d} h_k(t)w_k(t)
\end{equation}
where
\begin{equation}
h(t) = f_{\theta(t)}\left(h(t-1), x(t)\right)
\end{equation}
here $\theta(t)$ are the parameters of the RNN at time $t$, whereas $f_\theta$ is a function that represents the RNN. Given this notation, we can write the gradient of a prediction $y(t)$ w.r.t the parameters $\theta$ as
\begin{equation}
\begin{aligned}
\frac{\partial{y(t)}}{\partial{\theta}} & = \frac{\partial{y(t)}}{\partial{h(t)}}\frac{\partial{h(t)}}{\partial{\theta}}   \\
\end{aligned}
\end{equation}
We can expand the second term as:
\begin{equation}
\frac{\partial{h(t)}}{\partial{\theta}} =  \frac{\partial{h(t)}}{\partial{\theta(t)}} + \frac{\partial{h(t)}}{\partial{h(t-1)}} \frac{\partial{h(t-1)}}{\partial{\theta}}
\label{expansion}
\end{equation}
to get
\begin{equation}
\frac{\partial{y(t)}}{\partial{\theta}}  = \frac{\partial{y(t)}}{\partial{h(t)}}\left(\frac{\partial{h(t)}}{\partial{\theta(t)}} + \frac{\partial{h(t)}}{\partial{h(t-1)}} \frac{\partial{h(t-1)}}{\partial{\theta}}  \right)
\label{expanded}
\end{equation}
Note that we can use equation~\ref{expansion} to recursively expand equation~\ref{expanded}, until we can get rid of $\partial \theta$ term.
BPTT estimates the gradient of $y(t)$ by repeatedly using equation~\ref{expansion}.

BPTT extends the back-propagation algorithm for feed-forward networks---independently proposed by Werbos~(1974) and Rumelhart~\etal~(1986)---to RNNs by storing network activations from prior steps, and repeatedly applying the chain-rule starting from the output of the network and ending at the activations at the beginning of the sequence. More precisely, we can write the gradient of a prediction $y(t)$ w.r.t the parameters $\theta$ as:

where $h(t)$ is the internal state of the network at time $t$. Then, BPTT comput
BPTT is unsuitable for online learning as it requires memory proportional to the length of the sequence. Moreover, it delays gradient computation until the end of the sequence. For online learning, this sequence can be arbitrarily long or never-ending.
\begin{algorithm}
\caption{Back-propagation through time}\label{alg:cap}
\begin{algorithmic}
	\Require $x_1, x_2, x_3, x_4, \cdots, x_n$
	\Require $h_0$
	\Require $\theta_1$
	\State $X \gets x$
	\State $N \gets n$
	\For{$t$ in $1, 2, 3, \cdots, n$}
		\State $h_t = f_{\theta_t}(h_{t-1}, x_t)$
	\EndFor
	\State $y_t = \sum_{k=0}^{d} w_k h_k^t$
	\State grad = 0
	\For{$t$ in $n, n-1, \cdots, 1$}
	\State
	\EndFor

\end{algorithmic}
\end{algorithm}

\begin{algorithm}
	\caption{Real-time recurrent learning}\label{alg:cap}
	\begin{algorithmic}
		\Require $n \geq 0$
	\end{algorithmic}
\end{algorithm}

% The dependence of BPTT on storing activations from past hinders its application to online learning. Additionally, the computation in BPTT is not spread uniformly across time --- the learner accumulates activations for a sequence in the memory and delays the computation of the gradients until the end. This is incompatible with the goal of real-time learning --- the ability to incorporate feedback from the environment quickly for correcting mistakes.

RTRL---an alternative to BPTT---was proposed by Williams and Zipser~(1989). RTRL relies on forward-mode differentiation---using chain-rule to compute gradients in the direction of time---to compute gradients recursively. Unlike BPTT, RTRL does not delay gradient-computation until the final step. The memory requirement of RTRL also does not depend on the sequence length. As a result, it is more suitable for real-time online learning. Unfortunately, RTRL requires maintaining the Jacobian $\frac{\partial h(t)}{\partial \theta}$ at every step, which requires $O(|h||\theta|)$ memory, where $|h|$ is the size of state of the network and $|\theta|$ is the number of total parameters. The Jacobian is recursively updated by applying chain rule as:
$$\frac{\partial h(t+1)}{\partial \theta} =\frac{\partial h(t+1)}{\partial \theta(t+1)} +  \frac{\partial h(t+1)}{\partial h(t)}\frac{\partial h(t)}{\partial \theta}, $$
which requires  $O(|h|^2|\theta|)$ operations and scales poorly to large networks.

% For a more thorough explanation of the memory and time complexity of recurrent state-learning and meta-learning methods using RTRL, see Appendix A.

A promising direction to scale gradient-based credit-assignment to large networks is to approximate the gradient. Elman~(1990) proposed to ignore the influence of parameters on future predictions completely for training RNNs. This resulted in a scalable but biased algorithm. Williams and Peng~(1990) proposed a more general algorithm called Truncated BPTT (T-BPTT). T-BPTT tracks the influence of all parameters on predictions made up to $k$ steps in the future. T-BPTT limits the BPTT computation to last $k$ steps, and works well for a range of problems~(Mikolov~\etal, 2009, 2010; Sutskever, 2013 and Kapturowski~\etal, 2018). Its main limitation is that the resultant gradient is blind to long-range dependencies. Mujika~\etal~(2018) showed that on a simple copy task, T-BPTT failed to learn dependencies beyond the truncation window. Tallec~\etal~(2017) demonstrated T-BPTT can even diverge when a parameter has a negative long-term effect on a target and a positive short-term effect.

\begin{figure*}
	\centering
	\includegraphics[width=0.95\textwidth]{figures/three_types}
	\caption{TODO: Make the caption concise. Three structures of recurrent neural networks that can be trained without truncation. Recurent networks with a columnar structure can be trained end-to-end using gradients without any truncation, only requiring $O(n)$ operations and memory per step. However, columnar networks do not have hierarchical recurrent features---recurrent features made out of other recurrent features. Constructive networks have hierarchical recurrent features, however must be trained incrementally to prevent bias. Incremental learning is achieved by initializing all $w_i$ to zero, and learning $h_1$, $h_2$, and $h_3$ in order. Finally, columnar and constructive networks can be combined to get a hybrid network. The pairs $(h_1, h_2)$ and $(h_3, h_4)$ do not depend on each other, and can learn in parallel. Hoever, $(h_3, h_4)$ must be learned after $(h_1, h_2)$ have been learned and fixed.}
\end{figure*}

RTRL can also be approximated to reduce its computational overhead. Ollivier~\etal~(2015) and Tallec~\etal~(2017) proposed NoBacktrack and UORO. Both of these algorithms provide stochastic unbiased estimates of the gradient and scale well. However, their estimates have high variance and require extremely small step sizes for effective learning. Cooijmans, Martens~(2019) and Menick~\etal~(2021) showed that, for practical problems, UORO does not perform well compared to other biased approximations due to the high variance in its gradient estimates . Menick~\etal~(2021) proposed an approximation to RTRL called SnAp-$k$. SnAp-$k$ tracks the influence of a parameter on a state only if the parameter can influence the state within $k$ steps. It first identifies parameters whose influence on a state is zero for $k$ steps and then assumes the future influence to be zero as well. For the remaining parameters, it tracks their influence on all future predictions. The bias introduced by SnAp-k is fundamentally different than the bias introduced by T-BPTT. SnAp-1 can be computationally efficient but introduces significant bias. SnAp-$k$ for $k>1$ reduces bias but can be as expensive as RTRL for dense RNNs.  Menick~\etal~(2021) further proposed using sparse connections as a way to make SnAp more scalable. Connection sparsity reduces the number of parameters that can influence a state within $k$ steps.

Among all the approximations, SnAp-1 is the only method that scales linearly with number of parameters, is capable of learning long-term dependencies, and

\subsection{General value functions and }
\section{Problem Formulation}
We formulate the problem as predicting the discounted sum of a cumulant from an online stream of data. More formally, the agent sees a feature vector $x(t)\in \mathcal{R}^d$ at timestep $t$ and predicts the discounted sum of a cumulant $y_t$ given by:
\begin{equation}
\begin{aligned}
V(S_t) & = \mathbb{E}[G_t | S_t] \\
& = \mathbb{E}[\sum_{k = 0}^{\infty} \gamma^k y(k)]
\label{eq:return}
\end{aligned}
\end{equation}
This problem formulation can be used to represent temporal predictions, such as policy evaluation and general value functions. Additionally, by setting $\gamma = 0$, our problem formulation can represent classical supervised recurrent learning benchmarks. The goal of the agent is to minimize the mean squared error between the prediction $\hat{V}(x_{1:t})$ and $V(S)$. Note that for all our experiments, the data generation policy $\pi$ is fixed and can be ignored.

\subsection{Animal Learning Benchmarks}\label{animal_bench}
A suitable benchmark that fits this problem formulation is the trace patterning benchmark introduced by \cite{Rafiee2020TestbedsFR}. In this benchmark, we are provided with a set of binary stimuli $o_t \in \R^d$ at each timestep t, which includes a set of $n$ Conditioned Stimuli CS$_t $, one Unconditioned Stimuli US$_t$ and $m$ distractor signals. There are $k$ random sets of  configurations of CSs that will activate the US. If the active values of CS$_t$ match one of these activation patterns, the value US$_{t+\text{ISI}}$ will be set to 1, where ISI (\textit{inter-stimulus interval}) is a problem hyperparameter that is uniformly sampled from a certain range after every occurence of the US. In this problem, the goal is to learn eq. \ref{eq:return} using TD error where
\begin{equation}
	y(k) = \text{US}_{t+k+1}
\end{equation}

This benchmark is suitable for our evaluations since in order to do well, the learned model not only has to learn long-term temporal associations, it also has to identify the correct patterns of CSs that activate the US. Moreover, since this is a continuous online prediction problem, the algorithms need to scale well with time.

TODO: maybe we should also describe gamma here instead of the empirical evaluation section since it is also a problem parameter

TODO: maybe we should describe the ITI and how the probability of activation works.

\section{Divergence for truncated-backprop and Snap-1}
Because truncated backprop and Snap-1 use biased estimates of the gradients for learning updates, they can cause the prediction error and parameters to diverge under pathological cases. We present examples of divergence for both truncated-bptt and SnAp-1. In real-world case, such divergence happens rarely, if ever. However the examples still serve a useful pedagogical role, to clarify the limitation of the two algorithms.

For truncated backprop, we use a simple recurrent network with one hidden unit, one feature, and one prediction. The recurrence is defined as adding the value of the last hidden state to the current state. We fix the weight for the recurrent connection and the outgoing connection, and learn only the incoming weight. The optimal solution is $w_{in} = -10$. We let the recurrent network learn for a thousand steps using a step size of $10^{-3}$ and show the behavior pf th elearning algorithm in Figure~. Note that the learning
\section{Proposed methods}
We propose imposing a specific structure on either the connections of the recurrent network, or the learning algorithms, to achieve scalable learning without truncation. We propose columnar networks to ac

\section{Empirical evaluation}

\begin{figure*}[t]
	\centering
	% More versions of the left figure available in the figures folder
	\includegraphics[width=0.48\textwidth]{figures/plt_animal_preds400_tight_withaxlabel.eps}
	\includegraphics[width=0.48\textwidth]{figures/plt_animal.eps}
	\label{animal_results}
	\caption{ Left: Comparison of the predictions produced by the different methods for the last 400 timesteps of training. Right: Performance when an LSTM is trained using the proposed and other online baseline methods on the trace patterning benchmark (lower is better). All the runs are averaged using 30 seeds, and the error regions are the 95\% confidence intervals. T-BPTT uses a truncation length k=27. We can see from this plot that the Diagonal Jacobian method performs the worst while the hybrid networks achieve the lowest error. TODO: T-BPTT is only 15 runs while the remaining curves are 30 runs. TODO: Maybe hybrid networks should get a better name so that people can refer to it easily?}
\end{figure*}
We first evaluate the performance of the proposed methods and baselines on the trace patterning benchmark described in section \ref{animal_bench}. In these experiments, we set $\text{ISI} = \text{Unif}(14,26)$, $\g = \frac{1}{1-\E[ISI]}$, $n=6$ CSs, $k=10$ activation patterns with a $0.5$ probability of activation and $m=5$ distractor signals. The details of the remaining hyper-parameters and their tuning is given in the Appendix \ref{hyperparams}.

All of these methods are used to train a network with a single LSTM hidden layer, followed by a single output unit which predicts $y(k)$. The number of hidden recurrent units are adjusted for each method in order to ensure a fair comparison. This adjustment is done in order to ensure that the amount of computational resources consumed per step is the same for all the methods. A consequence of this constraint is that in the case of T-BPTT, we have to use a relatively smaller network since we need multiple passes to calculate the intermediate gradients for the computation of the eligibility traces. Despite this, we have found from our experiments that there is no significant difference in the performance of T-BPTT as compared to using a larger hidden state than we are using.

For the T-BPTT experiments, we used a truncation length of k=27 and a hidden size of  18.  This truncation length essentially makes the T-BPTT equivalent to BPTT since $k>\text{max(ISI)}$. The columnar network has 108 independent recurrent units in the hidden layer. The constructive networks begin with a single recurrent feature in the hidden layer and add a new feature every 400,000 steps. By the end of the training, this network ends up with a total of 23 recurrent features. The hybrid approach begins with 20 recurrent features in the hidden layer, and adds 20 features every 800,000 steps, until it reaches a maximum of 108 features. TODO: How should we describe Diagonal Jacobian here?

The results of the experiment are shown in Fig. \ref{animal_results}. We can see that the Diagonal Jacobian and the T-BPTT methods perform the worst on this benchmark. The columnar and the constructive approaches achieve a significantly lower error than these baselines. Furthermore, these two methods achieve an asymptotic performance that is similar to each other. Finally, the hybrid approach, which is a combination of the constructive and the columnar approaches, achieves the lowest asymptotic error out of all the methods presented.

From these experiments, we have shown that despite our methods limiting the class of the function approximators that we can use, the unbiased and low-variance gradient updates allow these methods to achieve a potentially better performance in the challenging online prediction tasks, while still maintaining linear scalability with time.

TODO: The T-BPTT curve here is not using eligibility traces. Should the size still be adjusted?

TODO: Where are the plots for columnar with 23 total features stored? I should plot those instead maybe.\\
\khurram{They are in khurram\_constructive\_nomralized\_2. Use the features column to select the size of the network}

\section{Temporally symmetric hierarchical recurrent learning using generate and test}
The primary limitation of the constructive and hybrid approaches is that they are not suitable for continual learning. Once a feature is frozen, it never changes, even if the environment does. In the constructive case, the agent can, at best, continually update one recurrent feature. In the hybrid case, the agent can update $k$ recurrent units added last. Inability to update all features after a certain number of time steps  make the agent lose the ability to  modify its feature representation. In a sense, the agent becomes a linear learner.

We can address this limitation by identifying features that are no longer useful, and pruning them. These pruned features can then be replaced with newly constructed features. Similar ideas have been proposed in literature under the term generate and test.

To effectively combine our method with generate and test, we need to answer two questions---(1) how useless features are identified, and (2) how new features replace them. It is unclear what an ideal algorithm to answer these two questions are. In this work, we limit ourselves to the simple algorithms for both questions. The goal here is to present one temporally symmetric version of the hybrid approach, and not to figure out the best temporally symmetric version.

To identify useless features, we simply look at how much a feature contributes to its outgoing connection, and use the sum of contribution to all outgoing connections as the utility of the feature. The contribution of a feature to an outgoing connection is defined as the

%  SnAp, combined with sparsity, is a promising research direction.

%Menick~\etal~(2021) showed that for highly sparse networks, SnAp-$k$ reduces the computational requirement of RTRL by over 95\% while keeping bias in check. One limitation of their work is that they do not provide a scalable method to identify parameters that would not influence a state within $k$ steps. Instead, they run the RNN for $k$ steps and look at the full Jacobian $\frac{\partial h(t)}{\partial \theta}$ to determine these parameters. For large networks, computing this Jacobian even once is not possible. Moreover, because they induce sparsity randomly in their networks, the resultant sparsity in the Jacobian is not structured and is not amenable to efficiency gains using existing hardware. Finally, SnAp does not scale well when used in conjunction with deep feature extractors. If internal states operate on a shared representation computed with a deep network with parameters $\theta$, even SnAp-1 could require $O(|\theta||h|)$ memory and $O(|\theta||h|^2)$ operations per-step. Our goal is to design an algorithm that requires $O(|\theta|)$ memory and operations.

% Acknowledgements should go at the end, before appendices and references

\section{Conclusions}
Use hybrid networks.

\acks{We would like to acknowledge support for this project
from the National Science Foundation (NSF grant IIS-9988642)
and the Multidisciplinary Research Program of the Department
of Defense (MURI N00014-00-1-0637). }

% Manual newpage inserted to improve layout of sample file - not
% needed in general before appendices/bibliography.

\newpage

\appendix

\section*{Appendix A.}
\section{Hyper-parameter Settings}
We performed an exhaustive search over all the hyper-parameter ranges for each of the methods that is presented in this paper. For each hyper-parameter configuration, we averaged its performance over at-least three runs and determined the best setting based on the lowest average final error. The sweep ranges for the different hyper-parameters are provided in Table \ref{tab:sweep-ranges} and the best hyper-parameters are provided in Table \ref{tab:best-params}.

\begin{table}[h]
	\centering
	\begin{tabular}{@{}cllcccc@{}}
		\toprule
		 &  Hyper-parameter & Sweep Ranges\\ \midrule
		$\a$  & Step-size     & $\{3^{-1}, 1^{-1}, 3^{-2}, 1^{-2}, 3^{-3}, 3^{-4}\}$ \\
		$\l$ & Eligibility Traces Decay Rate     & $\{0, 0.5, 0.9, 0.99\}$               \\
		$k$    & Truncation Length (T-BPTT)   & $\{1,10,27,50\}$   \\
		$n$& Number of Initial Features (Hybrid) & \{5, 10, 20\} \\
		$\eta$ & Feature Addition Interval (Constructive/Hybrid)    & $x*10,000 ;~  x \in \{4, 10, 20, 40, 80\}$ \\
		\bottomrule
	\end{tabular}
	\vspace{0.15cm}
	\caption{Sweep ranges for each hyper-parameter. TODO: The notation for last two rows is just placeholder. Replace it with something appropriate.}
	\label{tab:sweep-ranges}
\end{table}

\begin{table}[h]
	\centering
	\begin{tabular}{@{}cccccc@{}}
		\toprule
		Hyper-parameter &  T-BPTT & Diagonal Jacobian & Columnar &Constructive & Hybrid\\ \midrule
		$\a$     & $1^{-1}$					& $1^{-2}$		& $1^{-1}$ 	& $1^{-2}$	      &   $1^{-2}$     \\
		$\l$       & 0 				 		 & 0.99 					& 0.9 				& 0.99				&		0.99	\\
		$k$       & 27 						& \textbf{-}  		& \textbf{-} 	& \textbf{-} 	&		\textbf{-} 	\\
		$n$        & 18						& 108					& 108 				& 1						& 		20 	\\
		$\eta$        & \textbf{-} 	& \textbf{-}		& \textbf{-}	 & 400,000		& 		800,000	\\
		\bottomrule
	\end{tabular}
	\vspace{0.15cm}
	\caption{The best hyper-parameter configuration found for each method according to the average final error over multiple runs. Note that although we specify the $n$ parameter for T-BPTT, Diagonal Jacobian and Columnar approaches, no new feature will be added during their training. It is simply the width of the network. }
	\label{tab:best-params}
\end{table}

\section{Implementation Details}
We implemented the Diagonal Jacobian, Columnar, Constructive and the Hybrid Networks in C++. T-BPTT is implemented using Pytorch. The code for replicating these results (along with an optional python wrapper for the C++ code with python environment) is released at \href{https://github.com/khurramjaved96/constructive\_rnn}{https://github.com/khurramjaved96/constructive\_rnn}

%setcounter{section}{1}
\section*{Appendix B.}

\section{Forward-mode gradient computation of an LSTM cell }

Here we derive the update equations for recursively computing the gradients of a single LSTM based recurrent column. Each column has a single hidden unit. Because all columns are identical, the same update equations can be used to learn the complete Columnar Network.  The state of an LSTM column is updated using following equations:
% \begin{equation}
% \label{lstm}
\begin{align}
i(t) &= \sigma( W_{i}^T x_k(t) + u_{i} h(t-1) + b_i) \label{i} \\
f(t) &= \sigma( W_{f}^T x_k(t) + u_{f} h(t-1) + b_f) \label{f} \\
o(t) &= \sigma( W_{o}^T x_k(t) + u_{o} h(t-1) + b_o) \label{o} \\
g(t) &= \phi( W_{g}^T x_k(t) + u_{g} h(t-1) + b_g) \label{g} \\
c(t) &= f(t)  c(t-1) + i(t) g(t) \label{c} \\
h(t) &= o(t)  \phi(c(t)) \label{state_update}
\end{align}
% \end{equation}
where $\sigma$ and $\phi$ are the sigmoid and tanh activation functions, $h(t)$ is the state of the column at time $t$ and $ W_{i}^T x_k(t) = \sum_{k=1}^m W_{i_k} x_k(t)$. The derivative of $\sigma(x)$ and $\phi(x)$ w.r.t to $x$ are $\sigma(x)(1-\sigma(x))$ and $(1-\phi^2(x))$ respectively.

Let the length of input vector $x$ be $m$. Then, $W_{i}, W_{f}, W_{o}$ and $W_{g}$ are vectors of length $m$ whereas $u_i, b_i, u_f, b_f, u_o, b_o, u_g$ and $b_g$ are scalars. We want to compute gradient of $h(t)$ with respect to all the parameters. We derive the update equations for $\frac{\partial h(t)}{\partial W_{i}} ,\frac{\partial h(t)}{\partial u_{i}}, $$\frac{\partial h(t)}{\partial b_{i}}, \frac{\partial h(t)}{\partial W_{f}} $$,\frac{\partial h(t)}{\partial u_{f}}, \frac{\partial h(t)}{\partial b_{f}}, $$\frac{\partial h(t)}{\partial W_{o}} ,\frac{\partial h(t)}{\partial u_{o}},$$ \frac{\partial h(t)}{\partial b_{o}}, \frac{\partial h(t)}{\partial W_{g}} ,\frac{\partial h(t)}{\partial u_{g}},$ and $\frac{\partial h(t)}{\partial b_{g}}$ in the following sections.

\subsection{$\frac{\partial h(t)}{\partial W_{i}}$}

$W_i = (W_{i_1}, W_{i_2}, \cdots, W_{i_m})$ is a vector of length $m$. Since all elements of $W_i$ are symmetric, we show gradient derivation for $W_{i_j}$ without loss of generality. Let
% \begin{equation}
\begin{align}
TH_{W_{i_j}}(t)  & \vcentcolon = \frac{\partial h(t)}{\partial W_{i_j}}  && \textit{(By definition)} \label{def1} \\
TH_{W_{i_j}}(0)  & \vcentcolon = 0  && \textit{(By definition)} \label{def2} \\
TC_{W_{i_j}}(t)  & \vcentcolon = \frac{\partial c(t)}{\partial W_{i_j}}  && \textit{(By definition)} \label{def3} \\
TC_{W_{i_j}}(0)  & \vcentcolon = 0  && \textit{(By definition)}  \label{def4}
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial}{\partial W^{ij}_z}\left (  z^k(t)  \hat{h}^k(t) \right )  && \textit{Using definition 5}\\
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial \hat{h}^k(t)}{\partial W^{ij}_z} + \hat{h}^k(t) \frac{\partial z^k(t)}{\partial W^{ij}_z}  && \textit{test} \textit{Applying chain rule}
\end{align}
% \end{equation}
Then:
% \begin{equation}
\begin{align*}
TH_{W_{i_j}}(t) & = \frac{\partial}{\partial W_{i_j}} \left ( o(t) \phi(c(t)) \right ) && \textit{From equation \ref{state_update} and definition \ref{def1}} \\
& = o(t) \frac{\partial \phi(c(t))}{\partial W_{i_j}}  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{i_j}}   && \textit{Product rule of differentiation} \\
&= o(t) (1-\phi^{2}(c(t))) \frac{\partial c(t)}{\partial W_{i_j}}  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{i_j}}   && \textit{Derivative of $\phi(x)$ is (1-$\phi^2(x))$}\\
&= o(t) (1-\phi^{2}(c(t))) TC_{W_{i_j}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{i_j}}   && \textit{From definition \ref{def3}}\\
\frac{\partial o(t)}{\partial W_{i_j}} &=  \frac{\partial}{\partial W_{i_j}} \sigma(W_{o}^T x(t) + u_{o} h(t-1) + b_o) && \textit{From equation \ref{o}}\\
&= \sigma(y)(1-\sigma(y))u_o TH_{W_{i_j}}(t-1) && \textit{Where $y$ equals $W_{o}^T x(t) + u_{o} h(t-1) + b_o$} \\
TC_{W_{i_j}}(t) &=  \frac{\partial c(t)}{\partial W_{i_j}} && \textit{From definition \ref{def3}} \\
&=  \frac{\partial}{\partial W_{i_j}} (f(t)  c(t-1) + i(t) g(t))  && \textit{From equation \ref{c}} \\
&= f(t) TC_{W_{i_j}}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial W_{i_j}} + \frac{\partial}{\partial W_{i_j}} ( i(t) g(t) )   && \textit{Product rule and definition \ref{def3}}\\
&= f(t) TC_{W_{i_j}}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial W_{i_j}} + i(t)\frac{\partial g(t) }{\partial W_{i_j}}  + g(t)\frac{\partial i(t) }{\partial W_{i_j}}  && \textit{Product rule}
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial}{\partial W^{ij}_z}\left (  z^k(t)  \hat{h}^k(t) \right )  && \textit{Using definition 5}\\
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial \hat{h}^k(t)}{\partial W^{ij}_z} + \hat{h}^k(t) \frac{\partial z^k(t)}{\partial W^{ij}_z}  && \textit{test} \textit{Applying chain rule}
\end{align*}

% \end{equation}

Where gradient of $g(t)$  w.r.t $W_{i_j}$ is:
% \begin{equation}
\begin{align*}
\frac{\partial g(t)}{\partial W_{i_j}} &=  \frac{\partial}{\partial W_{i_j}} \phi(W_{g}^T x(t) + u_{g} h(t-1) + b_g) && \textit{From equation \ref{g}} \\
&= (1-\phi^2(y))u_g TH_{W_{i_j}}(t-1) && \textit{Where $y$ equals $W_{g}^T x(t) + u_{g} h(t-1) + b_g$} \\
\end{align*}
% \end{equation}
, gradient of $f(t)$  w.r.t $W_{i_j}$ is:
% \begin{equation}
\begin{align*}
\frac{\partial f(t)}{\partial W_{i_j}} &=  \frac{\partial}{\partial W_{i_j}} \sigma(W_{f}^T x(t) + u_{f} h(t-1) + b_f) && \textit{From equation \ref{f}} \\
&= \sigma(y)(1-\sigma(y))u_f TH_{W_{i_j}}(t-1) && \textit{Where $y$ equals $W_{f}^T x(t) + u_{f} h(t-1) + b_f$} \\
\end{align*}
% \end{equation}
and gradient of $i(t)$ w.r.t $W_{i_j}$ is:
% \begin{equation}
\begin{align*}
\frac{\partial i(t)}{\partial W_{i_j}} &=  \frac{\partial}{\partial W_{i_j}} \sigma(W_{i}^T x(t) + u_{i} h(t-1) + i_f) && \textit{From equation \ref{i}} \\
&= \sigma(y)(1-\sigma(y)) \left (x_j(t)+ u_i TH_{W_{i_j}}(t-1) \right )&& \textit{Where $y$ equals $W_{i}^T x(t) + u_{i} h(t-1) + b_i$} \\
\end{align*}
% \end{equation}
The derivation shows that using two traces per parameter of $W_i$, it is possible to compute the gradient of $h(t)$ w.r.t $W_i$ recursively. We provide the derivations for parameters $u_i$ and $b_i$ below. We skip the step-by-step derivations for the remaining parameters as they are similar.

\subsection{$\frac{\partial h(t)}{\partial u_i}$}

% \begin{equation}
\begin{align}
TH_{u_{i}}(t)  & \vcentcolon = \frac{\partial h(t)}{\partial u_i}  && \textit{(By definition)}  \label{thu}\\
TH_{u_{i}}(0)  & \vcentcolon = 0  && \textit{(By definition)}  \\
TC_{u_{i}}(t)  & \vcentcolon = \frac{\partial c(t)}{\partial u_i}  && \textit{(By definition)}  \label{tcu} \\
TC_{u_{i}}(0)  & \vcentcolon = 0  && \textit{(By definition)}
\end{align}
% \end{equation}

% \begin{equation}
\begin{align*}
TH_{u_i}(t) & = \frac{\partial}{\partial u_i} \left ( o(t) \phi(c(t)) \right ) && \textit{From equation \ref{state_update}} \\
& = o(t) \frac{\partial \phi(c(t))}{\partial u_i}  + \phi(c(t)) \frac{\partial o(t)}{\partial u_i}   && \textit{Product rule} \\
&= o(t) (1-\phi^{2}(c(t))) \frac{\partial c(t)}{\partial u_i}  + \phi(c(t)) \frac{\partial o(t)}{\partial u_i}   && \textit{Derivative of $\phi(x)$ is $1-\phi^2(x)$} \\
&= o(t) (1-\phi^{2}(c(t))) TC_{u_{i}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial u_i}   && \textit{Using definition \ref{tcu}} \\
\frac{\partial o(t)}{\partial u_i} &=  \frac{\partial}{\partial u_i} \sigma(W_{o}^T x(t) + u_{o} h(t-1) + b_o) && \textit{Using equations \ref{o}}\\
&= \sigma(x)(1-\sigma(x))u_o TH_{u_i}(t-1) && \textit{Where $x$ equal $W_{o}^T x(t) + u_{o} h(t-1) + b_o$} \\
TC_{u_i}(t) &=  \frac{\partial c(t)}{\partial u_i} && \textit{Definition \ref{tcu}} \\
&=  \frac{\partial}{\partial u_i} (f(t)  c(t-1) + i(t) g(t))  && \textit{From equation \ref{state_update}} \\
&= f(t) TC_{u_i}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial u_i} + \frac{\partial}{\partial u_i} \left ( i(t) g(t) ) \right )  && \textit{Product rule}\\
&= f(t) TC_{u_i}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial u_i} + i(t)\frac{\partial g(t) }{\partial u_i}  + g(t)\frac{\partial i(t) }{\partial u_i}  && \textit{Product rule}
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial}{\partial W^{ij}_z}\left (  z^k(t)  \hat{h}^k(t) \right )  && \textit{Using definition 5}\\
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial \hat{h}^k(t)}{\partial W^{ij}_z} + \hat{h}^k(t) \frac{\partial z^k(t)}{\partial W^{ij}_z}  && \textit{test} \textit{Applying chain rule}
\end{align*}
% \end{equation}
Gradient of  $g(t)$  w.r.t  $u_i$ is:
% \begin{equation}
\begin{align*}
\frac{\partial g(t)}{\partial u_i} &=  \frac{\partial}{\partial u_i} \phi(W_{g}^T x(t) + u_{g} h(t-1) + b_g) && \textit{From equations \ref{state_update}} \\
&= (1-\phi^2(y))u_g TH_{u_i}(t-1) && \textit{Where $y$ equals $W_{g}^T x(t) + u_{g} h(t-1) + b_g$} \\
\end{align*}
% \end{equation}
, gradient of $f(t)$  w.r.t $u_i$ is:
% \begin{equation}
\begin{align*}
\frac{\partial f(t)}{\partial u_i} &=  \frac{\partial}{\partial u_i} \sigma(W_{f}^T x(t) + u_{f} h(t-1) + b_f) && \textit{From equations 1} \\
&= \sigma(y)(1-\sigma(y))u_f TH_{u_i}(t-1) && \textit{Where $y$ equals $W_{f}^T x(t) + u_{f} h(t-1) + b_f$} \\
\end{align*}
% \end{equation}

and the gradient of $i(t)$  w.r.t $u_i$ is
% \begin{equation}
\begin{align*}
\frac{\partial i(t)}{\partial u_i} &=  \frac{\partial}{\partial u_i} \sigma(W_{i}^T x(t) + u_{i} h(t-1) + b_i) && \textit{Using equations 1} \\
&= \sigma(y)(1-\sigma(y)) \left (h(t-1)+ u_i TH_{u_{i}}(t-1) \right )&& \textit{Where $y$ equals $W_{i}^T x(t) + u_{i} h(t-1) + b_i$} \\
\end{align*}
% \end{equation}

\subsection{$\frac{\partial h(t)}{\partial b_i}$}

% \begin{equation}
\begin{align}
TH_{b_i}(t)  & \vcentcolon = \frac{\partial h(t)}{\partial b_i}  && \textit{(By definition)}  \label{thb}\\
TH_{b_i}(0)  & \vcentcolon = 0  && \textit{(By definition)}  \\
TC_{b_i}(t)  & \vcentcolon = \frac{\partial c(t)}{\partial b_i}  && \textit{(By definition)}  \label{tcb}\\
TC_{b_i}(0)  & \vcentcolon = 0  && \textit{(By definition)}
\end{align}
% \end{equation}

% \begin{equation}
\begin{align*}
TH_{b_i}(t) & = \frac{\partial}{\partial b_i} \left ( o(t) \phi(c(t)) \right ) && \textit{From equation \ref{state_update}} \\
& = o(t) \frac{\partial \phi(c(t))}{\partial b_i}  + \phi(c(t)) \frac{\partial o(t)}{\partial b_i}   && \textit{Product rule} \\
&= o(t) (1-\phi^{2}(c(t))) \frac{\partial c(t)}{\partial b_i}  + \phi(c(t)) \frac{\partial o(t)}{\partial b_i}   && \textit{Derivative of of $\phi(x)$ is $1-\phi^2(x)$}\\
&= o(t) (1-\phi^{2}(c(t))) TC_{b_i}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial b_i}   && \textit{From definition \ref{tcb} }\\
\frac{\partial o(t)}{\partial b_i} &=  \frac{\partial}{\partial b_i} \sigma(W_{o}^T x(t) + u_{o} h(t-1) + b_o) && \textit{From equations \ref{o}}\\
&= \sigma(y)(1-\sigma(y))u_o TH_{b_i}(t-1) && \textit{Where $y$ equal $W_{o}^T x(t) + u_{o} h(t-1) + b_o$} \\
TC_{b_i}(t) &=  \frac{\partial c(t)}{\partial b_i} && \textit{From definition \ref{tcb}} \\
&=  \frac{\partial}{\partial b_i} (f(t)  c(t-1) + i(t) g(t))  && \textit{From equation \ref{c}} \\
&= f(t) TC_{b_i}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + \frac{\partial}{\partial b_i}  i(t) g(t)   && \textit{Product rule}\\
&= f(t) TC_{b_i}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  && \textit{Product rule}
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial}{\partial W^{ij}_z}\left (  z^k(t)  \hat{h}^k(t) \right )  && \textit{Using definition 5}\\
% %   & = (1-z^k(t)) T_{w_{z}}^{ijk}(t-1) + z^k(t) \frac{\partial \hat{h}^k(t)}{\partial W^{ij}_z} + \hat{h}^k(t) \frac{\partial z^k(t)}{\partial W^{ij}_z}  && \textit{test} \textit{Applying chain rule}
\end{align*}
% \end{equation}
Where gadient of  $g(t)$  w.r.t $b_i$ is:
\begin{align*}
\frac{\partial g(t)}{\partial b_i} &=  \frac{\partial}{\partial b_i} \phi(W_{g}^T x(t) + u_{g} h(t-1) + b_g) && \textit{From equation \ref{g}} \\
&= (1-\phi^2(y))u_g TH_{b_i}(t-1) && \textit{Where $y$ equal $W_{g}^T x(t) + u_{g} h(t-1) + b_g$} \\
\end{align*}

, gradient of $f(t)$  w.r.t $b_i$ is:
% \begin{equation}
\begin{align*}
\frac{\partial f(t)}{\partial b_i} &=  \frac{\partial}{\partial b_i} \sigma(W_{f}^T x(t) + u_{f} h(t-1) + b_f) && \textit{From equation \ref{f}} \\
&= \sigma(y)(1-\sigma(y))u_f TH_{b_i}(t-1) && \textit{Where $y$ equal $W_{f}^T x(t) + u_{f} h(t-1) + b_f$} \\
\end{align*}
% \end{equation}

and gradient of $i(t)$  w.r.t $b_i$ is:
% \begin{equation}
\begin{align*}
\frac{\partial i(t)}{\partial b_i} &=  \frac{\partial}{\partial b_i} \sigma(W_{i}^T x(t) + u_i h(t-1) + b_i) && \textit{From equation \ref{i}} \\
&= \sigma(y)(1-\sigma(y)) \left (u_i TH_{b_i}(t-1) + 1 \right )&& \textit{Where $y$ equal $W_{i}^T x(t) + b_i h(t-1) + b_i$} \\
\end{align*}
% \end{equation}

\subsection{$\frac{\partial h(t)}{\partial W_{f_j}}$}
The derivations for the remaining parameters is analogous to what previous derivations. The final equations are as follows.
\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial W_{f_j}} &=  (1-\phi^2(y))(u_g TH_{W_{f_j}}(t-1))  \\
\frac{\partial f(t)}{\partial W_{f_j}} &=  \sigma(y)(1-\sigma(y))( x_j + u_f TH_{W_{f_j}}(t-1)) \\
\frac{\partial i(t)}{\partial W_{f_j}}  &= \sigma(y)(1-\sigma(y))( u_i TH_{W_{f_j}}(t-1))  \\
\frac{\partial o(t)}{\partial W_{f_j}} &=  \sigma(y)(1-\sigma(y))( u_o TH_{W_{f_j}}(t-1))  \\
TC_{W_{f_j}}&= f(t) TC_{f_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{W_{f_j}}&= o(t) (1-\phi^{2}(c(t))) TC_{W_{f_j}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial W_{o_j}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial W_{o_j}} &=  (1-\phi^2(y))(u_g TH_{W_{o_j}}(t-1))  \\
\frac{\partial f(t)}{\partial W_{o_j}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{W_{o_j}}(t-1)) \\
\frac{\partial i(t)}{\partial W_{o_j}}  &= \sigma(y)(1-\sigma(y)) u_i TH_{W_{o_j}}(t-1)  \\
\frac{\partial o(t)}{\partial W_{o_j}} &=  \sigma(x)(1-\sigma(x))(x_j + u_o TH_{W_{o_j}}(t-1))  \\
TC_{W_{o_j}}&= f(t) TC_{o_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{W_{o_j}}&= o(t) (1-\phi^{2}(c(t))) TC_{W_{o_j}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial W_{g_j}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial W_{g_j}} &=  (1-\phi^2(y))(x_j + u_g TH_{W_{g_j}}(t-1))  \\
\frac{\partial f(t)}{\partial W_{g_j}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{W_{g_j}}(t-1)) \\
\frac{\partial i(t)}{\partial W_{g_j}}  &= \sigma(y)(1-\sigma(y))(u_i TH_{W_{g_j}}(t-1))  \\
\frac{\partial o(t)}{\partial W_{g_j}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{W_{g_j}}(t-1))  \\
TC_{W_{g_j}}&= f(t) TC_{g_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{W_{g_j}}&= o(t) (1-\phi^{2}(c(t))) TC_{W_{g_j}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

% \subsection{$\frac{\partial h(t)}{\partial W_{i_j}}$}

% \begin{equation}
%     \begin{aligned}
%       \frac{\partial g(t)}{\partial W_{i_j}} &=  \sigma(y)(1-\sigma(y))(u_g TH_{W_{i_j}}(t-1))  \\
%       \frac{\partial f(t)}{\partial W_{i_j}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{W_{i_j}}(t-1)) \\
%       \frac{\partial i(t)}{\partial W_{i_j}}  &= \sigma(y)(1-\sigma(y))(x_j + u_i TH_{W_{i_j}}(t-1))  \\
%         \frac{\partial o(t)}{\partial W_{i_j}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{W_{i_j}}(t-1))  \\
%       TC_{W_{i_j}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
%       TH_{W_{i_j}}&= o(t) (1-\phi^{2}(c(t))) TC_{W_{i_j}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
%     \end{aligned}
% \end{equation}

% \subsection{$\frac{\partial h(t)}{\partial u_{i}}$}

% \begin{equation}
%     \begin{aligned}
%       \frac{\partial g(t)}{\partial u_{i}} &=  \sigma(y)(1-\sigma(y))(u_g TH_{u_{i}}(t-1))  \\
%       \frac{\partial f(t)}{\partial u_{i}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{u_{i}}(t-1)) \\
%       \frac{\partial i(t)}{\partial u_{i}}  &= \sigma(y)(1-\sigma(y))(u_i TH_{u_{i}}(t-1) + h(t-1))  \\
%         \frac{\partial o(t)}{\partial u_{i}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{u_{i}}(t-1))  \\
%       TC_{u_{i}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
%       TH_{u_{i}}&= o(t) (1-\phi^{2}(c(t))) TC_{u_{i}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
%     \end{aligned}
% \end{equation}

\subsection{$\frac{\partial h(t)}{\partial u_{o}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial u_{o}} &=  (1-\phi^2(y))(u_g TH_{u_{o}}(t-1))  \\
\frac{\partial f(t)}{\partial u_{o}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{u_{o}}(t-1)) \\
\frac{\partial i(t)}{\partial u_{o}}  &= \sigma(y)(1-\sigma(y))(u_i TH_{u_{o}}(t-1))  \\
\frac{\partial o(t)}{\partial u_{o}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{u_{o}}(t-1) + h(t-1))  \\
TC_{u_{o}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{u_{o}}&= o(t) (1-\phi^{2}(c(t))) TC_{u_{o}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial u_{f}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial u_{f}} &=
(1-\phi^2(y))(u_g TH_{u_{f}}(t-1))  \\
\frac{\partial f(t)}{\partial u_{f}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{u_{f}}(t-1) + h(t-1)) \\
\frac{\partial i(t)}{\partial u_{f}}  &= \sigma(y)(1-\sigma(y))(u_i TH_{u_{f}}(t-1))  \\
\frac{\partial o(t)}{\partial u_{f}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{u_{f}}(t-1))  \\
TC_{u_{f}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{u_{f}}&= o(t) (1-\phi^{2}(c(t))) TC_{u_{f}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial u_{g}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial u_{g}} &=
(1-\phi^2(y))(u_g TH_{u_{g}}(t-1)+ h(t-1))  \\
\frac{\partial f(t)}{\partial u_{g}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{u_{g}}(t-1)) \\
\frac{\partial i(t)}{\partial u_{g}}  &= \sigma(y)(1-\sigma(y))( u_i TH_{u_{g}}(t-1))  \\
\frac{\partial o(t)}{\partial u_{g}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{u_{g}}(t-1))  \\
TC_{u_{g}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{u_{g}}&= o(t) (1-\phi^{2}(c(t))) TC_{u_{g}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial b_{g}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial b_{g}} &=
(1-\phi^2(y))(u_g TH_{b_{g}}(t-1)+ 1)  \\
\frac{\partial f(t)}{\partial b_{g}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{b_{g}}(t-1)) \\
\frac{\partial i(t)}{\partial b_{g}}  &= \sigma(y)(1-\sigma(y))(u_i TH_{b_{g}}(t-1))  \\
\frac{\partial o(t)}{\partial b_{g}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{b_{g}}(t-1))  \\
TC_{b_{g}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{b_{g}}&= o(t) (1-\phi^{2}(c(t))) TC_{b_{g}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial b_{f}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial b_{f}} &=
(1-\phi^2(y))(u_g TH_{b_{f}}(t-1))  \\
\frac{\partial f(t)}{\partial b_{f}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{b_{f}}(t-1) + 1 ) \\
\frac{\partial i(t)}{\partial b_{f}}  &= \sigma(y)(1-\sigma(y))( u_i TH_{b_{f}}(t-1))  \\
\frac{\partial o(t)}{\partial b_{f}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{b_{f}}(t-1))  \\
TC_{b_{f}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{b_{f}}&= o(t) (1-\phi^{2}(c(t))) TC_{b_{f}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\subsection{$\frac{\partial h(t)}{\partial b_{o}}$}

\begin{equation}
\begin{aligned}
\frac{\partial g(t)}{\partial b_{o}} &=
(1-\phi^2(y))(u_g TH_{b_{o}}(t-1))  \\
\frac{\partial f(t)}{\partial b_{o}} &=  \sigma(y)(1-\sigma(y))( u_f TH_{b_{o}}(t-1)) \\
\frac{\partial i(t)}{\partial b_{o}}  &= \sigma(y)(1-\sigma(y))( u_i TH_{b_{o}}(t-1))  \\
\frac{\partial o(t)}{\partial b_{o}} &=  \sigma(x)(1-\sigma(x))( u_o TH_{b_{o}}(t-1) + 1 )  \\
TC_{b_{o}}&= f(t) TC_{i_j}(t-1) + c(t-1)\frac{\partial f(t)}{ \partial b_i} + i(t)\frac{\partial g(t) }{\partial b_i}  + g(t)\frac{\partial i(t) }{\partial b_i}  \\
TH_{b_{o}}&= o(t) (1-\phi^{2}(c(t))) TC_{b_{o}}(t)  + \phi(c(t)) \frac{\partial o(t)}{\partial W_{ij}}
\end{aligned}
\end{equation}

\bibliography{sample}

\end{document}


% BEGIN LICENSE BLOCK
% Version: CMPL 1.1
%
% The contents of this file are subject to the Cisco-style Mozilla Public
% License Version 1.1 (the "License"); you may not use this file except
% in compliance with the License.  You may obtain a copy of the License
% at www.eclipse-clp.org/license.
%
% Software distributed under the License is distributed on an "AS IS"
% basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.  See
% the License for the specific language governing rights and limitations
% under the License.
%
% The Original Code is  The ECLiPSe Constraint Logic Programming System.
% The Initial Developer of the Original Code is  Cisco Systems, Inc.
% Portions created by the Initial Developer are
% Copyright (C) 2006 Cisco Systems, Inc.  All Rights Reserved.
%
% Contributor(s):
%
% END LICENSE BLOCK

\chapter{Getting started with Interval Constraints}
\label{chapicintro}
%HEVEA\cutdef[1]{section}

The Interval Constraints (IC) library provides a constraint solver which
works with both integer and real interval variables.
This chapter provides a general introduction to the library, and then
focusses on its support for integer constraints.
For more detail on IC's real variables and constraints, please see
Chapter~\ref{chapreal}.

%----------------------------------------------------------------------
\section{Using the Interval Constraints Library}

To use the Interval Constraints Library, load the library using either of:
\begin{code}
:- lib(ic).
:- use_module(library(ic)).
\end{code}
Specify this at the beginning of your program.

%----------------------------------------------------------------------
\section{Structure of a Constraint Program}
The typical top-level structure of a constraint program is
\begin{code}
solve(Variables) :-
        read_data(Data),
        setup_constraints(Data, Variables),
        labeling(Variables).
\end{code}
where \verb0setup_constraints/20 contains the problem model. It creates the
variables and the constraints over the variables.
This is often, but not necessarily, deterministic.
The \verb0labeling/10 predicate is the search part of the program that
attempts to find solutions by trying all instantiations for the
variables. This search is constantly pruned by constraint propagation.

The above program will find all solutions.
If the best solution is wanted, a branch-and-bound
procedure can be wrapped around the search component of the program:
\begin{code}
solve(Variables) :-
        read_data(Data),
        setup_constraints(Data, Variables, Objective),
        branch_and_bound:minimize(labeling(Variables), Objective).
\end{code}

\See{The {\em branch\_and\_bound} library provides generic predicates
that support optimization in conjunction with any \eclipse{} solver.
Section~\ref{secbboptsearch} discusses these predicates.}

%----------------------------------------------------------------------
\section{Modelling}
The problem modelling code must:
    \begin{itemize}
    \item Create the variables with their initial domains
    \item Setup the constraints between the variables
    \end{itemize}

A simple example is the ``crypt-arithmetic'' puzzle,
\verb0SEND+MORE = MONEY0.
The idea is to associate a digit (0-9) with each letter so that the
equation is true. The \eclipse{} code is as follows:

\label{send-more-money}
\begin{code}
:- lib(ic).

sendmore(Digits) :-
    Digits = [S,E,N,D,M,O,R,Y],

% Assign a finite domain with each letter - S, E, N, D, M, O, R, Y -
% in the list Digits
    Digits :: [0..9],

% Constraints
    alldifferent(Digits),
    S #\verb0\0= 0,
    M #\verb0\0= 0,
                 1000*S + 100*E + 10*N + D
               + 1000*M + 100*O + 10*R + E
    #= 10000*M + 1000*O + 100*N + 10*E + Y,

% Search
    labeling(Digits).
\end{code}

%----------------------------------------------------------------------
\section{Built-in Constraints}
\label{icbics}

The following section summarises the built-in constraint predicates of the
{\em ic} library.

\quickref{Domain constraints}{
\begin{description}

\item [\biptxtrefni{Vars :: Domain}{::/2!ic}{../bips/lib/ic/NN-2.html}]
\index{::/2@\texttt{::/2}!ic}
Constrains Vars to take only integer or real values from the domain
specified by Domain.  Vars may be a variable, a list, or a submatrix (e.g.\
M[1..4, 3..6]); for a list or a submatrix, the domain is applied recursively
so that one can apply a domain to, for instance, a list of lists of
variables.  Domain can be specified as a simple range Lo .. Hi, or as a list
of subranges and/or individual elements (integer variables only).  The type
of the bounds determines the type of the variable (real or integer).  Also
allowed are the (untyped) symbolic bound values {\tt inf}, {\tt +inf} and
{\tt -inf}.

\item [\biptxtrefni{Vars \$:: Domain}{\$::/2!ic}{../bips/lib/ic/SNN-2.html}]
\index{\$::/2@\texttt{\$::/2}!ic}
Like \texttt{::/2}, but for declaring real variables (i.e.\ it never imposes
integrality, regardless of the types of the bounds).

\item [\biptxtrefni{Vars \#:: Domain}{\#::/2!ic}{../bips/lib/ic/HNN-2.html}]
\index{\#::/2@\texttt{\#::/2}!ic}
Like \texttt{::/2}, but for declaring integer variables.

\item [\biptxtrefni{reals(Vars)}{reals/1!ic}{../bips/lib/ic/reals-1.html}]
\index{reals/1@\texttt{reals/1}!ic}
Declares that the variables are IC variables (like declaring
{\tt Vars :: -inf..inf}).

\item [\biptxtrefni{integers(Vars)}{integers/1!ic}{../bips/lib/ic/integers-1.html}]
\index{integers/1@\texttt{integers/1}!ic}
Constrains the given variables to take integer values only.

\end{description}}

The most common way to declare an IC variable is to use the
\biptxtrefni{::/2}{::/2!ic}{../bips/lib/ic/NN-2.html} predicate (or
\biptxtrefni{\$::/2}{\$::/2!ic}{../bips/lib/ic/SNN-2.html} or
\biptxtrefni{\#::/2}{\#::/2!ic}{../bips/lib/ic/HNN-2.html}) to give it an
initial domain:

\begin{quote}\begin{verbatim}
?- X :: -10 .. 10.
X = X{-10 .. 10}
Yes

?- X :: -10.0 .. 10.0.
X = X{-10.0 .. 10.0}
Yes

?- X #:: -10 .. 10.
X = X{-10 .. 10}
Yes

?- X $:: -10 .. 10.
X = X{-10.0 .. 10.0}
Yes

?- X :: 0 .. 1.0Inf.
X = X{0 .. 1.0Inf}
Yes

?- X :: 0.0 .. 1.0Inf.
X = X{0.0 .. 1.0Inf}
Yes

?- X :: [1, 4 .. 6, 9, 10].
X = X{[1, 4 .. 6, 9, 10]}
Yes
\end{verbatim}\end{quote}

Note that for \biptxtrefni{::/2}{::/2!ic}{../bips/lib/ic/NN-2.html}
\index{\#::/2@\texttt{\#::/2}!ic}  the type
of the bounds defines the type of the variable (integer or real) but that
infinities are considered type-neutral.  To just declare the type of a
variable without restricting the domain at all, one can use the
\biptxtrefni{integers/1}{integers/1!ic}{../bips/lib/ic/integers-1.html}
\index{\#::/2@\texttt{\#::/2}!ic}
and \biptxtrefni{reals/1}{reals/1!ic}{../bips/lib/ic/reals-1.html}\index{\#::/2@\texttt{\#::/2}!ic}
.

The final way to declare that a variable is an IC variable is to just use it
in an IC constraint: this performs an implicit declaration.

\ignore{
\begin{code}
?- X :: -10 .. 10, get_domain_as_list(X, List).
   X = X\{-10 .. 10\}
   List = [-10, -9, -8, -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Yes

?- reals([X, Y]), X = Y, X$>= 1.5, X $=< 2.5, Y = 2.2.
   X = 2.2
   Y = 2.2
Yes

?- integers([X, Y]), X :: 0 .. 4, X + Y #= W, X = Y, W = 8.
   X = 4
   Y = 4
   W = 8
Yes
\end{code}
}

\quickref{Integral Arithmetic constraints\label{integer-constraints}}{
\begin{description}

\item [\biptxtrefni{ExprX \#= ExprY}{\#=/2!ic}{../bips/lib/ic/HE-2.html}]
\index{\#=/2@\texttt{\#=/2}!ic}
ExprX is equal to ExprY.  ExprX and ExprY are integer expressions, and the
variables and subexpressions are constrained to be integers.

\item [\biptxtrefni{ExprX \#>= ExprY}{\#>=/2!ic}{../bips/lib/ic/HGE-2.html}]
\index{\#>=/2@\texttt{\#>=/2}!ic}
ExprX is greater than or equal to ExprY.  ExprX and ExprY are integer
expressions, and the variables and subexpressions are constrained to be
integers.

\item [\biptxtrefni{ExprX \#=< ExprY}{\#=</2!ic}{../bips/lib/ic/HEL-2.html}]
\index{\#=</2@\texttt{\#=</2}!ic}
ExprX is less than or equal to ExprY.  ExprX and ExprY are integer
expressions, and the variables and subexpressions are constrained to be
integers.

\item [\biptxtrefni{ExprX \#> ExprY}{\#>/2!ic}{../bips/lib/ic/HG-2.html}]
\index{\#>/2@\texttt{\#>/2}!ic}
ExprX is greater than ExprY.  ExprX and ExprY are integer expressions, and
the variables and subexpressions are constrained to be integers.

\item [\biptxtrefni{ExprX \#< ExprY}{\#</2!ic}{../bips/lib/ic/HL-2.html}]
\index{\#</2@\texttt{\#</2}!ic}
ExprX is less than ExprY.  ExprX and ExprY are integer expressions, and the
variables and subexpressions are constrained to be integers.

%\item [\biptxtrefni{ExprX \#\bsl= ExprY}{\#\bsl=/2!ic}{../bips/lib/ic/HRE-2.html}]
\item [{\bf ExprX \#\bsl= ExprY}]
ExprX is not equal to ExprY.  ExprX and ExprY are integer
expressions, and the variables are constrained to be integers.

\item [\biptxtref{ac_eq(X, Y, C)}{ac_eq/3}{../bips/lib/ic/ac_eq-3.html}]
Arc-consistent implementation of \bipnoidx{X \#= Y + C}.  X and Y are
constrained to be integer variables and to have ``reasonable'' bounds.  C
must be an integer.

\end{description}}
% This belongs inside the quickref, but the \bsl gets garbled in the index
% if it's there.
\index{\#\=@\#\bsl=/2!ic}%

\quickref{Non-Integral Arithmetic Constraints\label{general-constraints}}{
\begin{description}

\item [\biptxtrefni{ExprX \$= ExprY}{\$=/2!ic}{../bips/lib/ic/SE-2.html}]
\index{\$=/2@\texttt{\$=/2}!ic} ExprX is equal to ExprY.  ExprX and ExprY are general expressions.

\item [\biptxtrefni{ExprX \$>= ExprY}{\$>=/2!ic}{../bips/lib/ic/SGE-2.html}]
\index{\$>=/2@\texttt{\$>=/2}!ic} ExprX is greater than or equal to ExprY.  ExprX and ExprY are general
expressions.

\item [\biptxtrefni{ExprX \$=< ExprY}{\$=</2!ic}{../bips/lib/ic/SEL-2.html}]
\index{\$=</2@\texttt{\$=</2}!ic} ExprX is less than or equal to ExprY.  ExprX and ExprY are general expressions.

\item [\biptxtrefni{ExprX \$> ExprY}{\$>/2!ic}{../bips/lib/ic/SG-2.html}]
\index{\$>/2@\texttt{\$>/2}!ic} ExprX is greater than ExprY.  ExprX and ExprY are general expressions.

\item [\biptxtrefni{ExprX \$< ExprY}{\$</2!ic}{../bips/lib/ic/SL-2.html}]
\index{\$</2@\texttt{\$</2}!ic} ExprX is less than ExprY.  ExprX and ExprY are general expressions.

\item [{\bf ExprX \$\bsl= ExprY}]
ExprX is not equal to ExprY.  ExprX and ExprY are general expressions.
\end{description}}
% This belongs inside the quickref, but the \bsl gets garbled in the index
% if it's there.
\index{\$\=@\$\bsl=/2!ic}%

The basic IC relational constraints come in two forms.  The first form is
for integer-only constraints, and is summarised in
Figure~\ref{integer-constraints}.  All of these constraints contain
\texttt{\#} in their name, which indicates that all numbers appearing in
them must be integers, and all variables \emph{and subexpressions} will be
constrained to be integral.  It is important to note that subexpressions are
constrained to be integral, because it means, for instance, that
\bipnoidx{X/2 + Y/2 \#= 1} and \bipnoidx{X + Y \#= 2} are different
constraints, since the former constrains X and Y to be even.

The second form is the general form of the constraints, and is summarised in
Figure~\ref{general-constraints}.  These constraints can be used with either
integer or real variables and numbers.  With the exception of integrality
issues, the two versions of each constraint are equivalent.  Thus if the
constants are integers and the variables and subexpressions are integral,
the two forms may be used interchangeably.

Most of the basic constraints operate by propagating bound information
(performing interval reasoning).  The exceptions are the disequality (not
equals) constraints and the \bipnoidx{ac_eq/3} constraint, which perform
domain reasoning (arc consistency).  An example:

\begin{quote}\begin{verbatim}
?- [X, Y] :: 0 .. 10, X #>= Y + 2.
X = X{2 .. 10}
Y = Y{0 .. 8}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

In the above example, since the lower bound of \texttt{Y} is 0 and
\texttt{X} must be at least 2 greater, the lower bound of \texttt{X} has
been updated to 2.  Similarly, the upper bound of \texttt{Y} has been
reduced to 8.  The delayed goal indicates that the constraint is still
active: there are still some combinations of values for \texttt{X} and
\texttt{Y} which violate the constraint, so the constraint remains until it
is sure that no such violation is possible.

Note that if a domain ever becomes empty as the result of propagation (no
value for the variable is feasible) then the constraint must necessarily
have been violated, and the computation backtracks.

For a disequality constraint, no deductions can be made until
there is only one variable left, at which point (if it is an integer
variable) the variable's domain can be updated to exclude the relevant
value:

\begin{quote}\begin{verbatim}
?- X :: 0 .. 10, X #\= 3.
X = X{[0 .. 2, 4 .. 10]}
Yes

?- [X, Y] :: 0 .. 10, X - Y #\= 3.
X = X{0 .. 10}
Y = Y{0 .. 10}
There is 1 delayed goal.
Yes

?- [X, Y] :: 0 .. 10, X - Y #\= 3, Y = 2.
X = X{[0 .. 4, 6 .. 10]}
Y = 2
Yes
\end{verbatim}\end{quote}

For the \bipref{ac_eq/3}{../bips/lib/ic/ac_eq-3.html} constraint, ``holes''
in the domain of one variable are propagated to the other:

\begin{quote}\begin{verbatim}
?- [X, Y] :: 0 .. 10, ac_eq(X, Y, 3).
X = X{3 .. 10}
Y = Y{0 .. 7}
There is 1 delayed goal.
Yes

?- [X, Y] :: 0 .. 10, ac_eq(X, Y, 3), Y #\= 4.
X = X{[3 .. 6, 8 .. 10]}
Y = Y{[0 .. 3, 5 .. 7]}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

Compare with the corresponding bounds consistency constraint:

\begin{quote}\begin{verbatim}
?- [X, Y] :: 0 .. 10, X #= Y + 3, Y #\= 4.
X = X{3 .. 10}
Y = Y{[0 .. 3, 5 .. 7]}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

\See{IC supports a range of mathematical operators beyond the basic
\texttt{+/2}, \texttt{-/2}, \texttt{*/2}, etc.  See the IC chapter in
the Constraint Library Manual for full details.}

\Note{If one wishes to construct an expression to use in an IC constraint at
run time, then one must wrap it in \texttt{eval/1}:}

\begin{quote}\begin{verbatim}
?- [X, Y] :: 0..10, Expr = X + Y, Sum #= Expr.
number expected in set_up_ic_con(7, 1, [0 * 1, 1 * Sum{-1.0Inf .. 1.0Inf}, -1 * (X{0 .. 10} + Y{0 .. 10})])
Abort

?- [X, Y] :: 0..10, Expr = X + Y, Sum #= eval(Expr).
X = X{0 .. 10}
Y = Y{0 .. 10}
Sum = Sum{0 .. 20}
Expr = X{0 .. 10} + Y{0 .. 10}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

{\em Reification} provides access to the logical truth of a constraint
expression and can be used by:

\begin{itemize}
\item The {\eclipse} system to infer the truth value, reflecting the value
into a variable.
\item The programmer to enforce the constraint or its negation by giving a
value to the truth variable.
\end{itemize}

This logical truth value is a boolean variable (domain \texttt{0..1}), where
the value 1 means the constraint is or is required to be true, and the value
0 means the constraint is or is required to be false.

When constraints appear in an expression context,
they evaluate to their reified truth value. Practically, this means that
the constraints are posted in a passive check but do not propagate mode.
In this mode no variable domains are modified but checks are made to
determine whether the constraint has become entailed (necessarily true) or
disentailed (necessarily false).

The simplest and arguably most natural way to reify a constraint is to
place it in an expression context  (i.e.\ on either side of a \verb|$=|,
\verb|#=|, etc.) and assign its truth value to a variable. For example:

\begin{quote}\begin{verbatim}
?- X :: 0 .. 10, TruthValue $= (X $> 4).
TruthValue = TruthValue{[0, 1]}
X = X{0 .. 10}
There is 1 delayed goal.
Yes

?- X :: 6 .. 10, TruthValue $= (X $> 4).
TruthValue = 1
X = X{6 .. 10}
Yes

?- X :: 0 .. 4, TruthValue $= (X $> 4).
TruthValue = 0
X = X{0 .. 4}
Yes
\end{verbatim}\end{quote}

All the basic relational constraint predicates also come in a three-argument
form where the third argument is the reified truth value, and this form can
also be used to reify a constraint directly.  For example:

\begin{quote}\begin{verbatim}
?- X :: 0 .. 10, $>(X, 4, TruthValue).
X = X{0 .. 10}
TruthValue = TruthValue{[0, 1]}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

As noted above the boolean truth variable corresponding to a constraint can
also be used to enforce the constraint (or its negation):

\begin{quote}\begin{verbatim}
?- X :: 0 .. 10, TruthValue $= (X $> 4), TruthValue = 1.
X = X{5 .. 10}
TruthValue = 1
Yes

?- X :: 0 .. 10, TruthValue $= (X $> 4), TruthValue = 0.
X = X{0 .. 4}
TruthValue = 0
Yes
\end{verbatim}\end{quote}

By instantiating the value of the reified truth variable, the constraint
changes from being {\em passive} to being {\em active}.  Once actively
true (or actively false) the constraint will prune domains as though
it had been posted as a simple non-reified constraint.

\ignore{
\Note{User defined constraints will be treated as reifiable if they appear
in an expression context and as such should provide a version with an extra
argument where this final argument is the reified truth value reflected
into a variable.}
}

\See{Additional information on reified constraints can be found in the
{\eclipse} Constraint Library Manual that documents {\em IC: A Hybrid
 Finite Domain / Real Number Interval Constraint Solver}.}

\quickref{Constraint Expression Connectives\label{expression-connectives}}{
\begin{description}

\item[{\bf and}]
     Constraint conjunction. e.g.\ {\tt X \$> 3 and X \$< 8}

\item[{\bf or}]
     Constraint disjunction. e.g.\ {\tt X \$< 3 or X \$> 8}

\item[{\bf =>}]
     Constraint implication. e.g.\ {\tt X \$> 3 => Y \$< 8}

\item[{\bf neg}]
     Constraint negation. e.g.\ {\tt neg X \$> 3}

\end{description}}

IC also provides a number of connectives useful for combining constraint
expressions.  These are summarised in Figure~\ref{expression-connectives}.
For example:

\begin{quote}\begin{verbatim}
?- [X, Y] :: 0 .. 10, X #>= Y + 6 or X #=< Y - 6.
X = X{0 .. 10}
Y = Y{0 .. 10}
There are 3 delayed goals.
Yes

?- [X, Y] :: 0 .. 10, X #>= Y + 6 or X #=< Y - 6, X #>= 5.
Y = Y{0 .. 4}
X = X{6 .. 10}
There is 1 delayed goal.
Yes
\end{verbatim}\end{quote}

In the above example, once it is known that \texttt{X \#=< Y - 6} cannot be
true, the constraint \texttt{X \#>= Y + 6} is enforced.

Note that these connectives exploit constraint reification, and actually
just reason about boolean variables.  This means that they can be used as
boolean constraints as well:

\begin{quote}\begin{verbatim}
?- A => B.
A = A{[0, 1]}
B = B{[0, 1]}
There is 1 delayed goal.
Yes

?- A => B, A = 1.
B = 1
A = 1
Yes

?- A => B, A = 0.
B = B{[0, 1]}
A = 0
Yes
\end{verbatim}\end{quote}

%----------------------------------------------------------------------

\section{Global constraints}
\label{secglobal}

\index{global constraints!ic|(}
\index{library!ic_global|(}

The IC constraint solver has some optional components which provide
so-called \emph{global} constraints.  These are high-level constraints that
tend to provide more global reasoning than the constraints in the main IC
library.  These optional components are contained in the \texttt{ic_global},
\texttt{ic_cumulative},\index{library!ic_cumulative}
\texttt{ic_edge_finder}\index{library!ic_edge_finder}
and \texttt{ic_edge_finder3}\index{library!ic_edge_finder3}
libraries.  The \texttt{ic_global} library provides a collection of general
global constraints, while the others provide constraints for
resource-constrained scheduling.

To use these global constraints, load the relevant optional library or
libraries using directives in one of these forms:
\begin{code}
:- lib(ic_global).
:- use_module(library(ic_global)).
\end{code}
Specify this at the beginning of your program.

Note that some of these libraries provide alternate implementations of
predicates which also appear in other libraries.  For example, the
\texttt{alldifferent/1} constraint is provided by both the standard
\texttt{ic} library and the \texttt{ic_global} library.  This means that if
you wish to use it, you must use the relevant module qualifier to specify
which one you want:
\biptxtrefni{ic:alldifferent/1}{alldifferent/1!ic}{../bips/lib/ic/alldifferent-1.html}
\index{alldifferent/1@\texttt{alldifferent/1}!ic} or \index{alldifferent/1@\texttt{alldifferent/1}!ic_global}
\biptxtrefni{ic_global:alldifferent/1}{alldifferent/1!ic_global}{../bips/lib/ic_global/alldifferent-1.html}.

% List a few example predicates from ic_global?

\See{See the ``Additional Finite Domain Constraints'' section of the Library
Manual for more details of these libraries and a full list of the predicates
they provide.}

\subsection{Different strengths of propagation}

The \texttt{alldifferent(List)} predicate imposes the constraint on the
elements of \texttt{List} that they all take different values.
The standard \biptxtrefni{alldifferent/1}{alldifferent/1!ic}{../bips/lib/ic/alldifferent-1.html}
\index{alldifferent/1@\texttt{alldifferent/1}!ic} predicate from the IC library provides a level of propagation equivalent to
imposing pairwise
%\biptxtrefni{\#\bsl=/2}{\#\=/2@\#\bsl=/2!ic}{../bips/lib/ic/HRE-2.html}
\ahref{../bips/lib/ic/HRE-2.html}{{\bf \#\bsl=/2}}
\index{\#\=@\#\bsl=/2!ic}%
constraints (though it does it more efficiently than that).  This means that
no propagation is performed until elements of the list start being made
ground.  This is despite the fact that there may be ``obvious'' inferences
which could be made.

Consider as an example the case of 5 variables with domains \texttt{1..4}.
Clearly the 5 variables cannot all be given different values, since there
are only 4 distinct values available.  However, the standard
\biptxtrefni{alldifferent/1}{alldifferent/1!ic}{../bips/lib/ic/alldifferent-1.html} constraint
cannot determine this:

\begin{quote}\begin{verbatim}
?- L = [X1, X2, X3, X4, X5], L :: 1 .. 4, ic:alldifferent(L).
X1 = X1{1 .. 4}
X2 = X2{1 .. 4}
X3 = X3{1 .. 4}
X4 = X4{1 .. 4}
X5 = X5{1 .. 4}
L = [X1{1 .. 4}, X2{1 .. 4}, X3{1 .. 4}, X4{1 .. 4}, X5{1 .. 4}]
There are 5 delayed goals.
Yes
\end{verbatim}\end{quote}

Consider another example where three of the variables have domain
\texttt{1..3}.  Clearly, if all the variables are to be different, then no
other variable can take a value in the range \texttt{1..3}, since each of
those values must be assigned to one of the original three variables.
Again, the standard
\biptxtrefni{alldifferent/1}{alldifferent/1!ic}{../bips/lib/ic/alldifferent-1.html} constraint
cannot determine this:

\begin{quote}\begin{verbatim}
?- [X1, X2, X3] :: 1 .. 3, [X4, X5] :: 1 .. 5,
   ic:alldifferent([X1, X2, X3, X4, X5]).
X1 = X1{1 .. 3}
X2 = X2{1 .. 3}
X3 = X3{1 .. 3}
X4 = X4{1 .. 5}
X5 = X5{1 .. 5}
There are 5 delayed goals.
Yes
\end{verbatim}\end{quote}

On the other hand, \texttt{ic_global}'s
\biptxtrefni{alldifferent/1}{alldifferent/1!ic_global}{../bips/lib/ic_global/alldifferent-1.html}
\index{alldifferent/1@\texttt{alldifferent/1}!ic_global}
constraint performs some stronger, more global reasoning, and for both of
the above examples makes the appropriate inference:

\begin{quote}\begin{verbatim}
?- L = [X1, X2, X3, X4, X5], L :: 1 .. 4, ic_global:alldifferent(L).
No

?- [X1, X2, X3] :: 1 .. 3, [X4, X5] :: 1 .. 5,
   ic_global:alldifferent([X1, X2, X3, X4, X5]).
X1 = X1{1 .. 3}
X2 = X2{1 .. 3}
X3 = X3{1 .. 3}
X4 = X4{[4, 5]}
X5 = X5{[4, 5]}
There are 2 delayed goals.
Yes
\end{verbatim}\end{quote}

Of course, there is a trade-off here: the stronger version of the constraint
takes longer to perform its propagation.  Which version is best depends on
the nature of the problem being solved.

\See{Note that even stronger propagation can be achieved if desired, by
using the Propia library (see Chapter~\ref{chappropiachr}).}

\index{library!ic_global|)}

In a similar vein, the \texttt{ic_cumulative}, \texttt{ic_edge_finder} and
\texttt{ic_edge_finder3} libraries provide increasingly strong versions of
constraints such as \texttt{cumulative/4}, but with increasing cost to do
their propagation (linear, quadratic and cubic, respectively).

\index{global constraints!ic|)}

%----------------------------------------------------------------------

\section{Simple User-defined Constraints}
User-defined, or `conceptual' constraints can easily be defined as
conjunctions of primitive constraints. For example, let us consider a set
of products and the specification that allows them to be colocated in a
warehouse. This should be done in such a way as to propagate possible
changes in the domains as soon as this becomes possible.

Let us assume we have a symmetric relation that defines which product
can be colocated with another and that products are distinguished by numeric
product identifiers:

\begin{code}
colocate(100, 101).
colocate(100, 102).
colocate(101, 100).
colocate(102, 100).
colocate(103, 104).
colocate(104, 103).
\end{code}

Suppose we define a constraint \verb0colocate_product_pair(X, Y)0 such
that any change of the possible values of $X$ or $Y$ is propagated to the
other variable. There are many ways in which this pairing
can be defined in {\eclipse}. They are different solutions with different
properties, but they yield the same results.

\subsection{Using Reified Constraints}
We can encode directly the relations between elements in the
domains of the two variables:
\begin{code}
colocate_product_pair(A, B) :-
    cpp(A, B),
    cpp(B, A).

cpp(A, B) :-
    [A,B] :: [100, 101, 102, 103, 104],
    A #= 100 => B :: [101, 102],
    A #= 101 => B #= 100,
    A #= 102 => B #= 100,
    A #= 103 => B #= 104,
    A #= 104 => B #= 103.
\end{code}

This method is quite simple and does not need any special analysis; on
the other hand it potentially creates a huge number of auxiliary
constraints and variables.

\subsection{Using Propia}
By far the simplest mechanism, that avoids this potential creation of large
numbers of auxiliary constraints and variables, is to load the Generalised
Propagation library ({\em propia}) and use arc-consistency ({\em ac})
propagation, viz:
\begin{quote}\begin{verbatim}
?- colocate(X,Y) infers ac
\end{verbatim}\end{quote}

\See{Additional information on {\em propia} can be found in
section~\ref{secpropia}, section~\ref{chappropiachr} and the {\eclipse} Constraint
Library Manual.}

\subsection{Using the {\em element} Constraint}
\label{icelement}
In this case we use the \verb0element/30 predicate,
that states in a list of integers that the element at
an index is equal to a value. Every time the index or the value is updated,
the constraint is activated and the domain of the other variable is updated
accordingly.

\begin{code}
relates(X, Xs, Y, Ys) :-
    element(I, Xs, X),
    element(I, Ys, Y).
\end{code}

We define a generic predicate, \verb0relates/40, that associates the
corresponding elements at a specific index of two lists, with one
another. The variable {\it I} is an index into the lists,
{\it Xs} and {\it Ys}, to yield the elements at this index,
in variables {\it X} and {\it Y}.

\begin{code}
colocate_product_pair(A, B) :-
    relates(A, [100, 100, 101, 102, 103, 104],
            B, [101, 102, 100, 100, 104, 103]).
\end{code}

The \verb0colocate_product_pair0 predicate simply calls \verb0relates/40
passing a list containing the product identifiers in the first argument
of \verb0colocate/20 as {\it Xs} and a list containing product identifiers
from the second argument of \verb0colocate/20 as {\it Ys}.

Behind the scenes, this is exactly the implementation used for
arc-consistency propagation by the Generalised Propagation library.

Because of the specific and efficient algorithm implementing the
\verb0element/30 constraint,  it is usually faster than the first
approach, using reified constraints.

%----------------------------------------------------------------------
\section{Searching for Feasible Solutions}
\begin{description}
\item[indomain(+DVar)]
\index{indomain/1}
This predicate instantiates the domain variable {\it DVar} to an
element of its domain; on backtracking the subsequent value is taken.
It is used, for example, to find a value of {\it DVar} which is consistent
with all currently imposed constraints.
If {\it DVar} is a ground term, it succeeds.
Otherwise, if it is not a domain variable, an error is raised.

\item[labeling(+List)]
\index{labeling/1}
\index{labeling!ic}
The elements of the {\it List} are instantiated using the
\verb0indomain/10 predicate.
\end{description}

\See{Additional information on search algorithms, heuristics and their
use in {\eclipse} can be found in chapter~\ref{chapsearch}.}

\section{Bin Packing}
This section presents a worked example using finite domains to solve a
bin-packing problem.

\subsection{Problem Definition}
In this type of problem the goal is to pack a certain amount of
different items into the minimal number of bins under specific constraints.
Let us solve an example given by Andre Vellino in the Usenet
group comp.lang.prolog, June 93:
\begin{itemize}
\item There are 5 types of items:

        {\em glass}, {\em plastic}, {\em steel}, {\em wood}, {\em copper}

\item There are three types of bins:

        {\em red}, {\em blue}, {\em green}

\item        The capacity constraints imposed on the bins are:

\begin{itemize}
\item        red   has capacity 3
\item        blue  has capacity 1
\item        green has capacity 4
\end{itemize}

\item The containment constraints imposed on the bins are:
\begin{itemize}
\item        red   can contain glass, wood, copper
\item        blue  can contain glass, steel, copper
\item        green can contain plastic, wood, copper
\end{itemize}

\item  The requirement constraints imposed on component types (for all bin
  types) are:

        wood requires plastic

\item Certain component types cannot coexist:

\begin{itemize}
\item        glass and copper exclude each other
\item        copper and plastic exclude each other
\end{itemize}

\item The following bin types have the following capacity constraints for certain
components:

\begin{itemize}
\item red   contains at most 1 wood item
\item blue  implicitly contains at most 1 wood item
\item green contains at most 2 wood items
\end{itemize}

\item Given the initial supply stated below, what is the minimum total
  number of bins required to contain the components?
\begin{itemize}
\item 1 glass item
\item 2 plastic items
\item 1 steel item
\item 3 wood items
\item 2 copper items
\end{itemize}
\end{itemize}

\subsection{Problem Model - Using Structures}

In modelling this problem we need to refer to an array of quantities
of glass items, plastic items, steel items, wood items and copper
items. We therefore introduce:

A structure to hold this array:
\begin{code}
:- local struct(contents(glass, plastic, steel, wood, copper)).
\end{code}

A structure that defines the colour for each of the bin types:
\begin{code}
:- local struct(colour(red, blue, green)).
\end{code}

By defining the bin colours as fields of a structure there is an implicit
integer value associated with each colour. This allows the readability of the code
to be preserved by writing, for example, {\tt red of colour} rather than
explicitly writing the colour's integer value `{\tt 1}'.

And a structure that represents the bin itself, with its colour,
capacity and contents:
\begin{code}
:- local struct(bin(colour, capacity, contents:contents)).
\end{code}

\Note{The {\tt contents} attribute of {\tt bin} is itself a {\tt
contents} structure. The {\tt contents} field declaration within the {\tt bin}
structure using '{\tt :}' allows field names of the {\tt contents}
structure to be used as if they were field names of the {\tt bin}
structure. More information on accessing nested structures and
structures with {\em inherited} fields can be found in
section~\ref{structures} and in the {\em Structure Notation} section of
the {\eclipse} User Manual.}

The predicate \verb0solve_bin/20 is the general predicate
that takes an amount of components packed into a \verb0contents0
structure and returns the solution.
\begin{code}
?- Demand = contents\{glass:1, plastic:2, steel:1, wood:3, copper:2\},
   solve_bin(Demand, Bins).
\end{code}

\subsection{ Handling an Unknown Number of Bins}

\verb0solve_bin/20  calls \verb0bin_setup/20 to
generate a list {\it Bins}.
It adds redundant constraints to remove symmetries (two
solutions are considered symmetrical if they are
the same, but with the bins in a different order).
Finally it labels all decision variables in the problem.
\begin{code}
solve_bin(Demand, Bins) :-
    bin_setup(Demand, Bins),
    remove_symmetry(Bins),
    bin_label(Bins).
\end{code}

The usual pattern for solving finite domain problems is to state
constraints on a set of variables, and then label them.
However, because the number of bins needed is not known initially, it
is awkward to model the problem with a fixed set of variables.

One possibility is to take a fixed, large enough, number of bins
and to try to find a minimum number of non-empty bins.
However, for efficiency, we choose to solve a sequence of problems,
each one with a - larger - fixed number of bins,
until a solution is found.

The predicate \verb0bin_setup/20, to generate a list of bins with appropriate
constraints, works as follows.
First it tries to match the (remaining) demand with zero,
and use no (further) bins.
If this fails, a new bin is added to the bin list;
appropriate constraints are imposed on all the new bin's
variables;
its contents are subtracted from the demand;
and the \verb0bin_setup/20 predicate calls itself recursively:

\begin{code}
bin_setup(Demand,[]) :-
        all_zeroes(Demand).
bin_setup(Demand, [Bin | Bins]) :-
        constrain_bin(Bin),
        reduce_demand(Demand, Bin, RemainingDemand),
        bin_setup(RemainingDemand, Bins).

all_zeroes(
           contents\{glass:0, plastic:0, wood:0, steel:0, copper:0\}
          ).

reduce_demand(
              contents\{glass:G, plastic:P, wood:W, steel:S, copper:C\},
              bin\{glass:BG, plastic:BP, wood:BW, steel:BS, copper:BC\},
              contents\{glass:RG, plastic:RP, wood:RW, steel:RS, copper:RC\}
             ) :-
       RG #= G - BG,
       RP #= P - BP,
       RW #= W - BW,
       RS #= S - BS,
       RC #= C - BC.
\end{code}

\subsection{Constraints on a Single Bin}

The constraints imposed on a single bin correspond exactly to the
problem statement:
\begin{code}
constrain_bin(bin\{colour:Col, capacity:Cap, contents:C\}) :-
        colour_capacity_constraint(Col, Cap),
        capacity_constraint(Cap, C),
        contents_constraints(C),
        colour_constraints(Col, C).
\end{code}

\paragraph{colour\_capacity\_constraint}
The colour capacity constraint relates the colour of the bin to its
capacity, we implement this using the \verb0relates/40 predicate (defined
in section~\ref{icelement}):
\begin{code}
colour_capacity_constraint(Col, Cap) :-
    relates(Col, [red of colour, blue of colour, green of colour],
            Cap, [3, 1, 4]).
\end{code}

\paragraph{capacity\_constraint}
The capacity constraint states the following:
\begin{itemize}
\item The number of items of each kind in the bin is non-negative.
\item The sum of all the items does not exceed the capacity of the bin.
\item and the bin is non-empty (an empty bin serves no purpose)
\end{itemize}

\begin{code}
capacity_constraint(Cap, contents\{glass:G,
                                   plastic:P,
                                   steel:S,
                                   wood:W,
                                   copper:C\}) :-
        G #>= 0, P #>= 0, S #>= 0, W #>= 0, C #>= 0,
        NumItems #= G + P + W + S + C,
        Cap #>= NumItems,
        NumItems #> 0.
\end{code}

\paragraph{contents\_constraints}
The contents constraints directly enforce the restrictions on items in
the bin: wood requires paper, glass and copper exclude each other, and
copper and plastic exclude each other:
\begin{code}
contents_constraints(contents\{glass:G, plastic:P, wood:W, copper:C\}) :-
        requires(W, P),
        exclusive(G, C),
        exclusive(C, P).
\end{code}

These constraints are expressed as logical combinations of constraints
on the number of items.
`requires' is expressed using implication, \verb0=>0.
`Wood requires paper' is expressed in logic as `If the number of wood
items is greater than zero, then the number of paper items
is also greater than zero':
\begin{code}
requires(W,P) :-
        W #> 0 => P #> 0.
\end{code}

Exclusion is expressed using disjunction, \verb0or0.
`X and Y are exclusive' is expressed as `Either the number of items of
kind $X$ is zero, or the number of items of kind $Y$ is zero':
\begin{code}
exclusive(X,Y) :-
        X #= 0 or Y #= 0.
\end{code}

\paragraph{colour\_constraints}
The colour constraint limits the number of wooden items in bins of
different colours.
Like the capacity constraint, the relation between the colour and
capacity, $WCap$, is expressed using the \verb0relates/40 predicate.
The number of wooden items is then constrained not to exceed the capacity:
\begin{code}
colour_constraints(Col, contents\{wood:W\}) :-
    relates(Col, [red of colour, blue of colour, green of colour],
            WCap, [1, 1, 2]),
    W #=< WCap.
\end{code}

This model artificially introduces a capacity of blue bins for
wood items (set simply at its maximum capacity for all items).

\subsection{Symmetry Constraints}
\label{binsym}
To make sure two solutions (a solution is a list of bins) are not
just different permutations of the same bins, we impose an order
on the list of bins:

\begin{code}
remove_symmetry(Bins) :-
        ( fromto(Bins, [B1, B2 | Rest], [B2 | Rest], [_Last])
        do
            lex_ord(B1, B2)
        ).
\end{code}

We order two bins by imposing lexicographic order onto lists computed
from their colour and contents, (recall that in defining the bin colours
as fields of a structure we have encoded them as integers, which allows
them to be ordered):
\begin{code}
lex_ord(bin\{colour:Col1, contents:Conts1\},
        bin\{colour:Col2, contents:Conts2\}) :-
        % Use `=..' to extract the contents of the bin as a list
        Conts1 =.. [_ | Vars1],
        Conts2 =.. [_ | Vars2],
        lexico_le([Col1 | Vars1], [Col2 | Vars2]).
\end{code}

The lexicographic order is imposed using \texttt{ic_global}'s
\biptxtrefni{lexico_le/2}{lexico_le/2!ic_global}{../bips/lib/ic_global/lexico_le-2.html}\index{lexico_le/1@\texttt{lexico_le/1}!ic_global} constraint.

\subsection{Search}

The search is done by first choosing a colour for each bin, and then
labelling the remaining variables.
\begin{code}
bin_label(Bins) :-
        ( foreach(bin\{colour:C\} Bins) do indomain(C) ),
        term_variables(Bins, Vars),
        search(Vars, 0, first_fail, indomain, complete, []).
\end{code}

The remaining variables are labelled by employing the first fail heuristic
(using the \verb0search/60 predicate of the {\em ic} library).

\See{Additional information on search algorithms, heuristics and their
use in {\eclipse} can be found in section~\ref{chapsearch}.}

\section{Exercises}

\begin{enumerate}

\item

A magic square is a $3 \times 3$ grid containing the digits 1 through 9
exactly once, such that each row, each column and the two diagonals sum to
the same number (15).  Write a program to find such magic squares.  (You may
wish to use the ``Send More Money'' example in section~\ref{send-more-money}
as a starting point.)

Bonus points if you can add constraints to break the symmetry, so that only
the one unique solution is returned.

\item

Fill the circles in the following diagram with the numbers 1 through 19 such
that the numbers in each of the 12 lines of 3 circles (6 around the outside,
6 radiating from the centre) sum to 23.

\begin{center}
%\epsfbox{archery.eps}
\resizebox{0.35\textwidth}{!}{\includegraphics{archery.eps}}
\end{center}

If the value of the sum is allowed to vary, which values of the sum have
solutions, and which do not?

(Adapted from Puzzle 35 in Dudeney's ``The Canterbury Puzzles''.)

\item

Consider the following code:

\begin{code}
foo(Xs, Ys) :-
        (
            foreach(X, Xs),
            foreach(Y, Ys),
            fromto(1, In, Out, 1)
        do
            In #= (X #< Y + Out)
        ).
\end{code}

Which constraint does this code implement?  (Hint: declaratively, it is the
same as one of the constraints from \texttt{ic_global}, but is implemented
somewhat differently.)  How does it work?

\end{enumerate}

%HEVEA\cutend