<reponame>robertodessi/SCAN-CNN<filename>writeups/acl2019/setup.tex
\section{SCAN}
\label{sec:setup}

SCAN studies compositionality in a simple command
execution environment framed as a supervised sequence-to-sequence
task. The neural network receives word sequences as input, and has to
produce the correspondence action sequence. Examples are
given in Table \ref{table:examples}.
  \newcite{Lake:Baroni:2017} originally introduced 4
train/test splits, of which we consider 2.\footnote{We also
  tested our CNNs on SCAN's \emph{length} split, where test commands
  require systematically longer actions than the training
  ones. Accuracy was near 0\%, as the learned positional
  embeddings of our CNN architecture do not 
  generalize beyond  training lengths. We leave the
  investigation of more flexible positional encodings \cite[as in,
  e.g.,][]{vaswani:etal:2017} to future work. We also experimented
  with SCAN's \emph{turn left} split, obtaining near-perfect
  generalization. As RNNs were already performing very well in this split,
  we focus in the paper on the more challenging
  \emph{jump} case.} In the \emph{random} split, the training set
includes 80\% of randomly selected distinct SCAN commands, with the
remaining 20\% in the test set. This requires generalization,
as no test command is encountered in training, but there is no
systematic difference between the commands in the two sets.  In the
\emph{jump} split, the \emph{jump} command is only seen in isolation
during training, and the test set consists of all composite commands
with \emph{jump}. A system able to extract compositional rules (such
as ``\emph{X twice} means to X and X'') should have no problem
generalizing them to a new verb, as in this
split. \newcite{Loula:etal:2018} proposed a set of new SCAN splits,
the most challenging one being the \emph{around-right}
split.  The training partition contains examples of \emph{around} and
\emph{right}, but never in combination. The test set
contains all possible \emph{around right} commands. Loula and
colleagues want to test ``second-order modification'', as models
must learn how to compositionally apply the \emph{around}
function to \emph{right}, which is in turn a first-order function
modifying simple action verbs.

\begin{table}[t!]
    \footnotesize
    \begin{center}
%      \begin{tabular}{| l | l | l |}
      \begin{tabular}{| c | p{2.2cm} | p{2.2cm} |}
            \hline \textbf{Split} & \textbf{Train Command} & \textbf{Test Command} \\ \hline
            \textit{random} & \textit{walk opposite left}; \textit{turn left twice and look} & 
                \textit{walk and jump right twice}; \textit{run and run thrice}  \\
            \hline
            \textit{jump} & \textit{\underline{jump}}; \textit{turn left twice \underline{after look}}  & 
            \textit{turn left twice \underline{after jump}}; \textit{run twice \underline{and jump}} \\% jump and turn opposite right} \\
            \hline
            \textit{around-right} & \textit{jump \underline{around left}}; \textit{turn \underline{opposite right} twice} & \textit{walk \underline{around right}};
            \textit{look \underline{around right} and jump left} \\
            %\textit{around-right} & \textit{jump around left thrice}; \textit{turn opposite right twice} & \textit{walk around right};
            %\textit{look around right and jump left} \\
            \hline
        \end{tabular} 
    \end{center}
    \caption{\label{table:examples} Training and test examples for the three splits used in our experiments.}
\end{table}

\section{Experimental setup}
\paragraph{Model} We use the fully convolutional encoder-decoder model
of \newcite{gehring:etal:2017} out of the box, using version 0.6.0 of the fairseq 
toolkit.\footnote{\url{https://github.com/pytorch/fairseq}} The model
uses convolutional filters and Gated Linear Units
\cite{dauphin:etal:2016} along with an attention mechanism
that connects the encoder and the decoder.  Attention is computed
separately for each encoder layer, and produces weighted sums over
encoder input embeddings and encoder outputs. See the original paper
for details.
\paragraph{Training} The shift in distribution between training and
test splits makes SCAN unsuitable for validation-set tuning. Instead,
following \newcite{Lake:Baroni:2017} and \newcite{Loula:etal:2018}, we
train on 100k random samples with replacement from the
training command set. We explore different batch sizes (in terms of number of tokens per 
batch: 25, 50, 100, 200, 500, 1000), learning rates (0.1, 0.01, 0.001), layer dimensionalities
(128, 256, 512), layer numbers (6 to 10), convolutional
kernel width (3, 4, 5) and amount of dropout used (0, 0.25, 0.5).
For all other hyperparameters, we accept recommended/default fairseq values. Each configuration is run with 5
seeds, and we report means and standard deviations.

