Environmental drivers of systematicity and generalization in a situated
  agent
  The question of whether deep neural networks are good at generalising beyond
their immediate training experience is of critical importance for
learning-based approaches to AI. Here, we consider tests of out-of-sample
generalisation that require an agent to respond to never-seen-before
instructions by manipulating and positioning objects in a 3D Unity simulated
room. We first describe a comparatively generic agent architecture that
exhibits strong performance on these tests. We then identify three aspects of
the training regime and environment that make a significant difference to its
performance: (a) the number of object/word experiences in the training set; (b)
the visual invariances afforded by the agent's perspective, or frame of
reference; and (c) the variety of visual input inherent in the perceptual
aspect of the agent's perception. Our findings indicate that the degree of
generalisation that networks exhibit can depend critically on particulars of
the environment in which a given task is instantiated. They further suggest
that the propensity for neural networks to generalise in systematic ways may
increase if, like human children, those networks have access to many frames of
richly varying, multi-modal observations as they learn.
