Title: DyNet: The Dynamic Neural Network Toolkit,
Abstract: We describe DyNet, a toolkit for implementing neural network models based on
dynamic declaration of network structure. In the static declaration strategy
that is used in toolkits like Theano, CNTK, and TensorFlow, the user first
defines a computation graph (a symbolic representation of the computation), and
then examples are fed into an engine that executes this computation and
computes its derivatives. In DyNet's dynamic declaration strategy, computation
graph construction is mostly transparent, being implicitly constructed by
executing procedural code that computes the network outputs, and the user is
free to use different network structures for each input. Dynamic declaration
thus facilitates the implementation of more complicated network architectures,
and DyNet is specifically designed to allow users to implement their models in
a way that is idiomatic in their preferred programming language (C++ or
Python). One challenge with dynamic declaration is that because the symbolic
computation graph is defined anew for every training example, its construction
must have low overhead. To achieve this, DyNet has an optimized C++ backend and
lightweight graph representation. Experiments show that DyNet's speeds are
faster than or comparable with static declaration toolkits, and significantly
faster than Chainer, another dynamic declaration toolkit. DyNet is released
open-source under the Apache 2.0 license and available at
this http URL.