Training recurrent neural networks (RNNs) that possess long-term memory is challenging. We provide insight into the trade-off between the smoothness of the cost function and the memory retention capabilities of the network. We express both aspects in terms of the Lipschitz constant of the dynamics modeled by the network. This allows us to make a distinction between three types of regions in the parameter space. In the first region, the network experiences problems in retaining long-term information, while at the same time the cost function is smooth and easy for gradient descent to navigate in. In the second region, the amount of stored information increases with time and the cost function is intricate and full of local minima. The third region is in between the two other regions and here the RNN is able to retain long-term information. Based on these theoretical findings we present the hypothesis that good parameter choices for the RNN are located in between the well-behaved and the ill-behaved cost function regions. The concepts presented in the paper are illustrated by artificially generated and real examples.
@cite_28: Chapter 1: The Real and Complex Number Systems Introduction Ordered Sets Fields The Real Field The Extended Real Number System The Complex Field Euclidean Spaces Appendix Exercises Chapter 2: Basic Topology Finite, Countable, and Uncountable Sets Metric Spaces Compact Sets Perfect Sets Connected Sets Exercises Chapter 3: Numerical Sequences and Series Convergent Sequences Subsequences Cauchy Sequences Upper and Lower Limits Some Special Sequences Series Series of Nonnegative Terms The Number e The Root and Ratio Tests Power Series Summation by Parts Absolute Convergence Addition and Multiplication of Series Rearrangements Exercises Chapter 4: Continuity Limits of Functions Continuous Functions Continuity and Compactness Continuity and Connectedness Discontinuities Monotonic Functions Infinite Limits and Limits at Infinity Exercises Chapter 5: Differentiation The Derivative of a Real Function Mean Value Theorems The Continuity of Derivatives L'Hospital's Rule Derivatives of Higher-Order Taylor's Theorem Differentiation of Vector-valued Functions Exercises Chapter 6: The Riemann-Stieltjes Integral Definition and Existence of the Integral Properties of the Integral Integration and Differentiation Integration of Vector-valued Functions Rectifiable Curves Exercises Chapter 7: Sequences and Series of Functions Discussion of Main Problem Uniform Convergence Uniform Convergence and Continuity Uniform Convergence and Integration Uniform Convergence and Differentiation Equicontinuous Families of Functions The Stone-Weierstrass Theorem Exercises Chapter 8: Some Special Functions Power Series The Exponential and Logarithmic Functions The Trigonometric Functions The Algebraic Completeness of the Complex Field Fourier Series The Gamma Function Exercises Chapter 9: Functions of Several Variables Linear Transformations Differentiation The Contraction Principle The Inverse Function Theorem The Implicit Function Theorem The Rank Theorem Determinants Derivatives of Higher Order Differentiation of Integrals Exercises Chapter 10: Integration of Differential Forms Integration Primitive Mappings Partitions of Unity Change of Variables Differential Forms Simplexes and Chains Stokes' Theorem Closed Forms and Exact Forms Vector Analysis Exercises Chapter 11: The Lebesgue Theory Set Functions Construction of the Lebesgue Measure Measure Spaces Measurable Functions Simple Functions Integration Comparison with the Riemann Integral Integration of Complex Functions Functions of Class L2 Exercises Bibliography List of Special Symbols Index
The notion of using entropy for a dynamical system is not new. One classical definition is that of the Kolmogorov-Sinai entropy @cite_28 which is related to how uncertainty (and information) increase with time in a chaotic attractor. However, this definition cannot be applied for regions of the parameter space that do not preserve volume. The definition presented in serves our purpose better, since it does not introduce such restrictions.