Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
  This article concerns the expressive power of depth in neural nets with ReLU
activations and bounded width. We are particularly interested in the following
questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of
width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous
function on the unit cube $[0,1]^d$ aribitrarily well? For ReLU nets near this
minimal width, what can one say about the depth necessary to approximate a
given function? Our approach to this paper is based on the observation that,
due to the convexity of the ReLU activation, ReLU nets are particularly
well-suited for representing convex functions. In particular, we prove that
ReLU nets with width $d+1$ can approximate any continuous convex function of
$d$ variables arbitrarily well. These results then give quantitative depth
estimates for the rate of approximation of any continuous scalar function on
the $d$-dimensional cube $[0,1]^d$ by ReLU nets with width $d+3.$
