Neural Networks and Complex Valued Inputs
The power of complex representations remains an open topic to me. I still do strive the understand Fourier transformations. 

An underlying question is, to me: why would complex transformations be useful for real data? More generally, when data dwell in a set $S$, is $S$ the most appropriate set of analysis, or is it more appropriate to resort to a bigger set $S^*$? For instance, for real-valued polynomials, we know that the field of complex numbers provides a more elegant extension. This might not be so distant, as $z$-transforms (extended polynomials) are tools of choice for real time-series analysis. Real linear-time invariant systems are root signals (here eigenvectors) that are complex (cisoids). To better separate frequencies, time-frequency analyses often employ analytic signals and the Hilbert transform, or analytic, dual-tree multiscale tools like wavelets. The recent paper Complex-Valued Signal Processing: The Proper Way to Deal With Impropriety deals with more stochastic observations:


  Complex-valued signals occur in many areas of science and engineering
  and are thus of fundamental interest. In the past, it has often been
  assumed, usually implicitly, that complex random signals are proper or
  circular. A proper complex random variable is uncorrelated with its
  complex conjugate, and a circular complex random variable has a
  probability distribution that is invariant under rotation in the
  complex plane. While these assumptions are convenient because they
  simplify computations, there are many cases where proper and circular
  random signals are very poor models of the underlying physics. When
  taking impropriety and noncircularity into account, the right type of
  processing can provide significant performance gains. There are two
  key ingredients in the statistical signal processing of complex-valued
  data: 1) utilizing the complete statistical characterization of
  complex-valued random signals; and 2) the optimization of real-valued
  cost functions with respect to complex parameters. In this overview
  article, we review the necessary tools, among which are widely linear
  transformations, augmented statistical descriptions, and Wirtinger
  calculus. We also present some selected recent developments in the
  field of complex-valued signal processing, addressing the topics of
  model selection, filtering, and source separation.


But going back in time (Oppenheim's works for instance), one knows that complex phase can capture non-stationarity, discontinuity (edges and jumps) and oscillatory behaviors (textures). My present belief is that, at a given scale, a complex feature computed as a whole is more efficient at capturing local behavior that a pair of real and imaginary parts computed somewhat separately, incorporating some invariances with respect to translation or rotation. 

As for neural networks, and of course deep learning, the recent theory of scattering networks, and subsequent works, have provided a solid ground for understanding how deep learning works, from a solid mathematical point of view, based on complex wavelet frames and non-linear operators. Another interesting paper for your question is:

M. Tygert et al., 2016, A Mathematical Motivation for Complex-Valued Convolutional Networks, Neural Computation


  Courtesy of the exact correspondence, the remarkably rich and rigorous
  body of mathematical analysis for wavelets applies directly to
  (complex-valued) convnets.
  
  Convolutional networks (convnets) have become increasingly important
  to artificial intelligence in recent years, as reviewed by LeCun,
  Bengio, and Hinton (2015).This note presents a theoretical argument
  for complex-valued convnets and their remarkable performance.
  Complex-valued convnets turn out to calculate data-driven multiscale
  windowed spectra characterizing certain stochastic processescommonin
  the modeling of time series (such as audio) and natural images
  (including patterns and textures).We motivate the construction of such
  multiscale spectra using local averages of multiwavelet absolute
  values or, more generally, nonlinear multiwavelet packets.


And maybe, complex numbers  are not enough, and we should turn to quaternions...