First, we propose a class of recurrent-like neural networks for NLP tasks that satisfy the differential equation DISPLAYFORM0

where DISPLAYFORM0

and where INLINEFORM0 and INLINEFORM1 are learned functions. INLINEFORM2 corresponds to traditional RNNs, with INLINEFORM3 . For INLINEFORM4 , this takes the form of RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state.  
 Question: What novel class of recurrent-like networks is proposed?
A network, whose learned functions satisfy a certain equation. The  network contains RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state.