deep learning for text and sequences understanding recurrent neural networks major characteristic of all neural networks you've seen so farsuch as densely connected networks and convnetsis that they have no memory each input shown to them is processed independentlywith no state kept in between inputs with such networksin order to process sequence or temporal series of data pointsyou have to show the entire sequence to the network at onceturn it into single data point for instancethis is what you did in the imdb examplean entire movie review was transformed into single large vector and processed in one go such networks are called feedforward networks in contrastas you're reading the present sentenceyou're processing it word by word--or rathereye saccade by eye saccade--while keeping memories of what came beforethis gives you fluid representation of the meaning conveyed by this sentence biological intelligence processes information incrementally while maintaining an internal model of what it' processingbuilt from past information and constantly updated as new information comes in recurrent neural network (rnnadopts the same principlealbeit in an extremely simplified versionit processes sequences by iterating through the sequence elements and maintaining state containing information relative output to what it has seen so far in effectan rnn is type of neural network that has an internal loop (see figure the state of the rnn is reset between processing two difrnn recurrent ferentindependent sequences (such as two different connection imdb reviews)so you still consider one sequence sininput gle data pointa single input to the network what changes is that this data point is no longer processed in figure recurrent single stepratherthe network internally loops over networka network wit loop sequence elements to make these notions of loop and state clearlet' implement the forward pass of toy rnn in numpy this rnn takes as input sequence of vectorswhich you'll encode as tensor of size (timestepsinput_featuresit loops over timestepsand at each timestepit considers its current state at and the input at (of shape (input_ features,)and combines them to obtain the output at you'll then set the state for the next step to be this previous output for the first timestepthe previous output isn' definedhencethere is no current state soyou'll initialize the state as an allzero vector called the initial state of the network in pseudocodethis is the rnn listing pseudocode rnn state_t the state at for input_t in input_sequenceiterates over sequence elements output_t (input_tstate_tstate_t output_t the previous output becomes the state for the next iteration licensed to