17. The system of claim 13 , wherein the neural-network-based encoder comprises a plurality of layers of nodes, and wherein a first value associated with a first node of a first layer of the plurality of layers is modified during processing the portion of the interaction history data, and wherein the internal state of the neural-network-based encoder comprises the first value.
13. A system comprising: a computer-readable memory storing executable instructions; and one or more computer processors in communication with the computer-readable memory, wherein the one or more computer processors are programmed by the executable instructions to perform a process comprising: generating language model input data regarding an utterance of a user, wherein the language model input data is generated using an acoustic model and audio data regarding the utterance; obtaining state data regarding an internal state of a neural-network-based encoder reached during generation of a prior interaction history vector for the user; generating an interaction history vector using the neural-network-based encoder and the state data, wherein the interaction history vector comprises an encoded representation of a plurality of prior interactions of the user, and wherein the plurality of prior interactions comprises: a verbal interaction, and a separate non-verbal interaction with displayed content; processing the language model input data and the interaction history vector using a neural-network-based decoder to generate scores for individual transcription hypotheses of a plurality of transcription hypotheses; and selecting a transcription of the utterance using the scores for the individual transcription hypotheses.