1. A system comprising: a computer-readable memory storing executable instructions; and one or more computer processors in communication with the computer-readable memory, wherein the one or more computer processors are programmed by the executable instructions to at least: obtain (1) a first transcription of a first utterance of the user and (2) a second transcription of a second utterance of the user; generate a first interaction history vector using a neural-network-based interaction history encoder, the first transcription, and data representing a non-verbal interaction of the user with displayed content, wherein the first interaction history vector comprises a first plurality of values, wherein individual values of the first plurality of values are determined using the first transcription and the data representing the non-verbal interaction, and wherein generating the first interaction history vector modifies a persistent internal state of the interaction history encoder; store state data regarding the persistent internal state of the interaction history encoder; obtain audio data regarding a current utterance of a user; generate a plurality of transcription hypotheses for the current utterance using the audio data and an acoustic model; restore the persistent internal state of the interaction history encoder using the state data; generate a second interaction history vector using the interaction history encoder and the second transcription, wherein the second interaction history vector comprises a second plurality of values, wherein individual values of the second plurality of values are determined using the persistent internal state of the interaction history encoder and the second transcription, and wherein the second interaction history vector represents a cluster of a plurality of clusters of users; generate a first score for a first transcription hypothesis of the plurality of transcription hypotheses using a neural-network-based language model decoder and the second interaction history vector, wherein the first score is generated as a function of (1) a first value representing at least a portion of the first transcription hypothesis, (2) a second value of the second plurality of values of the second interaction history vector, and (3) the cluster; and select the first transcription hypothesis as a transcription of the current utterance based on the first score being greater than a second score associated with a second transcription hypothesis of the plurality of transcription hypotheses.