5. A computer-implemented method comprising: as performed by one or more computing devices configured with specific computer-executable instructions, generating language model input data regarding an utterance of a user, wherein the language model input data is generated using an acoustic model and audio data regarding the utterance, and wherein the language model input data is associated with a plurality of transcription hypotheses for the utterance; obtaining interaction history data regarding a prior interaction of the user with a computing system; obtaining encoder state data regarding an internal state of a neural-network-based encoder reached during generation of a prior interaction history vector for the user; restoring the internal state of the neural-network-based encoder using the encoder state data; processing at least a portion of the interaction history data using the neural-network-based encoder to generate an interaction history vector, wherein the interaction history vector comprises an encoded representation of at least a portion of a plurality of prior interactions of the user, and wherein the plurality of prior interactions comprises a verbal interaction, and a separate non-verbal interaction with displayed content; processing the language model input data and the interaction history vector using a neural-network-based decoder to generate scores for individual transcription hypotheses of the plurality of transcription hypotheses; and selecting a transcription of the utterance using the scores for the individual transcription hypotheses.
5. A computer-implemented method comprising: as performed by one or more computing devices configured with specific computer-executable instructions, generating language model input data regarding an utterance of a user, wherein the language model input data is generated using an acoustic model and audio data regarding the utterance, and wherein the language model input data is associated with a plurality of transcription hypotheses for the utterance; obtaining interaction history data regarding a prior interaction of the user with a computing system; obtaining encoder state data regarding an internal state of a neural-network-based encoder reached during generation of a prior interaction history vector for the user; restoring the internal state of the neural-network-based encoder using the encoder state data; processing at least a portion of the interaction history data using the neural-network-based encoder to generate an interaction history vector, wherein the interaction history vector comprises an encoded representation of at least a portion of a plurality of prior interactions of the user, and wherein the plurality of prior interactions comprises a verbal interaction, and a separate non-verbal interaction with displayed content; processing the language model input data and the interaction history vector using a neural-network-based decoder to generate scores for individual transcription hypotheses of the plurality of transcription hypotheses; and selecting a transcription of the utterance using the scores for the individual transcription hypotheses. 8. The computer-implemented method of claim 5 , wherein the interaction history data comprises data representing at least one of: an utterance, a search request, a purchase, or content access.