when is substituted with Long short-term memory There has been success in using long short-term memory (LSTM) models to generate paraphrases. In short, the model consists of an encoder and decoder component, both implemented using variations of a stacked residual LSTM. First, the encoding LSTM takes a one-hot encoding of all the words in a sentence as input and produces a final hidden vector, which can represent the input sentence. The decoding LSTM takes the hidden vector as input and generates a new sentence, terminating in an end-of-sentence token. The encoder and decoder are trained to take a phrase and reproduce the one-hot distribution of a corresponding paraphrase by minimizing perplexity using simple stochastic gradient descent. New paraphrases are generated by inputting a new phrase to the encoder and passing the output to the decoder. Transformers With the introduction of Transformer models, paraphrase generation approaches improved their ability to generate text by scaling neural network parameters and heavily parallelizing training through feed-forward layers. These models are so fluent in generating text that human experts cannot identify if an example was human-authored or machine-generated. Transformer-based paraphrase generation relies on autoencoding, autoregressive, or sequence-to-sequence methods. Autoencoder models predict word replacement candidates with a one-hot distribution over the vocabulary, while autoregressive and seq2seq models generate new text based on the source predicting one word at a time. More advanced efforts also exist to make paraphrasing controllable according to predefined quality dimensions, such as semantic preservation or lexical diversity. Many Transformer-based paraphrase generation methods rely on unsupervised learning to leverage large amounts of training data and scale their methods. Paraphrase recognition Recursive Autoencoders Paraphrase recognition has been attempted by Socher et al through the use of recursive autoencoders. The main concept is to produce a vector representation of a sentence and its components by recursively using an autoencoder. The vector representations of paraphrases should have similar vector representations; they are processed, then fed as input into a neural network for classification. Given a sentence with words, the autoencoder is designed to take 2 -dimensional word embeddings as input and produce an -dimensional vector as output. The same autoencoder is applied to every pair of words in to produce vectors. The autoencoder is then applied recursively with the new vectors as inputs until a single vector is produced. Given an odd number of inputs, the first vector is forwarded as-is to the next level of