Understanding LSTM text input I am an beginner in text generation and and deep-learing but I like to get in touch with it. Currently I am learing about LSTM networks and VAEs for text generation. I would like to read a sequence at once and output another sequence. What I learned from this post is that the input shape should be 3D. Since my text data is 2D I want to reshape it. But this is now the confusing part. As I have training data with the shape of (nb_sequences, max_seq_lenth) = (274, 95). As the post pointed out, it would be important to reshape the data to a format of input = (nb_sequence, nb_timestep, nb_feature) with nb_sequence being the number of sequences (274), nb_timestep the length of the sequence (95) and nb_feature the number of features describing a sequence at a timestep, right?
I have two questions here:
1). How to reshape properly? My vocab size is 359 which I imagined (obviously wrong) to be the number of the features. --&gt; input = (274, 95, 359). This is not possible since an array of 274*95 = 26030 can't be reshaped into (274, 95, 359).
If I take the number of diffrent words my sequences are containing, wouldn't it lead to some erros since sentences do not consist out of the same words naturally? Meaning that I would have a variable feature number with each sequence?
2). How would one realize the problem I am facing? I read about one-hot encoding the sequences. This might solve the problem I have regarding question 1), but as far as I understand this would lead me to a quite inefficient way of predicting sequences as I want to handle more data. Is there any suggestion how to do it in a effient way?
The input to an LSTM must be a batch of sequences of vectors of real numbers, i.e. 3D tensor). Textual inputs are discrete tokens, so they are a batch of sequences of integers (indexes to the vocabulary table), i.e. a 2D tensor.
Before any reshaping, you must transform each integer value into a vector. For that, the usual approaches are one-hot encoding and embedding tables:

With one-hot encoding, you encode each integer index as a vector with 359 elements, where all are 0s except the position at the integer index, which is a 1.

With an embedding table, you keep a table with 359 vectors of an arbitrary dimensionality (the embedding size).


In either case, you get the &quot;extra dimension&quot; that you were lacking with integer indices.