Creating a python algorithm to train a keras model to predict a large sequence of integers
One hot encoding increases number of columns according to unique categories in data set. I think you should check the performance of model with just using tokenizer not both. Because most of the time tokenizer alone performs very well.
I'm new to machine learning but I'm trying to apply it to a project I have. I was able to train a model to convert words from one language to another using LSTM layers. Say I use A as input to my model and I get B as output. What I do is: 

'original word' -> word embedding -> one-hot encode (A) -> MODEL -> one-hot encoded output (B) -> word embedding -> 'translated word'

This is relatively simple as I'm using a character-level tokenizer to encode the words and that does not require much memory (small sequences, one for each word). 
However, I now have to train a model that takes B as input and gives me C (no longer a translation problem). C is later going to be used for different purposes. The difference is that C can have a length of say 315 numbers and each of them can be one of 5514 unique values i.e., shape(215, 5514). Generically what I want to do is, for example:

'banana' -> (some processing, word embedding or one-hot) -> MODEL -> [434, 434, 410, 321, 225, 146, 86, 43, 13, -8, -23, -32, -38, -41, -13, 101, 227, 332, 411, 470, 515, 550, 577, 597, 611, 622, 628, 622, 608, 593, 580, 570, 561, 554, 549, 547, 548, 548, 549, 555, 564, 572, 579, 584, 587, 589, 590, 591, 591, 591, 590, 590, 584, 567, 550, 535, 524, 516, 511, 506, 503, 503, 507, 511, 518, 530, 543, 553, 561, 568, 573, 577, 580, 582, 584, 585, 586, 586, 587, 587, 588, 588, 588, 588, 588, 586]

So the problem is that I don't have enough memory to perform a one-hot encoding of the output sequences. I tried using generators to load each sequence from the disk instead of loading all of them from memory but It doesn't seem to be working.
Do you have any suggestions as to how I should approach this problem?
EDIT:
The dataset I'm using has the following format: n lines, each line contains 2 columns separated by a tab. The first column is the input word and the second column is the sequence I want to obtain if the input is that word.