Creating sequence vector from text in Python
Question: I am now trying to prepare the input data for LSTM-based NN. I have some big
number of text documents and what i want is to make sequence vectors for each
document so i am able to feed them as train data to LSTM RNN.

My poor approach:

    
    
    import re
    import numpy as np
    #raw data
    train_docs = ['this is text number one', 'another text that i have']
    
    #put all docs together
    train_data = ''
    for val in train_docs:
        train_data += ' ' + val
    
    tokens = np.unique(re.findall('[a-zа-я0-9]+', train_data.lower()))
    voc = {v: k for k, v in dict(enumerate(tokens)).items()}
    

and then brutforce replace each doc with a "voc" dict.

Is there any libs which can help with this task?


Answer: You could use NLTK to tokenise the training documents. NLTK provides a
standard word tokeniser or allows you to define your own tokeniser (e.g.
RegexpTokenizer). Take a look
[here](http://www.nltk.org/api/nltk.tokenize.html) for more details about the
different tokeniser functions available.

[Here](http://www.nltk.org/book/ch01.html) might also be helpful for pre-
processing the text.

A quick demo using NLTK's pre-trained word tokeniser below:

    
    
    from nltk import word_tokenize
    
    train_docs = ['this is text number one', 'another text that i have']
    train_docs = ' '.join(map(str, train_docs))
    
    tokens = word_tokenize(train_docs)
    voc = {v: k for k, v in dict(enumerate(tokens)).items()}
    

