 Hello, I am very new to data science, machine learning, and stack overflow. Excuse me for being unclear or asking naive questions.  My question is as follows: From any given document, I am trying to classify it according to the emotions it evokes in readers, using a neural network. However, I am having difficulties with feature selection. I'm thinking of using NLTK and RAKE to extract keywords, but I don't know how I can translate them into features. Should I hash the keywords for one feature? Or, should I find a dictionary of english words (i.e. Wordnet), and use every word in the dictionary as a feature.
Which features do I select from text?
machine learning;classification;neural network;text mining;feature selection
Using NLTK in python you should first Tokenize the sentences into words, even you can use Ngram for 2-Gram or 3-Gram bags of word, the reason I am suggesting N-Gram is that let's suppose you have sentence like: I am not happy with this product, then 2-Gram tokenize it as ['not happy', 'happy with', 'with this', 'this product'] here I and am are assumed as STOPWORDS. Using HashingTF you can hash the sentence into a feature vector as ['word position': frequency of word, ...] i.e highly sparse vectors, For Hashing in PySpark check this documentation. Here below python code will help you to tokenize in bags of wordimport stringfrom nltk.tokenize import word_tokenizefrom nltk.corpus import stopwordsfrom nltk.stem.porter import PorterStemmerPUNCTUATION = set(string.punctuation)STOPWORDS = set(stopwords.words('english'))STEMMER = PorterStemmer()example = ['Hello Krishna Prasad, this is test file for spark testing',            'Another episode of star',            'There are far and away many stars'            'A galloping horse using two coconuts'            'My kingdom for a horse'            'A long time ago in a galaxy far']def tokenize(text):    tokens = word_tokenize(text)    lowercased = [t.lower() for t in tokens]    no_punctuation = []    for word in lowercased:        punct_removed = ''.join([letter for letter in word if not letter in PUNCTUATION])        no_punctuation.append(punct_removed)    no_stopwords = [w for w in no_punctuation if not w in STOPWORDS]    stemmed = [STEMMER.stem(w) for w in no_stopwords]    return [w for w in stemmed if w]tokenized_word = [tokenize(text) for text in example]for word in tokenized_word:    print wordOut of the above code as:$python WordFrequencyHash.py[u'hello', u'krishna', u'prasad', u'test', u'file', u'spark', u'test'][u'anoth', u'episod', u'star'][u'far', u'away', u'mani', u'starsa', u'gallop', u'hors', u'use', u'two', u'coconutsmi', u'kingdom', u'horsea', u'long', u'time', u'ago', u'galaxi', u'far']You can also use word2vec or countvectorizer for tokenization.