Why feature extraction of text don't return all possible feature names?
Here is the snippet of code from the book 
Natural Language Processing with PyTorch:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import seaborn as sns

corpus = ['Time flies flies like an arrow.', 'Fruit flies like a banana.']
one_hot_vectorizer = CountVectorizer()
vocab = one_hot_vectorizer.get_feature_names()


The value of vocab : 

vocab = ['an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time']


Why is not there an 'a' among the extracted feature names? If it is excluded as too common word automatically, why "an" is not excluded for the same reasons? How to make .get_feature_names() filter other words as well?

Very good question! Though this is not a pytorch question but a sklearn one =)

I encourage to first go through this https://www.kaggle.com/alvations/basic-nlp-with-nltk, esp. the "Vectorization with sklearn" section



TL;DR

If we use the CountVectorizer, 

from io import StringIO
from sklearn.feature_extraction.text import CountVectorizer

sent1 = "The quick brown fox jumps over the lazy brown dog."
sent2 = "Mr brown jumps over the lazy fox."

with StringIO('\n'.join([sent1, sent2])) as fin:
    # Create the vectorizer
    count_vect = CountVectorizer()
    count_vect.fit_transform(fin)

# We can check the vocabulary in our vectorizer
# It's a dictionary where the words are the keys and 
# The values are the IDs given to each word. 
print(count_vect.vocabulary_)


[out]:

{'brown': 0,
 'dog': 1,
 'fox': 2,
 'jumps': 3,
 'lazy': 4,
 'mr': 5,
 'over': 6,
 'quick': 7,
 'the': 8}


We didn't tell the vectorizer to remove punctuation and tokenize and lowercase, how did they do it?

Also, the is in the vocabulary, it's a stopword, we want it gone... 
And jumps isn't stemmed or lemmatized!

If we look at the documentation of the CountVectorizer in sklearn, we see:

CountVectorizer(
    input=’content’, encoding=’utf-8’, 
    decode_error=’strict’, strip_accents=None, 
    lowercase=True, preprocessor=None, 
    tokenizer=None, stop_words=None, 
    token_pattern=’(?u)\b\w\w+\b’, ngram_range=(1, 1), 
    analyzer=’word’, max_df=1.0, min_df=1, 
    max_features=None, vocabulary=None, 
    binary=False, dtype=&lt;class ‘numpy.int64’&gt;)


And more specifically:


  analyzer : string, {‘word’, ‘char’, ‘char_wb’} or callable
  
  Whether the feature should be made of word or character n-grams.
  Option ‘char_wb’ creates character n-grams only from text inside word
  boundaries; n-grams at the edges of words are padded with space. If a
  callable is passed it is used to extract the sequence of features out
  of the raw, unprocessed input.
  
  preprocessor : callable or None (default)
  
  Override the preprocessing (string transformation) stage while
  preserving the tokenizing and n-grams generation steps.
  
  tokenizer : callable or None (default)
  
  Override the string tokenization step while preserving the
  preprocessing and n-grams generation steps. Only applies if analyzer
  == 'word'.
  
  stop_words : string {‘english’}, list, or None (default)
  
  If ‘english’, a built-in stop word list for English is used. If a
  list, that list is assumed to contain stop words, all of which will be
  removed from the resulting tokens. Only applies if analyzer == 'word'.
  If None, no stop words will be used.
  
  lowercase : boolean, True by default
  
  Convert all characters to lowercase before tokenizing.


But in the case of the example from http://shop.oreilly.com/product/0636920063445.do, it's not exactly the stopwords causing the issue. 

If we explicitly use the English stopwords from https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py

&gt;&gt;&gt; from sklearn.feature_extraction.text import CountVectorizer
&gt;&gt;&gt; one_hot_vectorizer = CountVectorizer(stop_words='english')

&gt;&gt;&gt; one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=&lt;class 'numpy.int64'&gt;, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words='english',
        strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)

&gt;&gt;&gt; one_hot_vectorizer.get_feature_names()
['arrow', 'banana', 'flies', 'fruit', 'like', 'time']


So what exactly is happening in the case where the stop_words argument is left as None?

Lets try an experiment where I add some single character words to the input:

&gt;&gt;&gt; corpus = ['Time flies flies like an arrow 1 2 3.', 'Fruit flies like a banana x y z.']

&gt;&gt;&gt; one_hot_vectorizer = CountVectorizer()

&gt;&gt;&gt; one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=&lt;class 'numpy.int64'&gt;, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
        tokenizer=None, vocabulary=None)
&gt;&gt;&gt; one_hot_vectorizer.get_feature_names()                                         
['an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time']


They're all gone again!!! 

Now if we dig into the docs, https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L738


  token_pattern : string
          Regular expression denoting what constitutes a "token", only used
          if analyzer == 'word'. The default regexp select tokens of 2
          or more alphanumeric characters (punctuation is completely ignored
          and always treated as a token separator).


Ah ha, that's why all the single character tokens gets deleted!

The default pattern for CountVectorizer is token_pattern=r"(?u)\b\w\w+\b", to enable it to take single character, you can try:

&gt;&gt;&gt; one_hot_vectorizer = CountVectorizer(token_pattern=r"(?u)\b\w+\b")           
&gt;&gt;&gt; one_hot_vectorizer.fit(corpus)
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=&lt;class 'numpy.int64'&gt;, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), preprocessor=None, stop_words=None,
        strip_accents=None, token_pattern='(?u)\\b\\w+\\b', tokenizer=None,
        vocabulary=None)
&gt;&gt;&gt; one_hot_vectorizer.get_feature_names()
['1', '2', '3', 'a', 'an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time', 'x', 'y', 'z']

