Analysis of the most common and salient words in a text I've been trying to get more understanding and experience in the Natural Language Processing space and, to get some more practice, decided to attempt to make a simple high-level analysis of the "Industrial Society and its Future" (aka "Unabomber Manifesto"). 

Here is what I am trying to do:


download the text of the Manifesto from the Washington Post website 
preprocess it using nltk: 


tokenize into words
remove the non-alpha words
remove English stop words 
lemmatize the tokens

count most commonly used words
use the TF-IDF model from gensim library to compute the most popular and most rarely used / important words


The code:

from bs4 import BeautifulSoup

from gensim.models.tfidfmodel import TfidfModel
from gensim.corpora.dictionary import Dictionary

from nltk import word_tokenize, Counter, sent_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords

import requests


ENGLISH_STOPS = set(stopwords.words('english'))


def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    response = requests.get(url)

    soup = BeautifulSoup(response.content, "html.parser")
    return soup.get_text()


def preprocess(text):
    """Pre-processes the text, splits into tokens that are lower-cased, filtered and lemmatized."""
    tokens = word_tokenize(text)

    # lowering
    lower_tokens = [token.lower() for token in tokens]

    # Retain alphabetic words: alpha_only
    alpha_only = [t for t in lower_tokens if t.isalpha()]

    # Remove all stop words: no_stops
    no_stops = [t for t in alpha_only if t not in ENGLISH_STOPS]

    # Lemmatize all tokens into a new list: lemmatized
    wordnet_lemmatizer = WordNetLemmatizer()
    lemmatized = [wordnet_lemmatizer.lemmatize(t) for t in no_stops]

    return lemmatized


def tf_idf(text):
    """
    Computes the TF-IDF of a text - using every sentence as a separate "document".
    Returns a list of tuples with words and weights.
    """
    texts = [preprocess(sentence) for sentence in sent_tokenize(text)]

    dictionary = Dictionary(texts)
    corpus = [dictionary.doc2bow(text) for text in texts]

    tfidf = TfidfModel(corpus)
    corpus_tfidf = tfidf[corpus]

    tfidf_weights = {dictionary.get(id): value
                     for doc in corpus_tfidf
                     for id, value in doc}
    sorted_tfidf_weights = sorted(tfidf_weights.items(), key=lambda w: w[1])

    return sorted_tfidf_weights


if __name__ == '__main__':
    content = retrieve_text("http://www.washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm")

    tokens = preprocess(content)
    token_counter = Counter(tokens)

    most_common = token_counter.most_common(10)
    tf_idf_results = tf_idf(content)
    popular_terms, rare_terms = tf_idf_results[:10], tf_idf_results[-10:]

    # print out words only (without counts and weights)
    keys = lambda x: next(zip(*x))
    print(f"Most common words: {keys(most_common)}")
    print(f"Most popular terms: {keys(popular_terms)}")
    print(f"Most unique/important terms: {keys(rare_terms)}")


Prints:

Most common words: ('society', 'system', 'people', 'power', 'would', 'one', 'human', 'technology', 'leftist', 'need')
Most popular terms: ('society', 'people', 'freedom', 'whole', 'human', 'lead', 'system', 'necessity', 'process', 'use')
Most unique/important terms: ('license', 'simplification', 'personnel', 'carried', 'crossroad', 'eminent', 'exactly', 'paramount', 'danger', 'virtue')


I am still learning and would like my code to be reviewed. Would appreciate any feedback about the code quality, performance or any other possible improvements.

Also, I am not 100% sure the TF-IDF model is applied in the most appropriate way. Currently, I am treating each sentence as a separate document. Is this a good approach to detect the most "important"/"valuable" terms/words, or should I use other articles/posts as documents for the corpus?
First, I will start with PEP 8 specifications. The PEP 8 analysis shows the following:

E501:17:80:line too long (94 &gt; 79 characters)
E501:25:80:line too long (99 &gt; 79 characters)
E501:46:80:line too long (82 &gt; 79 characters)
E501:66:80:line too long (114 &gt; 79 characters)
E731:76:5:do not assign a lambda expression, use a def
W292:79:62:no newline at end of file


If I am not incorrect, one of the things that you are trying is to calculate the frequency distribution of words, namely the most common words and most rare words. I tried to re-write your code like viz.

from bs4 import BeautifulSoup
from nltk import FreqDist, re

import requests

def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    response = requests.get(url)

    soup = BeautifulSoup(response.content, "html.parser")
    return soup.get_text()


if __name__ == '__main__':

    content = retrieve_text("http://www.washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm")
    wordList = re.sub("[^\w]", " ", content).split()
    fdist = FreqDist(wordList)

    # print out words only (without counts and weights)
    print(fdist.most_common(20))


As you see in the output, I am obviously not skipping connectorwords, like the,  of etc.:

[('the', 1735), ('of', 1251), ('to', 1078), ('a', 736), ('and', 719), ('that', 657), ('is', 611), ('in', 546), ('be', 412), ('for', 351), ('it', 319), ('or', 290), ('are', 281), ('have', 246), ('society', 239), ('not', 235), ('will', 233), ('as', 230), ('by', 229), ('they', 224)]


but, the execution was much faster and I did not have to spend a huge time downloading the gensim library. Off course if you need something as sophisticated as gensim, I have no substitute, else for frequency, this maybe much faster and you can easily write a method to remove connector words.