Analysis of the most common and salient words in a text
First, I will start with PEP 8 specifications. The PEP 8 analysis shows the following:

E501:17:80:line too long (94 &gt; 79 characters)
E501:25:80:line too long (99 &gt; 79 characters)
E501:46:80:line too long (82 &gt; 79 characters)
E501:66:80:line too long (114 &gt; 79 characters)
E731:76:5:do not assign a lambda expression, use a def
W292:79:62:no newline at end of file


If I am not incorrect, one of the things that you are trying is to calculate the frequency distribution of words, namely the most common words and most rare words. I tried to re-write your code like viz.

from bs4 import BeautifulSoup
from nltk import FreqDist, re

import requests

def retrieve_text(url):
    """Downloads HTML content from a URL, parses with an HTML parser and returns text only."""
    response = requests.get(url)

    soup = BeautifulSoup(response.content, "html.parser")
    return soup.get_text()


if __name__ == '__main__':

    content = retrieve_text("http://www.washingtonpost.com/wp-srv/national/longterm/unabomber/manifesto.text.htm")
    wordList = re.sub("[^\w]", " ", content).split()
    fdist = FreqDist(wordList)

    # print out words only (without counts and weights)
    print(fdist.most_common(20))


As you see in the output, I am obviously not skipping connectorwords, like the,  of etc.:

[('the', 1735), ('of', 1251), ('to', 1078), ('a', 736), ('and', 719), ('that', 657), ('is', 611), ('in', 546), ('be', 412), ('for', 351), ('it', 319), ('or', 290), ('are', 281), ('have', 246), ('society', 239), ('not', 235), ('will', 233), ('as', 230), ('by', 229), ('they', 224)]


but, the execution was much faster and I did not have to spend a huge time downloading the gensim library. Off course if you need something as sophisticated as gensim, I have no substitute, else for frequency, this maybe much faster and you can easily write a method to remove connector words.