A simple PyPI crawler
Here are some comments/notes about the code and potential improvements:


use with context manager when opening files
parsing HTML with regular expressions has always been a very controversial thing to do. I would switch to an HTML parser like BeautifulSoup and lxml.html. For example, getting all the PyPI links with BeautifulSoup can be as straightforward as:

from bs4 import BeautifulSoup

with open('list.html') as directory:
    soup = BeautifulSoup(directory, "html.parser")
    for link in soup.select("a[href*=pypi]"):
        print(link.get_text())


where a[href*=pypi] is a CSS selector that would match all a elements that has pypi substring inside an href attribute.
instead of using requests.get() directly, initialize a "session" to reuse the underlying TCP connection:


  ..if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase..



If you want to scale this up, you would need to switch for a synchronous and blocking code/approach to asynchronous - look into using Scrapy web-scraping framework which is based on the twisted networking library.