Crawl multiple pages at once
While I can't answer to your main question about crawling multiple pages at once,
I have a couple of comments about this code.

get_page_links

In this code, it seems that base_url and the if that follows should be outside of the for loop:


links = []
for link in page[1].find_all('a'):
    url = link.get('href')
    base_url = (uparse(page[0]).scheme + '://' + uparse(page[0]).netloc)
    if base_url not in robot_set:
        robot_set.add(base_url)
        rp.set_url(urlparse.urljoin(base_url,'robots.txt'))
        rp.read()
    abs_url = urlparse.urljoin(base_url, url)
    if rp.can_fetch('*', abs_url): 
        links.append(abs_url)
links = set(links)



Also, instead of links = set(links),
it would be better to make links a set in the first place.

Rewriting the above code like this would be more efficient:

links = set()
base_url = uparse(page[0]).scheme + '://' + uparse(page[0]).netloc
if base_url not in robot_set:
    robot_set.add(base_url)
    rp.set_url(urlparse.urljoin(base_url, 'robots.txt'))
    rp.read()
for link in page[1].find_all('a'):
    url = link.get('href')
    abs_url = urlparse.urljoin(base_url, url)
    if rp.can_fetch('*', abs_url):
        links.add(abs_url)


get_page

It would be better to specify in the except clause the type of error that can be raised.

get_page_words

This expression contains the non-ascii character «,
and the posted code doesn't contain an encoding declaration:


to_remove = '{}[]«#$%^&amp;*._,1234567890+=&lt;&gt;/\()":;!?'



You can declare utf-8 encoding by adding this line at the top of the file:

# coding=utf-8





to_remove = '{}[]«#$%^&amp;*._,1234567890+=&lt;&gt;/\()":;!?'
for i in to_remove:
    words = words.replace(i, ' ').lower()



One problem here is that words is lowercased in every iteration,
when it would be enough to lowercase just once, before starting the loop.

But the bigger problem is the loop itself:
it would be much more efficient to use a regular expression to get rid of all the unwanted characters, for example like this:

import re
re_nonword = re.compile(r'[{}[\]«#$%^&amp;*._,1234567890+=&lt;&gt;/\()":;!?]')
words = re_nonword.sub(' ', words)


set literals

Instead of this:


bad_words = set(['http','www','com','https','gov','org','edu','imgs'])



As of Python 2.7, you can write using set literals, like this:

bad_words = {'http', 'www', 'com', 'https', 'gov', 'org', 'edu', 'imgs'}


I also put spaces after the commas, to follow PEP8.