How to Crawl Multiple Websites to find common Words (BeautifulSoup,Requests,Python3)
Question: I'm wondering how to crawl multiple different websites using beautiful
soup/requests without having to repeat my code over and over.

Here is my code right now:

    
    
    import requests
    from bs4 import BeautifulSoup
    from collections import Counter
    import pandas as pd
    Website1 = requests.get("http://www.nerdwallet.com/the-best-credit-cards")
    soup = BeautifulSoup(Website1.content)
    texts = soup.findAll(text=True)
    a = Counter([x.lower() for y in texts for x in y.split()])
    b = (a.most_common())
    makeaframe = pd.DataFrame(b)
    makeaframe.columns = ['Words', 'Frequency']
    print(makeaframe)
    

**What I am trying to do** is ideally crawl 5 different websites, find all of
the individual words on these websites, find the frequency of each word on
each website, ADD all the frequencies together for each particular word, then
combine all of this data into one dataframe that can be exported using Pandas.

Hopefully the output would look like this

    
    
    Word     Frequency
    the       200
    man       300
    is        400
    tired     300
    

My code can only do this for ONE website at a time right now and I'm trying to
avoid repeating my code.

Now, I can do this manually by repeating my code over and over and crawling
each individual website and then concatenating my results for each of these
dataframes together but that seems very unpythonic. I was wondering if anyone
had a faster way or any advice? Thank you!


Answer: Make a function:

    
    
    import requests
    from bs4 import BeautifulSoup
    from collections import Counter
    import pandas as pd
    
    cnt = Counter()
    def GetData(url):
     Website1 = requests.get(url)
     soup = BeautifulSoup(Website1.content)
     texts = soup.findAll(text=True)
     a = Counter([x.lower() for y in texts for x in y.split()])
     cnt.update(a.most_common())
    
    websites = ['http://www.nerdwallet.com/the-best-credit-cards','http://www.other.com']
    for url in websites:
     GetData(url)
    
    makeaframe = pd.DataFrame(cnt.most_common())
    makeaframe.columns = ['Words', 'Frequency']
    print makeaframe
    

