Using python and beautifulsoup to iterate through a list of websites to find a particular string
First of all, since you are applying a regular expression pattern to the complete source of the page, you don't need an HTML parser like BeautifulSoup - directly search inside the page.content.

And, if you need to go the HTML parsing route and speed matters, choose either lxml, or lxml parser with BeautifulSoup.

You may also look into reusing the same requests.Session() instance - it may have a positive impact on performance.



Overall though, your approach is blocking/synchronous - your code processes URLs one by one - it would not process the next URL until it is done with the current one. Look into tools like Scrapy to approach the problem in the asynchronous/non-blocking fashion.