Finding specific URLs from a list of URLs using Python
Question: I want find if specific links exist in a list of URLs by crawling through
them. I have written the following program and it works perfectly. However, I
am stuck at 2 places.

  1. Instead of using an array, how can I call the links from a text file.
  2. The crawler takes close to 4 minutes to crawl through 100 webpages.

Is there a way I can make that faster.

    
    
    from bs4 import BeautifulSoup, SoupStrainer
    import urllib2
    import re
    import threading
    
    start = time.time()
    #Links I want to find
    url = "example.com/one", "example.com/two", "example.com/three"]
    
    #Links I want to find the above links in...
    url_list =["example.com/1000", "example.com/1001", "example.com/1002",
    "example.com/1003", "example.com/1004"]
    
    print_lock = threading.Lock()
    #with open("links.txt") as f:
    #  url_list1 = [url.strip() for url in f.readlines()]
    
    def fetch_url(url):
        for line1 in url_list:
            print "Crawled" " " + line1
            try:
                html_page = urllib2.urlopen(line1)
                soup = BeautifulSoup(html_page)
                link = soup.findAll(href=True)
            except urllib2.HTTPError:
            pass
            for link1 in link:
                url1 = link1.get("href")
                for url_input in url:
                    if url_input in url1:
                        with print_lock:
                            print 'Found' " " +url_input+ " " 'in'+ " " + line1
    
    threads = [threading.Thread(target=fetch_url, args=(url,)) for url in url_list]
    for thread in threads:
    thread.start()
    for thread in threads:
    thread.join()
    print('Entire job took:',time.time() - start) 
    


Answer: If you want to read from a textfile, use the code you commented out.

As for the "performance" problem: Your code blocks at the read operation
`urlopen` until the content from the website is returned. You ideally want to
run those requests in parallel. You need a parallelized solution, by using
threads for example.

[Here's](https://github.com/gevent/gevent/blob/master/examples/concurrent_download.py)
an example using a different approach, using gevent (non-standard)

