How to compare an item in a queue to an item in a set?
Question: REDIT: Was trying to avoid just placing the entire block of code on the forum
and saying fix it for me, but here it is, to simply the process of determining
the error:

    
    
    #! /usr/bin/python2.6
    import threading
    import Queue
    import sys
    import urllib
    import urllib2
    from urlparse import urlparse
    from lxml.html import parse, tostring, fromstring
    
    THREAD_NUMBER = 1
    
    
    class Crawler(threading.Thread):
    
     def __init__(self, queue, mal_urls, max_depth):
        self.queue = queue
        self.mal_list = mal_urls
        self.crawled_links = []
        self.max_depth = max_depth
        self.count = 0 
        threading.Thread.__init__(self) 
    
     def run(self):
        while True:
            if self.count <= self.max_depth:
                self.crawled = set(self.crawled_links)
                url = self.queue.get()
                if url not in self.mal_list:
                    self.count += 1
                    self.crawl(url)
                else:
                    #self.queue.task_done()
                    print("Malicious Link Found: {0}".format(url))
                    continue
            else:
                self.queue.task_done()
                break
        print("\nFinished Crawling! Reached Max Depth!")
        sys.exit(2)
    
    def crawl(self, tgt):
        try:
            url = urlparse(tgt)
            self.crawled_links.append(tgt)
            print("\nCrawling {0}".format(tgt))
            request = urllib2.Request(tgt)
            request.add_header("User-Agent", "Mozilla/5,0")
            opener = urllib2.build_opener()
            data = opener.open(request)
    
        except: # TODO: write explicit exceptions the URLError, ValueERROR ...
            return
    
        doc = parse(data).getroot()
        for tag in doc.xpath("//a[@href]"):
            old = tag.get('href')
            fixed = urllib.unquote(old)
            self.queue_links(fixed, url)
    
    
     def queue_links(self, link, url):
    
        if link.startswith('/'):
            link = "http://" + url.netloc + link
    
        elif link.startswith("#"):
            return
    
        elif not link.startswith("http"):
            link = "http://" + url.netloc + "/" + link
    
    
        if link not in self.crawled_links:
            self.queue.put(link)
            self.queue.task_done()
        else:
            return
    
    
    def make_mal_list():
    """Open various malware and phishing related blacklists and create a list 
    of URLS from which to compare to the crawled links
    """
    
    hosts1 = "hosts.txt"
    hosts2 = "MH-sitelist.txt"
    hosts3 = "urls.txt"
    
    mal_list = []
    
    with open(hosts1) as first:
        for line1 in first:
            link = "http://" + line1.strip()
            mal_list.append(link)
    
    with open(hosts2) as second:
        for line2 in second:
            link = "http://" + line2.strip()
            mal_list.append(link)
    
    with open(hosts3) as third:
        for line3 in third:
            link = "http://" + line3.strip()
            mal_list.append(link)
    
    return mal_list
    
    def main():
        x = int(sys.argv[2])
        queue = Queue.Queue()
    
        mal_urls = set(make_mal_list())
        for i in xrange(THREAD_NUMBER):
            cr = Crawler(queue, mal_urls, x)
            cr.start()
    
    
       queue.put(sys.argv[1])
    
       queue.join()
    
    
    if __name__ == '__main__':
        main()
    

So what I've got going on here is a web spider, which first creates a set made
of the lines of several text files which contain 'malicious links'. Then
starts a thread, passing both the set of bad links, and sys.argv[1]. The
started thread, then calls teh crawl function which retrieves an lxml.html
parse from sys.argv[1], and then after parsing all the links out of that
initial page, places them in the queue. The loop continues, with each link
placed in the queue being removed with self.queue.get(). The corresponding
link is then SUPPOSED to be compared against the set of bad links. If the link
in found to be bad, the loop is supposed to output it to the screen and then
continue on to the next link, UNLESS it has already crawled that link.

If it is not bad, crawl it, parse it, place its links into the queue, etc,
incrementing a counter each time a link is crawled, and continuing until the
counter reaches a limit determined by the value passed as sys.argv[2]. The
problem is that, items it should be triggering the if/else statement for 'if
url not in mal_list' are not, and links that have been placed in the
'crawled_already' list, are being crawled a 2nd, 3rd, and forth time anyhow.


Answer: I don't understand one detail of this code: the queue is marked as `task_done`
if there is any new link found in `self.queue_links`, but not as a matter of
course in `self.crawl`. I'd have thought that this code would make more sense:

    
    
    def crawl(self, tgt):
        try:
            url = urlparse(tgt)
            self.crawled_links.append(tgt)
            print("\nCrawling {0}".format(tgt))
            request = urllib2.Request(tgt)
            request.add_header("User-Agent", "Mozilla/5,0")
            opener = urllib2.build_opener()
            data = opener.open(request)
            doc = parse(data).getroot()
            for tag in doc.xpath("//a[@href]"):
                old = tag.get('href')
                fixed = urllib.unquote(old)
                self.queue_links(fixed, url)
            self.queue.task_done()
        except: # TODO: write explicit exceptions the URLError, ValueERROR ...
            pass
    
    def queue_links(self, link, url):
        if not link.startswith("#"):
            if link.startswith('/'):
                link = "http://" + url.netloc + link
            elif not link.startswith("http"):
                link = "http://" + url.netloc + "/" + link
            if link not in self.crawled_links:
                self.queue.put(link)
    

I can't say, though, that I have a complete answer to your question.

* * *

Later: the [docs](http://docs.python.org/library/queue.html) for
`Queue.task_done` suggest that `task_done` should be 1:1 with `Queue.get`
calls:

> Queue.task_done()¶
>
> Indicate that a formerly enqueued task is complete. Used by queue consumer
> threads. For each get() used to fetch a task, a subsequent call to
> task_done() tells the queue that the processing on the task is complete.
>
> If a join() is currently blocking, it will resume when all items have been
> processed (meaning that a task_done() call was received for every item that
> had been put() into the queue).
>
> Raises a ValueError if called more times than there were items placed in the
> queue.

Were you getting [uncaught] `ValueError` exceptions? It looks like this might
be so.

