title: urllib2 and threading
body: <p>Iam trying to follow the multithreading example given in:
  <a href="https://stackoverflow.com/questions/3472515/python-urllib2-open-is-slow-need-a-better-way-to-read-several-urls">Python urllib2.urlopen() is slow, need a better way to read several urls</a> but I seem to get a "thread error" and I am not sure what this really means.</p>
  
  <pre><code>urlList=[list of urls to be fetched]*100
  def read_url(url, queue):
   my_data=[]
   try:
      data = urllib2.urlopen(url,None,15).read()
      print('Fetched %s from %s' % (len(data), url))
      my_data.append(data)
      queue.put(data)
  except HTTPError, e:
      data = urllib2.urlopen(url).read()
      print('Fetched %s from %s' % (len(data), url))
      my_data.append(data)
      queue.put(data)
  
  def fetch_parallel():
      result = Queue.Queue()
      threads = [threading.Thread(target=read_url, args = (url,result)) for url in urlList]
      for t in threads:
        t.start()
      for t in threads:
        t.join()
      return result
  
  res=[]  
  res=fetch_parallel()
  reslist = []
  while not res.empty: reslist.append(res.get())
  print (reslist)
  </code></pre>
  
  <p>I get the following first error:</p>
  
  <pre><code>Exception in thread Thread-1:
  Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
  self.__target(*self.__args, **self.__kwargs)
  File "demo.py", line 76, in read_url
  print('Fetched %s from %s' % (len(data), url))
  TypeError: object of type 'instancemethod' has no len()
  </code></pre>
  
  <p>On the other hand, I see that sometimes, it does seem to fetch data, but then I get the following second error:</p>
  
  <pre><code>Traceback (most recent call last):
  File "demo.py", line 89, in &lt;module&gt;
  print str(res[0])
  AttributeError: Queue instance has no attribute '__getitem__'
  </code></pre>
  
  <p>When it fetches data, why is the result not showing up in res[]? Thanks for your time.</p>
  
  <p><em>Update</em> After changing read to read() in the read_url() function, although the situation has improved (I now get many page fetches), but still got the error:</p>
  
  <pre><code>Exception in thread Thread-86:
  Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
  self.__target(*self.__args, **self.__kwargs)
  File "demo.py", line 75, in read_url
  data = urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
  return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 397, in open
  response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 510, in http_response
  'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 429, in error
  result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
  result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 605, in http_error_302
  return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 397, in open
  response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 510, in http_response
  'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 435, in error
  return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
  result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 518, in http_error_default
  raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  HTTPError: HTTP Error 502: Bad Gateway
  </code></pre>