title: How to do this (check requests status codes) in Python concurrently or in parallel?
body: <p>Here's what I'm doing:</p>
  
  <ul>
  <li>Get words from a text file - every word is on a separate line.</li>
  <li>Add <code>http://www.</code> and <code>.com</code> to words to create a url.</li>
  <li>Get the URL with requests.</li>
  <li>Find out if it's a free domain or not (based on status codes and
  error in connection/other error).</li>
  <li>Add free domains to a text file.</li>
  <li>Time it all.</li>
  </ul>
  
  <p>I've kind of made it work so far but it's very slow. The text file has 350 000 words. How would I go about doing this concurrently or in parallel? Also which would be a better choice for this task?</p>
  
  <p>Here's my code:</p>
  
  <pre><code>import requests, time
  
  start = time.time()
  
  with open('words1.txt','r') as f:
      words = []
      for item in f:
          words.append(item.strip())
  
  for w in words:
      url = 'http://www.'+w+'.com'
      try:
          header = {'User-Agent': 'Mozilla/5.0'}
          r = requests.get(url, headers=header)
          codes = [200,201,202,203,204,205,206,300,301,302,303,307,308,400,401,402,403,404,405,406,500,501,502,503]
          if r.status_code in codes:
              print(url,': Known Status Code &gt; Unavailable')
          else:
              print(url,': Unknown Status Code &gt; Probably Free')
              with open('available.txt','a') as myfile:
                  myfile.write(url+'\n')
      except requests.exceptions.ConnectionError:
          print(url,' : Connection Error &gt; Probably Free')
          with open('available.txt','a') as myfile:
              myfile.write(url+'\n')
      except requests.exceptions.HTTPError:
          print('http error')
      except requests.exceptions.Timeout:
          print('timeout error')
      except requests.exceptions.TooManyRedirects:
          print('too many redirects')
  
  end = time.time()
  print('\n')
  print(end-start, 'seconds')
  print((end-start)/60,'minutes')
  print(((end-start)/60)/60,'hours')
  </code></pre>
  
  <p>Thanks!</p>
  
  <p><strong>EDIT:</strong> I got it to work. Thanks for the help <strong>Kendas</strong> and <strong>DeepSpace</strong>!
  Here's a quick test:</p>
  
  <p>100 words - 22 sec</p>
  
  <p>1000 words - 285 sec</p>
  
  <p>Not too fast but way faster than my first try.</p>
  
  <p>Seems like gevent + socket is the way to go.</p>
  
  <p>Please let me know if you have any tips on making this better/faster.</p>
  
  <p>Here's the code:</p>
  
  <pre><code>import gevent,time
  from gevent import socket
  
  start = time.time()
  
  words = []
  with open('words1000.txt','r') as f:
      for item in f:
          words.append(item.strip())
  
  urls = ['www.{}.com'.format(w) for w in words]
  
  jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
  
  gevent.joinall(jobs)
  
  values = {url:job.value for (url,job) in zip(urls,jobs)}
  freeDomains = []
  for (v,job,url) in zip(values,jobs,urls):
      if job.value == None:
          freeDomains.append(url)
          with open('availableds.txt','a') as myFile:
              myFile.write(url+'\n')
  
  print(freeDomains)
  end = time.time()
  print(end-start,'seconds')
  print((end-start)/60,'minutes')
  print((end-start)/3600,'hours')
  </code></pre>