title: python parallel send 1000+ url requests and get content info
body: <p>I have a function that:
  get the title and content of a new article and append into a list.</p>
  
  <p>I want to:
  Call 1000+ urls and run that function</p>
  
  <p>Goal:
  with one run, I'll get the 1000+ title and content in a list without having to loop through each url and call in sequence.</p>
  
  <p>The code I have so far:</p>
  
  <p>setup</p>
  
  <pre><code>import requests
  from newspaper import Article
  import threading
  </code></pre>
  
  <p>1)This is the "do work" function</p>
  
  <pre><code>def get_url_info(url):
      try:
          r = requests.head(url) 
          if r.status_code &lt; 400: # if loads
              article = Article(url)
              article.download()
              article.parse()
              if detect(article.title) == 'en': #English only
                  if len(article.text)&lt;50: #filter out permission request
                      title = (article.title.encode('ascii', errors='ignore')) 
                      text = (article.text.encode('ascii', errors='ignore'))
                      test_url= url
      except Exception as e:    
          issue = url #storing issue urls
          print(e, url) 
  
  return title, text, test_url
  </code></pre>
  
  <p>2)This is the actual append into a list function:</p>
  
  <pre><code>def get_text_list():
      text_list = [] #article content list
      test_urls = [] #urls taht works
      title_list = [] #article titles
      url_list = get_tier_3()[:8000]  #get first 8000 english texts for testing
      threads = [threading.Thread(target=fetch_url, args=(url,)) for url in url_list]
      for thread in threads:
      #originally this was for url in url_list
          thread.start()
          """
          title, text, test_url = call do work here
          title_list.append(title)
          text_list.append(text)
          test_urls.append(test_url)
          """
          print (i) #counts number of urls from DB processed
  
  return text_list, test_urls, title_list
  </code></pre>
  
  <p>Question:
  I don't know how to proceed after setting up the threads and get information from each thread</p>