Why is my Python web scraper taking so long? I am trying to create a program to download a long list of websites using Python 3.7. By using multiprocessing for each request, my code runs much faster than when it's run synchronously. 

I've added a timer to my script so I can know the execution time for the script. The one I wrote below takes on average approximately 3.5 seconds to run. However, when I use this script for my long list of URL's, it takes approximately 8 minutes to execute.

I am wondering: is it taking this long because my internet is slow? Or is my script just bad? 

My internet speed is pretty low (3 mb/s download, 1 mb/s upload). If anyone has faster internet, could you post your execution time for this script along with your internet speed? This way, I can see if the bottleneck is internet speed or not.

If the bottleneck isn't internet speed, but simply that my script is bad, can anyone offer any suggestions for improving the script so that the execution time is better? Thanks coders!

from urllib.request import Request, urlopen
from multiprocessing.dummy import Pool as ThreadPool
import datetime
import time
import requests

start_time = time.time()

urls = [
  'https://en.wikipedia.org/wiki/Python_(programming_language)',
  'https://www.python.org/',
  'https://stackoverflow.com/questions/tagged/python',
  'https://github.com/python',
  'https://realpython.com/python-beginner-tips/',
  'https://pythonprogramming.net/introduction-to-python-programming/',

  ]

mylist = []

def my_function(url_to_parse):
  req = Request(url_to_parse, headers={'User-Agent': 'Mozilla/5.0'})
  resp = urlopen(req).read()
  mylist.append(resp)

#------------------------------------------------------------------------------------------

pool = ThreadPool(8)
results = pool.map(my_function, urls)
pool.close()
pool.join()

print("--- %s seconds --- REQUESTS" % (time.time() - start_time))
```
serializing
You are serializing all HTML text and sending it back to the parent.
That can be time consuming.
Consider having the child (my_function()) write HTML text to a file in /tmp,
and pass back the temporary filename to the parent.
Better, also pass back some timestamps: (start, end, filename)
stragglers
If you can predict that a page will be &quot;big&quot; or a site will be &quot;slow&quot;,
sort those to the front of your list of 500 URLs.
That way you won't have a handful of sluggish stragglers
impacting your overall time,
while most cores are idle and waiting for them to complete.