Why is my Python web scraper taking so long?
serializing
You are serializing all HTML text and sending it back to the parent.
That can be time consuming.
Consider having the child (my_function()) write HTML text to a file in /tmp,
and pass back the temporary filename to the parent.
Better, also pass back some timestamps: (start, end, filename)
stragglers
If you can predict that a page will be &quot;big&quot; or a site will be &quot;slow&quot;,
sort those to the front of your list of 500 URLs.
That way you won't have a handful of sluggish stragglers
impacting your overall time,
while most cores are idle and waiting for them to complete.