High Speed Python Web Scraper Optimised
If you want absolute performance


You could avoid printing anything at all. Since that can be slow in some cases. If you absolutly need to know at least what happened you could try flushing at the end of the process.
I know with python is delicated, but you could try visiting multiple pages at the same time? Threads or the equivalent in python