Scraping with asyncio/aiohttp and retrying the request on error
with back-connect proxies, is you do not always get a proxy that works.


There are several possibilities here. The origin webserver could be responsive, slow (slower than your timeout), or dead, and the proxy could be responsive, slow, or dead. Call those "health" details. It doesn't look like you are tracking health details of either one, so you don't adapt to such details. For faster processing, use adaptive timeouts and don't re-submit requests to dead hosts (don't contact known-dead proxies, and don't contact known-dead webservers).

EDIT: An example of an adaptive timeout would be triple the observed median response time of a host, which attempts to capture a large fraction of the pdf (probability distribution function) of its observed response times, conditioned on actually getting a response.

class Proxy:
    def __init__(self, headers, proxy):


I'm not crazy about that last identifier. Call it proxy_url, please. It clearly isn't an instance of the Proxy class.