I have asyncio crawler, that visits URLs and collects new URLs from HTML responses. I was inspired that great tool: <https://github.com/aio-libs/aiohttp/blob/master/examples/legacy/crawl.py>

Here is a very simplified piece of workflow, how it works:

```
import asyncio
import aiohttp

class Requester:

    def __init__(self):
        self.sem = asyncio.BoundedSemaphore(1)

    async def fetch(self, url, client):

            async with client.get(url) as response:
                data = (await response.read()).decode('utf-8', 'replace')
                print("URL:", url, " have code:", response.status)
                return response, data

    async def run(self, urls):
        async with aiohttp.ClientSession() as client:
            for url in urls:
                await self.sem.acquire()
                task = asyncio.create_task(self.fetch(url, client))
                task.add_done_callback(lambda t: self.sem.release())

    def http_crawl(self, _urls_list):
        loop = asyncio.get_event_loop()
        crawl_loop = asyncio.ensure_future(self.run(_urls_list))
        loop.run_until_complete(crawl_loop)

r = Requester()
_url_list = ['https://www.google.com','https://images.google.com','https://maps.google.com','https://mail.google.com','https://news.google.com','https://video.google.com','https://books.google.com']
r.http_crawl(_url_list)

```

What I need now is to add some very slow beautifulsoap based function. I need that function do not block main loop and work as background process. For instance, I will handle HTTP responses.

I read python docs about it and found that: <https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor>

I tried to add it to my code, but it does not work as should (I use cpu\_bound only for demo):

```
import asyncio
import aiohttp
import concurrent.futures

def cpu_bound():
    return sum(i * i for i in range(10 ** 7))

class Requester:

    def __init__(self):
        self.sem = asyncio.BoundedSemaphore(1)

    async def fetch(self, url, client):

            async with client.get(url) as response:
                data = (await response.read()).decode('utf-8', 'replace')
                print("URL:", url, " have code:", response.status)
                ####### Blocking operation #######
                loop = asyncio.get_running_loop()
                with concurrent.futures.ProcessPoolExecutor() as pool:
                    result = await loop.run_in_executor(pool, cpu_bound)
                    print('custom process pool', result)
                #################################
                return response, data

    async def run(self, urls):
        async with aiohttp.ClientSession() as client:
            for url in urls:
                await self.sem.acquire()
                task = asyncio.create_task(self.fetch(url, client))
                task.add_done_callback(lambda t: self.sem.release())

    def http_crawl(self, _urls_list):
        loop = asyncio.get_event_loop()
        crawl_loop = asyncio.ensure_future(self.run(_urls_list))
        loop.run_until_complete(crawl_loop)

r = Requester()
_url_list = ['https://www.google.com','https://images.google.com','https://maps.google.com','https://mail.google.com','https://news.google.com','https://video.google.com','https://books.google.com']
r.http_crawl(_url_list)

```

For now, it doesn't work as expected, it blocks HTTP requests every time:

```
URL: https://www.google.com  have code: 200
custom process pool 333333283333335000000
URL: https://images.google.com  have code: 200
custom process pool 333333283333335000000
URL: https://maps.google.com  have code: 200
custom process pool 333333283333335000000
URL: https://mail.google.com  have code: 200
custom process pool 333333283333335000000
URL: https://news.google.com  have code: 200
custom process pool 333333283333335000000
URL: https://video.google.com  have code: 200
custom process pool 333333283333335000000

```

How to correctly put the task in the background inside the main asyncio process?

Are there best practices on how to do that in a simple way, or I should use Redis for task planning?