Access partial results of a Celery task
Question: I'm not a Python expert, however I'm trying to develop some long-running
Celery-based tasks which I'm able to access their partial results instead of
waiting for the tasks to finish.

As you can see in the code below, given a multiplier, an initial and final
range, the _worker_ creates a list of size _final_range - initial_range + 1_.

    
    
    from celery import Celery
    app = Celery('trackers', backend='amqp', broker='amqp://')
    
    @app.task
    def worker(value, initial_range, final_range):
        if initial_range < final_range
            list_values = []
            for index in range(initial_frame, final_frame + 1):
                list_values.append(value * index)
            return list_values
        else
            return None
    

So, instead of waiting for all _four workers_ to finish, I would like to
access the to-be-returned values (_list_values_) before they are actually
returned.

    
    
    from trackers import worker
    
    res_1 = worker.delay(3, 10, 10000000)
    res_2 = worker.delay(5, 01, 20000000)
    res_3 = worker.delay(7, 20, 50000000)
    res_4 = worker.delay(9, 55, 99999999)
    

First of all, is it possible? If so, what sort of changes do I have to perform
to make it work?


Answer: You absolutely need to use an external storage such as SQL or Redis/Memcached
because different tasks can be executed on different servers in common case.

So in your example you should store list_values in some DB and update it
during the loop.

