title: Migrating multi-threaded program to Docker containers
body: <p>I have an old algorithm that executes two steps, say A and B. A deals with connecting to an external service and retrieve data points. A needs to be executed n times, and all the data so gathered is passed as input to B. To help scale the A part, it was implemented using multi-threading where 10 threads are spawned and each connects to n/10 endpoints. Once all threads complete execution, the complete dataset is provided as input to B.</p>
  
  <p>We are planning to create a Docker image of this algo. While we do that I would like to explore if we can do away with the multi threading and instead deploy multiple containers. This gives us better scalability as n is a variable and sometimes is very small or very large. </p>
  
  <p>I could use Kubernetes to orchestrate these containers. However, I see two challenges:
  1. How do I gather back all the data points into my core algo?
  2. How do I know that all containers have finished processing, and that I could move to step B?</p>
  
  <p>Any pointers or help is appreciated.</p>