Pattern(s) to scale node.js based ETL
I ended up following this path
Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to &quot;jobify&quot; the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)
A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container
Eventually, when I need to scale horizontally, the &quot;dispatch&quot; function will morph to running a job whose sole purpose is to post an event  (eg: Azure service bus).  I will have an orchestrator running various containers who read from the queue and process accordingly.