Need setup recommendation for parallel processing many contracts through many scenarios
From a simple conceptual perspective:

Write yourself a function that takes a contract and a scenario as parameters and performs the desired calculation
Use Python's multiprocessing to set up a worker pool
Create a Queue (from multiprocessing package) that is to be shared across workers
Fill the queue with all combinations (might be a good idea to use fixed indices and only push a tuple of the contract/scenario indices (C, S) to the queue to reduce required space
Map your function to the worker pool given the queue

There are more elaborate ways to do this (including amqp/celery/...) depending whether you want to distribute tasks across multiple machines or just to all of your locally available cores. This simple concept should contain all required keywords to build your first local multiprocessing on your own!
I need a recommendation from gurus out there on how to go about setting up a modeling application. I have thousands of scenarios to run on thousands for contracts for cash flow projections. Assuming I have 1000 scenarios and 1000 contracts I would need to run 1,000,000 projections (1000x1000). I'd like to do this in parallel using dask, ray or some other method. My data are in dataframes but I'm open to better suggestions. I can create 2 loops (scenario,contract) for each run but this would be sequential.
Scenario1 w Contract1
Scenario1 w Contract2
Scenario1 w Contract3
.
.
.
Scenario1000 w Contract1000
I'd like to distribute compute to multiple processor and multiple servers.
I'll save my question on the inner loop projections where I have to run 100 scenario projections at each time step of the 1,000,000 runs for next time.
Any suggestion to point me in the right direction would help.