Let me describe a simplified version of the problem in relatively domain neutral language:let's say I have a supermarket with 10s of thousands of different products on the shelvesI can get deliveries at any time of the day and customers can also purchase items throughout the day, impacting inventory. Volume wise it should in the hundreds / low thousands per minuteprices can fluctuate too, at a similar volume to inventoryitems can belong to categories, think aisle 6 or dairyusers need to have a real time view of the value of the inventory, but they are interested in specific categories, not individual items.by real time i mean the screen will need to have updates pushed to it by the systems, so updates from the inventory/prices will need to flow through. latency is not critical, but if you open the screen you don't want to have to recalculate everythingSo in my mind I see a lot of caching here:- we need to cache the inputs: inventory volume for each product + prices- cache the value for each product (price x volume for each product)- cache the value for each category (sum of all products in category price x volume)The main challenge thus is to keep the cache updated with minimal recalculations whilst having these updates flow through from the upstream systems to the front end.I've thought of a few ways to do this:- DIY: manage own cache in HashMap type of objects, using observer and/or pub/sub pattern to update the categories & the screen (the screens can all display different categories so will not be interested in all updates)- In memory data grid like hazelcast / gemfire: as there's a lot of caching, and these can have pub/sub logic in them it seems a relatively good choice- Use of actors (akka) to parallelize this Things to bear in mind:- volume of data is not insignificant but not huge: as mentioned maybe up to 100k product types, a few hundreds of categories, and number of users less than 100.- number of updates is also not that high: a few hundred/thousand /minuteSo I wonder if IMDG solutions or things like spark are not overkill.Akka seems interesting but im not sure its the right fit.Anyone had to work with similar problem or has interesting suggestion?Technology wise I'm looking for some java based stuff mostly; screen will be html5/js.
IMDG vs Actors vs Observer for ticking screen?
design;caching;system