My requirement is to build real-time aggregation pipeline using kafka stream with large volume of data. Based on the estimate, the possible unique keys will be ~ 3 to 4 billion and total message size ~5TB.

The high level architecture is, read from a kafka topic, aggregate it based on certain key columns and publish aggregated results into KTable (kafka compact topic). KTable is used to read the previous state and update with new aggregated results.

*Is KTable scalable with billions of unique keys?*