Consumer-producer pattern with pyarrow
Solution with lists:
Producer puts data into a list with LPUSH
Consumer takes data from this list with RPOP or BRPOP (blocking).
Limitations: only one consumer reads the message. If you have 2, only one of them will see the message.
Speed: for one pair of consumer-producer it will have the same speed. The more consumers (for this or other lists), the faster it will be than pub/sub.
What is the best way to implement a multi process based consumer producer pattern with pyarrow as a fast memory store for pandas dataframes?
Currently I am using redis pub sub but I think there might be a more efficient (faster) solution? Could you provide an example?