Distributing Real-Time Market Data Using ZeroMQ / NFS?
I'm pretty sure sending with ZeroMQ will be substantially quicker than saving and loading files.
There are other ways to send information over the network, such as raw sockets (lower level), AMQP implementations like RabbitMQ (more structured/complicated), HTTP requests/replies, and so on. ZeroMQ is a pretty good option, but it probably depends on your situation.
You could also look at frameworks for distributed computing, such as that in IPython.
Suppose that you have a machine that gets fed with real-time stock prices from the exchange. These prices need to be transferred to 50 other machines in your network in the fastest possible way, so that each of them can run its own processing on the data.
What would be the best / fastest way to send the data over to the other 50 machines?
I am looking for a solution that would work on linux, using python as the programming language. Some ideas that I had are:
(1) Send it to the other machines using python's ZeroMQ module
(2) Save the data to a shared folder and have the 50 machines read it using NFS
Any other ideas?