title: Streaming text logfiles into RabbitMQ, then reconstructing at other end?
body: <p><strong>Requirements</strong></p>
  
  <p>We have several servers (20-50) - Solaris 10 and Linux (SLES) - running a mix of different applications, each generating a bunch of log events into textfiles. We need to capture these to a separate monitoring box, where we can do analysis/reporting/alerts.</p>
  
  <p><strong>Current Approach</strong></p>
  
  <p>Currently, we use SSH with a remote "tail -f" to stream the logfiles from the servers onto the monitoring box. However, this is somewhat brittle.</p>
  
  <p><strong>New Approach</strong></p>
  
  <p>I'd like to replace this with  RabbitMQ. The servers would publish their log events into this, and each monitoring script/app could then subscribe to the appropriate queue.</p>
  
  <p>Ideally, we'd like the applications themselves to dump events directly into the RabbitMQ queue.</p>
  
  <p>However, assuming that's not an option in the short term (we may not have source for all the apps), we need a way to basically "tail -f" the logfiles from disk. I'm most comfortable in Python, so I was looking at a Pythonic way of doing that - the consensus seems to be to just use a loop with readline() and sleep() to emulate "tail -f".</p>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li><p>Is there an easier way of "tail -f" a whole bunch of textfiles directly onto a RabbitMQ stream? Something inbuilt, or an extension we could leverage on? Any other tips/advice here?</p></li>
  <li><p>If we do write a Python wrapper to capture all the logfiles and publish them - I'd ideally like a single Python script to concurrently handle all the logfiles, rather than manually spinning up a separate instance for each logfile. How should we tackle this? Are there considerations in terms of performance, CPU usage, throughput, concurrency etc.?</p></li>
  <li>We need to subscribe to the queues, and then possibly dump the events back to disk and reconstruct the original logfiles. Any tips/advice on this? And we'd also like a single Python script we could startup to handle reconstructing all of the logfiles - rather than 50 separate instances of the same script - is that easily achievable?</li>
  </ol>
  
  <p>Cheers,
  Victor</p>
  
  <p>PS: We did have a look at Facebook's Scribe, as well as Flume, and both seem a little heavyweight for our needs.</p>