How to store massive quantities of machine generated data? I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances.  Our project will need to keep up to 10 TB of data (maybe more in the future) and serve queries fairly quickly.  I have been reading about columnular rdms' as well as nosql options, which all seem promising.
There's a few systems specifically tailored to what you're requesting. I don't know exactly what kind of logs you're dealing with (application specific logs, system logs, web-server logs, a combination, etc.), so you may need to research a bit further.

Please pardon the lack of inline links - I currently don't have the reputation to post more than 2 links (you can find the writeup with inline links here).

Commercially there's:


Splunk
Sumo Logic
Loggly
Papertrail


On the open-source side there's:


Graylog2 (meant as an open-source alternative to Splunk)
OpenTSDB
Roll your own with Apache Flume and elasticsearch
Logstash


Personally, I'd look into Graylog2, Logstash, and (depending upon the requirements) OpenTSDB.

One thing to also look at is the underlying database system and how well it works with the hardware that you've already bought (i.e. MongoDB may not work very well with the hardware you have and the size of data that you have, however Hadoop/Hbase or a SQL server may work out better).