Should I use the Infobright database or is there a better aggregation solution? We are a MySQL front-end shop and have a system currently logging event data in a highly structured format. Think apache traffic log.

We need to be able to aggregate counts of this data (cube) vis a vi ad-hoc queries. We are currently sending the data to CouchDB. This is very fast, once a view has mapped completely, but our database size is only 31GB and CouchDB is taking FOREVER to map a new view ( 2+hours ). Retrieving JSON formatted data has worked well for us. But we dread creating new view documents. I'm thinking all Map-Reduce systems may have this problem.

We are evaluating Infobright as they claim to provide superior aggregation performance for ad-hoc queries and are a MySQL fork. 

We have evaluated Mondrian and it's not going to work for us.

MySQL/InnoDB is too slow.

We will be around 500GB of log data by the end of the year.

Is Infobright the right solution or should we be also evaluating something else?
Almost always (in my experience), the following is the solution to performance with large, log-like, data and semi ad hoc queries.

Characteristics of the data and the application:


continually arriving data
no updates to 'old' data
optionally purging old data (if so use PARTITION BY RANGE(TO_DAYS()))
queries tend to have a date range in the WHERE clause


Solution:  Build and maintain "summary tables".


pick a timeframe (usually day or hour)
after midnight (or top of hour), summarize yesterday's data from the raw (Fact) table into the Summary table.
PK of the Summary table usually includes some dimension(s) and the rounded-down date/hour.
rest of fields include aggregates like COUNT(), SUM(), etc, but not AVG().
AVG is calculated as SUM(sum_foo)/SUM(ct)
"reports" hit a summary table, not the fact table.
1-10 summary tables usually suffice for a given application
Querying the summary table usually gives you 10x performance improvement; I have seen 1000x in rare cases.


I say "semi ad hoc" because you will find out that users don't really query on everything.  Even if they do, you can have summary table(s) that at least help.

I mentioned the straight-forward way of completely summarizing "yesterday".  There are many other flavors.  One is "as needed", with the aid of INSERT ... ON DUPLICATE KEY UPDATE.