How do I properly perform a MySQL bake-off? I want to performance test (aka bake-off) MySQL server rpm against some other forks such as Percona server, MariaDB, and possibly some others. I'm hoping that by asking this question I can better understand the methodology behind setting up a proper performance test. I planned on using sysbench to run my actual test, but I'm open to anything.


What steps should I be taking to
ensure that the test results in a
apples-to-apples comparison and that
only the RDBMS is the variant? 
Where do I get started?  
How do I evaluate results?  
What advice can you give me?
Here is the first thing I thought about


Setup 3 DB servers (identical HW/OS configs) each installed with


MySQL
Percona
MariaDB

Get a Fourth Server installed with MONyog (eval version lasts 30 days)
Register the 3 DB servers in MONyog
Use the Performance Metrics Charts in MONyog or Set Up Your Own Charts in MONyog
Use SysBench  against all 3 DB Servers at your discretion


This is just a basic outline. Believe me, you can use Monyog right out of the box. Check out my testimonial at MonYog's Website (Look for my name on the page). I eventaully convinced my company to purchase the Utlimate Version outright after my testimonial, which they watched with me as it happened.

Caveat : Do no get Monyog until you have the 3 DB servers and Sysbench Ready to get the best 30-day usage of MonYog.

UPDATE 2011-06-20 13:18 EDT

If you keep the InnoDB Buffer Pool Small (default for innodb_buffer_pool_size is 8M) and keep the binary logs disabled (do no include log-bin in my.cnf), then these metrics should be measured


Innodb_buffer_pool_pages_flushed
Innodb_buffer_pool_write_requests
Innodb_row_lock_time_avg
Innodb_os_log_fsyncs
Queries, Questions
SELECTs ((Com_select + Qcache_hits) / Questions)
INSERTs ((Com_insert + Com_replace) / Questions)
UPDATEs (Com_update / Questions)
DELETEs (Com_delete / Questions)


Many of these are default graphs in MONyog's Monitors/Advisors Section, and more graphs can be created to check other desired MySQL metrics. MONyog also lets you export the graphs as CSV data.

UPDATE 2011-10-12 14:17 EDT

Percona performed a bake-off among multiple releases of MySQL


MySQL 4.1
MySQL 5.0
MySQL 5.1 (with built-in InnoDB)
MySQL 5.1 with InnoDB-plugin
MySQL 5.5
MySQL 5.6


All tests were performed with MySQL unconfigured. The results?


MySQL 4.1 performs the best single-threaded
MySQL 5.1 with InnoDB plug-in scales on multiple cores better than 5.1 InnoDB built, 5.5 and 5.6


What do I get out of this? Your must configure MySQL 5.5/5.6 to get multicore enhancements engaged.