Are there any MySQL Benchmarking tools? I've heard a long time ago that there is this tool that helps you tweek mysql settings for better performance but i can't seam to find it. I am aware that I can use ab for apache to simulate high traffic and it will generate me a slow log. However, if it crashes (already happened and it was in production mode), I don't know why it crashed and if it can be tweeked from the config.
Benchmarking or Tuning tool?

IMHO, there's no tool that will be specific to the latter unless you have a super generic usage. You need to identify your usage pattern and tune your database hosts to accommodate. If you're write-heavy, you will have a different configuration than a read-heavy scenario. Bottom line, your tuning follows your applications usage.

For benchmarking, I use Sysbench. Here's an example from my blog.

Added:
Here's my beef with config tools: With the changes in versions 4.x vs 5.x vs 5.5.x there are a lot of tweaks that don't jive between them. So you really need a savvy DBA to evaluate what's going on for each host. Load, storage, traffic, application specific requirements, there's a lot that can go into an optimal configuration. A tool may get you part of the way, but may leave out something or include something that may cause a failure. From buffers to flushing to plugins to threading, a config tool may provide you with false confidence that you are implementing the correct config.