best db and file system for high volume, high qps
Virtually any database server can meet your needs, whether it is MySQL, Microsoft SQL Server, Oracle, etc. Performance with all the rdbms' vary a lot based on the following criteria


I/O Performance of the disk (get good, fast, RAID'd disks over high throughput connections)
Performance and Amount of RAM (good and fast RAM, and lots of it)
File System Overhead (use a modern fs designed for databases. Ext4 works fine, but there's OCFS2 that is Oracle's file system, built for databases, or even Oracle's RAW storage where there isn't even a file system in the OS-sense... the database manages the disk entirely.)
Properly Indexed table structure (quick access to data depends upon your data being indexed correctly, where "correct" means most efficient for all your needs.)
Properly Optimized Queries... usually goes hand-in-hand with the correct indexes; the database can try to figure out the best optimization for your query, but it can only do so with correct indexes and statistics. Better yet, fine tune your queries so that the database doesn't have to do a lot of optimization on its own, and you'll be faster and happier for it.
I/O Performance of the network - if you are transferring a lot of data in or out, network performance will matter (otherwise the database has to wait when sending data across the wire).


A few things that really don't matter very much (as far as performance):


Your operating system. Windows, Mac OSX, Linux, and anything else that's modern will do just fine as the operating system layer. Most of your work is done in the database anyway, with occasional tasks needing done in the OS. Pick an OS that will simplify your life (and not complicate it), but modern databases are performant on most OSes.
Your database engine. Yes, there are critical differences between MySQL and Oracle and MS SQL, etc. Does it really matter? If there is a particular feature you must have, then by all means, go for it, but ultimately, all will be sufficient with the correctly configured hardware and tables.


Really, pick a database, an OS, a file system, and build your app. With good hardware and good table design (with indexes and optimization), things should go swimmingly.

All that said, here's what I'd pick (only because I'm most familiar with this type of configuration):


RHEL / OEL / Ubuntu (OS)
Oracle Standard One (or Standard, or EE, depending on your needs) (DB)
Ext4 Journaled File System (FS)
2 2xCore Processors (>2.5GHz)
8 GiB Memory (something around 1066 or faster MHz)
1 TiB HDD (SATA, with >=7200rpms)
At least gigabit ethernet


Swap out the OS, File system, and DB as desired. Your machine should scream and be just fine for a good number of databases, and is probably a bit overkill. Best thing, though, is that you can always add memory and disk space and upgrade your processors if you need more. If necessary, move to a new box, or add boxes (if your database supports it). (Even better: run your environment as a Virtual Machine/Cluster. Then moving to another physical machine is child's play.)