Recommendation for RDBMS for large table
The size of the tables really doesn't matter all that much. It's all about how well indexed the tables are and how many people are using the database at once.

In my case I've got clients with billions of rows in a single table where the table size is about 1 TB and another 2 billion row table which has blobs.  These tables crank without performance issues at all.

In this case the server has 1TB of RAM, 80 cores, and is running Microsoft SQL Server 2008 R2.  Granted not all that helpful if you want a *nix OS, but it gives you some ideas that most database platforms can handle just about everything.