How to force higher values of open_files_limit or table_open_cache?
From the looks of your numbers, you must have hit some bug or pre-GA behavior. Why?
According to the MySQL Documentation, the default for table_open_cache is -1 for GA releases of MySQL 5.6. This tells mysqld pick a fair starting value give current OS conditions. What makes that value weird is the fact that 400 is the default for MySQL 5.6.7. The first GA release is 5.6.8. It may be a coincidence, but my guess would be that you set table_open_cache a little too high. Evidently, 5000 is a tolerable value for your DB Server.
SUGGESTION
Try experimenting with table_open_cache_instances setting it to 2 or 4. Then restart mysql and see.