how much to keep table_open cache and what settings is available in mysql to release unwanted memory cache and table cache?
Each unit in table_open_cache is one table (not one byte).  Since you have a huge amount of RAM, the 'right' answer is probably to make it a little more than the number of tables in all the databases.

It is limited to open_files_limit, which, in turn, is limited by the OS.  Some OSs have an artificially low 1024, but it can be changed.

Suggest table_open_cache_instances = 16 if your version includes that relatively new feature.

If you would like a lengthier check, please provide SHOW VARIABLES; and SHOW GLOBAL STATUS; after mysqld has been running for more than a day, and preferably with a realistic load.

A common mistake is setting query_cache_size bigger than about 50M.  The code in the QC was not designed to scale.  Anyway, most production servers run better with the QC turned off.

The buffer_pool is the main cache in RAM.  There are a number of lesser ones.  Is this what you mean by "memory cache"?

Will this server be running anything other than MySQL?  If so, you should subtract that from the 672G when posing the question.