Is Drupal cache as set in the core enough for simple sites? Let's imagine a site which is almost completely static. There are news which are rarely updated and from time to time some new content in the form of more news. If there are only anonymous users apart from the admin... would the cache retrieval be so fast as simple HTML pages? I ask this because I have tested the config above with cache on and it goes very slow in comparison with HTML saved counterparts and I am not sure if it is a misconfiguration.
No it would be unlikely to be as fast as plain HTML, because the default caching is still using the database and some PHP.

However you can improve the speed significantly with a module or two. It is highly dependent on the speed of your servers - if you have a fast database server but a slow filesystem you might be better using the database although that is an unlikely situation. At minimum I recommend installing Advagg - correctly configured that will reduce load times for a lot of sites by around 25%. 

Depending on the site, Boost can also vastly improve performance although I have not found it as universally applicable as Advagg. Also depending on your setup an external cache like Varnish may be better although it requires careful configuration and some server admin knowhow.