What happened to Drupal's performance?
D8 requires more resources and yes, you should have hosting with opcache enabled, which is included by default in PHP 5.5.

Drupal 8 has so many files because it includes symfony and other projects and everything is a class.

FTP is a very bad idea for D8 and web in general in 2015. Look for a better way to do web hosting.

That said, it should not take that long. I suggest you open a bug report  and provide as many details as you can about your specific PHP version, hoster, the errors that you had, ..