Very large MySQL backup At our company we have a very large MySQL server with almost 1.000 databases with a total size of 500 GB. Every customer has is own database. And each database has about 120-125 tables! 

In the past we made backups with mysqldump. The server is very slow while creating backups and it almost takes 5 hours to take backups. The databases have for 99% Innodb tables. 

I understand taking backups of a lot of data takes a lot of time. But what is the best solution to take backups? The problem is that our customers are working 24/7 with our webapplications. The system is very slow while taking backups.
The conventional approach is to set up multiple replicated slaves.

You then backup from these slaves periodically (mysqldump is probably fine for this part still).

The master need not be blocked while the slaves are being backed-up; it will simply continue to replicate once those backups are done. Stagger the backups so you always have an up-to-date slave.

Replication is not CPU-expensive (unless you have a crazy level of database activity), and the slaves' backup process won't impact the master at all.

And everything should have redundant storage.



Anecdote: At a previous job we had a 600GB database that was backed up in the same way that you're doing it, but over a 10Mbit pipe to the target, and it would block the entire database for 11 hours every night! Eesh. Never did get replication in place there (lol) but upgrading the networking to 100Mbit meant it could be done in 2 hours or so instead, which was "fine". Of course, with the database locked during the backup (which it should be!), and thus the actual service unavailable for that duration, a performance hit from the backup process was not even observable from the outside. In your case you have multiple distinct databases so the situation isn't quite the same &mdash; you don't have to lock everything for the whole time.