Non-Blocking MySQL Backup
Indeed the bottleneck was IO. Since I was using tar -czf to compress the files and the CPU was so performant, the drive simply couldn't handle all the data coming so fast (although we use mirrored RAID volumes).

The mysqldump itself is no problem at all, but tar was slowing down the whole system massively.

I'm now using pv to shape the pipe stream:

tar -czf - ./ | pv -q -L 10m &gt; output.tgz



-q disables output of pv.
-L 10m limits the write operation to 10 MB per second. Just test which value you should use on your server by watching the gzip process via top. I wanted it to have around 50% CPU usage, so the IO impact is lowered to 50% as well.
CPU impact is lowered implicitly, because pv slows down everything (except of the dump itself, it is done before compression).


Many thanks to @Michael - sqlbot and others for pushing me into the right direction.