Should database directory and file access be restricted to only the db user?
I'd say this is a good security practice to restrict OS access to your DB files.  First you should definitely restrict write privileges to DB files at the OS level, if not already configured that way.  Don't allow anyone other than the OS user that runs the DB to write to the DB files.  Practically any DB software I can think of restricts this by default.

If you want to additionally restrict read access, this would be a good idea, especially if your database contains sensitive information.  Any unprivileged OS user that shares your OS, if he/she has read-access to your DB files, they basically can get read access to your DB.  They could simply steal copies of your DB files, send them to their own server, and setup the DB system and grant for themselves unlimited access to that clone.  I'd say the majority of DB software does NOT proactively restrict read access at the OS level to the datafiles.  It's up to you, the DBA to restrict this.  If your data is super sensitive, you might want to consider encryption, which is available in most major DBMS.  If the datafiles are encrypted, a would-be information thief would most likely be foiled even if he/she made off with your datafiles.

Don't forget to encrypt your backups, too, since they are just as sensitive as the DB itself.

Generally, you don't need to worry about the impact of cutting off access to other OS tools like backup software.  Most DBMS can't and don't use any OS-based backup tools since DB files are changing.  Rather you use specific tools like Oracle RMAN, SQL Server native backup, Litespeed, mysqldump, etc.  These tools know how to handle a database that is changing and to get a proper backup.  And some other OS tools like antivirus scanners run with root/system authority, so you can't keep them from reading your datafiles by permissions alone (encryption can).

If you're doing this on Unix, it can sometimes be a matter of simply chmod 700 (or perhaps 750) on the directory that contains your DB files.  On Windows you could likely do this via NTFS permissions.