How can I use the dvdbackup tool in Mac OS X?
From the man page:

dvdbackup -M -i/dev/dvd -o/my/dvd/backup/dir/ This action creates a valid DVD-Video structure that can be burned to a DVD-/+R(W) with help of mkisofs version 1.11a27 or later  http://pwet.fr/man/linux/commandes/dvdbackup

Honestly I have never used this tool on Mac OS X, if you want to do some dvd ripping and this keeps frustrating you, Handbrake is pretty good, or you can even use the built in Disk Utility to do a image backup of said DVD.