Is there a Terminal command that will force open the DVD tray?
First of all, try the "nice" way using the drive utility

drutil tray eject


If this doesn´t work (probably because the system can´t unmount the volume), the following command will forcefully unmount the Volume with name "untitled" (make sure you´re using the right mountpoint!!):

hdiutil detach -force /Volumes/untitled