How does one convert a dmg inside a DVD to individual videos or watch the dmg DVD file in an iphone?
DMG is a disk image file format.

Disk image files behave like a physical disk. In so doing, the operating system interprets them in the same way as if you had just inserted a CD-ROM, USB drive, etc into the computer. 

So, to play DVDs you've converted to a disk image such as .DMG, you'll need to do two things:


Expand the disk image so that the 'DVD' becomes available
Open the DVD with some sort of 'DVD player' software (because what the operating system sees is a DVD)


In other words, an iOS devise like an iPhone is not going to be able to play a DVD. So, backing up your DVDs to a disk image file is a great way of preserving/backing up your DVDs for future access, but it's not a way to actually play your DVDs without the appropriate operating system and software. You could certainly expand the .dmg disk images and then use a DVD Player app on a Mac to view the DVD, or burn them to a DVD to make them available to a physical DVD player.

If your intention is to be able to play them on an iPhone or iPad (or any other smartphone/tablet) what you'll need to do is rip the DVDs to a video file format (e.g. .mp4, .mov, etc) and then transfer these files to your device. Depending on the file format you use, you may also need to install 3rd party app to be able to play them.

Some further reading:


For more info on DMG files you may want to read my answer here.
Possible to rip a DVD as an iTunes film? 
Tiger DVD Ripper
How can I rip my DVD to play them on my TV?
Is it possible to connect an external CD/DVD drive to an iOS device?