Improving frame/image quality on Handbrake?
Under the video tab, you'll see two options. Video Codec, which is how the video is encoded (h.264 is the standard for .mp4 files), and Quality, which controls the level of video compression that occurs.

It's the Quality slider that you're mostly worried about if you want to keep it an mp4. Drag that slider to the right to improve the quality (this will also increase the file size) or to the left to decrease the file size (this will also decrease the quality). Or if you know exactly the size of file you want, divide the desired file size by the number of seconds and you can input the kb/s that you want the video to be compressed to (smaller file sizes means higher compression and lower quality, higher file sizes equals less compression and higher quality).

In order to test to make sure that you like your quality selection and file size, you can choose a single chapter of the film (if your cdr has that stored) and run that through the encoder and test the quality.

Another thing that is available to choose is the video resolution, but in my experience with handbrake, it's quite temperamental, so I only recommend messing with that if you really need to.