Why do I get different results for my H.264 encoded .mov videos depending on the file format + encoding settings i choose?
You appear to be confusing encoding systems with containers. AVI, MOV, etc. are containers. H.264, MPEG and many, many others are compression systems.

Blender's manual has some great reference info here: https://www.blender.org/manual/data_system/files/media/video_formats.html

VBR is better for most work since it allows the compresssor to alter the data rate (and therefore file size) according to the amount of detail. Rapidly changing shots require huge amounts of data whereas slowly changing ones require very little.

H.264 and MPEG-4 are very well supported and deliver excellent bang for your buck. (H.265 is better but requires serious processing power to encode.)

Another solution might be to render your video out in a low loss or (ideally) lossless format and then downconvert it in a dedicated system such as Handbrake (Www.handbrake.fr) which will give you presets or let you tinker with all manner of settings until you find something that fits your needs.