Why do I get different results for my H.264 encoded .mov videos depending on the file format + encoding settings i choose? TL;DR? --> read the bold parts only ;)

The question could also have been called: What is the difference between the settings "file format" in "output" and "Format" in "Encoding"? It is very close to this question and I think I got the answer from there.

But I am still left with my actual problem, which is this: from the VSE I want to render out an h.264 encoded video in a .mov container. as far as I see there are two options to achieve this:

1) select H.264 or MPEG in "output"--"file format to save the rendered image as" and then in "Encoding" select "Quicktime" as "Format" and "H.264" as "Codec".
(--> this will give you the option to select a target bitrate for the output file (nice!))

2) select "Qicktime" in "output"--"file format to save the rendered image as" and then H.264 in the new option "Video Codec" that pops up right below.
(--> this will give you a setting for the "Quality percentage" (awkward!))

When I compare similarly sized output files generated by 1) and 2), method 1) looks waaaaay better! (example: 2000 kb/s in 1) gives 4MB file vs. 25% in 2) 7MB file and still file 1) looks considerably better!)

The only problem with 1) is that the resulting .mov file is not fully compatible with OS X whereas file 2) is.
(Also, selecting "lossless output" in 1) generates a file that cannot be played with QT player in OS X (VLC will play it))

Why are 1) and 2) different (is it the VBR for only 1)?) and what should I do to get a high-quality, low-size fully QT-compatible file rendered out?
You appear to be confusing encoding systems with containers. AVI, MOV, etc. are containers. H.264, MPEG and many, many others are compression systems.

Blender's manual has some great reference info here: https://www.blender.org/manual/data_system/files/media/video_formats.html

VBR is better for most work since it allows the compresssor to alter the data rate (and therefore file size) according to the amount of detail. Rapidly changing shots require huge amounts of data whereas slowly changing ones require very little.

H.264 and MPEG-4 are very well supported and deliver excellent bang for your buck. (H.265 is better but requires serious processing power to encode.)

Another solution might be to render your video out in a low loss or (ideally) lossless format and then downconvert it in a dedicated system such as Handbrake (Www.handbrake.fr) which will give you presets or let you tinker with all manner of settings until you find something that fits your needs.