Setting format to H.264 in After Effects
I would not recommend trying to output your AE Comp using H.264. Unfortunately, regardless of the settings, you will get less than desirable results (typically terrible). 

The proven workflow / best practice in terms of output is:

Under your Output Module Settings:

Render out using 

Format:     Quicktime
Format Options Button -> Video Codec:    Animation 
Quality:    100
Options: You can select your Channels as to include or not include Alpha along with RGB. Including Alpha will increase file size. 
Audio Output: Check if needed. 

Now this will output your composition to a lossless QT Animation file, which yes, will be very large, especially if you include an alpha channel. But the output will be perfect in terms of what you are seeing in AE. 

NEXT...

Use AME (Adobe Media Encoder, or import the file into Premiere Pro and Sequence the comp), and render out using export/AME and H.264 using the MP4 option. 

NOTE:  There is absolutely no need and I don't recommend using the QT in premiere either. Use H.264 MP4. Again, H.264 and MOV simply are not friendly within the CS suite. 

However, after you have rendered out your H.264 MP4 at the desired bitrate and resolution; if you need the file to be QT compatible, or the client specifically wants an MOV file, simply change the extension from "FILE.mp4" to "FILE.mov". 

There is absolutely zero difference file structure wise, the MP4/MOV is simply the wrapper for the H.264 codec. 

This will give you perfect outputs every time. 

Hope this helps.