Change Encoding Settings through Python API
Set Output.file_format to FFMPG and adjust the attributes of Render.ffmpeg:

import bpy

scene = bpy.context.scene
rd = scene.render

# Set output type
rd.image_settings.file_format = "FFMPEG"

# Set output format
rd.ffmpeg.format = "MPEG4"

# Set the codec
rd.ffmpeg.codec = "H264"

...


Other options:


codec: MPEG1, MPEG2, MPEG4, HUFFYUV, DV, H264, THEORA, DNXHD, PNG
format: MPEG1, MPEG2, MPEG4, AVI, QUICKTIME, DV, H264, XVID, OGG, MKV
video_bitrate: 0 - infinity (integer)
maxrate: 0 - infinity (integer)
audio_channels: MONO, STEREO, SURROUND4, SURROUND51, SURROUND71
audio_bitrate: 32 - 384


For all available options, see the API on FFmpegSettings.