What are the pros and cons of a few of Photoshop's video format export options?
You don't need to worry about most of those formats. Over the last decades, different formats have emerged for different purposes. Since most of those purposes still exist somewhere, software such as the Adobe suit still supports most of them, but for general online video purposes, you don't need most of them. For a normal video that is intended to be watched on computers, uploaded to the internet et c., MP4 or AVI videos with the H.264 codec is what you will want to use, as it is supported by virtually all applications and hardware. QuickTime Movie (usually used with .mov files, though the codec can be used in other container formats as well) is another option.


  "Keyframe every x seconds"


In the context of encoding an H.264 video, the term 'keyframe' refers to I-frames. For compression purposes, many codecs don't save each individual frame fully. Instead, they save one full frame (similar to one image file) every few seconds (or in irregular intervals) and for all the frames in between, only the difference between the last I-frame (keyframe) is saved, reducing filesize. The keyframe interval controls the frequency of the keyframes. Most encoders can also place them irregularly whenever there is a large difference between two adjacent frames (which usually means a scene change), which is what you want if it is an option and you don't have a special reason why you need to set the keyframe interval manully.


  "Limit data rate to x"


Probably refers to the bitrate (I never understood the tendency of different developers to give their settings special names, even though it's the same damn thing every time. Talk about confusing users). The bitrate is the amount of bits (i.e. information) per second. The higher the bitrate, the better the quality (non-linearly though, after a certain point there are diminishing returns in quality increasement). And the higher the bitrate, the larger the resulting file. 

Edit: As Mulvya pointed out, this setting probably refers to the maximum bitrate rather than the target bitrate. That is, the encoder will never exceed that average bitrate, it may however use a lower value if feasible. In this case, the quality slider (see next section) would control the target bitrate. 


  Most of them offer a depth and quality setting ranging from low to high or best - I always choose the best quality, but I'm not sure it's necessary. 


Quality sliders usually control the compression level, which in turn is related to the bitrate. Higher bitrate = higher quality = higher filesize. You will have to experiment with that to find a setting where you don't notice any artifacts in the rendered file while the filesize is still acceptable.


  For example I just exported this in AVI mode at 1080p, but have a weird flicker.


Looks to me like a framerate issue, but could also be caused by messing with the keyframe interval. As you said, try a different format, preferably MP4/H.264. In general, don't meddle with a setting if you don't know what it does. Setting an unusual keyframe interval for instance can yield unexpected results.




  but this is a complex animation and even with a powerful graphics card the rendering takes 30 minutes to an hour


I had to wait about three days for an animation rendering to finish once (blender using cycles engine though), just to put that into perspective °v°