Setting up video for Drupal- to play in popup
Video is hard. One thing to think about is that if you are going to use Vimeo then you don't need to worry about formats since Vimeo will handle that on upload (and the player, and the HTML5). My personal preference would be to use embedded media field, and the media:vimeo module.

Then you end up with the video as a cck field and you can use something like colorbox to give you your popup.