Modal Popup for Vimeo when clicking on Vimeo Thumbnail Link
You want to have a look at the video_embed_field module and the colorbox module.
They combine well to achieve what you want.

Create a contenttype with the new field provided by video_embed_field.
Then for the thumbs you could make a view and select the video_embed_field.
Go into settings of this video field and in the selectbox labeled Formatting tool, select thumbnails/w colorbox
This should give you a nice grid with thumbnails all linking to a colorbox overlay with player embedded.