What is metastability?
Quick Answer:  If you violate the setup and hold time on the input of a flip flop, then the output will be unpredictable for some amount of time.  That unpredictable output is called meta-stable (or metastability).  

Long answer:  When the output is unpredictable, I mean that it's unpredictable.  It could be high, it could be low, it could be somewhere in between, or it could oscillate.  After this metastable period the output will be high or low, but we don't know which way it'll go until it happens.

The amount of time that it's unpredictable is somewhat predictable, however.  There are two main factors that determine the length of the metastable period:  The speed of the flip-flop, and how "close to the edge" you got the timing.  

Most of the metastable times are quite short, although the probability of having a long time is non-zero.  Theoretically you could have a metastable time on the order of seconds, although the odds of that happening are incredibly rare.  As the speed of the flip-flop increases, the average metastable time decreases-- all other things being equal.

There is an "imaginary" time in the flip-flop, relative to the clock edge, where you're most susceptible to metastability issues.  Exactly when that is depends on lots of factors like temperature, voltage, process, phase of the moon, animal sacrifices, and what political party you affiliate with.  Whenever that time is, the closer your data input edge is to that time the longer the metastability time will be.  

The best way to deal with metastability is to make all of your logic synchronous, and not violate any of your setup and hold times.  This is, of course, difficult to impossible for circuits of any complexity.  So what we do is try to limit the places where metastability could be an issue and then deal with those places.  

The normal method would be to "double-clock" the data.  Meaning, have two D Flip-Flops in series with the output of the first feeding the input of the second.  The hope is that if the first flip-flop goes metastable then the metastable period would be over before it violates the setup/hold time of the second.  In practice this works fairly well.  In super critical applications there might be some "triple-clocking" going on.