Quality loss in black&white conversion
Shouldn't it have thousands or millions of colors available?


There is a difference between how the compositor deals with imagery, versus the VSE. The majority of the VSE code paths use unique to the VSE code, and are designed for 8 bit per channel paths, or 256 discrete steps from 0.0 to 1.0. The compositor on the other hand, uses 32 bit float per channel, which is a decimal format and provides much larger granularity in between values.


  Have you any guess why it's sooo ugly when rendered in a video? 


The issue is quantisation.

When you apply your mask to an 8 bit image, which all motion pictures encoded in codecs are loaded as in the VSE, you are already constraining your work.

A vignette is a constriction of light, which is a simple multiplication operation. 8 bit imagery is encoded using integers, or whole numbers. This means that if your mask is trying to set a value between 24 and 25, it can't. It is forced to quantise the value to 24 or 25. So the first problem is working in the VSE which is hammering your effort with quantisation artifacts long before you get to encoding it into a video codec.


  What causes the loss of quality?


The second portion of the problem is a codec. Codecs are designed for a number of different needs. Many times, that need is to conserve bandwidth. This means trade offs at the quantisation level as well as compression artifacts. In this case, the motion picture signal is complexly scaled to less than 8 bits per channel, and then further compressed temporally and spatially. While it might not seem like much, that scaling, when placed on top of already sub-optimal 8 bit manipulations, results in a significant degradation in quality. Now stack compression on top of that and you can begin to see where the image begins to fall apart.

Worse still is that at the lower level values your demonstration has, the sRGB OETF is not terribly efficient at encoding. This means that there are very few code values allocated at the lower end, and they must be crafted well or they will exhibit posterization.

In the end, it is a four pronged destruction:


VSE 8 bit quantisation.
Codec signal scaling.
Codec compression.
Inefficiency of the sRGB OETF transfer function.


Solutions may include:


Use the compositor for all work.
Use lower compression levels in output encodes for codecs.
Avoid output / input encoding to file formats, especially codecs, while working on a project. If files must be saved, save individual frames as EXRs. Doing otherwise stacks repeated degradation on the work.