Specification for Eye Mask Creation?
This is a very crude plan, but it should get you started.

Your typical mask will look like this:



Now we have to figure out how to define each of the three regions shown in red. For that, we need to get the SERDES datasheet and look up the numbers.

The bar on the top, as you already know, indicates the maximum allowed voltage. It means that, if the input signal exceeds this voltage, the chip could malfunction or get damaged. That voltage could be found in the datasheet section called "Absolute Maximum Ratings" or something like this.

The bar on the bottom could be defined similarly.

For the middle region, you can use a timing diagram and pretty much copy the elongated hexagon in the middle.

The top level of the hexagon should correspond to the minimum level for input voltage for logical 1, VIH in the datasheet. 

The bottom level should correspond to the maximum level of the input voltage for logical 0, VIL.

The slopes of the edges should not exceed the maximum values specified for tLH and tHL.

The width of the hexagon should not be less than the sum of the minimum setup and hold times, tsu and thold, assuming that the sampling clock edge will be optimally placed somewhere in the middle.

Again, this is a very crude and simplistic approach, but it should give you an idea.