What Gerber file extension is used to specify PCIe edge connector plating?
I'm assuming you're after a thicker plating on the mating portion of the PCIE connector.

A gerber file, regardless of extension, is not capable of showing details like this.  A gerber file contains a sequence of commands that describe the two dimensional geometry of a single layer of a PCB.  The geometry is made up of all of the lines, arcs, and shapes, all with coordinates, that when taken together show all of the positive or negative space on the board.  Could be copper, solder mask, silkscreen, solder paste, etc.  Things like layer-to-layer spacing, hole plating, board material, plating thickness etc. are beyond the scope of a gerber file.

Furthermore (although less relevant and more pedantic) the use of extensions like .GTL to differentiate between layers is not part of the gerber standard.  Section 2.1 of the standard states that The standard extension is “.gbr” or “.GBR”.  Furthermore, section 6.2 of the specification "Most common bad practices" recommend against using non standard extensions, although this particular use is generally well understood by board houses.  Still, when you submit the gerbers you're generally required to specify what layer each gerber represents.  The board house will build the board based on your answer, not the extension of the individual gerber files.

Therefore:


It is not possible to represent this using a gerber file
Even if it were possible, you wouldn't use the file extension to do it.


Gerber files are allowed to contain comments, so you could write something in there like "Manufacture this layer with a 2oz plating" but even if you did, it's unlikely that another human being would ever read the comment.

Not very helpful so far, eh?

In order to get a special plating, you'll likely need to enlist the services of a full-service board house.  In addition to the gerber files, you'll deliver a drawing of the board (usually a PDF).  Somewhere in that drawing there will be text or pictures that tell the board house some additional information like:


Which layers of the board are represented by which gerber files
What order the layers go in in the board stack-up
The material that the board is made of
The thickness of the board
Any non-standard features of the board - In your case, you probably need to specify a different plating for the edge connector than the rest of the board.  This would probably be done by circling the area on the drawing, and writing some text specifying the plating.


The board house will have a real human being look at the drawing - possibly offering feedback depending on their capabilities and a quote to manufactuer the board.  Express shops are much more streamlined.  By standardizing the process and increasing the use of automation they are able to offer boards at lower prices than a full service shop.

There's a small possibility that the board house would want a gerber file with only the geometry to be special plated so that they could generate the correct mask.  In that case, you would include a gerber with only the PCIE contacts and specify to the board house what that particular file contains.